# Plan for Converting CameraManager.cs to Use OpenCvSharp Mat

## Overview

This document outlines the plan for converting the CameraManager.cs and related classes to use OpenCvSharp's Mat class throughout the codebase, eliminating the conversion to Godot Image.

## Changes Required

### 1. CameraManager.cs Changes

- Remove Godot Image-related fields and methods:
  - Remove `_lastFrame` and `_feedTexture` fields
  - Remove `MatToImage` conversion method
  - Remove `GetCameraTexture()` method
  
- Update method signatures:
  - Change `GetCameraFrame()` to return `Mat` instead of `Image`
  
- Update implementation:
  - Remove any code that converts between Mat and Image
  - Ensure proper Mat cloning when returning frames to prevent thread safety issues
  - Update resource cleanup to properly dispose of Mat objects

### 2. AcquisitionManager.cs Changes

- Update method signatures and implementations:
  - Modify `ProcessFrame()` to work with `Mat` instead of `Image`
  - Update any code that processes camera frames to work with Mat

- Update DataRecorder:
  - Modify `RecordFrame()` to accept Mat instead of Image
  - Update frame saving logic to use OpenCvSharp's `Cv2.ImWrite()` instead of Godot's Image saving

### 3. Visualizer Class Changes

- Update method signatures:
  - Change `UpdateVisualization()` to accept `Mat` instead of `Image`
  - Change `DrawEyePositions()` to work with `Mat`
  
- Replace Godot drawing with OpenCV drawing:
  - Replace `DrawCircle()` with OpenCV's `Cv2.Circle()`
  - Replace `SetPixelSafe()` with direct pixel access or OpenCV drawing functions
  
- Update visualization logic:
  - Modify any code that assumes Godot Image properties to use Mat properties instead

## Detailed Implementation Steps

### 1. CameraManager.cs Implementation

```csharp
// Key changes to CameraManager.cs
public partial class CameraManager : Node
{
	// Remove these fields
	// private Image _lastFrame;
	// private ImageTexture _feedTexture;
	
	// Keep or add these fields
	private VideoCapture _videoCapture;
	private bool _isInitialized = false;
	private Thread _captureThread;
	private bool _isCapturing = false;
	private Mat _currentFrame;
	private readonly object _frameLock = new object();
	
	// Update _Ready() method
	public override void _Ready()
	{
		GD.Print("CameraManager: Ready");
		// Remove: _feedTexture = new ImageTexture();
	}
	
	// Update GetCameraFrame() method
	public Mat GetCameraFrame()
	{
		if (!_isInitialized || _videoCapture == null || !_videoCapture.IsOpened())
		{
			GD.PrintErr("Cannot get frame: Camera not initialized");
			return null;
		}
		
		try
		{
			lock (_frameLock)
			{
				if (_currentFrame.Empty())
				{
					GD.PrintErr("Current frame is empty");
					return null;
				}
				
				// Return a clone of the current frame to prevent thread safety issues
				return _currentFrame.Clone();
			}
		}
		catch (Exception ex)
		{
			GD.PrintErr($"Error getting camera frame: {ex.Message}");
			GD.PrintErr(ex.StackTrace);
			return null;
		}
	}
	
	// Remove MatToImage method entirely
	
	// Remove GetCameraTexture method entirely
}
```

### 2. Visualizer Class Implementation

```csharp
public class Visualizer
{
	// Update UpdateVisualization method
	public void UpdateVisualization(int frameCount, Mat frame, EyeData eyeData)
	{
		if (!_initialized)
		{
			GD.PrintErr("Visualizer not initialized. Call InitRerun() first.");
			return;
		}
		
		// Draw eye positions on the frame for visualization
		DrawEyePositions(frame, eyeData);
		
		// In a real implementation, this would update a visualization window or stream
		GD.Print($"Visualizing frame {frameCount} with eye positions: " +
				 $"L({eyeData.LeftPos.X:F2}, {eyeData.LeftPos.Y:F2}), " +
				 $"R({eyeData.RightPos.X:F2}, {eyeData.RightPos.Y:F2})");
	}
	
	// Update DrawEyePositions method
	private void DrawEyePositions(Mat frame, EyeData eyeData)
	{
		// Convert normalized coordinates [0,1] to pixel coordinates
		int leftX = (int)(eyeData.LeftPos.X * frame.Width);
		int leftY = (int)(eyeData.LeftPos.Y * frame.Height);
		int rightX = (int)(eyeData.RightPos.X * frame.Width);
		int rightY = (int)(eyeData.RightPos.Y * frame.Height);
		
		// Draw circles at eye positions using OpenCV
		Cv2.Circle(frame, new Point(leftX, leftY), 10, new Scalar(0, 0, 255), -1); // Red circle for left eye
		Cv2.Circle(frame, new Point(rightX, rightY), 10, new Scalar(255, 0, 0), -1); // Blue circle for right eye
	}
	
	// Remove DrawCircle and SetPixelSafe methods as they're replaced by OpenCV functions
}
```

### 3. AcquisitionManager.cs Implementation

```csharp
public partial class AcquisitionManager : Node3D
{
	// Update ProcessFrame method
	private bool ProcessFrame(CancellationToken cancellationToken)
	{
		float loopStart = (float)Time.GetTicksMsec() / 1000.0f;
		
		try
		{
			// Check for cancellation
			if (cancellationToken.IsCancellationRequested)
				return false;
				
			// Capture frame
			Mat frame = _cameraManager.GetCameraFrame();
			if (frame == null || frame.Empty())
			{
				GD.PrintErr("Failed to capture frame");
				return true; // Continue trying
			}
			
			float frameTimestamp = (float)Time.GetTicksMsec() / 1000.0f;
			_frameCount++;
			
			// Get latest eye data
			EyeData eyeData = GetEyeTrackingData();
			if (eyeData != null)
			{
				_lastEyeData = eyeData;
				GD.Print($"Updated eye data: L({_lastEyeData.LeftPos.X:F2}, {_lastEyeData.LeftPos.Y:F2}), " +
						 $"R({_lastEyeData.RightPos.X:F2}, {_lastEyeData.RightPos.Y:F2})");
			}
			
			// Process synchronized data
			if (_lastEyeData != null && _eyeProcessor.ValidateEyeData(_lastEyeData, frameTimestamp))
			{
				GD.Print($"Synchronized frame {_frameCount} at {frameTimestamp:F3} " +
						 $"with eye data at {_lastEyeData.Timestamp:F3}");
				
				// Record data
				_dataRecorder.RecordFrame(frame, _frameCount);
				_dataRecorder.RecordEyeData(_frameCount, frameTimestamp, _lastEyeData);
				
				// Update visualization
				_visualizer.UpdateVisualization(_frameCount, frame, _lastEyeData);
			}
			else
			{
				GD.Print($"No fresh eye data for frame {_frameCount} at {frameTimestamp:F3}");
			}
			
			// Maintain frame rate
			float elapsed = ((float)Time.GetTicksMsec() / 1000.0f) - loopStart;
			int sleepMs = (int)Math.Max(0, (_frameInterval - elapsed) * 1000);
			
			if (sleepMs > 0)
			{
				Thread.Sleep(sleepMs);
			}
			
			return true;
		}
		catch (Exception ex)
		{
			GD.PrintErr($"Error in acquisition loop: {ex.Message}");
			GD.PrintErr(ex.StackTrace);
			return false;
		}
	}
}
```

### 4. DataRecorder Implementation

```csharp
public class DataRecorder
{
	// Update RecordFrame method
	public void RecordFrame(Mat frame, int frameNumber)
	{
		try
		{
			// Create directory if it doesn't exist
			string directory = "recorded_data/frames";
			if (!System.IO.Directory.Exists(directory))
			{
				System.IO.Directory.CreateDirectory(directory);
			}
			
			// Save the frame using OpenCV
			string filename = $"{directory}/frame_{frameNumber:D6}.jpg";
			Cv2.ImWrite(filename, frame);
			
			GD.Print($"Recorded frame {frameNumber} to {filename}");
		}
		catch (Exception ex)
		{
			GD.PrintErr($"Error recording frame: {ex.Message}");
		}
	}
}
```

## Implementation Steps

1. **CameraManager.cs**:
   - Update the class to use Mat throughout
   - Remove Image conversion code
   - Ensure proper resource management

2. **Visualizer Class**:
   - Replace Godot drawing functions with OpenCV equivalents
   - Update visualization logic to work with Mat

3. **AcquisitionManager.cs**:
   - Update to work with Mat objects from CameraManager
   - Modify frame processing logic

4. **DataRecorder**:
   - Update to save Mat objects directly

## Testing Strategy

1. Test CameraManager in isolation:
   - Verify camera initialization
   - Verify frame capture
   - Check resource cleanup

2. Test integration with AcquisitionManager:
   - Verify frame processing
   - Check eye tracking data integration

3. Test visualization:
   - Verify drawing functions work correctly with Mat

## Benefits

1. **Performance**: Eliminating conversions between Mat and Image will improve performance
2. **Functionality**: Direct access to OpenCV's rich feature set
3. **Consistency**: Using Mat throughout the codebase will make it more consistent
4. **Simplicity**: Reduced code complexity by removing conversion logic

## Potential Challenges

1. **Thread Safety**: Ensuring thread-safe access to Mat objects
2. **Memory Management**: Proper disposal of Mat objects to avoid memory leaks
3. **API Differences**: Adapting code that expects Godot Image properties to use Mat properties

## Next Steps

1. Switch to Code mode to implement these changes
2. Start with CameraManager.cs as it's the core component
3. Then update the Visualizer class
4. Finally, update AcquisitionManager.cs and DataRecorder
5. Test the changes thoroughly to ensure everything works as expected