OpenSeeFace Documentation

OpenSeeFace provides Unity components for receiving and visualizing tracking data:

• OpenSee: Receives UDP packets and exposes data via the trackingData public field. Important: Because UDP packets are received in a separate thread, you should always copy the trackingData field to a local variable before processing to avoid data being overwritten mid-frame.
• OpenSeeShowPoints: Visualizes the detected facial landmark points. Useful for debugging.
• OpenSeeIKTarget: Used with IK solutions (like FinalIK) to animate head motion.
• OpenSeeLauncher: Allows starting the tracker directly from Unity (designed for pyinstaller executables).

If you are using poetry for dependency management, you can install all required dependencies for the project into a separate virtual environment using the following command.

poetry install

To use OpenSeeFace in a Python environment, you need to install the following dependencies: onnxruntime, opencv-python, pillow, and numpy. You can install them all at once using pip.

pip install onnxruntime opencv-python pillow numpy

The face tracking is performed by the facetracker.py script (Python 3.7+) or the facetracker.exe provided in the Binary folder of release builds. The tracker processes webcam input or video files and sends tracking data over UDP, allowing for remote tracking on a separate PC.

If you installed dependencies using poetry, prefix your commands with poetry run or execute them within a poetry shell.

python facetracker.py --help

# Example: Tracking a video file with visualization
python facetracker.py --visualize 3 --pnp-points 1 --max-threads 4 -c video.mp4

The OpenSeeExpression component (used alongside OpenSeeFace) detects specific facial expressions based on user-specific calibration.

Calibration Process:

1. Enter a name for the expression.
2. Perform the expression and hold it, then tick the recording box.
3. While holding the expression, move your head in various directions and (if applicable) talk.
4. Untick the recording box.
5. Repeat for other expressions.
6. Tick the train box to generate the detection model.
7. Check statistics in the component to verify accuracy.

Management:

• Delete data: Enter the expression name and tick the Clear box.
• Save/Load: Enter a full file path in the Filename field and tick Save or Load.

The OpenSeeLauncher component provides the following public API for controlling the tracker from within Unity:

• public string[] ListCameras(): Returns an array of available camera names. The index in this array corresponds to the cameraIndex field. Setting cameraIndex to -1 disables webcam capturing.
• public bool StartTracker(): Starts the tracker. If a tracker is already running, it shuts down the current instance and starts a new one with the current settings.
• public void StopTracker(): Stops the tracker. The tracker is also automatically stopped when the application terminates or the object is destroyed.

To pass custom arguments, add them individually to the commandlineArguments array. For example, to pass -v 1, add "-v" and "1" as separate elements.

OpenSeeFace includes four pretrained models with different trade-offs between speed and tracking quality. Use the --model flag to select one:

• -1: Extremely fast, very low accuracy (intended for low-power devices).
• 0: Very fast, low accuracy.
• 1: Moderate speed, better accuracy.
• 2: Slower, good accuracy.
• 3 (default): Slowest, highest accuracy.