MediaPipe Unity Plugin

The RunningMode determines how the Task processes data and which execution method you should call:

• RunningMode.IMAGE: For single still images. Use Detect().
• RunningMode.VIDEO: For continuous video frames. Use DetectForVideo().
• RunningMode.LIVE_STREAM: For camera input where you want to avoid blocking the main thread. Results are received asynchronously via a callback. Use DetectAsync().

While using the Task API is generally recommended, you can still run a custom CalculatorGraph for legacy solutions (like Face Mesh Legacy). This involves initializing a CalculatorGraph with a configuration text (typically from a .pbtxt file) and starting its execution.

Warning: On Windows, certain code patterns used in custom graph implementations may cause UnityEditor to crash. Refer to the project's technical limitations for details.

TextAsset configAsset;

var graph = new CalculatorGraph(configAsset.text);
graph.StartRun();

To avoid copying input images from CPU to GPU (especially on Android with OpenGL ES), you can share the OpenGL context between MediaPipe and Unity. Use the GpuManager helper to initialize and retrieve GpuResources.

// ATTENTION!: It will fail if the Graphics API is set to OpenGL Core.
yield return GpuManager.Initialize();

using var gpuResources = GpuManager.IsInitialized ? GpuManager.GpuResources : null;
using var faceLandmarker = FaceLandmarker.CreateFromOptions(options, gpuResources);

By default, the plugin outputs logs at the INFO level. You can increase the verbosity of the plugin's internal logs by setting Mediapipe.Unity.Logger.MinLogLevel to a more granular level like Debug.

Mediapipe.Unity.Logger.MinLogLevel = Logger.LogLevel.Debug;

Building with Docker Windows Containers is recommended. Note: Hyper-V backend is required; Windows 10/11 Home is not supported.

1. Install Docker Desktop and switch to Windows Containers.
2. Build the image:

   docker build -t mediapipe_unity:windows . -f docker/windows/x86_64/Dockerfile

3. Run the container, mounting Packages and Assets:

   docker run --cpus=16 --memory=32g \
       --mount type=bind,src=%CD%\Packages,dst=C:\mediapipe\Packages \
       --mount type=bind,src=%CD%\Assets,dst=C:\mediapipe\Assets \
       -it mediapipe_unity:windows

4. Run the build script inside the container.

Alternative: You can use a Docker Linux Container on Windows to build libraries for Android only.

To use the Task API, you must generate a Task instance by providing configuration options. At a minimum, you need to specify the model asset and the execution delegate within BaseOptions. You should also set the RunningMode to match your use case (e.g., IMAGE or VIDEO).

Note: The default RunningMode is RunningMode.IMAGE.

using Mediapipe.Tasks.Vision.FaceLandmarker;

TextAsset modelAsset;

var options = new FaceLandmarkerOptions(
  baseOptions: new Tasks.Core.BaseOptions(
    Tasks.Core.BaseOptions.Delegate.CPU,
    modelAssetBuffer: modelAsset.bytes
  ),
  runningMode: Tasks.Vision.Core.RunningMode.VIDEO
);

using var faceLandmarker = FaceLandmarker.CreateFromOptions(options);

Building with Docker is the recommended method for Linux to ensure environment consistency and avoid GLibc version conflicts.

Note: The target machine must have GLibc version >= 2.31.

1. Install Docker and ensure you can run docker without sudo.
2. Build the Docker image:

   docker build --build-arg UID=$(id -u) -t mediapipe_unity:latest . -f docker/linux/x86_64/Dockerfile

3. Start the container, mounting the Packages and Assets directories so files are saved to your host:

   docker run \
       --mount type=bind,src=$PWD/Packages,dst=/home/mediapipe/Packages \
       --mount type=bind,src=$PWD/Assets,dst=/home/mediapipe/Assets \
       -it mediapipe_unity:latest

4. Run the build script inside the container:

   python build.py build --desktop gpu --opencv cmake --android arm64 -v

To enable glog flags, you must call Glog.Initialize when the application starts.

Warning: Glog.Initialize can only be called once. Calling it a second time will cause your application or the Unity Editor to crash.

To ensure logs are visible in Unity's log files (Editor.log or Player.log), you must set Glog.Logtostderr = true before or during initialization.

Glog.Initialize("MediaPipeUnityPlugin");

The Task API requires an Image instance as input. If you are using Unity's WebCamTexture, you must copy the pixel data to a Texture2D and then wrap it in an Image object.

Warning: Using SetPixels32 is slow because it reads pixel data on the CPU. For better performance, especially on Android, consider using Share OpenGL Context or the Experimental.TextureFrame approach to avoid unnecessary CPU-GPU data copying.

var tmpTexture = new Texture2D(webCamTexture.width, webCamTexture.height, TextureFormat.RGBA32, false);
tmpTexture.SetPixels32(webCamTexture.GetPixels32());
tmpTexture.Apply();
using var image = new Image(tmpTexture);

When building for Android, you must include libstdc++_shared.so in your APK to avoid DllNotFoundException. This is because mediapipe_android.aar contains libopencv_java4.so, which depends on it.

Option 1: Manual Placement Place libstdc++_shared.so directly in the Assets/Plugins/Android directory of your Unity project.

Option 2: Gradle Automation (Recommended) Add a task to your mainTemplate.gradle to automatically copy the library from your NDK. Use the appropriate snippet based on your NDK version.

MediaPipe requires timestamps to be provided in microseconds. If you provide values in milliseconds or other units, calculators that rely on absolute timestamp values will behave incorrectly.

A reliable way to generate timestamps in Unity is to use System.Diagnostics.Stopwatch to track elapsed time since startup.

using Stopwatch = System.Diagnostics.Stopwatch;

var stopwatch = new Stopwatch();
stopwatch.Start();

// Inside your loop:
var currentTimestamp = stopwatch.ElapsedTicks / ((double)System.TimeSpan.TicksPerMillisecond / 1000);
graph.AddPacketToInputStream("input_video", Packet.CreateImageFrameAt(imageFrame, (long)currentTimestamp));

To build locally on macOS, follow these steps:

1. Install Homebrew.
2. Install Python (version >= 3.9.0, < 3.13.0) and NumPy:

   brew install python
   export PATH=$PATH:"$(brew --prefix)/opt/python/libexec/bin"
   pip3 install --user six numpy

3. Install Bazelisk and NuGet:

   brew install bazelisk
   brew install nuget

4. Install Xcode (version >= 16.0) and Command Line Tools:

   sudo xcodebuild -license
   sudo xcode-select -s /Applications/Xcode.app
   xcode-select --install

5. Run the build script (python build.py ...).

brew install python
export PATH=$PATH:"$(brew --prefix)/opt/python/libexec/bin"
pip3 install --user six numpy
brew install bazelisk
brew install nuget
sudo xcode-select --install