DOCSEARCHbyAlgolia
Submit Docs

mediapipe-touchdesigner

repository·main·Indexed 25 days ago

https://github.com/torinmb/mediapipe-touchdesigner

A GPU-accelerated MediaPipe plugin for TouchDesigner (v0.5.2) that enables face, hand, pose, and object tracking. It utilizes a self-contained Chromium-based architecture to run vision tasks via JavaScript/WebAssembly, removing the need for external MediaPipe library installations on Mac and PC. The plugin includes the MediaPipe.tox component, real-time performance monitoring via CHOPs, and support for Spout (Windows) and Syphon (Mac) for video input.

Tokens
6.7K
Snippets
8
Records
43
Agent score
83%

What's inside mediapipe-touchdesigner

  1. Understand the MediaPipe Plugin architecture

    main

    The plugin runs MediaPipe vision tasks via a web browser to leverage GPU acceleration through JavaScript/WebAssembly without requiring local library installations. It uses three core layers:

    1. Web (and websocket) server: An embedded server that hosts web pages and acts as a WebSocket server for two-way communication between the browser and TouchDesigner.
    2. Web browser: An embedded Chromium instance within TouchDesigner that runs the MediaPipe detection components and renders the video stream.
    3. JSON decoders: The browser sends detection data to TouchDesigner in JSON format. Specialized .tox components then decode this JSON into usable TouchDesigner data (CHOPs, DATs, etc.).

  2. Create a release zip file using build_release COMP

    main

    The project includes a build_release COMP at the top level of the .toe file to automate the release process.

    Steps to create a release:

    1. Open MediaPipe TouchDesigner.toe.
    2. Navigate to your desired initial layout.
    3. Press Ctrl + Alt + B to trigger the build. Note: The project will lock up during this process.

    Automated Build Actions:

    • Recreates the release folder.
    • Runs yarn install and yarn build.
    • Loads _mpdist files into the MediaPipe COMP Virtual File System.
    • Removes external script references on DATs.
    • Exports all .tox files to release/toxes.
    • Saves the .toe file to the release folder.
    • Zips the release folder into release.zip.

    If errors occur, check the TouchDesigner text console to diagnose the issue.

  3. Install the MediaPipe TouchDesigner Plugin

    main

    The plugin is a GPU-accelerated, self-contained tool for Mac and PC that requires no local installation of MediaPipe libraries.

    Quick Setup

    1. Download the latest release.zip from the GitHub Releases section.
    2. Open the MediaPipe TouchDesigner.toe file.
    3. All components are located in the /toxes folder. The primary component is MediaPipe.tox.
    4. Important: When dragging the MediaPipe component into a new project, ensure you select "Enable External .tox" to prevent the .toe file size from becoming excessively large.

    Configuration

    Once MediaPipe.tox is loaded:

    • Select your webcam from the drop-down menu.
    • Toggle specific MediaPipe models on or off.
    • Enable/disable preview overlays.
    • Use sub-menus for model-specific customization.

  5. Use Spout to send TOPs from TouchDesigner (Windows)

    main

    On Windows, you can send any TOP to MediaPipe with minimal delay using Spout and SpoutCam.

    1. Setup SpoutCam

    • Download SpoutCam.
    • Run SpoutCam Settings.exe (no installer required).
    • Set the frame rate and resolution to match your TouchDesigner source.
    • In the Starting Sender box, enter: TDSyphonSpoutOut.
    • Click Register to create the virtual webcam.

    2. Setup TouchDesigner

    • Drop a Syphon Spout Out TOP into your project.
    • Connect your desired TOP to the input.

    3. Connect to MediaPipe

    • In the MediaPipe.tox component, select SpoutCam as your webcam source.

    Troubleshooting Spout Noise

    If Spout shows only noise, it is often due to multiple GPUs (e.g., integrated CPU graphics vs. dedicated GPU). Ensure both the sender and receiver processes are set to use the same graphics pipeline in Windows graphics settings.

  7. Debug the MediaPipe web page within TouchDesigner

    main

    To inspect the console logs and debug the embedded Chromium instance used by the MediaPipe component, you can connect a standard Chrome browser to the instance running inside TouchDesigner.

    1. Load the MediaPipe project within TouchDesigner.
    2. Open a Chrome browser on your desktop.
    3. Navigate to http://localhost:9222.

    Note: This method has a performance impact because the page is rendered twice (once in TouchDesigner and once in Chrome).

  9. Live debug a development page with yarn dev

    main

    You can use a development server to enable hot-reloading and live debugging in a separate Chrome tab while maintaining WebSocket communication with TouchDesigner.

    Workflow:

    1. Run yarn dev via the command line or VSCode console. This starts a server on port 5173.
    2. Load the TouchDesigner project and configure your settings.
    3. Enter the MediaPipe COMP and disable the webbrowser component (click the X).
    4. Copy the string from the current_url DAT.
    5. Paste that URL into a new Chrome tab.
    6. Edit the URL in Chrome to replace the original port number with 5173 (e.g., change localhost:3001 to localhost:5173).
    7. Press F12 in Chrome to open Developer Tools.

    Important: Parameter changes made in TouchDesigner will not automatically reflect in the Chrome tab unless you repeat the process of copying the updated current_url.

    yarn dev

  10. Configure hand gesture recognition settings via gestureState

    main

    The gestureState object allows you to configure the parameters for the MediaPipe Gesture Recognizer. These settings are used when initializing the landmarker via createGestureLandmarker.

    Key configuration properties include:

    • modelPath: Path to the .task model file.
    • numHands: Number of hands to detect.
    • minDetectionConfidence: Minimum confidence for hand detection.
    • minPresenceConfidence: Minimum confidence for hand presence.
    • minTrackingConfidence: Minimum confidence for hand tracking.
    • maxResults: Maximum number of gestures to return.
    • scoreThreshold: Minimum score for a gesture to be considered valid.
    • draw: A callback function for visualizing results on a canvas.
    export let gestureState = {
    modelTypes: {
        'full': './mediapipe/models/gesture_recognition/gesture_recognizer.task',
    },
    modelPath: './mediapipe/models/gesture_recognition/gesture_recognizer.task',
    detect: true,
    landmarker: undefined,
    results: undefined,
    resultsName: "gestureResults",
    numHands: 2,
    minDetectionConfidence: 0.5,
    minPresenceConfidence: 0.5,
    minTrackingConfidence: 0.5,
    maxResults: -1,
    scoreThreshold: 0.5,
    draw: (state, canvas) => drawHandGestures(state, canvas),
};

  11. Configure Pose Tracking settings via poseState

    main

    The poseState object controls the behavior and configuration of the pose landmarker. You can modify these properties to change the model being used, detection sensitivity, and the number of poses to track.

    Key configuration properties include:

    • modelPath: The path to the .task model file. Available types are 'lite', 'full', and 'heavy'.
    • numPoses: The number of poses to detect.
    • minDetectionConfidence: Minimum confidence for the initial detection.
    • minPresenceConfidence: Minimum confidence for the presence of landmarks.
    • minTrackingConfidence: Minimum confidence for tracking landmarks.
    • detect: Boolean to enable or disable detection.
    • draw: A function used for visualization.

  12. Configure face landmark detection via faceLandmarkState

    main

    The faceLandmarkState object allows you to configure the behavior and parameters of the face landmark detection process. These settings are used when initializing the landmarker via createFaceLandmarker.

    Key configuration properties include:

    • modelPath: Path to the .task model file.
    • numFaces: Number of faces to detect.
    • minDetectionConfidence: Minimum confidence score for detection.
    • minPresenceConfidence: Minimum confidence score for presence.
    • minTrackingConfidence: Minimum confidence score for tracking.
    • outputBlendshapes: Boolean to enable/disable facial blendshape output.
    • outputTransformationMatrixes: Boolean to enable/disable facial transformation matrix output.