DOCSEARCHbyAlgolia
Submit Docs

MediaPipe

repository·master·Indexed 12 days ago

https://github.com/google-ai-edge/mediapipe

An open-source framework for building efficient, on-device machine learning pipelines. It provides high-level 'Solutions' (Tasks and pre-trained models) for rapid deployment and a low-level 'Framework' for building custom perception pipelines across mobile, web, and edge devices, including support for Edge TPU and cross-compilation for ARM32 and ARM64.

Tokens
106.1K
Snippets
194
Records
353
Agent score
99%

What's inside MediaPipe

  1. Overview of MediaPipe Instant Motion Tracking

    master

    MediaPipe Instant Motion Tracking is a legacy Augmented Reality (AR) solution designed to provide AR tracking across devices and platforms without the need for initialization or calibration. Built upon the MediaPipe Box Tracking solution, it allows developers to place virtual 2D and 3D content on static or moving surfaces that interact seamlessly with the real-world environment.

    Note: Support for this MediaPipe Legacy Solution ended on March 1, 2023. Developers are encouraged to migrate to the modern MediaPipe Solutions.

  2. Overview of MediaPipe Pose

    master

    MediaPipe Pose is a machine learning solution designed for high-fidelity body pose tracking. It infers 33 3D landmarks and a background segmentation mask (distinguishing human from background) from RGB video frames.

    Key features include:

    • Real-time performance: Optimized for mobile phones, desktops, laptops, web, and Python environments.
    • High fidelity: Uses BlazePose research to provide accurate body tracking.
    • Applications: Suitable for physical exercise quantification, sign language recognition, full-body gesture control, and augmented reality overlays.

  3. Overview of MediaPipe Face Detection

    master

    MediaPipe Face Detection is a high-performance, ultrafast solution designed for detecting faces in real-time. It supports multi-face detection and provides 6 facial landmarks per face.

    Key technical details:

    • Model Architecture: Based on BlazeFace, a lightweight detector optimized for mobile GPU inference.
    • Performance: Designed for super-realtime performance, making it suitable for live viewfinder experiences.
    • Use Cases: It serves as a foundational step for other tasks, such as 3D facial keypoint estimation (Face Mesh), facial feature/expression classification, and face region segmentation.
    • Technical Foundation: Uses a lightweight feature extraction network (inspired by MobileNetV1/V2), a GPU-friendly anchor scheme (modified from SSD), and an improved tie resolution strategy instead of non-maximum suppression.

  4. Overview of MediaPipe Holistic

    master

    MediaPipe Holistic is a legacy solution designed for the simultaneous, real-time perception of human pose, face landmarks, and hand tracking. It is optimized for mobile devices and enables applications such as fitness/sport analysis, gesture control, sign language recognition, and augmented reality.

    Note: As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe Solution. For current implementations, refer to the MediaPipe Solutions site.

  5. Overview of MediaPipe Iris

    master

    MediaPipe Iris is a machine learning solution for real-time iris estimation using a single RGB camera. It tracks landmarks for the iris, pupil, and eye contours without requiring specialized hardware.

    Key capabilities include:

    • Iris Tracking: Accurate estimation of eye landmarks (eyelid) and iris landmarks.
    • Depth Estimation: Determines the metric distance between the subject and the camera with a relative error of less than 10%, based on the constant horizontal diameter of the human iris.
    • Cross-platform: Runs on mobile (Android, iOS), desktop (CPU/GPU), and web.

    Note: This solution was upgraded to a new MediaPipe Solution as of May 10, 2023. For current implementations, refer to the MediaPipe Solutions site.

  6. Overview of MediaPipe Selfie Segmentation

    master

    MediaPipe Selfie Segmentation is a solution designed to segment prominent humans in a scene. It is optimized for real-time performance on smartphones and laptops, making it ideal for selfie effects and video conferencing where the subject is within approximately 2 meters of the camera.

    Available Models

    MediaPipe provides two optimized models based on MobileNetV3:

    • General Model (model_selection=0): Operates on a 256x256x3 (HWC) tensor. This model powers ML Kit.
    • Landscape Model (model_selection=1): Operates on a 144x256x3 (HWC) tensor. It has fewer FLOPs and runs faster, making it suitable for landscape-oriented video (e.g., Google Meet).

  7. Overview of MediaPipe Modules

    master

    MediaPipe Modules are organized as subfolders, each providing specialized subgraphs and necessary resources (such as TFLite models) to perform specific domain tasks. These modules are used to build complex MediaPipe graphs for tasks like face, hand, and pose detection/tracking.

    Available modules include:

    • face_detection: Subgraphs for detecting faces.
    • face_geometry: Subgraphs for extracting face geometry.
    • face_landmark: Subgraphs for detecting and tracking face landmarks.
    • hand_landmark: Subgraphs for detecting and tracking hand landmarks.
    • holistic_landmark: Subgraphs for detecting and tracking holistic pose (combining pose, face, and hand landmarks).
    • iris_landmark: Subgraphs for detecting iris landmarks.
    • palm_detection: Subgraphs for detecting palms/hands.
    • pose_detection: Subgraphs for detecting poses.
    • pose_landmark: Subgraphs for detecting and tracking pose landmarks.
    • objectron: Subgraphs for detecting and tracking 3D objects.

  8. Overview of MediaPipe Hands

    master

    MediaPipe Hands is a high-fidelity hand and finger tracking solution that uses machine learning to infer 21 3D landmarks of a hand from a single frame. It is designed for real-time performance on mobile devices and can scale to multiple hands. This solution is useful for sign language understanding, hand gesture control, and augmented reality overlays.

    Key Features:

    • 21 3D Landmarks: Provides precise 3D coordinates for hand knuckles and fingers.
    • Real-time Performance: Optimized for mobile hardware.
    • Multi-hand Support: Capable of tracking multiple hands simultaneously.

  9. Overview of MediaPipe Face Mesh

    master

    MediaPipe Face Mesh is a real-time solution that estimates 468 3D face landmarks from a single camera input without requiring a depth sensor. It is designed for high performance on mobile devices using lightweight model architectures and GPU acceleration.

    Key features include:

    • 3D Facial Surface Estimation: Infers the 3D surface using machine learning.
    • Face Transform Module: A bundled module that bridges landmark estimation and Augmented Reality (AR) applications. It establishes a metric 3D space and provides face transform data, including a face pose transformation matrix and a triangular face mesh, using Procrustes Analysis on the CPU.

  10. Available Iris Landmark Subgraphs

    master

    The iris_landmark module provides several subgraphs for detecting iris landmarks, categorized by whether they detect a single eye or both eyes, and whether they execute on the CPU or GPU.

    Choose a subgraph based on your hardware target and detection requirements:

    • Single Eye Detection: Use IrisLandmarkCpu or IrisLandmarkGpu to detect landmarks for either the left or right eye.
    • Dual Eye Detection: Use IrisLandmarkLeftAndRightCpu or IrisLandmarkLeftAndRightGpu to detect landmarks for both eyes simultaneously.
    • Hardware Target: Select the Cpu variant for CPU-based input and inference, or the Gpu variant for GPU-based input and inference.
    | Subgraph | Details |
| :--- | :--- |
| `IrisLandmarkCpu` | Detects iris landmarks for left or right eye. (CPU input, and inference is executed on CPU.) |
| `IrisLandmarkGpu` | Detects iris landmarks for left or right eye. (GPU input, and inference is executed on GPU) |
| `IrisLandmarkLeftAndRightCpu` | Detects iris landmarks for both left and right eyes. (CPU input, and inference is executed on CPU) |
| `IrisLandmarkLeftAndRightGpu` | Detects iris landmarks for both left and right eyes. (GPU input, and inference is executed on GPU.) |

  11. MediaPipe Objectron Overview

    master

    MediaPipe Objectron is a legacy mobile real-time 3D object detection solution designed for everyday objects. It detects objects in 2D images and estimates their 3D poses (size, position, and orientation) using machine learning models trained on the Objectron dataset.

    Note: Support for this MediaPipe Legacy Solution ended on March 1, 2023. Users are encouraged to migrate to the current MediaPipe Solutions site.

  12. Use Selfie Segmentation subgraphs

    master

    MediaPipe provides two specialized subgraphs for selfie segmentation, which separates a person from the background in selfie-style images. Choose the subgraph based on your hardware requirements and input type:

    • SelfieSegmentationCpu: Use this when your input is on the CPU and you want to execute inference on the CPU.
    • SelfieSegmentationGpu: Use this when your input is on the GPU and you want to execute inference on the GPU.

    These subgraphs are defined via .pbtxt configuration files.

    | Subgraph | Details |
| :--- | :--- |
| `SelfieSegmentationCpu` | Segments the person from background in a selfie image. (CPU input, and inference is executed on CPU.) |
| `SelfieSegmentationGpu` | Segments the person from background in a selfie image. (GPU input, and inference is executed on GPU.) |