DOCSEARCHbyAlgolia
Submit Docs

Audio2X SDK

repository·main·Indexed 41 days ago

https://github.com/nvidia/audio2face-3d-sdk

A high-performance, GPU-accelerated toolkit for audio-driven animation and emotion detection. It consists of the Audio2Face SDK for generating facial animations (skin, tongue, jaw, and eyes) and the Audio2Emotion SDK for detecting emotional states from speech. Leveraging CUDA and TensorRT, it supports both offline batch processing and real-time interactive modes on Windows and Linux.

Tokens
6.3K
Snippets
7
Records
28
Agent score
53%

What's inside Audio2X

  1. Overview of the Audio2X SDK

    main

    The Audio2X SDK is a high-performance toolkit for audio-driven animation and emotion detection. It is designed for applications requiring fast processing (faster than 60 FPS) and leverages NVIDIA GPU acceleration via CUDA and TensorRT. It supports both Windows and Linux and is suitable for both batch (offline) and interactive (real-time/authoring) processing modes.

    The SDK is divided into two primary functional components:

    • Audio2Emotion SDK: Detects and classifies emotional states from speech audio.
    • Audio2Face SDK: Generates facial animations (skin, tongue, jaw, and eyes) from speech audio using either regression-based or diffusion-based models.

  2. Overview of Audio2X SDK

    main

    The Audio2X SDK is a toolkit for audio-driven animation and emotion detection. It consists of two primary components:

    • Audio2Emotion SDK: Analyzes speech input to detect and classify emotional states.
    • Audio2Face SDK: Generates facial animations from speech input using regression and diffusion-based approaches.

    The SDK is optimized for high performance using NVIDIA GPU acceleration (CUDA and TensorRT), supporting faster-than-real-time processing (over 60 FPS), multi-track audio streams, and both batch and interactive processing modes on Windows and Linux.

  3. Overview of Audio2Face SDK Samples

    main

    The samples directory provides three distinct application levels to demonstrate the Audio2Face SDK capabilities, ranging from high-level executor APIs to low-level manual control.

    • sample-a2f-executor: The recommended starting point. It uses the high-level Executor API to demonstrate geometry executor creation with bundles, audio loading, and neutral emotion setup. This sample does not require Audio2Emotion (A2E) and supports both regression and diffusion models.
    • sample-a2f-a2e-executor: Demonstrates integration between Audio2Face and Audio2Emotion (A2E). It showcases combined processing workflows and supports both offline and streaming modes.
    • sample-a2f-low-level-api-fullface: An advanced sample for maximum control. It demonstrates direct usage of core components, including manual inference engine setup, tensor manipulation, and loading data from NPY files.

  5. How Audio Accumulators work

    main

    The SDK uses a streaming architecture where audio is processed via IAudioAccumulator objects. Audio is passed as floating-point values using the Accumulate() method.

    Key lifecycle and management methods:

    • Accumulate(): Adds audio samples to the accumulator.
    • Close(): Signals that no more audio will be streamed. This allows executors to process the final frames using zero padding.
    • Reset(): Clears the accumulator to prepare for a new audio track. Existing GPU buffers are returned to an internal pool for reuse.
    • DropSamplesBefore(): Allows the application to drop already-processed samples, enabling the streaming of very long audio tracks without unbounded GPU memory growth.

    Accumulators can be shared between different executors (e.g., between Audio2Emotion and Audio2Face).

  6. Understand the Build Output Structure

    main

    After building, artifacts are located in the _build directory. The audio2x-sdk directory is the most important for integration as it provides a unified shared library combining Audio2Emotion, Audio2Face, and Common components.

    Structure Overview:

    • _build/release/audio2emotion-sdk/: Binaries and static libraries for emotion detection.
    • _build/release/audio2face-sdk/: Binaries and static libraries for facial animation.
    • _build/release/audio2x-common/: Common utilities.
    • _build/release/audio2x-sdk/: Unified SDK
      • bin/: audio2x.dll (Windows) or shared library.
      • include/: Header files.
      • lib/: Import libraries (Windows) or libaudio2x.so (Linux).

  7. Geometry vs Blendshape Executors

    main

    The SDK provides two primary ways to output facial animation:

    Geometry Executors

    These generate facial animation as moving vertices from a mesh with known topology.

    Blendshape Executors

    These output blendshape weights or rig controls. There are two implementation types:

    1. GPU Blendshape Solve: Runs on the GPU. It behaves like a geometry executor: it schedules the computation on the GPU and returns immediately. Callbacks are triggered from the same thread that called Execute(). Users must synchronize with the provided CUDA stream to ensure data is ready.
    2. CPU Blendshape Solve: Runs on a user-controllable thread pool via the nva2f::IJobRunner interface (or a default std::thread pool). Unlike the GPU version, nva2x::IExecutor::Execute() may return before any result callbacks have been called. To ensure all CPU tasks are finished, use nva2f::IBlendshapeExecutor::Wait().

  8. Understand the Audio2Emotion model and emotion remapping

    main

    The Audio2Emotion model is an ONNX-based classifier that detects emotions from audio.

    Key Technical Details:

    • Input: Expects a 16000 Hz sampling rate.
    • Detected Emotions: The raw network outputs: angry, disgust, fear, happy, neutral, and sad.
    • Remapping: Because Audio2Face uses a different emotion set, a post-processing stage is required. Configuration for this remapping (mapping inference output to Audio2Face indices) is stored in model_config.json.
    • Inference Window: The model detects the emotion at the center of an audio window. It requires audio samples available for at least half the inference window size ahead of the target timestamp.

    Implementation Tip: While you can manage inference windows manually via the low-level API, it is recommended to provide a desired frame rate to the executor, which handles window progress and timestamps automatically.

  9. How Emotion Accumulators work

    main

    Emotion accumulators store multi-valued emotions associated with specific timestamps. They are used to provide emotion input to executors, whether those values come from DCC animation curves, constant values, or another executor (like Audio2Emotion).

    Usage rules:

    • Ordering: Emotions must be accumulated in increasing timestamp order.
    • Querying: Querying a timestamp returns the exact value if it exists, or performs linear interpolation between the two nearest timestamps.
    • Boundary Behavior: Reading before the first emotion returns the first emotion. Reading past the last emotion returns the last emotion only if the accumulator has been Close()-ed. If not closed, querying past the last emotion returns an error because the next incoming emotion could change the interpolated result.

  10. Compare Audio2Face Regression-based vs Diffusion-based animation

    main

    The Audio2Face SDK offers two distinct deep learning approaches for generating facial animations:

    Regression-based Animation

    • Mechanism: Provides facial animation at the center of the provided audio sample window.
    • Capabilities: Supports random-access in the audio buffer and can produce frames at any arbitrary frame rate.
    • Inputs: Takes audio and emotion data. Emotions can be implicit (specified via source_frame and source_shot in model_config.json) or explicit (provided by the user or Audio2Emotion).
    • Output: Provides PCA coefficients. These must be reconstructed into vertex position deltas using shape matrices from model_data.npz.
    • Buffer Info: Inference buffer size is defined by buffer_len in network_info.json.

    Diffusion-based Animation

    • Mechanism: Computes many frames of animation within a single inference pass. It generates a block of frames, truncates the edges, and uses the center frames.
    • Capabilities: Generates 60 frames per second. Because of the truncation, each inference is typically stepped by half a second to ensure contiguous 'good' frames.
    • Inputs: Requires emotion data to be available for all 'good' frames in the window. It also requires an identity index to select from available identities (listed in network_info.json).
    • Output: Directly provides vertex position deltas (no PCA reconstruction required).
    • Statefulness: Unlike regression, the diffusion model is stateful. Inferences must be run sequentially and contiguously, passing the state from the previous inference to the next.

  11. Understand the Audio2X SDK API levels

    main

    The SDK provides two distinct API layers depending on the required level of control:

    High-level API (Executors)

    Recommended for most use cases. It provides turn-key model setup and execution. You interact with Executors which abstract the underlying models and handle the end-to-end pipeline (initialization, execution, and data flow).

    • Multi-track Executors: Optimized for high-throughput offline processing. They support streaming audio, allowing you to accumulate audio, trigger execution, and receive results via callbacks.
    • Interactive Executors: Optimized for interactive authoring. They are designed to compute single frames quickly after parameter edits or compute entire tracks rapidly after final values are set. They typically require the full audio track to be available upfront.

    Low-level API (Building Blocks)

    Provides fine-grained control over buffer management, inference window progress, and post-processing. This is intended for advanced users who need to compose custom pipelines using individual components like ICudaStream, IInferenceEngine, or specific post-processors (e.g., IMultiTrackAnimatorSkin, IBlendshapeSolver).

  12. How Multi-Track Execution works

    main

    Audio2Emotion and Audio2Face executors support multi-track execution to improve GPU throughput by batching multiple audio tracks together.

    Behavioral differences between executors:

    • Audio2Emotion: If not all tracks have enough data, the executor runs a minimal batch size corresponding only to the number of ready tracks.
    • Audio2Face: Always performs inference using the full batch size allocated at the time of executor creation. This optimization is intended to avoid the overhead of moving state associated with partial batches.