DOCSEARCHbyAlgolia
Submit Docs

Open Avatar Chat Documentation

repository·main·Indexed 25 days ago

https://github.com/humanaigc-engineering/openavatarchat

A modular implementation of interactive digital human dialogues supporting multi-modal interactions (text, voice, video). It allows developers to swap core components including ASR, LLM, TTS, and avatar technologies such as LiteAvatar, LAM, MuseTalk, and FlashHead. Features include a Beta Chat Agent mode integrated with OpenClaw for tool calling, personality memory, and visual perception, as well as Docker Compose deployment options.

Tokens
61.6K
Snippets
111
Records
412
Agent score
86%

What's inside Open Avatar Chat

  2. Overview of MuseTalk Avatar Handler

    main
    MuseTalk is a diffusion-based real-time digital human lip-sync driving solution. It consumes an AVATAR_AUDIO audio stream from upstream TTS (Text-to-Speech) and uses a multi-threaded pipeline to generate lip-synced video frames and audio frames. The output consists of AVATAR_VIDEO and AVATAR_AUDIO, which are provided to downstream Client Handlers for rendering.

  3. Overview of Open Avatar Chat

    main

    Open Avatar Chat is a modular implementation of interactive digital human dialogue systems. It is designed to run a complete digital human conversation system on a single PC.

    Key features include:

    • Multimodal Interaction: Supports text, voice, and video for natural human-computer dialogue.
    • Modular Architecture: Highly modular design allows for flexible replacement of core components such as ASR (Automatic Speech Recognition), LLM (Large Language Model), TTS (Text-to-Speech), and Avatar technologies.
    • Diverse Digital Human Avatars: Supports multiple technologies including LiteAvatar, LAM, MuseTalk, and FlashHead.
    • Low Latency Optimization: Optimized via VAD (Voice Activity Detection), speech buffering, and frame rate control, achieving an average response time of approximately 2.2 seconds.

  4. MuseTalk Backpressure and Error Handling

    main

    To prevent memory exhaustion and handle runtime errors, the MuseTalk pipeline uses the following strategies:

    Backpressure Control

    • Output Queue: If the _output_queue depth exceeds batch_size * 5, _collect_batch() pauses collection.
    • Frame ID Queue: The Frame Collector allocates frame_id based on actual FPS; the inference thread must acquire a frame_id before processing.

    Fault Tolerance

    • Inference Errors: If a GPU inference error occurs, the pipeline returns zero frames/latents to prevent a total crash (this may result in temporary black faces).
    • Audio Overflow: If the add_audio() queue is full, data is dropped after a 1-second timeout.
    • Resource Cleanup: create_context() uses try/except blocks to prevent Processor leaks, and destroy_context() follows a strict stop-then-release sequence.

  5. Get Started with Open Avatar Chat

    main
    Open Avatar Chat is a modular interactive digital human conversation system designed to run a complete digital human dialogue system on a single PC. It features multimodal interaction (text, audio, video), a modular architecture for swapping ASR, LLM, TTS, and Avatar components, and is optimized for low latency (~2.2s average response time).

  7. Understand MuseTalk Concurrency and Thread Safety

    main

    The MuseTalk implementation uses a hybrid concurrency model to balance GPU and CPU resources:

    • GPU Operations: Serialized via _inference_lock in the MuseTalkAlgoV15 instance to prevent hardware contention.
    • CPU Operations: Tasks like res2combined() and frame collection run in parallel across multiple Processor instances.
    • Multi-Session Support: A MuseTalkProcessorPool manages multiple Processor instances. Each processor has its own independent queues and worker threads, allowing multiple concurrent user sessions.

    Key Locks:

    • _inference_lock (MuseTalkAlgoV15): Protects all GPU operations.
    • _lock (MuseTalkProcessorPool): Protects acquire/release of processors.
    • _generation_lock (AvatarMuseTalkProcessor): Protects _generation_id.
    • _frame_id_lock (AvatarMuseTalkProcessor): Protects queue clearing operations.
    • _stream_key_lock (AvatarMuseTalkContext): Protects _current_tts_stream_key.

  8. Understand the MuseTalk Avatar Data Flow

    main

    The MuseTalk Avatar Handler processes audio from a TTS engine to generate synchronized video frames. The data flow follows this pipeline:

    1. Input: Receives TTS audio (expected: AVATAR_AUDIO, 24kHz, float32, shape [1, N]).
    2. Handler Processing: HandlerAvatarMuseTalk.handle() detects stream key changes, validates input, and slices audio into 1-second segments (based on output_audio_sample_rate).
    3. Processor Pipeline:
      • Feature Extraction: Resamples audio from 24kHz to 16kHz using librosa, extracts Whisper features, and aligns frames.
      • Frame Generation: Uses UNet and VAE (optionally multi-threaded) to generate latents and decode them into video frames.
      • Composition: Combines processed frames into a final format.
      • Frame Collection: Acts as the output clock, emitting video and audio frames synchronized to the configured fps.
    4. Output: Emits AVATAR_VIDEO and AVATAR_AUDIO via callbacks to the downstream client (e.g., WebRTC).

  9. Understand the MuseTalk Avatar Handler Pipeline

    main

    The MuseTalk Avatar Handler uses a multi-threaded pipeline to transform audio into video frames. The pipeline consists of five main stages connected by queues:

    1. Feature Extractor Worker: Resamples audio (e.g., 24kHz to 16kHz), extracts Whisper features via GPU, and splits data into frame-aligned chunks.
    2. UNet Worker: Collects chunks into batches and performs UNet inference on the GPU to generate latents.
    3. VAE Worker: Decodes UNet latents into facial crops using the VAE on the GPU.
    4. Compose Worker: Performs CPU-bound image composition (res2combined) to merge facial crops into full frames.
    5. Frame Collector Worker: Acts as the output metronome, ensuring frames are output at the correct fps. It handles both speaking frames and idle static frames.

    Output Callbacks:

    • on_video_frame()
    • on_audio_frame()
    • on_speech_end()

  10. Understand MuseTalk Architecture and Threading Model

    main

    The MuseTalk implementation uses a multi-session processor pool architecture. The AvatarMuseTalkProcessor manages several daemon worker threads for a pipeline:

    1. _feature_extractor_worker: Whisper feature extraction.
    2. _frame_generator_worker: Single-threaded inference (UNet+VAE).
    3. _frame_generator_unet_worker: Multi-threaded UNet inference.
    4. _frame_generator_vae_worker: Multi-threaded VAE inference.
    5. _compose_worker: Res2Combined synthesis.
    6. _frame_collector_worker: Periodic frame output (FPS control).

    Threading Contexts:

    • handle(): Runs in the engine's handler_pumper thread (one per handler).
    • on_signal(): Runs in the global signal_distribute_thread.
    • on_speech_end callback: Runs in the _frame_collector_worker thread.

    Queue Flow: _audio_queue → _whisper_queue → [_unet_queue] → _compose_queue → _output_queue (with _frame_id_queue managed by the collector). All internal queues are unbounded Queue() objects.

  11. Understand the Open Avatar Chat modular architecture

    main

    Open Avatar Chat uses a modular Handler architecture. You can compose different digital human dialogue systems by combining various ASR, LLM, TTS, and Avatar modules via configuration files.

    The core modules are:

    • Client Handler: Manages WebRTC audio/video stream ingestion and transmission.
    • VAD Handler: Performs Voice Activity Detection to identify when a user starts and stops speaking.
    • ASR Handler: Converts user speech into text.
    • LLM Handler: Performs language model inference to generate dialogue responses.
    • Agent Handler (Beta): A multi-turn tool-calling Agent that can replace the traditional LLM Handler.
    • TTS Handler: Synthesizes speech from the generated response text.
    • Avatar Handler: Drives the digital human by generating corresponding facial animations based on the audio.

  12. MuseTalk Thread Safety and Concurrency

    main

    The MuseTalk handler implements several mechanisms to ensure stability during concurrent use:

    • GPU Serialization: All GPU operations are serialized through _inference_lock to allow safe sharing across multiple sessions.
    • TTS Stream Protection: The _current_tts_stream_key is protected by _stream_key_lock.
    • Race Condition Prevention: A generation_id mechanism is used to resolve race conditions between add_audio() and interrupt() calls.
    • Queue Integrity: _frame_id_lock protects queue clearing operations to prevent conflicts between interrupt() and _clear_queues().