DH_live (mini) Documentation

DH_live (mini) is a lightweight 2D video digital human solution designed for web, mobile, and real-time applications. It is optimized for extremely low computational power (39 Mflops per frame) and minimal storage (web resources under 3MB), allowing it to run in real-time on mobile browsers without requiring a GPU.

Key features include:

• No training required: Out-of-the-box usage.
• Low latency: Optimized for mobile and web environments.
• Cross-platform: Supports Windows, Linux, macOS, Web, Mini-programs, and App (via WebView).

DH_live_mini is optimized for low-end devices and web environments, whereas the standard DH_live requires high-end GPUs (30-series and above).

| Solution Name | Single Frame Compute (Mflops) | Usage Method | Face Resolution | Applicable Devices |
|------------------------------|-------------------|------------|-------------------|--------------------|
| Ultralight-Digital-Human（mobile） | 1100 | Individual Training | 160 | Mid-to-high-end mobile APPs |
| DH_live_mini | 39 | No Training Required | 128 | All devices, Web & APP & Mini-program |
| DH_live | 55046 | No Training Required | 256 | GPUs 30-series and above |
| duix.ai | 1200 | Individual Training | 160 | Mid-to-high-end mobile APPs |

To use DH_live, create a Conda environment with Python 3.11 and install the necessary dependencies.

Note: If you do not have a GPU, install the CPU version of PyTorch instead of the CUDA version.

1. Create and activate the environment:

   conda create -n dh_live python=3.11
   conda activate dh_live

2. Install PyTorch (CUDA 12.4 example):

   pip install torch --index-url https://download.pytorch.org/whl/cu124

3. Install other requirements:

   pip install -r requirements.txt

4. Download and extract model files into the checkpoint/ directory.

To enable full real-time voice dialogue, you must configure the LLM interface in voiceapi/llm.py using the OpenAI API format. You can use providers like Doubao or DeepSeek.

Required Models:

• ASR model: sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20
• TTS model: sherpa-onnx-vits-zh-ll

# Example DeepSeek configuration
from openai import OpenAI
base_url = "https://api.deepseek.com"
api_key = ""
model_name = "deepseek-chat"

llm_client = OpenAI(
    base_url=base_url,
    api_key=api_key,
)

For a simple demonstration service without full real-time voice capabilities, run the following command from the DH_live root directory:

python web_demo/server.py

Once the server is running, access the interface in your browser at: http://localhost:8888/static/MiniLive.html

To start the real-time voice dialogue service, ensure you have downloaded the local ASR and TTS models and configured your OpenAI API for LLM interaction. Run the following command from the DH_live root directory:

python web_demo/server_realtime.py

Once the server is running, access the interface in your browser at: http://localhost:8888/static/MiniLive_RealTime.html

The project requires specific model checkpoints to function. These are organized into the following directory structure:

• checkpoint/DINet_mini/epoch_40_new.pth: Video generation model
• checkpoint/lstm/lstm_model_epoch_325.pkl: Speech feature model
• checkpoint/rvm_resnet50.pth: Green screen removal (matting) model

Download files from BaiduDrive or GoogleDrive.

Before running the digital human, you must process your source video to generate the required assets. This involves two steps: generating processed video data (with optional matting) and generating web-compatible data.

1. Process video with matting:

   python data_preparation_mini.py <input_video_path> <output_dir> --matting

2. Prepare web assets:

   python data_preparation_web.py <output_dir>

Processed information will be stored in the ./video_data directory.

python data_preparation_mini.py video_data/000002/video.mp4 video_data/000002 --matting
python data_preparation_web.py video_data/000002

To host a web-based demonstration of the digital human:

1. Replace the files in the assets folder with your new character's assets (e.g., from video_data/000002/assets).
2. Start the server:

   python web_demo/server.py

3. Access the demo at localhost:8888/static/MiniLive.html.

To customize the web behavior, edit the CONFIG object at the top of web_demo/static/js/MiniLive2.js.

Generate digital human video from an audio file.

Warning: This step is not supported on Linux or macOS.

Requirement: The audio file must be a single-channel 16K Hz .wav file.

python demo_mini.py video_data/000002/assets video_data/audio0.wav 1.mp4