HeyGem-Linux-Python-Hack

To set up the project, clone the repository, navigate to the directory, and run the provided download script to fetch necessary assets.

git clone https://github.com/Holasyb918/HeyGem-Linux-Python-Hack
cd HeyGem-Linux-Python-Hack
bash download.sh

First, clone the repository and download the necessary models using the provided script:

git clone https://github.com/Holasyb918/HeyGem-Linux-Python-Hack
cd HeyGem-Linux-Python-Hack
bash download.sh

CLI Usage

You can run the demo using the provided sample audio and video files:

python run.py

To use your own data, pass the --audio_path and --video_path arguments. Note: Paths must be local and relative.

python run.py --audio_path example/audio.wav --video_path example/video.mp4

Gradio Web UI

To launch the Gradio interface:

python app.py

Important: Wait for the model initialization to complete before submitting any tasks.

The tts component uses Fish Speech. While it supports Python 3.8, higher versions are generally preferred. To set it up, clone the tts-fish-speech repository and download the models using the huggingface-cli.

Note: If you cannot satisfy all environment requirements in a single environment, you may need to run the pipeline (from text to digital human) in separate steps.

git clone https://github.com/Holasyb918/tts-fish-speech
cd tts-fish-speech
# Download tts models
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5/

This project requires a specific environment. It only supports Linux and Python 3.8.

To install dependencies, it is recommended to run the code first and observe error messages, then attempt to install specific requirements based on those errors. A bulk installation via requirements.txt may not succeed due to hardware-specific dependencies like onnxruntime-gpu and torch which must match your local CUDA version.

If you encounter environment issues, you can use requirements_0.txt as a fallback or refer to community solutions for environments like AutoDL.

pip install -r requirements.txt

The face2face component requires Python 3.8. To set it up, clone the repository and run the provided download script to fetch the necessary models.

Note: The face2face component specifically supports and requires Python 3.8.

git clone https://github.com/Holasyb918/HeyGem-Linux-Python-Hack
cd HeyGem-Linux-Python-Hack
# Download f2f models
bash download.sh

When you instantiate VideoProcessor, it automatically triggers _initialize_service(). This method includes a mandatory sleep period of 5 seconds to allow the underlying trans_dh_service to stabilize.

Any calls to process_video made before this initialization is complete will enter a polling loop, checking self.is_initialized every 1 second until the service is ready.

The video generation process follows these steps:

1. Initialization: VideoProcessor waits for trans_dh_service to be ready.
2. Task Creation: A unique work_id (UUID) is generated to track the job.
3. Core Processing: The self.task.work() method is called with the audio and video paths.
4. Post-processing: The system retrieves the raw result, moves it to a organized result/{work_id} directory, and cleans up temporary files.
5. Output: The final path to the processed video is returned.

Initialization errors are often caused by a mismatch in onnxruntime-gpu versions.

1. Verify environment: Run the check script to see if CUDA is successfully detected:

   python check_env/check_onnx_cuda.py

2. Fixing mismatches: Try to find a compatible combination of cudatoolkit and onnxruntime-gpu. A known working pair is:
   • cudatoolkit: 11.8.0
   • onnxruntime-gpu: 1.16.0
3. Alternative approach: If issues persist, uninstall onnxruntime-gpu and onnxruntime, use conda to install the cudatoolkit environment, and then attempt to pip install onnxruntime-gpu again.

Initialization errors are often caused by onnxruntime-gpu version mismatches.

1. Verify environment: Run python check_env/check_onnx_cuda.py and check if it outputs successfully.
2. Fixing mismatches: If it fails, try matching your cudatoolkit with a compatible onnxruntime-gpu version.

Recommended working version:

If issues persist, uninstall both onnxruntime-gpu and onnxruntime, install cudatoolkit via Conda, and then reinstall onnxruntime-gpu via pip.

This error indicates a missing dependency. Install typeguard to resolve it:

pip install typeguard

If you see errors like Could not load library libcublasLt.so.11, it means a required CUDA library is missing or not in your path.

1. Locate the file:

   sudo find /usr -name "libcublasLt.so.11"

2. If not found: You must install the corresponding CUDA version.
3. If found: Add the directory containing the file to your LD_LIBRARY_PATH. For example:

   export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

   To make this permanent, add the export line to your `~/.bashrc` and run `source ~/.bashrc`.