HunyuanVideo-Foley

HunyuanVideo-Foley uses a hybrid architecture to achieve high-fidelity audio-visual synchronization:

• Multimodal Transformer Blocks: Process visual and audio streams simultaneously.
• Unimodal Transformer Blocks: Focus on refining the audio stream.
• Visual Encoding: Uses a pre-trained encoder to extract features from video frames.
• Text Processing: Extracts semantic features via a pre-trained text encoder.
• Audio Encoding: Uses latent representations with Gaussian noise perturbation.
• Temporal Alignment: Employs a Synchformer-based frame-level synchronization mechanism with gated modulation.

The repository is organized as follows:

• hunyuanvideo_foley/models/: Model implementations (e.g., hifi_foley.py, dac_vae/, synchformer/).
• hunyuanvideo_foley/utils/: Utilities for configuration, features, model loading, and media processing.
• hunyuanvideo_foley/constants.py: Centralized project constants.

hunyuanvideo_foley/
├── models/                 # Model implementations
│   ├── hifi_foley.py      # Main model
│   ├── nn/                # Neural network layers
│   ├── dac_vae/           # Audio VAE
│   └── synchformer/       # Synchronization model
├── utils/                 # Utilities
│   ├── config_utils.py    # Configuration handling
│   ├── feature_utils.py   # Feature extraction
│   ├── model_utils.py     # Model loading/saving
│   └── media_utils.py     # Audio/video processing
└── constants.py           # Project constants

When contributing code, adhere to these standards:

Error Handling

• Use custom exceptions for domain-specific errors.
• Validate inputs at function boundaries.
• Log errors using appropriate levels (ERROR, WARNING, INFO).
• Provide helpful error messages.

Type Hints

• Add type hints to all function parameters and return values.
• Use Optional[Type] for nullable parameters.
• Import types from the typing module.

Documentation

• Use Google-style docstrings for all public functions and classes.
• Document parameters, return values, and exceptions.

Follow these steps to set up the environment for generating Foley audio from video.

System Requirements

• CUDA: 12.4 or 11.8 recommended
• Python: 3.8+
• OS: Linux (primary support)
• VRAM Requirements:
  • XXL model: 20GB (or 12GB with --enable_offload)
  • XL model: 16GB (or 8GB with --enable_offload)

Installation Steps

1. Clone the repository:

   git clone https://github.com/Tencent-Hunyuan/HunyuanVideo-Foley
   cd HunyuanVideo-Foley

2. Install dependencies:

   pip install -r requirements.txt

3. Download Pretrained Models (from Huggingface):

   # Using git-lfs
   git clone https://huggingface.co/tencent/HunyuanVideo-Foley
   
   # OR using huggingface-cli
   huggingface-cli download tencent/HunyuanVideo-Foley

Tests should be placed in the tests/ directory and named using the test_*.py pattern. Use pytest to execute them.

Commands:

• Run all tests: python -m pytest
• Run a specific test file: python -m pytest tests/test_feature_utils.py
• Run with coverage: python -m pytest --cov=hunyuanvideo_foley

# Run all tests
python -m pytest

# Run specific test file
python -m pytest tests/test_feature_utils.py

# Run with coverage
python -m pytest --cov=hunyuanvideo_foley

After installation, verify that the package is correctly installed and the CLI tool is accessible.

# Check package installation and version
python -c "import hunyuanvideo_foley; print(hunyuanvideo_foley.__version__)"

# Check CLI tool help
hunyuanvideo-foley --help

The community has developed several ComfyUI nodes to integrate HunyuanVideo-Foley into existing workflows:

• if-ai/ComfyUI_HunyuanVideoFoley: Supports CPU offloading and FP8 quantization.
• phazei/ComfyUI-HunyuanVideo-Foley: An alternative implementation supporting different precision modes.

If you need to format code or run quality checks manually without pre-commit, use the following commands. Note that the project uses a 120 character line length.

# Format all Python files
black --line-length 120 .

# Sort imports
isort --profile black --line-length 120 .

# Check code style
flake8 --max-line-length 120

# Type checking
mypy --ignore-missing-imports .

The project uses pre-commit to automatically format code before commits. This ensures compliance with the project's style guidelines (Black, isort, flake8, mypy).

To install the hooks, run:

pip install pre-commit
pre-commit install

To begin developing on HunyuanVideo-Foley, create a virtual environment, activate it, and install the project in editable mode along with its dependencies.

Steps:

1. Create and activate a virtual environment.
2. Install requirements via pip install -r requirements.txt.
3. Install the package in editable mode using pip install -e ..

python -m venv venv
source venv/bin/activate  # Linux/Mac
# or
venv\Scripts\activate     # Windows

pip install -r requirements.txt
pip install -e .