FoleyCrafter

To set up the FoleyCrafter environment, use the provided environment.yaml file to create a new Conda environment and then activate it. You must also install and initialize git-lfs to ensure checkpoints can be downloaded correctly.

# install conda environment
conda env create -f requirements/environment.yaml
conda activate foleycrafter

# install GIT LFS for checkpoints download
conda install git-lfs
git lfs install

You can guide the audio generation process using text prompts.

• Positive Prompt: Use --prompt to specify desired sounds (e.g., 'seagulls').
• Negative Prompt: Use --nprompt to specify sounds you want to exclude (e.g., 'river flows').
• Seed: Use --seed to ensure reproducibility.

# Example: Generating audio with a specific prompt
python inference.py \
--input=input/PromptControl/case1/ \
--seed=10201304011203481429 \
--prompt='noisy, people talking' \
--save_dir=output/PromptControl/case1_prompt/

# Example: Generating audio while excluding specific sounds
python inference.py \
--input=input/PromptControl/case3/ \
--seed=10041042941301238011 \
--nprompt='river flows' \
--save_dir=output/PromptControl/case3_nprompt/

Checkpoints are automatically downloaded when running inference.py. However, you can download them manually using git clone.

Manual Download Steps:

1. Clone the Auffusion text-to-audio base model into checkpoints/auffusion.
2. Clone the FoleyCrafter model into the checkpoints/ directory.

Required Directory Structure: Ensure your checkpoints folder follows this structure:

└── checkpoints
    ├── semantic
    │   ├── semantic_adapter.bin
    ├── vocoder
    │   ├── vocoder.pt
    │   ├── config.json
    ├── temporal_adapter.ckpt
    └── timestamp_detector.pth.tar

# Download the text-to-audio base model (Auffusion)
git clone https://huggingface.co/auffusion/auffusion-full-no-adapter checkpoints/auffusion

# Download FoleyCrafter
git clone https://huggingface.co/ymzhang319/FoleyCrafter checkpoints/

When using the FoleyCrafter interface, two key scales control the generation quality:

1. Visual Content Scale (ip_adapter_scale): Determines the level of semantic alignment between the visual content of the video and the generated audio.
2. Temporal Align Scale (temporal_scale): Determines how strongly the audio synchronizes with temporal visual cues. If the input video has strong temporal movements (e.g., a hammer hitting a nail), increasing this scale helps the audio match those moments.

FoleyCrafter provides a Gradio-based web interface for generating synchronized audio from silent videos. You can launch the server using the app.py entrypoint with several CLI arguments to configure the server and model paths.

CLI Arguments:

• --config: Path to the configuration file (default: example/config/base.yaml).
• --server-name: The hostname to bind the server to (default: 0.0.0.0).
• --port: The port number to listen on (default: 7860).
• --share: Boolean flag to enable a public Gradio share link.
• --save-path: Directory where generated samples are stored (default: samples).
• --ckpt: Directory containing model checkpoints (default: checkpoints/).

python app.py --config example/config/base.yaml --port 7860

Use inference.py to generate audio from input videos. By default, this performs standard video-to-audio generation.

python inference.py --save_dir=output/sora/

To ensure the generated audio aligns better with visual cues in the video, use the --temporal_align flag. This utilizes a temporal adapter to synchronize audio events with visual changes.

python inference.py \
--temporal_align \
--input=input/avsync \
--save_dir=output/avsync/

You can interact with FoleyCrafter through a web-based Gradio interface by running app.py. Use the --share flag to generate a public URL if you want to share the interface.

python app.py --share

The inference.py script accepts the following command-line arguments for controlling the audio generation process.

options:
  -h, --help            show this help message and exit
  --prompt PROMPT       prompt for audio generation
  --nprompt NPROMPT     negative prompt for audio generation
  --seed SEED           ramdom seed
  --temporal_align TEMPORAL_ALIGN
                        use temporal adapter or not
  --temporal_scale TEMPORAL_SCALE
                        temporal align scale
  --semantic_scale SEMANTIC_SCALE
                        visual content scale
  --input INPUT         input video folder path
  --ckpt CKPT           checkpoints folder path
  --save_dir SAVE_DIR   generation result save path
  --pretrain PRETRAIN   generator checkpoint path
  --device DEVICE