FireRedVAD Documentation

The NCNN runtime is designed for multi-platform support, with a focus on mobile devices. It provides three primary capabilities:

• Stream VAD: Frame-by-frame processing using a packed cache.
• Non-stream VAD: Processing of an entire audio sequence at once.
• Non-stream AED: Audio Event Detection supporting three classes: speech, singing, and music.

For detailed implementation and usage, refer to the ncnn/README.md file within the repository.

The Rust implementation provides several high-level audio processing capabilities:

• Preprocessing: Uses pure Rust Mel filterbank and CMVN (Cepstral Mean and Variance Normalization) preprocessing, ensuring no C dependencies are required for audio feature extraction.
• Resampling: Automatically resamples input audio from any sample rate to the required 16kHz.
• Frame Management: Uses a FrameAdapter to handle frame buffering. This allows you to feed chunks of any size into the system while receiving correctly sized 10ms frames for the model.
• Format Support: Compatible with any WAV file format, including mono/stereo and varying sample rates.

Models can be downloaded via ModelScope (recommended for users in China) or Hugging Face.

Using ModelScope:

pip install -U modelscope
modelscope download --model xukaituo/FireRedVAD --local_dir ./pretrained_models/FireRedVAD

Using Hugging Face:

pip install -U "huggingface_hub[cli]"
huggingface-cli download FireRedTeam/FireRedVAD --local-dir ./pretrained_models/FireRedVAD

# ModelScope
modelscope download --model xukaituo/FireRedVAD --local_dir ./pretrained_models/FireRedVAD

# Hugging Face
huggingface-cli download FireRedTeam/FireRedVAD --local-dir ./pretrained_models/FireRedVAD

Use the provided shell scripts to build the runtime for your current host or for Android.

Host Build

Run ./build.sh to build for your host machine. Artifacts are collected in out/:

• libfirered_vad_stream.* (Library)
• test_vad_stream (Executable)
• test_vad_non_stream (Executable)
• test_aed_non_stream (Executable)

Android Cross-Build

1. Build NCNN for Android (defaults to 32-bit): ./3rd/build_android.sh 20260113 $ANDROID_NDK
2. Cross-build runtime: ./build_android.sh $ANDROID_NDK

Artifacts are collected in out_android/armeabi-v7a/.

# Build for host
./build.sh

# Build for Android
./build_android.sh $ANDROID_NDK

FireRedVAD requires audio in 16kHz 16-bit mono PCM format. If your audio is in a different format, use ffmpeg to convert it:

ffmpeg -i <input_audio_path> -ar 16000 -ac 1 -acodec pcm_s16le -f wav <output_wav_path>

You can install FireRedVAD as a Python package using pip. For users with Nvidia GPUs, install the GPU-enabled version to leverage hardware acceleration.

# Standard installation
pip install fireredvad

# Installation with PyTorch GPU support
pip install fireredvad[gpu]

To build and run the FireRedVAD NCNN runtime, ensure your environment meets the following requirements:

• CMake: 3.10+
• C++ Toolchain: Clang or GCC
• Utilities: curl, wget, and unzip
• Python: 3.10+ (required for model conversion in ../convert)
• Android (Optional): Android NDK r21+ if you intend to cross-compile for Android.

You can convert models from the runtime/convert/ directory using the following Python scripts. The outputs will be placed in runtime/convert/out/.

• Stream VAD (packed cache): python export_packed_cache_stream_vad.py
• Non-stream VAD: python export_non_stream_vad.py
• Non-stream AED (3-class): python export_aed.py

Alternatively, you can download pre-converted models from: https://github.com/lhwcv/FireRedVAD-NCNN-streaming/

# Example: Convert Stream VAD with packed cache
cd runtime/convert
python export_packed_cache_stream_vad.py

If you do not need to perform the conversion yourself, you can download the already converted models from the following repository:

https://github.com/lhwcv/FireRedVAD-NCNN-streaming/

To install from source, first clone the repository and create a clean environment. Depending on your platform, install the specific requirements file.

Setup steps:

1. Create and activate a Python 3.10 environment:

   conda create --name fireredvad python=3.10
   conda activate fireredvad
   git clone https://github.com/FireRedTeam/FireRedVAD.git
   cd FireRedVAD

2. Install requirements:
   • macOS (ARM64): pip install -r requirements-macos-arm64.txt
   • Windows/Linux (Nvidia GPU): pip install -r requirements.txt
3. Set up environment variables:

   export PATH=$PWD/fireredvad/bin/:$PATH
   export PYTHONPATH=$PWD/:$PYTHONPATH

# macOS ARM64
pip install -r requirements-macos-arm64.txt

# Windows/Linux Nvidia GPU
pip install -r requirements.txt 

export PATH=$PWD/fireredvad/bin/:$PATH
export PYTHONPATH=$PWD/:$PYTHONPATH

To convert models for runtime, follow these steps:

1. Download the base models from Hugging Face into the pretrained_models/FireRedVAD directory within the FireRedVAD repository root.
2. Install dependencies (see the dependency list below).
3. Run the conversion scripts located in the ./runtime/convert/ directory.

Dependencies

Ensure you have Python >= 3.10 and the following packages installed:

• fireredvad
• torch>=2.0.0
• onnx>=1.14.0
• onnxsim>=0.4.0
• onnxruntime
• huggingface_hub
• pnnx

# 1. Download models
cd ~/FireRedVAD
huggingface-cli download FireRedTeam/FireRedVAD --local-dir ./pretrained_models/FireRedVAD

# 2. Run conversion scripts
cd ./runtime/convert/
python export_packed_cache_stream_vad.py
python export_non_stream_vad.py
python export_aed.py

Use the export_onnx.py script to convert FireRedVAD PyTorch models into ONNX format. The script automatically downloads models from HuggingFace (FireRedTeam/FireRedVAD) if they are not found locally. It supports exporting non-streaming models (VAD, AED) and streaming models (Stream-VAD) in two different modes: one where caches are returned as outputs, and another where caches are provided as both inputs and outputs.

# Export all models (downloads from HuggingFace automatically)
python export_onnx.py --all

# Export a specific model
python export_onnx.py --task vad
python export_onnx.py --task stream_vad
python export_onnx.py --task aed

# Use a local model directory instead of downloading
python export_onnx.py --task vad --model-dir pretrained_models/FireRedVAD/VAD