DOCSEARCHbyAlgolia
Submit Docs

FireRedASR2S Documentation

repository·main·Indexed 20 days ago

https://github.com/fireredteam/fireredasr2s

An industrial-grade, all-in-one speech processing system featuring high-accuracy Automatic Speech Recognition (ASR), Voice Activity Detection (VAD), Spoken Language Identification (LID), and Punctuation prediction. It supports multiple languages, 20+ Chinese dialects, and specialized tasks like singing transcription. The system offers two ASR architectures: FireRedASR2-LLM (Encoder-Adapter-LLM) for maximum accuracy and FireRedASR2-AED (Attention-based Encoder-Decoder) for computational efficiency.

Tokens
12.8K
Snippets
31
Records
50
Agent score
69%

What's inside FireRedASR2S

  1. Overview of FireRedASR2S System

    main

    FireRedASR2S is an industrial-grade, all-in-one Automatic Speech Recognition (ASR) system. It is composed of four specialized modules designed to work together for high-performance speech processing:

    • FireRedASR2: The core ASR module. It supports speech and singing transcription for Chinese (Mandarin and 20+ dialects/accents), English, and code-switching. It is available in two variants:
      • FireRedASR2-LLM: Uses an Encoder-Adapter-LLM framework for state-of-the-art performance and end-to-end speech interaction.
      • FireRedASR2-AED: Uses an Attention-based Encoder-Decoder architecture, balancing high performance with computational efficiency.
    • FireRedVAD: Voice Activity Detection (VAD) supporting 100+ languages. It supports non-streaming, streaming, and Multi-label VAD (mVAD) for distinguishing between speech, singing, and music.
    • FireRedLID: Spoken Language Identification (LID) supporting 100+ languages and 20+ Chinese dialects/accents.
    • FireRedPunc: Punctuation Prediction (Punc) for Chinese and English.

  2. FireRedVAD Capabilities

    main

    FireRedVAD is a DFSMN-based module that provides several modes of operation:

    • Streaming/Non-streaming: Supports both real-time and batch processing.
    • Multi-label VAD (mVAD): Acts as a lightweight Audio Event Detection (AED) system to categorize sounds into speech, singing, or music.

  3. FireRedASR2 Model Variants

    main

    FireRedASR2 is optimized for different use cases through two distinct architectures:

    1. FireRedASR2-LLM: Best for maximum accuracy and seamless end-to-end speech interaction. It utilizes an Encoder-Adapter-LLM framework.
    2. FireRedASR2-AED: Best for balancing performance and efficiency. It is suitable as a speech representation module in other LLM-based models and uses an Attention-based Encoder-Decoder (AED) architecture.

  4. Serve FireRedASR2-LLM with vLLM

    main

    For high-performance serving of the LLM-based ASR model, use vllm.

    1. Start the server:
    vllm serve allendou/FireRedASR2-LLM-vllm -tp=2 --dtype=float32
    1. Use an OpenAI-compatible client to transcribe audio:
    python3 examples/online_serving/openai_transcription_client.py --repetition_penalty=1.0 --audio_path=/root/hello_zh.wav
    vllm serve allendou/FireRedASR2-LLM-vllm -tp=2 --dtype=float32

  5. Deploy FireRedASR-AED in Client-Server Mode

    main

    For scalable deployment, use NVIDIA Triton Inference Server. This can be done via Docker Compose or manual Docker container management.

    Quick Start with Docker Compose

    Launch the service immediately by providing your Hugging Face token:

    HF_TOKEN="hf_your_token" docker compose up

    Using Docker Images

    Pull pre-built image:

    docker pull soar97/triton-fireredasr:25.06

    Build from source:

    docker build . -f Dockerfile -t soar97/triton-fireredasr:25.06

    Run the container: Ensure you mount a directory and set the shared memory size appropriately:

    your_mount_dir=/mnt:/mnt
docker run -it --name "fireredasr-aed-server" --gpus all --net host -v $your_mount_dir --shm-size=2g soar97/triton-fireredasr:25.06
    HF_TOKEN="hf_your_token" docker compose up

  6. Download FireRedASR2S models

    main

    You need to download several pretrained models to run the system. You can use either ModelScope (recommended for users in China) or Hugging Face.

    The required models are:

    • FireRedASR2-AED
    • FireRedVAD
    • FireRedLID
    • FireRedPunc
    • FireRedASR2-LLM
    # Option 1: Download via ModelScope
pip install -U modelscope
modelscope download --model xukaituo/FireRedASR2-AED --local_dir ./pretrained_models/FireRedASR2-AED
modelscope download --model xukaituo/FireRedVAD --local_dir ./pretrained_models/FireRedVAD
modelscope download --model xukaituo/FireRedLID --local_dir ./pretrained_models/FireRedLID
modelscope download --model xukaituo/FireRedPunc --local_dir ./pretrained_models/FireRedPunc
modelscope download --model xukaituo/FireRedASR2-LLM --local_dir ./pretrained_models/FireRedASR2-LLM

# Option 2: Download via Hugging Face
pip install -U "huggingface_hub[cli]"
huggingface-cli download FireRedTeam/FireRedASR2-AED --local-dir ./pretrained_models/FireRedASR2-AED
huggingface-cli download FireRedTeam/FireRedVAD --local-dir ./pretrained_models/FireRedVAD
huggingface-cli download FireRedTeam/FireRedLID --local-dir ./pretrained_models/FireRedLID
huggingface-cli download FireRedTeam/FireRedPunc --local-dir ./pretrained_models/FireRedPunc
huggingface-cli download FireRedTeam/FireRedASR2-LLM --local-dir ./pretrained_models/FireRedASR2-LLM

  7. Perform Offline Inference with FireRedASR2-AED

    main

    Offline inference allows for direct local execution of the model without setting up a server. This mode is useful for local testing and benchmarking.

    Setup

    Install dependencies using uv:

    uv sync

    Export TensorRT Engine

    To accelerate inference, you must first download the pre-built weights and export the TensorRT engines:

    1. Download the weights from Hugging Face.
    2. Run the export script.
    huggingface-cli download yuekai/FireRedASR2-AED-TensorRT --local-dir ./FireRedASR2-AED-TensorRT
uv run bash scripts/export_tensorrt.sh ./FireRedASR2-AED-TensorRT

    Run Inference

    Execute inference using torchrun on a dataset (e.g., yuekai/aishell):

    uv run torchrun infer.py --huggingface_dataset yuekai/aishell
    uv run torchrun infer.py --huggingface_dataset yuekai/aishell

  8. Setup FireRedASR2S environment

    main

    To use FireRedASR2S, follow these steps to prepare your Python environment, install dependencies, and configure your paths.

    1. Create a Python 3.10 environment using conda.
    2. Install dependencies via pip install -r requirements.txt.
    3. Configure Environment Variables: You must add the project directory to your PATH and PYTHONPATH to ensure the CLI and modules are discoverable.

    Audio Requirements: Ensure your input audio is in 16kHz 16-bit mono PCM format. You can use ffmpeg to convert files if necessary.

    # Environment setup
$ conda create --name fireredasr2s python=3.10
$ conda activate fireredasr2s
$ git clone https://github.com/FireRedTeam/FireRedASR2S.git
$ cd FireRedASR2S

# Dependencies and Paths
$ pip install -r requirements.txt
$ export PATH=$PWD/fireredasr2s/:$PATH
$ export PYTHONPATH=$PWD/:$PYTHONPATH

# Audio conversion (if needed)
$ ffmpeg -i <input_audio_path> -ar 16000 -ac 1 -acodec pcm_s16le -f wav <output_wav_path>

  9. Stream VAD processing modes

    main

    The stream_vad.py tool implements three distinct ways to process audio through the FireRedStreamVad engine:

    1. Framewise Mode: Uses stream_vad.detect_frame(audio_frame) to process audio one frame at a time. This is the most granular mode, suitable for real-time streaming simulations.
    2. Chunkwise Mode: Uses stream_vad.detect_chunk(audio_chunk) to process audio in blocks of N frames (defined by --stream_chunk_frame). This balances latency and computational efficiency.
    3. Full Mode: Uses stream_vad.detect_full(wav_path) to process the entire file in one pass. This is typically the most efficient for offline batch processing of complete files.

    Results from all modes can be converted to timestamps using stream_vad.results_to_timestamps(results).

  10. Use the stream_vad CLI tool

    main

    The stream_vad.py script provides a command-line interface for performing Voice Activity Detection (VAD) on audio files using FireRedStreamVad. It supports multiple processing modes (framewise, chunkwise, and full) and can output results as JSON, TextGrid files, or split audio segments.

    Input Options

    • --wav_path: Path to a single WAV file.
    • --wav_paths: List of multiple WAV file paths.
    • --wav_scp: Path to a .scp file containing audio mappings.
    • --wav_dir: Directory containing WAV files.

    Output Options

    • --output: Path to save the JSON results (default: vad_output).
    • --write_textgrid: If set to 1, writes a TextGrid file for the audio.
    • --save_segment_dir: Directory where detected speech segments will be saved as individual files.

    VAD Modes (--stream_vad_mode)

    • framewise: Processes audio frame by frame.
    • chunkwise: Processes audio in chunks of a specified size.
    • full: Processes the entire file at once.
    • all: Runs all available modes.

    Example Usage

    python3 stream_vad.py --wav_path test.wav --output results.json --write_textgrid 1 --save_segment_dir ./segments

  11. Use the FireRedASR2S CLI for speech processing

    main

    The fireredasr2s_cli.py script provides a command-line interface to run the full FireRedASR2S system, which includes Voice Activity Detection (VAD), Language Identification (LID), Automatic Speech Recognition (ASR), and Punctuation (Punc) modules.

    Users can provide input via a single file, multiple files, a directory, or an .scp file. The system can output results in JSONL format, TextGrid, or SRT formats, and can optionally save segmented audio files based on VAD timestamps.

    # Example: Process a directory of wav files and save results to 'my_output'
python fireredasr2s_cli.py --wav_dir ./audio_data --outdir my_output --write_srt 1 --write_textgrid 1

  12. Deploy FireRedASR2 with Triton and TensorRT via Docker Compose

    main

    You can deploy the FireRedASR2 service using Docker Compose. The setup uses the soar97/triton-fireredasr:25.06 image and requires an NVIDIA GPU.

    Configuration Details

    • Ports: The service exposes ports 8000, 8001, and 8002.
    • Shared Memory: The container is configured with 1gb of shm_size.
    • Environment Variables:
      • HF_TOKEN: Required to authenticate with Hugging Face for model downloads.
      • PYTHONIOENCODING: Set to utf-8.
    • GPU Requirements:
      • Requires the nvidia driver.
      • Uses device_ids: ['0'] by default.

    Startup Sequence

    When the container starts, it automatically:

    1. Logs into Hugging Face using the provided $HF_TOKEN.
    2. Clones the FireRedASR2S repository (branch triton).
    3. Navigates to the runtime/triton_tensorrt directory.
    4. Executes bash run.sh 0 2 to start the service.
    services:
  asr:
    image: soar97/triton-fireredasr:25.06
    shm_size: '1gb'
    ports:
      - "8000:8000"
      - "8001:8001"
      - "8002:8002"
    environment:
      - PYTHONIOENCODING=utf-8
      - HF_TOKEN=${HF_TOKEN}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    command: >
      /bin/bash -c "huggingface-cli login --token $HF_TOKEN && git clone https://github.com/yuekaizhang/FireRedASR2S.git -b triton && cd FireRedASR2S/runtime/triton_tensorrt && bash run.sh 0 2"