DOCSEARCHbyAlgolia
Submit Docs

SenseVoice Documentation

repository·main·Indexed 25 days ago

https://github.com/funaudiollm/sensevoice

A high-performance speech foundation model for multilingual Automatic Speech Recognition (ASR), Spoken Language Identification (LID), Speech Emotion Recognition (SER), and Audio Event Detection (AED). Supports inference via FunASR's AutoModel, deployment as a standalone binary using llama.cpp (GGUF), and export to ONNX or Libtorch. Includes capabilities for speaker diarization when composed with FSMN-VAD and CAM++, as well as tools for finetuning and data preparation using sensevoice2jsonl.

Tokens
12.3K
Snippets
41
Records
67
Agent score
94%

What's inside SenseVoice

  1. Overview of SenseVoice capabilities

    main

    SenseVoice is an audio foundation model designed for comprehensive audio understanding. It provides high-precision capabilities in:

    • Multilingual Speech Recognition (ASR): The SenseVoiceSmall checkpoint supports Chinese, Cantonese, English, Japanese, and Korean.
    • Language Identification (LID): Automatically detects the spoken language.
    • Speech Emotion Recognition (SER): Detects emotional states in speech.
    • Audio Event Detection (AED): Identifies acoustic events such as music, clapping, laughter, crying, coughing, and sneezing.

    Note: Speaker diarization is not a native output of the SenseVoiceSmall checkpoint; it is achieved via a pipeline combining FunASR with independent FSMN-VAD and CAM++ models.

  2. Overview of FunASR on llama.cpp / GGUF runtime

    main

    The runtime/llama.cpp directory provides a C++ / ggml runtime designed to run FunASR models (Fun-ASR-Nano, SenseVoiceSmall, and Paraformer) on CPU and edge devices without requiring PyTorch or Python. It uses quantized GGUF weights and is optimized for offline, on-device, and embedded deployment (e.g., laptops, phones, Raspberry Pi) using CPU SIMD.

    Supported Models

    ModelHead / DecoderAutoregressive?Output Units
    Fun-ASR-Nanoadaptor + Qwen3-0.6B LLMyes (LLM)Qwen3 BPE
    SenseVoiceSmallCTCnospectok BPE (25055)
    ParaformerCIF + SAN-M decoderno (parallel)char/BPE (8404)

    Directory Structure

    Each model has its own directory containing llama.cpp example sources, a GGUF export script, and a model-specific README:

    • fun-asr-nano/ (includes funasr-cli, funasr-encoder, funasr-embd, export_encoder_gguf.py)
    • sensevoice/ (includes funasr-sensevoice, export_sensevoice_gguf.py, detok.py)
    • paraformer/ (includes funasr-paraformer, export_paraformer_gguf.py, detok_paraformer.py)

  3. Compare FunASR (llama.cpp/GGUF) vs whisper.cpp for Chinese ASR

    main

    For Chinese (Mandarin) speech recognition on CPU, FunASR models (SenseVoice, Paraformer, Fun-ASR-Nano) significantly outperform whisper.cpp in both accuracy (measured by micro-CER) and speed (Real-Time Factor).

    Key Performance Insights:

    • Accuracy: FunASR models achieve ~8-10% CER, whereas whisper.cpp models range from 22-31% CER.
    • Speed: FunASR models like SenseVoiceSmall and Paraformer achieve ~20-21x real-time speed on CPU (8 threads), while whisper.cpp models are slower (3.2x to 9.9x real-time).
    • Model Selection:
      • Fun-ASR-Nano: Most accurate (uses an autoregressive 0.6B LLM decoder).
      • SenseVoiceSmall: High speed (~20x RTF) and provides language ID, emotion, and audio-event detection.
      • Paraformer: High speed (~21x RTF) and specialized for Mandarin.

  4. Third-party deployment options for SenseVoice

    main

    SenseVoice can be deployed using several optimized third-party implementations depending on your performance and platform requirements:

    • Triton (GPU) Deployment: Uses Triton + TensorRT (tested with FP32). Achieved a 526x speedup on V100 GPUs. FP16 support is in progress.
    • Sherpa-onnx: Supports 10 programming languages (C++, C, Python, C#, Go, Swift, Kotlin, Java, JavaScript, Dart) and platforms including iOS, Android, and Raspberry Pi.
    • Orca: Integrates SenseVoice for local, offline speech recognition with automatic language detection (Chinese, English, Japanese, Korean, Cantonese) on macOS, Linux, and Windows. Available in v1.4.159-rc.1 pre-release or later.
    • SenseVoice.cpp: A pure C/C++ implementation based on GGML. Supports 3-bit, 4-bit, 5-bit, and 8-bit quantization with no third-party dependencies.
    • streaming-sensevoice: Implements chunk-based inference using truncated attention for pseudo-streaming. Supports CTC prefix beam search and hotword enhancement.
    • OmniSenseVoice: Optimized for ultra-fast inference and batch processing.
    • SenseVoice Hotword: Supports neural network-based hotword enhancement.

  5. Perform Speaker Diarization with SenseVoiceSmall

    main

    To get speaker labels, compose SenseVoiceSmall with fsmn-vad, cam++ (for speaker labels), and ct-punc (for punctuation) via FunASR.

    Note: This requires installing FunASR from source: pip install git+https://github.com/modelscope/FunASR.git.

    from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess

model = AutoModel(
    model="iic/SenseVoiceSmall",
    trust_remote_code=True,
    remote_code="./model.py",
    vad_model="fsmn-vad",
    vad_kwargs={"max_single_segment_time": 30000},
    spk_model="cam++",
    punc_model="ct-punc",
    device="cuda:0",
)
res = model.generate(
    input="example.wav",
    cache={},
    language="auto",
    use_itn=True,
    batch_size_s=60,
    merge_vad=True,
    merge_length_s=15,
)
# Per-sentence results with speaker labels
for sent in res[0]["sentence_info"]:
    text = rich_transcription_postprocess(sent["text"])
    print(f"Speaker {sent['spk']}: [{sent['start']}ms - {sent['end']}ms] {text}")

  6. Install SenseVoice dependencies

    main

    Install the required dependencies for SenseVoice using pip:

    pip install -r requirements.txt

    Note: For SenseVoiceSmall examples and speaker diarization via FunASR, you must have funasr>=1.3.26. If you have an older version, upgrade it using:

    pip install -U "funasr>=1.3.26"

  7. Run SenseVoice on CPU/Edge via llama.cpp/GGUF

    main

    SenseVoice can run as a self-contained binary using llama.cpp without requiring Python or a GPU. It includes built-in FSMN-VAD.

    1. Download the model: bash runtime/llama.cpp/download-funasr-model.sh sensevoice ./gguf
    2. Run inference: llama-funasr-sensevoice -m <model_path> --vad <vad_model_path> -a <audio_path>
    bash runtime/llama.cpp/download-funasr-model.sh sensevoice ./gguf
llama-funasr-sensevoice -m ./gguf/sensevoice-small-f16.gguf --vad ./gguf/fsmn-vad.gguf -a audio.wav

  9. Convert SenseVoice PyTorch models to GGUF

    main

    Use the export_sensevoice_gguf.py script to convert PyTorch checkpoints and MVN files to the GGUF format. You can export in f32 (default) or f16 (half size) precision.

    # Export f32 (~936 MB)
python runtime/llama.cpp/export_sensevoice_gguf.py \
    --model_pt <model>/model.pt --mvn <model>/am.mvn \
    --out sensevoice-small.gguf

# Export f16 (half size)
python runtime/llama.cpp/export_sensevoice_gguf.py --wtype f16 \
    --model_pt <model>/model.pt --mvn <model>/am.mvn \
    --out sensevoice-small-f16.gguf

  10. Perform inference with AutoModel

    main

    Use the funasr.AutoModel class to perform speech recognition on audio files of any format and length. For long audio, it is recommended to enable VAD (Voice Activity Detection) to segment the audio. You can post-process the results using rich_transcription_postprocess to clean up the text.

    from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess

model_dir = "iic/SenseVoiceSmall"

model = AutoModel(
    model=model_dir,
    trust_remote_code=True,
    remote_code="./model.py",  
    vad_model="fsmn-vad",
    vad_kwargs={"max_single_segment_time": 30000},
    device="cuda:0",
)

# en
res = model.generate(
    input=f"{model.model_path}/example/en.mp3",
    cache={},
    language="auto",  # "zh", "en", "yue", "ja", "ko", "nospeech"
    use_itn=True,
    batch_size_s=60,
    merge_vad=True,
    merge_length_s=15,
)
text = rich_transcription_postprocess(res[0]["text"])
print(text)

  11. Build the SenseVoice llama.cpp runtime

    main

    To build the standalone llama-funasr-sensevoice binary, you must clone llama.cpp, copy the funasr-common and funasr-sensevoice directories into the examples/ folder, and update the examples/CMakeLists.txt to include the new subdirectory. Then, use CMake to build the project.

    git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cp -r /path/to/runtime/llama.cpp/funasr-common examples/   # shared audio loader (miniaudio)
cp -r /path/to/runtime/llama.cpp/funasr-sensevoice examples/
echo 'add_subdirectory(funasr-sensevoice)' >> examples/CMakeLists.txt
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
cmake --build build -j --target llama-funasr-sensevoice

  12. Configure Paraformer inference with CMVN

    main

    Unlike SenseVoiceSmall, the Paraformer model requires CMVN (Cepstral Mean and Variance Normalization) to be applied to the fbank features during inference. The transformation follows the formula (fbank + shift) · scale, where shift and scale are 560-length vectors derived from am.mvn.

    Note on parsing am.mvn: The am.mvn configuration contains three bracketed blocks: [Splice idx], [AddShift=shift], and [Rescale=scale]. Ensure you parse these by length rather than index, as naive indexing may incorrectly grab the shift vector as the scale vector, leading to incorrect token counts.