DOCSEARCHbyAlgolia
Submit Docs

FunASR Speech Recognition Toolkit

repository·main·Indexed 12 days ago

https://github.com/modelscope/funasr

An industrial-grade speech recognition toolkit for offline, streaming, and edge deployment. It supports ASR, VAD, punctuation, speaker diarization, emotion recognition, and audio-event detection. Features include the AutoModel class for inference, support for multi-language models like Qwen3-ASR and GLM-ASR-Nano, and tools for speaker embedding extraction and model training via torchrun.

Tokens
238.7K
Snippets
665
Records
1K
Agent score
97%

What's inside FunASR

  1. Overview of FunASR on llama.cpp / GGUF

    main
    FunASR on llama.cpp allows running FunASR models on the ggml stack. This enables execution on CPU, edge devices, and environments without a Python runtime or GPU. It uses quantized weights (GGUF format) to provide a lightweight alternative to PyTorch/ONNX/vLLM paths, making it suitable for laptops, phones, and embedded C/C++ applications.

  2. Overview of the M2MeT2.0 Challenge

    main

    The M2MeT2.0 (Multi-Channel Multi-Party Meeting Transcription) challenge is a specialized task focused on speaker-attributed automatic speech recognition (ASR). Unlike previous iterations that were speaker-independent, M2MeT2.0 requires systems to solve the 'who spoke what and when' problem.

    Core Tasks

    • Speaker Diarization: Identifying who spoke when during a meeting.
    • Multi-speaker ASR: Transcribing speech from multiple speakers simultaneously, specifically addressing challenges like overlapping speech and acoustic interference.

    Challenge Tracks

    • Fixed training conditions
    • Open training conditions

  3. Overview of FunASR on llama.cpp / GGUF runtime

    main

    The runtime/llama.cpp directory provides a C++ / ggml runtime designed to run FunASR models (Fun-ASR-Nano, SenseVoiceSmall, and Paraformer) on CPU and edge devices without requiring PyTorch or Python. It uses quantized GGUF weights (2–8 bit) and leverages CPU SIMD for efficient inference on hardware like laptops, phones, or Raspberry Pi.

    Supported Models

    ModelHead / DecoderAutoregressive?Output Units
    Fun-ASR-Nanoadaptor + Qwen3-0.6B LLMyes (LLM)Qwen3 BPE
    SenseVoiceSmallCTCnospectok BPE (25055)
    ParaformerCIF + SAN-M decoderno (parallel)char/BPE (8404)

    Directory Structure

    Each model has its own directory containing llama.cpp example sources, a GGUF export script, and model-specific documentation:

    • fun-asr-nano/: funasr-cli, funasr-encoder, funasr-embd, export_encoder_gguf.py
    • sensevoice/: funasr-sensevoice, export_sensevoice_gguf.py, detok.py
    • paraformer/: funasr-paraformer, export_paraformer_gguf.py, detok_paraformer.py

  4. Overview of FunASR Service Deployment Options

    main

    FunASR provides several community-developed software packages to facilitate industrial deployment. The available service types include:

    • Chinese Offline Transcription (CPU): For long-form audio/video to punctuated text.
    • Chinese Real-time Speech Dictation (CPU): For low-latency, high-precision real-time transcription.
    • English Offline Transcription (CPU): For English-language long-form transcription.
    • Chinese Offline Transcription (GPU): High-performance version for large-scale offline tasks.

    These services are designed to bridge the gap between raw models and production-ready business applications.

  5. Overview of SenseVoice

    main

    SenseVoice is a speech foundation model designed for comprehensive speech understanding. It provides high-precision capabilities across several domains:

    • Multilingual Automatic Speech Recognition (ASR): Supports over 50 languages (trained on 400k+ hours of data), outperforming Whisper in several benchmarks, particularly for Chinese and Cantonese.
    • Rich Transcription: Includes advanced Speech Emotion Recognition (SER) and Acoustic Event Detection (AED) (e.g., detecting music, applause, laughter, coughing, etc.).
    • Efficient Inference: Uses a non-autoregressive end-to-end framework. The SenseVoice-Small model is optimized for low latency (e.g., ~70ms for 10s of audio), making it significantly faster than Whisper-Large.
    • Deployment: Supports Python, C++, HTML, Java, and C# clients, with pipelines for multi-concurrent request handling.

  6. Compare FunASR (llama.cpp/GGUF) vs whisper.cpp for Chinese ASR

    main

    This benchmark compares the FunASR llama.cpp runtime against whisper.cpp for Mandarin Chinese speech recognition on CPU.

    Key Findings for Chinese ASR on CPU:

    • Accuracy: FunASR is approximately 2.7× more accurate than whisper.cpp across all model tiers.
    • Speed: FunASR models (SenseVoiceSmall and Paraformer) achieve ~20-21× real-time speed, significantly outperforming whisper.cpp.
    • Model Tiers:
      • Fun-ASR-Nano: Highest accuracy (uses an autoregressive 0.6B LLM decoder).
      • SenseVoiceSmall: High speed (~20× RTF) and includes language ID, emotion, and audio-event detection.
      • Paraformer: High speed (~21× RTF) and specialized for Mandarin.

    Comparison Summary (Micro-CER ↓):

    SystemCER (approx)Speed (RTF)
    FunASR SenseVoiceSmall~8%~20×
    FunASR Paraformer~10%~21×
    whisper.cpp base~31%9.9×
    whisper.cpp small~22%4.6×
    whisper.cpp large-v3-turbo~23%3.2×

  7. Explore the FunASR Model Zoo

    main

    FunASR provides a variety of pre-trained models for speech processing tasks, including speech recognition, speaker diarization, punctuation restoration, and voice activity detection (VAD). Models are available on both ModelScope (⭐) and HuggingFace (🤗).

    Available Speech Recognition Models

    Model NameTask DetailsTraining DataParameters
    paraformer-zhSpeech recognition (non-streaming, with timestamps)60,000h Mandarin220M
    paraformer-zh-spkSpeech recognition with speaker diarization (non-streaming, with timestamps)60,000h Mandarin220M
    paraformer-zh-onlineStreaming speech recognition60,000h Mandarin220M
    paraformer-enSpeech recognition (non-streaming, with timestamps)50,000h English220M
    conformer-enSpeech recognition (non-streaming)50,000h English220M

    Supporting Models

    Model NameTask DetailsTraining DataParameters
    ct-puncPunctuation restorationMandarin & English1.1G
    fsmn-vadVoice Activity Detection (VAD)Mandarin & English0.4M
    fa-zhTimestamp prediction5,000h Mandarin38M

    For a complete list and detailed dataset information, visit the ModelScope Speech Recognition task page.

  8. Use the FunASR OpenAI-Compatible API

    main

    The FunASR speech API provides an OpenAI-compatible interface for audio transcription. You can use the provided openapi.json to inspect, mock, or import the API into various tools like Swagger Editor, Redoc, Postman, or workflow engines like Dify and n8n. For client generation, ensure the multipart file field is mapped to a binary upload.

    Server URL Configuration: Replace the default local examples (http://localhost:8000 or http://funasr-api:8000) with the actual URL reachable from your application or runtime.

  9. FunASR Integration Features

    main

    When using FunASR via its API, you can leverage the following capabilities:

    • Language Support: 50+ languages including Chinese dialects, English, Japanese, and Korean.
    • Speaker Diarization: Enable via spk=true.
    • Word-level Timestamps: Available when using response_format="verbose_json".
    • Hotword Boosting: Support for improving recognition of specific terms.
    • Performance: Up to 170x realtime speed, fully local, and MIT licensed.

  10. Overview of FunASR models and capabilities

    main

    FunASR is a toolkit rather than a single model, offering different models optimized for specific use cases:

    • Fun-ASR-Nano: Optimized for speed (up to 340x real-time with vLLM). Supports Chinese, English, Japanese, and Chinese dialects/accents. Best for GPU deployment.
    • Fun-ASR-MLT-Nano: Supports 31 languages.
    • SenseVoiceSmall: Provides ASR for 5 languages along with emotion and audio event recognition. Highly efficient on CPU (17x real-time).
    • Paraformer: Designed for low-latency streaming via WebSocket.

    Key advantages over Whisper include built-in VAD + Speaker ID pipelines, emotion recognition, and significantly higher speeds on both CPU and GPU.

  11. Understand JSONL streaming and batching behavior

    main

    When running FunASR ONNX binaries in batch or streaming modes, keep the following behaviors in mind:

    • Output Streams: JSON records are written to stdout. Diagnostic information and progress logs are written to stderr.
    • Input Processing: When using a wav.scp file, one JSON line is output for every successfully completed key.
    • Concurrency and Ordering: For RTF (Real-Time Factor) binaries, single JSON lines are guaranteed not to be interleaved by multiple threads. However, the completion order of different tasks is not guaranteed. Always use the key field to associate results with inputs rather than relying on line order.
    • Default Format: If --output-format is not specified, it defaults to --output-format log (the original logging behavior).

  12. Understand the FunASR + vLLM inference architecture

    main

    The FunASR vLLM integration splits the ASR model into two distinct parts to leverage vLLM's high-throughput capabilities:

    1. PyTorch Component (Single GPU): Handles the audio frontend. It processes audio through an Audio Encoder (e.g., SenseVoice/Whisper) and an Adaptor (Transformer/MLP) to produce Audio Embeddings.
    2. vLLM Engine: Receives the audio embeddings and text prompts. It uses PagedAttention, Continuous Batching, and Tensor Parallelism to decode the text using the LLM (e.g., Qwen3-0.6B or Llama-2B).

    Key Technical Concepts:

    • EmbedsPrompt: Instead of passing token IDs, Fun-ASR-Nano passes the continuous audio embedding vectors directly to vLLM by setting enable_prompt_embeds=True. The audio embeddings are concatenated with text embeddings in the sequence dimension.
    • Weight Separation: LLM weights are extracted from the original model.pt and converted to HuggingFace safetensors format for vLLM compatibility.
    • Performance: vLLM provides significantly higher throughput (RTFx 340+) compared to standard PyTorch generate() (RTFx ~20) due to optimized KV Cache management and batching.