DOCSEARCHbyAlgolia
Submit Docs

CrispASR Documentation

repository·main·Indexed 19 days ago

https://github.com/crispstrobe/crispasr

A lightweight on-device speech recognition interface supporting Whisper, Qwen3-ASR, and FastConformer. It provides a safe Rust wrapper and multiple language bindings, including Go, Java (JNI), Node.js, and Ruby, for performing speech-to-text transcription and Voice Activity Detection (VAD).

Tokens
401.2K
Snippets
1.2K
Records
1.9K
Agent score
66%

What's inside CrispASR

  1. Overview of Text-to-Speech (TTS) engines in CrispASR

    main
    CrispASR provides access to fourteen different open-weights TTS engines within the crispasr binary. Each engine offers a different trade-off between voice quality, language support, and computational footprint. Engines vary in their ability to perform voice cloning (zero-shot or via presets) and their output sample rates.

  2. Overview of CrispASR Space capabilities

    main

    The CrispASR Space is a Gradio-wrapped HTTP server designed for Hugging Face Docker Spaces. It runs a C++ inference engine on port :8080 and a Gradio UI on port :7860.

    Exposed Services

    FeatureBackend(s)API EndpointNotes
    Transcribe (ASR)whisper, parakeet, moonshine, moonshine-de, wav2vec2 (EN+DE), parakeet-ctc-0.6b, cohere, qwen3POST /v1/audio/transcriptionsFootprint: 37 MB – 550 MB per model
    Speak (TTS)kokoro (82M, multilingual)POST /v1/audio/speech, GET /v1/voicesFootprint: ~85 MB
    Detect language (text)CLD3, GlotLID-V3, LID-176N/A (uses crispasr-lid subprocess)Footprint: 440 KB – 250 MB
    System Infostatic capability tableGET /backends, GET /health—

    Model Management

    Models are hot-swapped via POST /load. Only one model is resident in memory at a time; switching backends triggers a download (on first use) and a load operation.

    Note on Large Models: Large speech-LLM backends (e.g., Voxtral 2.5 GB, MiMo-ASR 4.5 GB, Granite-4.1 3 GB, omniasr-llm) are omitted from the default demo to avoid exceeding the 16 GB RAM ceiling. To use these, you must build the image locally.

  3. Overview of supported TTS architectures

    main

    CrispASR supports a variety of Text-to-Speech (TTS) backends, each with different characteristics:

    • MeloTTS: VITS2 architecture (~52M params). Supports 4 English speakers and optional BERT conditioning via melotts_load_bert() to improve contextual disambiguation.
    • Piper: VITS architecture (~60M params). Lightweight and fast, using espeak-ng for phonemization.
    • IndexTTS-1.5: GPT-2 AR mel-code generator + BigVGAN vocoder. Supports zero-shot voice cloning from a reference WAV.
    • OuteTTS: OLMo-0.5B LM backbone + WavTokenizer codec. Generates speech tokens autoregressively.
    • VoxCPM2: Qwen2-2B backbone. Two-stage process (AR text-to-semantic-tokens followed by flow matching). Supports zero-shot voice cloning.
    • CosyVoice3: Three-stage pipeline (Qwen2-0.5B AR $\rightarrow$ DiT flow matching $\rightarrow$ HiFT vocoder). Supports 9 languages and 18 Chinese dialects.
    • KugelAudio: Hybrid AR + diffusion based on VibeVoice. Uses a Qwen2.5-7B LM and a 4-layer DiT prediction head.
    • Pocket-TTS: Llama-1B backbone generating Mimi RVQ codec tokens. Designed for streaming.
    • F5-TTS: Diffusion Transformer (DiT) for flow-matching. Supports zero-shot voice cloning.
    • LFM2-Audio: End-to-end multimodal (ASR + TTS + S2S) using a 1.5B LFM backbone.

  4. Overview of crispasr.wasm

    main

    The crispasr.wasm example provides a WebAssembly (WASM) port of the CrispASR implementation of OpenAI's Whisper ASR model. This allows for local, browser-based audio transcription and translation where audio data never leaves the user's machine.

    Key Characteristics:

    • Privacy: All processing is done locally on the client machine.
    • Performance: On modern CPUs/browsers, tiny and base models can achieve ~2x to 3x real-time performance (e.g., transcribing 60s of audio in 20-30s).
    • Requirements: Requires a browser that supports WASM SIMD 128-bit intrinsics.
    • Capabilities: Supports all models up to small size, both transcription and translation, and the Greedy sampling strategy.
    • Limitations: Maximum audio length is 120 seconds.

  5. Use GigaAM-v3 for Russian ASR

    main

    GigaAM-v3 is a 16-layer rotary Conformer (220M parameters) designed for Russian Automatic Speech Recognition (ASR). It supports both CTC and RNN-T heads.

    Model Variants & Quantization:

    • GGUF models are available in f16, q8_0, and q4_k variants.
    • The registry defaults to q8_0 to balance transcript accuracy and file size (~249 MB).
    • The quantizer preserves joint.*, decoder.*, and head.ctc.* at source precision to ensure head stages are unaffected by encoder quantization.

    Performance Note: On M1 Metal, e2e_rnnt at q8_0 achieves approximately 43.4× realtime, while ctc at q8_0 achieves 35.0× realtime.

  6. License and attribution for Canary CTC aligner

    main

    The Canary-1B-v2 CTC aligner GGUF files are licensed under CC-BY-4.0, inherited from the original nvidia/canary-1b-v2 model.

    Important: Use of these GGUF files must comply with the CC-BY-4.0 license, which includes mandatory attribution to the NVIDIA NeMo team.

    Note that while the model weights are CC-BY-4.0, the conversion and runtime code is licensed under MIT (matching the crispasr base).

  7. HTDemucs Parity Validation Overview

    main

    HTDemucs (Meta Demucs v4) is a model with 42M parameters and 533 tensors. Parity validation ensures that the C++ runtime implementation produces outputs that are mathematically identical (via cosine similarity) to the original Python reference implementation.

    Successful validation is defined by:

    • Full End-to-End F32 Parity: All 45 stages passing with cosine similarity $\ge 0.999989$.
    • Decoded-output Roundtrip: The ASR (Automatic Speech Recognition) of the separated vocals stem must be byte-identical to the ASR of the original mix.
    • F16 Verification: All 45 stages passing with a minimum cosine similarity of $0.999961$.

  8. What is the Qwen3-TTS Tokenizer 12Hz?

    main

    The Qwen3-TTS-Tokenizer-12Hz is a separate speech tokenizer and codec used by the Qwen3-TTS family within the CrispASR ecosystem. It provides a 12.5 Hz, 16-codebook speech representation.

    Key Functions:

    • Encoding: Converts reference audio into ref_code.
    • Baking: Used for baking or loading voice-pack GGUFs.
    • Decoding: Converts generated codes back into 24 kHz mono WAV audio.

    Supported Languages: zh, en, ja, ko, de, fr, ru, pt, es, it.

  9. What is Voxtral-Mini-3B-2507?

    main

    Voxtral-Mini-3B is a 3-billion parameter speech-LLM based on the Mistral Ministral 3B backbone. Unlike standard Whisper models that use a dedicated decoder, Voxtral uses a projector to feed audio embeddings into a general-purpose Llama 3 LLM. This architecture enables advanced capabilities beyond simple transcription, such as:

    • Audio Q&A and Summarization: Ask questions about the audio content directly.
    • Function Calling: Trigger backend functions based on spoken intents.
    • Long-form Context: Supports up to 30 minutes of audio for transcription and 40 minutes for audio understanding.
    • Multilingual Support: Natively handles English, French, German, Spanish, Italian, Portuguese, Dutch, and Hindi with automatic detection.

  10. What is Tiron (Multi-speaker Meeting ASR)

    main

    Tiron is a specialized Whisper large-v3 architecture checkpoint designed for joint transcription and speaker attribution in a single decode pass. Unlike standard Whisper, it is fine-tuned to emit inline speaker turn markers (<|speakerN|>) and timestamps (<|t.tt|>) directly within the transcript.

    Key Characteristics:

    • Architecture: Whisper large-v3 (1280 d_model, 32 encoder/decoder layers, 20 heads, 128 mel bins).
    • Output Format: An inline transcript containing speaker markers and 20ms timestamps. Example: <|speaker1|><|0.00|> Thanks everyone for joining.<|2.96|><|3.52|> Let's get started.<|4.80|><|speaker2|><|2.98|> Morning!<|3.40|>
    • Speaker Scope: By default, speaker indices are local to each 30s window (e.g., the first person to speak in any window is <|speaker1|>).
    • License: Apache-2.0.

  11. Overview of Backend Architectures

    main

    CrispASR supports a wide variety of backend architectures for ASR, TTS, and translation. Understanding the architecture family helps determine performance characteristics (speed vs. accuracy) and resource requirements (CPU vs. GPU).

    Architecture Families

    • Feedforward CTC (e.g., wav2vec2, omniasr-CTC, fc-ctc, firered-asr): No decoder and no KV cache. These are the fastest backends but do not provide native punctuation.
    • Encoder-decoder (e.g., whisper, canary, cohere, moonshine): Uses cross-attention KV cache and an autoregressive text decoder.
    • Audio-LLM (e.g., granite, voxtral, qwen3, glm-asr, gemma4-e2b): Audio features are injected into the LLM embedding space, utilizing KV-cached autoregressive decoding.
    • Transducer (e.g., parakeet, reazonspeech): Uses an LSTM predictor and joint network with frame-synchronous TDT/RNNT decoding. Supports greedy decoding (default), label-looping beam search (-bs N), and MAES (Modified Adaptive Expansion Search) via CRISPASR_PARAKEET_MAES=1.
    • Codec + LM (e.g., kyutai-stt): Uses a neural audio codec (RVQ) followed by a token-based LM.
    • TTS Pipelines:
      • Discrete-token codec + vocoder: Talker LM emits codec tokens, and a separate GGUF decoder renders audio (e.g., qwen3-tts, orphus).
      • Flow-matching mel + iSTFT vocoder: Emits mel-spectrograms via flow-matching, then renders audio via HiFTGenerator (e.g., chatterbox).
      • Realtime σ-VAE: Uses a base LM, TTS LM, and σ-VAE decoder (e.g., vibevoice).
      • StyleTTS2 / iSTFTNet: Single-shot, non-autoregressive (e.g., kokoro).
    • Text-to-text translation (e.g., m2m100, t5_translate, madlad): Uses transformer encoder-decoder architectures. Driven by --text "..." -sl <src> -tl <tgt>.

  12. Compare MAES vs Greedy decoding performance

    main

    MAES is a 'Pareto improvement' over standard beam search for transducers. While it is slower than greedy decoding, it significantly improves accuracy for certain model types and audio qualities.

    Performance Characteristics:

    • Accuracy: Highly effective at recovering full sentences and correcting factual errors in smaller models (e.g., tdt-0.6b-v2 with 1K vocab). It provides diminishing returns on larger models (1.1b) or larger vocabularies (8K) where the greedy baseline is already strong.
    • Speed: On CPU, MAES is approximately 35% slower than greedy decoding for 60s audio. The overhead is due to extra predictor LSTM and joint forward calls per non-blank expansion.
    • GPU Impact: On GPU, the decoding time is a tiny fraction of the total time (as the encoder dominates), making the impact of MAES negligible.
    • Model Suitability: MAES helps most on smaller models with smaller vocabularies. It cannot fix errors caused by insufficient model capacity (e.g., very small 110M models).