DOCSEARCHbyAlgolia
Submit Docs

sherpa

repository·master·Indexed 21 days ago

https://github.com/k2-fsa/sherpa

An open-source speech-to-text inference framework built on PyTorch, specialized in deploying end-to-end (E2E) transducer and CTC-based models. It provides C++ and Python interfaces for speech transcription and supports integration with NVIDIA Triton Inference Server, TensorRT, and TensorRT-LLM. The framework includes tools for converting OpenAI Whisper models, deploying pyannote segmentation models, and a web UI for upload, streaming, and offline recording.

Tokens
330.7K
Snippets
1K
Records
1.3K
Agent score
73%

What's inside sherpa

  1. Overview of the sherpa inference framework

    master

    sherpa is an open-source speech-to-text inference framework built on PyTorch. It is designed exclusively for end-to-end (E2E) models, specifically transducer- and CTC-based models.

    Key characteristics:

    • Focus: Deployment of pre-trained models for speech transcription (not for training or fine-tuning).
    • APIs: Provides both C++ and Python interfaces.
    • Model Types: Supports transducer-based and CTC-based E2E models.

    If you need to train or fine-tune models, use the icefall repository instead.

  2. Overview of FunASR Nano models

    master

    FunASR Nano models (specifically Fun-ASR-Nano-2512) are speech recognition models converted to ONNX format for use with sherpa-onnx.

    Supported Languages:

    • Chinese (including 7 dialects like Wu, Cantonese, Min, etc., and 26+ regional accents)
    • English (including various regional accents)
    • Japanese (including various regional accents)

    Specialized Capabilities:

    • Lyric recognition
    • Rap speech recognition

    Supported Platforms via sherpa-onnx:

    • Linux: x64, aarch64, arm, riscv64
    • macOS: x64, arm64
    • Windows: x64, x86, arm64
    • Android: arm64-v8a, armv7-eabi, x86, x86_64
    • iOS: arm64

  3. Overview of sherpa-ncnn for local speech recognition

    master
    sherpa-ncnn is a self-contained implementation that uses the ncnn framework for neural network computation instead of PyTorch. This allows for speech recognition to be performed entirely locally on a device without requiring internet access. The project is designed to be compiled from source and supports a wide range of platforms including Linux, macOS, Windows, embedded systems, Android, and iOS.

  4. Overview of VITS pre-trained models

    master

    VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) is a supported model architecture in sherpa-onnx. There are over 100 pre-trained models available across more than 40 languages.

    Key resources:

    • Model Repository: Find all pre-trained models at sherpa-onnx releases.
    • Interactive Demo: Try models in a HuggingFace Space.
    • Android Support: Android APKs for various models are available here (refer to the APK documentation page).

  5. Overview of sherpa-onnx

    master

    sherpa-onnx

    sherpa-onnx is a self-contained project that uses onnxruntime to perform neural network computations, serving as a replacement for PyTorch.

    Key Features:

    • Local Processing: During speech recognition, no internet access is required; all processing happens locally on the device.
    • Self-Contained: The entire project can be compiled from source.
    • Cross-Platform Support: It can be built for Linux, macOS, Windows, embedded systems, Android, and iOS.

    To use sherpa-onnx, you can either use pre-trained models or export your own models from icefall to the onnx format.

  7. Overview of sherpa speech recognition

    master

    sherpa is a speech recognition framework that uses PyTorch for neural network computation. A key feature is that it performs all speech recognition processing locally on your device; it does not require internet access during runtime.

    If you are developing custom models, you must export them using TorchScript (via torch.jit.script) to be compatible with the framework. Detailed instructions for model export can be found in the icefall documentation.

    https://k2-fsa.github.io/icefall/model-export/export-with-torch-jit-script.html

  9. Use WebSocket for real-time speech recognition with sherpa-onnx

    master

    The sherpa-onnx project provides both a WebSocket server and client implementation to enable real-time speech recognition. The server is implemented in C++ using websocketpp and asio, and notably does not depend on boost.

    There are two primary modes of operation available via WebSocket:

    1. Online WebSocket: For streaming, real-time speech recognition.
    2. Offline WebSocket: For processing pre-recorded audio files.

    Refer to the specific guides for online-websocket or offline-websocket to implement your specific use case.

  10. Use Conformer transducer based non-streaming ASR

    master

    Sherpa provides non-streaming Automatic Speech Recognition (ASR) using Conformer transducer models. This allows for offline transcription of audio files.

    There are two primary ways to interact with these models:

    1. Server/Client Model: Running a server that hosts the model and a client that sends audio for transcription.
    2. Pre-trained Models: You can use pre-trained models for specific languages/datasets without performing any training yourself.

    Supported datasets for demonstration include:

    • aishell: A Chinese dataset using Chinese characters as modeling units (vocabulary size: 4336).
    • LibriSpeech: An English dataset using Byte Pair Encoding (BPE) as modeling units (vocabulary size: 500).

  11. Compare sherpa sub-projects to choose a deployment target

    master

    The sherpa ecosystem consists of three main sub-projects tailored for different deployment needs, hardware constraints, and language bindings. Choose the one that best fits your target platform and required features:

    k2-fsa/sherpa

    • Best for: High-performance research or server-side deployment where PyTorch is available.
    • NN Library: PyTorch.
    • Hardware: x86, x86_64 (with CUDA support for NVIDIA GPUs).
    • OS: Linux, Windows, macOS.
    • Capabilities: Supports batch sizes > 1; streaming and non-streaming ASR.
    • APIs: C++, Python.
    • Installation Difficulty: Hard.

    k2-fsa/sherpa-onnx

    • Best for: Cross-platform deployment with wide language support and mobile/edge compatibility.
    • NN Library: onnxruntime.
    • Hardware: x86, x86_64, arm32, arm64, RISC-V (with CUDA support for NVIDIA GPUs).
    • OS: Linux, Windows, macOS, iOS, Android.
    • Capabilities: Supports batch sizes > 1; streaming ASR, non-streaming ASR, text-to-speech, speaker diarization, speaker identification, speaker verification, spoken language identification, speech denoising, speech enhancement, audio tagging, VAD, and keyword spotting. Also supports RKNN.
    • APIs: C, C++, Python, C#, Java, Kotlin, Swift, Go, JavaScript, Dart, Pascal, Rust.
    • Installation Difficulty: Easy.

    k2-fsa/sherpa-ncnn

    • Best for: Extremely lightweight edge deployment on ARM/RISC-V devices.
    • NN Library: ncnn.
    • Hardware: x86, x86_64, arm32, arm64, RISC-V (with Vulkan support for ARM GPUs).
    • OS: Linux, Windows, macOS, iOS, Android.
    • Capabilities: Does not support batch sizes > 1; streaming ASR, non-streaming ASR, text-to-speech, and VAD. Also supports RKNN.
    • APIs: C, C++, Python, C#, Kotlin, Swift, Go.
    • Installation Difficulty: Super easy.