DOCSEARCHbyAlgolia
Submit Docs

sherpa-rs

repository·main·Indexed 43 days ago

https://github.com/thewh1teagle/sherpa-rs

Rust bindings to the sherpa-onnx engine, providing capabilities for speech processing tasks including speech-to-text (STT), text-to-speech (TTS), voice activity detection (VAD), speaker diarization, audio tagging, and keyword spotting. Version 0.6.8. Note: This crate is deprecated in favor of the official sherpa-onnx Rust API.

Tokens
12.4K
Snippets
45
Records
53
Agent score
54%

What's inside sherpa-rs

  1. Overview of sherpa-rs features

    main

    The sherpa-rs crate provides Rust bindings for various speech and audio processing tasks, including:

    • Spoken language detection
    • Speaker embedding (labeling)
    • Speaker diarization
    • Speech to text
    • Text to speech
    • Text punctuation
    • Voice activity detection
    • Audio tagging
    • Keyword spotting

  2. Build sherpa-rs from source

    main

    First, prepare the repository by cloning it recursively:

    git clone --recursive https://github.com/thewh1teagle/sherpa-rs
cd sherpa-rs

    Then, perform a standard release build:

    cargo build --release
    git clone --recursive https://github.com/thewh1teagle/sherpa-rs
cd sherpa-rs
cargo build --release

  3. Build for Android

    main

    To build for Android, you must have the NDK installed via Android Studio.

    rustup target add aarch64-linux-android
cargo install cargo-ndk
export NDK_HOME="$HOME/Library/Android/sdk/ndk/27.0.12077973" # Adjust path as needed
cargo ndk -t arm64-v8a build --release
    rustup target add aarch64-linux-android
cargo install cargo-ndk
export NDK_HOME="$HOME/Library/Android/sdk/ndk/27.0.12077973"
cargo ndk -t arm64-v8a build --release

  4. Install prerequisites for building sherpa-rs

    main

    Before building, ensure you have Cargo, Clang, and Cmake installed. Depending on your operating system, use the following commands:

    Linux

    sudo apt-get update
sudo apt-get install -y pkg-config build-essential clang cmake

    Windows

    Use winget to install necessary utilities:

    winget install -e --id GnuWin32.Tar
winget install -e --id JernejSimoncic.Wget

    Note: When running examples on Windows, use wget.exe instead of wget.

    sudo apt-get update
sudo apt-get install -y pkg-config build-essential clang cmake

  5. Build for iOS

    main

    Ensure Xcode command line tools are installed via xcode-select --install.

    1. Install the required toolchains:
    # iOS
rustup target add aarch64-apple-ios
# Intel chip emulator
rustup target add x86_64-apple-ios
# Apple chip emulator
rustup target add aarch64-apple-ios-sim
    1. Build for iOS:
    export RUSTFLAGS="-C link-arg=-fapple-link-rtlib"
cargo build --release --target aarch64-apple-ios
    export RUSTFLAGS="-C link-arg=-fapple-link-rtlib"
cargo build --release --target aarch64-apple-ios

  6. Fast compilation using prebuilt sherpa-onnx binaries

    main

    You can bypass the automatic download and build of sherpa-onnx by manually providing the library path via the SHERPA_LIB_PATH environment variable.

    Important Notes:

    • To link sherpa-onnx libraries dynamically, set SHERPA_BUILD_SHARED_LIBS=1.
    • Disable rust-analyzer during this process to prevent it from triggering long rebuilds on every save.
    • On Linux, when linking statically, set RUSTFLAGS="-C relocation-model=dynamic-no-pic".

    macOS (arm64/x86-64)

    wget https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.10.28/sherpa-onnx-v1.10.28-osx-universal2-static.tar.bz2
tar xf sherpa-onnx-v1.10.28-osx-universal2-static.tar.bz2
export SHERPA_LIB_PATH="$(pwd)/sherpa-onnx-v1.10.28-osx-universal2-static"
cargo build

    Windows (x86-64)

    wget.exe https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.10.28/sherpa-onnx-v1.10.28-win-x64-static.tar.bz2
tar.exe xf sherpa-onnx-v1.10.28-win-x64-static.tar.bz2
$env:SHERPA_LIB_PATH="$pwd/sherpa-onnx-v1.10.28-win-x64-static"
cargo build

    Linux (x86-64)

    wget https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.10.28/sherpa-onnx-v1.10.28-linux-x64-static.tar.bz2
tar xf sherpa-onnx-v1.10.28-linux-x64-static.tar.bz2
export SHERPA_LIB_PATH="$(pwd)/sherpa-onnx-v1.10.28-linux-x64-static"
export RUSTFLAGS="-C relocation-model=dynamic-no-pic"
cargo build
    export SHERPA_LIB_PATH="$(pwd)/sherpa-onnx-v1.10.28-linux-x64-static"
export RUSTFLAGS="-C relocation-model=dynamic-no-pic"
cargo build

  7. Resample WAV files to 16kHz

    main

    Sherpa-rs requires audio files to be resampled to 16kHz, mono, PCM 16-bit LE format. Use ffmpeg to prepare your files:

    ffmpeg -i <input_file> -ar 16000 -ac 1 -c:a pcm_s16le <output_file>
    ffmpeg -i <file> -ar 16000 -ac 1 -c:a pcm_s16le <out>

  8. Prepare the punctuation model for Android

    main

    To use the punctuation model in the Tauri Android application, you must download the model, extract the .onnx file, and push it to the device. Note that the APK currently has a hardcoded path for the model location on the Android filesystem.

    1. Download and extract the model.
    2. Move model.onnx to the root of the src-tauri directory.
    3. Push the model to the device using adb to the path /data/local/tmp/model.onnx.
    cd src-tauri
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/punctuation-models/sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12.tar.bz2
tar xvf sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12.tar.bz2
mv sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12/model.onnx model.onnx
rm -rf sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12
rm sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12.tar.bz2
adb push model.onnx /data/local/tmp/model.onnx

  9. Build the Tauri + sherpa-rs Android application

    main

    Building the application requires setting up Android environment variables, installing UI dependencies via bun, and compiling the Rust core using cargo ndk. You must also manually link the shared libraries (.so files) into the Android jniLibs directory so the app can access onnxruntime and the sherpa-onnx-c-api.

    # 1. Setup environment variables
export JAVA_HOME="/Applications/Android Studio.app/Contents/jbr/Contents/Home"
export ANDROID_HOME="$HOME/Library/Android/sdk"
export NDK_HOME="$HOME/Library/Android/sdk/ndk/27.0.12077973"

# 2. Setup UI
bun install
bunx tauri icon src-tauri/icons/icon.png

# 3. Build Rust core for Android
cd src-tauri
export CARGO_TARGET_DIR="$(pwd)/target"
cargo ndk -t arm64-v8a build

# 4. Link shared libraries to jniLibs
mkdir -p gen/android/app/src/main/jniLibs/arm64-v8a
ln -s $(pwd)/target/aarch64-linux-android/debug/libonnxruntime.so $(pwd)/gen/android/app/src/main/jniLibs/arm64-v8a/libonnxruntime.so
ln -s $(pwd)/target/aarch64-linux-android/debug/libsherpa-onnx-c-api.so $(pwd)/gen/android/app/src/main/jniLibs/arm64-v8a/libsherpa-onnx-c-api.so

# 5. Run development server
bun run tauri android dev

  12. Available TTS engines in sherpa-rs

    main

    The sherpa-rs crate provides several text-to-speech engine implementations. Each engine has its own specific TTS struct and configuration struct:

    • Kitten: KittenTts, KittenTtsConfig
    • Kokoro: KokoroTts, KokoroTtsConfig
    • Matcha: MatchaTts, MatchaTtsConfig
    • Vits: VitsTts, VitsTtsConfig
    • ZipVoice: ZipVoiceTts, ZipVoiceTtsConfig