whisper-rs Documentation

repository·master·Indexed 21 days ago

https://github.com/tazz4843/whisper-rs

Rust bindings for whisper.cpp, enabling high-performance speech-to-text transcription. Version 0.14.3 provides tools for loading WhisperContext, managing transcription states, and converting audio data. It supports hardware acceleration via feature flags for CUDA, Metal, Vulkan, OpenBLAS, and hipBLAS, and includes utilities for audio sample conversion and system information inspection.

Tokens
8K
Snippets
28
Records
39
Agent score
73%

What's inside whisper-rs

  1. Quickstart with whisper-rs examples

    master

    To quickly test the library, you can clone the repository and run the provided examples. Note that you will need a model file to pass as an argument.

    git clone --recursive https://github.com/tazz4843/whisper-rs.git
    cd whisper-rs
    
    # Run the basic usage example
    cargo run --example basic_use
    
    # Run the audio transcription example
    cargo run --example audio_transcription
    git clone --recursive https://github.com/tazz4843/whisper-rs.git
    cd whisper-rs
    cargo run --example basic_use
    cargo run --example audio_transcription
  2. Build whisper-rs on Windows using MSYS2

    master

    Follow these steps to build using the MSYS2 toolchain:

    1. Install MSYS2/MinGW following the standard setup.
    2. Install the toolchain in MSYS2 ucrt64: pacman -S --needed base-devel mingw-w64-x86_64-toolchain
    3. Add C:\msys64\ucrt64\bin to your system PATH.
    4. Install make in MSYS2 ucrt64: pacman -S make
    5. Install the GNU toolchain for Rust in Windows PowerShell/Cmd: rustup toolchain install stable-x86_64-pc-windows-gnu
    6. Create a .cargo/config.toml file in the project root with the following configuration to point to the MSYS2 binaries:
    [target.x86_64-pc-windows-gnu]
    linker = "C:\\msys64\\ucrt64\\bin\\gcc.exe"
    ar = "C:\\msys64\\ucrt64\\bin\\ar.exe"
    1. Run cargo run in Windows PowerShell/Cmd.
  3. Build whisper-rs on M1 OSX

    master

    To build on M1 Mac (Apple Silicon), you must install cmake (e.g., via Homebrew) and configure your .cargo/config.toml to include the necessary framework flags for the aarch64-apple-darwin target.

    brew install cmake
    [target.aarch64-apple-darwin]
    rustflags = "-lc++ -l framework=Accelerate"
  4. Build whisper-rs on Windows with CUDA enabled

    master

    To enable CUDA support on Windows, perform the following setup:

    1. Download and install CUDA.
    2. Download Visual Studio with Desktop C++ and Clang enabled.
    3. Download CLANG.
    4. Download CMAKE.
    5. Locate your clang installation using where.exe clang.
    6. Set the LIBCLANG_PATH environment variable to your clang bin directory (e.g., setx LIBCLANG_PATH "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\x64\bin").
    7. Restart your shell to apply changes.
    8. Run cargo build.
  5. How the manual transcription pipeline works (PCM -> Encode -> Decode)

    master

    If you need granular control over the transcription process, you can manually step through the pipeline using WhisperState. The sequence must be:

    1. Prepare Spectrogram: Call pcm_to_mel (converts raw PCM to log mel spectrogram) or set_mel (if you already have a spectrogram).
    2. Encode: Call encode to run the Whisper encoder on the stored spectrogram.
    3. Decode: Call decode to run the decoder and obtain logits/probabilities for the next token.

    This approach is useful for streaming or custom decoding logic.

    // 1. Convert PCM to Mel
    state.pcm_to_mel(&pcm_data, threads)?;
    
    // 2. Run Encoder
    state.encode(0, threads)?;
    
    // 3. Run Decoder
    state.decode(&tokens, n_past, threads)?;
  6. Configure whisper-rs feature flags

    master

    Use feature flags to enable hardware acceleration or logging backends. All features are disabled by default.

    FeatureDescription
    raw-apiExpose whisper-rs-sys directly. Warning: May break semver compliance in patch releases.
    cudaEnable CUDA support (implicitly enables GPU flag at runtime).
    hipblasEnable ROCm/hipBLAS support (Linux only; implicitly enables GPU flag).
    openblasEnable OpenBLAS support.
    metalEnable Metal support (macOS; implicitly enables GPU flag).
    vulkanEnable Vulkan support (implicitly enables GPU flag).
    log_backendHook into whisper.cpp logs and send to the log backend.
    tracing_backendHook into whisper.cpp logs and send to the tracing backend.
  7. Choose a SamplingStrategy for decoding

    master

    When creating FullParams, you must specify a SamplingStrategy to determine how the model selects tokens during decoding.

    Available strategies:

    • SamplingStrategy::Greedy { best_of: c_int }: Selects the most likely token at each step. best_of determines how many candidates are considered.
    • SamplingStrategy::BeamSearch { beam_size: c_int, patience: c_float }: Explores multiple paths. Note that patience is currently not implemented in whisper.cpp (v1.2.0).
    // Greedy sampling
    let greedy_params = SamplingStrategy::Greedy { best_of: 1 };
    
    // Beam search sampling
    let beam_params = SamplingStrategy::BeamSearch { beam_size: 5, patience: 1.0 };
  8. Install Whisper-level logging hooks

    master

    To capture logs emitted by the underlying Whisper engine (via whisper.cpp), call install_whisper_logging_hook(). This function sets up a global trampoline that redirects Whisper's internal logging to the Rust logging ecosystem.

    Note that the actual logging behavior depends on which backend features are enabled in your Cargo.toml:

    • If log_backend is enabled, logs are routed to the log crate.
    • If tracing_backend is enabled, logs are routed to the tracing crate.
    • If neither is enabled, the logging calls effectively become no-ops.
  9. Fix binding generation panics during build

    master

    If you encounter a panic during the binding generation phase of the build, you can skip the generation process and use the existing pre-packaged bindings by setting the WHISPER_DONT_GENERATE_BINDINGS environment variable to 1.

    WHISPER_DONT_GENERATE_BINDINGS=1 cargo build
  10. Basic usage of whisper-rs for transcription

    master

    To perform transcription, follow these steps:

    1. Load a WhisperContext using WhisperContext::new_with_params and a path to a model file.
    2. Create a FullParams object using a SamplingStrategy (e.g., SamplingStrategy::Greedy).
    3. Create a state using ctx.create_state().
    4. Run the model on audio data (32-bit floating point samples, 16KHz, mono) using state.full(params, &audio_data).
    5. Iterate through segments using state.full_n_segments() and retrieve text and timestamps via full_get_segment_text, full_get_segment_t0, and full_get_segment_t1.
    use whisper_rs::{WhisperContext, WhisperContextParameters, FullParams, SamplingStrategy};
    
    fn main() {
    	let path_to_model = std::env::args().nth(1).unwrap();
    
    	// load a context and model
    	let ctx = WhisperContext::new_with_params(
    		path_to_model,
    		WhisperContextParameters::default()
    	).expect("failed to load model");
    
    	// create a params object
    	let params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
    
    	// assume we have a buffer of audio data
    	// here'll make a fake one, floating point samples, 32 bit, 16KHz, mono
    	let audio_data = vec![0_f32; 16000 * 2];
    
    	// now we can run the model
    	let mut state = ctx.create_state().expect("failed to create state");
    	state
    		.full(params, &audio_data[..])
    		.expect("failed to run model");
    
    	// fetch the results
    	let num_segments = state
    		.full_n_segments()
    		.expect("failed to get number of segments");
    	for i in 0..num_segments {
    		let segment = state
    			.full_get_segment_text(i)
    			.expect("failed to get segment");
    		let start_timestamp = state
    			.full_get_segment_t0(i)
    			.expect("failed to get segment start timestamp");
    		let end_timestamp = state
    			.full_get_segment_t1(i)
    			.expect("failed to get segment end timestamp");
    		println!("[{} - {}]: {}", start_timestamp, end_timestamp, segment);
    	}
    }