whisper.cpp

whisper.wasm is a WebAssembly (WASM) port of the whisper.cpp implementation that allows running OpenAI's Whisper ASR model directly inside a web browser.

Key Characteristics:

• Privacy: Audio data is processed locally on the user's machine and does not leave the computer.
• Performance: Achieves approximately 2x to 3x real-time transcription for tiny and base models on modern CPUs. Performance degrades for models larger than small.
• Capabilities: Supports both transcription and translation using the Greedy sampling strategy.
• Input Support: Supports loading audio from a file or recording via microphone (maximum 120 seconds).
• Requirements: Requires a browser that supports WASM SIMD 128-bit intrinsics.

Voice Activity Detection (VAD) can be enabled using the --vad argument in whisper-cli. This process first passes audio through a VAD model to detect speech segments, then only processes those segments through Whisper, which can significantly speed up transcription. A VAD model must be provided via the --vad-model (or -vm) flag.

./build/bin/whisper-cli --file ./samples/jfk.wav --model ./models/ggml-base.en.bin --vad --vad-model ./models/silero.bin

Guided mode allows you to provide a list of allowed command strings. The transcription process is then guided to classify the input into one of the strings from your list. This is highly efficient for devices that only need to recognize a specific subset of commands.

To use this mode, provide a text file containing the allowed commands using the -cmd flag.

# Run in guided mode, the list of allowed commands is in commands.txt
./whisper-command -m ./models/ggml-base.en.bin -cmd ./examples/command/commands.txt

# On Raspberry Pi, in guided mode you can use "-ac 128" for extra performance
./whisper-command -m ./models/ggml-tiny.en.bin -cmd ./examples/command/commands.txt -ac 128 -t 3 -c 0

chessboard.js is a standalone JavaScript chessboard component designed with a "just a board" API. It is responsible for the visual representation of a chessboard and does not contain any chess logic.

Key distinction:

• chessboard.js: Handles the UI/visual board.
• chess.js (recommended): Handles the game logic (legal moves, turn tracking, check/mate detection, PGN parsing).

Because chessboard.js does not understand how chess is played, you should pair it with a library like chess.js to implement game rules, move validation, or engine integration.

Whisper models are converted into a custom ggml binary format. This format packs all necessary components into a single file, including:

• model parameters
• mel filters
• vocabulary
• weights

Models can be downloaded via models/download-ggml-model.sh or manually from Hugging Face.

You can enable a sliding window mode combined with a basic Voice Activity Detector (VAD) by setting the --step argument to 0.

In this mode, the tool waits for speech activity before transcribing. When silence is detected, it transcribes the last --length milliseconds of audio, producing a transcription block suitable for parsing.

Use the -vth argument to adjust the VAD threshold: higher values make the detector more sensitive to silence (detecting silence more often). A value of 0.6 is a recommended starting point.

./build/bin/whisper-stream -m ./models/ggml-base.en.bin -t 6 --step 0 --length 30000 -vth 0.6

Models can be loaded using shorthand names for pre-converted models, local file paths, or URIs.

• Pre-converted models: Use shorthand like "tiny", "base.en", etc. You can see available keys via Whisper::Model.pre_converted_models.keys.
• Local files: Pass the path to a .bin file.
• Remote files: Pass a string URI or a URI object; the gem will download it automatically on the first use.

To clear the downloaded cache for a specific model, use #clear_cache on the model object.

# Use shorthand
whisper = Whisper::Context.new("base.en")

# Use local path
whisper = Whisper::Context.new("path/to/your/model.bin")

# Use URI (auto-downloads)
whisper = Whisper::Context.new("https://example.net/uri/of/your/model.bin")

# Clear cache
Whisper::Model.pre_converted_models["base"].clear_cache

For a full voice-to-voice experience, the tool requires a TTS engine to convert LLaMA's text responses into audio.

By default, the tool is configured to use:

• macOS: say command
• Windows: SpeechSynthesizer

You can use any other TTS engine by editing the speak script provided in the example directory to suit your requirements.

The whisper-stream tool requires the SDL2 library to capture audio from the microphone. Follow these steps to install dependencies and build the project using CMake.

1. Install SDL2

Debian-based Linux:

sudo apt-get install libsdl2-dev

Fedora Linux:

sudo dnf install SDL2 SDL2-devel

macOS:

brew install sdl2

2. Build with CMake

Enable the WHISPER_SDL2 option during configuration to ensure audio capture support is included.

cmake -B build -DWHISPER_SDL2=ON
cmake --build build --config Release

Once installed and configured, you can use speech-to-text with the following workflow:

1. Trigger Transcription: Press Ctrl-G in INSERT, VISUAL, or NORMAL mode.
2. Speak: Say your desired text.
3. Finish: Press Ctrl-C to end the transcription.

The transcribed text will be automatically inserted under your cursor. The underlying mechanism works by running the whisper.nvim script, which calls the stream binary. The transcription results are continuously written to /tmp/whisper.nvim, and the Neovim mapping extracts the final line from that file to insert it into your buffer.