WhisperLiveKit Documentation

To sync code, it is recommended to upload a compressed tarball. Avoid including large directories like runs/, data/, model checkpoints, or __pycache__ to save time and bandwidth.

Workflow:

1. Clean local __pycache__.
2. Create a tarball of specific source files and directories.
3. Use jl upload to move the tarball to the VM.
4. Use jl exec to extract the tarball into the target directory.

WhisperLiveKit provides several compatible interfaces for easy integration.

OpenAI-compatible REST API

Use standard curl commands to hit the transcription endpoint.

OpenAI Python SDK

Configure the OpenAI client to use the WhisperLiveKit local server.

Deepgram-compatible WebSocket

Point any Deepgram-compatible SDK to localhost:8000 to use the service.

Native WebSocket

Connect directly to the real-time streaming endpoint at ws://localhost:8000/asr.

# OpenAI-compatible REST API
curl http://localhost:8000/v1/audio/transcriptions -F file=@audio.wav

# Works with the OpenAI Python SDK
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

# Deepgram-compatible WebSocket (use any Deepgram SDK)
# Just point your Deepgram client at localhost:8000

# Native WebSocket for real-time streaming
ws://localhost:8000/asr

The AlignAtt MT engine requires a CUDA environment with approximately 40 GB of VRAM. Setup is performed within a clone of the Alignatt4LLM repository.

1. Run the bootstrap script to install the pinned vLLM stack and Python 3.13.
2. Accept the license for Gemma-4-E4B on Hugging Face.
3. Download the model using huggingface-cli.
4. Start the server using the alignatt-mt-server binary with a low-latency preset.

# In a clone of Alignatt4LLM
tools/bootstrap/setup_inference_qwen_asr_vllm.sh

huggingface-cli download google/gemma-4-E4B-it

.venv-inference/bin/alignatt-mt-server --preset gemma_low_latency --port 8765

Always pause your VM when GPU work is complete to avoid unnecessary costs. Use the --yes flag to bypass confirmation prompts.

Verification: After pausing, run jl list to ensure the instance status is Paused.

If you are using a diarization model that predicts up to 4 speakers but require a constrained output of exactly 2 speakers, use the SortFormer mapping algorithm. This transforms input predictions of shape (x, x, 4) into constrained predictions of shape (x, x, 2).

Algorithm Steps

1. Identify Top Speakers: For each time step, extract the top 2 speaker predictions using np.argsort.
2. Attribution Logic: Use the distance between detected speakers to map the 4 predictions to 2 attributed speakers (AS_0 and AS_1). If the distance between the first and second detected speakers is smaller than the distance to other candidates, they are likely the same speaker, and the attribution must be adjusted accordingly.

To achieve faster inference on Apple Silicon (M4/M3/etc.), use the MLX_WHISPER implementation via Simulstreaming. This approach decouples the encoder from the decoder, allowing you to load only the encoder weights for optimized frameworks.

On Apple Silicon M4, MLX_WHISPER significantly outperforms standard Whisper and Faster-Whisper:

• base.en: ~0.07s (vs 0.35s for standard Whisper)
• small: ~0.20s (vs 1.09s for standard Whisper)

For tiny.en using mlx_whisper, the memory footprint is reduced by loading only the necessary weights:

• Decoder weights: 59,110,771 bytes
• Encoder weights: 15,268,874 bytes

Voxtral Mini is a 4B-parameter model that handles 100+ languages with reliable automatic language detection. It uses its own streaming policy.

• Apple Silicon (MLX): Use the voxtral-mlx backend.
• Linux/GPU (Transformers): Use the voxtral backend.

# Apple Silicon (native MLX, recommended)
pip install -e ".[voxtral-mlx]"
wlk --backend voxtral-mlx

# Linux/GPU (HuggingFace transformers)
pip install transformers torch
wlk --backend voxtral

To resume a paused instance without interactive prompts, use the --yes flag.

Warning: The instance ID may change upon resumption (e.g., 420638 becomes 420777). Always run jl list after resuming to capture the new ID for subsequent commands.

Wait for SSH readiness: After resuming, wait 10-30 seconds before executing commands. You can validate readiness by checking the hostname and GPU status via jl exec.

When performing translation (e.g., using NLLB-200 models), choose your inference engine based on your system performance.

On MacBook M3 systems, benchmarks indicate that Standard Transformers are faster than CTranslate2 for NLLB-200-distilled-600M.

Recommendation:

• Use Standard Transformers if running on MacBook M3/M4.
• Ideally, look for an MLX implementation for optimal performance on Apple Silicon.

You can replace the in-process NLLB translator with the alignatt backend. This uses Alignatt4LLM as a sidecar, where a decoder-only LLM (defaulting to Gemma-4-E4B) drafts translations. The AlignAtt policy only commits target tokens when the ASR has actually committed the corresponding source words, ensuring an append-only output that never requires retraction.

To use this, pass --translation-backend alignatt to the wlk CLI and provide the URL of your running sidecar using --alignatt-url.

wlk --backend qwen3-streaming --language en \
    --qwen3-streaming-audio-backend causal \
    --target-language de \
    --translation-backend alignatt \
    --alignatt-url ws://gpu-host:8765

You can build your own client (browser, mobile, or desktop) to communicate with a running WhisperLiveKit server.

1. Start the server: Use the CLI or instantiate TranscriptionEngine manually.

   wlk --model small --language en --host 0.0.0.0 --port 9000

2. Connect via WebSocket: Open a connection to ws(s)://<host>:<port>/asr.
3. Send Audio: Send either MediaRecorder/Opus WebM blobs or raw PCM. If using raw PCM, ensure the server is started with the --pcm-input flag to instruct the client to use the AudioWorklet.
4. Consume Data: Parse the JSON payload as defined in the API documentation.