Delayed Streams Modeling (DSM)

Kyutai STT models are optimized for real-time streaming inference, support batching for high efficiency, and provide word-level timestamps.

Available models:

• kyutai/stt-1b-en_fr: English and French, ~1B parameters, 0.5s delay, includes semantic Voice Activity Detection (VAD).
• kyutai/stt-2.6b-en: English-only, ~2.6B parameters, 2.5s delay.

The Rust implementation is designed for production streaming via websockets.

1. Install the server crate (requires CUDA feature for GPU support):

cargo install --features cuda moshi-server

2. Start the server using the appropriate config file:

• For stt-1b-en_fr: configs/config-stt-en_fr.hf.toml
• For stt-2.6b-en: configs/config-stt-en-hf.toml

moshi-server worker --config configs/config-stt-en_fr-hf.toml

3. Transcribe audio:

• From microphone: uv run scripts/stt_from_mic_rust_server.py
• From file: uv run scripts/stt_from_file_rust_server.py audio/bria.mp3 (use --rtf 1000 to process as fast as possible).

Select an implementation based on your target environment:

• PyTorch: Best for research, tinkering, and Python-based experimentation.
• Rust: Best for production environments. Provides a robust server with streaming access over websockets.
• MLX: Best for on-device inference on Apple silicon (Mac and iPhone).

For on-device inference on Mac or iPhone, use the moshi-mlx package (version 0.2.6 or later).

Run inference on a file:

python -m moshi_mlx.run_inference --hf-repo kyutai/stt-2.6b-en-mlx audio/bria.mp3 --temp 0

Run inference from microphone:

python scripts/stt_from_mic_mlx.py

Note: MLX models can also be used in Swift via the moshi-swift codebase.

To set up the development environment with pre-commit hooks, run the following commands:

pip install pre-commit
pre-commit install

If you are using uv, you can use:

uvx pre-commit install

The PyTorch implementation is intended for research and experimentation. You can use the tts_pytorch.py script for standard generation (waits for all text before starting) or tts_pytorch_streaming.py for a fully streaming implementation.

Requirements:

• Requires the moshi package.
• You can install it via pip install moshi or use uvx --with moshi to run without a manual install step.

Usage Examples:

The PyTorch implementation requires the moshi package (version 0.2.6 or later).

To run inference on an audio file directly:

python -m moshi.run_inference --hf-repo kyutai/stt-2.6b-en audio/bria.mp3

If you use uv, you can run it without manual installation:

uvx --with moshi python -m moshi.run_inference --hf-repo kyutai/stt-2.6b-en audio/bria.mp3

The MLX implementation is optimized for on-device inference on iPhone and Mac using Apple's ML framework for hardware acceleration.

Requirements:

• Requires the moshi-mlx package.
• Install via pip install moshi-mlx or use uvx --with moshi-mlx.

Usage: Use the tts_mlx.py script. If the model cannot maintain real-time performance, use the --quantize flag with values 4 or 8 to speed up inference.

Usage Examples:

Install the moshi library via pip to use the Kyutai STT and TTS models.

!pip install moshi

You can install the moshi package using two different methods depending on your requirements:

Fast install (minimal dependencies): Use this if you want to avoid downloading large packages like torch and cuda immediately, but note it might break in the future.

Slow install (future proof): Use this to ensure all necessary dependencies like torch and cuda are correctly installed.

# Fast install
!pip install 'safetensors<0.6'
!pip install 'sphn<0.2'
!pip install --no-deps "moshi==0.2.11"

# Slow install (recommended for stability)
!pip install "moshi==0.2.11"

# Fast install
!pip install 'safetensors<0.6'
!pip install 'sphn<0.2'
!pip install --no-deps "moshi==0.2.11"

# Slow install
# !pip install "moshi==0.2.11"