DOCSEARCHbyAlgolia
Submit Docs

MLX-Audio

repository·main·Indexed 27 days ago

https://github.com/blaizzy/mlx-audio

An audio processing library optimized for Apple Silicon using the MLX framework. It provides high-performance Text-to-Speech (TTS), Speech-to-Text (STT), and Speech-to-Speech (STS) capabilities, including implementations for LFM2.5-Audio, DeepFilterNet, and Mel-Band-RoFormer for audio source separation.

Tokens
89.8K
Snippets
298
Records
461
Agent score
92%

What's inside mlx-audio

  1. Overview of MLX Audio capabilities

    main

    MLX Audio is a high-performance audio library built on Apple's MLX framework, optimized for M-series chips (M1/M2/M3/M4). It provides native acceleration and quantization support (3-bit to 8-bit) for several audio tasks:

    • Text-to-Speech (TTS): Generate speech using models like Kokoro, Qwen3-TTS, Voxtral TTS, CSM, and Dia. Supports multilingual output, voice cloning, and speed control.
    • Speech-to-Text (STT): Transcribe audio using Whisper, Parakeet, Voxtral Realtime, Qwen3-ASR, and VibeVoice. Supports streaming and word-level timestamps.
    • Speech-to-Speech (STS): Includes source separation (SAM-Audio), speech enhancement (MossFormer2, DeepFilterNet), and conversational AI (Liquid2.5-Audio).
    • OpenAI-Compatible API: Provides a REST API server compatible with existing OpenAI client libraries, including a web UI with 3D audio visualization.
    • Swift/iOS Support: On-device TTS is available for macOS and iOS via the mlx-audio-swift package.

  2. Overview of MLX Audio Python API modules

    main

    The MLX Audio public Python API is organized into three primary modules:

    • mlx_audio.tts: Handles Text-to-Speech tasks, including model loading, audio generation, and related utilities.
    • mlx_audio.stt: Handles Speech-to-Text tasks, including model loading and transcription.
    • mlx_audio.audio_io: Provides utilities for reading and writing audio files.

  4. Compare Speech-to-Text (STT) models

    main

    MLX Audio supports various STT models optimized for Apple Silicon. Use the following guide to select a model based on your requirements:

    • Best multilingual coverage: Whisper (99+ languages) or MMS (1000+ languages).
    • Best accuracy for English: Parakeet v2 or Whisper large-v3-turbo.
    • Best for European languages: Parakeet v3 (25 languages) or Canary.
    • Lowest latency / streaming: Voxtral Realtime (4bit variant).
    • Smallest footprint: Moonshine tiny (27M parameters).
    • Speaker diarization built-in: MOSS-Transcribe-Diarize or VibeVoice-ASR.
    • Word-level alignment: Qwen3-ForcedAligner.
    • Emotion / event detection: SenseVoice.

  5. Use OmniVoice for zero-shot multilingual TTS

    main

    OmniVoice is a zero-shot multilingual Text-to-Speech (TTS) system supporting over 646 languages. It allows for high-quality speech generation and optional voice cloning using a reference audio file.

    Recommended Model Weights: Use mlx-community/OmniVoice-bfloat16 for MLX-optimized weights on Apple Silicon.

    Requirements for Voice Cloning: If you intend to use the voice cloning feature, you must install torchaudio and transformers:

    pip install torchaudio transformers

  6. Use Liquid2.5-Audio for multimodal tasks

    main

    Liquid2.5-Audio is a multimodal foundation model supporting Text-to-Speech (TTS), Speech-to-Text (ASR), and Speech-to-Speech (STS) using a ChatState object to manage interleaved text and audio turns.

    Available Models:

    • mlx-community/LFM2.5-Audio-1.5B-4bit (4-bit quantized)
    • mlx-community/LFM2.5-Audio-1.5B-8bit (8-bit quantized)

  7. Compare Text-to-Speech (TTS) models

    main

    MLX-Audio supports various TTS models optimized for Apple Silicon. Choose a model based on your requirements for speed, size, language support, voice cloning, or streaming capabilities.

    Model Selection Guide

    • Fastest / Smallest: Kokoro (82M) - ideal for quick generation with many voice presets.
    • Voice Cloning: CSM, Qwen3-TTS, Higgs Audio v3, or OmniVoice - allows cloning a voice from reference speech.
    • Multilingual: Voxtral TTS (9 languages, 20 voices) or Chatterbox (16 languages).
    • Dialogue: Dia - includes built-in support for multi-speaker conversations using speaker tags.
    • Emotion / Style Control: Qwen3-TTS (CustomVoice or VoiceDesign variants).

  8. System Requirements for MLX-Audio

    main

    To run MLX-Audio, ensure your environment meets the following requirements:

    • Hardware: Apple Silicon Mac (M1/M2/M3/M4)
    • Python: 3.10 or higher
    • Framework: MLX
    • Dependencies: ffmpeg (required for MP3, FLAC, OGG, Opus, and Vorbis encoding)

  9. Use Granite Speech for Speech-to-Text and Translation

    main

    Granite Speech is an MLX implementation of IBM's Granite Speech model. It supports both ASR (Automatic Speech Recognition/Transcription) and AST (Automatic Speech Translation).

    Supported Languages: English, French, German, Spanish, Portuguese, Japanese.

    Available Model:

    • ibm-granite/granite-4.0-1b-speech (~1B parameters)

  12. Generate dialogue with MOSS-TTSD-v1.0

    main

    The MOSS-TTSD-v1.0 model supports multi-speaker dialogue using speaker tags like [S1] and [S2] in the text. You can provide multiple reference audio/text pairs via the CLI using --ref_audio and --ref_text to define different speakers.

    # Single speaker dialogue
python -m mlx_audio.tts.generate \
  --model OpenMOSS-Team/MOSS-TTSD-v1.0 \
  --text "[S1] Hello. [S2] Hi, this is MOSS-TTSD running on MLX."
  --output_path outputs

# Multi-speaker with specific references
python -m mlx_audio.tts.generate \
  --model OpenMOSS-Team/MOSS-TTSD-v1.0 \
  --text "[S1] This uses the first reference. [S2] This uses the second."
  --ref_audio speaker_1.wav \
  --ref_text "Reference transcript for speaker one." \
  --ref_audio speaker_2.wav \
  --ref_text "Reference transcript for speaker two." \
  --output_path outputs