MLX Audio Swift

Echo TTS is a diffusion-based text-to-speech model that supports voice cloning using a short reference audio clip. You can use it via the command line or integrate it into Swift applications using the MLXAudioTTS framework. The model requires a reference audio file to clone a specific voice.

mlx-audio-swift-tts \
  --model mlx-community/echo-tts-base \
  --text "Hello from Echo TTS." \
  --ref-audio speaker.wav

SenseVoice is a speech foundation model that provides multiple capabilities in a single model: Automatic Speech Recognition (ASR), Spoken Language Identification (LID), Speech Emotion Recognition (SER), and Audio Event Detection (AED). It is a non-autoregressive CTC model.

Supported Model:

• mlx-community/SenseVoiceSmall

MLXAudioLID provides Swift implementations of spoken language identification (LID) models. It can identify languages from raw audio waveforms using two primary model architectures: Wav2Vec2ForSequenceClassification (MMS-LID-256) and EcapaTdnn (ECAPA-TDNN).

Both models require 16 kHz mono audio. It is recommended to use loadAudioArray(from:) from MLXAudioCore to handle automatic resampling.

import MLXAudioCore
import MLXAudioLID

// Example using MMS-LID-256
let model = try await Wav2Vec2ForSequenceClassification.fromPretrained("facebook/mms-lid-256")
let (_, audio) = try loadAudioArray(from: audioURL)
let output = model.predict(waveform: audio, topK: 5)

print("Language: \(output.language) (\(output.confidence * 100)%)")

Irodori TTS is a Japanese flow-matching text-to-speech model (part of the Echo-TTS family). It uses a Rectified-Flow DiT over continuous latents at 48 kHz. A key feature is VoiceDesign, which allows you to describe a voice using a Japanese caption instead of providing a reference audio clip.

Key Characteristics

• Language: Japanese-only. Pass Japanese text directly.
• Voice Control: Use the voice parameter with a Japanese description (e.g., "a calm, natural female voice") to design the speaker's characteristics.
• Model Weights: Available on Hugging Face (e.g., mlx-community/Irodori-TTS-600M-v3-VoiceDesign-8bit).

MOSS-TTS provides non-quantized full OpenMOSS text-to-speech capabilities. It utilizes a Qwen3 backbone, multi-codebook audio generation, and the MOSS Audio Tokenizer. The system supports three primary model variants:

1. Standard Model: The base delay-pattern model.
2. Dialogue Model: Optimized for multi-speaker dialogue.
3. Local-Transformer Variant: A specific transformer-based architecture.

The OpenMOSS-Team/MOSS-Audio-Tokenizer is automatically loaded as part of the process.

Chatterbox TTS is a two-stage speech synthesis system that supports voice cloning. It works by first converting text to speech tokens (T3) and then converting those tokens into 24kHz audio using S3Gen and HiFi-GAN.

There are two main variants:

• Regular: Uses LLaMA 520M, supports 23 languages, and includes emotion control.
• Turbo: Uses GPT-2 Medium, is English-only, and is optimized for faster generation.

Temporal anchors allow you to guide the separation by specifying when a target sound is or is not present. An anchor follows the format (token: String, startTime: Float, endTime: Float).

Token Meanings:

• "+": The target sound is present during this time span.
• "-": The target sound is NOT present during this time span.

Note: Anchors are currently not supported in chunked/long/streaming separation methods.

// Example: Target is present from 1.0s to 2.5s, and absent from 4.0s to 6.0s
anchors: [[("+", 1.0, 2.5), ("-", 4.0, 6.0)]]