Seamless Communication

SeamlessStreaming is a multilingual streaming translation model designed for real-time communication. It provides the following capabilities:

• Streaming Automatic Speech Recognition (ASR): Supports 96 languages.
• Simultaneous Translation (Speech-to-Text): Supports 101 source languages for speech input and 96 target languages for text output.
• Simultaneous Translation (Speech-to-Speech): Supports 36 target languages for speech output.

For a complete list of supported languages, refer to the SeamlessM4T README.

Seamless is a family of AI models for natural multilingual communication:

• SeamlessM4T: A foundational multimodal model supporting ~100 languages for S2ST, S2TT, T2ST, T2TT, and ASR. v2 features the UnitY2 architecture for improved quality and lower latency.
• SeamlessExpressive: A speech-to-speech model that preserves prosody (rate, pauses) and voice style.
• SeamlessStreaming: A streaming model supporting speech input and speech/text output for simultaneous translation and ASR.
• Seamless: A unified model combining multilinguality, real-time streaming, and expressivity.

SeamlessExpressive is a model architecture designed for expressive speech-to-speech translation. It consists of two primary modules:

1. Prosody UnitY2: A prosody-aware speech-to-unit translation model based on the UnitY2 architecture. It handles phrase-level prosody, such as speech rate and pauses, by injecting expressivity embeddings from PRETSSEL into the unit generation.
2. PRETSSEL (Paralinguistic REpresentation-based TextleSS acoustic modEL): An expressive unit-to-speech generator. It disentangles semantic and expressivity components, allowing for the transfer of utterance-level expressivity, such as a speaker's specific voice style.

Together, these modules enable the transfer of both phrase-level prosody and utterance-level vocal style across languages.

The m4t_finetune script supports three distinct modes, which determine which parts of the model weights are updated and which are frozen:

1. SPEECH_TO_SPEECH: Finetunes both the Speech-to-Text (S2T) and Text-to-Unit (T2U) parts of the model. All model weights except the text encoder are engaged.
2. TEXT_TO_SPEECH: Finetunes only the Text-to-Unit (T2U) part; all other weights are frozen.
3. SPEECH_TO_TEXT: Finetunes only the Speech-to-Text (S2T) part; all other weights are frozen.

The project uses three distinct license categories. Ensure your usage complies with the specific license for the component you are using:

• MIT License: Applies to the code, W2v-BERT 2.0 speech encoder, the text-only portion of the mExpresso dataset, the UnitY2 forced alignment extractor, the speech toxicity tool (etox), and the MuTox dataset/detector.
• CC-BY-NC 4.0 License: Applies to SeamlessM4T models (v1 and v2) and SeamlessStreaming models.
• Seamless License: Applies to Seamless models and SeamlessExpressive models.

The SeamlessAlignExpressive dataset provides metadata for expressive speech alignment, allowing for the discovery of audio segments that share both meaning and expressivity. The metadata is provided in space-separated, gzipped files. Each file represents a specific alignment direction, denoted by a two-letter language code followed by 'A' (e.g., frA, enA, deA).

For example, a file named deA-enA contains information for reconstructing German speech to English speech alignments.

Seamless metadata files are tab-separated, gzipped files used for reconstructing datasets for model training. Each file represents a specific alignment direction.

Naming Convention

Language codes determine the content type:

• Text: Uses 3-letter codes (e.g., fra, eng, tur).
• Audio: Uses 2-letter codes followed by 'A' (e.g., frA, enA, trA).

Examples of directions:

• eng-trA: English text to Turkish speech.
• enA-jpn: English speech to Japanese text.
• enA-frA: English speech to French speech.

The MuTox classifier uses multi-modal and multilingual SONAR encoders. The output predictions are provided as logits.

To convert these logits into probabilities, apply a sigmoid transform. A logit value of 0 corresponds to a 50% predicted toxicity probability and can be used as a default threshold for detection.

The mExpresso (Multilingual Expresso) dataset is an expressive S2ST dataset covering seven styles (default, happy, sad, confused, enunciated, whisper, and laughing) across English and five other languages (French, German, Italian, Mandarin, and Spanish).

You can create an English-to-other-language speech-to-text dataset from scratch using the prepare_mexpresso CLI module. This process downloads the English Expresso dataset, downsamples audio to 16k Hz, and joins it with text translations to form a manifest.

The output manifests will be located at <OUTPUT_FOLDER>/{dev,test}_mexpresso_eng_{spa,fra,deu,ita,cmn}.tsv.

python3 -m seamless_communication.cli.expressivity.data.prepare_mexpresso \
    <OUTPUT_FOLDER>

If you modify the C API signatures in ggml headers, you can regenerate the Python bindings (via cffi) and IDE stubs using regenerate.py.

Basic Regeneration

pip install -q cffi
python regenerate.py

Note: This assumes llama.cpp is located in ../../../llama.cpp relative to the script.

Customizing Include Directory

If llama.cpp is in a different location, use the C_INCLUDE_DIR environment variable:

C_INCLUDE_DIR=$LLAMA_CPP_DIR python regenerate.py

Limiting API Scope

To generate bindings only for a specific header (e.g., just the core ggml.h without llama.cpp extensions), use the API variable:

API=../../include/ggml/ggml.h python regenerate.py