DOCSEARCHbyAlgolia
Submit Docs

Omnilingual ASR Modeling Library

repository·main·Indexed 25 days ago

https://github.com/facebookresearch/omnilingual-asr

An open-source multilingual speech recognition system supporting over 1,600 languages. It provides a flexible model family including SSL, CTC, and LLM-based architectures (such as omniASR_LLM_7B_v2) for high-accuracy transcription. The library includes the ASRInferencePipeline for processing audio files, binary data, or decoded audio dictionaries, and supports zero-shot generation with context examples via the omniASR_LLM_7B_ZS model. It integrates with fairseq2 for model management and HuggingFace for dataset evaluation.

Tokens
11.2K
Snippets
21
Records
69
Agent score
83%

What's inside omnilingual-asr

  1. Understand Omnilingual ASR Model Architectures

    main

    The project provides three main types of model architectures based on a Wav2Vec2 encoder foundation:

    1. W2V (Wav2Vec2 SSL): Produces contextualized audio embeddings. Useful as a foundation for custom architectures.
    2. CTC (Connectionist Temporal Classification): A non-autoregressive model that projects embeddings to vocabulary logits for parallel prediction. Best for on-device transcription.
    3. LLM (Large Language Model): An encoder-decoder architecture that projects audio embeddings into Llama space (4096-dim) for autoregressive text generation via beam search. Offers the highest transcription flexibility.

    All models expect raw audio waveforms at 16kHz as input.

  4. Use LLM+LID Unlimited Length Models

    main

    For transcribing long-form audio, use the LLM+LID, Unlimited length variant.

    Inference Behavior:

    • The model processes audio in segments of N=15 seconds.
    • During inference, segments are decoded iteratively, where each segment is conditioned on the previous M=1 decoded segments.
    • Note: While the current inference pipeline does not support real-time/streaming, the underlying checkpoints can be extended for streaming applications.

  5. Model Download and Storage

    main

    Models in the Omnilingual ASR suite are managed automatically:

    • Automatic Download: Models are downloaded automatically upon their first use during either training or inference.
    • Storage Location: Assets are stored in the fairseq2 asset store at ~/.cache/fairseq2/assets/.

  6. Quick Start with ASRInferencePipeline

    main

    To perform basic speech transcription, initialize the ASRInferencePipeline with a model_card and call the .transcribe() method. The pipeline handles audio decoding, resampling to 16kHz, mono-channel conversion, and normalization.

    from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline
pipeline = ASRInferencePipeline(model_card="omniASR_CTC_1B_v2")
transcriptions = pipeline.transcribe(["/path/to/audio1.flac"], batch_size=1)
print(transcriptions[0])

  7. Use LLM+ZS (Zero-Shot with Context) Models

    main

    The LLM+ZS variant is designed for few-shot learning using context examples.

    Requirements:

    • You must provide exactly 10 context examples (audio-text pairs) for proper inference.
    • Constraint: If you have fewer than 10 examples, you must repeat the available examples to reach the required count of 10 to satisfy the model's input validation.

    These context examples are passed as part of the input batch using special tokens.

  9. Use LLM+LID (Language Conditioning) Models

    main

    The LLM+LID variant is designed for language-conditioned transcription. It supports two input modes:

    • Audio + language_id: Providing the language ID explicitly.
    • Audio-only: The model is robust enough to handle inputs without language identification tokens.

    This variant is ideal when the language of the audio is known or when you want to provide a hint to the model.

  10. Use MixtureParquetStorage for weighted multilingual sampling

    main
    The MixtureParquetStorage class implements the StorageInterface and allows reading multiple splits simultaneously with sampling based on weights provided in a dataset summary. It is designed for training on large-scale multilingual datasets where different corpora or languages need to be balanced using beta parameters.

  12. Evaluate using the HuggingFace dataset

    main

    To use the facebook/omnilingual-asr-corpus dataset with the inference pipeline, first install the data extra: pip install "omnilingual-asr[data]". You can then load the dataset using the datasets library and format the audio data for the ASRInferencePipeline.transcribe method.

    from datasets import load_dataset
from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline

# Load dataset for a specific language (e.g., Ligurian)
omni_dataset = load_dataset("facebook/omnilingual-asr-corpus", "lij_Latn", split="train", streaming=True)
batch = next(omni_dataset.iter(5))

# Convert to pipeline input format
audio_data = [{"waveform": x["array"], "sample_rate": x["sampling_rate"]}
              for x in batch["audio"]]

# Run inference
pipeline = ASRInferencePipeline(model_card="omniASR_LLM_7B_v2")
transcriptions = pipeline.transcribe(audio_data, batch_size=2)

# Display results
for i, (transcription, original_text) in enumerate(zip(transcriptions, batch["raw_text"]), 1):
    print(f"\n Sample {i}:")
    print(f"   Ground Truth: {original_text}")
    print(f"   Predicted:    {transcription}")