DOCSEARCHbyAlgolia
Submit Docs

Fairseq Sequence Modeling Toolkit

repository·main·Indexed 12 days ago

https://github.com/facebookresearch/fairseq

A sequence modeling toolkit for training and using models for translation, summarization, language modeling, and other text generation tasks. It provides reference implementations for state-of-the-art research papers, including support for Adaptive Span, Transformer-XL, multilingual and multi-domain Automatic Speech Recognition (ASR) with attention head selection, and end-to-end Natural Language Understanding (NLU) using wav2vec or hubert.

Tokens
202K
Snippets
481
Records
617
Agent score
99%

What's inside Fairseq

  1. Overview of the MoE (Mixture of Experts) Language Models

    main

    The models described in this documentation are part of the research paper "Efficient Large Scale Language Modeling with Mixtures of Experts". They include both dense and sparse (MoE-based) architectures:

    • Dense models: Parameter counts range from 125M to 13B.
    • Sparse (MoE) models: Parameter counts range from 15B to 1.1 Trillion.

    The 1.1 Trillion parameter MoE model is designed for efficiency through sparse computation. It leverages the fact that only a small fraction of parameters are active for any given input. For example, the 1.1T MoE model requires only ~30% more FLOPS compared to a 6.7B parameter dense model, despite having a 160x increase in total parameters.

  3. Overview of the XGLM multilingual model

    main
    XGLM is a family of multilingual autoregressive language models developed by FAIR. The models range in size from 564 million to 7.5 billion parameters and are trained on the CC100-XL dataset, which covers 134 languages. The models are designed to learn tasks from natural language descriptions and a few examples (few-shot learning).

  4. Overview of Speech Synthesis (S^2) in fairseq

    main
    Speech Synthesis (S^2) is a toolkit within fairseq designed for scalable and integrable speech synthesis tasks. It supports both autoregressive and non-autoregressive models and is capable of multi-speaker synthesis. The toolkit includes audio preprocessing capabilities such as denoising and Voice Activity Detection (VAD) to handle less curated datasets. It also provides automatic metrics for model development and uses a data configuration style similar to the Speech-to-Text (S2T) module.

  5. Overview of Generative Spoken Language Modeling (GSLM)

    main

    Generative Spoken Language Modeling (GSLM) is a speech-to-speech system composed of three primary components that work together to process and generate speech. The system utilizes Log Mel Filterbank, Modified CPC, HuBERT Base, and Wav2Vec 2.0 Large for its underlying representations.

    The pipeline consists of:

    1. speech2unit: Quantizes raw speech into learned discrete speech units.
    2. ulm (Unit Language Model): A generative language model trained on those discrete speech units.
    3. unit2speech: Synthesizes speech from the discrete speech units.

    For more detailed information on each component, refer to their specific sub-directories in the repository.

  6. Overview of Unit to Speech Model (unit2speech)

    main

    The unit2speech model is a modified Tacotron2 architecture designed to synthesize speech from discrete speech units rather than text. These models are trained on quantized LJSpeech data.

    To use these models, you must ensure that the quantized audio units you provide for inference match the upstream units used during the model's training (e.g., Log Mel Filterbank, Modified CPC, HuBERT Base, or wav2vec 2.0 Large).

  7. Overview of fairseq features

    main

    Fairseq is a sequence modeling toolkit designed for training custom models for translation, summarization, language modeling, and text generation.

    Key features include:

    • Multi-GPU training: Supports data and model parallelism on single or multiple machines.
    • Fast generation: Supports multiple search algorithms on CPU and GPU, including:
      • Beam search
      • Diverse Beam Search
      • Sampling (unconstrained, top-k, and top-p/nucleus)
      • Lexically constrained decoding
    • Large mini-batch training: Uses gradient accumulation to enable large batches on single GPUs.
    • Mixed precision training: Supports FP16 training for faster performance and reduced memory usage on NVIDIA tensor cores.
    • Extensibility: Allows easy registration of new models, criterions, tasks, optimizers, and learning rate schedulers.
    • Flexible configuration: Uses the Hydra framework, supporting combinations of code, command-line, and file-based configurations.
    • Distributed training optimizations: Supports full parameter and optimizer state sharding and CPU offloading.

  8. Overview of Fairseq command-line tools

    main

    Fairseq provides a suite of command-line interfaces for the complete machine learning lifecycle, including data preparation, training, inference, and evaluation:

    • fairseq-preprocess: Pre-processes data by building vocabularies and binarizing training data.
    • fairseq-train: Trains new models on single or multiple GPUs.
    • fairseq-generate: Performs translation/inference on pre-processed data using a trained model.
    • fairseq-interactive: Performs translation/inference on raw text using a trained model.
    • fairseq-score: Calculates BLEU scores for generated translations against reference translations.
    • fairseq-eval-lm: Evaluates language models.

  9. Overview of Wav2Vec 2.0

    main

    Wav2Vec 2.0 is a framework for self-supervised learning of speech representations from unlabeled data. It has been extended for:

    • Multilingual learning: Learning representations across multiple languages (XLSR).
    • Self-training: Combining self-supervised pre-training with self-training.
    • Domain robustness: Using data from multiple domains to handle domain shift.
    • Zero-shot cross-lingual phoneme recognition: Using XLSR-53 to transcribe unseen languages.

  10. Overview of Speech Resynthesis from Discrete Disentangled Self-Supervised Representations

    main

    This project implements the research from the paper Speech Resynthesis from Discrete Disentangled Self-Supervised Representations. It focuses on using self-supervised discrete representations to synthesize speech in a controllable manner by separately extracting low-bitrate representations for:

    1. Speech content
    2. Prosodic information
    3. Speaker identity

    The method allows for high-quality speech resynthesis and voice conversion, and can be used to build ultra-lightweight speech codecs (achieving rates as low as 365 bits per second).

    Key Resources:

  11. Explore Speech-to-Speech Translation (S2ST) implementations

    main

    Fairseq provides implementations and resources for several research works focused on Speech-to-Speech Translation (S2ST). You can explore specific implementations based on the following methodologies:

    • Direct speech-to-speech translation with discrete units: Implementation based on Lee et al. 2021.
    • Textless Speech-to-Speech Translation on Real Data: Implementation based on Lee et al. 2021 focusing on textless approaches using real-world data.
    • Enhanced Direct Speech-to-Speech Translation: Implementation using self-supervised pre-training and data augmentation techniques.

  12. Joint Speech Text training in Fairseq

    main

    This project is an extension of the Fairseq Speech-to-Text (S2T) project. It enhances the standard speech-to-text task by incorporating a co-trained text-to-text mapping task. This joint training approach allows the model to leverage text data to improve speech translation and recognition performance.

    For foundational information on the core speech-to-text capabilities, refer to the main Fairseq S2T documentation.