UniLM (Unified Language Model)

PFPO (Preference Optimization for Reasoning with Pseudo Feedback) is a method introduced for ICLR 2025 that generates pseudo feedback for reasoning tasks. It frames the labeling of solutions as an evaluation against associated test cases.

Key features include:

• Pseudo Feedback Generation: Uses two forms: feedback generated by frontier LLMs and feedback generated by extending self-consistency to multi-test-case scenarios.
• Task Support: Optimized for mathematical reasoning and coding tasks.
• Proven Performance: Demonstrated significant improvements in models like Mathstral-7B (on MATH) and Deepseek-coder-7B-v1.5 (on LiveCodeBench).

Speech Synthesis (S^2) provides implementations for speech synthesis tasks using fairseq. It supports:

• Model Architectures: Both autoregressive and non-autoregressive models.
• Speaker Support: Multi-speaker synthesis capabilities.
• Preprocessing: Tools for audio preprocessing.
• Evaluation: Automatic metrics for speech quality.

The data configuration patterns are similar to the Speech-to-Text (S2T) implementation in this repository.

LongNet is a model architecture designed to scale Transformers to handle up to 1,000,000,000 tokens. It is part of Microsoft's research into large-scale self-supervised pre-training across various tasks and modalities.

Note: The actual implementation and code release for LongNet are maintained in the torchscale repository rather than this specific directory.

The models described in this documentation are autoregressive English language models based on the paper "Efficient Large Scale Language Modeling with Mixtures of Experts". They include both dense and sparse (MoE) architectures.

Model Scales

• Dense models: Range from 125M to 13B parameters.
• Sparse (MoE) models: Range from 15B to 1.1 Trillion parameters.

Efficiency via Sparse Computation

MoE models achieve high parameter counts with significantly lower computational costs compared to dense models of similar scale. For example, the 1.1T parameter MoE model requires only approximately 30% more FLOPS than a 6.7B parameter dense model, despite having a 160x increase in parameters. This is achieved by leveraging sparse computation, where only a small fraction of parameters are active for any given input.

UniLM is a repository dedicated to large-scale self-supervised pre-training across various tasks, languages, and modalities. The project focuses on 'The Big Convergence': unifying predictive and generative tasks, supporting over 100 languages, and integrating multiple modalities including language, image, audio, layout/format, and vision+language combinations.

Key research areas include:

• Foundation Architecture: Developing stable, efficient, and general-purpose architectures (e.g., DeepNet, Magneto, X-MoE).
• Foundation Models: Multimodal Large Language Models (MLLMs) like the Kosmos series.
• Multimodal Integration: Combining text, layout, and images for Document AI (e.g., LayoutLM series) and vision-language pre-training (e.g., BEiT-3, VLMo).

This project implements research from the paper Speech Resynthesis from Discrete Disentangled Self-Supervised Representations. It focuses on using self-supervised discrete representations to synthesize speech in a controllable manner by separately extracting three types of low-bitrate representations:

1. Speech Content: The linguistic information.
2. Prosodic Information: The rhythm, stress, and intonation.
3. Speaker Identity: The unique characteristics of the speaker.

Key applications include high-quality speech resynthesis, voice conversion, and ultra-lightweight speech codecs (achieving rates as low as 365 bits per second).

Fairseq is a sequence modeling toolkit built on PyTorch. It is designed for training custom models across various text generation and sequence tasks, including:

• Translation
• Summarization
• Language modeling
• Other text generation tasks

GSLM is a generative speech-to-speech system composed of three primary components that work together to process and generate spoken language. The system utilizes various audio representations including Log Mel Filterbank, Modified CPC, HuBERT Base, and Wav2Vec 2.0 Large for building and evaluating its models.

The three core components are:

1. speech2unit: Quantizes raw speech into learned discrete speech units.
2. ulm (Unit Language Model): A generative language model trained specifically on discrete speech units.
3. unit2speech: Synthesizes speech from the discrete speech units.

For detailed implementation and usage of each component, refer to their respective sub-directories in the repository.

The unit2speech model is a modified Tacotron2 architecture designed to synthesize speech from discrete speech units rather than text. These models are trained on quantized LJSpeech data.

When using a unit2speech model, ensure that the quantized audio units you provide for synthesis match the upstream units the specific model was trained with (e.g., HuBERT Base, wav2vec 2.0 Large, or Log Mel Filterbank combined with specific K-means centroids like KM50, KM100, etc.).