fairseq2 Documentation

The fairseq2.data module provides high-performance data processing pipelines and utilities for machine learning workflows. It is designed for high-throughput workloads and utilizes both Python and C++ implementations for performance-critical operations.

Key capabilities include:

• High-Performance Data Pipelines: Optimized C++-based loading and processing.
• Text Processing: Tokenization and preprocessing utilities.
• Audio Processing: Tools for loading audio data and extracting features.
• Structured Data: Support for formats like Parquet, JSON, and CSV.
• Memory Efficiency: Support for streaming and batched processing of large datasets.

The fairseq2.nn module provides the fundamental building blocks and utilities for constructing transformer-based models and other neural network architectures. It is designed to support efficient neural network operations through a unified batch layout system and core architectural components.

Key functional areas include:

• Batch Layout: A unified system for managing tensor layouts across different dimensions.
• Core Layers: Implementations of embeddings, position encoders, normalization, projections, and residual connections.
• State Management: Utilities for handling incremental states (useful for autoregressive decoding).
• Utilities: General-purpose neural network operation helpers.

fairseq2 is a modular sequence modeling toolkit designed for research and production. Key features include:

• Training: Multi-GPU, multi-node training using DDP, FSDP, and tensor parallelism (supports 70B+ models).
• Inference: Native support for vLLM, with built-in sampling and beam search sequence generators.
• Extensibility: Uses a setuptools extension mechanism to register new models, optimizers, LR schedulers, and trainer units without forking the library.
• Data Pipeline: A high-throughput, streaming-based C++ API supporting speech and video decoding.
• Configuration: Flexible, deterministic configuration via a built-in structured API.
• Asset Management: Programmatic asset cards for version-controlled access to models, datasets, and tokenizers.
• Modern Tooling: Built on modern PyTorch features like torch.compile and PyTorch FSDP.

The fairseq2.models.hg module provides seamless integration with HuggingFace Transformers models. It allows you to load and use any HuggingFace model within fairseq2's training and inference pipelines.

Prerequisites This module requires the transformers library. Install it via:

pip install transformers

Note on Security Some models require trust_remote_code=True for custom architectures. Only use this with trusted model sources.

The fairseq2.assets module provides a model hub system designed for loading pre-trained models, tokenizers, and other essential assets.

While the documentation is currently under development, the module is architected to handle:

• Asset store and registry: Managing available assets.
• Model downloading and caching: Automatically fetching and storing models locally.
• Asset metadata management: Handling information associated with specific assets.
• Pre-trained model catalogs: Accessing collections of ready-to-use models.

For specific implementation details, refer to the source code or available usage examples in the repository.

A Gang is an abstraction representing a set of processes capable of performing collective communication operations (e.g., all_reduce, broadcast). It encapsulates device management, process ranks, and total process count into a single object.

There are two primary implementations:

1. ProcessGroupGang: Used for real distributed environments, wrapping PyTorch's ProcessGroup (using NCCL or Gloo backends).
2. FakeGang: Used for testing or single-process scenarios, simulating the distributed API without requiring a multi-process setup.

import torch
from fairseq2.gang import ProcessGroupGang

device = torch.get_default_device()
# Creates a ProcessGroup with NCCL or Gloo backend depending on device
gang = ProcessGroupGang.create_default_process_group(device)

tensor = torch.ones((8, 8), device=gang.device)
gang.all_reduce(tensor, ReduceOperation.SUM)

fairseq2 v0.5 introduces a new checkpoint format that serves as a lightweight alternative to PyTorch DCP.

Key features include:

• Dynamic Resharding: Supports 3-D model sharding APIs for both offline (checkpoint) and online (training) settings.
• No Process Groups: Unlike DCP, you do not need to set up process groups for saving or loading checkpoints.
• Inspectability: Checkpoints are generated as regular, user-inspectable PyTorch tensor files (.pt).
• Asynchronous Management: A new asynchronous checkpoint manager is included, which is fully deterministic and handles NFS lookup caches to prevent race conditions during async evaluation jobs.
• Model-Only Saving: You can choose to save only models instead of entire checkpoints to reduce disk overhead during short-running post-training jobs.

Packed Batches are more memory efficient and better for highly variable sequence lengths because they only allocate space for actual elements. However, they require more careful indexing.

Padded Batches are simpler to implement and better for uniform attention operations but may waste memory by allocating max_seq_len for every sequence in the batch.

# Memory comparison
seq_lens = [100, 50, 75, 25]  # Variable length sequences

# Padded: allocates max_len for all sequences
padded_layout = BatchLayout((4, 100), seq_lens=seq_lens, packed=False)
padded_memory = 4 * 100  # 400 positions

# Packed: only allocates needed positions
packed_layout = BatchLayout((250,), seq_lens=seq_lens, packed=True)
packed_memory = sum(seq_lens)  # 250 positions (37.5% savings)