DOCSEARCHbyAlgolia
Submit Docs

Hugging Face Transformers Documentation

repository·main·Indexed Apr 15, 2026

https://github.com/huggingface/transformers

Repository for fine-tuning and training state-of-the-art machine learning models on text, vision, audio, and multimodal tasks. Supports PyTorch, JAX, and TensorFlow with the Trainer API and native Accelerate loops. Includes comprehensive benchmarking v2 framework for performance evaluation, model definition compatibility across frameworks like DeepSpeed and vLLM, and example scripts for causal, masked, and permutation language modeling.

Tokens
41.5K
Snippets
83
Records
99
Agent score
100%

What's inside transformers

  1. Overview of Benchmarking v2

    main

    Benchmarking v2 is a comprehensive framework for evaluating transformer model performance. It supports multiple execution modes (eager, compiled, kernelized), collects detailed performance metrics, and outputs results in a structured JSON format. The framework allows users to run all benchmarks or specific subsets, customize measurement parameters, and automatically upload results to a HuggingFace Dataset for tracking.

    Sources: benchmark_v2/README.md

  2. Overview of VaultGemma

    main

    VaultGemma is a text-only decoder model derived from Gemma 2. Key architectural differences from Gemma 2 include:

    • It drops the norms after the Attention and MLP blocks.
    • It uses full attention for all layers instead of alternating between full and local sliding attention.

    Privacy Features: VaultGemma was trained from scratch using sequence-level differential privacy (DP) with DP stochastic gradient descent (DP-SGD). It provides a $(\epsilon \le 2.0, \delta \le 1.1e-10)$-sequence-level DP guarantee, where a sequence consists of 1024 consecutive tokens.

  4. Overview of GLM-OCR

    main

    GLM-OCR is a lightweight (0.9B parameters) multimodal OCR model designed for complex document understanding. It combines a CogViT visual encoder, a cross-modal connector with token downsampling, and a GLM-0.5B language decoder.

    It is capable of multi-tasking across:

    • Text recognition
    • Formula recognition
    • Table recognition
    • Information extraction

  5. Overview of torchao features and quantization techniques

    main

    torchao is a PyTorch architecture optimization library designed for high-performance data types, quantization, and sparsity. It is designed to be composable with torch.compile for accelerated inference and training.

    Supported Quantization Techniques

    • A16W8 Float8 Dynamic Quantization
    • A16W8 Float8 WeightOnly Quantization
    • A8W8 Int8 Dynamic Quantization
    • A16W8 Int8 Weight Only Quantization
    • A16W4 Int4 Weight Only Quantization
    • A16W4 Int4 Weight Only Quantization + 2:4 Sparsity
    • Autoquantization

    Key Features

    • Quantization Aware Training (QAT): Training with minimal accuracy loss.
    • Float8 Training: High-throughput training using float8 formats.
    • Sparsity Support: Semi-structured (2:4) sparsity for faster inference.
    • Optimizer Quantization: 4 and 8-bit variants of Adam to reduce memory.
    • KV Cache Quantization: Lower memory usage for long context inference.
    • FSDP2 Compatibility: Composable with FSDP2 for training.

  6. Overview of the TAPAS model

    main

    TAPAS (Table Parsing) is a BERT-based model designed specifically for answering questions about tabular data. Unlike standard BERT, TAPAS uses relative position embeddings and 7 token types to encode the structural information of a table.

    For question answering, TAPAS utilizes two specialized heads:

    1. Cell selection head: Identifies relevant cells in the table.
    2. Aggregation head: Performs optional mathematical operations (like sum or count) on the selected cells.

    TAPAS is pre-trained on Masked Language Modeling (MLM) using millions of Wikipedia tables and has been fine-tuned on datasets such as SQA (Sequential Question Answering), WTQ (Wiki Table Questions), and WikiSQL.

  7. Overview of Sapiens2 models

    main

    Sapiens2 is a family of high-resolution vision transformers designed for human-centric computer vision tasks. It is pretrained on approximately 1 billion curated human images and supports tasks such as:

    • Pose estimation
    • Body-part segmentation
    • Surface normal estimation
    • Pointmap estimation
    • Albedo estimation

    Key technical characteristics:

    • Scalability: Models range from 0.4B to 5B parameters.
    • Resolution: Supports training at native 1K resolution, with hierarchical 4K variants available for extended spatial reasoning.
    • Architecture: Uses Rotary Position Embeddings (RoPE) to support arbitrary input resolutions. It employs Grouped Query Attention (GQA) in middle layers, while the first and last 8 layers use full multi-head attention.
    • Register Tokens: Uses 8 register tokens by default to reduce high-norm artifacts in patch tokens, which improves attention map cleanliness and performance on dense prediction tasks.

  8. Overview of Qwen3 ASR models

    main

    Qwen3 ASR is an automatic speech recognition (ASR) model family from Alibaba's Qwen team. It utilizes a Whisper-style audio encoder paired with a Qwen3 language model decoder for speech-to-text transcription.

    Key features include:

    • Multilingual Transcription: Supports automatic language detection and transcription across many languages.
    • Forced Aligner: A specialized model (Qwen3-ForcedAligner-0.6B-hf) that can timestamp a provided transcript against audio by predicting word lengths using the audio encoder and a classification head.

    Available Checkpoints:

    • Qwen/Qwen3-ASR-1.7B-hf (Multilingual ASR)
    • Qwen/Qwen3-ASR-0.6B-hf (Multilingual ASR)
    • Qwen/Qwen3-ForcedAligner-0.6B-hf (Forced Alignment)

  9. Access official and community Transformers notebooks

    main
    The notebooks/ directory contains a collection of official Hugging Face notebooks and community-contributed content. These notebooks provide practical examples and tutorials for using the Transformers library, including PyTorch examples and documentation-driven guides.

  10. Use SEW-D for speech tasks

    main

    SEW-D (Squeezed and Efficient Wav2Vec with Disentangled attention) is a speech model designed for performance-efficiency trade-offs in automatic speech recognition (ASR).

    Key usage requirements:

    • Input Format: The model accepts a float array representing the raw waveform of the speech signal.
    • Decoding: When using SEWDForCTC, the model is fine-tuned using Connectionist Temporal Classification (CTC). Therefore, you must decode the model output using the Wav2Vec2CTCTokenizer to obtain text.

  11. Optimizing LLMs for Speed and Memory

    main

    Large Language Models (LLMs) face challenges in real-world deployment due to high memory demands from billions of parameters and the need to manage long input sequences. To address these challenges, several optimization techniques can be employed:

    1. Lower Precision (Quantization): Using reduced numerical precision, such as 8-bit or 4-bit, to achieve computational advantages and reduce memory footprint with minimal performance loss.
    2. Flash Attention: An optimized attention algorithm that improves memory efficiency and speed by optimizing GPU memory utilization.
    3. Architectural Innovations: Utilizing specialized model architectures designed for efficient autoregressive text generation and long context handling. Key innovations include:
      • Alibi
      • Rotary embeddings
      • Multi-Query Attention (MQA)
      • Grouped-Query-Attention (GQA)

  12. Overview of the Bark model architecture

    main

    Bark is a transformer-based text-to-speech model composed of four main sub-models that work sequentially:

    1. BarkSemanticModel (text model): A causal auto-regressive transformer that predicts semantic text tokens from input text.
    2. BarkCoarseModel (coarse acoustics model): A causal auto-regressive transformer that predicts the first two audio codebooks for EnCodec.
    3. BarkFineModel (fine acoustics model): A non-causal autoencoder transformer that predicts the remaining codebooks.
    4. EncodecModel: Decodes the predicted codebook channels into the final output audio array.

    Each of the first three modules can support conditional speaker embeddings to produce specific predefined voices.