DOCSEARCHbyAlgolia
Submit Docs

Hugging Face Blog Documentation

repository·main·Indexed 25 days ago

https://github.com/huggingface/blog

Guidelines and technical tutorials for the official Hugging Face Blog. Includes instructions for contributing articles using Markdown, managing assets, and embedding Gradio Spaces. Also provides technical guides on 4-bit quantization using bitsandbytes and QLoRA, as well as configuring DeepSpeed ZeRO Stage-2 and Stage-3 with the Accelerate library.

Tokens
206.6K
Snippets
538
Records
851
Agent score
86%

What's inside huggingface-blog

  1. Overview of Llama 3.2 Models

    main

    Llama 3.2 is a collection of multimodal and small text-only models from Meta.

    Multimodal (Vision) Models:

    • 11B: Designed for efficient deployment on consumer-grade GPUs. Supports tile sizes of 448 (base) or 560 (instruct).
    • 90B: Designed for large-scale applications. Uses tile sizes of 560.
    • Both come in base and instruction-tuned variants.
    • Supports visual reasoning, grounding, document QA, and image-text retrieval.
    • Context length is 128k tokens. Note: The transformers implementation attends only to the last image provided to preserve memory and quality.

    Small Text-Only Models:

    • 1B and 3B: Designed for on-device execution. Available in base and instruct variants.
    • Llama Guard 3 (1B): A small safeguard model for classifying model inputs and generations, including multimodal prompts.

    Integrations:

    • Hugging Face transformers and TGI for Vision models.
    • Inference Endpoints, Google Cloud, Amazon SageMaker, and DELL Enterprise Hub.

  2. Overview of Transformers support for MoEs

    main

    To support MoEs as first-class citizens, the transformers library has undergone redesigns in several key areas of its architecture to move beyond the assumptions made for dense models:

    • Weight Loading: Refactored to handle sparse architectures.
    • Expert Backend: Specialized execution models for expert layers.
    • Expert Parallelism: Distributed abstractions for parallelizing across experts.
    • Training: Support for training MoE models within the transformers ecosystem.

  3. Overview of DecodingTrust evaluation perspectives

    main

    DecodingTrust is a unified evaluation platform for assessing LLM trustworthiness across eight specific perspectives. Each perspective uses tailored red-teaming methodologies:

    • Toxicity: Uses optimization algorithms and 33 challenging system prompts (e.g., role-play, task reformulation) to test model responses.
    • Stereotype Bias: Evaluates bias across 24 demographic groups and 16 stereotype topics using prompt variations.
    • Adversarial Robustness: Tests robustness against five adversarial attack algorithms across diverse tasks.
    • OOD (Out-of-Distribution) Robustness: Evaluates performance under style transformations (e.g., Shakespearean) or when required knowledge is absent from training data.
    • Robustness against Adversarial Demonstrations: Tests performance against misleading information like counterfactual examples, spurious correlations, and backdoor attacks.
    • Privacy: Assesses privacy leakage from pretraining data, during conversations, and the understanding of privacy-related words/events.
    • Ethics: Uses ETHICS and Jiminy Cricket datasets to evaluate recognition of immoral behavior and resistance to jailbreaking.
    • Fairness: Evaluates fairness in zero-shot and few-shot settings by controlling protected attributes across tasks.

  4. Overview of Llama 3.1 Models

    main

    Llama 3.1 is a family of open-weight models from Meta available on the Hugging Face Hub. It includes three primary sizes:

    • 8B: Optimized for efficient deployment on consumer-grade GPUs.
    • 70B: Designed for large-scale AI native applications.
    • 405B: A massive dense model intended for synthetic data generation, LLM as a Judge, or distillation.

    Each size is available in both Base (pre-trained) and Instruct (fine-tuned) variants. Additionally, Meta released safety models:

    • Llama Guard 3: A safeguard model to classify LLM inputs and generations for safety.
    • Prompt Guard: A small (279M parameter) BERT-based classifier to detect prompt injections and jailbreaks.

  5. Overview of DABStep: Data Agent Benchmark for Multi-step Reasoning

    main

    DABStep is a benchmark designed to evaluate the capabilities of Large Language Models (LLMs) and AI agents in performing complex, multi-step data analysis tasks. It consists of over 450 real-world tasks extracted from actual workloads (specifically from Adyen) to test how well agents can reason over both structured (databases) and unstructured (documents) data.

    Key characteristics of DABStep include:

    • Real-world tasks: Avoids synthetic toy problems in favor of practical data analysis challenges.
    • Hybrid data handling: Requires reasoning across multiple datasets and unstructured documentation.
    • Simple setup: Unlike other benchmarks (e.g., SWE-bench), DABStep only requires a code execution environment to generate answers.
    • Objective evaluation: Uses a factoid evaluation method where task outcomes are mapped to a binary (right or wrong) result, removing the need for subjective interpretation.

  6. Overview of 4-bit quantization and QLoRA

    main

    LLMs can be made more accessible using 4-bit quantization and QLoRA (Quantized Low-Rank Adaptation). This approach allows running and fine-tuning large models on consumer hardware by compressing the pretrained model into 4-bit precision while adding small, trainable Low-Rank Adapters (LoRA).

    Key benefits include:

    • Memory Efficiency: Enables fine-tuning a 33B parameter model on a single 24GB GPU and a 65B parameter model on a single 46GB GPU.
    • Performance: QLoRA backpropagates gradients through a frozen, 4-bit quantized model into LoRA adapters, preserving performance comparable to full 16-bit fine-tuning.
    • Data Types: Uses a storage data type (typically 4-bit NormalFloat/NF4) for base weights and a computation data type (16-bit BrainFloat/bfloat16) for operations.

  7. Overview of Hugging Face Kernels for ROCm

    main
    Hugging Face's kernels library provides a streamlined way to build and share high-performance custom GPU kernels. It supports multiple backends including CUDA, ROCm, Metal, and XPU. For AMD users, the library facilitates building ROCm-compatible kernels that are optimized for AMD GPUs and seamlessly integrated with PyTorch, helping to avoid common issues with CMake/Nix, compiler errors, and ABI compatibility.

  8. Overview of Megatron-LM for Language Model Training

    main

    Megatron-LM is a high-performance framework developed by NVIDIA for pre-training large transformer models on GPUs. It is highly optimized for speed and efficiency through several key mechanisms:

    • Efficient DataLoader: Tokenizes and shuffles data before training. It pre-computes data indices for numbered sequences based on training parameters, which smoothes the learning curve and reduces computation time compared to standard epoch-based iteration.
    • Fused CUDA Kernels: Combines multiple operations (e.g., $f$, $g$, and $h$) into a single hardware operation. This reduces memory movement by keeping intermediate results in GPU registers instead of writing them back to memory.
    • Fused Optimizers: Utilizes fused implementations of optimizers like AdamW from NVIDIA Apex, which are faster than standard PyTorch implementations.

    While more complex to use than Hugging Face Accelerate or the Trainer API, Megatron-LM can provide significant speedups for large-scale GPU training.

  9. Overview of LeRobot Datasets v3.0

    main

    LeRobotDataset v3.0 introduces a chunked episode format and streaming capabilities designed for massive datasets (e.g., > 400GB like Open X Embodiment).

    Key improvements include:

    • Chunked Episodes: Supports large-scale data.
    • Efficient Video Storage + Streaming: Faster loading and seamless streaming.
    • Unified Parquet Metadata: Replaces scattered JSONs with structured Parquet files for episode metadata.
    • Performance: Reduced initialization times and improved memory efficiency.

  10. Overview of ControlNet in Diffusers

    main

    ControlNet is a framework that allows for adding spatial context conditioning (such as depth maps, segmentation maps, scribbles, or keypoints) to Diffusion models like Stable Diffusion.

    In the diffusers implementation, ControlNet works by using a 'trainable copy' of the Diffusion model's parameters (connected via 'zero convolution' layers) alongside a 'locked copy' that preserves the original model's knowledge.

    For inference, you need both the pre-trained diffusion model weights and the specific trained ControlNet weights for your chosen conditioning type. Because the base diffusion model remains locked, you can switch between different ControlNet weights (e.g., switching from a Canny edge model to a depth map model) within a single application while keeping the same base model loaded.

  11. Overview of the 🤗 Kernels ecosystem

    main

    The 🤗 Kernels project standardizes how custom kernels are packaged, distributed, and consumed on the Hugging Face Hub. It introduces a new repository type called kernel which allows users to discover kernels based on supported accelerators, operating systems, and backend versions.

    Key components:

    • kernels library/CLI: Used for loading and preparing kernels for use in Python environments.
    • kernel-builder CLI: Used for scaffolding, building, and benchmarking kernels.
    • Kernel Hub: A dedicated space to browse available kernels at https://huggingface.co/kernels.

  12. Overview of EmbeddingGemma

    main

    EmbeddingGemma is a state-of-the-art multilingual embedding model developed by Google, designed for efficiency and on-device use cases (e.g., mobile RAG pipelines and agents).

    Key Specifications:

    • Parameters: 308M
    • Context Window: 2048 tokens
    • Output Dimension: 768-dimensional vector (default)
    • Language Support: Over 100 languages
    • Architecture: Based on the Gemma3 backbone, modified with bi-directional attention (encoder architecture) and a mean pooling layer.
    • Matryoshka Representation Learning (MRL): Supports on-demand truncation of the 768-dimensional output to 512, 256, or 128 dimensions to reduce memory and disk usage.