DOCSEARCHbyAlgolia
Submit Docs

Smol Models

repository·main·Indexed 24 days ago

https://github.com/huggingface/smollm

A family of efficient, lightweight, and fully open AI models from Hugging Face, including SmolLM3 (language models) and SmolVLM (vision-language models) designed for high performance on-device. The repository includes resources for text generation, visual QA, image description, and visual storytelling, as well as pipelines for data curation (FineMath), evaluation using lighteval, and fine-tuning instructions.

Tokens
27.6K
Snippets
72
Records
138
Agent score
86%

What's inside smollm

  1. Overview of SmolLM family

    main

    The SmolLM family consists of compact, high-performance language models designed for efficiency and on-device usage.

    • SmolLM3: A 3B parameter model supporting dual-mode reasoning (think/no_think), 6 languages (English, French, Spanish, German, Italian, Portuguese), and long context up to 128k tokens.
    • SmolLM2: A family of models available in 135M, 360M, and 1.7B parameter sizes, optimized for lightweight and on-device applications.

  2. Overview of the FineMath pipeline

    main
    The FineMath pipeline is a multi-phase process designed to identify, extract, and curate high-quality mathematical content from CommonCrawl. It aims to produce datasets like FineMath-3+ (34B tokens) by refining content extraction to preserve LaTeX notation and mathematical reasoning, which are often filtered out by standard web-crawling pipelines.

  3. Overview of the `data_collection` pipeline

    main

    The data_collection module is designed to process web data for vision-language model training. The pipeline follows these stages:

    1. Simplify HTML DOM trees: Reducing complexity of the raw HTML.
    2. Convert DOM trees: Transforming simplified trees into structures optimized for extraction.
    3. Perform extraction: Extracting either web documents or image-text pairs.
    4. Perform filtering: Applying filters to the extracted web documents or image-text pairs.
    5. Visualize results: Using Streamlit apps to inspect the data.

  4. Overview of Data Processing Pipelines

    main

    The project utilizes two versions of data processing pipelines:

    1. Split + Parallel/Slurm: Uses sharding (split) combined with parallel or Slurm arrays of processing scripts (e.g., Python or Rust).
    2. Apache Beam + Dataflow: Uses Apache Beam to define processing pipelines and Google Cloud Dataflow for horizontal scaling.

  5. Understand the data loading and packing strategy

    main

    The project uses on-the-fly data processing (notably packing) rather than pre-processing datasets. This is implemented using iterable datasets to avoid the overhead of loading entire datasets into memory.

    Key components of the data pipeline include:

    • DataLoaderForIterableWrapperDataset: The highest-level dataloader used by the training loop.
    • CustomChainDataset: Handles the mixing of multiple datasets based on mixture proportions defined in the configuration.
    • IterableWrapperDataset: Handles a single dataset type.

    To ensure efficient training, the system implements custom sharding and packing logic to avoid data overlap between processes and minimize wastage during batch formation.

  6. Run SmolLM2 and SmolVLM locally using various inference engines

    main

    You can run SmolLM2 (Language Models) and SmolVLM (Vision Language Models) on your local machine by leveraging several different inference libraries and frameworks. Supported engines include:

    • llama.cpp: For high-performance CPU/GPU inference.
    • MLX: Optimized for Apple Silicon.
    • MLC: For cross-platform deployment.
    • Transformers.js: For running models directly in the browser or via Node.js.

  7. Explore the Smol Models repository structure

    main

    The repository is organized into the following directories:

    • text/: Contains SmolLM3, SmolLM2, and SmolLM1 related code and resources.
    • vision/: Contains SmolVLM related code and resources.
    • tools/: Contains shared utilities and inference tools, including:
      • smol_tools/: Lightweight AI-powered tools.
      • smollm_local_inference/: Local inference for SmolLM.
      • smolvlm_local_inference/: Local inference for SmolVLM.

  8. Understand the evaluation dataset split strategy for SmolVLM

    main

    The evaluation datasets for SmolVLM are split into four disjoint subsets to facilitate few-shot learning performance estimation (adapting on support samples and evaluating on query samples). This follows the methodology described in the DeepMind Flamingo paper.

    Dataset Split Definitions

    • validation support: Contains support samples used for validation.
    • validation query: Contains query samples used for validation. (Note: For benchmarks without official validation sets, a subset of the original training set is used).
    • test support: Contains support samples used for final performance estimation.
    • test query: Contains query samples used for final performance estimation. (Note: This subset is chosen to match prior works for apples-to-apples comparison).

    Split Sizes by Task Type

    • Captioning Tasks:
      • COCO: 5,000 samples.
      • VATEX: 1,024 samples.
    • Question-Answering Tasks:
      • 1,024 samples.
    • Image Classification Tasks:
      • ImageNet: 10,000 samples (10 images per class).
      • Kinetics700: 7,000 samples (10 images per class).

    Support Set Sizes

    • General Tasks: 2,048 samples for both validation and test support.
    • Classification Tasks: 32 samples per class for both validation and test support.

  9. Setup environment for SmolLM2 fine-tuning

    main

    To prepare for fine-tuning SmolLM2 using the provided custom scripts, install PyTorch, install the project requirements, and authenticate with wandb, HuggingFace Hub, and accelerate.

    pip install -r requirements.txt

wandb login
huggingface-cli login
accelerate config

git clone https://github.com/huggingface/smollm
cd smollm/finetune