DOCSEARCHbyAlgolia
Submit Docs

Candle ML Framework

repository·main·Indexed 12 days ago

https://github.com/huggingface/candle

A minimalist machine learning framework for Rust designed for high performance and ease of use. Candle provides core primitives for building and running ML models with strong GPU support (CUDA/Metal) and WASM compatibility.

Tokens
97.1K
Snippets
411
Records
489
Agent score
97%

What's inside Candle

  2. Explore Candle features and supported models

    main

    Candle provides a high-performance ML framework with the following capabilities:

    Core Features

    • Syntax: PyTorch-like API with support for model training and user-defined kernels (e.g., Flash Attention v2).
    • Backends:
      • CPU: Optimized with optional MKL (x86) or Accelerate (macOS) support.
      • CUDA: GPU acceleration with multi-GPU distribution via NCCL.
      • WASM: Support for running models in web browsers.
    • File Formats: Load models from safetensors, npz, ggml, or PyTorch files.
    • Quantization: Support for llama.cpp quantized types.

    Supported Model Categories

    • Language Models: LLaMA (v1, v2, v3), Falcon, StarCoder, Phi (1, 1.5, 2, 3), Gemma, Mistral, Mixtral, Qwen, etc.
    • Quantized LLMs: Llama (7b to 70b), Mistral, Mixtral, Zephyr, OpenChat, Qwen3 MoE.
    • Text-to-Text: T5 variants, Marian MT.
    • Text-to-Image: Stable Diffusion (v1.5, v2.1, XL v1.0).
    • Image-to-Text: BLIP, TrOCR.
    • Audio: Whisper, EnCodec, MetaVoice-1B, Parler-TTS.
    • Computer Vision: DINOv2, YOLO (v3, v8), Segment-Anything (SAM), etc.

  4. SmolLM Model Family Overview

    main

    The SmolLM family includes several model architectures:

    SmolLM2

    Uses the standard Llama3 architecture (implemented in models/llama.rs).

    • Variants: HuggingFaceTB/SmolLM2-135M, HuggingFaceTB/SmolLM2-360M, HuggingFaceTB/SmolLM2-1.7B.

    SmolLM3

    Introduces the NoPE architecture and supports long context.

    • Implementations: smollm3.rs (Full precision) and quantized_smollm3.rs (GGUF with weight reconstruction).
    • Variants: HuggingFaceTB/SmolLM3-3B (Instruct), HuggingFaceTB/SmolLM3-3B-Base, and quantized versions from unsloth/SmolLM3-3B-GGUF.

    SmolVLM (Planned)

    Vision-language model variant.

  8. Inspect Parquet dataset files

    main

    Candle training workflows often utilize standardized parquet files (found on the refs/convert/parquet branch of datasets on Hugging Face). You can use parquet::file::serialized_reader::SerializedFileReader to inspect the contents of these files.

    When inspecting a dataset like MNIST, you will typically see columns representing the features and labels, for example:

    • label: The integer class identifier.
    • image: The raw bytes of the image data.
    Column id 1, name label, value 6
Column id 0, name image, value {bytes: [137, ....]}
Column id 1, name label, value 8
Column id 0, name image, value {bytes: [137, ....]}

  9. Understand the Candle repository structure

    main

    The Candle ecosystem is organized into several specialized crates:

    • candle-core: The foundation containing core operations, device abstractions, and the Tensor struct.
    • candle-nn: High-level tools and modules for building neural network models.
    • candle-examples: Realistic implementation examples.
    • candle-datasets: Data loading and dataset management.
    • candle-transformers: Utilities specifically for transformer architectures.
    • candle-flash-attn: Implementation of the Flash Attention v2 layer.
    • candle-kernels: Custom CUDA kernels.
    • candle-onnx: Tools for evaluating ONNX models.
    • candle-pyo3: Python bindings.

  10. Configure SmolLM3 Thinking Mode

    main

    SmolLM3 supports an explicit reasoning mode using <think> tags. This can be controlled via the prompt template or the --thinking CLI flag in the example implementation.

    • Enabled: The model generates reasoning inside <think> tags. The prompt prefix is <|im_start|>assistant\n<think>\n.
    • Disabled: The model skips reasoning and goes straight to the answer. The prompt prefix is <|im_start|>assistant\n<think>\n\n</think>\n.

  11. Process multiple images with PaddleOCR-VL

    main

    PaddleOCR-VL supports multi-image processing for multi-page documents. You can use two different methods:

    1. Combined Output: Pass multiple --image flags. This processes images together in a single prompt (defaults to --task ocr).
    2. Batch Processing: Use the --batch flag to process images sequentially with distinct outputs. This supports shell glob expansion.
    # Combined output (multi-page document in one prompt)
cargo run --example paddleocr-vl --release -- \
    --image candle-examples/examples/paddleocr-vl/test_ocr.png \
    --image candle-examples/examples/paddleocr-vl/test_ocr_page2.png

# Batch processing (sequential with distinct output)
# Using explicit files
cargo run --example paddleocr-vl --release -- \
    --batch candle-examples/examples/paddleocr-vl/test_ocr.png candle-examples/examples/paddleocr-vl/test_ocr_page2.png

# Using shell glob expansion
cargo run --example paddleocr-vl --release -- \
    --batch candle-examples/examples/paddleocr-vl/test_ocr*.png

  12. Select RWKV v7 model variants

    main

    RWKV v7 models follow the naming convention rwkv7{variant}-g{generation}{dataset}-{size}.

    • Base models (rwkv7-g1d): Fastest models, available in sizes from 0.1B to 13.3B.
    • rwkv7a (DeepEmbed): Adds token-dependent gating in the FFN layer for better context awareness. Minimal overhead.
    • rwkv7b (DEA): Adds Deep Embedding Attention (a full quadratic attention mechanism) alongside linear attention. Slower but better for precise token relationships.
    # v7a with DeepEmbed
cargo run --example rwkv --release -- \
  --which rwkv7a-g1d-0.1b --template chat \
  --prompt "Summarize this: The quick brown fox jumps over the lazy dog."

# v7b with DEA
cargo run --example rwkv --release -- \
  --which rwkv7b-g1b-0.1b --template chat \
  --prompt "What is 2+2?"