DOCSEARCHbyAlgolia
Submit Docs

ggml Tensor Library

repository·master·Indexed 12 days ago

https://github.com/ggml-org/ggml

A low-level, cross-platform tensor library for machine learning designed for high performance. It features integer quantization, automatic differentiation, and zero memory allocations during runtime. The library supports CPU-based inference for models like GPT-2, Cerebras-GPT, and GPT-J 6B, and provides tools for converting models to ggml and GGUF formats, as well as training and evaluating MNIST models.

Tokens
10.2K
Snippets
34
Records
51
Agent score
97%

What's inside ggml

  1. Overview of ggml features

    master

    ggml is a low-level tensor library for machine learning designed with the following characteristics:

    • Low-level cross-platform implementation: Optimized for various operating systems.
    • Integer quantization support: Enables efficient model compression.
    • Broad hardware support: Compatible with diverse hardware architectures.
    • Automatic differentiation: Supports gradient-based learning.
    • Optimizers: Includes ADAM and L-BFGS optimizers.
    • No third-party dependencies: Minimizes external requirements.
    • Zero memory allocations during runtime: Ensures predictable performance and stability during inference/training.

  2. Difference between simple-ctx and simple-backend examples

    master

    The repository provides two distinct simple examples for understanding usage:

    1. simple-ctx: A basic example demonstrating matrix multiplication and core ggml usage. Note that simple-ctx does not support GPU acceleration.
    2. simple-backend: An example designed to demonstrate how to utilize hardware-specific backends such as CUDA and Metal.

  3. What is GGUF and its key features

    master

    GGUF is a successor to the GGJT format designed for machine learning model deployment. It is optimized for single-file distribution, extensibility, and performance.

    Key features include:

    • Single-file deployment: All necessary information is contained within one file.
    • Extensibility: New metadata can be added without breaking compatibility with older executors.
    • mmap compatibility: Supports memory-mapped loading for fast I/O.
    • Ease of use: Designed to be loaded/saved with minimal code and no external libraries.
    • Full information: Contains all hyperparameters and metadata required for inference.

  4. What is GGUF and when to use it

    master

    GGUF is a binary file format designed for storing models for inference with GGML and GGML-based executors. It is optimized for fast loading, fast saving, and ease of reading.

    Developers typically use GGUF when they have models developed in frameworks like PyTorch and need to convert them for use within the GGML ecosystem. GGUF is the successor to the GGML, GGMF, and GGJT formats and is designed to be unambiguous by including all necessary metadata required to load a model, ensuring that no external information is needed. It is also extensible, allowing new metadata or features to be added without breaking backward compatibility.

  5. Comparison of GGML, GGMF, and GGJT Formats

    master

    Before the standardization of GGUF, three primary formats were used for LLMs. Understanding these is useful for context when dealing with legacy models:

    • GGML (unversioned): The baseline format; lacks versioning and alignment.
    • GGMF (versioned): An extension of GGML that includes versioning.
    • GGJT: A format that aligns tensors to allow for mmap usage. While v1-v3 are structurally similar, later versions use quantization schemes incompatible with earlier ones.

    Common Structure of these formats:

    1. Magic number (with optional version).
    2. Model-specific hyperparameters (metadata like layer count, heads, and ftype).
    3. Embedded vocabulary (list of strings with length prepended).
    4. List of tensors (name, type, and data).

  6. GGUF Naming Convention

    master

    GGUF files follow a specific naming convention to allow humans to identify model details at a glance. The format is: [<Sidecar>]<BaseName><SizeLabel><FineTune><Version><Encoding><Type><Shard>.gguf.

    Components:

    • Sidecar (Optional): Prefix for auxiliary modules (e.g., mmproj for multimodal projectors, mtp for Multi-Token Prediction heads).
    • BaseName: Descriptive name of the architecture (e.g., Llama-3).
    • SizeLabel: Parameter weight class (e.g., 8B, 70B, 8x7B).
    • FineTune (Optional): The tuning goal (e.g., Instruct, Chat).
    • Version (Optional): Formatted as v<Major>.<Minor> (e.g., v1.0). Defaults to v1.0 if missing.
    • Encoding: The weight encoding scheme (e.g., F16, Q4_K_M).
    • Type (Optional): Indicates if the file is a LoRA adapter or contains only vocab data.
    • Shard (Optional): Indicates split files, formatted as 00001-of-00005 (5-digit padded).

  7. How matrix multiplication works in ggml

    master

    In ggml, matrix multiplication is performed using the ggml_mul_mat operation. Unlike traditional row-by-column multiplication ($A imes B = C$), ggml expects the second matrix ($B$) to be provided in its transposed form, and the resulting matrix ($C$) is also returned in a transposed state.

    The mathematical relationship used is:

    ggml_mul_mat(A, B^T) = C^T

    This approach is optimized for the library's backend handling and memory layout.

    // Traditional: A (m x k) * B (k x n) = C (m x n)
// ggml: ggml_mul_mat(A, B_transposed) = C_transposed

  8. Use hardware acceleration in MNIST examples

    master

    The training and evaluation code is hardware-agnostic. GGML will use available backends (like CUDA) if the operations are implemented.

    To preferentially use a specific backend, append the backend name to the command. If a backend does not support a specific operation, GGML will fall back to the CPU, which may impact performance.

  9. LLM Architecture Metadata Keys

    master

    LLM metadata uses a namespaced approach [llm]. where [llm] is the architecture name (e.g., llama.context_length).

    Common LLM Keys

    • [llm].context_length: uint64: (n_ctx) The trained context length limit.
    • [llm].embedding_length: uint64: (n_embd) Embedding layer size.
    • [llm].block_count: uint64: Number of attention+feed-forward layers.
    • [llm].feed_forward_length: uint64: (n_ff) Length of the feed-forward layer.
    • [llm].tensor_data_layout: string: Describes tensor rearrangement. Default is reference.
    • [llm].expert_count: uint32: Number of experts (for MoE models).

    Attention Keys

    • [llm].attention.head_count: uint64: (n_head) Number of attention heads.
    • [llm].attention.head_count_kv: uint64: Number of heads per group (for GQA).
    • [llm].attention.layer_norm_epsilon: float32: Layer normalization epsilon.
    • [llm].attention.layer_norm_rms_epsilon: float32: Layer RMS normalization epsilon.

    RoPE Scaling Keys

    For adjusting context length via RoPE scaling:

    • [llm].rope.scaling.type: string: none, linear, or yarn.
    • [llm].rope.scaling.factor: float32: Scale factor.
    • [llm].rope.scaling.original_context_length: uint32_t: Original base model context length.
    • [llm].rope.scaling.finetuned: bool: Whether the model was finetuned with scaling.

  10. GGUF File Structure and Alignment

    master

    GGUF files are structured sequentially and use a global alignment specified in the general.alignment metadata field (ALIGNMENT). Fields and arrays are padded with 0x00 bytes to reach the next multiple of this alignment value.

    Core Components:

    1. Header: Contains the magic number (GGUF), version, tensor count, and metadata count.
    2. Metadata KV Pairs: A collection of key-value pairs for hyperparameters.
    3. Tensor Infos: An array describing each tensor's name, dimensions, type, and its offset within the tensor_data block.
    4. Padding: Ensures the transition from tensor info to tensor data respects ALIGNMENT.
    5. Tensor Data: The raw binary weights of the model.

    Endianness: Models are little-endian by default. If a model is big-endian, all values (metadata and tensors) will be big-endian.

  11. Convert GPT-J models to ggml format

    master

    If you have the original GPT-J model weights (e.g., from Hugging Face) and want to convert them to the ggml format yourself, follow these steps:

    1. Download the full GPT-J model (e.g., the 6B version, which is ~72 GB).
    2. Use the convert-h5-to-ggml.py script to perform the conversion.

    This will generate a ggml-model.bin file compatible with the gpt-j inference tool.

  12. Convert Magika H5 models to GGUF format

    master

    To use Google Magika with GGML, you must first convert the original Magika model from H5 format to GGUF format using the provided convert.py script.

    1. Obtain the Magika model in .h5 format (e.g., from the official Magika repository).
    2. Run the conversion script pointing to your .h5 file.
    python examples/magika/convert.py /path/to/model.h5