DOCSEARCHbyAlgolia
Submit Docs

gemma.cpp

repository·main·Indexed 27 days ago

https://github.com/google/gemma.cpp

A lightweight, standalone C++ inference engine for Google's Gemma foundation models (Gemma 2, Gemma 3, and PaliGemma 2), optimized for research and experimentation on CPUs. It utilizes the Google Highway library for portable SIMD CPU inference and supports mixed-precision GEMM (fp8, bf16, fp32, fp64) and weight compression. The project includes C++ APIs, Python bindings via pybind11, a command-line application, and a local HTTP API server.

Tokens
4.5K
Snippets
12
Records
27
Agent score
92%

What's inside gemma.cpp

  1. Overview of gemma.cpp

    main
    gemma.cpp is a lightweight, standalone C++ inference engine designed for the Gemma foundation models (Gemma 2, Gemma 3, and PaliGemma 2). It is intended for experimentation and research, providing a minimalist implementation that is easy to embed and modify. It focuses on simplicity and directness rather than full generality, utilizing the Google Highway library for portable SIMD CPU inference.

  2. Core capabilities of gemma.cpp

    main

    gemma.cpp provides several key features for LLM research and inference:

    LLM Inference

    • CPU-only inference for Gemma 2-3 and PaliGemma 2.
    • Sampling methods including TopK and temperature.
    • Support for backward pass (VJP) and Adam optimizer for research purposes.

    Optimizations

    • Mixed-precision GEMM: Supports fp8, bf16, fp32, and fp64. Includes automatic runtime autotuning for 7 parameters per matrix shape.
    • Weight Compression: Integrated directly into GEMM, supporting custom fp8 (with 2..3 mantissa bits and tensor scaling), bf16, f32, and non-uniform 4-bit (NUQ).

    Infrastructure

    • SIMD: Single implementation via Highway, choosing the ISA at runtime.
    • Parallelism: CCX-aware tensor parallelism with a multi-socket thread pool.
    • Disk I/O: Supports memory mapping or parallel reads.
    • Portability: Supports Linux, Windows, and OS X via CMake or Bazel on any CPU.

    Frontends

    • C++ APIs with streaming support for single query and batched inference.
    • Basic interactive command-line application.
    • Basic Python bindings via pybind11.

  3. Incorporate gemma.cpp into a CMake project

    main

    The recommended way to use gemma.cpp as a library is via CMake's FetchContent. This automatically handles gemma.cpp and its required dependencies: sentencepiece and highway.

    Add the following to your CMakeLists.txt:

    include(FetchContent)

FetchContent_Declare(sentencepiece GIT_REPOSITORY https://github.com/google/sentencepiece GIT_TAG 53de76561cfc149d3c01037f0595669ad32a5e7c)
FetchContent_MakeAvailable(sentencepiece)

FetchContent_Declare(gemma GIT_REPOSITORY https://github.com/google/gemma.cpp GIT_TAG origin/main)
FetchContent_MakeAvailable(gemma)

FetchContent_Declare(highway GIT_REPOSITORY https://github.com/google/highway.git GIT_TAG 2a16a50ff61071bb25ddef0ce35d92b0e2b9c579)
FetchContent_MakeAvailable(highway)

    Then, link the libraries to your executable and include the necessary source directories:

    target_link_libraries([Executable Name] libgemma hwy hwy_contrib sentencepiece)
FetchContent_GetProperties(gemma)
FetchContent_GetProperties(sentencepiece)
target_include_directories([Executable Name] PRIVATE ${gemma_SOURCE_DIR})
target_include_directories([Executable Name] PRIVATE ${sentencepiece_SOURCE_DIR})
    include(FetchContent)

FetchContent_Declare(sentencepiece GIT_REPOSITORY https://github.com/google/sentencepiece GIT_TAG 53de76561cfc149d3c01037f0595669ad32a5e7c)
FetchContent_MakeAvailable(sentencepiece)

FetchContent_Declare(gemma GIT_REPOSITORY https://github.com/google/gemma.cpp GIT_TAG origin/main)
FetchContent_MakeAvailable(gemma)

FetchContent_Declare(highway GIT_REPOSITORY https://github.com/google/highway.git GIT_TAG 2a16a50ff61071bb25ddef0ce35d92b0e2b9c579)
FetchContent_MakeAvailable(highway)

target_link_libraries([Executable Name] libgemma hwy hwy_contrib sentencepiece)
FetchContent_GetProperties(gemma)
FetchContent_GetProperties(sentencepiece)
target_include_directories([Executable Name] PRIVATE ${gemma_SOURCE_DIR})
target_include_directories([Executable Name] PRIVATE ${sentencepiece_SOURCE_DIR})

  4. Start the local Gemma.cpp API server

    main

    Run the gemma_api_server binary to host a local HTTP server that implements the Google API protocol. You must provide paths to the tokenizer and model weights.

    Required arguments:

    • --tokenizer: Path to the tokenizer file (.spm).
    • --weights: Path to the model weights file (.sbs).

    Optional arguments:

    • --port: Port to listen on (default: 8080).
    • --model: Model name for API endpoints (default: gemma3-4b).
    ./build/gemma_api_server \
  --tokenizer path/to/tokenizer.spm \
  --weights path/to/model.sbs \
  --port 8080

  5. Build the Gemma.cpp API server and client

    main

    The API server and unified client are built as part of the main gemma.cpp project using CMake. After building, the binaries are located in the build/ directory.

    Binaries produced:

    • build/gemma_api_server: The local HTTP API server.
    • build/gemma_api_client: A unified client that can interact with either the local server or the public Google API.
    # Configure the build
cmake -B build -DCMAKE_BUILD_TYPE=Release

# Build the API server and client
cmake --build build --target gemma_api_server gemma_api_client -j 8

  6. Use the Unified Gemma.cpp Client

    main

    The gemma_api_client can be used to interact with your local server or the public Google API.

    Interacting with a Local Server:

    • Interactive mode: ./build/gemma_api_client --interactive 1 --host localhost --port 8080
    • Single prompt: ./build/gemma_api_client --prompt "Your prompt here"

    Interacting with the Public Google API:

    • Via Environment Variable: export GOOGLE_API_KEY="your-api-key-here" then run ./build/gemma_api_client --interactive 1
    • Via CLI Flag: ./build/gemma_api_client --api_key "your-api-key" --interactive 1

  7. Convert Keras or PyTorch weights to SBS format

    main

    To use models in gemma.cpp, you need the .sbs (stripped down binary blob) weight format.

    1. From Keras to PyTorch: Use the export_gemma_to_torch_xla.py script from the keras-team/keras-nlp repository.
    2. From PyTorch to SBS: Use the compression/convert_weights.py script provided in this repository to generate uncompressed weights.
    3. For PaliGemma: Use python/convert_from_safetensors to create an SBS file directly.

  8. Build gemma.cpp as a shared library

    main

    If you are not using FetchContent, you can build the libgemma library manually using make.

    1. Configure the build directory with cmake: cmake -B build

    2. Build the libgemma target using make: cd build && make -j [number of threads] libgemma

    On Unix platforms, this produces libgemma.a in the build/ directory.

    cmake -B build
cd build
make -j 4 libgemma

  9. Build gemma.cpp

    main

    The build process depends on your platform. It is recommended to delete the build/ directory if re-running CMake with different settings (rm -rf build/*).

    Unix-like Platforms (Linux/macOS)

    cmake -B build
cmake --preset make
cmake --build --preset make -j [number of parallel threads]

    Note: On WSL, set the number of parallel threads to 1 to avoid errors.

    Windows

    cmake --preset windows
cmake --build --preset windows -j [number of parallel threads]

    Bazel

    bazel build -c opt --cxxopt=-std=c++20 :gemma

    Make

    A Makefile is available via jart/gemma3.

    cmake -B build
cmake --preset make
cmake --build --preset make -j $(nproc)

  10. Install System Requirements

    main

    Before building gemma.cpp, ensure the following are installed:

    Windows Native Build Requirements: Building natively on Windows requires Visual Studio 2012 Build Tools with the Clang/LLVM C++ frontend (clang-cl). You can install these via winget:

    winget install --id Kitware.CMake
winget install --id Microsoft.VisualStudio.2022.BuildTools --force --override "--passive --wait --add Microsoft.VisualStudio.Workload.VCTools;installRecommended --add Microsoft.VisualStudio.Component.VC.Llvm.Clang --add Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset"
    winget install --id Kitware.CMake
winget install --id Microsoft.VisualStudio.2022.BuildTools --force --override "--passive --wait --add Microsoft.VisualStudio.Workload.VCTools;installRecommended --add Microsoft.VisualStudio.Component.VC.Llvm.Clang --add Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset"

  11. Convert fine-tuned checkpoints to .sbs format

    main

    To convert PaliGemma 2 checkpoints from safetensors format to .sbs, use python/convert_from_safetensors.py. This requires a Bazel build of the compression library.

    1. Build the compression library:
    bazel build //compression/python:compression
    1. Link the built site-packages to your environment:
    BAZEL_OUTPUT_DIR="${PWD}/bazel-bin/compression"
ln -s $BAZEL_OUTPUT_DIR [path_to_your_site_packages]/compression
    1. Run the conversion:
    python3 python/convert_from_safetensors.py --load_path [...].safetensors.index.json
    bazel build //compression/python:compression
BAZEL_OUTPUT_DIR="${PWD}/bazel-bin/compression"
python3 -c "import site; print(site.getsitepackages())"
# Use your sites-packages file here:
ln -s $BAZEL_OUTPUT_DIR [...]/site-packages/compression
python3 python/convert_from_safetensors.py --load_path [...].safetensors.index.json

  12. Obtain Model Weights and Tokenizer

    main
    1. Visit the Kaggle page for Gemma-2 and select Model Variations |> Gemma C++.
    2. Choose a variation from the dropdown. For faster inference, use -sfp (8-bit switched floating point) weights. For higher fidelity, use bfloat16.
    3. Recommendation: Start with gemma2-2b-it-sfp.
    4. Download the archive.tar.gz file and extract it using tar:
    tar -xf archive.tar.gz

    This will produce model weights (e.g., 2b-it-sfp.sbs) and a tokenizer file (tokenizer.spm).

    tar -xf archive.tar.gz