chatllm.cpp Documentation

GGMM is an extension of the GGML format designed for chatllm.cpp. It works by prepending a meta field to the existing Config structure. The meta field contains a JSON string that supports non-ASCII characters.

Currently, this metadata is used to provide information that can customize the model's presentation, such as the model name displayed in the application banner. While the long-term goal is to use this metadata to eliminate the Config structure entirely, it currently serves as a way to embed model-specific identity information directly into the file.

{
    "model_name": "...",
    "model_native_name": "..."
}

ChatLLM.cpp provides a Python script, model_downloader.py, to download quantized models on demand.

There are two ways to trigger a download:

1. Via CLI: When using main_nim, provide a model name starting with a colon (e.g., -m :model_id:version) to the -m option. If the model file does not exist locally, the downloader will attempt to fetch it.
2. Via Script: Run the downloader script directly to see the list of available quantized models.

To see all available models that can be downloaded, run:

python model_downloader.py

The Linear lens applies a rotation matrix $M$ to the normalized activations before decoding: $len(l) = W_U \space {norm}(M h_l)$. This is typically used with pre-fitted Jacobian lenses to better interpret model internals.

To use a Linear lens, you must first provide a path to a converted Jacobian lens file. The command format is --lens linear LAYERS /path/to/quantized_name.bin.

# Example: Apply linear lens to all layers using a specific lens file
./chatllm --lens linear all /path/to/quantized_name.bin

You can build the project using CMake. The resulting executable will be located at ./build/bin/main.

Standard build:

cmake -B build
cmake --build build -j --config Release

To enable specific hardware acceleration or features, use GGML_... flags. For example, to enable Vulkan acceleration, RPC, all CPU variants, and backend dynamic loading:

cmake -B build -DGGML_VULKAN=1 -DGGML_RPC=1 -DGGML_CPU_ALL_VARIANTS=1 -DGGML_BACKEND_DL=1

To chat with a quantized model in an interactive session, use the -i flag. In interactive mode, your chat history is preserved as context for the next turn.

Linux or WSL

It is recommended to use rlwrap for a better terminal experience:

rlwrap ./build/bin/main -m model.bin -i

Windows

.\build\bin\Release\main -m model.bin -i

Basic execution (non-interactive)

To run a single prompt or specific seed:

./build/bin/main -m llama2.bin --seed 100

Use ./build/bin/main -h to see all available options.

To enable GPU acceleration, you must build the project with the appropriate backend flag enabled via CMake. Supported backends include CUDA (for Nvidia GPUs) and Vulkan (for general GPU support).

To build with Vulkan:

cmake -B build -DGGML_VULKAN=1
cmake --build build --config Release

To build with CUDA:

cmake -B build -DGGML_CUDA=1
cmake --build build --config Release

The following Python scripts provide implementation demos for tool calling (function calling or code interpretation) across different model architectures. Note that for some models like DeepSeek-Coder v2, function calling is officially unsupported but may still be demonstrated.

# QWen v1.5 MoE
python tool_qwen.py -i -m :qwen1.5:moe

# QWen v2
python tool_qwen.py -i -m :qwen2:1.5b

# DeepSeek-Coder v2
python tool_deepseekcoder.py -i -m :deepseek-coder-v2:light

# InternLM 2.5
python tool_internlm.py -i -m :internlm2.5 --temp 0

# CodeGeeX4
python tool_codegeex4.py -m :codegeex4

# NuminaMath (Note: F16 performs better than Q8)
python tool_numinamath.py --temp 0 -m :numinamath

# LlaMA3-Groq
python tool_groq.py --temp 0 -m /path/to/llama3-groq-tool-8b.bin

To enable multimodal capabilities (processing images, audio, or video), ChatLLM.cpp relies on external tools. Ensure the following are installed and available in your OS's searching path:

• ffmpeg (v7.0.2 or compatible): Used for video and audio I/O via ffmpeg and ffplay.
• ImageMagick (v7.1.1 or compatible): Used for image I/O via the magick command.

Before using a pre-fitted Jacobian lens (e.g., from HuggingFace) with the Linear lens, you must convert it using the convert_j_lens.py script.

Workflow:

1. Download your model and convert it for chatllm.cpp.
2. Download the pre-fitted Jacobian lens for your specific model.
3. Run the conversion script to create a .bin file compatible with chatllm.cpp.

python convert_j_lens.py -i /path/to/model/file -o quantized_name.bin --name <lens_name>