BeeLlama.cpp Documentation

GBNF (GGML BNF) is a format used to constrain model outputs in llama.cpp by defining formal grammars. It allows you to force models to follow specific structures, such as valid JSON, specific notation (like chess moves), or restricted character sets (like emojis).

GBNF grammars can be applied via:

• llama-cli and llama-completion using the --grammar or --grammar-file flags.
• llama-server completion endpoints by passing a grammar field in the request body.

To test a grammar file with the CLI, use:

./llama-cli -m <model> --grammar-file grammars/some-grammar.gbnf -p 'Some prompt'

The common library provides a PEG (Parsing Expression Grammar) parser implementation specifically designed for parsing model output. It supports partial parsing of streaming input, built-in JSON parsing, and AST (Abstract Syntax Tree) generation using 'tagged' nodes.

There are two main types of parsers:

• common_peg_*: General-purpose PEG parser types.
• common_chat_peg_*: Specialized helpers optimized for model output (e.g., handling reasoning content, tool calls, and chat formats).

The source code is organized as follows:

• src/lib/components/: UI components (subdivided into app/ and ui/).
• src/lib/hooks/: Svelte hooks.
• src/lib/stores/: State management logic.
• src/lib/services/: API and database services.
• src/lib/types/: TypeScript interfaces.
• src/lib/utils/: Utility functions.
• src/routes/: SvelteKit routes.
• src/styles/: Global styles.
• static/: Static assets.
• tests/: Test files.
• docs/: Architecture diagrams (high-level and feature-specific flows).
• .storybook/: Storybook configuration.

The Web UI provides a comprehensive interface for model interaction, including:

• Chat interface: Supports streaming responses.
• Multi-model support (ROUTER mode): Allows switching between models with automatic loading upon selection.
• Modality validation: Automatically checks if the selected model supports specific attachments like images or audio.
• Conversation management: Supports branching, regeneration, and editing while preserving history.
• Attachment support: Handles images, audio, and PDFs (using vision or text fallbacks).
• Configurable parameters: Syncs parameters like temperature and top_p with server defaults.
• Theming: Supports both dark and light modes.

libmtmd

libmtmd is the modern replacement for the original llava.cpp implementation. It is designed to provide a unified interface for various multimodal models, supporting multiple input types (text, audio, and images) while handling complex model-specific chat templates. Its API is inspired by the Processor class in the Hugging Face transformers library.

mtmd-cli

mtmd-cli is the consolidated command-line tool powered by libmtmd. It replaces various model-specific binaries (like qwen2vl-cli, minicpmv-cli, and gemma3-cli) into a single, unified interface for interacting with multimodal models.

The ggml-rpc-server allows you to expose ggml devices (like GPUs or CPUs) on a remote host. The RPC backend in llama.cpp communicates with one or more ggml-rpc-server instances via TCP to offload computations, enabling distributed LLM inference across multiple machines.

IMPORTANT

This is a proof-of-concept. The functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!

The GGML-VirtGPU backend allows GGML applications (like llama.cpp) to run machine learning computations on host hardware while the application itself is running inside a virtual machine (Guest VM). It achieves this by using host-guest shared memory for zero-copy data transfer and virtio-gpu hypercalls to trigger remote execution.

The architecture is split into two parts:

1. Guest-side Frontend (ggml-virtgpu/): Implements the GGML backend interface and forwards operations to the host.
2. Host-side Backend (ggml-virtgpu/backend/): Receives forwarded operations and executes them on actual hardware (e.g., Metal, Vulkan, CUDA, or CPU).

Multimodal support in llama.cpp functions by encoding images into embeddings using a separate model component, which are then fed into the language model.

To run a multimodal model, you typically need two distinct GGUF files:

1. The standard language model file: The core LLM.
2. A multimodal projector (mmproj) file: A specific component that handles image encoding and projection.

This separation allows for independent development of vision architectures (like Vision Transformers) without modifying the core libllama library.

KVarN (Key-Value Quantization with Residuals) differs from standard quantized caches in several ways:

Key Characteristics

• Intrinsic Suffix: KVarN always resolves at least a 128-token exact suffix. Positive requests for tail tokens round upward to 128.
• Eager Writing: Completed compressed records are written eagerly even while tokens coexist in the exact suffix.
• Memory/RAM Management:
  • Unified KVarN: Requires an exclusive structured stream for RAM save/restore. Contention during save results in no cache entry; contention during restore results in a cache miss.
  • Non-unified KVarN: Multi-slot RAM caching and group-aligned historical reuse are unconditional.
• Compatibility: BeeLlama v0.3.x sessions and v11 KVarN state are incompatible with v0.4.0 cache type IDs. Restore will fail closed; there is no migration shim.

In JSON_NATIVE mode, the entire tool call (function name, arguments, and values) is contained within a JSON structure. The parser detects this when the function name appears inside a JSON context (e.g., preceded by { or :).

Examples:

Standard OpenAI-style:

<tool_call>
{"name": "get_weather", "arguments": {"location": "Paris", "unit": "celsius"}}
</tool_call>

Mistral Nemo with array wrapper:

[TOOL_CALLS]
[{"name": "calculate", "arguments": {"expr": "2+2"}}]

Function name as JSON key (Apertus style):

{"get_weather": {"location": "Paris"}}