llama.cpp

llama.cpp is a high-performance LLM (and VLM) inference engine written in plain C/C++ with minimal dependencies.

Key Features:

• Hardware Optimization: Optimized for Apple Silicon (Metal, Accelerate, ARM NEON), x86 (AVX, AVX2, AVX512, AMX), and RISC-V.
• Quantization: Supports 1.5-bit through 8-bit integer quantization to reduce memory usage and increase speed.
• GPU Acceleration: Supports NVIDIA (CUDA), AMD (HIP), Moore Threads (MUSA), Vulkan, and SYCL.
• Hybrid Inference: Supports CPU+GPU hybrid inference for models larger than available VRAM.
• Backend Support: Extensive support including OpenCL, WebGPU, and ZenDNN.

The LLaMA.cpp HTTP Server is a fast, lightweight REST API and web UI built using httplib, nlohmann::json, and llama.cpp. It provides a set of endpoints to interact with Large Language Models (LLMs) via inference on both CPU and GPU, supporting both F16 and quantized models.

Key capabilities include:

• API Compatibility: Supports OpenAI-compatible routes (chat completions, responses, embeddings) and Anthropic Messages API-compatible chat completions.
• Advanced Inference: Features continuous batching, parallel decoding for multi-user support, speculative decoding, and schema-constrained JSON response formats.
• Specialized Endpoints: Includes a Reranking endpoint, multimodal support, and monitoring endpoints.
• Tool Use: Supports function calling and tool use for most models.
• User Interface: Includes an easy-to-use web UI for direct interaction.

The llama.cpp Jinja Engine is a C++ implementation of the Jinja template engine, designed for processing chat templates. It is architected to be lightweight and secure, featuring a decoupled design from nlohmann::json (which is only used optionally for JSON-to-internal type translation).

Key characteristics include:

• Minimal Primitive Types: Supports int, float, bool, string, array, object, none, and undefined.
• Detailed Logging: Provides source tracing during errors.
• Security: Implements input marking to prevent special token injection attacks.

llama-ui is a modern, feature-rich web interface for llama-server built with SvelteKit. It provides an intuitive chat interface with support for advanced file handling, conversation management, and comprehensive model interaction. It operates in two distinct modes:

• MODEL mode: Standard single-model operation using a single llama-server instance.
• ROUTER mode: Multi-model operation that allows for dynamic loading and unloading of models.

Key features include streaming responses, support for reasoning/thinking blocks, file attachments (images, PDFs, audio, text), conversation branching, and advanced Markdown rendering (KaTeX, syntax highlighting).

The llama-server Web UI provides a comprehensive interface for model interaction, including:

• Chat Interface: Supports streaming responses and conversation management (branching, regeneration, and editing with history preservation).
• Multi-model Support (ROUTER mode): Allows switching between models with automatic loading upon selection.
• Modality Validation: Validates that the selected model supports specific attachments like images, audio, or PDFs.
• Attachment Support: Handles images, audio, and PDFs (utilizing vision/text fallbacks).
• Parameter Configuration: Syncs parameters like temperature and top_p with server defaults.
• Theming: Supports both dark and light modes.

The tools/ui directory is organized as follows:

• src/lib/components/: UI components (split into app/ and ui/).
• src/lib/hooks/: Svelte hooks.
• src/lib/stores/: State management logic.
• src/lib/services/: API and database services.
• src/lib/types/: TypeScript interfaces.
• src/lib/utils/: Utility functions.
• src/routes/: SvelteKit routes.
• src/styles/: Global styles.
• static/: Static assets.
• tests/: Test files.
• docs/: Architecture diagrams and feature flows.

The GGML-VirtGPU backend allows GGML applications (like llama.cpp) to run machine learning computations on the host hardware while the application itself is running inside a virtual machine (Guest VM).

It achieves this by splitting the backend into two parts:

1. Guest-side Frontend (ggml-virtgpu/): Implements the GGML backend interface and forwards operations to the host via hypercalls and shared memory.
2. Host-side Backend (ggml-virtgpu/backend/): Receives forwarded operations and executes them using actual hardware backends like Metal, Vulkan, or CUDA.

This setup is particularly useful for running high-performance inference in isolated environments like MacOS containers.

GBNF (GGML BNF) is a formal grammar format used in llama.cpp to constrain Large Language Model (LLM) outputs. By defining a grammar, you can force the model to follow specific structures, such as valid JSON, specific programming languages, or even restricted sets like emojis or chess notation.

GBNF is supported in:

• llama-cli and llama-completion via CLI flags.
• llama-server via the grammar field in completion API requests.
• test-gbnf-validator for testing grammars against strings.

# Example usage with llama-cli
./llama-cli -m <model> --grammar-file grammars/some-grammar.gbnf -p 'Some prompt'

libmtmd

libmtmd is the modern library designed to replace the legacy llava.cpp implementation. It provides a unified interface for handling multimodal inputs (text, audio, and images) and is designed with an API inspired by the Hugging Face transformers Processor class. It is built to handle the complex chat templates required by various vision-capable models.

mtmd-cli

mtmd-cli is the consolidated command-line interface powered by libmtmd. It replaces various model-specific binaries (like qwen2vl-cli or minicpmv-cli) with a single, unified tool for interacting with different multimodal architectures.

GBNF uses production rules to define how patterns are constructed.

• Production Rule: Defined as nonterminal ::= sequence....
• Non-Terminals: Rule names that represent a pattern. They must be lowercase, dashed words (e.g., move, check-mate).
• Terminals: Actual characters or Unicode code points.
  • Can be literal strings: "1. " or "O-O".
  • Can be character ranges: [1-9] or [NBKQR].
  • Can be negated ranges: [^ ]+ (matches anything except a newline).
  • Support Unicode escapes: \xXX (8-bit), \uXXXX (16-bit), or \UXXXXXXXX (32-bit).
• The root Rule: Every full grammar must include a root rule, which defines the pattern for the entire model output.

root ::= ("- " item)+
item ::= [^
]+ "\n"