llama-cpp-turboquant Documentation

The GGML-VirtGPU backend allows GGML applications (like llama.cpp) running inside a virtual machine (Guest) to perform machine learning computations on the host hardware. It achieves this by splitting the backend into two parts:

1. Guest-side Frontend (ggml-virtgpu/): Implements the GGML backend interface and forwards operations to the host via hypercalls and shared memory.
2. Host-side Backend (ggml-virtgpu/backend/): Receives forwarded operations and executes them on actual hardware backends (e.g., Metal, Vulkan, CUDA, or CPU).

This architecture enables high-performance computation in virtualized environments using zero-copy data transfers via host-guest shared memory.

GBNF (GGML BNF) is a grammar format used to constrain model outputs in llama.cpp. It allows you to force a model to follow specific syntax, such as valid JSON, specific notation (like chess moves), or restricted character sets (like emojis).

GBNF grammars are supported via:

• llama-cli and llama-completion using --grammar or --grammar-file flags.
• llama-server completion endpoints via the grammar body field.
• test-gbnf-validator for testing grammars against strings.

./llama-cli -m <model> --grammar-file grammars/some-grammar.gbnf -p 'Some prompt'

The common library provides a Parsing Expression Grammar (PEG) implementation specifically designed for parsing model outputs. It supports partial parsing of streaming input, built-in JSON parsing, and AST generation with semantic tagging.

There are two main namespaces of types:

• common_peg_*: General-purpose PEG types for various parsing tasks.
• common_chat_peg_*: Specialized helpers optimized for parsing model chat outputs (e.g., extracting reasoning, content, and tool calls).

The Web UI provides several capabilities for interacting with llama-server:

• Chat Interface: Supports streaming responses.
• Multi-model Support (ROUTER mode): Allows switching between models with automatic loading upon selection.
• Modality Validation: Automatically checks if the selected model supports specific attachments (images, audio).
• Conversation Management: Supports branching, regeneration, and editing while preserving history.
• Attachment Support: Handles images, audio, and PDFs (utilizing vision or text fallbacks).
• Configurable Parameters: Syncs parameters like temperature and top_p with server defaults.
• Theming: Supports both Dark and Light modes.

libmtmd is the modern library designed to replace the original llava.cpp implementation for handling multimodal inputs. It is built upon clip.cpp and provides several key improvements:

• Unified Interface: Consolidates interaction for various multimodal models into a single system.
• Improved UX/DX: Offers a more intuitive API inspired by the Processor class in the Hugging Face transformers library.
• Flexibility: Supports multiple input types, including text, audio, and images, while managing the diverse chat templates required by different models.

In JSON_NATIVE mode, the entire tool call (function name, arguments, and values) is contained within a JSON structure. The function name is detected when it appears inside a JSON structure (e.g., preceded by { or :).

// Standard OpenAI-style
<tool_call>
{"name": "get_weather", "arguments": {"location": "Paris", "unit": "celsius"}}
</tool_call>

// Mistral Nemo with array wrapper
[TOOL_CALLS]
[{"name": "calculate", "arguments": {"expr": "2+2"}}]

// Function name as JSON key (Apertus style)
{"get_weather": {"location": "Paris"}}

In TAG_WITH_TAGGED mode, both the function name and the argument names are contained within XML-style tags. String values are unquoted, while non-string values (like objects or booleans) use JSON formatting.

<!-- Qwen/Hermes XML format -->
<function=get_weather>
<param=location>Paris</param>
<param=unit>celsius</param>
</function>

<!-- Mixed types -->
<function=calculate>
<param=expr>2+2</param>
<param=precision>2</param>
<param=options>{"round": true}</param>
</function>

When agenticConfig.enabled is true and MCP servers are connected, the chatStore delegates the request to agenticStore.runAgenticFlow. This initiates a multi-turn loop:

1. Turn Execution: The agent calls ChatService.sendMessage().
2. Tool Detection: If the API response contains tool_calls, the loop continues.
3. Tool Execution: For each tool_call, the mcpStore.executeTool() is called.
4. Result Integration: Tool results (including base64 attachments) are appended to the conversation as new messages.
5. Loop/Termination: The loop repeats with the updated message history until either no more tool_calls are returned or maxTurns is reached.
6. Finalization: The final response and accumulated timings are returned to the chatStore.