repository·master·Indexed 27 days ago
https://github.com/ericlbuehler/mistral.rsA high-performance LLM inference engine written in Rust providing zero-config inference for Hugging Face models. It supports multimodality (text, vision, video, audio), advanced quantization (GGUF, GPTQ, AWQ, HQQ, FP8, Bnb), and agentic capabilities including code and shell execution. The ecosystem includes crates for core model implementation (mistralrs-core), audio processing (mistralrs-audio), FlashAttention-2 CUDA kernels (mistralrs-flash-attn), and a Model Context Protocol client (mistralrs-mcp). It offers a CLI for interactive chat and API serving, a Python SDK, and hardware acceleration for CUDA, Metal, and MKL.
mistralrs-core is the central crate of the mistral.rs project. It provides the fundamental model implementations and the associated executors required for LLM inference. For detailed API documentation, refer to the official docs.rs page.mistralrs-flash-attn package provides a fork of Hugging Face Candle's FA2 (FlashAttention-2) CUDA kernels. It is specifically modified to support mistral.rs-owned attention paths, enabling optimized attention mechanisms within the mistral.rs ecosystem.mistral.rs allow interleaving non-text content parts (image_url, audio, video) with text. The engine preserves the order of content parts in the request body, ensuring that text surrounding a media element appears correctly on either side of the media tokens in the final uniform token stream processed by the transformer.The mistralrs engine supports a wide variety of model types, each accessible via a specific builder pattern:
ModelBuilderTextModelBuilderMultimodalModelBuilderGgufModelBuilderDiffusionModelBuilderSpeechModelBuilderEmbeddingModelBuilderModel::generate_structuredTool and ToolChoiceAgentBuilderLoraModelBuilder or XLoraModelBuilderAnyMoeModelBuilderMcpClientConfigConcurrency in mistral.rs is managed at the engine thread level:
The mistralrs Python package provides access to the core inference engine. The API is organized into several functional areas:
mistralrs-audio crate provides audio utilities for the mistral.rs ecosystem. It is designed to handle audio-related tasks such as reading audio files, resampling, managing audio channels, and computing mel spectrogram features. It functions similarly to mistralrs-vision but for audio data.The project provides two versions of calibration data sourced from a GitHub Gist. Choosing between them involves a trade-off between accuracy and processing time:
calibration_datav3.txt: The full original dataset containing 64,181 tokens. Use this for the most accurate calibration, though it will take the longest to process.calibration_datav3_small.txt: A subset containing up to line 1241 of the original data (30,173 tokens). Use this for faster calibration if full accuracy is not the primary concern.Agentic sessions in mistral.rs manage state for tool-calls, tool responses, and multimodal payloads (images, videos). This state is stored in memory and reconciled with incoming requests to maintain context across turns.
Requests match an existing session via:
session_id: A direct lookup using the provided ID.session_id is provided, the engine scans stored sessions and returns the first one where the user-visible message prefix matches the incoming messages. Note that if multiple sessions match, the returned one is arbitrary (not necessarily the longest match).When a session is matched, the engine performs 'splicing' to merge stored history with the new request:
The mistral.rs architecture is organized into three distinct layers to separate concerns between networking, orchestration, and hardware-specific execution:
To extend the project:
The mistralrs Python package allows you to run the mistral.rs inference engine in-process directly from Python. This enables high-performance LLM inference within your Python applications without needing to manage a separate server process.
To begin using the SDK, follow these steps:
UQFF is the native mistral.rs quantized file format. A UQFF export is a self-contained directory that must be distributed as a single unit. A single .uqff shard file is not loadable on its own; it requires the accompanying residual tensors and model assets located in the same directory.
Required Directory Contents:
<stem>-<shard>.uqff files containing quantized layers.residual.safetensors for unquantized tensors (e.g., norms, dense embeddings).config.json, tokenizer.json, tokenizer_config.json, generation_config.json, modules.json, chat_template.jinja, processor_config.json, preprocessor_config.json).Loading Mechanism:
A loader is pointed at one or more shard files using from_uqff. The loader automatically locates the residual.safetensors and JSON assets via sibling-path lookup.