DOCSEARCHbyAlgolia
Submit Docs

LARQL Documentation

repository·main·Indexed 21 days ago

https://github.com/chrishayuk/larql

LARQL treats transformer models as queryable graph databases by decompiling them into a 'vindex' format, allowing users to browse, edit, and perform inference on neural network weights using the LQL language. The ecosystem includes larql-cli for vindex lifecycle management, diagnostics, and weight-space extraction, as well as larql-boundary, a confidence-gated boundary reference codec for managing a two-tier memory model of hot KV windows and compressed residual checkpoints.

Tokens
543K
Snippets
1.4K
Records
2.3K
Agent score
78%

What's inside LARQL

  1. Overview of larql-server features

    main

    The larql-server is a high-performance server for vindex knowledge queries and inference. It is built on the thesis that transformer FFN layers are compilable knowledge databases.

    Core Capabilities:

    • Browse Endpoints: DESCRIBE, WALK, SELECT, RELATIONS, and STATS (requires no weights).
    • Inference: Full forward pass using WalkFfn (weights are lazy-loaded on the first request).
    • Knowledge Management: Apply Patch overlays to edit model behavior at runtime without retraining, and use Relation labels (probe-confirmed via feature_labels.json).
    • Deployment: Supports multi-model serving from a directory, HuggingFace loading (hf:// paths), API key (Bearer token) authentication, TLS (via rustls), and configurable CORS/concurrency limits.

  2. Overview of larql-boundary

    main

    larql-boundary is a confidence-gated boundary reference codec designed for the LARQL memory model. It manages the 'cold-storage' half of a two-tier memory system by transforming transformer final-layer residuals into compact protocol objects.

    The Two-Tier Memory Model

    • Hot KV Window: Recent tokens using normal attention (fast).
    • BOUNDARY Refs: Older context stored as compressed residual checkpoints (compact).

    Each BOUNDARY reference represents the final-layer residual at the end of a 512-token chunk. The system uses a three-phase pipeline to decide whether to compress these residuals or fall back to higher precision to maintain accuracy.

  3. Overview of larql-router

    main

    The larql-router is a layer-sharding router designed for distributed larql-server deployments. It manages the fan-out of POST /v1/walk-ffn calls across multiple shards. It supports two primary sharding topologies:

    1. Dense (layer-pipeline): Each shard owns a contiguous range of transformer layers (e.g., shard-a owns layers 0-14, shard-b owns 15-29). Requests specify layers via {layer: N} or {layers: [...]}.
    2. MoE (per-expert-shard): Designed for trillion-parameter Mixture-of-Experts models. Each shard owns a contiguous range of experts within a layer range. Requests specify experts via {layer: N, experts: [E1, E2, ...]} or {layer_experts: [{layer, experts}, ...]}.

    The router is a specialized coordinator, not a full transparent reverse proxy. It exposes specific endpoints for fan-out, health, and metrics, while other API endpoints (like /v1/stats or /v1/models) remain accessible directly on the individual shards.

  4. Use primary LARQL CLI commands

    main

    The larql-cli provides several primary verbs for interacting with models, managing caches, and running inference.

    Primary Verbs:

    • run: Execute inference.
    • chat: Start an interactive chat session.
    • pull: Download models/vindices.
    • model: Manage model configurations.
    • link: Link local files to the cache.
    • list: List available resources.
    • show: Display details of a resource.
    • slice: Perform model slicing.
    • publish: Publish resources.
    • rm: Remove resources.
    • bench: Run benchmarks.
    • shannon: Run Shannon correctness checks.
    • serve: Start a local server.

    Build & Diagnostic Verbs:

    • extract, build, compile, convert, verify, hf: Build-related commands.
    • diag, parity: Diagnostic commands for checking model correctness and divergence.

  5. Understand the larql-vindex crate structure

    main

    The larql-vindex crate is organized into several functional modules that handle the lifecycle of a Vindex (Vector Index). Key modules include:

    • config/: Defines configuration types like VindexConfig, VindexLayerInfo, and quantization settings (QuantFormat, Precision).
    • index/: The core in-memory KNN engine. It uses zero-copy mmap and provides capability traits for different data access modes (e.g., NativeFfnAccess for f32/f16, QuantizedFfnAccess for Q4/Q6, and Fp4FfnAccess for FP4/FP8).
    • format/: Handles Vindex file I/O, including loading weights, tokenizers, and managing the VindexManifest.
    • extract/: Contains the build pipeline to transform a raw model into a Vindex, supporting both full builds and streaming extraction.
    • patch/: Implements the patch system, allowing for VindexPatch operations (Insert/Update) and PatchedVindex overlays.
    • engine/: Manages the storage engine and L2 MEMIT cycles.
    • quant/: Provides quantization codec registries and conversion utilities (e.g., f32/f16 to Q4_K).
    • vindexfile/: A declarative system for building Vindices using a vindexfile (supporting FROM, PATCH, INSERT, and DELETE commands).

  6. Understand Multi-Modal Support in LARQL

    main

    LARQL supports multi-modal inputs (vision and audio) primarily through an embed-splice integration pattern. This pattern projects encoder outputs (from models like SigLIP or Whisper) into the Language Model's (LM) residual dimension and inserts them into the input sequence at specific placeholder positions.

    Supported Model Families & Patterns:

    • Gemma 3/4: Uses SigLIP/USM-like encoders with linear projection (Embed-splice).
    • Granite Vision: Uses SigLIP2 with a 2-layer MLP (GELU) connector (Embed-splice per-tile).
    • Qwen2-VL / 2.5-VL: Uses custom ViT with MLP connector (Embed-splice dynamic) and M-RoPE position encoding.
    • Qwen2-Audio: Uses Whisper with linear projection (Embed-splice).
    • Llama 3.2 Vision: Uses ViT-H with cross-attention (Note: This is currently out of scope for v1 and requires a forward-pass refactor).

    Key Capabilities (v1):

    • Accept image and audio file paths via CLI and Python bindings.
    • Load vision/audio encoders as opaque safetensors.
    • Generate text conditioned on multi-modal input (output is text-only).
    • vindex storage is already modality-agnostic and requires no changes.

  7. Understand the larql-kv crate layout

    main

    The larql-kv crate is organized into several functional areas:

    • engines/: Contains the core engine implementations:
      • standard.rs: Production K/V tensor cache (default).
      • no_cache.rs: Full re-forward per step (debug fallback).
      • apollo/: Boundary-residual injection (~4,000× compression).
      • markov_residual/: Residual-stream KV replacement (KL = 0).
      • turbo_quant/: WHT + Lloyd-Max K/V codec (3- or 4-bit).
      • unlimited_context/: Windowed re-prefill from checkpoints.
    • accuracy_suite/: Evaluation tools for parametric, in-context, and conflict-based knowledge testing.
    • accuracy.rs: Helpers for calculating cosine, MSE, KL, JS, and compare_hidden metrics.
    • vindex_compare.rs: Tools for A/B comparison of two vindexes on the same model.
    • profiler.rs: Per-stage decode timing accumulators.
    • generation.rs: Parity oracles for comparing generate_with_engine and generate_cached_* outputs.

  8. Explore Crate-Specific Roadmaps

    main

    The LARQL project is modular, with different crates managing specific domains. You can find detailed roadmaps for each component in their respective directories:

    • larql-compute: Metal GPU kernels, MoE prefill, and platform expansion.
    • larql-inference: Forward pass, generation quality, and KV engines.
    • larql-server: HTTP API, gRPC grid, and remote expert protocol.
    • larql-router: Grid routing, self-balancing, and QUIC transport.
    • larql-cli: CLI UX, sampling flags, and streaming display.
    • larql-lql: LQL grammar, and INSERT/SELECT/USE extensions.
    • larql-core: Graph data model, algorithms, and serialization.
    • larql-vindex: Vindex format, storage, and extraction.
    • larql-models: Architecture definitions and model loading.
    • larql-boundary: Confidence-gated BOUNDARY ref codec and cold-context residual storage.

  9. Key features of larql-vindex

    main

    The larql-vindex format provides several high-performance features for model manipulation:

    • Efficient Extraction: Stream extraction from safetensors, GGUF, or MLX models without a full model load.
    • High-Speed Querying: Supports Gate KNN via BLAS matmul, Q4 matvec (CPU/Metal/CUDA), or HNSW approximate search, as well as LM head KNN for top-K token lookup.
    • Memory Efficiency: Uses zero-copy mmap (gate vectors are sliced directly from disk) and f16 storage to reduce footprint. Vindex inference uses adaptive residency, meaning only necessary layers are resident in RAM.
    • Non-Destructive Editing: Uses a Patch overlay system where all mutations (INSERT/DELETE/UPDATE) are stored in stackable, reversible .vlp files, leaving the base vindex files read-only.
    • Advanced Formats: Supports MoE (Mixtral, DeepSeek) by treating experts as contiguous features, uses binary down_meta (instead of JSONL) for compactness, and includes SHA256 checksums for integrity.
    • Ecosystem Integration: Supports the hf://user/repo URI scheme for downloading/publishing via the HuggingFace Hub and uses a declarative Vindexfile (similar to a Dockerfile) for model builds.

  10. Understand the larql-router-protocol crate layout

    main

    The larql-router-protocol crate is a narrow package containing only the gRPC proto contracts and a QUIC transport wrapper. It does not contain orchestration logic (which is in larql-router) or server-side cache logic (which is in larql-server).

    Crate Structure:

    • proto/: Contains the .proto definitions for GridService, ExpertService, and ShardService.
    • src/lib.rs: Re-exports the generated tonic/prost code.
    • src/transport/quic.rs: Provides QUIC server/client endpoints with fingerprint-pinned TLS.
    • build.rs: Handles protoc invocation during build.

  11. What is a Vindex?

    main

    A vindex (vector index) is a directory containing a neural network's weights reorganized for specific queryable operations. Unlike standard formats (safetensors, GGUF) that store all weights together, a vindex separates weights by their functional purpose. This allows users to load only the specific weights needed for a task, significantly reducing memory and storage requirements.

    Key Principle: The files in a vindex are the canonical storage for the weights. For example, gate_vectors.bin is the actual W_gate matrix. The COMPILE operation reads these files directly to reconstruct standard formats like safetensors without data duplication.

  12. What is the MarkovResidualEngine?

    main

    The MarkovResidualEngine is a specialized decode path for transformer Language Models (LMs) that replaces the traditional per-token K/V cache with the residual stream itself as the persistent inference state.

    Unlike standard engines that grow a K/V cache linearly with context, this engine uses the residual stream as a 'sufficient statistic' to reconstruct the necessary state for the next token. This allows for significant memory savings at long contexts, as the engine bounds memory by storing a 'hot window' of residuals and a 'cold tier' of token IDs, rather than full K/V tensors.

    Key Characteristics:

    • Not a compression scheme: It is a fundamentally different approach to state persistence.
    • Inference-only: It is not designed for training or gradient flow through cold-replayed residuals.
    • Context handling: It bounds memory, but not compute; the cost of 'cold-replay' (recomputing K/V from residuals) grows as the number of tokens beyond the hot window increases.