DOCSEARCHbyAlgolia
Submit Docs

hipfire

repository·master·Indexed 19 days ago

https://github.com/kaden-schutt/hipfire

A high-performance LLM inference engine optimized for AMD RDNA GPUs (RDNA1 through RDNA4). Written as a single Rust binary, it avoids the Python/PyTorch stack in the hot path to provide an Ollama-like experience for consumer and professional AMD hardware. The repository includes tools for generating calibration corpora, NIAH (Needle-in-a-Haystack) fixtures, and a quantization quality evaluation harness for comparing model variants against BF16 references.

Tokens
485.5K
Snippets
1.1K
Records
2.2K
Agent score
67%

What's inside hipfire

  1. Overview of hipfire-arch-qwen2

    master

    The hipfire-arch-qwen2 crate provides a plain Qwen2 dense text decoder implementation for the hipfire ecosystem. It is identified by arch_id = 7.

    Key Technical Specifications

    • Attention: GQA (Grouped Query Attention) support (e.g., 12 query heads and 2 KV heads for the 1.5B model).
    • Normalization: RMSNorm with eps=1e-6.
    • Activation: SwiGLU FFN.
    • Positional Embeddings: 1-D RoPE with theta = 1_000_000.
    • Bias: Uses attention_bias = true on Q/K/V projections, following the Qwen2 modeling default.
    • Embeddings: Supports variable tie_word_embeddings (set to true for 1.5B-Instruct and false for dots.ocr).

  2. Overview of hsa-bridge

    master

    The hsa-bridge crate provides a thin Rust wrapper around libhsa-runtime64.so, serving as a low-level alternative to hip-bridge. It is used to interface directly with the HSA (Heterogeneous System Architecture) runtime.

    Key capabilities include:

    • Loading the library via dlopen.
    • Exposing agents, queues, signals, and memory pools.
    • Managing executables.
    • Providing an AQL (Architectural Queuing Language) packet builder.

    The total surface area is approximately 30 FFI entry points.

  3. Overview of hipfire-arch-dots-ocr

    master

    The hipfire-arch-dots-ocr crate provides an Architecture trait implementation for dots.ocr, a layout-analysis Vision Language Model (VLM) from the Qwen2-VL family (arch_id = 8).

    It functions by pairing a plain Qwen2 text decoder (delegated to hipfire-arch-qwen2) with a custom 42-block DotsVisionTransformer. The vision component utilizes RMSNorm, SwiGLU, 2-D RoPE, and non-causal attention, alongside a LayerNorm-based PatchMerger. The model is designed to produce JSON, Markdown, or SVG layout outputs via a custom chat template.

  4. MTP Research Artifacts Status

    master

    The MTP implementation is currently classified as research-artifact infrastructure.

    • Status: The code is built and runs when invoked manually via mtp_extract, mtp_head, mtp_spec, and mtp_compose.
    • Production Status: NOT enabled in production. The daemon path remains unchanged.
    • Integration: No PR has been merged to master. All work is contained in the worktree-mtp-qualcomm-probe branch for archival purposes.

  5. Review hipfire prefill optimization levers

    master

    This document provides a detailed inventory of performance optimization 'levers' used to improve prefill performance in the hipfire engine, specifically for the Qwen3.6-35B-A3B-PARO architecture on gfx1151 hardware. It compares hipfire implementations against hipEngine to identify parity, advantages, and remaining opportunities for optimization (Quick Wins).

    Optimization Categories

    • Embedding & Input Layer: Optimizations for embedding lookups, RMSNorm, and AWQ pre-rotation fusion.
    • LinearAttention: Fusions for QKV projection, conv1d/SiLU/split, and gating.
    • Gated DeltaNet (GDN): High-leverage optimizations like chunkwise recurrence (T2).
    • FullAttention: Optimizations for QKV projection, RoPE (Rotary Positional Embeddings), KV cache writes, and Flash Attention (T5).
    • MoE (Mixture of Experts): Optimizations for router GEMV, top-k selection, shared-expert projections, and routed-expert dispatch/projection.
    • LM Head: Optimizations for the language model head and sampling/argmax.
    • Cross-cutting: Methodology and build-level optimizations like hipGraph capture, unrolling thresholds, and auto-tuned chunk sizes.

  6. Understand the Hipfire engine bug sweep (Hunt-3)

    master

    The Hunt-3 engine bug sweep (2026-06-03) was an adversarial multi-agent bug-hunt using 109 agents to identify issues across four specific surfaces:

    1. Multi-GPU / Pipeline-Parallel (generate_multi, pp>1)
    2. Sampling + MoE dispatch
    3. Daemon concurrency / streaming (Rust JSON-lines)
    4. Bun CLI → backend entry (TS + IPC boundary)

    This sweep identified several high-priority bugs related to sibling-path divergence (where generate_multi and generate_vl lack fixes applied to the single-GPU generate() path), Bun single-pipe concurrency issues, and primitive coercion errors at boundaries.

  7. Review the hipfire codebase quality assessment

    master
    The docs/reviews/2026-06-01-nils-qa-assessment.md file contains a comprehensive quality assessment report for the hipfire codebase. It evaluates security, test coverage, architecture, DevOps, code style, documentation, and dependency health. This report is useful for developers assessing the stability, technical debt, and current development priorities of the project.

  8. Understand the hipfire crate structure

    master

    The project is organized into several specialized crates. Understanding these helps you locate where to implement new architectures, kernels, or quantization logic:

    • hipfire-runtime: The core inference orchestrator, managing KV cache, samplers, and loaders.
    • hipfire-arch-*: Per-family forward pass implementations (e.g., hipfire-arch-qwen35, hipfire-arch-llama).
    • rdna-compute: Handles kernel dispatch, hipGraph capture, and the JIT loader.
    • hip-bridge: Provides safe Rust FFI over libamdhip64.so.
    • hipfire-quantize: A standalone CPU-side encoder for converting safetensors/GGUF to .mq4 or .hf4 formats.
    • cli: The entrypoint for running models via the command line.

  9. What is DeltaNet (Qwen 3.5 hybrid)?

    master

    Qwen 3.5+ models use a hybrid architecture that alternates between FullAttention and DeltaNet linear-attention layers.

    DeltaNet replaces standard softmax attention with a recurrent gated linear update. This provides $O(1)$ per-token compute and uses a fixed-size state instead of a traditional KV cache for those specific layers. A 1D causal convolution is applied across the time axis before the linear-attention update for local mixing.

  10. Implementation requirements for multi-weight Q8 WMMA dispatch

    master

    When implementing or modifying dispatch logic for multi-weight Q8 WMMA kernels, follow these safety requirements to prevent silent data corruption:

    1. DType Consistency: You must assert that all weights in a multi-weight dispatch have consistent per-projection dtypes. Use debug_assert! at every dispatch site.
    2. Tail Handling: Ensure that the number of K elements is a multiple of 32 (K % 32 == 0). Use debug_assert_eq! in all 4 dispatch helpers to prevent kernels from silently dropping the tail.
    3. Residual Invariant: The residual += operation relies on the invariant that grid tiles do not overlap. If implementing a variant that splits K (e.g., _ksplit), ensure this invariant is maintained or use atomicAdd to prevent races.

  11. Understand the KLD vs PPL trade-off in quantization

    master

    In Hipfire quantization, Perplexity (PPL) and Kullback–Leibler divergence (KLD) can provide conflicting signals about model quality. This is a known phenomenon where a more precise quantization (lower weight noise) does not always result in better PPL.

    • PPL (Perplexity): Measures only the probability assigned to the true next token. A quantization can achieve better PPL if its argmax token happens to align with the reference model's argmax, even if the rest of the probability distribution is poorly aligned.
    • KLD (Kullback–Leibler divergence): Measures the match of the entire probability distribution against the reference (e.g., HF-bf16). Lower KLD indicates higher distributional fidelity.

    Example Case: The MQ6-q8conv1d variant has a KLD 6.1× lower than MQ4-Lloyd, indicating much better distribution matching, yet its PPL is 1.1% higher. This occurs because MQ4-Lloyd's codebook may nudge the argmax toward the reference token at the expense of the tail distribution, whereas MQ6-q8conv1d preserves the tail mass more accurately.

  12. Analyze kernel performance using PMC on gfx1151

    master

    On gfx1151 hardware, Performance Monitoring Counters (PMC) are available to cross-check analytical bandwidth models. This allows you to observe hardware-level metrics like FETCH_SIZE, GL2C_HIT, GL2C_MISS, and SQ_WAVES during kernel execution.

    Key Metrics for Attention Kernels:

    • FETCH_SIZE: Total bytes fetched.
    • GL2C_HIT / GL2C_MISS: L2 cache hit and miss rates.
    • SQ_WAVES: Number of wavefronts/waves dispatched.
    • VALU: Vector ALU utilization.

    Note for gfx1100 users: PMC counters (specifically FETCH/GL2C/SQ) may return zero on gfx1100 (7900 XTX). For those platforms, use the ratio of kernel-trace wall time to KV byte volume to estimate bandwidth constraints.