DOCSEARCHbyAlgolia
Submit Docs

Ziraph Documentation

website·Indexed 31 days ago

https://ziraph.com/docs/

A performance profiling tool for Apple Silicon on-device AI workloads. Ziraph measures ANE, GPU, and CPU telemetry to provide metrics such as tokens per second and energy per token. It includes features for A/B testing via multi-variant campaigns, hardware capability calibration, and trace recording in NDJSON format to help developers prove speedups and identify bottlenecks.

Tokens
48.4K
Snippets
171
Records
370
Agent score
55%

What's inside Ziraph

  1. Overview of Multi-variant Campaigns in Ziraph

    A campaign in Ziraph is a declarative orchestration layer used to run N-variant by M-metric experiments. Instead of manually running traces, aggregating results, and comparing them for each variant, a campaign uses a TOML configuration file to define a grid of variants (e.g., different models, quantizations, or runtimes) and the metrics to be measured.

    Key features include:

    • Statistical Stability: Each variant is run K times to calculate the mean and sigma, determining if performance differences are statistically significant or merely run-to-run variance.
    • Reproducibility: The TOML config serves as the experiment definition, allowing other developers to reproduce the exact same grid on different hardware.
    • Composition: Campaigns orchestrate existing Ziraph primitives (ziraph trace, ziraph aggregate, and ziraph compare) without replacing their underlying measurement or reduction logic.

  2. Overview of Ziraph

    Ziraph is a tracer-profiler for Apple Silicon designed to provide a broad view of chip activity and the ability to zoom into specific performance details. It was evolved from a narrower tool called 'anesight' after the developer realized that monitoring only the Apple Neural Engine (ANE) was insufficient, as modern local AI workloads (such as Ollama, llama.cpp, and MLX) primarily utilize the GPU rather than the ANE.

  3. Overview of Ziraph performance measurement capabilities

    Ziraph is a system monitoring tool for Apple Silicon that captures high-resolution traces to answer specific performance and efficiency questions. It surfaces signals that standard monitors typically miss, including per-process ANE (Apple Neural Engine) usage, chip throttle and clamp states, DVFM frequency-residency for accelerators, and energy split by pipeline stage.

    Ziraph specifically enables developers to determine:

    1. Which accelerator executed a workload (ANE, GPU, CPU, or Media Engine).
    2. Per-PID process attribution without requiring sudo.
    3. System bottlenecks via the Bottleneck Discovery Engine.
    4. Statistical significance of changes using $\sigma$-aware A/B and N×M campaigns.
    5. Cost metrics including power, bandwidth, and energy-per-token.

  4. Overview of Ziraph AI workload profiling

    Ziraph is a profiling tool that wraps local AI workloads (such as LLMs, Whisper transcriptions, CoreML vision models, or custom programs) to measure chip-level performance and energy. It scopes the measurement to a specific command (per-PID, no sudo required), runs a Bottleneck Discovery Engine over the trace, and provides $\sigma$-aware A/B verdicts across multiple runs to separate signal from noise.

  5. Understand ziraph Bottleneck Discovery Engine hints

    After every ziraph trace, the Bottleneck Discovery Engine classifies the trace to identify which accelerator performed the work, any measurement biases, and the location of the bottleneck. Hints are actionable messages with a severity level (high, medium, low, or advisory) and a recommended next step. They appear in the post-run TUI replay, ziraph compare output, and the saved trace JSON.

    Note: Hints marked as 'advisory' may include a 'heuristic - not yet field-calibrated' note, indicating the threshold is based on heuristics rather than field-calibrated constants.

  6. Understand ziraph trace token-count detection logic

    ziraph trace uses a stateful line scanner to monitor the combined stdout and stderr of child processes. It identifies the LLM runner based on a priority list (Ollama > llama.cpp > MLX > LM Studio > MLC-LLM > ds4-bench > llama-bench). Once a runner is matched, the parser is 'pinned' to that runner for the remainder of the session to prevent accidental cross-pattern matches. The parser tracks the most recent token count observed; it does not calculate a cumulative sum because most runners report the total for the current message.

  7. Understand the impact of memory bandwidth on LLM decode speed

    Autoregressive decoding is memory-bandwidth-bound rather than compute-bound. Each token generation requires streaming the model weights through the GPU once. Consequently, tokens per second is governed by DRAM read speed rather than FLOPs. Runtimes like MLX attempt to optimize this by treating unified memory as a primitive and using lazy evaluation to fuse operations, but on hardware with low bandwidth (like the base M1), both optimized and standard runtimes will eventually hit the same hardware ceiling.

  8. Understand how Ziraph measures Apple Silicon workloads

    Ziraph measures hardware counters by reading from macOS's private IOReport framework (the same source used by powermetrics). It employs a 'subscribe-sample-delta' lifecycle and a 'wrap-and-subtract' model to isolate specific workload measurements from ambient system noise.

  9. Ziraph internal architecture overview

    Ziraph is distributed as a single Zig binary with no daemon, kernel extension, or installer. It is structured into three layers:

    1. Platform Layer: Handles IOKit and IOReport bindings and converts raw hardware counters into millijoules or milliwatts.
    2. Runtime Layer: Implements the shared 'subscribe-sample-delta' loop used across all measurement modes.
    3. Resolver: Maps chip-specific available channels to the stable metrics surfaced by the tool.

  10. Identify Media Engine activity in multimodal pipelines

    In multimodal or video pipelines, the Media Engine handles hardware video decode/encode. This work is separate from the neural model execution. Ziraph reports Media Engine power as a distinct line item, preventing it from being conflated with general GPU compute power.

  11. Inspect model metadata with ziraph inspect

    The ziraph inspect command reads the header or manifest of a model artifact to extract metadata without loading full weights. It is used to verify quantization, architecture, and on-disk size before running a workload. It supports .mlmodelc (CoreML compiled bundles), .gguf (llama.cpp), and .safetensors (HuggingFace) formats. For extensionless files (like Ollama blobs), Ziraph identifies the format by sniffing magic bytes.
    # Inspect a GGUF model
ziraph inspect ~/models/llama-3.gguf

# Inspect a compiled CoreML bundle
ziraph inspect ~/models/mistral.mlmodelc

# Inspect an extensionless Ollama blob
ziraph inspect ~/.ollama/models/blobs/sha256-...

  12. Measure Apple Silicon power and bandwidth using IOReport

    Ziraph utilizes macOS's private IOReport framework to read hardware-counter snapshots from the kernel. It tracks per-component energy and memory-controller bandwidth for the ANE, GPU, CPU, and Media Engine. Because channel names and layouts vary by chip generation (M1, M2, M4, M5), Ziraph uses an internal resolver to map the correct channels at runtime.

    Data is collected in three phases:

    1. Subscribe: Ziraph subscribes to the telemetry stream at startup.
    2. Sample: It reads cumulative-since-boot counters at a specific rate (default 1 Hz, configurable via --rate).
    3. Delta: It uses the framework's internal delta function to calculate per-interval changes. Power is calculated as: power_mw = energy_delta_mj / interval_s.