DOCSEARCHbyAlgolia
Submit Docs

mtplx

repository·main·Indexed 19 days ago

https://github.com/youssofal/mtplx

A high-performance runtime for running local LLMs on Apple Silicon using Multi-Token Prediction (MTP), specifically designed for Qwen3-Next. It features a native macOS foundation (MTPLXApp), an OpenAI-compatible API, and tools for auto-tuning MTP draft depth and converting Hugging Face repositories into MTP models via the forge command. Version 2.4.2 supports various decoding modes including Turbo, Sustained, and Sustained Max to optimize performance based on hardware thermals and context length.

Tokens
36.9K
Snippets
136
Records
199
Agent score
77%

What's inside mtplx

  1. Overview of Metal Kernel Sources in vLLM-Metal

    main
    The vllm_metal project utilizes specialized Metal paged-attention shaders vendored from mistral.rs to accelerate LLM inference on Apple Silicon. These kernels are compiled and dispatched by paged_ops.cpp using MLX's native Metal command encoder. The current active kernel set is located in kernels_v1/ and provides optimized operations for paged attention, KV cache management, and quantization support.

  2. Understand the MTPLX benchmark asset structure

    main

    MTPLX benchmark assets are organized into three functional directories to facilitate testing different aspects of model performance:

    • prompts/: Contains prompt suites for specific testing domains, including code, warm-code, JSON/tool, prose, reasoning, and long-context tests.
    • validators/: Contains logic for verifying model outputs, including JSON, tool calls, code, cache equivalence, and stochastic distribution checks.
    • runners/: Contains the executable benchmark harnesses used to execute the tests.

    Note: There is no tracked reports/ directory. Generated summaries and large raw outputs should be placed in ignored output folders.

  3. Understand the MTPLXApp Native Backend Foundation

    main

    MTPLXApp serves as the native macOS foundation for MTPLX V1. It is designed with a clear separation of concerns: Swift acts as the supervisor, client, and state manager, while the heavy lifting of inference remains within the existing mtplx daemon.

    Key architectural components include:

    • MTPLXAppCore: Handles daemon supervision, typed HTTP/SSE clients, settings, logs, and observable state.
    • MTPLXApp: A minimal SwiftUI host target used primarily for debug launching.
    • MTPLXBackendStore: The intended binding surface for future UI development, replacing raw JSON parsing.

  5. Dashboard Features and Monitoring Capabilities

    main

    The dashboard provides real-time visibility into several aspects of the MTPLX inference engine:

    • Overview: Live decode TPS (Tokens Per Second) gauge, 5-minute TPS time-series, tokens-served lifetime, in-flight request count, and context-window utilization.
    • Speculative Decoding: Per-depth acceptance bars, verify-cycle waterfall (decomposing verify_time_s into components like forward, logits, hidden, etc.), and a comparison panel against a vLLM oracle (when Qwen3.6-27B is loaded).
    • Cache: A 24-slot SessionBank grid showing session IDs, prefix lengths, hits, bytes, and age. Includes an eviction button and CacheMissReason tooltips (e.g., POLICY_MISMATCH, TEMPLATE_MISMATCH).
    • Memory & Thermal: Hardware info (chip, machine model, unified memory) and MLX memory usage (active, cache, peak, headroom). Thermal monitoring (fan rings) is available if --enable-thermal-poll is enabled.
    • Requests: A live list of in-flight requests with a Cancel button to trigger a best-effort cancellation via the worker's cancel_event.

  6. Requirements for running MTPLX

    main

    To use MTPLX, ensure your system meets the following criteria:

    • Hardware: Apple Silicon Mac.
    • OS: macOS with MLX support.
    • Software: Python 3.11+.
    • Storage: Sufficient disk space for your selected models.

    Note: While commands like mtplx help, mtplx doctor, mtplx inspect, and mtplx init work without MLX installed, actual model generation and serving require MLX and a verified model.

  7. Understand the Control Chess Reanimation mechanic

    main

    Control Chess is a browser-based chess variant featuring a Reanimation mechanic.

    Core Rules:

    • Capture to Pool: When you capture a non-reanimated enemy piece, it switches to your color and enters your 'pool' with a cooldown.
    • Cooldown: A piece captured during round R becomes available to drop in round R+3. The UI tracks this with a countdown (3 → 2 → 1 → ready).
    • Dropping Pieces: Instead of a standard move, you may spend your entire turn dropping one ready piece from your pool onto an empty square that is attacked by at least one of your on-board pieces.
    • One-Shot Limit: Every dropped piece is flagged as reanimated. If a reanimated piece is captured, it is sent to the graveyard and cannot be reanimated again.
    • Game End: Checkmate and stalemate detection must include legal drops. A ready drop that prevents check or stalemate is considered a legal move.
    // Logic for drop readiness
droppable = currentFullmove >= capturedOnRound + 3;

  8. Select an MTPLX execution profile

    main

    MTPLX uses profiles to determine how models are executed, specifically managing the balance between speed, quality, and thermal management. You can select a profile to match your hardware or use case (e.g., high-speed decoding vs. long-context stability).

    Available Profiles

    ProfilePurpose
    turboFastest decode profile. Optimized for quantized 27B and 9B flagships. Includes Sustained plus NAX verify kernels and context-routed compiled verify. Matches macOS app launch presets.
    sustainedDefault mode for most models. Uses the native-MTP long-context path with chunked prefill, final-token logits, and request-sized paged KV. Uses standard Apple fan control.
    performance-coldLegacy burst path. Provides high performance without fan boost. Not recommended for context lengths beyond 8K.
    stableConservative fallback. A hidden alias for the exact/staged long-reply path and compatibility fallback.
    exactQA/Release mode. Used for exactness checks during QA and releases.
    max-diagnosticDiagnostic mode. Used for fan-control diagnostics only.

    Product Modes (Combinations)

    You can combine profiles with the --max flag to create specific product modes:

    • Sustained: Use the sustained profile.
    • Sustained Max: Use sustained + --max. Uses the long-context path with ThermalForge/TG Pro fans pinned while MTPLX runs.
    • Burst: Use performance-cold + --max. Uses the legacy max-fan path (not recommended for >8K context).

  9. Understand the MTPLX architecture and component relationships

    main

    MTPLX is structured to decouple the user interface from the underlying model mechanics. The architecture consists of the following layers:

    1. CLI Surface: The primary entry point for users.
    2. Profiles: Manages configuration and selection of execution profiles.
    3. Speculative Sampling: A backend-agnostic layer that handles the logic of speculative execution.
    4. Compatibility Registry: Maps models and configurations to the appropriate backend.
    5. Backends: Architecture-specific implementations (e.g., Qwen3NextMTPBackend) that provide the actual proposal and verification mechanics for specific model families.
    6. SessionBank: Manages state/sessions for the server.
    7. OpenAI Server: Provides a standardized API interface for interacting with the system.

    Key design principle: The speculative sampler is designed to be backend-agnostic, meaning it does not need to know the specifics of the model family; it relies on the backend to provide the necessary proposal and verification mechanics.

  10. Understand the `forge publish` progress reporting mechanism

    main

    Progress for the publish subcommand is reported via a single JSON file in the <output-dir>/<run-id>/ directory.

    publish.json Schema

    • bytes_uploaded: Total bytes sent.
    • total_bytes: Total size of the artifact.
    • mb_per_s: Upload speed.
    • repo_created: The Hugging Face repository name (e.g., user/repo) if creation succeeded.
    • revision: The specific git revision.
    • finished: Boolean status.
    • repo: The final repository path.
    {
  "bytes_uploaded": 12345678,
  "total_bytes": 17000000000,
  "mb_per_s": 18.6,
  "repo_created": "youssofal/Qwen3.6-35B-A3B-MTPLX-Speed",
  "revision": "abc1234",
  "finished": false,
  "repo": "youssofal/Qwen3.6-35B-A3B-MTPLX-Speed"
}

  11. Numerical behavior and bit-exactness of Turbo verify kernels

    main

    Turbo verify kernels are not bit-exact compared to stock MLX kernels due to different accumulation orders. However, they are functionally equivalent for most use cases:

    • Argmax: Identical on all probed positions.
    • Sampling: At standard product sampler settings (temp 0.6 / top_p 0.95 / top_k 20), sample agreement is 1.0.
    • Speculative Acceptance: Remains mathematically exact with respect to the verify-computed target distribution.

    Warning: Do not use these kernels for bit-exactness QA. Use mtplx qa exactness reference runs for batch-equivalence testing.

  12. Requirements for AR-only model tier

    main

    The AR-only tier is highly restrictive. It is currently limited to models that match the exact mixed-precision geometry and storage map of mlx-community/Laguna-S-2.1-oQ4e at revision 8e3f5cad513746264940c1c4195de48d7ea345a5.

    To be admitted to the local cache, the following components must be present and verified:

    • The pinned source marker
    • All 13 shards at their reviewed sizes
    • The index
    • The tokenizer
    • The generation config
    • The special tokens map
    • The Poolside chat template

    Note: Other Laguna variants (including the earlier uniform-4bit build) are currently blocked until they receive specific construction-time validation and runtime evidence.