DOCSEARCHbyAlgolia
Submit Docs

llmfit

repository·main·Indexed 10 days ago

https://github.com/alexsjones/llmfit

A terminal tool and library for right-sizing LLM models to specific hardware (RAM, CPU, GPU). It provides hardware detection, model scoring for fit, speed, quality, and context, and supports local runtimes like Ollama, llama.cpp, and MLX. Features include an interactive TUI, a REST API, MCP server support for AI agents, and community benchmark sharing.

Tokens
55.6K
Snippets
183
Records
265
Agent score
98%

What's inside llmfit

  1. Overview of llmfit

    main

    llmfit is a terminal tool that matches LLM models to your specific hardware (RAM, CPU, and GPU). It automatically detects your hardware and scores models across four dimensions: quality, speed, compatibility, and context.

    Key Features:

    • Hardware Detection: Automatically assesses your system capabilities.
    • Multi-GPU & MoE Support: Handles complex architectures and multi-GPU configurations.
    • Local Provider Support: Compatible with Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio.
    • Interactive TUI: A default terminal user interface for exploring models.
    • Community Benchmarks: Access real-world performance data (tok/s, TTFT, VRAM) from users with similar hardware via the Community Leaderboard (b).
    • Benchmark Sharing: Run llmfit bench --share to contribute real performance data from your machine back to the project via PRs, helping improve accuracy for everyone.

  2. Understand platform and GPU support for llmfit

    main

    llmfit provides varying levels of hardware detection depending on your operating system and GPU vendor.

    Supported Platforms

    • Linux: Full support. Detects NVIDIA, AMD, Intel Arc, and Ascend GPUs.
    • macOS (Apple Silicon): Full support. Uses Metal GPU acceleration and treats unified memory (system RAM) as VRAM.
    • macOS (Intel): Supports RAM/CPU detection; detects discrete NVIDIA GPUs if nvidia-smi is present.
    • Windows: Supports RAM/CPU detection; detects NVIDIA GPUs via nvidia-smi.
    • Android / Termux / PRoot: Supports CPU and RAM detection, but GPU autodetection is not supported. Mobile GPUs (e.g., Adreno) are typically not visible to the probing interfaces used by llmfit.

    GPU Detection and VRAM Reporting

    | Vendor | Detection method | VRAM reporting | | :--- | :--- | : | | NVIDIA | nvidia-smi | Exact dedicated VRAM | | AMD | rocm-smi | Detected (VRAM may be unknown) | | Intel Arc (discrete) | sysfs (mem_info_vram_total) | Exact dedicated VRAM | | Intel Arc (integrated) | lspci | Shared system memory | | Apple Silicon | system_profiler | Unified memory (= system RAM) | | Ascend | npu-smi | Detected (VRAM may be unknown) |

  3. Identify Audio Models via pipeline_tag and Capabilities

    main

    In llmfit, audio models (specifically Whisper variants) are identified by the pipeline_tag: "automatic-speech-recognition" or by including Capability::Audio in their capabilities list. When working with the model data, audio-specific metadata is provided via the following fields:

    • _audio_rtf_gpu: Real-Time Factor on GPU (e.g., 0.007 means 7x realtime).
    • _audio_rtf_cpu: Real-Time Factor on CPU.
    • _audio_vram_gb: VRAM required at F16 precision.
    • _audio_backends: A list of supported servers.

  4. Understand Audio Model Scoring (RTF)

    main

    Unlike text-based LLMs that use tokens per second (tok/s), audio models are evaluated using Real-Time Factor (RTF).

    • RTF Definition: A lower RTF indicates a faster model (e.g., 0.1 RTF means the model processes 10 seconds of audio in 1 second).
    • Scoring Logic: The llmfit scoring for audio models follows the formula: score = accuracy_tier - latency_penalty - vram_penalty.
    • Goal: Lower RTF results in a better (higher) score.

  5. Understand the llmfit project structure

    main

    The llmfit project is organized into several key modules and directories that handle hardware detection, model management, and the user interface:

    Core Logic (src/)

    • main.rs: Entrypoint, CLI argument parsing, and TUI launch.
    • hardware.rs: Detects system RAM, CPU, and GPU (including multi-GPU and backend identification).
    • models.rs: Manages the model database, quantization hierarchy, and dynamic quantization selection.
    • fit.rs: Handles multi-dimensional scoring (Q/S/F/C), speed estimation, and MoE offloading.
    • providers.rs: Integrates runtime providers (Ollama, llama.cpp, MLX, Docker Model Runner, LM Studio), detects installations, and manages pulls/downloads.
    • display.rs: Handles CLI table rendering and JSON output.

    Terminal User Interface (src/)

    • tui_app.rs: Manages TUI application state, filters, and navigation.
    • tui_ui.rs: Handles TUI rendering using ratatui.
    • tui_events.rs: Handles keyboard events using crossterm.

    Data and Scripts

    • llmfit-core/data/hf_models.json: The embedded model database used at compile time.
    • scripts/: Contains utility scripts like scrape_hf_models.py (HuggingFace API scraper), update_models.sh (database updates), and install-openclaw-skill.sh.

  6. Supported Runtime Providers in llmfit

    main

    llmfit detects and interacts with several local runtime providers to manage and run models. Supported providers include:

    • Ollama: Daemon/API based pulls and model detection.
    • llama.cpp: Direct GGUF downloads from Hugging Face and local cache detection.
    • MLX: Apple Silicon support using mlx-community/* repos on HuggingFace.
    • Docker Model Runner: Docker Desktop's built-in model serving.
    • LM Studio: Local model server with REST API for management and downloads.

    If multiple compatible providers are available for a specific model, you can use the provider picker modal in the TUI by pressing d.

  7. How community benchmark submissions work

    main

    Community benchmarks are collected in the community/ directory, organized by hardware slug and timestamped filenames (e.g., community/<hardware-slug>/<unix-timestamp>-<hash>.json).

    Submission Lifecycle

    1. Idempotency: Filenames include a content hash. If you have an open PR, new results are appended to it rather than creating a new one. Retrying a failed submission will skip files that have already been successfully submitted.
    2. Aggregation: Once merged, results are aggregated by llmfit-core/build.rs and embedded directly into the llmfit binary.
    3. Impact: Merged results appear on the benchmark page (attributed to llmfit community) and are used to provide measured ✓ tok/s values and calibrated estimates for other users on identical hardware (same CPU + GPU).

  8. Project Structure Overview

    main

    The project is organized as follows:

    • src/main.rs: CLI parsing, entry point, and TUI startup.
    • src/hardware.rs: System RAM/CPU/GPU detection.
    • src/models.rs: Model database and quantization logic.
    • src/fit.rs: Scoring, speed estimation, and MoE offloading.
    • src/providers.rs: Runtime provider integrations (Ollama, llama.cpp, MLX, etc.).
    • src/display.rs: CLI table rendering and JSON output.
    • src/tui_app.rs & src/tui_ui.rs: TUI application state and rendering.
    • data/hf_models.json: The embedded model database.
    • scripts/: Maintenance scripts (scraping and updates).
    • skills/llmfit-advisor/: OpenClaw skill for hardware-aware recommendations.

  9. How llmfit categorizes model fit levels

    main

    llmfit classifies how well a model fits your hardware into four levels:

    • Perfect: Meets recommended memory requirements on the GPU. Requires GPU acceleration.
    • Good: Fits with headroom. Ideal for MoE offloading or CPU+GPU modes.
    • Marginal: Tight fit, or running on pure CPU (pure CPU runs are capped at this level).
    • Too Tight: Insufficient VRAM and system RAM for the model.

  10. How the llmfit OpenClaw skill works

    main

    The llmfit skill automates the process of hardware-aware model selection and configuration through the following workflow:

    1. Hardware Detection: The agent calls llmfit --json system to detect your current hardware specifications.
    2. Recommendation: The agent calls llmfit recommend --json to retrieve a ranked list of recommended models.
    3. Mapping: The skill maps HuggingFace model names to specific tags used by Ollama, vLLM, or LM Studio.
    4. Configuration: The agent offers to update the models.providers.ollama.models section within your openclaw.json file with the optimal model choices.

  11. How llmfit handles dynamic quantization

    main

    Instead of assuming a fixed quantization level, llmfit performs Dynamic Quantization. It attempts to find the highest quality quantization that fits within your available memory by walking down a hierarchy:

    1. It starts at Q8_0 (highest quality).
    2. It moves down through the hierarchy to Q2_K (most compressed).
    3. It selects the highest quality version that fits in available memory.

    Context Fallback: If a model cannot fit at its full context window, llmfit will attempt to fit it again using half the context window.

  12. Supported Runtime Providers

    main

    llmfit supports several local runtime providers. If multiple compatible providers are available for a specific model, you can open a provider selection modal in the TUI by pressing d.

    - **Ollama** (Daemon/API based pull)
- **llama.cpp** (Direct GGUF download from Hugging Face + local cache detection)
- **MLX** (Apple Silicon / mlx-community model cache + optional server)
- **Docker Model Runner** (Docker Desktop built-in model serving)
- **LM Studio** (Local model server with REST API for management/downloads)