llamafile

The llama.cpp.patches/ directory contains the tools and files necessary to modify llama.cpp for Llamafile compatibility:

• apply-patches.sh: Script to apply all patches to the llama.cpp submodule.
• fetch-ui-assets.sh: Downloads and validates prebuilt web UI assets.
• renames.sh: Script for file renames and moves.
• llamafile-files/: Contains additional files to be copied into llama.cpp, including BUILD.mk (Makefile for building with cosmocc), README.llamafile, and common/license.cpp.
• patches/: Contains the actual patch files for upstream sources.

llamafile is a single-file executable framework that allows you to distribute and run Large Language Models (LLMs) locally on most operating systems and CPU architectures with no installation required.

It works by combining llama.cpp with Cosmopolitan Libc to collapse the complexity of LLM deployment into a single file.

Key features include:

• Single-file distribution: Everything needed to run the model is contained in one file.
• Cross-platform: Runs on most OSs and CPU architectures.
• No installation: Works as a standalone executable.
• Whisperfile: A related single-file speech-to-text tool built on whisper.cpp for transcription and translation.

Whisperfile provides the following capabilities:

• Supported Formats: Transcribes WAV, MP3, FLAC, and Ogg Vorbis.
• GPU Acceleration: Supports Apple Metal, NVIDIA CUDA, and AMD ROCm.
• Translation: Translates speech from any language into English.
• Remote Access: Includes an HTTP server with a REST API for remote transcription.
• Portability: Bundles the binary and model weights into a single portable executable.

llamafile relies on llama.cpp for core functionality. To incorporate bugfixes and support for new models, you must periodically bump the llama.cpp submodule. This process involves updating the submodule, triaging existing patches that may conflict with the new upstream version, reconciling those conflicts via in-place edits, and regenerating the patch set.

Warning: This procedure is designed around a specific set of single-purpose tools. Do not improvise or use standard git apply/git diff commands for patch production or verification, as this can lead to silent failures or stale build objects.

LocalScore is an open-source benchmarking tool that measures LLM performance on specific hardware. It evaluates three key metrics:

1. Prompt Processing Speed: Tokens per second (TPS) for input text.
2. Generation Speed: Tokens per second (TPS) for generated text.
3. Time to First Token (TTFT): Latency in milliseconds before the first response appears.

These metrics are combined into a single LocalScore value using a geometric mean:

$$\text{score} = 10 \cdot \sqrt[3]{\text{avg_prompt_tps} \cdot \text{avg_gen_tps} \cdot \frac{1000}{\text{avg_ttft_ms}}}$$

Score Guidelines:

• 1,000+: Excellent
• 250: Acceptable to good
• 100: Relatively poor

llamafile is a system for creating single-file, cross-platform executables (Actually Portable Executables or APE) that run LLMs, speech-to-text, and image generation models locally without installation.

It achieves this by bundling multiple inference engines into a single binary using Cosmopolitan Libc:

• llamafile/ (Core Library): Handles the HTTP server, TUI (Terminal User Interface), syntax highlighting, and GPU integration (Metal, CUDA, ROCm).
• llama.cpp/: The LLM inference engine providing GGUF support and tensor operations via ggml.
• whisper.cpp/: Provides speech-to-text capabilities.
• stable-diffusion.cpp/: Provides image generation capabilities.

Key features include runtime CPU dispatch (automatically selecting SSE, AVX, or NEON instructions) and dynamic GPU loading, which falls back to CPU if a compatible GPU is not found.

When building the CUDA backend, you can choose between a smaller binary or full feature support:

1. Minimized Build (Default for releases): Use the --minimize-size flag. This defines GGML_CUDA_NO_IQ_QUANTS, which compiles out most IQ-quant CUDA paths to reduce library size.

   • Consequence: The library will report IQ-quant operations as unsupported via ggml_backend_cuda_device_supports_op, and these operations will fall back to the CPU.

2. Full Feature Build: Omit the --minimize-size flag. This keeps the full IQ mul_mat set enabled for the GPU.

Note: ROCm, Vulkan, and Metal builds do not use this size-minimization strategy and keep IQ support enabled by default.

To avoid requiring users to pass flags like -m /zip/... every time, you can create a .args file. This file contains the default arguments for the executable, with each argument on its own line.

Important: Include the special token ... at the end of the file. This token is replaced by any additional arguments the user provides at runtime.

Example .args content:

-m
/zip/ggml-tiny.en-q5_1.bin
...

After creating the file, embed it into your executable using zipalign:

o//third_party/zipalign/zipalign whisper-tiny .args

GPU backends (CUDA, ROCm, Vulkan) are dynamically-loaded libraries that export the ggml C ABI. They are managed by a shared probe core in llamafile/gpu_backend.c using a GpuBackendDesc and a link thunk.

The loading lifecycle follows these steps:

1. Load the DSO.
2. Log-suppress output.
3. Device-count gate: The core rejects DSOs that report 0 devices to ensure AUTO mode falls back correctly.
4. Register the backend.

A SIGSEGV/SIGABRT crash guard is implemented around the foreign probe call because driver initialization can fault across the Cosmopolitan/ms_abi boundary.

Note on Metal: Metal is kept separate by design; it is runtime-compiled and does not use the device gate or the ms_abi split.