MLC LLM

MLC LLM is a machine learning compiler and high-performance deployment engine designed for large language models (LLMs). It enables the development, optimization, and native deployment of AI models across various platforms.

At its core, MLC LLM utilizes MLCEngine, a unified high-performance inference engine. MLCEngine provides an OpenAI-compatible API that can be accessed via:

• REST server
• Python
• JavaScript
• iOS
• Android

MLC LLM is a machine learning compiler and high-performance deployment engine designed for large language models (LLMs). It uses a unified engine called MLCEngine to provide high-performance inference across a wide variety of hardware platforms.

Key features include:

• Universal Deployment: Supports diverse hardware including NVIDIA GPUs (CUDA), AMD GPUs (ROCm/Vulkan), Apple Silicon (Metal), Intel GPUs (Vulkan/Metal), Web Browsers (WebGPU/WASM), iOS/iPadOS (Metal), and Android (OpenCL).
• Unified API: Provides an OpenAI-compatible API accessible via REST server, Python, JavaScript, iOS, and Android, all backed by the same underlying engine and compiler.

MLC LLM provides a universal deployment engine that can be targeted toward various platforms. Depending on your needs, you can explore the following deployment paths:

• Web/Browser: Deployment via WebGPU.
• Mobile: Native deployment for iOS and Android.
• Server/API: Running a REST server or using the Python API.
• CLI: Using the Chat CLI for local interaction.

To run your own models, you will need to follow the workflows for converting model weights to MLC format and compiling model libraries for your specific target platform (e.g., .wasm for web, .dll for Windows, or .tar for mobile).

@mlc-ai/web-llm is a high-performance in-browser LLM inference engine designed for web applications and agents. It uses WebGPU for local acceleration, provides an OpenAI-compatible API, and supports web workers to prevent heavy computation from blocking the UI thread.

Requirements:

• A WebGPU-compatible browser (e.g., latest Google Chrome).
• You can verify WebGPU support at webgpureport.org.

Installation: Available via npm or CDN.

Usage: For a quick start without setup, you can use a JSFiddle example. For advanced features like JSON mode and streaming, refer to the official examples repository.

MLC-LLM uses short codes to specify quantization modes, supporting both weight-only and weight-activation quantization.

Weight-only Quantization

Format: qAfB(_id), where A is the weight bit-width, B is the activation bit-width, and _id is an algorithm identifier (e.g., symmetric, AWQ).

Available options:

• q0f16 (Weight-only, 0-bit weights, FP16 activations)
• q0f32 (Weight-only, 0-bit weights, FP32 activations)
• q3f16_1 (3-bit weights, FP16 activations)
• q4f16_1 (4-bit weights, FP16 activations)
• q4f32_1 (4-bit weights, FP32 activations)
• q4f16_awq (4-bit AWQ, currently unstable)

Weight-activation Quantization (FP8 on CUDA)

Both weights and activations are quantized to FP8. Layer outputs are in higher precision (FP16) before being requantized to FP8.

Available options:

• e4m3_e4m3_f16
• e5m2_e5m2_f16

When deciding how to implement a model in MLC LLM, distinguish between a new architecture and a model variant:

1. Model Variant: If the architecture is already supported by MLC LLM (e.g., adding CodeLlama when llama-2 is already supported, or adding OpenHermes Mistral when mistral is supported), you only need to convert the weights.
2. New Architecture: If the model uses a new architecture or requires new inference logic, you must follow the full implementation process using tvm.nn.module as described in the architecture guide.

Both MLCEngine and AsyncMLCEngine accept a mode argument in their constructors to pre-define configurations for different use cases. The choice of mode affects request concurrency and KV cache capacity, which in turn impacts GPU memory usage.

• "local" (default): Low request concurrency and low KV cache capacity. Best for saving GPU memory when concurrent requests are minimal.
• "interactive": Request concurrency is set to 1 with low KV cache capacity. Designed for chat and conversational interfaces.
• "server": High request concurrency and high KV cache capacity. Designed to fully utilize GPU memory for large-scale server scenarios and system benchmarks.

If a model is not compatible with prebuilt MLC model libraries (due to different architecture, quantization, metadata, or platform requirements), you must build your own model library. A model library is defined by four components:

• Model Architecture: e.g., llama-3, gpt-neox, phi-3.
• Quantization: e.g., q4f16_1, q0f32.
• Metadata: Affects memory planning (e.g., context_window_size, sliding_window_size, prefill-chunk-size). Note that prefill-chunk-size currently affects the compiled model.
• Platform: e.g., cuda, webgpu, iOS.

To compile for webgpu, you must build mlc_llm from source and follow the install-web-build instructions to avoid the RuntimeError: Cannot find libraries: wasm_runtime.bc error.

Running mlc_llm package generates a ./dist/ directory containing the necessary components for the iOS app:

dist
├── bundle                   # Contains app configuration and optionally model weights
│   ├── mlc-app-config.json  # The app configuration JSON file
│   └── [optional model weights]
└── lib
    ├── libmlc_llm.a          # Interface for LLM, tokenizer, and TVM runtime
    ├── libmodel_iphone.a     # The compiled model library
    ├── libsentencepiece.a    # SentencePiece tokenizer
    ├── libtokenizers_cpp.a   # Huggingface tokenizer
    └── libtvm_runtime.a     # TVM runtime

A model library is defined by four primary components:

1. Model Architecture: (e.g., llama-2, gpt-neox)
2. Quantization: (e.g., q4f16_1, q0f32)
3. Metadata: Parameters like context_window_size, sliding_window_size, and prefill-chunk-size which influence memory planning.
4. Platform: The target hardware/runtime (e.g., cuda, webgpu, iOS, android, metal, vulkan).

These parameters are encapsulated in the mlc-chat-config.json file produced by the gen_config step. The compile step then reads this JSON to generate the optimized binary (e.g., .so, .dylib, .dll, .wasm, or .tar for mobile) for the target device.