DOCSEARCHbyAlgolia
Submit Docs

WebLLM

repository·main·Indexed 12 days ago

https://github.com/mlc-ai/web-llm

A high-performance in-browser LLM engine that leverages WebGPU to run large language models directly in the browser. Version 0.2.84 supports hardware-accelerated chats, multiple persistent cache backends (Cache API, IndexedDB, OPFS, and Cross-Origin Storage), and OpenAI-compatible function calling.

Tokens
26.9K
Snippets
100
Records
132
Agent score
97%

What's inside WebLLM

  1. Overview of WebLLM

    main

    WebLLM is a high-performance, in-browser language model inference engine designed to run Large Language Models (LLMs) directly in web browsers using hardware acceleration via WebGPU.

    Key capabilities include:

    • In-Browser Inference: Runs models locally, removing the need for server-side processing and enhancing privacy.
    • WebGPU Acceleration: Leverages local hardware for optimal performance.
    • OpenAI API Compatibility: Provides an API that is compatible with standard OpenAI workflows, allowing for easier integration.
    • Web Worker Support: Built-in support for running inference in web workers to prevent heavy computations from blocking the main UI thread.
    • Model Support: Compatible with various model families including Llama, Phi, Gemma, and Mistral.

  2. Key Features of WebLLM

    main

    WebLLM provides several core capabilities for developers building AI-powered web applications:

    • In-Browser Inference: Uses WebGPU for hardware-accelerated LLM operations without a backend server.
    • Full OpenAI API Compatibility: Supports standard OpenAI API functionalities including JSON-mode, function-calling, and streaming.
    • Extensive Model Support: Natively supports models such as Llama, Phi, Gemma, RedPajama, Mistral, and Qwen.
    • Custom Model Integration: Allows deployment of custom models provided in the MLC format.
    • Plug-and-Play Integration: Can be installed via NPM, Yarn, or directly through a CDN.
    • Streaming & Real-Time Interactions: Supports streaming chat completions for interactive UI experiences.
    • Web Worker & Service Worker Support: Enables offloading computations to separate threads to maintain UI responsiveness.
    • Chrome Extension Support: Can be used to build custom Chrome extensions.

  3. Override model defaults with ChatConfig

    main

    Every model has a baseline configuration loaded from mlc-chat-config.json. You can override these defaults when loading a model using MLCEngine.reload() by passing a ChatOptions object (which is a Partial<ChatConfig>).

    Configurable fields include:

    • tokenizer_files & tokenizer_info: Tokenizer initialization.
    • conv_template & conv_config: Conversation formatting and role templates.
    • context_window_size, sliding_window_size, & attention_sink_size: KV-cache and memory settings.
    • Default generation knobs: repetition_penalty, frequency_penalty, presence_penalty, top_p, and temperature.
    await engine.reload("Llama-3.1-8B-Instruct", {
    temperature: 0.7,
    repetition_penalty: 1.1,
    context_window_size: 4096,
});

  4. OpenAI-compatible function calling with WebLLM

    main

    For a more standardized and easier-to-use experience, you can use the OpenAI-compatible function calling implementation in WebLLM. This method leverages the standard tools, tool_choice, and tool_call fields.

    While this is more user-friendly and follows established patterns, it is less flexible than the manual approach because it uses pre-defined system prompts to guide the model's behavior.

  5. Project structure of the WebLLM Chrome Extension

    main

    This extension is built using Manifest V3 and consists of the following core components:

    • manifest.json: Defines the extension's structure, permissions, and behavior.
    • popup.ts: The script responsible for the extension's pop-up window UI.
    • background.ts: The service worker script. It is loaded by the browser when needed and unloaded when dormant.
    • content.js: A content script that runs in the context of web pages to interact with the DOM.

  8. Implement MCP-style tool calls using structural tags

    main

    This pattern allows you to force WebLLM to output tool calls within specific XML-like structural tags, following an MCP-style format. The workflow consists of three main steps:

    1. Define the Structural Tag: Configure a structural tag that forces the model to wrap tool calls in <tool_call>...</tool_call> blocks containing JSON payloads with {"name": ..., "arguments": ...}.
    2. Request and Parse: Call WebLLM using response_format.type = "structural_tag". When the model responds, parse the <tool_call> block to extract the tool name and arguments, then dispatch these to your tool implementation (or a stub).
    3. Return Results: Send the tool's output back to the model using a message of type tool, then request a final natural-language response from the assistant.

  9. Manual function calling with WebLLM

    main

    You can implement function calling manually to achieve maximum flexibility, especially when using models like Llama3.1 or Hermes2. This approach does not rely on the standard tools, tool_choice, or tool_call fields. Instead, you must follow the specific prompting instructions provided by the model releaser and handle the parsing of the model's output yourself.

    Note that parsing logic is model-specific. For example, Hermes2 models wrap tool calls in <tool_call> and </tool_call> tags, whereas other models may use entirely different formats.

  10. Use LogitProcessor to manipulate raw logits

    main

    The LogitProcessor allows you to manipulate raw logits before the token sampling step. This is useful for tasks like forcing the model to sample specific tokens by setting their logits to inf or preventing certain tokens from being sampled by setting them to -inf.

    Key features demonstrated in the example include:

    • Customization: Implementing your own logic in a class that extends or implements the processor interface.
    • Statefulness: LogitProcessor can maintain internal state during the generation process.
    • State Management: You can clear the internal state of a processor using the LogitProcessor.resetState() method.
    // Conceptual usage based on the example description
// To customize, implement your own logic (e.g., in my_logit_processor.ts)
// Example: setting a specific token's logit to a high value to force sampling

// To clear state:
// logitProcessor.resetState();

  11. Add custom language models to WebLLM

    main

    WebLLM supports serving custom language models that have been compiled using MLC-LLM. To add a custom model, you must first compile it using the MLC-LLM toolchain and then serve the resulting artifacts through the WebLLM runtime.

    For the complete step-by-step procedure on compiling and preparing your models for WebLLM, refer to the official MLC-LLM documentation.