WebLLM Chat

WebLLM Chat is a private AI chat interface that runs large language models (LLMs) natively in the browser using WebGPU acceleration. It leverages WebLLM to eliminate the need for server-side processing, ensuring that data and conversations stay local to the user's hardware.

Key capabilities include:

• Browser-Native AI: Uses WebGPU for local execution.
• Privacy: All data processing happens within the browser.
• Offline Support: Works offline after the initial model download.
• Vision Support: Ability to upload and chat with images.
• Custom Models: Support for connecting to local models via MLC-LLM REST APIs.

You can build the application in two ways depending on your deployment needs:

• Next.js Build: Use yarn build for a standard Next.js deployment.
• Static Site: Use yarn export to generate a static site.

To run the project locally for development, ensure you have nodejs and yarn installed. You must also configure your local environment variables in a .env.local file before running the application.

Run the following commands to install dependencies and start the development server:

yarn install
yarn dev

You can connect WebLLM Chat to custom language models running in your local environment using MLC-LLM.

Follow these steps:

1. Compile the model: (Optional) Convert your model into MLC format following the MLC-LLM compilation instructions.
2. Host a REST API: Deploy the model using the MLC-LLM REST API deployment guide.
3. Configure WebLLM Chat:
   • Open the WebLLM Chat interface.
   • Select Settings in the sidebar.
   • Set Model Type to MLC-LLM REST API (Advanced).
   • Enter your local REST API endpoint URL.

You can containerize the application using Docker.

Basic deployment:

docker build -t webllm_chat .
docker run -d -p 3000:3000 webllm_chat

Deployment with a proxy: If you need to run the service behind a proxy, use the PROXY_URL environment variable. If the proxy requires authentication, include the credentials in the URL.

# Basic proxy setup
docker build -t webllm_chat .
docker run -d -p 3000:3000 \
   -e PROXY_URL=http://localhost:7890 \
   webllm_chat

# Proxy with username and password
docker run -d -p 3000:3000 \
   -e PROXY_URL="http://127.0.0.1:7890 user pass" \
   webllm_chat

Use getClientConfig() to retrieve the application's build configuration. The function is environment-aware:

• Client-side (Browser): It attempts to extract the configuration from a <meta name='config'> tag in the document head and parses it as a BuildConfig object.
• Server-side (Node.js/SSR): It calls getBuildConfig() to return the server-side build configuration.

This ensures that the application uses the correct build-time constants regardless of whether the code is executing in the browser or during server-side rendering.

import { getClientConfig } from "./app/config/client";

const config = getClientConfig();
// config is of type BuildConfig
console.log(config);

The localization module provides utilities to detect, retrieve, and change the application's language. It prioritizes language selection in the following order:

1. URL Parameter: The ?lang=... query parameter.
2. LocalStorage: The previously saved language preference.
3. Browser Settings: The user's navigator.language.
4. Default: Falls back to en (English).

When a language is selected, it is persisted in localStorage under the key lang and the page is reloaded to apply changes.

import { getLang, changeLang, getISOLang } from "@/app/locales";

// Get the current active language key (e.g., 'en', 'cn')
const currentLang = getLang();

// Change the language and reload the page
changeLang('jp');

// Get the ISO language string (e.g., 'zh-Hans' for 'cn')
const isoLang = getISOLang();

The following constants provide the list of supported language keys (Lang) and their corresponding human-readable display names used in UI selection menus.

Supported Language Keys (Lang): cn, en, tw, pt, jp, ko, id, fr, es, it, tr, de, vi, ru, cs, no, ar, bn, sk

Language Display Names (ALL_LANG_OPTIONS):

export const ALL_LANG_OPTIONS: Record<Lang, string> = {
  cn: "简体中文",
  en: "English",
  pt: "Português",
  tw: "繁體中文",
  jp: "日本語",
  ko: "한국어",
  id: "Indonesia",
  fr: "Français",
  es: "Español",
  it: "Italiano",
  tr: "Türkçe",
  de: "Deutsch",
  vi: "Tiếng Việt",
  ru: "Русский",
  cs: "Čeština",
  no: "Nynorsk",
  ar: "العربية",
  bn: "বাংলা",
  sk: "Slovensky",
};

The BUILTIN_TEMPLATE_ID constant is the starting integer used to generate unique identifiers for built-in chat templates. This ensures that system templates occupy a distinct ID space from user-defined templates.

export const BUILTIN_TEMPLATE_ID = 100000;

The module exports types for defining and partially implementing translation objects. LocaleType represents the full set of required translation keys, while PartialLocaleType allows for incomplete translation objects (useful for overrides or partial implementations).

import type { LocaleType, PartialLocaleType } from "@/app/locales";

const myPartialConfig: PartialLocaleType = {
  // ... partial keys
};

The getISOLang() function returns the ISO language code for the currently active language. This is particularly useful for handling Chinese language variants where the key is simplified.

• If lang is cn, returns zh-Hans.
• If lang is tw, returns zh-Hant.
• For all other supported languages, it returns the key itself (e.g., en, fr).

import { getISOLang } from "@/app/locales";

const iso = getISOLang(); // returns 'zh-Hans' if current lang is 'cn'