wllama

Wllama uses two primary mechanisms to read GGUF files without making unnecessary copies. Note that Wllama only accepts Blob as input data.

1. Async File Read

This method hooks into fopen, fseek, and fread to forward calls to the main thread, where Blob.slice() is used to read data. This requires JSPI / Asyncify.

• Mechanism: When fread() is called, it triggers an fs.read_req message. The main thread reads the slice and returns it via fs.read_res.
• Optimization: To prevent bottlenecks from small metadata reads, the minimum read size is 1MB. If a request is smaller than 1MB, the full 1MB block is cached for subsequent reads.
• Activation: Set the environment variable USE_ASYNC_FILE to signal the Wasm module to use this mode.

2. HeapFS

HeapFS is a wrapper around Emscripten's FS driver designed to allow mmap() to map to existing data instead of copying it.

• Mechanism: A ReadableStream for the Blob is created in the main thread. Data is streamed to the worker via fs.write messages. Once complete, model loading is triggered with mmap = true.
• Limitation: On WebGPU, even if tensors are offloaded to the GPU, the full model must still be allocated in main memory (e.g., a 4GB model occupies 4GB of main memory).

V3.0 introduces breaking changes due to the architectural shift to server-context:

Removed APIs

• tokenize / detokenize (Low-level tokenizer API removed)
• decode / encode (Replaced by OAI completion API)
• samplingInit / samplingAccept / samplingSample (Sampling is now per-request)
• Sequence shift/remove operations

Configuration Changes

• Sampling Parameters: No longer passed at model load time. They must be provided per-request in createChatCompletion or createCompletion.
• Context Length: n_ctx_auto is removed. You must set n_ctx explicitly at load time.
• Multimodal Loading: Use mmprojUrl in loadModelFromUrl instead of separate file selection.
• Chat Templates: Chat templates are now parsed with Jinja. Enable this by setting jinja: true at load time, or override with chat_template.

Error Handling

• A new WllamaError type 'kv_cache_full' is available when the context runs out of space.

In v2.0, the Wllama constructor has been simplified. You no longer need to provide .js files in the configuration paths; you only need to specify the *.wasm files for each thread configuration.

Standard Configuration

Provide a mapping of thread types to their corresponding .wasm files:

const CONFIG_PATHS = {
  'single-thread/wllama.wasm': '../../esm/single-thread/wllama.wasm',
  'multi-thread/wllama.wasm' : '../../esm/multi-thread/wllama.wasm',
};
const wllama = new Wllama(CONFIG_PATHS);

Using CDN (Not recommended for production)

If you cannot embed WASM files in your project, you can use the provided CDN helper:

import WasmFromCDN from '@wllama/wllama/esm/wasm-from-cdn.js';
const wllama = new Wllama(WasmFromCDN);

Constructor Options

The constructor accepts an optional second parameter of type WllamaConfig. Note that many options previously passed to loadModelFromUrl (like parallelDownloads and allowOffline) have moved here.

const CONFIG_PATHS = {
  'single-thread/wllama.wasm': '../../esm/single-thread/wllama.wasm',
  'multi-thread/wllama.wasm' : '../../esm/multi-thread/wllama.wasm',
};

const wllama = new Wllama(CONFIG_PATHS, {
  parallelDownloads: 5, // maximum concurrent downloads
  allowOffline: false, // whether to allow offline model loading
});

Tool calling is supported for models that include tool-call templates (e.g., Qwen, Llama). You define tools in the tools array and pass them to createChatCompletion. If the model returns a finish_reason of 'tool_calls', you must execute the tool locally and feed the result back to the model in a subsequent turn using the 'tool' role.

const tools = [
  {
    type: 'function',
    function: {
      name: 'get_weather',
      description: 'Get the current weather for a given city.',
      parameters: {
        type: 'object',
        properties: {
          city: { type: 'string', description: 'City name' },
        },
        required: ['city'],
      },
    },
  },
];

const messages = [{ role: 'user', content: 'What is the weather in Tokyo?' }];

// First turn: model decides to call a tool
const response = await wllama.createChatCompletion({
  messages,
  tools,
  tool_choice: 'auto',
  max_tokens: 256,
});

const choice = response.choices[0];
if (choice.finish_reason === 'tool_calls') {
  const toolCall = choice.message.tool_calls[0];
  const args = JSON.parse(toolCall.function.arguments);
  const result = { condition: 'rain', temperature_celsius: 21 };

  // Second turn: feed tool result back
  messages.push(choice.message);
  messages.push({
    role: 'tool',
    tool_call_id: toolCall.id,
    content: JSON.stringify(result),
  });

  const final = await wllama.createChatCompletion({ messages, max_tokens: 256 });
  console.log(final.choices[0].message.content);
}

Models with a vision projector (mmproj) can process image and audio inputs. You can load these models from Hugging Face or explicit URLs using loadModelFromHF or loadModelFromUrl. When calling createChatCompletion, pass the media as an ArrayBuffer within the messages array using the image or audio type.

// Load from Hugging Face
await wllama.loadModelFromHF({
  repo: 'user/model-GGUF',
  quant: 'Q4_K_M',
  mmprojQuant: 'Q8_0',
});

// Or load from explicit URLs
await wllama.loadModelFromUrl({
  url: 'https://example.com/model.gguf',
  mmprojUrl: 'https://example.com/mmproj.gguf',
});

// Pass an image as ArrayBuffer alongside text
const imageData = await fetch('./photo.jpg').then(r => r.arrayBuffer());

const response = await wllama.createChatCompletion({
  messages: [
    {
      role: 'user',
      content: [
        { type: 'image', data: imageData },
        { type: 'text', text: 'Describe this image.' },
      ],
    },
  ],
  max_tokens: 512,
});