tiktoken Documentation

The repository provides two main ways to use tiktoken in JavaScript environments:

1. tiktoken: WASM bindings for the original Python library. It provides full 1-to-1 feature parity with the Python version. Best for NodeJS and environments with full WASM support.
2. js-tiktoken: A pure JavaScript port of the core functionality. Use this in environments where WASM is not supported or desired (e.g., certain edge runtimes).

To use the pure JS port of the tiktoken BPE tokenizer in your project, install the js-tiktoken package from NPM.

npm install js-tiktoken

If your application requires access to all available OpenAI tokenizers, you can import the full library.

Caution: This will significantly increase your bundle size as it includes all tokenizer data.

import { getEncoding, encodingForModel } from "js-tiktoken";

const enc = getEncoding("gpt2");
const tokens = enc.encode("hello world");
const text = enc.decode(tokens);

To use the WASM-based bindings for NodeJS and other JS runtimes, install the tiktoken package via npm.

npm install tiktoken

Since standard Create React App does not support WASM ESM modules, you must use craco. In your craco.config.js, enable Webpack experiments and exclude .wasm files from the standard asset/resource rule so Webpack can handle them.

module.exports = {
  webpack: {
    configure: (config) => {
      config.experiments = {
        asyncWebAssembly: true,
        layers: true,
      };

      // turn off static file serving of WASM files
      // we need to let Webpack handle WASM import
      config.module.rules
        .find((i) => "oneOf" in i)
        .oneOf.find((i) => i.type === "asset/resource")
        .exclude.push(/\.wasm$/);

      return config;
    },
  },
};

To use tiktoken with Vite, you must install and configure vite-plugin-wasm and vite-plugin-top-level-await in your vite.config.js.

import wasm from "vite-plugin-wasm";
import topLevelAwait from "vite-plugin-top-level-await";
import { defineConfig } from "vite";

export default defineConfig({
  plugins: [wasm(), topLevelAwait()],
});

If you want to minimize your bundle size, use the Lite version of the library. This requires you to manually load only the specific ranks (encodings) you need. You can either import ranks directly from the package or fetch them dynamically from the Cloudflare Pages CDN.

Option 1: Direct Import Import Tiktoken from js-tiktoken/lite and the desired rank from js-tiktoken/ranks/.

Option 2: Dynamic Fetching Import Tiktoken from js-tiktoken/lite and use fetch to retrieve the encoding JSON from https://tiktoken.pages.dev/js/[encoding_name].json.

import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";

const enc = new Tiktoken(o200k_base);
// enc.encode() and enc.decode() are now available

Vercel Edge Runtime requires importing the WASM binary with a ?module suffix and using the init function from tiktoken/lite/init.

// @ts-expect-error
import wasm from "tiktoken/lite/tiktoken_bg.wasm?module";
import model from "tiktoken/encoders/cl100k_base.json";
import { init, Tiktoken } from "tiktoken/lite/init";

export const config = { runtime: "edge" };

export default async function (req: Request) {
  await init((imports) => WebAssembly.instantiate(wasm, imports));

  const encoding = new Tiktoken(
    model.bpe_ranks,
    model.special_tokens,
    model.pat_str
  );

  const tokens = encoding.encode("hello world");
  encoding.free();

  return new Response(`${tokens}`);
}

To use tiktoken in the Electron main process, ensure the WASM binary is copied into your application package. If using Electron Forge with @electron-forge/plugin-webpack, add the CopyPlugin to your webpack.main.config.js.

const CopyPlugin = require("copy-webpack-plugin");

module.exports = {
  // ...
  plugins: [
    new CopyPlugin({
      patterns: [
        { from: "./node_modules/tiktoken/tiktoken_bg.wasm" },
      ],
    }),
  ],
};

SimpleBytePairEncoding implements the BPE algorithm through these steps:

1. Regex Splitting: The input text is split into chunks (approximately words) using a provided regex pattern.
2. Byte Conversion: Each chunk is converted into its UTF-8 byte representation.
3. Iterative Merging: The algorithm looks at all adjacent pairs of tokens and finds the pair with the highest priority (lowest rank) in the mergeable_ranks dictionary. It merges that pair into a single new token and repeats the process until no more mergeable pairs remain.
4. Decoding: Tokens are mapped back to their original byte sequences using a decoder dictionary. When converting bytes back to a string, invalid UTF-8 sequences are handled using the replacement character � to prevent errors.

To support WASM in Next.js (both API routes and Pages), update your next.config.js to enable asyncWebAssembly and layers experiments in Webpack.

// next.config.js
const config = {
  webpack(config, { isServer, dev }) {
    config.experiments = {
      asyncWebAssembly: true,
      layers: true,
    };

    return config;
  },
};

Special tokens are artificial tokens used to unlock model capabilities (like fill-in-the-middle). Because they can be used to trick a model, tiktoken implements a safety mechanism:

1. Default Behavior: encode() raises a ValueError if it encounters a string that matches a special token.
2. Mitigation: If you want to allow these strings to be treated as normal text, you must explicitly pass disallowed_special=() or a specific set of tokens to disallowed_special.
3. Intentional Use: If you want to trigger a special token, you must explicitly pass it in allowed_special or set allowed_special="all".