gpt-tokenizer

To run the showcase playground locally, navigate to the demo directory, install the dependencies, and start the Vite development server.

1. cd demo
2. npm install
3. npm run dev

The development server will be available at http://localhost:5173.

By default, importing from gpt-tokenizer uses the o200k_base encoding. To use a different model or encoding, import from the specific subpath.

Import by Model

Use the model name to get the correct encoding automatically:

import { encode, decode } from 'gpt-tokenizer/model/gpt-3.5-turbo'

Import by Encoding

If the model is not explicitly listed, you can load the BPE encoding directly:

import { encode, decode } from 'gpt-tokenizer/encoding/cl100k_base'

Lazy Loading

For asynchronous loading (e.g., to reduce initial bundle size), use dynamic imports:

const { encode, decode } = await import('gpt-tokenizer/model/gpt-3.5-turbo')

Note: If your environment does not support package.json exports resolution, you may need to use the cjs or esm directories (e.g., gpt-tokenizer/cjs/model/gpt-3.5-turbo).

The demo project uses a local file dependency for the core package: "gpt-tokenizer": "file:..".

When deploying, you must either:

1. Install the gpt-tokenizer package from the repository root first.
2. Replace the local file dependency with a published version of gpt-tokenizer from a registry like npm.

For browser environments without a bundler, you can include the library via a <script> tag from unpkg. The global object name is a concatenation of GPTTokenizer_ and the specific encoding name (e.g., GPTTokenizer_cl100k_base).

If you need a specific encoding, fetch the corresponding script URL:

• o200k_base.js: Modern models (gpt-5, gpt-4o, gpt-4.1, o1, etc.)
• o200k_harmony.js: Open-weight Harmony models (gpt-oss-20b, gpt-oss-120b)
• cl100k_base.js: gpt-4 and gpt-3.5
• p50k_base.js
• p50k_edit.js
• r50k_base.js

<script src="https://unpkg.com/gpt-tokenizer"></script>

<script>
  // the package is now available as a global:
  const { encode, decode } = GPTTokenizer_cl100k_base
</script>

To use gpt-tokenizer in a Node.js or web project, install it as an NPM package.

npm install gpt-tokenizer

When working with chat-based models, the library uses specific separators for roles and messages to ensure token counting matches the actual model behavior.

• GPT-3.5 models use \n for both messageSeparator and roleSeparator.
• GPT-4 models use an empty string '' for messageSeparator and a special token ImSep for roleSeparator.

Use encode to convert text into a sequence of integer tokens, and decode to convert tokens back into text.

import { encode, decode } from 'gpt-tokenizer'

const text = 'Hello, world!'

// Encode text into tokens
const tokens = encode(text)

// Decode tokens back into text
const decodedText = decode(tokens)

Use isWithinTokenLimit(text, tokenLimit, encodeOptions?) to check if a string or an iterable of ChatMessage objects fits within a specific token limit.

• Returns false if the limit is exceeded.
• Returns the number of tokens if it is within the limit.

This method is optimized to check limits without necessarily encoding the entire input.

import { isWithinTokenLimit, ALL_SPECIAL_TOKENS } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokenLimit = 10
const withinTokenLimit = isWithinTokenLimit(text, tokenLimit)

const withinTokenLimitWithSpecial = isWithinTokenLimit(text, tokenLimit, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})

Use decode(tokens) to convert a sequence of numeric tokens back into a human-readable string. This is useful for converting GPT model outputs back into text.

import { decode } from 'gpt-tokenizer'

const tokens = [18435, 198, 23132, 328]
const text = decode(tokens)

The tokenizer uses an LRU (Least Recently Used) cache to speed up encoding for similar strings. By default, it stores up to 100,000 merged token pairs.

Managing the Cache

• setMergeCacheSize(size): Adjust the cache size. Set to 0 to disable caching completely.
• clearMergeCache(): Explicitly clear the cache to free up memory.

Increasing the size improves speed for similar strings but increases memory consumption.

import { setMergeCacheSize, clearMergeCache } from 'gpt-tokenizer'

// Set to 5000 entries
setMergeCacheSize(5000)

// Disable caching completely
setMergeCacheSize(0)

// Clear the cache
clearMergeCache()