DOCSEARCHbyAlgolia
Submit Docs

tiktoken

repository·main·Indexed 12 days ago

https://github.com/openai/tiktoken

A fast BPE (Byte Pair Encoding) tokenizer used by OpenAI models. Version 0.13.0 provides tools to retrieve encodings for specific models via encoding_for_model(), convert text to tokens using Encoding.encode(), and decode tokens back to text. It includes support for batch encoding, NumPy array output, and a plugin mechanism via tiktoken_ext for registering custom encodings.

Tokens
4.4K
Snippets
22
Records
25
Agent score
95%

What's inside tiktoken

  2. Register custom encodings using the `tiktoken_ext` plugin mechanism

    main

    If you want your custom encodings to be discoverable via tiktoken.get_encoding(), you must use the tiktoken_ext plugin mechanism. This involves creating a namespace package.

    Project Layout

    Your project must follow this structure (do not include a tiktoken_ext/__init__.py file):

    my_tiktoken_extension
├── tiktoken_ext
│   └── my_encodings.py
└── setup.py

    Implementation Steps

    1. Create my_encodings.py: This module must contain a variable named ENCODING_CONSTRUCTORS. This is a dictionary where keys are encoding names and values are functions that return the arguments required by tiktoken.Encoding.
    2. Configure setup.py: Use setuptools with find_namespace_packages to ensure the tiktoken_ext namespace is correctly handled.
    3. Install: Install your package using pip install ./my_tiktoken_extension. Do not use an editable install.

    Example setup.py

    from setuptools import setup, find_namespace_packages

setup(
    name="my_tiktoken_extension",
    packages=find_namespace_packages(include=['tiktoken_ext*']),
    install_requires=["tiktoken"],
    ...
)

  3. Extend tiktoken by creating custom `Encoding` objects

    main

    If you need to create a custom encoding (for example, to add special tokens), you can instantiate a tiktoken.Encoding object directly.

    Note: In production, it is recommended to load arguments directly rather than accessing private attributes (like _pat_str or _mergeable_ranks). For reference on how to construct specific encodings, see openai_public.py in the repository.

    When adding special tokens, ensure you use a unique name for the encoding to avoid collisions.

    import tiktoken

cl100k_base = tiktoken.get_encoding("cl100k_base")

# Create a custom encoding by passing arguments to the Encoding constructor
enc = tiktoken.Encoding(
    name="cl100k_im",
    pat_str=cl100k_base._pat_str,
    mergeable_ranks=cl100k_base._mergeable_ranks,
    special_tokens={
        **cl100k_base._special_tokens,
        "<|im_start|>": 100264,
        "<|im_end|>": 100265,
    }
)

  4. How special tokens affect encoding safety

    main

    Special tokens are artificial tokens used to unlock model capabilities (like fill-in-the-middle). Because they can be used to manipulate model behavior, tiktoken implements safety checks.

    When calling encode(), the library checks if the input text matches any known special tokens. If it does, and that token is not in allowed_special, a ValueError is raised. This prevents users from accidentally (or maliciously) injecting control tokens into a prompt.

    To bypass this, you must explicitly allow the tokens via allowed_special or disable the check entirely via disallowed_special=().

  5. Learn BPE with the educational submodule

    main

    tiktoken includes an educational submodule tiktoken._educational designed to help users understand Byte Pair Encoding (BPE). It allows you to train a simple encoding on small text samples or visualize how standard encoders (like cl100k_base) process text.

    from tiktoken._educational import *

# Train a BPE tokeniser on a small amount of text
enc = train_simple_encoding()

# Visualise how the GPT-4 encoder encodes text
enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")
enc.encode("hello world aaaaaaaaaaaa")

  6. Get an encoding by name using `get_encoding()`

    main

    Use tiktoken.get_encoding(encoding_name) to retrieve a specific encoding object by its name (e.g., "o200k_base"). Once you have the encoding object, you can use .encode() to convert text to tokens and .decode() to convert tokens back to text.

    import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

  7. Get an encoding for a specific OpenAI model using `encoding_for_model()`

    main

    If you want to use the exact tokenizer that corresponds to a specific OpenAI model (like gpt-4o), use tiktoken.encoding_for_model(model_name). This is the recommended way to ensure your token counts match the model's actual behavior.

    import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

  8. Get the encoding object for a specific model

    main

    Use encoding_for_model(model_name) to directly obtain an Encoding object (the tokenizer) used by a specific OpenAI model. This is a convenience wrapper that combines model name resolution with encoding retrieval.

    If the model name is not recognized, it raises a KeyError.

    from tiktoken.model import encoding_for_model

# Example usage
encoding = encoding_for_model("gpt-3.5-turbo")
# Now you can use the encoding object to encode/decode text
tokens = encoding.encode("Hello, world!")

  10. Decode tokens into bytes with Encoding.decode_bytes()

    main

    If you need the raw byte representation of tokens (useful for visualization or low-level processing), use decode_bytes. This returns a bytes object.

    >>> enc.decode_bytes([31373, 995])
b'hello world'

  11. Batch encode strings with Encoding.encode_batch()

    main

    To encode a list of strings in parallel, use encode_batch. This method uses a ThreadPoolExecutor to distribute the work across multiple threads. You can tune performance using the num_threads parameter (defaults to 8).

    >>> enc.encode_batch(["hello world", "goodbye world"])
[[31373, 995], [11274, 16390, 995]]

  12. Get an encoding with get_encoding()

    main

    Use get_encoding(encoding_name: str) to retrieve a specific Encoding instance by its name (e.g., 'cl100k_base'). The function handles caching internally, so subsequent calls for the same encoding name return the same instance. If the requested encoding is not found in the core library or any installed tiktoken_ext plugins, it raises a ValueError listing available plugins.

    import tiktoken

encoding = tiktoken.get_encoding("cl100k_base")
# Use the encoding to encode/decode text
tokens = encoding.encode("Hello, world!")