TokenDagger

To verify TokenDagger against TikToken or run benchmarks, you can use the provided test scripts. You must have the original tiktoken installed for comparison tests. Use the --tokenizer flag to specify the tokenizer type (e.g., llama or mistral).

# Build the project
make clean && make

# Install tiktoken for comparison
pip3 install tiktoken

# Run compatibility tests
python3 tests/test_tokendagger_vs_tiktoken.py --tokenizer llama
python3 tests/test_tokendagger_vs_tiktoken.py --tokenizer mistral

# Run performance benchmarks
python3 tests/performance_benchmark.py --tokenizer llama
python3 tests/performance_benchmark.py --tokenizer mistral

# Run code-specific performance benchmarks
python3 tests/code_performance_benchmark.py --tokenizer llama

If you are developing or building from source, follow these steps to set up the environment. Note that libpcre2-dev is a required system dependency for the optimized regex engine.

git clone git@github.com:M4THYOU/TokenDagger.git
sudo apt install libpcre2-dev
git submodule update --init --recursive
sudo apt update && sudo apt install -y python3-dev

To use TokenDagger as a high-performance replacement for OpenAI's TikToken, install it using pip:

pip install tokendagger

TokenDagger is designed to be a drop-in replacement for the tiktoken library. To switch, install tokendagger and change your import statement from import tiktoken to import tokendagger as tiktoken. The Encoding class and its parameters remain compatible.

import tokendagger as tiktoken

# The usage remains identical to tiktoken
tokenizer = tiktoken.Encoding(
    name=name,
    pat_str=pattern,
    mergeable_ranks=vocab,
    special_tokens=special_tokens,
)

For high-performance processing of multiple strings, use the batch methods which utilize a ThreadPoolExecutor for parallel execution.

• encode_batch(text: Sequence[str], num_threads: int = 8, ...): Returns a list of lists of token IDs.
• decode_batch(tokens: Sequence[Sequence[int]], num_threads: int = 8, errors: str = "replace"): Returns a list of decoded strings.

texts = ["hello", "world", "token dagger"]

# Parallel encoding
batch_tokens = tokenizer.encode_batch(texts, num_threads=4)

# Parallel decoding
strings = tokenizer.decode_batch(batch_tokens, errors="replace")

TokenDagger provides several convenience functions to instantiate a Tokenizer:

• load_tokenizer(name, vocab_file, pattern, special_tokens_file=None): Loads a tokenizer from JSON files.
• create_tokenizer(name, pattern, vocab, special_tokens=None): Creates a tokenizer from in-memory data.
• Encoding(name, *, pat_str, mergeable_ranks, special_tokens=None): A specialized factory function for tiktoken-compatible data structures.

from tokendagger import load_tokenizer, Encoding

# From files
tok = load_tokenizer("my-tok", "vocab.json", r"<regex>", "special.json")

# From tiktoken-style dicts
tok = Encoding(
    name="tiktoken-style",
    pat_str=r"<regex>",
    mergeable_ranks={b"abc": 123}
)

The Tokenizer class provides several utility methods to inspect the vocabulary:

• n_vocab (property): Returns the total vocabulary size.
• special_tokens() (method): Returns a list of special token strings.
• special_tokens_set (property): Returns a set of special token strings.
• is_special_token(token: int) (method): Returns True if the provided token ID is a special token.

TokenDagger provides a high-performance Python interface for tokenization via the Tokenizer and Encoding classes. You can initialize a tokenizer using create_tokenizer or load a pre-configured one using load_tokenizer. All errors raised by the library are of type TokenDaggerError.

from tokendagger import create_tokenizer, load_tokenizer, Tokenizer, Encoding, TokenDaggerError

# Example usage pattern (conceptual based on exported symbols):
# tokenizer = create_tokenizer(model_name='gpt-4')
# encoding = tokenizer.get_encoding()
# tokens = tokenizer.encode("Hello world")

Use decode() to convert a sequence of token IDs back into a UTF-8 string, or decode_bytes() to get the raw bytes.

• decode(tokens: Sequence[int], errors: str = "replace"): Converts tokens to a string. The errors parameter accepts 'replace', 'ignore', or 'strict' (standard Python error handling modes).
• decode_bytes(tokens: Sequence[int]): Converts tokens directly to bytes.

tokens = [104, 101, 108, 108, 111]

# To string
text = tokenizer.decode(tokens)

# To bytes
raw_bytes = tokenizer.decode_bytes(tokens)

Use encode() to convert a string into a list of token IDs. This method supports handling special tokens via allowed_special and disallowed_special parameters.

• allowed_special: Can be set to "all" to allow all registered special tokens, or a specific set of strings. If not specified, defaults to an empty set.
• disallowed_special: Can be set to "all" to raise an error if any special token (not in allowed_special) is found in the text. If a disallowed token is encountered, a ValueError is raised.

# Encode text allowing specific special tokens
tokens = tokenizer.encode("hello <|endoftext|>", allowed_special={"<|endoftext|>"})

# Encode text and raise error if any special tokens are present
tokens = tokenizer.encode("hello <|endoftext|>", disallowed_special="all")

Most failures within the tokenizer (such as C++ extension errors or encoding failures) raise a TokenDaggerError. This is the base exception for the library. You should catch this when performing operations that rely on the underlying C++ CoreBPE engine.

from tokendagger import Tokenizer, TokenDaggerError

try:
    tokens = tokenizer.encode("some text")
except TokenDaggerError as e:
    print(f"Tokenizer operation failed: {e}")