JTokkit Documentation

JTokkit is a high-performance Java tokenizer designed for natural language processing tasks involving OpenAI models (such as gpt-3.5-turbo). It aims to provide JVM-based capabilities similar to OpenAI's tiktoken library.

Key Features

• Supported Encodings: Implements r50k_base, p50k_base, p50k_edit, and cl100k_base for encoding and decoding.
• Extensibility: Provides an easy way to extend the library with custom encoding algorithms.
• Performance: Optimized for high throughput, reaching 2-3 times the throughput of comparable tokenizers.
• Lightweight: Zero external dependencies and supports Java 8+.

To use JTokkit in your project, add the following dependency to your build configuration file. JTokkit has zero dependencies and supports Java 8 and above.

<!-- Maven -->
<dependency>
    <groupId>com.knuddels</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.0</version>
</dependency>

<!-- Gradle -->
dependencies {
	implementation 'com.knuddels:jtokkit:1.1.0'
}

To use JTokkit, you must first create an EncodingRegistry. You should maintain a single reference to this registry throughout your application's lifecycle because creating it is an expensive operation (it loads vocabularies from the classpath). The registry is thread-safe and handles caching of loaded encodings.

There are two ways to initialize a registry:

1. Default Registry: Loads all vocabularies for all encodings immediately upon creation.
2. Lazy Loading Registry: Only loads vocabularies for encodings that are actually accessed. This is more efficient if you only use a subset of available encodings.

// Default registry (loads all vocabularies immediately)
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();

// Lazy loading registry (loads vocabularies only when accessed)
EncodingRegistry registry = Encodings.newLazyEncodingRegistry();

To support completely custom encodings, implement the Encoding interface and register your implementation with an EncodingRegistry.

Requirements:

• The name returned by Encoding#getName() must be unique.
• Your implementation must be thread-safe, as the EncodingRegistry caches and reuses instances.

Once registered, you can retrieve your encoding using registry.getEncoding("your-encoding-name").

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding customEncoding = new CustomEncoding();
registry.register(customEncoding);

// Get the encoding from the registry
Encoding encodingFromRegistry = registry.getEncoding("custom-name");

If you want to add a new byte pair encoding (BPE) without implementing the full Encoding interface from scratch, you can use GptBytePairEncodingParams to define the necessary parameters and register it via registry.registerGptBytePairEncoding(params).

This method is useful for creating variations of existing GPT-style encodings by providing custom patterns and maps.

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
GptBytePairEncodingParams params = new GptBytePairEncodingParams(
        "custom-name",
        Pattern.compile("some custom pattern"),
        encodingMap,
        specialTokenEncodingMap
);
registry.registerGptBytePairEncoding(params);

// Get the encoding from the registry
Encoding encodingFromRegistry = registry.getEncoding("custom-name");

You can limit the number of tokens produced by passing a maxTokens integer to the encode method. This will truncate the output to the specified length.

JTokkit automatically handles Unicode characters that might be split by truncation; it ensures that if a character is split, the partial token is removed so that the resulting decoded string remains valid.

// Truncate to 3 tokens
IntArrayList encoded = encoding.encode("This is a sample sentence.", 3);
// encoded = [2028, 374, 264]

// Truncate while handling Unicode (e.g., emojis)
IntArrayList encodedEmoji = encoding.encode("I love 🍕", 4);
// The library ensures the emoji is not split incorrectly

If you only need the number of tokens without performing the full encoding, use Encoding#countTokens or Encoding#countTokensOrdinary. These methods are optimized for speed and are faster than calling encode and checking the list size.

Use countTokensOrdinary if the text might contain special tokens that you want to treat as normal text.

// Fast token counting
int tokenCount = encoding.countTokens("This is a sample sentence.");

// Fast token counting including special tokens as ordinary text
int tokenCountOrdinary = encoding.countTokensOrdinary("hello <|endoftext|> world");

An Encoding instance allows you to convert text into a list of token IDs (IntArrayList) and back into a string. The Encoding object is thread-safe.

Handling Special Tokens: By default, Encoding#encode does not support special tokens (e.g., <|endoftext|>). If it encounters one, it throws an UnsupportedOperationException. To treat special tokens as ordinary text, use Encoding#encodeOrdinary instead.

// Standard encoding/decoding
IntArrayList encoded = encoding.encode("This is a sample sentence.");
String decoded = encoding.decode(encoded);

// Handling special tokens as ordinary text
// encoding.encode("hello <|endoftext|> world") -> throws UnsupportedOperationException
IntArrayList encodedOrdinary = encoding.encodeOrdinary("hello <|endoftext|> world");