DOCSEARCHbyAlgolia
Submit Docs

tiktoken-rs

repository·main·Indexed 19 days ago

https://github.com/zurawiki/tiktoken-rs

A Rust library for encoding and decoding text using OpenAI's tiktoken encodings. It provides tools for counting tokens, calculating max_tokens for chat and text completion requests, and retrieving context window sizes for various OpenAI models. The library supports multiple encodings including cl100k_base, o200k_base, and o200k_harmony, and offers singleton instances for high-performance reuse. It also includes optional integration with the async-openai crate.

Tokens
6.3K
Snippets
23
Records
27
Agent score
63%

What's inside tiktoken-rs

  2. Integrate with async-openai

    main

    If you are using the async-openai crate, tiktoken-rs provides a feature async-openai that allows you to work directly with async_openai types.

    Within the async_openai module, you can use:

    • num_tokens_from_messages: Accepts &[async_openai::types::chat::ChatCompletionRequestMessage].
    • get_chat_completion_max_tokens: Accepts &[async_openai::types::chat::ChatCompletionRequestMessage].

    Note: Only text content is counted; non-text parts are skipped.

  3. Count token length in text

    main

    You can count tokens by initializing a specific encoding (e.g., o200k_base) and calling encode_with_special_tokens.

    For performance in applications with repeated calls, use the _singleton version to avoid the overhead of re-initializing the tokenizer on every call.

    use tiktoken_rs::o200k_base_singleton;

let bpe = o200k_base_singleton();
let tokens = bpe.encode_with_special_tokens(
  "This is a sentence   with spaces"
);
println!("Token count: {}", tokens.len());

  4. Calculate max_tokens for async-openai requests

    main

    If you are using the async-openai crate, enable the async-openai feature in your Cargo.toml. You can then use tiktoken_rs::async_openai::get_chat_completion_max_tokens which accepts async_openai message types.

    use tiktoken_rs::async_openai::get_chat_completion_max_tokens;
use async_openai::types::chat::{
    ChatCompletionRequestMessage, ChatCompletionRequestSystemMessage,
    ChatCompletionRequestSystemMessageContent, ChatCompletionRequestUserMessage,
    ChatCompletionRequestUserMessageContent,
};

let messages = vec![
    ChatCompletionRequestMessage::System(ChatCompletionRequestSystemMessage {
        content: ChatCompletionRequestSystemMessageContent::Text(
            "You are a helpful assistant that only speaks French.".to_string(),
        ),
        name: None,
    }),
    ChatCompletionRequestMessage::User(ChatCompletionRequestUserMessage {
        content: ChatCompletionRequestUserMessageContent::Text(
            "Hello, how are you?".to_string(),
        ),
        name: None,
    }),
];
let max_tokens = get_chat_completion_max_tokens("o1-mini", &messages).unwrap();
println!("max_tokens: {}", max_tokens);

  5. Calculate max_tokens for a Chat Completion request

    main

    Use get_chat_completion_max_tokens to calculate the required max_tokens parameter for a list of ChatCompletionRequestMessage objects. This helps ensure your request fits within the model's context window.

    use tiktoken_rs::{get_chat_completion_max_tokens, ChatCompletionRequestMessage};

let messages = vec![
    ChatCompletionRequestMessage {
        content: Some("You are a helpful assistant that only speaks French.".to_string()),
        role: "system".to_string(),
        ..Default::default()
    },
    ChatCompletionRequestMessage {
        content: Some("Hello, how are you?".to_string()),
        role: "user".to_string(),
        ..Default::default()
    },
    ChatCompletionRequestMessage {
        content: Some("Parlez-vous francais?".to_string()),
        role: "system".to_string(),
        ..Default::default()
    },
];
let max_tokens = get_chat_completion_max_tokens("o1-mini", &messages).unwrap();
println!("max_tokens: {}", max_tokens);

  6. Use CoreBPE::encode with special tokens

    main

    The CoreBPE::encode method mirrors the upstream tiktoken implementation and returns a Result. You must propagate or unwrap this result. It requires a slice of allowed special tokens.

    use tiktoken_rs::o200k_base;

let bpe = o200k_base().unwrap();
let allowed = bpe.special_tokens();
let (tokens, last_piece_token_len) = bpe.encode("hello <|endoftext|>", &allowed).unwrap();

  7. Reference: Supported Encodings and OpenAI Models

    main

    The following encodings are supported by tiktoken-rs and correspond to specific OpenAI model families:

    | Encoding name           | OpenAI models                                                                  |
| ----------------------- | ------------------------------------------------------------------------------ |
| `o200k_harmony`         | `gpt-oss-20b`, `gpt-oss-120b`                                                                  |
| `o200k_base`            | GPT-5 series, `o1`/`o3`/`o4` series, `gpt-4o`, `gpt-4.5`, `gpt-4.1`, `codex-*` |
| `cl100k_base`           | `gpt-4`, `gpt-3.5-turbo`, `text-embedding-ada-002`, `text-embedding-3-*`         |
| `p50k_base`             | Code models, `text-davinci-002`, `text-davinci-003`                             |
| `p50k_edit`             | Edit models like `text-davinci-edit-001`, `code-davinci-edit-001`              |
| `r50k_base` (or `gpt2`) | GPT-3 models like `davinci`                                                                    |

  8. Reference: Model Context Window Sizes

    main

    Use this table to understand the context window limits for various OpenAI models supported by the library:

    | Model                                                               | Context window |
| ------------------------------------------------------------------- | ----------------- |
| `gpt-5.4`, `gpt-5.4-pro`                                            | 1,050,000      |
| `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`                           | 1,047,576      |
| `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-5.4-mini`, `gpt-5.4-nano` | 400,000        |
| `o1`, `o3`, `o3-mini`, `o3-pro`, `o4-mini`                          | 200,000        |
| `codex-mini`                                                        | 200,000        |
| `gpt-oss`                                                           | 131,072        |
| `gpt-4o`, `gpt-4o-mini`                                            | 128,000        |
| `o1-mini`, `gpt-5.3-codex-spark`                                   | 128,000        |
| `gpt-3.5-turbo`                                                    | 16,385         |
| `gpt-4`                                                             | 8,192           |

  9. Calculate max tokens for text completions

    main

    Use get_text_completion_max_tokens to determine how many tokens are available for a legacy text/prompt completion (single string input). It calculates context_size - prompt_tokens for the specified model.

    Note: For chat completions, use get_chat_completion_max_tokens instead.

    use tiktoken_rs::get_text_completion_max_tokens;

let max_tokens = get_text_completion_max_tokens("gpt-4o", "Translate to French: '").unwrap();

  10. Get BPE tokenizer for a model or tokenizer type

    main

    You can retrieve a cached, thread-safe reference to a CoreBPE singleton using either a model name or a specific Tokenizer enum variant.

    • Use bpe_for_model(model: &str) to look up the tokenizer associated with a model name (e.g., "gpt-4o").
    • Use bpe_for_tokenizer(tokenizer: Tokenizer) to get the BPE instance for a specific tokenizer type.
    // By model name
use tiktoken_rs::bpe_for_model;
let bpe = bpe_for_model("gpt-4o").unwrap();
let tokens = bpe.encode_with_special_tokens("hello world");

// By Tokenizer variant
use tiktoken_rs::{bpe_for_tokenizer, tokenizer::Tokenizer};
let bpe = bpe_for_tokenizer(Tokenizer::O200kBase).unwrap();
let tokens = bpe.encode_with_special_tokens("hello world");

  11. Use the tiktoken-rs public API

    main

    The tiktoken-rs crate provides a high-performance Rust interface for OpenAI's tiktoken tokenizer. The public API is primarily exposed through the api module and includes core functionality for encoding text into tokens and decoding tokens back into text.

    Key components include:

    • api: Contains the primary high-level functions for interacting with tokenizers.
    • model: Defines the supported encoding models (e.g., cl100k_base, p50k_base).
    • tokenizer: Provides the underlying tokenizer implementation.
    • singleton: Manages singleton instances of tokenizers for efficient reuse.
    • openai_public: Provides extensions and utilities compatible with OpenAI's public encoding patterns.

  12. Count tokens in chat messages

    main

    Use num_tokens_from_messages to estimate the total number of tokens required to encode a sequence of chat messages for a specific model. This is useful for managing context windows and estimating API costs.

    Important Details:

    • It accounts for message framing overhead, name fields, function/tool calls, and reply priming.
    • Non-text content: Only text content is counted. Non-text parts (images, audio, files) are silently skipped because they use a separate token formula. If your messages contain non-text content, the returned count will be lower than the actual API token usage.
    • Supported Tokenizers: Chat token counting is only supported for models using Cl100kBase, O200kBase, or O200kHarmony tokenizers.
    use tiktoken_rs::{num_tokens_from_messages, ChatCompletionRequestMessage};

// Example usage depends on your specific message structure
let num_tokens = num_tokens_from_messages("gpt-4o", &messages).unwrap();