DOCSEARCHbyAlgolia
Submit Docs

tiktoken_ruby

repository·main·Indexed 41 days ago

https://github.com/iapark/tiktoken_ruby

A Ruby wrapper around OpenAI's BPE tokenizer used to count and manage tokens for GPT models. Version 0.1.0 provides functionality to retrieve encodings by name or model, encode text with configurable special token handling, and decode tokens into UTF-8 strings or raw bytes.

Tokens
2.6K
Snippets
12
Records
12
Agent score
51%

What's inside tiktoken_ruby

  2. Encode text with different special token handling

    main

    The library provides three ways to encode text, which differ in how they treat OpenAI special tokens (like <|endoftext|> or <|fim_prefix|>):

    1. encode_ordinary(text): Always treats special tokens as literal text (no special token recognition).
    2. encode(text, allowed_special: []): Treats special tokens as text unless they are explicitly listed in the allowed_special array.
    3. encode_with_special_tokens(text): Recognizes and parses all special tokens.

    Example of the differences:

    enc = Tiktoken.get_encoding("cl100k_base")
text = "Hello<|endoftext|>World"

# Treats <|endoftext|> as literal characters (9 tokens)
enc.encode_ordinary(text)

# Same as encode_ordinary by default
enc.encode(text)

# Recognizes the specified special token (3 tokens)
enc.encode(text, allowed_special: ["<|endoftext|!"])

# Recognizes ALL special tokens (3 tokens)
enc.encode_with_special_tokens(text)
    enc = Tiktoken.get_encoding("cl100k_base")
text = "Hello<|endoftext|>World"

# encode_ordinary: treats <|endoftext|> as literal characters (9 tokens)
enc.encode_ordinary(text)
#=> [9906, 27, 91, 8862, 728, 428, 91, 29, 10343]

# encode: same as encode_ordinary by default
enc.encode(text)
#=> [9906, 27, 91, 8862, 728, 428, 91, 29, 10343]

# encode with allowed_special: recognizes the specified special token (3 tokens)
enc.encode(text, allowed_special: ["<|endoftext|!"])
#=> [9906, 100257, 10343]

# encode_with_special_tokens: recognizes ALL special tokens (3 tokens)
enc.encode_with_special_tokens(text)
#=> [9906, 100257, 10343]

  3. Retrieve an encoding by name or model

    main

    You can obtain an encoding instance using Tiktoken.get_encoding(encoding_name) or Tiktoken.encoding_for_model(model_name). This instance provides the methods for encoding and decoding text.

    require 'tiktoken_ruby'

# By encoding name
enc = Tiktoken.get_encoding("cl100k_base")

# By model name
enc = Tiktoken.encoding_for_model("gpt-4")

  4. Decode tokens into strings or bytes

    main

    Decoding converts token arrays back into text. Because BPE tokens are byte-level, truncating a token array can result in invalid UTF-8 sequences (e.g., when splitting an emoji). You can control how these errors are handled using the errors: option in decode.

    • decode(tokens, errors: :strict): Decodes to a UTF-8 string. :strict (default) raises Tiktoken::UnicodeError on invalid sequences. :replace substitutes invalid sequences with � (matching Python's tiktoken behavior).
    • decode_bytes(tokens): Decodes tokens into raw bytes (ASCII-8BIT string) without UTF-8 validation.
    enc = Tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("🦄") # the emoji spans multiple tokens

# :strict (default) - raise Tiktoken::UnicodeError on invalid UTF-8
enc.decode(tokens.first(2))                    #=> raises Tiktoken::UnicodeError

# :replace - substitute invalid sequences with "�" (matches Python tiktoken's default)
enc.decode(tokens.first(2), errors: :replace)  #=> "�"

# decode_bytes - get the raw bytes and handle them yourself
enc.decode_bytes(tokens.first(2))              #=> "\xF0\x9F\xA6" (ASCII-8BIT)

  6. Retrieve an encoding by name with Tiktoken.get_encoding

    main

    Use Tiktoken.get_encoding(name) to load a specific encoding instance. If the encoding is already loaded, the method reuses the existing instance; otherwise, it loads it.

    Parameters:

    • name (Symbol|String): The name of the encoding (e.g., :cl100k_base).

    Returns:

    • Tiktoken::Encoding: The encoding instance.
    • nil: If the requested encoding name is not supported.
    enc = Tiktoken.get_encoding("cl100k_base")
enc.decode(enc.encode("hello world")) #=> "hello world"

  7. Decode tokens back into text

    main

    To convert an array of integer tokens back into a string, use the decode method. Because BPE tokens are byte-level, truncating token arrays can result in invalid UTF-8 sequences. You can control how these errors are handled using the errors option:

    • :strict (default): Raises a Tiktoken::UnicodeError if invalid UTF-8 is encountered.
    • :replace: Substitutes invalid sequences with the Unicode replacement character (�), matching the default behavior of Python's tiktoken library.
    encoding = Tiktoken::Encoding.for_name_cached(:cl100k_base)
tokens = [15496, 1234]

# Default behavior (raises error on invalid UTF-8)
text = encoding.decode(tokens, errors: :strict)

# Replace invalid sequences with �
text = encoding.decode(tokens, errors: :replace)

  8. Decode tokens into raw bytes

    main

    If you need to handle byte sequences manually without UTF-8 validation, use decode_bytes(tokens). This returns a string with ASCII-8BIT (binary) encoding. This is useful when working with partial tokens or when you want to perform your own error handling, matching Python's decode_bytes behavior.

    encoding = Tiktoken::Encoding.for_name_cached(:cl100k_base)
tokens = [15496, 1234]

# Returns a binary (ASCII-8BIT) string
bytes = encoding.decode_bytes(tokens)

  9. Encode text into tokens

    main

    The Tiktoken::Encoding class provides several ways to convert text into an array of integer tokens, depending on how you want to handle special tokens (like <|endoftext|>):

    • encode_ordinary(text): Treats special tokens as literal text. It does not recognize special non-text tokens.
    • encode(text, allowed_special: []): Treats special tokens as text unless they are explicitly included in the allowed_special array of strings.
    • encode_with_special_tokens(text): Recognizes and encodes all special tokens present in the text.
    encoding = Tiktoken::Encoding.for_name_cached(:cl100k_base)

# 1. Treat special tokens as literal text
tokens = encoding.encode_ordinary("hello <|endoftext|>")

# 2. Allow specific special tokens
tokens = encoding.encode("hello <|endoftext|>", allowed_special: ["<|endoftext|>"])

# 3. Recognize all special tokens
tokens = encoding.encode_with_special_tokens("hello <|endoftext|>")

  11. Load a Tiktoken::Encoding instance

    main

    You can obtain an encoding instance using either for_name or for_name_cached.

    • for_name(encoding): Returns a new instance for the specified encoding name (passed as a Symbol).
    • for_name_cached(encoding): Returns an existing instance if it has already been loaded, otherwise creates a new one. This is recommended for performance to avoid redundant loading.
    # Load a new instance
encoding = Tiktoken::Encoding.for_name(:cl100k_base)

# Load or retrieve from cache (recommended)
encoding = Tiktoken::Encoding.for_name_cached(:cl100k_base)

  12. Retrieve an encoding for an OpenAI model with Tiktoken.encoding_for_model

    main

    Use Tiktoken.encoding_for_model(model_name) to automatically retrieve the correct encoding associated with a specific OpenAI model name. This handles both exact model matches and prefix-based matching (e.g., matching gpt-4o-2024-05-13 to cl100k_base).

    Parameters:

    • model_name (Symbol|String): The name of the OpenAI model.

    Returns:

    • Tiktoken::Encoding: The encoding instance for that model.
    • nil: If no encoding is found for the provided model name.
    enc = Tiktoken.encoding_for_model("gpt-4")
enc.encode("hello world").length #=> 2