DOCSEARCHbyAlgolia
Submit Docs

tokenizers-ffi

repository·main·Indexed 42 days ago

https://github.com/daulet/tokenizers

FFI bindings for HuggingFace tokenizers with tiktoken support, providing high-performance text encoding and decoding capabilities. It includes a UnifiedTokenizer interface for both HuggingFace and tiktoken BPE encoders, Go bindings, and a WASM module for client-side tokenization. Version 1.26.0.

Tokens
6.7K
Snippets
28
Records
34
Agent score
53%

What's inside tokenizers-ffi

  1. Install the tokenizers Go bindings

    main

    To use the bindings, you must first build the static library libtokenizers.a. Run the following command in the repository root:

    make build

    When running your Go application, you must inform the linker where to find the static library. You can do this in two ways:

    1. Via go run flags:

      go run -ldflags="-extldflags '-L./path/to/libtokenizers/directory'" .

    2. Via the CGO_LDFLAGS environment variable (recommended): Set this variable to avoid specifying the path every time you run or build your application.

      CGO_LDFLAGS="-L./path/to/libtokenizers/directory"

  2. Use the Tokenizers WASM Web interface

    main

    The web application allows for client-side tokenization and decoding using HuggingFace tokenizer.json files.

    Encoding Text

    1. Load a tokenizer: Provide a URL to a HuggingFace tokenizer.json file (e.g., https://huggingface.co/bert-base-uncased/resolve/main/tokenizer.json) or upload a local file.
    2. Encode: Enter text and click "Encode".
      • Add special tokens: Toggle to include/exclude tokens like [CLS] or [SEP].
      • Show tokens: Toggle to see the string representation of tokens.
      • Show offsets: Toggle to see character offsets for each token.

    Decoding Token IDs

    1. Decode: Enter comma or space-separated token IDs.
      • Skip special tokens: Toggle to exclude special tokens from the resulting text output.

  3. Run the Go example

    main

    To run the Go example, you must first obtain the pre-built Rust library (libtokenizers). The execution requires passing the library path to the linker via -ldflags so the Go binary can locate the shared object.

    # If the library is in the current directory
go run -ldflags="-extldflags '-L$(pwd)'" main.go

# If you are running from the example directory after a 'make build' in the parent
go run -ldflags="-extldflags '-L..'" main.go

  5. Download the Rust library for M1+ Mac

    main

    If you are on an M1+ Mac (ARM64 architecture), download and extract the required libtokenizers.darwin-arm64.tar.gz binary from the latest release to use with the Go example.

    curl -fsSL https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.darwin-arm64.tar.gz | tar xvz

  8. Use pre-built binaries for tokenizers

    main

    If you do not want to install the Rust toolchain locally, you can use pre-built binaries from the releases page.

    Alternatively, you can build the binaries using Docker:

    docker build --platform=linux/amd64 -f release/Dockerfile .

  9. Use the UnifiedTokenizer for HuggingFace and Tiktoken

    main

    The UnifiedTokenizer provides a single interface to interact with both HuggingFace Tokenizer objects and tiktoken BPE encoders. This allows you to write code that is agnostic to the underlying tokenizer implementation.

    Supported Tokenizer Types

    • HuggingFace: Uses the standard HuggingFace Tokenizer implementation.
    • Tiktoken: Uses tiktoken_rs for BPE encoding, commonly used for models like GPT-4 or Llama 3.

    Core Methods

    • encode(text, add_special_tokens): Converts text into a vector of token IDs (Vec<u32>).
    • encode_with_details(text, add_special_tokens): Returns an EncodingDetails struct containing IDs, tokens, attention masks, and offsets. Note that Tiktoken support provides fewer details than HuggingFace.
    • decode(ids, skip_special_tokens): Converts token IDs back into a string. For Tiktoken, this includes robust handling of partial UTF-8 sequences by inserting replacement characters (�).
    • vocab_size(): Returns the size of the vocabulary.
    • set_encode_special_tokens(bool): Configures whether to add special tokens during encoding (supported by HuggingFace, ignored by Tiktoken).
    // Conceptual usage of UnifiedTokenizer
let ids = unified_tokenizer.encode("Hello world", true)?;
let text = unified_tokenizer.decode(&ids, false)?;

  10. The Encoding data structure

    main

    When using EncodeWithOptions, the result is returned as an Encoding struct containing:

    • IDs []uint32: The numerical token IDs.
    • TypeIDs []uint32: Token type IDs (used in models like BERT).
    • SpecialTokensMask []uint32: A mask indicating which tokens are special tokens.
    • AttentionMask []uint32: A mask for the attention mechanism.
    • Tokens []string: The string representation of each token.
    • Offsets []Offset: The start and end character positions in the original string for each token. An Offset is a [2]uint array where [0] is the start and [1] is the end.

  11. Encode and decode text with tokenizers

    main

    The library provides methods to convert text into token IDs (encoding) and convert token IDs back into text (decoding).

    • Encode(text string, addSpecialTokens bool): Returns the IDs and tokens. Use true for the second argument to include special tokens (like [CLS] or [SEP]).
    • Decode(ids []uint32, skipSpecialTokens bool): Converts a slice of IDs back to a string.
    • VocabSize(): Returns the size of the vocabulary.

    For explicit error handling during these operations, use EncodeErr, EncodeWithOptionsErr, or DecodeErr instead of the standard methods.

    import "github.com/daulet/tokenizers"
import "fmt"

// ... (tokenizer initialization) ...

fmt.Println("Vocab size:", tk.VocabSize())

// Basic encoding (without special tokens)
fmt.Println(tk.Encode("brown fox jumps over the lazy dog", false))

// Encoding with special tokens
fmt.Println(tk.Encode("brown fox jumps over the lazy dog", true))

// Decoding tokens back to text
fmt.Println(tk.Decode([]uint32{2829, 4419, 14523, 2058, 1996, 13971, 3899}, true))

  12. Load a tokenizer from a file or HuggingFace

    main

    You can initialize a tokenizer using a local JSON configuration file or by pulling a pre-trained model directly from HuggingFace.

    Important: Always call tk.Close() to release the underlying native resources when the tokenizer is no longer needed.

    import "github.com/daulet/tokenizers"

// Load from a local JSON config
tk, err := tokenizers.FromFile("./data/bert-base-uncased.json")
if err != nil {
    return err
}
defer tk.Close()

// Load from HuggingFace
tk, err := tokenizers.FromPretrained("google-bert/bert-base-uncased")
if err != nil {
    return err
}
defer tk.Close()