DOCSEARCHbyAlgolia
Submit Docs

huggingface/tokenizers

repository·main·Indexed 27 days ago

https://github.com/huggingface/tokenizers

A high-performance library implemented in Rust for training and using state-of-the-art tokenizers in research and production environments. It provides bindings for Python and NodeJS, supporting various models including BPE, WordPiece, and Unigram. The library includes specialized classes like CharBPETokenizer and BertWordPieceTokenizer, and supports free-threaded CPython (3.14t) for concurrent encoding.

Tokens
21K
Snippets
39
Records
180
Agent score
94%

What's inside tokenizers

  1. Overview of Hugging Face Tokenizers

    main
    Hugging Face Tokenizers is a high-performance library designed for both research and production environments. It provides implementations of modern, widely-used tokenizers used in the 🤗 Transformers ecosystem. The library is written in Rust, enabling extremely fast training and tokenization (e.g., processing 1GB of text in less than 20 seconds on a server CPU).

  2. Overview of tokenizers features and bindings

    main

    The tokenizers library provides high-performance implementations of common tokenization algorithms (BPE, WordPiece, Unigram) designed for both research and production.

    Key Features:

    • Extremely fast Rust-based implementation (can tokenize ~1GB of text in <20s on server CPU).
    • Supports training new vocabularies.
    • Normalization includes alignment tracking to map tokens back to original sentence spans.
    • Includes pre-processing capabilities like Truncation, Padding, and special token handling.

    Supported Language Bindings:

    • Rust (Original implementation)
    • Python
    • Node.js
    • Ruby (via external tokenizers-ruby repository)

  7. Understand the Tokenization Pipeline

    main

    When calling Tokenizer.encode or Tokenizer.encode_batch, the input text undergoes a four-step pipeline:

    1. Normalization: Cleaning the raw string (e.g., stripping whitespace, lowercasing).
    2. Pre-Tokenization: Splitting text into smaller objects (like words) to set an upper bound for final tokens.
    3. Model: Splitting pre-tokens into sub-tokens and mapping them to vocabulary IDs.
    4. Post-Processing: Adding special tokens or performing final transformations on the Encoding.

    You can also use decode to convert token IDs back into text.

  9. Key Features of Tokenizers

    main

    The library provides several core capabilities for NLP workflows:

    • Vocabulary Training & Tokenization: Train new vocabularies and tokenize text using state-of-the-art algorithms.
    • High Performance: Optimized Rust implementation for speed.
    • Alignment Tracking: Full support for alignment tracking, allowing you to map any token back to its corresponding part in the original sentence, even after destructive normalization.
    • Pre-processing: Built-in support for truncation, padding, and adding special tokens required by specific models.

  10. Understand the Tokenizer pipeline

    main

    A Tokenizer operates as a pipeline that processes raw text into an Encoding. The pipeline consists of four main stages:

    1. Normalizer: Normalizes the input text (e.g., applying Unicode normalization like NFD or NFKC).
    2. PreTokenizer: Performs initial splits of the text, such as splitting on whitespace.
    3. Model: Performs the actual tokenization logic (e.g., BPE or WordPiece).
    4. PostProcessor: Post-processes the Encoding to add necessary elements like special tokens required by language models.