DOCSEARCHbyAlgolia
Submit Docs

SentencePiece

repository·master·Indexed 11 days ago

https://github.com/google/sentencepiece

An unsupervised text tokenizer and detokenizer designed for neural network-based text generation systems like LLMs. It supports subword units such as BPE and Unigram, working directly on raw Unicode sequences without language-specific preprocessing. Version 0.2.3 includes a Python wrapper with the SentencePieceProcessor for encoding/decoding, a trainer for model creation, and an optional high-performance NLCodec for fast BPE training.

Tokens
33.3K
Snippets
98
Records
122
Agent score
91%

What's inside SentencePiece

  1. Handle BOS/EOS token errors for USER_DEFINED symbols

    master

    In versions since v0.2.2, SentencePiece enforces strict verification for BOS (Beginning of Sentence) and EOS (End of Sentence) tokens.

    If you attempt to add BOS or EOS tokens during encoding (e.g., using add_bos=True or add_eos=True in Python) but the tokens are defined as USER_DEFINED or any type other than CONTROL, the operation will fail.

    • Python: Passing add_bos=True or add_eos=True to encode() will raise a ValueError.
    • C++: SetEncodeExtraOptions("bos") or SetEncodeExtraOptions("eos") will return an error.

  2. Understand vocabulary size and special symbols

    master

    Special symbols (including default symbols like <s>, </s>, UNK, and custom ones) occupy slots in your vocabulary.

    If you set a vocab_size (e.g., 32000) and define a number of special symbols (e.g., 100), the remaining slots available for subwords learned from the training corpus will be vocab_size - num_special_symbols (e.g., 31900).

  3. Prevent prompt injection using Control Symbols

    master

    SentencePiece provides built-in protection against prompt injection (jailbreaking) by ensuring that Control Symbols cannot be parsed from raw user input.

    If a malicious user attempts to inject a control sequence like </s> to end a sequence prematurely, SentencePiece will treat the characters < / s > as normal text pieces rather than the special </s> control token. This prevents users from hijacking the model's control flow. To use control symbols, your application must insert their specific IDs into the token stream after the user text has been tokenized.

  4. How SentencePiece handles vocabulary piece constraints

    master

    SentencePiece is designed to operate directly on raw, un-pretokenized text. Instead of using an external pre-tokenizer at runtime (inference), it applies piece constraints during training to define which subwords are valid candidates for the vocabulary.

    This approach ensures:

    • Language Independence: No need for external morphological analyzers (like MeCab or Jieba).
    • Security: Avoids ReDoS (Regular Expression Denial of Service) risks during inference because no regex is run at runtime.
    • Portability: The model is self-contained and behaves identically across different environments/runtimes.

    Constraints applied during training act as hard boundaries that prevent subwords from crossing certain markers (like script changes or whitespace). These boundaries are baked into the model, so no external splitter is required during encode.

  5. Understand the difference between Control and User-Defined symbols

    master

    SentencePiece distinguishes between two types of special symbols to manage model control flow and security:

    Control Symbols

    • Purpose: Guide decoder or model control flow (e.g., <s>, </s>).
    • Encoding: They are never recognized if they appear in raw input text. If a user inputs <control1>, SentencePiece will tokenize it as normal text (splitting it into characters) rather than the special token ID.
    • Decoding: Decodes to an empty string "".
    • Usage: Must be inserted programmatically by your application logic into the token sequence (e.g., [BOS_ID] + tokenize(input) + [EOS_ID]).

    User-Defined Symbols

    • Purpose: Treat specific strings as single, indivisible tokens (e.g., HTML tags, emojis, or domain-specific tokens).
    • Encoding: If the symbol (e.g., <user1>) appears in the input text, it is guaranteed to be tokenized as that single token.
    • Decoding: Decodes back to its original surface string (e.g., <user1>).

  6. Inspect control symbols in decoded output

    master

    By design, SentencePieceProcessor.decode maps CONTROL symbols to empty strings. They are intended for model control flow and do not appear in the final text output.

    To inspect or verify control symbols, you must either look at the token IDs directly or convert them to pieces individually using id_to_piece(id).

    # If sp.decode([14, 6, 3, 6, 24]) -> "hello world"
# Use id_to_piece to see the control symbols:
pieces = [sp.id_to_piece(i) for i in [14, 6, 3, 6, 24]]
# pieces -> [' hello', ' ', '<control1>', ' ', 'world']

  7. Understand SentencePiece performance scaling and GIL bottlenecks

    master

    When using SentencePiece in Python, performance scales via multi-threading in the underlying C++ implementation, but it is subject to a non-linear scaling bottleneck due to the Python Global Interpreter Lock (GIL).

    How Scaling Works

    1. Parallel Core Execution: The core tokenization algorithms run in parallel and release the GIL, allowing efficient utilization of multiple CPU cores.
    2. Sequential Python Conversion: Once the native threads complete, the resulting native arrays (e.g., std::vector<std::vector<int>>) must be converted into Python objects (e.g., list[list[int]]).
    3. The Bottleneck: This conversion step must occur on the Python main thread and requires the GIL. Consequently, adding more threads provides diminishing returns as the time spent in the sequential conversion step becomes the dominant factor.

    Performance Advantage

    SentencePiece typically outperforms Hugging Face in Python environments because it converts directly to nested lists of raw integers, whereas Hugging Face instantiates heavier Python Encoding wrapper objects for each sentence.

  8. Use offset mapping for character or byte alignment

    master

    To align tokens with the original text (e.g., for highlighting), use return_type='offset_mapping'.

    Unicode Character Offsets

    By default, offsets are Unicode character indices. You can slice the original string using these (start, end) tuples.

    Raw Byte Offsets

    To get byte-level offsets and pieces, use the return_bytes=True parameter (only valid when return_type='offset_mapping'). This is useful for binary protocols or bypassing Unicode overhead.

    Byte Fallback Behavior

    If byte_fallback=True is enabled in the model, unknown characters are decomposed into UTF-8 byte tokens. The first $N-1$ byte tokens are assigned a zero-width span (start, start), and the final token is assigned the full span of the character. This ensures slicing remains valid.

    import sentencepiece as spm

sp = spm.SentencePieceProcessor(model_file='test/test_model.model')
text = "吾輩は猫である。"

# Encoding with Unicode offsets
enc_res = sp.encode(text, return_type='offset_mapping')
for piece, (start, end) in zip(enc_res['pieces'], enc_res['offsets']):
    print(f"Piece: {piece} -> Surface: {text[start:end]}")

# Decoding with raw byte offsets
dec_res_bytes = sp.decode(enc_res['ids'], return_type='offset_mapping', return_bytes=True)
print(dec_res_bytes['text']) # Reconstructed text as bytes

  9. Thread-safety and Free-Threading (NoGIL) support

    master

    SentencePiece provides experimental support for free-threaded (NoGIL) Python environments (introduced in v0.2.1).

    Thread-safety rules:

    • Thread-safe methods: const and static methods such as encode(), decode(), and train() are designed to work in a free-threaded environment.
    • Non-thread-safe methods: Non-const methods like load() are not thread-safe and can cause data races. You must implement your own locking mechanism when calling these from multiple threads.

  10. Difference between user_defined_symbols and required_chars

    master

    When configuring your model, it is important to distinguish between these two types of special symbols:

    • User-defined symbols: Treated as a single, indivisible token. They are never split into smaller pieces and are always matched from the input text if present.
    • Required characters (specified via --required_chars): Forced to be in the model's alphabet (preventing them from being mapped to UNK), but they are treated as normal characters during training and can be split or merged into larger subwords.

  11. How SentencePiece achieves lossless tokenization

    master

    SentencePiece treats input text as a raw sequence of Unicode characters and escapes whitespaces with a meta-symbol (U+2581). Because whitespace is treated as a regular symbol, the tokenization process is reversible and lossless.

    To reconstruct the original text from pieces, you can join the pieces and replace the meta-symbol with a standard space:

    original_text = "".join(pieces).replace(" ", " ")
    # Lossless detokenization
original_text = "".join(pieces).replace(" ", " ")