YouTokenToMe Documentation

YouTokenToMe is designed for high-speed BPE training and tokenization, particularly for languages with large alphabets. It supports multithreading for both training and tokenization processes.

Threading Behavior

• Training: Performance scaling for training typically plateaus after 8 threads. The library effectively uses min(8, n_threads) for training operations.
• Tokenization: Scales more linearly with thread count compared to training.

Recommended Configuration

To allow the library to automatically determine the optimal number of threads for your hardware, use n_threads=-1.

Install the youtokentome package using pip to access the BPE tokenizer via Python or the CLI.

pip install youtokentome

The BPE class provides methods to inspect the learned vocabulary:

• vocab(): Returns a list of vocab_size strings representing the subwords.
• vocab_size(): Returns the total number of tokens in the vocabulary.
• subword_to_id(subword): Returns the integer ID for a given subword string, or unk_id if not found.
• id_to_subword(id): Returns the subword string corresponding to a specific integer ID.

The encode method converts a list of strings into tokens. You can choose between integer IDs or string subwords.

Arguments:

• sentences (list[str]): The input sentences.
• output_type (yttm.OutputType): Use yttm.OutputType.ID for integer IDs or yttm.OutputType.SUBWORD for subword strings.
• bos (bool): If True, add the 'beginning of sentence' token.
• eos (bool): If True, add the 'end of sentence' token.
• reverse (bool): If True, reverse the output sequence.
• dropout_prob (float): BPE-dropout probability [0, 1].

Returns: A list of lists (either list[list[int]] or list[list[str]]).

import youtokentome as yttm

bpe = yttm.BPE(model="model.bin")

# Encode to IDs
ids = bpe.encode(["hello world"], output_type=yttm.OutputType.ID)

# Encode to Subwords
subwords = bpe.encode(["hello world"], output_type=yttm.OutputType.SUBWORD)

Convert a list of token IDs back into a concatenated string using the decode method.

import youtokentome as yttm

bpe = yttm.BPE(model="model.bin")

# ids is a list of lists of integers
decoded_text = bpe.decode([[10, 25, 3], ignore_ids=[1, 2]])

Initialize a youtokentome.BPE object by passing the path to a previously trained model file.

import youtokentome as yttm

bpe = yttm.BPE(model="example.model", n_threads=-1)

Options for the encode command:

• --model PATH (required): Path to the learned model file.
• --output_type TEXT (required): Either id or subword.
• --n_threads INTEGER: Number of threads [default: -1].
• --bos: Add 'begin of sentence' token.
• --eos: Add 'end of sentence' token.
• --reverse: Reverse the output sequence of tokens.
• --stream: Process each line one by one (ignores --n_threads).
• --dropout_prob: BPE-dropout probability [default: 0].

yttm encode --help

Options for the decode command:

• --model PATH (required): Path to the learned model file.
• --ignore_ids LIST: List of indices to ignore (e.g., --ignore_ids=1,2,3).

yttm decode --help

Options for the vocab command:

• --model PATH (required): Path to the learned model file.
• --verbose: Add merging rules to the output.

yttm vocab --help

Options for the bpe command to train a model:

• --data PATH (required): Training data file path.
• --model PATH (required): Output model file path.
• --vocab_size INTEGER (required): Number of tokens in the final vocabulary.
• --coverage FLOAT: Fraction of characters covered [default: 1.0].
• --n_threads INTEGER: Number of threads [default: -1].
• --pad_id INTEGER: Padding token id [default: 0].
• --unk_id INTEGER: Unknown token id [default: 1].
• --bos_id INTEGER: 'Begin of sentence' token id [default: 2].
• --eos_id INTEGER: 'End of sentence' token id [default: 3].

yttm bpe --help

When using YouTokenToMe, you can control the parallelism of the training and tokenization processes using the n_threads parameter.

• Set n_threads to a specific integer to use that exact number of threads.
• Set n_threads=-1 to enable automatic thread detection, which is recommended for most use cases.