ttok

ttok can count tokens for text provided as command-line arguments or via standard input (stdin).

Using arguments

ttok Hello world

Using stdin

When piping text, use echo -n to prevent adding a newline character, which would otherwise increase the token count.

echo -n "Hello world" | ttok

Appending text to piped input

To pipe in text and then append additional tokens from arguments, use the -i - option:

echo -n "Hello world" | ttok more text -i -

You can install ttok using pip or Homebrew.

Using pip

pip install ttok

Using Homebrew

brew install simonw/llm/ttok

ttok provides utilities to inspect the integer token IDs used by models.

View integer token IDs

Use --encode to see the integer IDs for the input text:

ttok Hello world --encode

View detailed token breakdown

Add the --tokens flag to see the actual byte strings for each token:

ttok Hello world --encode --tokens

Convert token IDs back to text

Use the --decode option to reverse the process:

ttok 9906 1917 --decode

The ttok command-line tool allows you to count tokens in text or truncate text to a specific token limit. You can provide text as positional arguments or via stdin using the --input flag or a pipe.

Basic Usage

Count tokens from arguments:

ttok one two three

Count tokens from stdin:

cat input.txt | ttok

Truncate text to a specific token count:

cat input.txt | ttok -t 100

Truncate using a specific model:

cat input.txt | ttok -t 100 -m gpt2

The following options are available for the ttok command:

• --version: Show the version and exit.
• -i, --input FILENAME: Specify an input file.
• -t, --truncate INTEGER: Truncate to this many tokens.
• -m, --model TEXT: Which model to use.
• --encode, --tokens: Output token integers.
• --decode: Convert token integers to text.
• --tokens: Output full tokens.
• --allow-special: Do not error on special tokens.
• --help: Show this message and exit.

You can use ttok to inspect the underlying tokenization of a string.

To view token integers: Use --encode to see the integer representation of the tokens.

cat input.txt | ttok --encode

To see the details of the tokens (as bytes): Use --tokens to see the actual token content (e.g., [b'hello', b' world']).

ttok "hello world" --tokens

To convert token integers back to text: Provide a string of integers and use the --decode flag.

ttok 9906 1917 --decode

By default, ttok uses the tokenizer model for GPT-3.5 and GPT-4. You can specify different models using the -m or --model option. For example, to use the GPT-2 model:

ttok boo Hello there this is -m gpt2

Use the -t or --truncate option to limit the output text to a specific number of tokens.

ttok This is too many tokens -t 3

The following options are available for the ttok command:

Constraints

• You cannot use --decode and --encode together.
• --allow-special requires either --encode or --tokens to be set.

ttok --help