tiktoken-go/tokenizer

To use the tokenizer in your Go application, use tokenizer.Get() to retrieve an encoding instance (e.g., tokenizer.Cl100kBase). Once you have an encoding instance, you can use .Encode(text) to convert a string into a slice of token IDs, and .Decode(ids) to convert token IDs back into a string.

Note that Encode returns three values: the slice of IDs, a slice of special tokens (if any), and an error. Decode returns the decoded string and an error.

package main

import (
    "fmt"
    "github.com/tiktoken-go/tokenizer"
)

func main() {
    enc, err := tokenizer.Get(tokenizer.Cl100kBase)
    if err != nil {
        panic("oh oh")
    }

    // this should print a list of token ids
    ids, _, _ := enc.Encode("supercalifragilistic")
    fmt.Println(ids)

    // this should print the original string back
    text, _ := enc.Decode(ids)
    fmt.Println(text)
}

The tokenizer CLI tool supports the following flags:

• -decode string: tokens to decode
• -encode string: text to encode
• -token string: text to calculate token

Usage of tokenizer:
  -decode string
        tokens to decode
  -encode string
        text to encode
  -token string
        text to calculate token

This example demonstrates the full workflow: obtaining a tokenizer via Get, encoding a string into token IDs, and decoding those IDs back into the original text.

package main

import (
	"fmt"
	"github.com/tiktoken-go/tokenizer"
)

func main() {
	enc, err := tokenizer.Get(tokenizer.Cl100kBase)
	if err != nil {
		panic("oh oh")
	}

	// this should print a list of token ids
	ids, _, _ := enc.Encode("supercalifragilistic")
	fmt.Println(ids)

	// this should print the original string back
	text, _ := enc.Decode(ids)
	fmt.Println(text)
}

The library includes a command-line tool for encoding, decoding, or calculating token counts from the terminal. Use the -h flag to view available commands.

> tokenizer -encode supercalifragilistic

Use tokenizer.ForModel(model Model) to retrieve the correct Codec for a specific OpenAI model. This abstracts away the underlying encoding format used by that model.

Supported Model examples include:

• O1, O1Preview, O1Mini, O3, O3Mini, O4Mini (use O200kBase)
• GPT4, GPT35, GPT35Turbo, TextEmbeddingAda002 (use Cl100kBase)
• TextDavinci003, CodeDavinci001, etc. (use P50kBase)
• GPT2 (use GPT2Enc)

enc, err := tokenizer.ForModel(tokenizer.GPT4o)
if err != nil {
    // handle error
}

Use tokenizer.Get(encoding Encoding) to retrieve a Codec implementation for a specific encoding scheme. This is useful if you know the exact encoding format required (e.g., Cl100kBase).

Supported Encoding values:

• O200kBase
• Cl100kBase
• R50kBase
• P50kBase
• P50kEdit
• GPT2Enc (via GPT2 model mapping)

enc, err := tokenizer.Get(tokenizer.Cl100kBase)
if err != nil {
    // handle error
}

The Codec interface is the primary way to interact with tokenizers in this package. It allows you to convert strings to token IDs (encoding) and convert token IDs back into strings (decoding). You can also use it to count the number of tokens in a string without performing a full encoding.

// Example of using a Codec instance
ids, tokens, err := enc.Encode("hello world")
text, err := enc.Decode(ids)
count, err := enc.Count("hello world")

The following flags are available for the tokenizer CLI tool:

The following encoding formats are explicitly supported by the Get function:

const (
	GPT2Enc    Encoding = "gpt2"
	R50kBase   Encoding = "r50k_base"
	P50kBase   Encoding = "p50k_base"
	P50kEdit   Encoding = "p50k_edit"
	Cl100kBase Encoding = "cl100k_base"
	O200kBase  Encoding = "o200k_base"
)

The following errors are returned when a requested model or encoding is not available:

var (
	ErrModelNotSupported    = errors.New("model not supported")
	ErrEncodingNotSupported = errors.New("encoding not supported")
)