tokenx

You can install tokenx using your preferred package manager to use it in your TypeScript or JavaScript projects.

# npm
npm install tokenx

# pnpm
pnpm add tokenx

# yarn
yarn add tokenx

You can tune the heuristic estimation by passing a TokenEstimationOptions object to the API functions. This is useful for different LLM families or specific language requirements.

Options:

• defaultCharsPerToken (number): The default average characters per token when no language-specific rule applies. Defaults to 7.
• languageConfigs (Array<LanguageConfig>): Custom rules to override defaults. These are checked before built-in heuristics.

LanguageConfig structure:

• pattern (RegExp): A regular expression to detect the language.
• averageCharsPerToken (number): The character-to-token ratio for that pattern.

interface TokenEstimationOptions {
  defaultCharsPerToken?: number
  languageConfigs?: LanguageConfig[]
}

interface LanguageConfig {
  pattern: RegExp
  averageCharsPerToken: number
}

The tokenx CLI accepts input through file paths or standard input (stdin).

• Files: Provide one or more file paths as arguments. The CLI will read each file using UTF-8 encoding and use the relative path as a label.
• Standard Input (stdin): Use a single dash - as an argument, or provide no arguments at all, to read from stdin.
• Constraint: You cannot mix file paths and stdin in a single command. Attempting to provide both will result in a CliError.

Input is processed into InputDocument objects containing a label (the file path or 'stdin') and the text content.

# Example: Reading from stdin
echo "hello world" | tokenx

# Example: Reading from a specific file
tokenx path/to/file.txt

# Example: Reading multiple files
tokenx file1.txt file2.txt

The tokenx CLI uses specific exit codes to communicate the result of token estimation:

• 0: Success (the input was processed normally).
• 1: General failure (e.g., file not found, invalid arguments).
• 2: The input exceeded the specified --limit (ran fine, but the input is over the budget).

Use estimateTokenCount to get a heuristic estimate of the number of tokens in a string. The estimates are calibrated against OpenAI's o200k_base encoding. For high-precision requirements, consider a full tokenizer like gpt-tokenizer.

import { estimateTokenCount } from 'tokenx'

const estimatedTokens = estimateTokenCount('Hello, world!')

// With custom options
const customEstimate = estimateTokenCount('Bonjour le monde!', {
  defaultCharsPerToken: 4,
  languageConfigs: [
    { pattern: /[éèêëàâîï]/i, averageCharsPerToken: 3 }
  ]
})

Extract a portion of text based on token positions, behaving similarly to Array.prototype.slice(). It supports both positive and negative indices.

import { sliceByTokens } from 'tokenx'

const text = 'Hello, world! This is a test sentence.'

const firstThree = sliceByTokens(text, 0, 3)
const fromSecond = sliceByTokens(text, 2)
const lastTwo = sliceByTokens(text, -2)
const middle = sliceByTokens(text, 1, -1)

Use isWithinTokenLimit to determine if a string's estimated token count falls below a specific threshold.

import { isWithinTokenLimit } from 'tokenx'

const withinLimit = isWithinTokenLimit('Check this text against a limit', 100)

// With custom options
const customCheck = isWithinTokenLimit('Text', 50, { defaultCharsPerToken: 3 })

Split text into an array of chunks based on a target token count. This is useful for RAG (Retrieval-Augmented Generation) or batch processing.

Important details:

• tokensPerChunk is a target, not a hard maximum. A chunk closes once it reaches the target, but a single segment (split by whitespace/punctuation) can push a chunk beyond the target.
• Chunks never break words apart.
• Use the overlap option to maintain semantic continuity between consecutive chunks.

import { splitByTokens } from 'tokenx'

const text = 'Long text that needs to be split into smaller chunks...'

// Basic splitting
const chunks = splitByTokens(text, 100)

// With overlap for semantic continuity
const overlappedChunks = splitByTokens(text, 100, { overlap: 10 })

// With custom options
const customChunks = splitByTokens(text, 50, {
  defaultCharsPerToken: 4,
  overlap: 5
})

The tokenx binary allows you to count tokens, enforce limits, and chunk documents directly from your terminal. You can run it via npx without installation or install it globally.

Key behaviors:

• Only results (counts as integers, chunks as JSON arrays) are sent to stdout.
• Notices and errors are sent to stderr.
• Use tokenx --help for all available options.

# Count tokens in a file, a pipe, or several files at once
tokenx README.md
cat article.md | tokenx
tokenx count src/*.ts

# Fail a script when a prompt outgrows its budget (exit code 2)
tokenx prompt.txt --limit 8000

# Extract a token range, or chunk a document for RAG
tokenx slice article.md --end 500
tokenx split article.md --size 500 --overlap 50

When using SplitByTokensOptions to chunk documents for RAG or other purposes, you can specify an overlap value. This defines the number of tokens that will overlap between consecutive chunks. The value is automatically clamped to ensure it does not exceed the target chunk size.

const splitOptions: SplitByTokensOptions = {
  defaultCharsPerToken: 7,
  overlap: 50
};

To handle different languages or formats with varying token densities, use LanguageConfig. You define a pattern (a RegExp) used to detect the language in a text block, and an averageCharsPerToken value that applies to that detected language.

const languageConfigs: LanguageConfig[] = [
  {
    pattern: /\w+/,
    averageCharsPerToken: 4
  }
];