TokenLens

TokenLens provides utilities to manage model metadata, context limits, and cost estimation. You can fetch model providers and then use helpers to calculate context caps, costs, or combined usage data.

Note that ID normalization is automatic: you can use provider/model, provider:id, or just the model name. Version dots are automatically normalized to dashes (e.g., claude-3.5-haiku becomes claude-3-5-haiku).

import { fetchModels } from 'tokenlens';
import { getContext, getTokenCosts, getUsage } from '@tokenlens/helpers';

// 1) Get providers
const openai = await fetchModels('openai');

// 2) Define model and usage
const modelId = 'openai/gpt-4o-mini';
const usage = { prompt_tokens: 3200, completion_tokens: 400 };

// Context caps
const { maxInput, maxOutput, maxTotal } = getContext(modelId, openai);

// Cost breakdown
const { 
  inputUSD, 
  outputUSD, 
  totalUSD 
} = getTokenCosts(modelId, usage, openai);

// Or both together
const usageData = getUsage(modelId, usage, openai);

console.log({ maxInput, maxOutput, maxTotal, inputUSD, outputUSD, totalUSD, usageData });

The @tokenlens/helpers package provides utilities for calculating context window limits and estimating token costs.

Dependency Injection Pattern: These helpers are DI-first. You must pass a providers object to the functions. You can obtain this object by calling fetchModels() from the tokenlens package (for dynamic provider data) or getModels() (for a static set).

Core Functions:

• getContext({ modelId, providers }): Returns context window limits.
• getTokenCosts({ modelId, usage, providers }): Returns estimated costs for input, output, reasoning, and caching.
• getUsage({ modelId, usage, providers, reserveOutput? }): Returns a combined usage summary.

Model ID Formats: Model IDs can be specified as provider/model, provider:id, or a providerless model. Note that version dots (e.g., 1.5) are normalized to dashes (e.g., 1-5) in the model segment.

import { fetchModels } from 'tokenlens';
import { getContext, getTokenCosts, getUsage } from '@tokenlens/helpers';

const openai = await fetchModels('openai');
const modelId = 'openai/gpt-4o-mini';
const usage = { prompt_tokens: 1000, completion_tokens: 500 };

// Get context window limits
const { maxInput, maxOutput, maxTotal } = getContext(modelId, openai);

// Get estimated costs
const {
  inputUSD,
  outputUSD,
  reasoningUSD,
  cacheReadUSD,
  cacheWriteUSD,
  totalUSD,
} = getTokenCosts(modelId, usage, openai);

// Get combined usage summary
const summary = getUsage(modelId, usage, openai);

TokenLens provides types to handle usage data from various SDKs and providers. You can work with raw usage shapes or normalized structures.

• UsageLike: A partial record of common token field names (e.g., prompt_tokens, completionTokens, totalTokens) used to ingest data from different provider SDKs.
• NormalizedUsage: A provider-agnostic structure used by helpers, containing input, output, and an optional total.
• TokenBreakdown: An extended version of NormalizedUsage that includes cacheReads, cacheWrites, and reasoningTokens for more granular analysis.

export type UsageLike = Partial<
  Record<
    | "prompt_tokens"
    | "completion_tokens"
    | "total_tokens"
    | "input_tokens"
    | "output_tokens"
    | "reasoning_tokens"
    | "promptTokens"
    | "completionTokens"
    | "totalTokens"
    | "reasoningTokens",
    number
  >
>;

export type NormalizedUsage = {
  input: number;
  output: number;
  total?: number;
};

export type TokenBreakdown = NormalizedUsage & {
  cacheReads?: number;
  cacheWrites?: number;
  reasoningTokens?: number;
};