Token-Oriented Object Notation (TOON)

TOON is a compact, human-readable encoding of the JSON data model designed specifically for LLM prompts. It aims to reduce token consumption while maintaining or improving retrieval accuracy compared to standard JSON.

Key characteristics include:

• Token Efficiency: Uses significantly fewer tokens than JSON (e.g., ~42.6% reduction in benchmarks).
• JSON Compatibility: Encodes the same objects, arrays, and primitives as JSON with deterministic, lossless round-trips.
• LLM Guardrails: Uses explicit [N] lengths and {fields} field lists in headers to provide models with a clear schema, improving parsing reliability.
• Minimal Syntax: Employs indentation instead of braces and minimizes quoting, combining YAML-like readability with CSV-style compactness.
• Tabular Forms: Optimizes uniform arrays of objects into tables that declare field lists once and stream row values line by line.

TOON (Token-Oriented Object Notation) is a compact, human-readable encoding of the JSON data model designed to minimize tokens for LLM inputs.

When to use TOON:

• Uniform arrays of objects: Data where items share the same structure. TOON uses tabular forms to declare fields once and stream data, significantly reducing token overhead.
• LLM Prompts: It provides deterministic structure, explicit array lengths ([N]), and field lists ({fields}) which act as validation guardrails to detect truncation or malformed model output.

When NOT to use TOON:

• Deeply nested or non-uniform structures: Standard JSON is often more token-efficient for complex, irregular configurations.
• Pure tabular data: CSV is more compact for flat tables; TOON adds ~5-10% overhead to provide structural metadata.
• Latency-critical applications: While TOON reduces tokens, some models may process compact JSON faster. Always benchmark your specific environment.

TOON is designed to be more byte-efficient (and thus more token-efficient for LLMs) than compact JSON in specific structural patterns.

Best use cases for TOON:

• Simple Objects: Flat objects with primitive values. TOON avoids braces and quotes on keys/simple values.
• Primitive Arrays: Arrays of strings or numbers. TOON uses a single length marker and omits most quotes.
• Tabular Arrays (Sweet Spot): Uniform arrays of objects. TOON declares the field list once and streams rows as bare values, avoiding repeated keys.
• Root Arrays: Arrays at the top level of the document.

When JSON may be more efficient:

• Arrays of Arrays: TOON pays a fixed overhead per inner array (e.g., "- " and "[m]: "), whereas JSON uses simple commas.
• Deeply Nested Configurations: Each level of nesting in TOON adds indentation (2 spaces per level), which can eventually exceed the cost of JSON braces.
• Low Tabularity: If data does not follow a uniform schema, the overhead of TOON's structural markers may not be offset by savings.

Arrays always declare their length in brackets [N]. TOON chooses the most efficient representation based on the content.

Primitive Arrays (Inline Form)

Used for arrays of strings, numbers, booleans, or null.

tags[3]: admin,ops,dev

Arrays of Objects (Tabular Form)

Used when all objects in an array share the same set of primitive-valued keys. The header defines the length [N] and the field names {field1,field2}.

items[2]{sku,qty,price}:
  A1,2,9.99
  B2,1,14.5

Nested Field Groups in Tabular Arrays

You can fold uniform sub-objects into the header using curly braces. This keeps rows flat while representing nested data.

orders[2]{id,customer{name,country},total}:
  1,Ada,DK,99

Mixed and Non-Uniform Arrays (List Form)

If an array contains mixed types or non-uniform objects, it uses the hyphenated list form.

items[3]:
  - 1
  - a: 1
  - text

Empty Arrays

Empty arrays are represented as key: [] or [] at the root.

The accuracy benchmark uses two types of datasets to test LLM comprehension:

Primary Datasets

Eight datasets covering various shapes to test tabular eligibility:

• Employee records: Uniform objects (best case for tabular).
• E-commerce orders: Nested objects and arrays.
• Time-series analytics: Dates and numeric values.
• GitHub repositories: Real-world data with long strings.
• Event logs: Semi-uniform data (flat and nested).
• Nested config: Deep nesting (worst case for tabular).
• Feature flags: Map of uniform objects (tests keyed tabular form).
• Contacts: Uniform records with nested groups (tests nested field groups).

Structural Validation Datasets

Five variants of a 20-row dataset used to test if models can detect corruption in the encoded text. Unlike JSON/YAML/XML, TOON's declared length ([N]) allows it to detect:

• Truncated: Last 3 rows removed.
• Extra rows: 3 rows appended past the declared [20].
• Width mismatch: A cell dropped from a row.
• Missing fields: Values removed from records.