DOCSEARCHbyAlgolia
Submit Docs

TONL (Token-Optimized Notation Language)

repository·main·Indexed 21 days ago

https://github.com/tonl-dev/tonl

A text-first, LLM-friendly serialization format designed to reduce JSON size and token costs. Version 2.5.2 provides a data platform with a rich API for querying, modifying, and optimizing data, featuring high-performance caching, comprehensive validation, streaming, and browser support. Includes the TONLDocument class for JSONPath-like queries, TSL (TONL Schema Language) for validation, and a CLI for encoding and decoding files.

Tokens
94.8K
Snippets
322
Records
411
Agent score
73%

What's inside tonl

  1. Overview of TONL Browser Examples

    main

    The examples/browser directory contains several demonstration files that showcase different integration patterns:

    • 01-basic-usage.html: Demonstrates basic JSON/TONL conversion and displays compression statistics.
    • 02-document-api.html: Demonstrates the TONLDocument API, including query operations (wildcards, filters, recursive), modification operations (set, push, delete), and document statistics.
    • 03-react-example.html: A full React 18 application demonstrating CRUD operations and real-time TONL output.
    • 04-vue-example.html: A Vue 3 task manager application demonstrating task CRUD, filtering, and real-time TONL sync.

  2. Overview of TONL CLI commands

    main

    The TONL CLI provides eight main commands for managing TONL data:

    • tonl encode: Convert JSON to TONL with optimization options
    • tonl decode: Convert TONL to JSON
    • tonl format: Format and prettify TONL files
    • tonl validate: Validate TONL data against schema
    • tonl generate-types: Generate TypeScript types from schema
    • tonl stats: Analyze and compare data formats
    • tonl query: Query TONL files with JSONPath expressions
    • tonl get: Get specific values from TONL files (alias for query)

  3. TONL Implementation Roadmap

    main

    The development of the TONL Schema Language (TSL) is organized into the following phases:

    • Phase 1: Parser (v0.4.0): Focuses on schema format design, file parsing, basic type validation, and error reporting.
    • Phase 2: Constraints (v0.4.0): Implementation of string (min, max, pattern), numeric (min, max, range), and array (min, max, unique) constraints, plus custom validators.
    • Phase 3: Code Generation (v0.4.0): Focuses on TypeScript type generation, JSDoc annotations, runtime validators, and JSON Schema export.
    • Phase 4: Advanced (v0.5.0+): Includes schema evolution/migration, conditional validation, references/relationships, and IDE integration (VS Code extension).

  4. What is TONL and its core design principles

    main

    TONL (Token-Optimized Notation Language) is a text-based serialization format designed for high token efficiency in LLM contexts (32-45% smaller than JSON), human readability, and bidirectional compatibility with JSON.

    Core design principles include:

    • Tabular for uniform arrays: Reduces redundancy for arrays of similar objects.
    • Nested blocks for objects: Uses clear hierarchy for nested structures.
    • Smart delimiter selection: Minimizes the need for quoting.
    • Optional type hints: Enables validation.
    • Minimal syntax overhead: Maximizes compactness.

  5. Overview of the TONL Public API and Architecture

    main

    The TONL library is organized into several functional layers. The primary entry points for developers are the Public API layer, which includes the core conversion functions and the CLI.

    Public API Layer:

    • TONLDocument: Represents the document structure.
    • encodeTONL: Converts JSON data to TONL format.
    • decodeTONL: Converts TONL format back to JSON.
    • CLI: Command-line interface for interacting with the format.

    Core Capabilities:

    • Query System: JSONPath-like queries for data retrieval.
    • Modification API: Supports CRUD operations on the data.
    • Schema Validation: Ensures data adheres to defined structures.
    • Streaming: Handles large files via streaming processing.
    • Navigation: Provides tree traversal capabilities.

  6. What is TONL and why use it?

    main

    TONL (Token-Optimized Notation Language) is a text-first serialization format designed to reduce token usage in Large Language Model (LLM) prompts. It achieves significant token reduction (typically 32-45% compared to JSON) by using a schema-based tabular format that eliminates redundant keys and excessive punctuation.

    Key Benefits:

    • Token Efficiency: Reduces overhead from repeated JSON keys and structural characters.
    • Human Readable: Designed to be editable in text editors and diffable in version control.
    • High Fidelity: Guarantees 100% round-trip fidelity with JSON (decodeTONL(encodeTONL(json)) === json).
    • Zero Dependencies: A pure TypeScript library with no runtime dependencies.

  7. Handle heterogeneous arrays in TONL

    main

    When an array contains objects with different keys (e.g., optional fields), you have two primary patterns in TONL:

    1. Tabular with null: Best if most objects share a similar structure. Define all possible columns in the header and use null for missing values.
    2. Mixed array: Best if objects are significantly different. Define the array header without columns, then specify the structure for each index individually using [index]{columns}.

    Recommendation: Use Solution 1 for similar structures and Solution 2 for highly varied structures.

    # Solution 1: Tabular with null
#version 1.0
events[3]{type,user,timestamp,duration}:
  login, alice, 1699100000, null
  logout, alice, 1699110000, 3600
  login, bob, 1699120000, null

# Solution 2: Mixed array
#version 1.0
events[3]:
  [0]{type,user,timestamp}: type: login user: alice timestamp: 1699100000
  [1]{type,user,timestamp,duration}: type: logout user: alice timestamp: 1699110000 duration: 3600
  [2]{type,user,timestamp}: type: login user: bob timestamp: 1699120000

  8. Representing primitive types in TONL

    main

    TONL maps JSON primitives to a compact notation. Primitives can often be placed on a single line if they are not nested.

    • Strings: Unquoted if they contain no special characters/delimiters and don't look like other types; otherwise, use quotes.
    • Numbers: Represented directly (e.g., 42, 3.14).
    • Special Numbers: Infinity, -Infinity, and NaN are supported as unquoted tokens.
    • Booleans: true and false are supported as unquoted tokens.
    • Null: null is supported as an unquoted token.
    # Basic Primitives
root{string,number,float,boolean,null_value}: string: hello number: 42 float: 3.14 boolean: true null_value: null

  9. Handle string quoting and unquoting in TONL

    main

    TONL uses specific quoting rules to ensure data integrity, especially when strings contain special characters or reserved literals.

    When to Quote

    Strings must be quoted if they contain:

    • Empty strings ("")
    • Reserved literals: true, false, null, undefined
    • Special numeric strings: Infinity, -Infinity, NaN, or any integer/decimal/scientific notation string.
    • Special characters: The delimiter, :, {, }, #, `

    , , `, or leading/trailing whitespace.

    Quote Types

    1. Single Double Quotes ("..."): Used for values with delimiters or special characters (excluding newlines).
      • Existing double quotes are escaped by doubling them ("").
      • Backslashes are escaped (\\).
    2. Triple Quotes ("""..."""): Used for multi-line strings or strings that contain the sequence """.
      • Triple quotes are escaped as \""".
      • Backslashes are escaped (\\).

    Unquoting Logic

    • Single quotes: Removes surrounding ", unescapes "" to ", and \\ to \.
    • Triple quotes: Removes surrounding """, unescapes \""" to """, and \\ to \.
    /* Example of Triple Quotes for multi-line input */

// Input:
Line 1
Line 2

// TONL Output:
description: """Line 1
Line 2"""

  10. Understand the TONL Data Model

    main

    TONL models data using the same primitives and structures as JSON:

    • Primitives: strings, numbers, booleans, and null.
    • Objects: Mappings from string keys to values.
    • Arrays: Ordered sequences of values.

    A TONL document can take three root forms:

    1. Root object (most common): Fields appear at depth 0 with no parent key.
    2. Root array: Begins with key[N]{fields}: at depth 0.
    3. Root primitive: A single primitive value (string, number, boolean, or null).

  11. Choose a TONL bundle format

    main

    TONL provides three different bundle formats depending on your integration method and environment requirements:

    • ESM (tonl.esm.js): Best for modern browsers and projects using bundlers (like Vite or Rollup).
    • UMD (tonl.umd.js): Compatible with older module loaders like Webpack or RequireJS.
    • IIFE (tonl.iife.js): Best for direct inclusion via a <script> tag in a standard HTML document.

  12. Understand type inference and coercion in TONL

    main

    If type hints are omitted, the TONL decoder uses inference rules to determine the data type:

    • Unquoted numbers: Parsed as numbers.
    • Quoted numbers: Parsed as strings.
    • Keywords: true, false, and null are parsed as booleans or null.
    • Others: Everything else is parsed as a string.

    Type Coercion in Strict Mode: When the decoder is running in strict mode, it will attempt to coerce values to match the specified type hints:

    • age:u32: "25" $\rightarrow$ 25 (number)
    • price:f64: "19.99" $\rightarrow$ 19.99 (float)
    • flag:bool: "true" $\rightarrow$ true (boolean)