TOON (Token-Oriented Object Notation)

Key folding collapses single-key object chains into dotted paths (e.g., data: { metadata: { items: [...] } } becomes data.metadata.items[...]). This significantly reduces token usage for LLMs.

To use it, enable KeyFoldingMode::Safe in EncodeOptions. You can also limit the depth of folding using with_flatten_depth(n).

Safety Rules: Folding only applies if segments are valid identifiers (a-z, A-Z, 0-9, _), each level has exactly one key, and there are no collisions with sibling keys.

use serde_json::json;
use toon_format::{encode, EncodeOptions, KeyFoldingMode};

let data = json!({
    "data": {
        "metadata": {
            "items": ["a", "b"]
        }
    }
});

// Enable key folding
let opts = EncodeOptions::new().with_key_folding(KeyFoldingMode::Safe);
let toon = encode(&data, &opts)?;
// Output: data.metadata.items[2]: a,b

Path expansion allows the decoder to automatically expand dotted keys into nested objects. This allows you to provide highly compact TOON input that still maps to complex nested structures.

To use it, enable PathExpansionMode::Safe in DecodeOptions.

Example: Input a.b.c: 1 will be decoded into {"a": {"b": {"c": 1}}}.

Note: Quoted keys (e.g., "c.d": 2) are treated as literal keys and will NOT be expanded.

use serde_json::Value;
use toon_format::{decode, DecodeOptions, PathExpansionMode};

let toon = "a.b.c: 1\na.b.d: 2";

// Enable path expansion
let opts = DecodeOptions::new().with_expand_paths(PathExpansionMode::Safe);
let json: Value = decode(toon, &opts)?;
// Result: {"a": {"b": {"c": 1, "d": 2}}}

The library provides high-level functions encode_default and decode_default that work with any type implementing serde::Serialize or serde::Deserialize (including custom structs and serde_json::Value).

Using Custom Structs

use serde::{Serialize, Deserialize};
use toon_format::{encode_default, decode_default};

#[derive(Serialize, Deserialize, Debug, PartialEq)]
struct User {
    name: String,
    age: u32,
    email: String,
}

fn main() -> Result<(), toon_format::ToonError> {
    let user = User { name: "Alice".to_string(), age: 30, email: "alice@example.com".to_string() };

    // Encode to TOON
    let toon = encode_default(&user)?;
    
    // Decode back to struct
    let decoded: User = decode_default(&toon)?;
    assert_eq!(user, decoded);
    Ok(())
}

You can use TOON as a Rust library for programmatic encoding/decoding or as a standalone command-line tool.

As a Library

Add toon-format to your Cargo.toml dependencies:

cargo add toon-format

As a CLI Tool

Install the toon binary globally using cargo:

cargo install toon-format

# Library
cargo add toon-format

# CLI
cargo install toon-format

The TUI provides a split-screen environment for real-time conversion:

• Input Panel (Left): Enter your JSON or TOON data here.
• Output Panel (Right): View the converted result instantly.
• Navigation: Use Tab to switch focus between the input and output panels.
• Modes: Toggle between Encode Mode (JSON $\to$ TOON) and Decode Mode (TOON $\to$ JSON) using Ctrl+E or Ctrl+M.
• Live Statistics: The bottom of the screen displays real-time token count comparisons, byte size comparisons, and percentage savings.

Press Ctrl+R to enter the REPL (Read-Eval-Print Loop) mode. This provides an interactive command-line interface for executing specific conversion commands and managing variables.

REPL Commands

Variable Usage

• Use $_ to reference the last result.
• Use $varname to reference a stored variable.
• Example: let $data = {...} followed by encode $data.

Press Ctrl+P to open the Interactive Settings Panel. You can adjust parameters on-the-fly using the following controls:

Encode Settings

• Delimiter: Cycle through Comma (,), Tab (\t), or Pipe (|) by pressing d.
• Indentation: Adjust spaces using + / = to increase and - / _ to decrease.
• Key Folding: Toggle with f (e.g., collapses {a:{b:1}} $\to$ a.b: 1).
• Flatten Depth: Control folding depth with [ / ] keys; press u for unlimited depth.

Press Ctrl+P to open the Interactive Settings Panel. Use these keys to adjust decoding behavior:

• Strict Mode: Toggle with s to enforce spec compliance.
• Type Coercion: Toggle with c to auto-convert strings to their appropriate types.
• Path Expansion: Toggle with p to expand folded paths (e.g., a.b:1 $\to$ {"a":{"b":1}}).

The toon CLI tool is used to encode JSON into the token-efficient TOON format or decode TOON back into JSON. It supports file input/output, stdin/stdout piping, and an interactive TUI mode.

Basic Usage Patterns:

• Encode JSON to TOON: toon input.json -o output.toon or cat data.json | toon -e
• Decode TOON to JSON: toon input.toon or toon input.toon --json-indent 2
• Interactive Mode: toon --interactive (launches a TUI)

# Encode JSON to TOON
toon input.json -o output.toon

# Decode TOON to JSON with indentation
toon input.toon --json-indent 2

# Pipe JSON to TOON and see token savings
cat data.json | toon -e --stats

# Launch interactive TUI
toon --interactive

The repository includes several example files that demonstrate different usage patterns. You can run them using cargo run --example <name>.

Key examples include:

• structs.rs: Custom struct serialization.
• tabular.rs: Tabular array formatting.
• arrays.rs & arrays_of_arrays.rs: Various single and nested array formats.
• objects.rs: Object encoding.
• mixed_arrays.rs: Handling mixed-type arrays.
• delimiters.rs: Using custom delimiters.
• round_trip.rs: Testing encode/decode cycles.
• decode_strict.rs: Using strict validation.
• empty_and_root.rs: Handling edge cases like empty or root-level data.

# Run all examples
cargo run --example examples