toon-python

TOON supports the following primitive types:

• Numbers:
  • Integers (e.g., 42, -17).
  • Floats (e.g., 3.14).
  • Note: Encoders must use decimal form, not scientific notation. Non-finite values (Infinity, NaN) are encoded as null.
  • Large integers (>2^53-1) should be quoted for compatibility (e.g., "9007199254740993").
• Booleans: Represented as lowercase true and false.
• Null: Represented as lowercase null.

To maximize the 30-60% token reduction offered by TOON, follow these structural guidelines:

1. Prefer Tabular Format: Use [N,]{fields}: instead of lists of objects. It is significantly more efficient.
2. Minimize Nesting: Deeply nested structures increase token overhead; flatten data where possible.
3. Use Compact Keys: Use short, descriptive keys (e.g., id instead of user_identification_number).
4. Be Consistent: Do not mix JSON and TOON in the same conversation context.

The project follows Semantic Versioning:

• MAJOR (X.0.0): Incompatible API changes.
• MINOR (0.X.0): New functionality, backward compatible.
• PATCH (0.0.X): Bug fixes, backward compatible.

Current Roadmap Examples:

• 0.9.x: Serializer, spec compliance, PyPI publishing.
• 1.0.0-rc.x: Production readiness candidates.
• 1.0.0: Official stable release.

TOON (Token-Oriented Object Notation) uses an indentation-based structure similar to YAML for representing objects. Objects consist of key: value pairs.

Simple Objects

Represented as direct key-value pairs:

name: Alice
age: 30
active: true

Nested Objects

Nesting is achieved through indentation (default is 2 spaces):

user:
  name: Alice
  settings:
    theme: dark

Object Keys

Keys follow identifier rules. If a key contains spaces, leading/trailing whitespace, structural characters (like :, [, {, -), or is a reserved keyword/numeric-looking string, it must be quoted.

name: Alice
age: 30
active: true

user:
  name: Alice
  settings:
    theme: dark

"with space": 4
"123": 3

To maximize the efficiency and reliability of using TOON with Large Language Models, follow these patterns:

• Explicit Prompting: Clearly instruct the model to use the TOON format.
• Few-Shot Examples: Provide concrete examples of the desired TOON structure within your prompt.
• Tabular Arrays: For maximum token efficiency (reducing costs by 30-60%), instruct the model to use tabular arrays for uniform objects.
• Error Recovery: Always wrap your decode() calls in error handling to manage potential model hallucinations or syntax errors gracefully.
• Consistency: Maintain the use of TOON throughout the entire conversation context.

Once the build is verified, follow these steps to trigger the automated publishing process:

1. Commit version changes to pyproject.toml and src/toon_format/__init__.py.
2. Create and push a Git tag (e.g., vX.Y.Z).
3. Publish to PyPI:
   • Go to GitHub Releases.
   • Create a new release using the tag you just pushed.
   • Add release notes and click "Publish release".
   • The GitHub Action will automatically build and publish the package to PyPI.

# Commit version changes
git add pyproject.toml src/toon_format/__init__.py
git commit -m "Bump version to X.Y.Z"

# Create and push tag
git tag -a vX.Y.Z -m "Release version X.Y.Z"
git push origin main
git push origin vX.Y.Z

You can use lengthMarker during encoding to provide explicit validation hints to the LLM. By setting lengthMarker to a character like #, the encoded TOON string will include markers (e.g., items[#3]: a,b,c) that tell the model exactly how many items are expected in an array.

Prompting Tip: Tell the model: "Array lengths are prefixed with #. Ensure your response matches these counts exactly."

from toon_format import encode

data = {"items": ["a", "b", "c"]}
toon = encode(data, {"lengthMarker": "#"})
# Output: items[#3]: a,b,c

When decoding LLM-generated strings, always wrap the decode() call in a try-except block to catch ToonDecodeError. This is critical because LLMs may occasionally produce invalid syntax or incorrect array lengths.

from toon_format import decode, ToonDecodeError

def safe_decode(toon_str):
    try:
        return decode(toon_str)
    except ToonDecodeError as e:
        print(f"TOON decode error: {e}")
        # Fall back to asking model to regenerate or use default
        return None

To ensure an LLM responds in TOON (Token-Oriented Object Notation), provide explicit format instructions in your system or user prompt. It is recommended to define the syntax rules and provide a structural example.

Recommended Prompting Rules:

• Use key: value for objects.
• Use indentation for nesting.
• Use [N] to indicate array lengths.
• Use tabular format [N,]{fields}: for uniform arrays.
• Always wrap the response in ```toon code blocks to help the model distinguish the format from natural language.

Respond using TOON format (Token-Oriented Object Notation):
- Use `key: value` for objects
- Use indentation for nesting
- Use `[N]` to indicate array lengths
- Use tabular format `[N,]{fields}:` for uniform arrays

Example:
users[2,]{id,name}:
  1,Alice
  2,Bob