DOCSEARCHbyAlgolia
Submit Docs

toon-go: TOON Format for Go

repository·main·Indexed 40 days ago

https://github.com/toon-format/toon-go

A Go implementation of the Token-Oriented Object Notation (TOON) format, designed to minimize token consumption when communicating structured data to Large Language Models. Provides functions for marshaling and unmarshaling Go structs and maps, as well as a Decode function for dynamic representation of TOON data.

Tokens
1.7K
Snippets
8
Records
8
Agent score
50%

What's inside toon-go

  1. What is TOON Format?

    main

    Token-Oriented Object Notation (TOON) is a compact, human-readable data format specifically designed to reduce token usage when passing structured data to Large Language Models (LLMs). It provides a significantly more dense representation compared to JSON.

    users[2]{id,name,role}:
  1,Alice,admin
  2,Bob,user

  2. Unmarshal TOON data into Maps

    main

    Similar to the encoding/json package, toon.Unmarshal can be used to populate dynamic maps (map[string]any) when the exact schema is unknown at compile time.

    var doc map[string]any
if err := toon.Unmarshal(encoded, &doc); err != nil {
    panic(err)
}
fmt.Printf("users: %#v\n", doc["users"])

  3. Marshal and Unmarshal TOON data

    main

    Use toon.Marshal to convert Go structs into TOON format and toon.Unmarshal to populate structs from TOON data. To map struct fields to TOON keys, use the toon struct tag.

    toon.Marshal accepts an optional functional option toon.WithLengthMarkers(bool) to include length markers in the output.

    package main

import (
    "fmt"
    "github.com/toon-format/toon-go"
)

type User struct {
    ID    int    `toon:"id"`
    Name  string `toon:"name"`
    Role  string `toon:"role"`
}

type Payload struct {
    Users []User `toon:"users"`
}

func main() {
    in := Payload{
        Users: []User{
            {ID: 1, Name: "Alice", Role: "admin"},
            {ID: 2, Name: "Bob", Role: "user"},
        },
    }

    // Marshal with length markers enabled
    encoded, err := toon.Marshal(in, toon.WithLengthMarkers(true))
    if err != nil {
        panic(err)
    }
    fmt.Println(string(encoded))

    // Unmarshal into a struct
    var out Payload
    if err := toon.Unmarshal(encoded, &out); err != nil {
        panic(err)
    }
    fmt.Printf("first user: %+v\n", out.Users[0])
}

  4. Decode TOON data without structs

    main

    If you do not have a destination struct or map ready, use toon.Decode to obtain a dynamic representation of the TOON data directly from a byte slice.

    package main

import (
    "fmt"
    "github.com/toon-format/toon-go"
)

func main() {
    raw := []byte("users[2]{id,name,role}:\n  1,Alice,admin\n  2,Bob,user\n")
    decoded, err := toon.Decode(raw)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%+v\n", decoded)
}

  5. Create ordered TOON Objects

    main

    The Object type preserves the encounter order of its fields, ensuring deterministic emission by the encoder. Use NewObject to construct an ordered Object from Field key/value pairs.

    // Field is a key/value pair
field := toon.Field{Key: "name", Value: "example"}
obj := toon.NewObject(field)

  6. Unmarshal TOON data into Go types

    main

    Use Unmarshal to decode TOON bytes into a non-nil pointer to a Go value. Struct fields are mapped using toon struct tags (supporting omitempty). Alternatively, use Decode or DecodeString to parse TOON data into generic Go types (float64 for numbers, map[string]any for objects, and []any for arrays).

    Available DecoderOption functions:

    • WithStrictMode(strict bool): Enables/disables strict-mode diagnostics.
    • WithDecoderIndent(spaces int): Configures the expected indentation step.
    • WithDecoderDocumentDelimiter(delimiter Delimiter): Configures the delimiter for delimiter-aware string parsing when no array header is active.
    // Unmarshal into a struct
type User struct {
    Name string `toon:"name"` 
}
var u User
err := toon.Unmarshal(data, &u)

// Decode into generic types
val, err := toon.Decode(data)

  7. Marshal Go values to TOON format

    main

    Use Marshal to convert a Go value into a TOON-formatted byte slice, or MarshalString to get a string. You can pass EncoderOptions to customize the output, such as indentation or delimiters.

    Available EncoderOption functions:

    • WithIndent(spaces int): Sets the number of spaces per indentation level.
    • WithDocumentDelimiter(delimiter Delimiter): Configures the delimiter for quoting decisions outside array scopes.
    • WithArrayDelimiter(delimiter Delimiter): Sets the default delimiter for arrays that don't explicitly override it.
    • WithLengthMarkers(enabled bool): Enables/disables optional # markers in array headers.
    • WithTimeFormatter(formatter func(time.Time) string): Provides a custom formatter for time.Time values.
    import "github.com/toon-format/toon-go"

// Marshal to []byte
data, err := toon.Marshal(myStruct, toon.WithIndent(2))

// Marshal to string
str, err := toon.MarshalString(myStruct, toon.WithLengthMarkers(true))

  8. Reference: TOON Delimiters

    main

    Delimiters identify the character used to split values inside array scopes. These can be used with WithArrayDelimiter, WithDocumentDelimiter, or their decoder counterparts.

    const (
	DelimiterComma = codec.DelimiterComma // Default
	DelimiterTab  = codec.DelimiterTab  // HTAB
	DelimiterPipe = codec.DelimiterPipe // '|'
)