repository·main·Indexed 27 days ago
https://github.com/bytedance/sonicA high-performance JSON library for Go that utilizes JIT and SIMD acceleration for fast serialization and deserialization. It features runtime object binding, a complete set of APIs for JSON manipulation via ast.Node, and streaming IO support. Sonic provides multiple configuration profiles (ConfigDefault, ConfigStd, ConfigFastest) to balance speed and compatibility with encoding/json.
Sonic is a high-performance JSON serialization and deserialization library for Go. It is accelerated using JIT (just-in-time compiling) and SIMD (single-instruction-multiple-data) to achieve high throughput. Key features include:
Sonic provides high-speed JSON operations with the following characteristics:
Before using Sonic, ensure your environment meets the following requirements:
-ldflags="-checklinkname=0".Sonic uses a JIT assembler that compiles schemas at runtime. For very large schemas or latency-sensitive applications, the first-hit compilation can cause request timeouts or Out-of-Memory (OOM) errors. To mitigate this, use sonic.PretouchMany() during application initialization to pre-compile the required types.
You can control the compilation depth using:
option.WithCompileRecursiveDepth(depth): Sets the recursion depth for deeply nested types.option.WithCompileMaxInlineDepth(depth): Sets the maximum inline depth to reduce compilation time for large structs.import (
"reflect"
"github.com/bytedance/sonic"
"github.com/bytedance/sonic/option"
)
func init() {
var v1 HugeStruct1
var v2 HugeStruct2
// For most large types (nesting depth <= option.DefaultMaxInlineDepth)
sonic.PretouchMany([]reflect.Type{reflect.TypeOf(v1), reflect.TypeOf(v2)},
// If the type is too deep nesting (nesting depth > option.DefaultMaxInlineDepth),
// you can set more recursive loops in Pretouch for fully sufficient JIT.
option.WithCompileRecursiveDepth(loop),
// For a large struct, try to set a smaller depth to reduce compiling time.
option.WithCompileMaxInlineDepth(depth),
)
}For scenarios where you only need specific parts of a JSON document, combine Get() with Unmarshal() or use ast.Node as a generic container instead of map or interface{}. ast.Node uses an array-based storage that is more efficient for insertion and scanning than maps, and it supports lazy loading (parsing values only on demand).
Concurrency Note: ast.Node is not inherently thread-safe due to its lazy loading design. To use a node concurrently, you must call Node.Load() or Node.LoadAll() first.
import "github.com/bytedance/sonic"
// Example: Combining Get and Unmarshal for partial schema
node, err := sonic.GetFromString(_TwitterJson, "statuses", 3, "user")
var user User
err = sonic.UnmarshalString(node.Raw(), &user)
// Example: Using ast.Node as a generic container
root, err := sonic.GetFromString(_TwitterJson)
user := root.GetByPath("statuses", 3, "user")
err = user.Check()
// err = user.LoadAll() // Call this to make 'user' safe for concurrent use
go someFunc(user)[]byte APIs for compatibility with encoding/json, converting strings to byte slices can incur performance costs for very large JSON. For better performance when your source data is already a string, use UnmarshalString() and GetFromString(). Additionally, use MarshalString() for encoding to allow safe zero-copy type conversion of the resulting JSON bytes.Sonic uses golang-asm as a JIT assembler. Running large JSON patterns for the first time at runtime can cause request timeouts or process memory overflows due to compilation overhead. To improve stability in latency-sensitive applications, run sonic.PretouchMany() with your target types before calling Marshal() or Unmarshal().
import (
"reflect"
"github.com/bytedance/sonic"
"github.com/bytedance/sonic/option"
)
func init() {
var v1 HugeStruct1
var v2 HugeStruct2
// For most large types (nesting depth <= option.DefaultMaxInlineDepth)
sonic.PretouchMany([]reflect.Type{reflect.TypeOf(v1), reflect.TypeOf(v2)},
// If the type is too deep nesting (nesting depth > option.DefaultMaxInlineDepth),
// you can set more recursive loops in Pretouch for fully sufficient JIT.
option.WithCompileRecursiveDepth(loop),
// For a large struct, try to set a smaller depth to reduce compiling time.
option.WithCompileMaxInlineDepth(depth),
)
}ast.Node is a good generic container for partial parsing, it is still an intermediate representation. For extreme performance requirements where you want to parse JSON directly into your custom types without intermediate steps, use ast.Visitor. Note that ast.Visitor is a complex API that requires manual implementation of the visitor pattern and careful management of the tree hierarchy.To avoid the performance penalty of string-to-byte copying when handling large JSON data:
UnmarshalString() and GetFromString() when your source data is a string.MarshalString() for convenient encoding of JSON bytes, as Sonic's output is always unique and safe for this purpose.[]byte and you can safely perform a no-copy cast, use the specialized APIs to avoid the overhead of encoding/json alignment.To reduce memory consumption and handle multiple JSON values, Sonic supports streaming via io.Reader and io.Writer.
sonic.ConfigDefault.NewEncoder(w) to encode objects into a writer. You can call Encode multiple times to stream multiple JSON objects.sonic.ConfigDefault.NewDecoder(r) to decode from a reader. You can call Decode multiple times to process a stream of JSON objects.// Encoder example
var o1 = map[string]interface{}{
"a": "b",
}
var o2 = 1
var w = bytes.NewBuffer(nil)
var enc = sonic.ConfigDefault.NewEncoder(w)
enc.Encode(o1)
enc.Encode(o2)
// Decoder example
var o = map[string]interface{}{}
var r = strings.NewReader(`{"a":"b"}{"1":"2"}`)
var dec = sonic.ConfigDefault.NewDecoder(r)
dec.Decode(&o)
dec.Decode(&o)decoder.CopyString() option.Sonic is a high-performance JSON serialization/deserialization library accelerated by JIT (Just-In-Time compilation) and SIMD.
--ldflags="-checklinkname=0"