DOCSEARCHbyAlgolia
Submit Docs

mus-go

repository·main·Indexed 41 days ago

https://github.com/mus-format/mus-go

A high-performance, flexible binary serialization library for Go. It provides low-level primitives for building custom binary formats or implementing existing ones like Protobuf, featuring modular packages for varint, raw, ordinary (ord), unsafe, pointer mapping (pm), and typed serialization.

Tokens
2.5K
Snippets
8
Records
16
Agent score
50%

What's inside mus

  1. Overview of mus encoding packages

    main

    The mus library is modular. You choose a package based on your performance and data requirements:

    PackageUse CaseKey StrengthTrade-off
    varintNumbersSpace efficientSlight CPU cost
    rawNumbers, TimeFast encodingHigher space usage for small numbers
    ord (ordinary)Pointers, Strings, Slices, MapsVariable-length types supportStandard allocations
    unsafeHigh-perf Numbers, Time, Strings, ArraysZero-allocationUses unsafe type conversions
    pm (pointer mapping)Pointers, Cyclic Graphs, Linked ListsPreserves pointer equalitySlightly more complex than ord
    typedInterface/VersioningTyped serializationRequires DTM definition

  2. Run mus tests and fuzz tests

    main

    To verify your implementation or the library itself:

    Standard Tests:

    go test ./...

    Fuzz Testing: To run the provided fuzzing script:

    ./fuzz.sh 10s

    To run a specific fuzz test using the Go toolchain:

    go test -v -fuzz="^FuzzVarint_Byte$" ./varint -fuzztime 10s

  3. Quick Start: Serialize a number with varint

    main

    To perform basic serialization of a number using the varint package, follow these steps:

    1. Calculate the required buffer size using varint.Int.Size(num).
    2. Pre-allocate a byte slice of that size.
    3. Use varint.Int.Marshal to write the value to the buffer.
    4. Use varint.Int.Unmarshal to read the value back.
    package main

import (
  "fmt"
  "github.com/mus-format/mus-go/varint"
)

func main() {
  num  := 100

  // Pre-allocate a buffer.
  var (
    size = varint.Int.Size(num)
    bs   = make([]byte, size)
  )
  
  // Marshal.
  varint.Int.Marshal(num, bs)
  
  // Unmarshal.
  num, _, err := varint.Int.Unmarshal(bs)
  if err != nil {
    panic(err)
  }
}

  4. Implement the mus.Serializer interface

    main

    To make a custom type serializable with mus, you must implement the mus.Serializer interface. This allows you to define exactly how each field of your type is encoded and decoded using the primitives provided in the library's packages.

    Required methods for the mus.Serializer interface:

    • Marshal(v Type, bs []byte) (n int): Serializes the value into the provided buffer and returns the number of bytes used.
    • Unmarshal(bs []byte) (v Type, n int, err error): Deserializes the value from the buffer, returning the value, bytes used, and any error.
    • Size(v Type) (size int): Returns the number of bytes required to serialize the value.
    • Skip(bs []byte) (n int, err error): Skips the serialized value in the buffer, returning the number of bytes skipped and any error.
    import "github.com/mus-format/mus-go"

// YourTypeMUS is a MUS serializer for YourType.
var YourTypeMUS = yourTypeMUS{}

// yourTypeMUS implements the mus.Serializer interface.
type yourTypeMUS struct{}

func (s yourTypeMUS) Marshal(v YourType, bs []byte) (n int)              {...}
func (s yourTypeMUS) Unmarshal(bs []byte) (v YourType, n int, err error) {...}
func (s yourTypeMUS) Size(v YourType) (size int)                         {...}
func (s yourTypeMUS) Skip(bs []byte) (n int, err error)                  {...}

  5. Use the unsafe package for zero-allocation performance

    main
    The unsafe package provides maximum performance using unsafe type conversions. This is particularly effective for the string type, where modifying the underlying byte slice after unmarshalling will also change the string's contents. It supports byte, bool, string, array, byte slice, time.Time, and all uint, int, and float types.

  6. Use the pm (pointer mapping) package for cyclic graphs

    main
    The pm package preserves pointer equality after unmarshalling. This allows you to serialize complex data structures like cyclic graphs or linked lists where multiple pointers point to the same memory address, ensuring ptr1 == ptr2 after deserialization.

  8. Use the ord (ordinary) package for variable-length types

    main

    The ord package handles bool, string, byte slice, slice, map, and pointer types. Variable-length data is encoded as length + data. By default, length is encoded using varint.PositiveInt.

    Note: For slice and map types, only constructors are available.

  9. Use the varint package for space-efficient numbers

    main
    The varint package provides Varint serializers for all uint (e.g., uint64, uint32, ...), int, float, and byte data types. It also includes the PositiveInt serializer (Varint without ZigZag) for efficiently encoding positive int values.

  10. Use the raw package for fast number and time encoding

    main

    The raw package contains Raw serializers for byte, uint, int, float, and time.Time data types. It is optimized for speed at the cost of higher space usage for small numbers.

    For time.Time, available serializers include:

    • TimeUnix (seconds)
    • TimeUnixMilli (milliseconds)
    • TimeUnixMicro (microseconds)
    • TimeUnixNano (nanoseconds)

    To ensure deserialized time.Time values are in UTC, use UTC-specific serializers like TimeUnixUTC or TimeUnixMilliUTC, or set the TZ environment variable to UTC.

  11. Serialize a Marshaller using Marshal()

    main

    The Marshal function is a convenience helper that creates a byte slice of the correct size and fills it with the serialized data of a type implementing the Marshaller interface.

    Usage: Pass any object that implements Marshaller to Marshal(v). It internally calls v.SizeMUS() to allocate the buffer and v.MarshalMUS(bs) to perform the encoding.

    // Assuming 'v' implements Marshaller
bs := mus.Marshal(v)

  12. Serialize a MarshallerTyped using MarshalTyped()

    main

    The MarshalTyped function is a convenience helper for types implementing the MarshallerTyped interface. It is used when you need the specialized typed serialization features provided by the mus/typed package.

    Usage: Pass any object that implements MarshallerTyped to MarshalTyped(v). It internally calls v.SizeTypedMUS() to allocate the buffer and v.MarshalTypedMUS(bs) to perform the encoding.

    // Assuming 'v' implements MarshallerTyped
bs := mus.MarshalTyped(v)