uPickle Documentation

repository·main·Indexed 20 days ago

https://github.com/com-lihaoyi/upickle

A high-performance, lightweight JSON and MessagePack serialization library for Scala. Designed to be zero-dependency and compatible with both the JVM and ScalaJS, uPickle provides tools for converting Scala objects to and from JSON (via the ujson package) and binary MessagePack formats (via the upack package), including support for AST manipulation, custom configuration via the Config trait, and sealed trait type tagging.

Tokens
8K
Snippets
22
Records
40
Agent score
73%

What's inside uPickle

  1. Configure upickle serialization behavior via the Config trait

    main

    To customize how upickle serializes and deserializes data globally, you can implement or override the Config trait. This allows you to control field naming, type tagging for sealed traits, handling of default values, and how Option types are represented in JSON.

    Common configuration areas include:

    • Sealed Trait Tagging: Control the key used to distinguish subtypes (defaults to $type) and whether to use fully-qualified names.
    • Field Naming: Use objectAttributeKeyWriteMap and objectAttributeKeyReadMap to transform Scala field names to/from JSON keys globally.
    • Default Values: Control whether fields matching their default values are omitted from the output.
    • Option Handling: Decide if Option types are serialized as unboxed values/nulls or as 0-or-1-element arrays.
    • Unknown Keys: Determine if de-serialization should fail or skip when encountering unknown JSON keys.
  2. Understand MessagePack vs JSON data representation differences

    main

    Converting between MessagePack and JSON is lossy because some MessagePack constructs (like binary data) cannot be exactly represented in JSON.

    Key differences include:

    • Large Longs: In JSON, values $> 2^{53}$ are represented as ujson.Str (strings). In MessagePack, they are represented as upack.Int64 or upack.UInt64.
    • Byte Arrays: Array[Byte] is represented as a list of numbers in JSON, but as upack.Binary in MessagePack.

    Note: Round-tripping between Scala data types <-> JSON and Scala data types <-> MessagePack is always safe, but JSON <-> MessagePack is not.

  3. MessagePack data types in upack

    main

    The Msg trait is implemented by several case classes representing the MessagePack primitives. Note that while MessagePack has many specific integer and string types, upack collapses them into a unified in-memory model:

    • Null: Represents a null value.
    • True / False: Represent boolean values.
    • Int32(value: Int): 32-bit signed integer.
    • Int64(value: Long): 64-bit signed integer.
    • UInt64(value: Long): 64-bit unsigned integer.
    • Float32(value: Float): 32-bit float.
    • Float64(value: Double): 64-bit float.
    • Str(value: String): String data.
    • Binary(value: Array[Byte]): Binary data.
    • Arr(value: mutable.ArrayBuffer[Msg]): An array of Msg elements.
    • Obj(value: LinkedHashMap[Msg, Msg]): A map of key-value pairs where both keys and values are Msg.
    • Ext(tag: Byte, data: Array[Byte]): Extension type with a specific tag.

    Constructing Collections

    • Arrays: Use Arr(items: Msg*).
    • Objects: Use Obj(firstPair: (Msg, Msg), remainingPairs: (Msg, Msg)*) or Obj() for an empty object.
    import upack._
    import scala.collection.mutable
    
    // Creating an Array
    val myArr = Arr(Int32(1), Str("two"), Bool(true))
    
    // Creating an Object
    val myObj = Obj(
      (Str("key1"), Int32(100)),
      (Str("key2"), Str("value2"))
    )
    
    // Creating an empty Object
    val emptyObj = Obj()
  4. Customize field keys with @key

    main

    Use the @key("...") annotation to control the name used for a field during serialization. This is useful for:

    • Backwards Compatibility: Renaming a field in Scala while keeping the old name in the JSON.
    • Tag Overriding: Changing the type tag used for members of a sealed trait hierarchy (default is $type).
    • Compatibility: Changing the key used for sealed hierarchy members to match other JSON libraries.
  5. Use @flatten to flatten case classes or collections

    main

    The @flatten annotation allows you to merge fields of a nested structure into the parent structure during serialization and deserialization.

    Use Cases

    1. Case Classes: Flatten fields of a nested case class into the parent.
    2. Iterables: Flatten key-value pairs of an Iterable[(String, _)] (like a Map[String, String]) into the parent structure.

    Limitations

    • You cannot flatten more than two collections to the same level.
    • When flattening an Iterable, the key type must be String.
    case class A(i: Int, @@flatten b: B)
    case class B(msg: String)
    implicit val rw: ReadWriter[A] = macroRW
    implicit val rw: ReadWriter[B] = macroRW
    write(A(1, B("Hello"))) // {"i":1, "msg": "Hello"}
  6. The ujson.Value AST structure

    main

    The ujson.Value hierarchy consists of the following types:

    • ujson.Str(value: String): A JSON string.
    • ujson.Obj(value: LinkedHashMap[String, Value]): A JSON object.
    • ujson.Arr(value: mutable.ArrayBuffer[Value]): A JSON array.
    • ujson.Num(value: Double): A JSON number.
    • ujson.Bool: A JSON boolean (ujson.True or ujson.False).
    • ujson.Null: A JSON null value.

    ujson.Obj and ujson.Arr provide apply methods for convenient construction.

    // Constructing an object
    val obj = ujson.Obj("a" -> 1, "b" -> "two")
    
    // Constructing an array
    val arr = ujson.Arr(1, 2, 3)
    
    // Constructing a boolean
    val b = ujson.Bool(true)
  7. Serialize and Deserialize Scala Objects

    main

    uPickle allows you to write Scala objects to JSON strings or MessagePack byte arrays, and read them back.

    Note on Scala Versions:

    • In Scala 3, use the convenient upickle.{read, write}.
    • In Scala 2.x, you must use the more verbose upickle.default.{read, write}.

    Supported Types

    • Primitives: Boolean, Byte, Char, Short, Int, Long, Float, Double.
    • Collections: Seq, List, Vector, Set, SortedSet, Option, Array, Map, Tuple (1-22).
    • Others: Duration, Either, UUID, null, and sealed trait/sealed class hierarchies.
    • Case Classes: Up to 64 fields.

    Serialization Formats

    • JSON: Human-readable text.
    • MessagePack: Binary format via upack, providing faster serialization and smaller payloads.
  8. Upgrade to uPickle 4.0.0 from 3.x

    main

    uPickle 4.0.0 is a major breaking change affecting serialization formats (e.g., Options, sealed traits) and binary compatibility.

    To upgrade while preserving your existing serialization format:

    1. Define a Custom Configuration.
    2. Replace all usages of upickle.default with your custom configuration.
    3. In your configuration, set:
      • override def objectTypeKeyWriteFullyQualified = true (forwards and backwards compatible).
      • override def optionsAsNulls = false (Note: This is not forwards or backwards compatible; use this if you cannot perform a 'big bang' upgrade of all data/code simultaneously).
  9. Use upickle.default.web for high-performance JSON on Scala.js

    main
    On Scala.js/Node.js, standard Scala-based JSON parsers can be slow. uPickle provides a upickle.default.web API that leverages the JavaScript runtime's built-in JSON.parse and JSON.stringify. This is typically 4-6x faster than the pure Scala implementations.
  10. Derive JSON Schema in Scala 3

    main

    Using the experimental upickle-jsonschema module, you can derive JSON Schema documents from types that implement ReadWriter or JsonSchema. This is currently targeting Scala 3.

    1. Add the dependency:
      • SBT: "com.lihaoyi" %% "upickle-jsonschema" % "4.4.3"
      • Mill: ivy"com.lihaoyi::upickle-jsonschema:4.4.3"
    2. Define JsonSchema instances using JsonSchema.derived.
    3. Call upickle.default.schema[T] to get the schema as a ujson.Value.

    Recursive and mutually-recursive types are supported using lazy given JsonSchema[T] = JsonSchema.derived.

    import upickle.default.*
    import upickle.jsonschema.*
    
    case class Address(street: String, zip: Int) derives ReadWriter
    case class Person(name: String, address: Address) derives ReadWriter
    
    given JsonSchema[Address] = JsonSchema.derived
    given JsonSchema[Person] = JsonSchema.derived
    
    val schemaJson: ujson.Value = upickle.default.schema[Person]
    println(schemaJson.render(indent = 2))