uPickle Documentation
repository·main·Indexed 20 days ago
https://github.com/com-lihaoyi/upickleA high-performance, lightweight JSON and MessagePack serialization library for Scala. Designed to be zero-dependency and compatible with both the JVM and ScalaJS, uPickle provides tools for converting Scala objects to and from JSON (via the ujson package) and binary MessagePack formats (via the upack package), including support for AST manipulation, custom configuration via the Config trait, and sealed trait type tagging.
What's inside uPickle
- uPickle is a simple Scala library designed for both JSON and Binary (MessagePack) serialization. It provides a lightweight way to convert Scala objects to and from these formats.
Learn uPickle via tutorials and books
mainFor developers looking to master uPickle, the following resources are recommended:
- Blog Post: How to work with JSON in Scala provides a hands-on introduction.
- Book: Hands-on Scala Programming contains an entire chapter (Chapter 8) dedicated to JSON and Binary Data Serialization using uPickle.
Configure upickle serialization behavior via the Config trait
mainTo customize how upickle serializes and deserializes data globally, you can implement or override the
Configtrait. This allows you to control field naming, type tagging for sealed traits, handling of default values, and howOptiontypes are represented in JSON.Common configuration areas include:
- Sealed Trait Tagging: Control the key used to distinguish subtypes (defaults to
$type) and whether to use fully-qualified names. - Field Naming: Use
objectAttributeKeyWriteMapandobjectAttributeKeyReadMapto transform Scala field names to/from JSON keys globally. - Default Values: Control whether fields matching their default values are omitted from the output.
- Option Handling: Decide if
Optiontypes are serialized as unboxed values/nulls or as 0-or-1-element arrays. - Unknown Keys: Determine if de-serialization should fail or skip when encountering unknown JSON keys.
- Sealed Trait Tagging: Control the key used to distinguish subtypes (defaults to
Understand MessagePack vs JSON data representation differences
mainConverting between MessagePack and JSON is lossy because some MessagePack constructs (like binary data) cannot be exactly represented in JSON.
Key differences include:
- Large Longs: In JSON, values $> 2^{53}$ are represented as
ujson.Str(strings). In MessagePack, they are represented asupack.Int64orupack.UInt64. - Byte Arrays:
Array[Byte]is represented as a list of numbers in JSON, but asupack.Binaryin MessagePack.
Note: Round-tripping between
Scala data types <-> JSONandScala data types <-> MessagePackis always safe, butJSON <-> MessagePackis not.- Large Longs: In JSON, values $> 2^{53}$ are represented as
MessagePack data types in upack
mainThe
Msgtrait is implemented by several case classes representing the MessagePack primitives. Note that while MessagePack has many specific integer and string types,upackcollapses them into a unified in-memory model:Null: Represents a null value.True/False: Represent boolean values.Int32(value: Int): 32-bit signed integer.Int64(value: Long): 64-bit signed integer.UInt64(value: Long): 64-bit unsigned integer.Float32(value: Float): 32-bit float.Float64(value: Double): 64-bit float.Str(value: String): String data.Binary(value: Array[Byte]): Binary data.Arr(value: mutable.ArrayBuffer[Msg]): An array ofMsgelements.Obj(value: LinkedHashMap[Msg, Msg]): A map of key-value pairs where both keys and values areMsg.Ext(tag: Byte, data: Array[Byte]): Extension type with a specific tag.
Constructing Collections
- Arrays: Use
Arr(items: Msg*). - Objects: Use
Obj(firstPair: (Msg, Msg), remainingPairs: (Msg, Msg)*)orObj()for an empty object.
import upack._ import scala.collection.mutable // Creating an Array val myArr = Arr(Int32(1), Str("two"), Bool(true)) // Creating an Object val myObj = Obj( (Str("key1"), Int32(100)), (Str("key2"), Str("value2")) ) // Creating an empty Object val emptyObj = Obj()Customize field keys with @key
mainUse the
@key("...")annotation to control the name used for a field during serialization. This is useful for:- Backwards Compatibility: Renaming a field in Scala while keeping the old name in the JSON.
- Tag Overriding: Changing the type tag used for members of a
sealed traithierarchy (default is$type). - Compatibility: Changing the key used for sealed hierarchy members to match other JSON libraries.
Use @flatten to flatten case classes or collections
mainThe
@flattenannotation allows you to merge fields of a nested structure into the parent structure during serialization and deserialization.Use Cases
- Case Classes: Flatten fields of a nested case class into the parent.
- Iterables: Flatten key-value pairs of an
Iterable[(String, _)](like aMap[String, String]) into the parent structure.
Limitations
- You cannot flatten more than two collections to the same level.
- When flattening an
Iterable, the key type must beString.
case class A(i: Int, @@flatten b: B) case class B(msg: String) implicit val rw: ReadWriter[A] = macroRW implicit val rw: ReadWriter[B] = macroRW write(A(1, B("Hello"))) // {"i":1, "msg": "Hello"}The ujson.Value AST structure
mainThe
ujson.Valuehierarchy consists of the following types:ujson.Str(value: String): A JSON string.ujson.Obj(value: LinkedHashMap[String, Value]): A JSON object.ujson.Arr(value: mutable.ArrayBuffer[Value]): A JSON array.ujson.Num(value: Double): A JSON number.ujson.Bool: A JSON boolean (ujson.Trueorujson.False).ujson.Null: A JSON null value.
ujson.Objandujson.Arrprovideapplymethods for convenient construction.// Constructing an object val obj = ujson.Obj("a" -> 1, "b" -> "two") // Constructing an array val arr = ujson.Arr(1, 2, 3) // Constructing a boolean val b = ujson.Bool(true)Serialize and Deserialize Scala Objects
mainuPickle allows you to write Scala objects to JSON strings or MessagePack byte arrays, and read them back.
Note on Scala Versions:
- In Scala 3, use the convenient
upickle.{read, write}. - In Scala 2.x, you must use the more verbose
upickle.default.{read, write}.
Supported Types
- Primitives:
Boolean,Byte,Char,Short,Int,Long,Float,Double. - Collections:
Seq,List,Vector,Set,SortedSet,Option,Array,Map,Tuple(1-22). - Others:
Duration,Either,UUID,null, andsealed trait/sealed classhierarchies. - Case Classes: Up to 64 fields.
Serialization Formats
- JSON: Human-readable text.
- MessagePack: Binary format via
upack, providing faster serialization and smaller payloads.
- In Scala 3, use the convenient
Upgrade to uPickle 4.0.0 from 3.x
mainuPickle 4.0.0 is a major breaking change affecting serialization formats (e.g.,
Options,sealed traits) and binary compatibility.To upgrade while preserving your existing serialization format:
- Define a Custom Configuration.
- Replace all usages of
upickle.defaultwith your custom configuration. - In your configuration, set:
override def objectTypeKeyWriteFullyQualified = true(forwards and backwards compatible).override def optionsAsNulls = false(Note: This is not forwards or backwards compatible; use this if you cannot perform a 'big bang' upgrade of all data/code simultaneously).
Use upickle.default.web for high-performance JSON on Scala.js
mainOn Scala.js/Node.js, standard Scala-based JSON parsers can be slow. uPickle provides aupickle.default.webAPI that leverages the JavaScript runtime's built-inJSON.parseandJSON.stringify. This is typically 4-6x faster than the pure Scala implementations.Derive JSON Schema in Scala 3
mainUsing the experimental
upickle-jsonschemamodule, you can derive JSON Schema documents from types that implementReadWriterorJsonSchema. This is currently targeting Scala 3.- Add the dependency:
- SBT:
"com.lihaoyi" %% "upickle-jsonschema" % "4.4.3" - Mill:
ivy"com.lihaoyi::upickle-jsonschema:4.4.3"
- SBT:
- Define
JsonSchemainstances usingJsonSchema.derived. - Call
upickle.default.schema[T]to get the schema as aujson.Value.
Recursive and mutually-recursive types are supported using
lazy given JsonSchema[T] = JsonSchema.derived.import upickle.default.* import upickle.jsonschema.* case class Address(street: String, zip: Int) derives ReadWriter case class Person(name: String, address: Address) derives ReadWriter given JsonSchema[Address] = JsonSchema.derived given JsonSchema[Person] = JsonSchema.derived val schemaJson: ujson.Value = upickle.default.schema[Person] println(schemaJson.render(indent = 2))- Add the dependency: