ZIO Schema

repository·main·Indexed 41 days ago

https://github.com/zio/zio-schema

A Scala library for modeling data structures as first-class runtime values, providing powerful serialization, deserialization, and data migration capabilities. It supports automatic schema derivation via zio-schema-derivation and provides codecs for various protocols including JSON, Protobuf, Avro, BSON, MsgPack, and Thrift. Key features include the ability to calculate diffs between objects, generate optics for nested data manipulation, and transform DTO schemas into domain schemas.

Tokens
50.2K
Snippets
147
Records
204
Agent score
50%

What's inside ZIO Schema

  1. Overview of ZIO Schema operations

    main

    Once a schema is defined, ZIO Schema provides a rich set of operations to manipulate, transform, and interact with data. These operations allow you to combine schemas, transform data types, handle optionality, perform migrations, and manage data patches. Common categories of operations include:

    • Composition: Combining schemas using product (<*>, zip) or sum (<+>, orElseEither) types.
    • Transformation: Converting between types using transform or transformOrFail.
    • Data Manipulation: Converting to/from DynamicValue via toDynamic and fromDynamic.
    • Validation & Patching: Validating values with validate and applying changes via diff and patch.
    • Schema Evolution: Migrating data between schemas using migrate or coercing schemas with coerce.
    • Metadata: Accessing the schema's structure via ast or adding metadata via annotate.
    sealed trait Schema[A] {
      def ? : Schema[Option[A]]
      def <*>[B](that: Schema[B]): Schema[(A, B)]
      def <+>[B](that: Schema[B]): Schema[scala.util.Either[A, B]]
      def defaultValue: scala.util.Either[String, A]
      def annotations: Chunk[Any]
      def ast: MetaSchema 
      def annotate(annotation: Any): Schema[A]
      def coerce[B](newSchema: Schema[B]): Either[String, Schema[B]]
      def diff(thisValue: A, thatValue: A): Patch[A]
      def patch(oldValue: A, diff: Patch[A]): scala.util.Either[String, A]
      def fromDynamic(value: DynamicValue): scala.util.Either[String, A] 
      def makeAccessors(b: AccessorBuilder): Accessors[b.Lens, b.Prism, b.Traversal]
      def migrate[B](newSchema: Schema[B]): Either[String, A => scala.util.Either[String, B]]
      def optional: Schema[Option[A]]
      def ordering: Ordering[A]
      def orElseEither[B](that: Schema[B]): Schema[scala.util.Either[A, B]]
      def repeated: Schema[Chunk[A]]
      def serializable: Schema[Schema[A]]
      def toDynamic(value: A): DynamicValue
      def transform[B](f: A => B, g: B => A): Schema[B]
      def transformOrFail[B](f: A => scala.util.Either[String, B], g: B => scala.util.Either[String, A]): Schema[B]
      def validate(value: A)(implicit schema: Schema[A]): Chunk[ValidationError]
      def zip[B](that: Schema[B]): Schema[(A, B)]
    }
  2. What is a ZIO Schema and its core types

    main

    A Schema[A] is the core data type in ZIO Schema and is invariant in A. It is used to both produce and consume values of type A.

    ZIO Schemas are built from three primary sealed traits and one case class:

    • Primitive[A]: Represents scalar data types (the 'leaves' of a data tree).
    • Record[R]: Represents product types (data structures composed of multiple fields).
    • Enum[A]: Represents sum types (types that can be one of several different cases).
    • Sequence[Col, Elem]: Represents collections of elements.

    Key transformation operators include zip for combining schemas and transform for mapping values while maintaining schema integrity.

    sealed trait Schema[A] {
      def zip[B](that: Schema[B]): Schema[(A, B)]
      def transform[B](f: A => B, g: B => A): Schema[B]
    }
  3. What is a BinaryCodec and how does it differ from a Codec?

    main

    A BinaryCodec[A] is a specialized version of a Codec specifically designed for binary formats (where the output is Chunk[Byte] and the input elements are Byte).

    Beyond simple single-value encoding/decoding, a BinaryCodec[A] provides built-in support for streaming. It allows you to transform a stream of values of type A into a stream of bytes (and vice versa) using ZPipeline.

    Key capabilities of BinaryCodec[A]:

    • encode(value: A): Chunk[Byte]: Encodes a single value.
    • decode(whole: Chunk[Byte]): Either[DecodeError, A]: Decodes a single chunk.
    • streamEncoder: A ZPipeline to transform a stream of A into a stream of Byte.
    • streamDecoder: A ZPipeline to transform a stream of Byte into a stream of A.
    trait BinaryCodec[A] {
      def encode(value: A): Chunk[Byte]
      def decode(whole: Chunk[Byte]): Either[DecodeError, A]
    
      def streamEncoder: ZPipeline[Any, Nothing, A, Byte]
      def streamDecoder: ZPipeline[Any, DecodeError, Byte, A]
    }
  4. What is DynamicValue and how does it work?

    main

    A DynamicValue is a self-describing representation of data that combines both the actual values and their structural type information. Unlike relational models where schema and data are separate, DynamicValue bakes the structure into the data itself, similar to how JSON can be extended with type information.

    This allows you to introspect, manipulate, and migrate data structures dynamically without needing the original typed context immediately available.

  5. What is ZIO Schema?

    main

    ZIO Schema is a ZIO-based library that models the schema of data structures as first-class values. It reifies the concept of structure for data types, turning compiled-time constructs (types) into runtime constructs (values) that can be read, manipulated, and composed.

    Key capabilities include:

    • Codec Generation: Creating serialization and deserialization codecs for protocols like JSON, Protobuf, CSV, etc.
    • Metaprogramming: Deriving standard type classes (Eq, Show, Ordering) and default values from data structure shapes without relying on macros or reflection.
    • ETL Automation: Providing tools for diffing (comparing values), patching (applying updates), and migrating values between different types.
    • Distributed Computing: Enabling computations to be treated as data.
  6. What is the Transform type class in ZIO Schema?

    main

    In ZIO Schema, transformations are modeled by the Transform[A, B] type class. It encapsulates a codec for type A and two functions that handle the conversion between A and B while allowing for error reporting via Either[String, T].

    Transform[A, B] extends Schema[B].

    object Schema {
      final case class Transform[A, B](
          codec: Schema[A],
          f: A => Either[String, B],
          g: B => Either[String, A]
        ) extends Schema[B]
      }
    }
  7. What is DynamicValue and how to use it

    main

    A DynamicValue is a self-describing representation of data that combines both the value and its structural type information. Unlike relational models where schema and data are separate, a DynamicValue carries its own structure, making it useful for document-based models (like JSON) or for inspecting data without knowing the original type at compile time.

    You can create a DynamicValue from a typed value using DynamicValue.fromSchemaAndValue(schema, value) or by calling schema.toDynamic(value). To convert it back to a typed value, use DynamicValue#toTypedValue[A](implicit schema: Schema[A]).

    import zio.schema._
    
    case class Person(name: String, age: Int)
    object Person {
      implicit val schema: Schema[Person] = DeriveSchema.gen[Person]
    }
    
    val person = Person("John Doe", 42)
    // Create DynamicValue
    val dynamicPerson = DynamicValue.fromSchemaAndValue(Person.schema, person)
    // OR
    val dynamicPersonAlt = Person.schema.toDynamic(person)
    
    // Convert back to typed value
    val recoveredPerson: Either[String, Person] = dynamicPerson.toTypedValue[Person](Person.schema)
    // Note: toTypedValue requires an implicit schema for the target type A
  8. What is ZIO Schema and why use it?

    main

    ZIO Schema is a library designed to reify data structures. In this context, reification means transforming abstract concepts (like the structure of a data type) into first-class values that can be manipulated, composed, and used by other libraries.

    While ZIO reifies side-effects into values and Optics reify field access into values, ZIO Schema reifies the description of entire data structures into values. This allows you to treat the structure of a type (its name, fields, and field types) as data that can be used for serialization, validation, and integration with other systems like ZIO SQL, ZIO DynamoDB, or ZIO Redis.

  9. What is a ZIO Schema Codec

    main

    A Codec is a utility used to encode or decode a value of a specific type A to and from a specific format (such as JSON, binary, or XML).

    In ZIO Schema, unlike many other libraries, the Codec trait itself does not carry a type parameter. Instead, the encoding and decoding operations are parameterized by a Schema[A]. This allows a single codec instance to handle various schemas by providing the necessary structural information during the encoding/decoding process.

    trait Codec {
      def encoder[A](schema: Schema[A]): ZTransducer[Any, Nothing, A, Byte]
      def decoder[A](schema: Schema[A]): ZTransducer[Any, String, Byte, A]
    
      def encode[A](schema: Schema[A]): A => Chunk[Byte]
      def decode[A](schema: Schema[A]): Chunk[Byte] => Either[String, A]
    }
  10. How Schema#migrate works for mapping DTOs

    main

    The Schema#migrate method allows you to map one schema to another, which is useful for schema evolution or converting DTOs to domain objects without full serialization.

    When you call sourceSchema.migrate(targetSchema), it returns an Either[String, source => Either[String, target]].

    • If the migration logic is valid, you receive a Right containing a function. This function takes an instance of the source type and returns an Either[String, target], allowing for potential transformation errors.
    • If the migration logic is invalid, you receive a Left with an error message.

    This is particularly useful when you want to perform the transformation within a ZIO effect, handling potential errors gracefully.

    // Example of using migrate to transform a DTO to a Person
    def fromPersonDTO(p: PersonDTO): IO[String, Person] =
      ZIO.fromEither(
        PersonDTO.schema
          .migrate(personDTOMapperSchema) // Returns Either[String, PersonDTO => Either[String, Person]]
          .flatMap(_(p))                 // Applies the transformation function to the DTO
      )
  11. How ZIO Schema works

    main

    ZIO Schema reifies the structure of a data type, turning a compile-time construct (the type) into a runtime construct (a Schema value). This Schema value can be read, manipulated, and composed at runtime.

    By treating schemas as first-class values, you can:

    • Generate Codecs: Automatically create serialization/deserialization logic for protocols like JSON, Protobuf, CSV, etc.
    • Derive Type Classes: Derive standard type classes like Eq, Show, or Ordering from the data structure.
    • Automate ETL: Perform operations like diffing (comparing two values), patching (applying diffs), and migration (moving values between types).
    • Metaprogramming: Avoid macros or reflection by using the schema value to drive computations.