ProtoQuill

repository·master·Indexed 42 days ago

https://github.com/zio/zio-protoquill

A Scala 3 implementation of Quill providing compile-time language-integrated queries. It leverages Scala 3 metaprogramming, such as `inline` and `given`, to provide type-safe query logic for JDBC, ZIO, and Cassandra. Features include support for batch queries, Co-Product rows via enums and sealed traits, custom parsing for business logic, and integration with Caliban GraphQL for filter and predicate pushdown.

Tokens
8.6K
Snippets
26
Records
32
Agent score
53%

What's inside ProtoQuill

  1. Share code between Scala and ProtoQuill queries

    master

    Because quotation is optional for inline def expressions, you can write logic once and use it in both standard Scala collections and ProtoQuill queries. This allows for highly reusable and type-safe code.

    // Define logic once
    inline def onlyJoes(p: Person) = p.name == "Joe"
    
    // Use in ProtoQuill
    run( query[Person].filter(p => onlyJoes(p)) )
    
    // Use in standard Scala
    val people: List[Person] = ...
    val joes = people.filter(p => onlyJoes(p))
  2. How quotation and inline def work in ProtoQuill

    master

    ProtoQuill uses Scala 3 inline def to achieve compile-time query generation.

    1. Compile-Time Queries: Use inline def for your query expressions. If all parts of a query are inline def, the quote { ... } block is optional.
    2. Dynamic Queries: If you use standard def (not inline), the query becomes a Dynamic query, which is evaluated at runtime.
    3. Contexts: You do not need to import a context to create a quotation (only io.getquill._ is required). Contexts are only required when performing lifting (e.g., lift(value)).
    // Compile-time: quotation is optional if everything is inline
    inline def people = query[Person]
    inline def joes = people.filter(p => p.name == "Joe")
    
    // Dynamic: quotation is required if parts are not inline
    inline def people = quote { query[Person] }
    val joes = quote { people.filter(p => p.name == "Joe") }
  3. Implement Custom Parsing for Business Logic

    master

    ProtoQuill allows you to define custom parsing logic so that regular Scala methods (not just quoted ones) can be used within quote blocks. This prevents the need to define duplicate logic for both Scala and Quill.

    Workflow:

    1. Define Business Logic: Create an object containing the methods you want to use. Ensure the implementation is not inline in this object.
    2. Define a Parser: Implement a Parser.Clause[Ast] that intercepts the specific syntax (e.g., an infix operator) and translates it into a Quill Ast (like Infix).
    3. Register the Parser: Create a ParserLibrary that includes your custom parser in the operationsParser series.
    4. Use in Application: Define your custom parser as a given in your application code to enable the syntax within quote blocks.

    Important: The parser definition must reside in a separate compilation unit (e.g., a separate SBT project) that is compiled before your main application code.

    // 1. Define business logic (NOT inline)
    object MyBusinessLogic:
      extension (i: Int)
        def **(exponent: Int) = Math.pow(i, exponent)
    
    // 2. Define the parser (in a separate compilation unit)
    import io.getquill.parser._
    import io.getquill.ast.{ Ast, Infix }
    import io.getquill.quat.Quat
    
    case class CustomOperationsParser(root: Parser[Ast] = Parser.empty)(override implicit val qctx: Quotes) extends Parser.Clause[Ast] {
      import quotes.reflect._
      import CustomOps._
      def reparent(newRoot: Parser[Ast]) = this.copy(root = newRoot)
      def delegate: PartialFunction[Expr[_], Ast] =
        case '{ ($i: Int)**($j: Int) } =>
          Infix(
            List("power(", " ,", ")"),
            List(astParse(i), astParse(j)), true, Quat.Value)
    }
    
    object CustomParser extends ParserLibrary:
      import Parser._
      override def operationsParser(using qctx: Quotes) =
        Series.of(new OperationsParser, new CustomOperationsParser)
    
    // 3. Use in application
    given myParser: CustomParser.type = CustomParser
    import MyBusinessLogic._
    
    inline def q = quote { query[Person].map(p => p.age ** 2) }
  4. Implement Co-Product Rows using Enums and Sealed Traits

    master

    ProtoQuill supports Co-Products (sum types) using a "Table Per Class-Hierarchy" model, where all data for the co-product must be encodable within a single row.

    Requirements:

    1. Static Enums: Currently, all Enums used for Co-Product rows must be defined in a static namespace (e.g., inside an object).
    2. Sealed Traits: Any sealed traits used must be sealed in a separate object to ensure the sum-type mirror is found.

    Steps to implement:

    1. Define the Co-Product (e.g., an enum inside an object).
    2. Create a RowTyper (a given RowTyper[T]) that inspects a database row (e.g., via a type column) to determine which element of the co-product to decode.
    3. Run your query using quote and ctx.run.
    // 1. Create the Coproduct in a static namespace
    object StaticEnumExample {
      enum Shape(val id: Int):
        case Square(override val id: Int, width: Int, height: Int) extends Shape(id)
        case Circle(override val id: Int, radius: Int) extends Shape(id)
    }
    
    // 2. Create a row-typer to decode the row
    given RowTyper[Shape] with
      def apply(row: Row) =
        row.apply[String]("type") match
          case "square" => classTag[Shape.Square]
          case "circle" => classTag[Shape.Circle]
    
    // 3. Run the query
    inline def q = quote { query[Shape].filter(s => s.id == 18) }
    val result: List[Shape] = ctx.run(q)
  5. Integrate ProtoQuill with Caliban GraphQL

    master

    The quill-caliban module allows you to create Caliban GraphQL endpoints that support filter-pushdown (filtering by columns) and predicate-pushdown (including/excluding columns). This ensures that GraphQL requests only fetch and filter the data requested by the client at the database level.

    Key Methods:

    • .filterColumns(columns: List[String]): Limits the SQL SELECT clause to the specified columns.
    • .filterByKeys(filters: Map[String, String]): Adds WHERE clause conditions based on the provided key-value pairs.
    • quillColumns(productArgs): A helper from io.getquill.CalibanIntegration to extract requested columns from Caliban arguments.

    Implementation Pattern:

    1. Add .filterColumns and .filterByKeys to your quote block.
    2. In your Caliban resolver, use quillColumns(productArgs) to pass the requested columns and productArgs.keyValues to pass the filters to your data service.
    import io.getquill._
    import io.getquill.CalibanIntegration._
    
    // 1. Define query with pushdown methods
    inline def peopleAndAddresses(inline columns: List[String], inline filters: Map[String, String]) =
      quote {
        query[Person].leftJoin(query[Address]).on((p, a) => p.id == a.ownerId)
          .map((p, a) => PersonAddress(p.id, p.first, p.last, p.age, a.map(_.street)))
          .filterColumns(columns)
          .filterByKeys(filters)
      }
    
    // 2. Use in Caliban Resolver
    val endpoint = graphQL(
      RootResolver(
        Queries(
          personAddress => (productArgs => 
            DataService.personAddress(
              quillColumns(productArgs), // Extract columns from Caliban
              productArgs.keyValues      // Extract filters from Caliban
            )
          )
        )
      )
    ).interpreter
  6. Migrate from Scala2-Quill to ProtoQuill

    master

    When migrating from Scala2-Quill to ProtoQuill, observe the following changes:

    1. Inline Queries: ProtoQuill relies on inline def for compile-time queries. You must change val declarations to inline def for any quoted expressions intended to be compile-time queries.

      • Old: val people = quote { query[Person] }
      • New: inline def people = quote { query[Person] }
      • Note: Using val will result in a Dynamic Query.
    2. Imports: If you were previously importing methods from a context (e.g., import ctx._), you must now import from io.getquill. The core methods like query and quote are exported from the Dsl object to the io.getquill package.

      • Minimal required imports: io.getquill.quote and io.getquill.query.
  7. Install ProtoQuill via SBT

    master

    To use ProtoQuill, add the appropriate module to your SBT libraryDependencies. Choose the module based on your required database context (JDBC, ZIO, or Cassandra) and whether you need ZIO support or Caliban integration.

    libraryDependencies ++= Seq(
      // Syncronous JDBC Modules
      "io.getquill" %% "quill-jdbc" % "4.7.3",
      // Or ZIO Modules
      "io.getquill" %% "quill-jdbc-zio" % "4.7.3",
      // Or Cassandra
      "io.getquill" %% "quill-cassandra" % "4.7.3",
      // Or Cassandra + ZIO
      "io.getquill" %% "quill-cassandra-zio" % "4.7.3",
      // Add for Caliban Integration
      "io.getquill" %% "quill-caliban" % "4.7.3"
    )
  8. Use TypedEntityQuery for record-based queries

    master

    The TypedEntityQuery[T] API provides a way to build queries using Record[T, Col] for structural field access. This approach is designed to offer better compile times compared to the standard macro-based query[T] API by deriving schema information via type-level NamedTuple instead of heavy AST inspection.

    Key characteristics:

    • Field Access: Uses Record[T, Col] in lambdas to allow accessing fields of type T.
    • Result Type: The map operation currently only supports column selection or reordering within the same entity type T. It does not yet support projecting to a different type (e.g., a different case class).
    • Integration: Use .toQuoted to convert the builder into a Quoted[EntityQuery[T]], which can then be passed to existing ctx.run() infrastructure.
    // Conceptual usage pattern
    val query = TypedQueryMacro[User]
      .filter(r => r.id === 1)
      .sortBy(r => r.name)
      .take(10)
      .toQuoted
    
    ctx.run(query)
  9. Understand the ProtoContextSecundus abstraction

    master

    In ProtoQuill, ProtoContextSecundus serves as the foundational base for all other contexts. It provides a common API that allows Scala 3 macros to call low-level execution methods. It acts as a bridge to ensure that the internal APIs for both Scala 2-Quill and ProtoQuill remain consistent, preventing API drift.

    It is a RowContext that defines several associated types for query results, actions, and sessions, which are specialized by the specific implementation (e.g., a specific SQL dialect).

  10. Configure a JDBC DataSource

    master

    For JDBC contexts (e.g., Postgres), configure your database connection in application.conf. Example for Postgres:

    testPostgresDB.dataSourceClassName=org.postgresql.ds.PGSimpleDataSource
    testPostgresDB.dataSource.databaseName=<my-database>
    testPostgresDB.dataSource.url=<my-jdbc-url>
  11. Use `implicitly` to provide environment context to ZIO queries

    master

    When using Ctx.run(query), the resulting ZIO effect often requires a specific environment (like a DataSource or a Session) to execute. Instead of calling .provide(Has(ds)) manually for every single query, you can use the implicitly extension method.

    To use this, you must:

    1. Import io.getquill.context.qzio.Ctx._ and io.getquill.context.qzio.ImplicitSyntax._.
    2. Define an implicit val env = Implicit(Has(yourDependency)) in your scope.
    3. Call .implicitly on the query effect, passing the required Tag.

    This is particularly useful in JDBC contexts where you need to provide a DataSource, or in specialized drivers like quill-zio-cassandra where you only need to provide the session.

    import io.getquill.context.qzio.Ctx._
    import io.getquill.context.qzio.ImplicitSyntax._
    
    case class MyQueryService(ds: DataSource with Closeable) {
      // Define the implicit context wrapper
      implicit val env = Implicit(Has(ds))
    
      // Use .implicitly to automatically provide the environment to the query
      val joes = Ctx.run(query[Person].filter(p => p.name == "Joe")).onDataSource.implicitly
    }
  12. Configure database services via docker-compose.yml

    master

    The project uses a docker-compose.yml file to orchestrate a local development environment containing multiple database engines. You can use this file to spin up the necessary infrastructure for testing or development with zio-protoquill.

    Supported services include:

    • Postgres: Port 15432.
    • MySQL: Port 13306.
    • Cassandra: Port 19042.
    • OrientDB: Port 12424.
    • SQL Server: Port 11433.
    • Oracle: Port 11521.

    Note that some services like cassandra and orientdb have specific JVM memory settings configured to enable NativeMemoryTracking and manage CompressedClassSpaceSize for local builds.

    version: '2.2'
    
    services:
      postgres:
        ports:
          - "15432:5432"
        environment:
          - POSTGRES_USER=postgres
          - POSTGRES_HOST_AUTH_METHOD=trust
    
      mysql:
        ports:
          - "13306:3306"
        environment:
          - MYSQL_ROOT_PASSWORD=root
          - MYSQL_ALLOW_EMPTY_PASSWORD=yes
          - MYSQL_ROOT_HOST=%"
    
      cassandra:
        ports:
          - "19042:9042"
        environment:
          - MAX_HEAP_SIZE=256m
          - HEAP_NEWSIZE=64m
          - JVM_OPTS=-XX:NativeMemoryTracking=summary -XX:CompressedClassSpaceSize=200m
    
      sqlserver:
        ports:
          - "11433:1433"
        environment:
          - ACCEPT_EULA=Y
          - SA_PASSWORD=QuillRocks!