next.jdbc

repository·develop·Indexed 21 days ago

https://github.com/seancorfield/next-jdbc

A high-performance, low-level Clojure wrapper for JDBC-based database access designed as a modern successor to clojure.java.jdbc. It provides a streamlined API focusing on qualified keywords, transducers, and the IReduceInit approach. Key features include efficient result set streaming via `plan`, eager realization with `execute!`, and single-row retrieval with `execute-one!`. It supports flexible DataSource creation via db-specs and provides convenience wrappers for common SQL operations in the next.jdbc.sql namespace.

Tokens
29.7K
Snippets
76
Records
116
Agent score
74%

What's inside next.jdbc

  1. Migrate :identifiers and :qualifier to next.jdbc.result-set

    develop

    In next.jdbc, identifier and qualifier handling is managed via :builder-fn and next.jdbc.result-set/as-* functions.

    Replacing :identifiers

    • If you used :identifiers identity (default in clojure.java.jdbc), use as-unqualified-maps.
    • If you used a custom string transformation, use as-unqualified-modified-maps and pass your transformation function as the :label-fn option.

    Replacing :qualifier

    • To achieve the effect of :qualifier, use as-modified-maps and pass :qualifier-fn (constly "your_qualifier") along with the appropriate :label-fn (e.g., identity or clojure.string/lowercase).
  2. Use friendly SQL functions for basic CRUD

    develop

    The next.jdbc.sql namespace provides "friendly" functions for common CRUD (Create, Read, Update, Delete) operations. These functions simplify basic SQL tasks by accepting table names as keywords and using Clojure data structures (like maps and vectors) instead of requiring you to write full SQL strings for every operation.

    Available functions:

    • insert! and insert-multi! (Create)
    • query (Read - an alias for execute! when using a vector of SQL and parameters)
    • update! (Update)
    • delete! (Delete)
    • find-by-keys and get-by-id (Specific Read operations)
    • aggregate-by-keys (Aggregate Read operation)

    Note: These functions are intended for simple, common operations. For complex queries, it is recommended to use the primary API (plan, execute!, execute-one!) or a SQL DSL like HoneySQL.

    ;; Example of the pattern
    (sql/insert! ds :table-name {:column "value"})
    (sql/update! ds :table-name {:column "new-value"} {:id 1})
  3. Handle SQL reserved names and quoting

    develop

    By default, next.jdbc.sql functions use the provided keywords exactly as-is. If your table or column names are SQL reserved words, you must use quoting functions via the :table-fn and :column-fn options.

    Quoting Functions (next.jdbc.quoted):

    • ansi, postgres, oracle: Double quotes ("name")
    • mysql: Back ticks (`name`)
    • sql-server: Square brackets ([name])
    • schema: A modifier to support schema-qualified names (e.g., (schema sql-server) produces [dbo].[table]).

    Pre-configured Options:

    • jdbc/snake-kebab-opts: Uses camel-snake-kebab to transform keywords to snake_case for both tables and columns, and performs the reverse transformation on results.
    • jdbc/unqualified-snake-kebab-opts: Similar to above, but for unqualified names.
    ;; Using MySQL backtick quoting
    (sql/insert! ds :my-table {:some "data"} {:table-fn next.jdbc.quoted/mysql})
    
    ;; Using snake_case transformations
    (sql/insert! ds :my-table {:some-data "val"} jdbc/snake-kebab-opts)
    
    ;; Using schema-qualified SQL Server quoting
    (sql/insert! ds :dbo.table {:col "val"} {:table-fn (next.jdbc.quoted/schema next.jdbc.quoted/sql-server)})
  4. Choose the right SQL execution strategy

    develop

    The API is designed around three specific usage scenarios:

    • Streaming/Efficient Reduction: Use plan when you want to process rows one-by-one as they are read from the database. This is the fastest approach and allows for resource cleanup even if the reduction is short-circuited via reduced.
    • Single Row/DDL: Use execute-one! when you need exactly one result (like a single user record) or when performing DDL/updates where you only care about the first result or the update count.
    • Eager Result Sets: Use execute! when you want a full, realized vector of all rows in memory. This is useful for smaller datasets where you want to work with the entire collection at once.
  5. Understand the motivation for next.jdbc

    develop

    next.jdbc is a low-level Clojure wrapper for JDBC designed to improve upon clojure.java.jdbc in three key areas:

    1. Performance: Reduces overhead in converting ResultSet objects to sequences of hash maps and simplifies the db-spec parsing logic.
    2. Modern API: Focuses on using qualified keywords, transducers, and the IReduceInit approach to provide a streamlined, consistent interface.
    3. Simplicity: Replaces the inconsistent execution methods (like query, execute!, and db-do-commands) with a more predictable, protocol-based design. It also provides a clearer path from db-spec to DataSource and Connection to encourage better connection reuse.

    Additionally, next.jdbc includes built-in datafy/nav support as the default behavior for execute! and execute-one!.

  6. Handle Result Sets and Row Functions in next.jdbc

    develop

    Unlike clojure.java.jdbc, next.jdbc never exposes result sets lazily.

    When you call execute! or execute-one!, you receive a fully-realized data structure. If you need to process large result sets without loading everything into memory, you must use the plan approach with reduce or transducing functions to process the stream eagerly.

    Note on early termination: You can terminate a reduction early by wrapping the final value in the reduced function.

  7. Efficiently process large result sets with `plan`

    develop

    For high-performance data processing, use jdbc/plan instead of execute!. plan returns an IReduceInit object that can be used with reduce, transduce, or into.

    Key Advantages:

    • Low Overhead: It avoids creating Clojure hash maps for every row. Instead, it provides an abstraction over the underlying ResultSet. You can access columns via their SQL labels (e.g., (:unit_price row)) directly.
    • Automatic Resource Management: The database connection is opened when the reduction starts and closed automatically when the reduction completes.

    Important Notes:

    • The row object in a reduction is not a standard Clojure map. Using functions like assoc, keys, or vals on it will force the row to be fully realized into a hash map, negating the performance benefits.
    • To get a fully realized, navigable hash map within a reduction, use next.jdbc.result-set/datafiable-row.
    • Do not use plan for DDL or statements that only return update counts; it is intended for result set processing.
    ;; Using reduce with plan
    (reduce (fn [acc row] (+ acc (:unit_price row) (:unit_count row))) 
            0 
            (jdbc/plan ds ["SELECT unit_price, unit_count FROM invoice"]))
    
    ;; Using transduce with plan
    (transduce (map #(apply * [% :unit_price :unit_count])) 
               + 
               0 
               (jdbc/plan ds ["SELECT unit_price, unit_count FROM invoice"]))
    
    ;; Using into with plan
    (into #{} (map :product) (jdbc/plan ds ["SELECT product FROM invoice"]))
  8. How next.jdbc.datafy affects result-set metadata

    develop

    Requiring the next.jdbc.datafy namespace changes the behavior of next.jdbc.result-set/metadata when used inside the reducing function of a plan call:

    • Without next.jdbc.datafy: The function returns a raw java.sql.ResultSetMetaData object. This object must not leak outside the reducing function.
    • With next.jdbc.datafy: The function returns a Clojure data structure describing the columns in the result set instead of the raw Java object.
  9. Best practices for Times, Dates, and Timezones

    develop

    To minimize confusion when working with databases and JDBC, follow these best practices:

    1. Store in UTC: Always store dates and timestamps in UTC. In Postgres, use TIMESTAMP (without time zone) and ensure the application sends UTC values.
    2. Application-level Timezones: Treat timezone conversion and logic as an application concern rather than a database concern.
    3. Consistent Libraries: When using libraries like clojure.java-time, save (java.time/instant) into timestamp columns to maintain consistency.
    4. System Settings: For maximum safety, set your database and servers to the UTC timezone.
  10. Understand the RowBuilder Protocol

    develop

    The RowBuilder protocol is used by next.jdbc to materialize a single row from a ResultSet into a Clojure data structure. The following functions are part of the protocol:

    • (->row builder): Produces a new row (defaults to (transient {})).
    • (column-count builder): Returns the number of columns in the row.
    • (with-column builder row i): Fetches column i from the ResultSet, converts it to a Clojure value, and adds it to the row. For as-maps, this involves read-column-by-index and assoc!.
    • (with-column-value builder row col v): A low-level utility to add a column name col and value v to the row. Builders/adapters should generally use this to control value handling.
    • (row! builder row): Completes the row (defaults to (persistent! row)).

    execute!, execute-one!, and plan utilize these functions to build the resulting data.

  11. Understand JDBC Datafication with next.jdbc.datafy

    develop

    By requiring next.jdbc.datafy, the Datafiable protocol is extended to several JDBC object types. This allows you to call datafy on these objects to convert them into Clojure hash maps using Java Bean introspection (via clojure.java.data/from-java-shallow).

    Key datafied types include:

    • java.sql.Connection: Datafies as a bean. The :metaData property returns a datafiable java.sql.DatabaseMetaData object.
    • DatabaseMetaData: Datafies as a bean with an additional :all-tables property. It includes six navigable properties that produce fully-realized datafiable result sets:
      • :all-tables (via .getTables)
      • :catalogs (via .getCatalogs)
      • :clientInfoProperties (via .getClientInfoProperties)
      • :schemas (via .getSchemas)
      • :tableTypes (via .getTableTypes)
      • :typeInfo (via .getTypeInfo)
    • ParameterMetaData: Datafies as a vector of parameter description maps. Each map contains: :class, :mode (:in, :in-out, or :out), :nullability (:null, :not-null, or :unknown), :precision, :scale, :type, and :signed.
    • ResultSet: Datafies as a bean. If the ResultSet is associated with a Statement and a Connection, an additional :rows key is provided containing a datafied result set (using next.jdbc.result-set/datafiable-result-set with default options).
    • ResultSetMetaData: Datafies as a vector of column description maps. Each map contains metadata such as :auto-increment, :case-sensitive, :catalog, :class, :currency, :definitely-writable, :display-size, :label, :name, :nullability, :precision, :read-only, :searchable, :signed, :scale, :schema, :table, :type, and :writable.
    • Statement: Datafies as a bean.