DOCSEARCHbyAlgolia
Submit Docs

Migratus Documentation

repository·master·Indexed 20 days ago

https://github.com/yogthos/migratus

A general-purpose migration framework for Clojure supporting both SQL scripts and arbitrary Clojure code. It uses 14-digit timestamps instead of incrementing integers to prevent collisions in distributed teams. Features include support for transactional DDL, property substitution, SQL assertions, and a programmatic API via migratus.core. It can be managed through a CLI, a Leiningen plugin (migratus-lein), or Clojure deps.edn aliases.

Tokens
4K
Snippets
17
Records
19
Agent score
22%

What's inside Migratus

  1. How code-based migrations work

    master

    Code-based migrations allow you to run Clojure functions instead of raw SQL. This is useful for complex logic like data backfilling or dynamic DDL.

    Implementation Steps:

    1. Create an EDN descriptor file in your migrations directory (e.g., YYYYMMDDHHMMSS-name.edn). This file maps the migration to a namespace and specific functions.
    2. Implement the functions in a Clojure namespace. The up and down functions must accept a single argument: the Migratus config map.

    EDN Descriptor Format:

    • :ns: The namespace containing the migration functions.
    • :up-fn: The function to run for the 'up' migration. Can be a symbol or a vector [function extra-arg1 extra-arg2].
    • :down-fn: The function to run for the 'down' migration.
    • :transaction?: Boolean to enable/disable transactions (defaults to true).

    Note: You can mix SQL and code-based migrations in the same directory; they will execute in the order of their timestamps.

    ;; resources/migrations/20170331141500-import-users.edn
{:ns app.migrations.import-users
 :up-fn migrate-up
 :down-fn migrate-down
 :transaction? true}

;; src/app/migrations/import_users.clj
(ns app.migrations.import-users)

(defn migrate-up [config]
   ;; logic here)

(defn migrate-down [config]
   ;; logic here)

  2. Quickstart with Clojure (deps.edn)

    master

    To use Migratus in a native Clojure project using deps.edn, follow these steps:

    1. Add the clj-migratus alias to your deps.edn.
    2. Create a configuration file (migratus.edn or migratus.clj). It is recommended to use .clj to allow environment variable substitution for credentials.
    3. Run commands via clj -M:migrate <command>.

    Example deps.edn configuration:

    :aliases {:migrate {:extra-deps {com.github.paulbutcher/clj-migratus {:git/tag "v1.0.0"
                                                                      :git/sha "67d0fe5"}}
                     :main-opts ["-m" "clj-migratus"]}}

    Example migratus.clj using environment variables:

    {:store :database
 :db {:jdbcUrl (get (System/getenv) "JDBC_DATABASE_URL")}}

    Example commands:

    $ clj -M:migrate init
$ clj -M:migrate migrate
$ clj -M:migrate create create-user-table

  3. Use Migratus with Leiningen

    master

    Migratus provides a Leiningen plugin (migratus-lein) for CLI-based management.

    1. Add migratus-lein and migratus to your project.clj dependencies.
    2. Add a :migratus configuration key to your project.clj.

    Commands:

    • lein migratus migrate: Apply pending migrations.
    • lein migratus rollback: Rollback the last migration.
    ;; project.clj
:plugins [[migratus-lein "<VERSION>"]]
:dependencies [[migratus "<VERSION>"]]

:migratus {:store :database
           :migration-dir "migrations"
           :db {:dbtype "mysql"
                :dbname "//localhost/migratus"
                :user "root"
                :password ""}}

  4. Disable transactions for a specific migration

    master

    Migratus runs migrations within a transaction by default. If your database does not support transactional DDL, add the following line to the very start of your migration file:

    -- :disable-transaction

    -- :disable-transaction
CREATE TABLE foo (id INT);

  5. Quick Start with SQL Migrations

    master

    To use Migratus for SQL-based migrations:

    1. Add the migratus dependency to your project.
    2. Create a migration directory (e.g., resources/migrations/).
    3. Create .up.sql and .down.sql files using a 14-digit timestamp prefix to avoid collisions in distributed teams.

    Example file structure:

    • resources/migrations/20111206154000-create-foo-table.up.sql containing CREATE TABLE IF NOT EXISTS foo(id BIGINT);
    • resources/migrations/20111206154000-create-foo-table.down.sql containing DROP TABLE IF EXISTS foo;
    -- resources/migrations/20111206154000-create-foo-table.up.sql
CREATE TABLE IF NOT EXISTS foo(id BIGINT);

-- resources/migrations/20111206154000-create-foo-table.down.sql
DROP TABLE IF EXISTS foo;

  6. Execute multiple SQL statements in a single migration

    master

    Because JDBC does not support sending multiple SQL commands in one execution, Migratus requires you to separate multiple statements in a single .sql file using the --;; delimiter. Migratus will split these commands and attempt to execute them within a transaction.

    Note: Databases like MySQL that do not support transactional DDL will not be able to roll back all statements if one fails.

    CREATE TABLE IF NOT EXISTS quux(id bigint, name varchar(255));
--;;
CREATE INDEX quux_name on quux(name);

  7. Configure Migratus for Database Migrations

    master

    Migratus is configured using a configuration map. To run migrations against a database, set :store to :database.

    Key Configuration Options

    • :migration-dir: The directory on the classpath containing SQL migration files. Files must follow the pattern [id]-[name].[direction].sql (e.g., 202310271000-create-users.up.sql).
    • :db: A next.jdbc database spec, a java.sql.Connection, or a javax.sql.DataSource.
    • :exclude-scripts: A collection of script name globs to exclude from migrations.
    • :command-separator: The separator used to split commands within a transaction.
    • :expect-results?: If true, allows using -- expect n comments in SQL to assert the number of rows affected.
    • :migration-table-name: Custom name for the migration tracking table (defaults to schema_migrations).
    • :init-script: A string pointing to a script to run during database initialization.
    • :init-in-transaction?: Defaults to true. Set to false if your database does not support schema initialization within a transaction.
    • :tx-handles-ddl?: If true, skips the automatic down migration that occurs on an exception.
    {:store :database
 :migration-dir "migrations"
 :exclude-scripts ["*.clj"]
 :db {:dbtype "mysql"
      :dbname "migratus"
      :user "root"
      :password ""}}

  8. Use property substitution in SQL migrations

    master

    Migratus can replace placeholders in the format ${property.name} with values from the environment or a provided map. This feature is enabled by setting the :properties flag in your configuration.

    • Environment Variables: Shell variables are normalized (e.g., FOO_BAR becomes foo.bar).
    • Default Properties: migratus.schema, migratus.user, migratus.database, and migratus.timestamp.
    • Customization: Use :env to specify environment variable keys or :map to provide a custom property map.
    {:store :database
 :properties {:env ["database.table"]
              :map {:database {:user "bob"}}}
 :db {:dbtype   "h2"
      :dbname   "site.db"}}

    Template usage:

    GRANT SELECT,INSERT ON ${database.table} TO ${database.user};

  9. Use SQL assertions with :expect-results?

    master

    When :expect-results? is enabled in your configuration, you can add assertions to your SQL migration files to verify that a specific number of rows were affected by a statement. Use the -- expect n syntax.

    -- expect 17
update foobar set thing = 'c' where thing = 'a';

--;;

-- expect 1
delete from foobar where thing = 'c';

  10. Programmatic API for Migratus

    master

    You can use Migratus programmatically by calling functions in the migratus.core namespace.

    FunctionDescription
    migratus.core/initRuns an initialization script (e.g., creating a new schema).
    migratus.core/createCreates a new migration file with the current date.
    migratus.core/migrateRuns up for all pending migrations. Returns nil on success or :ignore if the table is reserved.
    migratus.core/rollbackRuns down for the most recent migration.
    migratus.core/rollback-until-just-afterRuns down for all migrations after a specific migration-id.
    migratus.core/upRuns up for specific migration IDs (skips already applied ones).
    migratus.core/downRuns down for specific migration IDs (skips already reverted ones).
    migratus.core/resetResets the database by running down on all migrations, then up on all.
    migratus.core/pending-listReturns a list of migrations that have not yet been applied.
    migratus.core/migrate-until-just-beforeRuns up for pending migrations preceding a specific ID (useful for testing).
    migratus.core/squashing-listReturns a list of migrations between from-id and to-id (inclusive) to be squashed.
    migratus.core/create-squashGenerates a new squashed migration file from a range of IDs and removes the originals.
    migratus.core/squash-betweenMarks old IDs as squashed in the migration table and replaces them with a new ID (no migration applied).

  11. Customize SQL execution with :modify-sql-fn

    master
    If you need to process SQL strings before execution (for example, when using extensions like pglogical that require DDL to be called via specific functions), provide a :modify-sql-fn in your configuration. This function should accept a SQL string and return either a single modified SQL string or a sequence of SQL strings.

  12. Generate migration files

    master

    Use migratus.core/create to automatically generate the .up.sql and .down.sql files with the current timestamp prefix.

    ;; Create SQL migration
(migratus/create config "create-user")

;; Create EDN (code-based) migration
(migratus/create config "import-users" :edn)
    (migratus/create config "create-user")