Strong Migrations

repository·master·Indexed 26 days ago

https://github.com/ankane/strong_migrations

A tool for Ruby on Rails applications that detects and prevents potentially dangerous database migrations during development to avoid production downtime. It supports PostgreSQL, MySQL, and MariaDB, identifying operations that could block reads/writes or cause application errors. The library provides a `safety_assured` block to bypass checks and offers guidance on safely removing columns, changing types, renaming tables, adding foreign keys, and performing concurrent index creation in Postgres.

Tokens
6.7K
Snippets
25
Records
42
Agent score
89%

What's inside strong_migrations

  1. Configure App Timeouts in database.yml

    master

    To prevent connections from hanging and individual queries from consuming excessive resources, add timeouts to your config/database.yml.

    Postgres: Set connect_timeout, statement_timeout, and lock_timeout. Note: If using a connection pooler like PgBouncer in transaction mode, set statement_timeout and lock_timeout on the database user instead.

    MySQL: Set connect_timeout, read_timeout, write_timeout, max_execution_time (in ms), and lock_wait_timeout (in seconds).

    MariaDB: Set connect_timeout, read_timeout, write_timeout, max_statement_time (in seconds), and lock_wait_timeout (in seconds).

    # Postgres example
    production:
      connect_timeout: 5
      variables:
        statement_timeout: 15s
        lock_timeout: 10s
    
    # MySQL example
    production:
      connect_timeout: 5
      read_timeout: 5
      write_timeout: 5
      variables:
        max_execution_time: 15000 # ms
        lock_wait_timeout: 10 # sec
    
    # MariaDB example
    production:
      connect_timeout: 5
      read_timeout: 5
      write_timeout: 5
      variables:
        max_statement_time: 15 # sec
        lock_wait_timeout: 10 # sec
  2. Add indexes concurrently in Postgres

    master

    In Postgres, adding an index non-concurrently blocks writes. To avoid this, use algorithm: :concurrently and you must call disable_ddl_transaction! in your migration. Note that indexes on new tables created within the same migration do not require this.

    class AddSomeIndexToUsers < ActiveRecord::Migration[8.1]
      disable_ddl_transaction!
    
      def change
        add_index :users, :some_column, algorithm: :concurrently
      end
    end
  3. Safely rename a column or table

    master

    Renaming columns or tables in use causes application errors. Use the following deployment pattern:

    For Columns:

    1. Create a new column.
    2. Write to both columns.
    3. Backfill data from the old column to the new column.
    4. Move reads from the old column to the new column.
    5. Stop writing to the old column.
    6. Drop the old column.

    For Tables:

    1. Create a new table.
    2. Write to both tables.
    3. Backfill data from the old table to the new table.
    4. Move reads from the old table to the new table.
    5. Stop writing to the old table.
    6. Drop the old table.
  4. Safely add a check constraint

    master

    Adding a check constraint blocks reads and writes while rows are checked.

    Postgres Strategy:

    1. Add the constraint without validating existing rows using validate: false.
    2. Validate the constraint in a separate migration using validate_check_constraint.

    MySQL/MariaDB: Currently, enforcing check constraints blocks writes. There is no recommended safe way provided in the documentation for this database engine.

    # Postgres: Step 1
    class AddCheckConstraint < ActiveRecord::Migration[8.1]
      def change
        add_check_constraint :users, "price > 0", name: "price_check", validate: false
      end
    end
    
    # Postgres: Step 2
    class ValidateCheckConstraint < ActiveRecord::Migration[8.1]
      def change
        validate_check_constraint :users, name: "price_check"
      end
    end
  5. Safely add a foreign key

    master

    Adding a foreign key normally blocks writes on both tables.

    Postgres Strategy:

    1. Add the foreign key without validating existing rows using validate: false.
    2. Validate the foreign key in a separate migration using validate_foreign_key.

    MySQL/MariaDB Strategy: If you are certain all rows are valid and not using a connection pooler, wrap the operation in a safety_assured block and temporarily disable foreign_key_checks.

    # Postgres: Step 1
    class AddForeignKeyOnUsers < ActiveRecord::Migration[8.1]
      def change
        add_foreign_key :users, :orders, validate: false
      end
    end
    
    # Postgres: Step 2
    class ValidateForeignKeyOnUsers < ActiveRecord::Migration[8.1]
      def change
        validate_foreign_key :users, :orders
      end
    end
    
    # MySQL/MariaDB
    class AddForeignKeyOnUsers < ActiveRecord::Migration[8.1]
      def up
        safety_assured do
          begin
            execute "SET SESSION foreign_key_checks = 0"
            add_foreign_key :users, :orders
          ensure
            execute "SET SESSION foreign_key_checks = 1"
          end
        end
      end
    
      def down
        remove_foreign_key :users, :orders
      end
    end
  6. Add a JSONB column instead of JSON in Postgres

    master

    In Postgres, the json column type lacks an equality operator, which can cause errors for existing SELECT DISTINCT queries. Use jsonb instead for better performance and compatibility.

    class AddPropertiesToUsers < ActiveRecord::Migration[8.1]
      def change
        add_column :users, :properties, :jsonb
      end
    end
  7. Add a column with a volatile default value safely

    master

    Adding a column with a volatile default (e.g., gen_random_uuid()) to an existing table causes a full table rewrite, blocking reads and writes. Instead, add the column without a default, change the default in a separate step, and then backfill the data.

    class AddSomeColumnToUsers < ActiveRecord::Migration[8.1]
      def up
        add_column :users, :some_column, :uuid
        change_column_default :users, :some_column, "gen_random_uuid()"
      end
    
      def down
        remove_column :users, :some_column
      end
    end
  8. Rename an enum value safely

    master

    Renaming an enum value in use causes application errors. Instead: 1. Add the new enum value. 2. Update application code to handle both values and write the new value. 3. Backfill data from the old value to the new value.

    class AddCompletedToStatus < ActiveRecord::Migration[8.1]
      def up
        add_enum_value :status, "completed", after: "done"
      end
    end
  9. Add references concurrently in Postgres

    master

    Rails adds indexes non-concurrently to references by default, which blocks writes in Postgres. To prevent this, pass the concurrent algorithm in the index options.

    class AddReferenceToUsers < ActiveRecord::Migration[8.1]
      disable_ddl_transaction!
    
      def change
        add_reference :users, :city, index: {algorithm: :concurrently}
      end
    end
  10. Safely remove a column

    master

    Dropping a column directly can cause application errors because Active Record caches column information. To remove a column safely, follow these steps:

    1. Ignore the column in your model: Add the column to the ignored_columns list to prevent Active Record from using it.
    2. Deploy the code change.
    3. Run the migration: Use a safety_assured block in your migration to bypass the check.
    4. Deploy and run the migration.
    5. Remove the ignored column line from your model code.
    # 1. In your model
    class User < ApplicationRecord
      self.ignored_columns += ["some_column"]
    end
    
    # 3. In your migration
    class RemoveSomeColumnFromUsers < ActiveRecord::Migration[8.1]
      def change
        safety_assured { remove_column :users, :some_column }
      end
    end
  11. Backfill data safely

    master

    Backfilling data within a standard migration transaction keeps the table locked for the duration of the backfill. To backfill safely, follow these principles:

    1. Disable DDL transactions: Use disable_ddl_transaction! in your migration.
    2. Batching: Use in_batches to process records in chunks.
    3. Throttling: Use sleep between batches to avoid overwhelming the database.
    4. Reset Column Information: If using methods other than update_all, call User.reset_column_information to ensure the model has updated schema info.

    Note: Strong Migrations does not automatically detect dangerous backfills.

    class BackfillSomeColumn < ActiveRecord::Migration[8.1]
      disable_ddl_transaction!
    
      def up
        User.unscoped.in_batches(of: 10000) do |relation|
          relation.where(some_column: nil).update_all some_column: "default_value"
          sleep(0.01) # throttle
        end
      end
    end
  12. Add unique constraints concurrently in Postgres

    master

    Adding a unique constraint directly creates a unique index that blocks reads and writes. The safe pattern is to create a unique index concurrently first, then apply the constraint using that index.

    class AddUniqueConstraint < ActiveRecord::Migration[8.1]
      def up
        add_index :users, :some_column, unique: true, algorithm: :concurrently
        add_unique_constraint :users, using_index: "index_users_on_some_column"
      end
    
      def down
        remove_unique_constraint :users, :some_column
      end
    end