DOCSEARCHbyAlgolia
Submit Docs

OnlineMigrations Documentation

repository·master·Indexed 20 days ago

https://github.com/fatkodima/online_migrations

A tool for Ruby on Rails and PostgreSQL designed to detect unsafe migrations during development and provide safe patterns for production. It prevents dangerous operations that block database access or cause application errors, offering helpers for safely removing columns, adding columns with default values, backfilling data in batches, changing column types, renaming tables and columns, and managing indexes and constraints concurrently.

Tokens
20.8K
Snippets
87
Records
93
Agent score
73%

What's inside OnlineMigrations

  1. Compare OnlineMigrations with strong_migrations

    master

    While online_migrations shares the same APIs as strong_migrations, it provides several advanced features:

    1. Code Helpers: Instead of just providing text guidance, it provides actual migration helpers for renaming tables/columns, changing column types (e.g., integer to bigint), adding columns with default values, backfilling data, and adding constraints.
    2. Background Migrations: Includes a framework for running both data migrations and schema migrations in the background on very large tables.
    3. Enhanced Checks: Implements more checks for unsafe changes.
    4. PostgreSQL Focus: Currently optimized specifically for PostgreSQL.
    5. Flexible Configuration: Offers more granular configuration options.

  2. Best practices for writing Data Migrations

    master

    When writing data migrations, adhere to these two principles to ensure system stability:

    1. Isolation: Do not use application code (like models defined in app/models). Because migrations run over long periods, the application code they depend on might change or be removed in newer deployments while the migration is still running.
    2. Idempotence: The process method must be safe to run multiple times on the same item. If a migration fails halfway through, it may be retried, and the same elements might be processed again. Ensure your logic guarantees data integrity even during retries.

  3. How OnlineMigrations detects and handles dangerous operations

    master

    OnlineMigrations classifies an operation as dangerous if it meets either of these criteria:

    1. It blocks reads or writes for more than a few seconds after a lock is acquired.
    2. It has a high probability of causing application errors.

    When a dangerous operation is detected during a migration, the gem prevents it from running by default and provides a detailed error message in the console. This message includes specific instructions and code helpers (such as safety_assured) to perform the operation safely.

    For example, when dropping a column, the gem recommends a multi-step deployment process:

    1. Add the column to ignored_columns in the model.
    2. Deploy the code.
    3. Wrap the remove_column call in a safety_assured { ... } block within the migration.
    4. Deploy.
    5. Remove the ignored_columns configuration and deploy again.
    # Example of using the safety_assured helper provided in error messages
class RemoveColumn < ActiveRecord::Migration[8.0]
  def change
    safety_assured { remove_column :users, :name }
  end
end

  4. Monitor migration states

    master

    Background Schema Migrations can exist in the following states:

    • pending: Created by the user but not yet started.
    • running: Currently being executed by a migration executor.
    • errored: Raised an error during the last run.
    • failed: Raised an error and exceeded maximum retry attempts.
    • succeeded: Finished without error.
    • cancelled: Cancelled by the user.
    • delayed: Created with delay: true, waiting for user approval.

  5. Implement custom enumerators for external resources

    master

    If you need to iterate over data not stored in your database (e.g., an external API), implement the build_enumerator(cursor:) method.

    This method must return an Enumerator that yields pairs of [item, cursor]. The cursor is passed as a String. Online Migrations persists this cursor so that if the migration is interrupted, it can resume from the last known position.

    module OnlineMigrations
  module DataMigrations
    class CustomEnumeratorMigration < OnlineMigrations::DataMigration
      def build_enumerator(cursor:)
        after_id = cursor&.to_i
        # Yield pairs of [item, cursor_value]
        PostAPI.index(after_id: after_id).map { |post| [post, post.id] }.to_enum
      end

      def process(post)
        Post.create!(post)
      end
    end
  end
end

  6. Implement migration throttling

    master

    To prevent background migrations from taxing your database, you can define a throttling mechanism. If the condition provided to the throttler block evaluates to true, the migration will be interrupted and retried during the next Scheduler cycle run.

    Common use cases include checking PostgreSQL replication lag, DB thread counts, or general database health.

    # config/initializers/online_migrations.rb
OnlineMigrations.config.throttler = -> { DatabaseStatus.unhealthy? }

  7. Add indexes concurrently

    master

    Adding indexes non-concurrently blocks writes. To add an index without blocking, use the algorithm: :concurrently option.

    Important: You must call disable_ddl_transaction! in the migration.

    class AddIndexOnUsersEmail < ActiveRecord::Migration[8.0]
  disable_ddl_transaction!

  def change
    add_index :users, :email, unique: true, algorithm: :concurrently
  end
end

  8. Rename a column safely

    master

    Renaming a column in use causes application errors. OnlineMigrations uses a database VIEW and column aliasing to allow both old and new names to work simultaneously.

    Implementation Steps:

    1. Configure Rails: Set OnlineMigrations.config.column_renames and enable partial writes/inserts in config/application.rb.
      • For AR >= 7: config.active_record.partial_inserts = true
      • For AR < 7: config.active_record.partial_writes = true
    2. Initialize Rename: Use initialize_column_rename in a migration.
      initialize_column_rename :users, :name, :first_name
    3. Deploy: Update codebase to use the new name.
    4. Ignore Old Column: If using ignored_columns, add the old name to the list and deploy.
    5. Finalize Rename: Remove the config and use finalize_column_rename to drop the VIEW and rename the actual column.
      finalize_column_rename :users, :name, :first_name
    # config/application.rb
config.active_record.partial_inserts = true

# Migration
class InitializeRenameUsersNameToFirstName < ActiveRecord::Migration[8.0]
  def change
    initialize_column_rename :users, :name, :first_name
  end
end

# Finalize
class FinalizeRenameUsersNameToFirstName < ActiveRecord::Migration[8.0]
  def change
    finalize_column_rename :users, :name, :first_name
  end
end

  9. Upgrade to online_migrations v0.27.0

    master

    Upgrading to v0.27.0 involves a significant refactor of background data migration internals. The gem now relies on Sidekiq's Iteration feature, making Sidekiq 7.3.3+ a hard requirement for background data migrations to function.

    Standard Upgrade Steps

    1. Update your Gemfile to use version ~> 0.27.0.
    2. Update your initializer in config/online_migrations.rb to match the new template.

    If you do not use background data migrations or background schema migrations, these steps are sufficient.

    gem 'online_migrations', '~> 0.27.0'

  11. Install and set up Background Schema Migrations

    master

    To use background schema migrations, first generate the necessary migration files using the Rails generator.

    After installation, you must run a scheduler to execute the migrations. The scheduler performs only one migration at a time to prevent database overload. If you have multiple enqueued migrations or shards, you must call the runner method multiple times.

    Important: Ensure the scheduler process does not terminate until the migration is complete.

    $ bin/rails generate online_migrations:install

    Example: Using the whenever gem to run the scheduler every minute

    every 1.minute do
  runner "OnlineMigrations.run_background_schema_migrations"
end

    Example: Running the scheduler manually from Rails console

    OnlineMigrations.run_background_schema_migrations

  12. Enqueue a Data Migration via ActiveRecord Migration

    master

    To start a background migration, create a standard ActiveRecord migration and use the enqueue_background_data_migration helper in the up method. Use remove_background_data_migration in the down method to clean up.

    Passing Arguments: If your data migration class requires arguments in its initialize method, pass them as additional parameters to the enqueue/remove helpers.

    # Enqueueing a simple migration
class EnqueueBackfillProjectIssuesCount < ActiveRecord::Migration[8.0]
  def up
    enqueue_background_data_migration("BackfillProjectIssuesCount")
  end

  def down
    remove_background_data_migration("BackfillProjectIssuesCount")
  end
end

# Enqueueing with custom arguments
class EnqueueMyMigration < ActiveRecord::Migration[8.0]
  def up
    enqueue_background_data_migration("MyMigrationWithArgs", arg1, arg2)
  end

  def down
    remove_background_data_migration("MyMigrationWithArgs", arg1, arg2)
  end
end