DOCSEARCHbyAlgolia
Submit Docs

activerecord-import

repository·master·Indexed 26 days ago

https://github.com/zdennis/activerecord-import

A high-performance bulk insertion library for ActiveRecord that optimizes SQL statement generation and avoids N+1 insert problems by following associations. It supports bulk imports via Model.import and Model.import!, providing advanced conflict handling (upserts) for PostgreSQL, MySQL, and SQLite3, as well as memory synchronization for ActiveRecord instances.

Tokens
5.6K
Snippets
11
Records
35
Agent score
88%

What's inside activerecord-import

  1. Overview of Activerecord-Import

    master
    Activerecord-Import is a library designed for bulk inserting data into a database using ActiveRecord. It optimizes performance by following ActiveRecord associations and generating the minimal number of SQL insert statements required, effectively avoiding the N+1 insert problem. For example, instead of performing millions of individual inserts for nested associations (e.g., Publishers -> Books -> Reviews), it can reduce the operation to just a few SQL statements (one per table).

  2. Key features of Activerecord-Import

    master

    The library provides several high-level features for efficient data ingestion:

    • Raw columns and arrays of values: The fastest method for bulk insertion.
    • Model objects: Works with instantiated model objects (faster than standard ActiveRecord saves).
    • Validations: Supports performing model validations during the import process.
    • On duplicate key updates: Supports performing updates when a duplicate key is encountered (requires MySQL, SQLite 3.24.0+, or Postgres 9.5+).

  3. Install and Load activerecord-import

    master

    Using Bundler (Recommended)

    Add to your Gemfile:

    gem 'activerecord-import'

    Manual Loading

    If you use require: false in your Gemfile, you must manually require the necessary components. If you are in a Rails environment with an established connection, you may need to load the specific adapter:

    require 'activerecord-import/base'
require 'activerecord-import/active_record/adapters/postgresql_adapter' # or mysql2, sqlite3, etc.

    If you are not using a bundler-managed environment, require it after active_record is loaded:

    require 'active_record'
require 'activerecord-import'

  4. Handle duplicate keys with upsert (on_duplicate_key_update)

    master

    Allows you to specify fields to update if a conflict occurs. Support varies by database:

    • MySQL: Uses ON DUPLICATE KEY UPDATE.
    • PostgreSQL (9.5+): Requires specifying a conflict_target (the columns where the conflict occurs).
    • SQLite (3.24.0+): Models support after PostgreSQL.

    PostgreSQL Specifics

    For PostgreSQL, you can provide a conflict_target (the column/index that triggers the conflict) and the columns to update. You can also use index_predicate for partial indexes or constraint_name for unique constraints.

    # MySQL: Basic update
Book.import [book], on_duplicate_key_update: [:title]

# PostgreSQL: Basic update (conflict target must be primary key)
Book.import [book], on_duplicate_key_update: [:title]

# PostgreSQL: Explicit target and columns
Book.import [book], on_duplicate_key_update: {conflict_target: [:id], columns: [:title]}

# PostgreSQL: Using a value from another column
Book.import [book], on_duplicate_key_update: {
  conflict_target: [:id], 
  columns: {author: :title}
}

# PostgreSQL: Using partial indexes
Book.import [book], on_duplicate_key_update: {
  conflict_target: [:id],
  index_predicate: "published_at IS NOT NULL",
  columns: [:author]
}

# PostgreSQL: Using constraints
Book.import [book], on_duplicate_key_update: {
  constraint_name: :for_upsert, 
  columns: [:published_at]
}

  5. Handle ActiveRecord Callbacks

    master

    Standard ActiveRecord callbacks (like before_create, after_save, etc.) are NOT called during an import because it is a mass-import operation.

    If you need callbacks to run, you must manually trigger them on your objects before importing, or loop through and validate/run callbacks manually.

    # Manually running callbacks before import
books.each do |book|
  book.run_callbacks(:save) { false }
  book.run_callbacks(:create) { false }
end
Book.import(books)

  6. Handle duplicate keys with on_duplicate_key_ignore

    master

    Supported by MySQL, SQLite, and PostgreSQL (9.5+). This option allows you to skip records if a primary or unique key constraint is violated.

    Note: This cannot be used with recursive: true on PostgreSQL.

    book = Book.create! title: "Book1", author: "George Orwell"
book.title = "Updated Book Title"

# Skips the update because the title already exists
Book.import [book], on_duplicate_key_ignore: true

  7. Configure ActiveRecord Timestamps

    master

    By default, timestamps: true is enabled, and created_at/updated_at columns are automatically set.

    If you want to manually specify these columns in specific records, you can still use timestamps: true; the gem will only update the timestamp if the field is nil in the record. To disable automatic timestamping entirely, use timestamps: false.

  10. Batching imports with batch_size

    master

    Use the batch_size option to control how many rows are inserted per SQL statement. This is useful for very large datasets to avoid memory issues or long-running transactions.

    You can also provide a callable to the batch_progress option to monitor progress during large imports.

    # 2 INSERT statements for 4 records
Book.import columns, books, batch_size: 2

# Monitoring progress
my_proc = ->(rows_size, num_batches, current_batch_number, batch_duration_in_secs) {
  # logic to report progress
}
Book.import columns, books, batch_size: 2, batch_progress: my_proc

  11. Recursive imports (PostgreSQL only)

    master

    If you are using PostgreSQL, you can use recursive: true to import has_many or has_one associations in a single operation. This does not work with raw hashes or arrays; you must pass ActiveRecord objects.

    books = []
10.times do |i|
  book = Book.new(name: "book #{i}")
  book.reviews.build(title: "Excellent")
  books << book
end
Book.import books, recursive: true

  12. Import ActiveRecord Models

    master

    You can pass an array of ActiveRecord model instances directly to import. The gem will automatically extract attributes based on the model's columns.

    books = [
  Book.new(title: "Book 1", author: "George Orwell"),
  Book.new(title: "Book 2", author: "Bob Jones")
]

# Import models (validations enabled by default)
Book.import books