Shopify Maintenance Tasks

Maintenance Tasks relies on Active Job to execute tasks.

Important: It is strongly recommended to use a persistent queuing backend (like Sidekiq, Resque, etc.) rather than the default async adapter. Using a non-persistent backend means task progress will be lost during code deployments or infrastructure restarts.

The web UI tracks the lifecycle of a task through several states:

• new: Not yet run.
• enqueued: Waiting to be performed.
• running: Currently being performed by a job worker.
• pausing: Requested to pause; waiting to finish current work.
• paused: Paused; can be resumed.
• interrupted: Momentarily interrupted by job infrastructure.
• cancelling: Requested to cancel; waiting to finish current work.
• cancelled: Cancelled; cannot be resumed.
• succeeded: Finished successfully.
• errored: Encountered an unhandled exception.

When implementing Task#process, follow these guidelines to ensure reliability and compatibility with queue adapters (like Sidekiq):

1. Keep process short: The execution of a single element in process should take less than 25 seconds (or your queue adapter's timeout). Short batches allow tasks to be safely interrupted and resumed.
2. Ensure Idempotency: process must be safe to run multiple times for the same element. If a task errors or a job is re-enqueued due to a timeout, the same element may be processed again.
3. Be careful with memoization: A Task object lives for the duration of a single job. Memoized values in a Task instance will persist across multiple calls to process within that same job. Use this for throttling or reporting, but do not rely on it for state that must be fresh for every element.

CSV Tasks allow you to iterate over rows in a CSV file uploaded via Active Storage. Note: This requires Active Storage to be configured in your application.

Generate a CSV task using the --csv flag:

bin/rails generate maintenance_tasks:task import_posts --csv

Your task must implement #process(row), where row is a CSV::Row object. The implicit #count method will parse the entire file to determine the row count, which may be slow for very large files. You can override #count to return an approximation or nil to skip counting.

# app/tasks/maintenance/import_posts_task.rb

module Maintenance
  class ImportPostsTask < MaintenanceTasks::Task
    csv_collection

    def process(row)
      Post.create!(title: row["title"], content: row["content"])
    end
  end
end

To install the Maintenance Tasks engine and set up the necessary database tables and routing, add the gem to your bundle and run the provided generator.

Running the generator will:

1. Create and run a migration to add the required tables to your database.
2. Mount the Maintenance Tasks web UI in your config/routes.rb (accessible by default at /maintenance_tasks).

bundle add maintenance_tasks
bin/rails generate maintenance_tasks:install

If your task performs a single operation (like enqueuing a job or calling an API) and does not need to iterate over a collection, generate it with the --no-collection flag. You only need to implement the #process method.

bin/rails generate maintenance_tasks:task no_collection_task --no-collection

module Maintenance
  class NoCollectionTask < MaintenanceTasks::Task
    no_collection

    def process
      SomeAsyncJob.perform_later
    end
  end
end

To create a task that iterates over a collection (like an Active Record Relation or an Array), use the Rails generator. The generated task must implement #collection to define the data source and #process to define the work performed on a single record.

Workflow:

1. Generate the task class.
2. Implement collection (returns an enumerable).
3. Implement process(record) (logic for one item).
4. Run the task via Web UI, CLI, or Ruby.

bin/rails generate maintenance_tasks:task update_posts

module Maintenance
  class UpdatePostsTask < MaintenanceTasks::Task
    def collection
      Post.all
    end

    def process(post)
      post.update!(content: "New content!")
    end
  end
end

It is recommended to test the #process method at a minimum. For tasks with parameters, you must instantiate the task class and assign attributes before calling #process.

# Testing a standard task
module Maintenance
  class UpdatePostsTaskTest < ActiveSupport::TestCase
    test "#process performs a task iteration" do
      post = Post.new
      Maintenance::UpdatePostsTask.process(post)
      assert_equal "New content!", post.content
    end
  end
end

# Testing a task with parameters
module Maintenance
  class UpdatePostsViaParamsTaskTest < ActiveSupport::TestCase
    setup do
      @task = UpdatePostsViaParamsTask.new
      @task.updated_content = "Testing"
    end

    test "#process performs a task iteration" do
      assert_difference -> { Post.first.content } do
        @task.process(Post.first)
      end
    end
  end
end