sidekiq-batch

repository·master·Indexed 18 days ago

https://github.com/breamware/sidekiq-batch

A lightweight implementation of Sidekiq Batch jobs designed as a drop-in replacement for the Sidekiq PRO Batch API. It allows developers to group multiple Sidekiq jobs into a batch, track their progress via Sidekiq::Batch::Status, and define callbacks for :success and :complete events. The library includes middleware for bid propagation and a configurable :batch_push_interval to manage callback timing and Redis load.

Tokens
2.9K
Snippets
12
Records
13
Agent score
62%

What's inside sidekiq-batch

  1. Configure :batch_push_interval to manage batch callback timing

    master

    When a worker (Worker A) queues other jobs (Worker B) within the same batch, there is a race condition where Worker B might finish before Worker A. By default, this can cause on(:success) or on(:complete) callbacks to fire prematurely.

    You can control this behavior using the :batch_push_interval option in your sidekiq.yml configuration.

    Configuration Options

    ValueBehavior
    Absent (Legacy)Worker A only increments the batch job count after Worker A completes.
    0Worker A increments the batch count immediately when WorkerB.perform_async is called.
    Positive IntegerWorker A waits up to $N$ seconds before pushing the increment to Redis, or until it finishes, whichever comes first.

    Best Practices

    • To prevent premature callbacks when Worker A is slow: Set :batch_push_interval: 0. This ensures the batch status is updated immediately so callbacks don't fire while Worker A is still active.
    • To avoid high Redis load when queueing many jobs: If Worker A is queueing thousands of jobs, set :batch_push_interval to a small positive number (e.g., 3). To ensure the batch callback doesn't trigger until Worker A and the last Worker B job are finished, use perform_in for the child jobs:
    # Example: Using perform_in to ensure Worker A finishes first
    WorkerB.perform_in(4.seconds, some_arg)
    :batch_push_interval: 0
  2. Configure Sidekiq::Batch middleware

    master

    To enable batch functionality, you must configure both the Sidekiq client and server middleware. This ensures that batch IDs (bid) are correctly propagated from the client to the worker and that job success/failure is reported back to the batch.

    Calling Sidekiq::Batch::Middleware.configure automatically sets up the following:

    1. Client Middleware: Adds Sidekiq::Batch::Middleware::ClientMiddleware to both client and server configurations to inject bid into job payloads.
    2. Server Middleware: Adds Sidekiq::Batch::Middleware::ServerMiddleware to the server configuration to track job completion.
    3. Worker Extension: Includes Sidekiq::Batch::Extension::Worker into Sidekiq::Worker to support batch-aware worker behavior.
    # Simply call the configuration method to set up the middleware and extensions
    Sidekiq::Batch::Middleware.configure
  3. Define and use Batch callbacks

    master

    Sidekiq::Batch allows you to define callbacks that trigger when a batch reaches specific states. You can define these callbacks in two ways:

    1. Class-based callbacks: Specify a class and a method name. The method must follow the naming convention on_#{event}, where event is either success or complete.
    2. Instance-based callbacks: Specify a class and a method name using the Class#method syntax.

    When a callback is triggered, the method receives three arguments:

    • status: An instance of Sidekiq::Batch::Status representing the batch state.
    • opts: An options hash passed during the callback registration.
    • (For class-based callbacks) The method signature is method_name(status, opts).

    Callback Events:

    • complete: Triggered when all jobs in the batch have finished (regardless of whether they succeeded or failed).
    • success: Triggered only when all jobs in the batch have finished successfully (no failures).

    Note on Execution Order: If a batch is successful, the complete callback is triggered first, followed by the success callback. This order is intentional because success callbacks may add more jobs to a parent batch.

    # Example of a callback class
    class MyBatchCallback
      def on_success(status, opts)
        puts "Batch #{status.bid} succeeded with options: #{opts}"
      end
    
      def on_complete(status, opts)
        puts "Batch #{status.bid} is complete."
      end
    end
    
    # When creating a batch, you would register this class
    # (Assuming the Batch API allows registration via a method like .on)
    # batch.on('MyBatchCallback#on_success', { some_opt: 'value' })
  4. Invalidate a batch

    master

    The invalidate_all method marks a batch as invalid in Redis. This is useful for cancelling the effects of a batch or preventing its callbacks from running if the batch's context is no longer relevant.

    batch = Sidekiq::Batch.new
    # ... perform work ...
    
    # Mark this batch as invalid
    batch.invalidate_all
  5. Define batch callbacks with `on`

    master

    You can register callbacks that trigger automatically when a batch reaches certain lifecycle events. Supported events are:

    • :success: Triggered when all jobs in the batch have completed successfully.
    • :complete: Triggered when all jobs in the batch have finished (regardless of whether they succeeded or failed).

    The on method accepts the event name, the callback class/method string, and an optional hash of arguments.

    batch = Sidekiq::Batch.new
    
    # Callback when all jobs succeed
    batch.on(:success, 'MyCallbackClass#perform')
    
    # Callback with specific options
    batch.on(:complete, 'MyCallbackClass#perform', { 'some_option' => 'value' })
    
    batch.jobs do
      MyWorker.perform_async(1)
    end
  6. Retrieve failure details using Sidekiq::Batch::Status#failure_info

    master

    To inspect specifically which jobs failed within a batch, call failure_info. This returns an array of members from the Redis set BID-{bid}-failed. If no failures exist, it returns an empty array [].

    status = Sidekiq::Batch::Status.new(bid)
    if status.failures > 0
      puts "Failed job details: #{status.failure_info}"
    end
  7. Check if a batch is still valid

    master

    A batch can be invalidated (e.g., via invalidate_all). Use the valid? method to check if the current batch and all its parent batches are still valid. If a batch or any of its ancestors has been invalidated, valid? will return false.

    batch = Sidekiq::Batch.new(some_bid)
    
    if batch.valid?
      # Proceed with batch-related logic
    else
      # The batch or its parent has been invalidated
    end
  8. Create and manage Sidekiq::Batch instances

    master

    Use Sidekiq::Batch.new to create a new batch. You can wrap multiple Sidekiq jobs within a jobs block to associate them with the batch. The batch tracks the progress of these jobs and can trigger callbacks when the batch reaches specific states.

    Key attributes:

    • bid: The unique batch identifier.
    • description: A string describing the batch's purpose.
    • parent: Returns the parent Sidekiq::Batch instance if this batch is a child of another.
    batch = Sidekiq::Batch.new
    batch.description = "Processing user reports"
    
    batch.jobs do
      # Any Sidekiq jobs pushed here will be part of this batch
      MyWorker.perform_async(1)
      MyWorker.perform_async(2)
    end
  9. Access batch information from within a Sidekiq worker

    master

    The Sidekiq::Batch::Extension::Worker module provides helper methods to access the current batch context from inside a Sidekiq worker. When a worker is running as part of a batch, you can use these methods to retrieve the batch ID or check the batch's validity.

    Available methods:

    • bid: Returns the current batch ID (bid).
    • batch: Returns the current Sidekiq::Batch object.
    • valid_within_batch?: Returns true if the current batch is valid, false otherwise.
    class MyWorker
      include Sidekiq::Batch::Extension::Worker
    
      def perform(*args)
        if valid_within_batch?
          puts "Running in batch: #{bid}"
        end
      end
    end
  10. Query batch progress with Sidekiq::Batch::Status

    master

    Use Sidekiq::Batch::Status to inspect the real-time progress, completion state, and error information of a specific batch using its Batch ID (bid).

    Key metrics available:

    • total: The total number of jobs in the batch.
    • pending: The number of jobs still remaining to be processed.
    • failures: The count of jobs that failed.
    • complete?: Returns true if the batch has finished processing.
    • child_count: The number of child batches associated with this batch.
    • created_at: The timestamp when the batch was created.
    • parent_bid: The ID of the parent batch, if this is a nested batch.
    • failure_info: An array of information regarding failed jobs.

    You can retrieve all these metrics at once as a hash using the data method.

    status = Sidekiq::Batch::Status.new(bid)
    
    puts "Progress: #{status.total - status.pending}/#{status.total}"
    puts "Complete? #{status.complete?}"
    puts "Failures: #{status.failures}"
    
    # Get all data as a hash
    batch_data = status.data