sidekiq-batch
repository·master·Indexed 18 days ago
https://github.com/breamware/sidekiq-batchA lightweight implementation of Sidekiq Batch jobs designed as a drop-in replacement for the Sidekiq PRO Batch API. It allows developers to group multiple Sidekiq jobs into a batch, track their progress via Sidekiq::Batch::Status, and define callbacks for :success and :complete events. The library includes middleware for bid propagation and a configurable :batch_push_interval to manage callback timing and Redis load.
What's inside sidekiq-batch
- Sidekiq::Batch provides a simple implementation of batch jobs. It is designed to be a drop-in replacement for the Sidekiq PRO Batch API. For detailed usage patterns regarding how batches function, refer to the Sidekiq PRO Batch documentation.
Install sidekiq-batch
masterTo use
sidekiq-batchin your Ruby application, add it to yourGemfileand runbundle, or install it directly via the gem command.```ruby gem 'sidekiq-batch'Or via CLI
$ gem install sidekiq-batch
Configure :batch_push_interval to manage batch callback timing
masterWhen a worker (Worker A) queues other jobs (Worker B) within the same batch, there is a race condition where Worker B might finish before Worker A. By default, this can cause
on(:success)oron(:complete)callbacks to fire prematurely.You can control this behavior using the
:batch_push_intervaloption in yoursidekiq.ymlconfiguration.Configuration Options
Value Behavior Absent (Legacy) Worker A only increments the batch job count after Worker A completes. 0Worker A increments the batch count immediately when WorkerB.perform_asyncis called.Positive Integer Worker A waits up to $N$ seconds before pushing the increment to Redis, or until it finishes, whichever comes first. Best Practices
- To prevent premature callbacks when Worker A is slow: Set
:batch_push_interval: 0. This ensures the batch status is updated immediately so callbacks don't fire while Worker A is still active. - To avoid high Redis load when queueing many jobs: If Worker A is queueing thousands of jobs, set
:batch_push_intervalto a small positive number (e.g.,3). To ensure the batch callback doesn't trigger until Worker A and the last Worker B job are finished, useperform_infor the child jobs:
# Example: Using perform_in to ensure Worker A finishes first WorkerB.perform_in(4.seconds, some_arg):batch_push_interval: 0- To prevent premature callbacks when Worker A is slow: Set
Configure Sidekiq::Batch middleware
masterTo enable batch functionality, you must configure both the Sidekiq client and server middleware. This ensures that batch IDs (
bid) are correctly propagated from the client to the worker and that job success/failure is reported back to the batch.Calling
Sidekiq::Batch::Middleware.configureautomatically sets up the following:- Client Middleware: Adds
Sidekiq::Batch::Middleware::ClientMiddlewareto both client and server configurations to injectbidinto job payloads. - Server Middleware: Adds
Sidekiq::Batch::Middleware::ServerMiddlewareto the server configuration to track job completion. - Worker Extension: Includes
Sidekiq::Batch::Extension::WorkerintoSidekiq::Workerto support batch-aware worker behavior.
# Simply call the configuration method to set up the middleware and extensions Sidekiq::Batch::Middleware.configure- Client Middleware: Adds
Define and use Batch callbacks
masterSidekiq::Batch allows you to define callbacks that trigger when a batch reaches specific states. You can define these callbacks in two ways:
- Class-based callbacks: Specify a class and a method name. The method must follow the naming convention
on_#{event}, whereeventis eithersuccessorcomplete. - Instance-based callbacks: Specify a class and a method name using the
Class#methodsyntax.
When a callback is triggered, the method receives three arguments:
status: An instance ofSidekiq::Batch::Statusrepresenting the batch state.opts: An options hash passed during the callback registration.- (For class-based callbacks) The method signature is
method_name(status, opts).
Callback Events:
complete: Triggered when all jobs in the batch have finished (regardless of whether they succeeded or failed).success: Triggered only when all jobs in the batch have finished successfully (no failures).
Note on Execution Order: If a batch is successful, the
completecallback is triggered first, followed by thesuccesscallback. This order is intentional becausesuccesscallbacks may add more jobs to a parent batch.# Example of a callback class class MyBatchCallback def on_success(status, opts) puts "Batch #{status.bid} succeeded with options: #{opts}" end def on_complete(status, opts) puts "Batch #{status.bid} is complete." end end # When creating a batch, you would register this class # (Assuming the Batch API allows registration via a method like .on) # batch.on('MyBatchCallback#on_success', { some_opt: 'value' })- Class-based callbacks: Specify a class and a method name. The method must follow the naming convention
Invalidate a batch
masterThe
invalidate_allmethod marks a batch as invalid in Redis. This is useful for cancelling the effects of a batch or preventing its callbacks from running if the batch's context is no longer relevant.batch = Sidekiq::Batch.new # ... perform work ... # Mark this batch as invalid batch.invalidate_allDefine batch callbacks with `on`
masterYou can register callbacks that trigger automatically when a batch reaches certain lifecycle events. Supported events are:
:success: Triggered when all jobs in the batch have completed successfully.:complete: Triggered when all jobs in the batch have finished (regardless of whether they succeeded or failed).
The
onmethod accepts the event name, the callback class/method string, and an optional hash of arguments.batch = Sidekiq::Batch.new # Callback when all jobs succeed batch.on(:success, 'MyCallbackClass#perform') # Callback with specific options batch.on(:complete, 'MyCallbackClass#perform', { 'some_option' => 'value' }) batch.jobs do MyWorker.perform_async(1) endRetrieve failure details using Sidekiq::Batch::Status#failure_info
masterTo inspect specifically which jobs failed within a batch, call
failure_info. This returns an array of members from the Redis setBID-{bid}-failed. If no failures exist, it returns an empty array[].status = Sidekiq::Batch::Status.new(bid) if status.failures > 0 puts "Failed job details: #{status.failure_info}" endCheck if a batch is still valid
masterA batch can be invalidated (e.g., via
invalidate_all). Use thevalid?method to check if the current batch and all its parent batches are still valid. If a batch or any of its ancestors has been invalidated,valid?will returnfalse.batch = Sidekiq::Batch.new(some_bid) if batch.valid? # Proceed with batch-related logic else # The batch or its parent has been invalidated endCreate and manage Sidekiq::Batch instances
masterUse
Sidekiq::Batch.newto create a new batch. You can wrap multiple Sidekiq jobs within ajobsblock to associate them with the batch. The batch tracks the progress of these jobs and can trigger callbacks when the batch reaches specific states.Key attributes:
bid: The unique batch identifier.description: A string describing the batch's purpose.parent: Returns the parentSidekiq::Batchinstance if this batch is a child of another.
batch = Sidekiq::Batch.new batch.description = "Processing user reports" batch.jobs do # Any Sidekiq jobs pushed here will be part of this batch MyWorker.perform_async(1) MyWorker.perform_async(2) endAccess batch information from within a Sidekiq worker
masterThe
Sidekiq::Batch::Extension::Workermodule provides helper methods to access the current batch context from inside a Sidekiq worker. When a worker is running as part of a batch, you can use these methods to retrieve the batch ID or check the batch's validity.Available methods:
bid: Returns the current batch ID (bid).batch: Returns the currentSidekiq::Batchobject.valid_within_batch?: Returnstrueif the current batch is valid,falseotherwise.
class MyWorker include Sidekiq::Batch::Extension::Worker def perform(*args) if valid_within_batch? puts "Running in batch: #{bid}" end end endQuery batch progress with Sidekiq::Batch::Status
masterUse
Sidekiq::Batch::Statusto inspect the real-time progress, completion state, and error information of a specific batch using its Batch ID (bid).Key metrics available:
total: The total number of jobs in the batch.pending: The number of jobs still remaining to be processed.failures: The count of jobs that failed.complete?: Returnstrueif the batch has finished processing.child_count: The number of child batches associated with this batch.created_at: The timestamp when the batch was created.parent_bid: The ID of the parent batch, if this is a nested batch.failure_info: An array of information regarding failed jobs.
You can retrieve all these metrics at once as a hash using the
datamethod.status = Sidekiq::Batch::Status.new(bid) puts "Progress: #{status.total - status.pending}/#{status.total}" puts "Complete? #{status.complete?}" puts "Failures: #{status.failures}" # Get all data as a hash batch_data = status.data