DOCSEARCHbyAlgolia
Submit Docs

Bull Job Queue

repository·develop·Indexed 12 days ago

https://github.com/optimalbits/bull

A high-performance, Redis-based job manager for Node.js optimized for stability and atomicity. Bull 4.16.5 features a polling-free design, support for delayed and repeatable jobs (cron), priority queues, rate limiting, and sandboxed processors to prevent stalled jobs. It implements an 'at least once' delivery strategy and is currently in maintenance mode, with BullMQ recommended for new TypeScript implementations.

Tokens
25.4K
Snippets
77
Records
103
Agent score
95%

What's inside Bull

  1. Overview of Bull features

    develop

    Bull is a Redis-based queue for Node.js designed for stability and atomicity. Key features include:

    • Polling-free design: Minimal CPU usage.
    • Job Management: Delayed jobs, repeatable jobs (cron specification), and priority support.
    • Flow Control: Rate limiting, concurrency control, and pause/resume capabilities (globally or locally).
    • Reliability: Automatic recovery from process crashes and sandboxed (threaded) processing functions.
    • Flexibility: Multiple job types per queue.

  2. Reference the Bull API structure

    develop

    The Bull API is organized into three primary domains: Queue, Job, and Events.

    • Queue: Methods for managing the lifecycle of the queue itself, adding jobs, controlling execution (pause/resume), and querying job counts or metrics.
    • Job: Methods for interacting with individual job instances, such as updating progress, logging, changing state (retry/discard/promote), and managing locks.
    • Events: A system for listening to global or queue-specific lifecycle events.

  3. Create repeatable jobs

    develop

    Repeatable jobs are configured to run indefinitely, or until a specific limit (date or count) is reached, using either a time interval (every) or a cron specification.

    Key Behaviors:

    • Deduplication: Bull will not add the same repeatable job if the repeat options are identical.
    • Caution on Job IDs: Since a job ID is considered part of the repeat options, passing a specific jobId will allow multiple jobs with the same cron pattern to be inserted.
    • Worker Availability: If no workers are running, repeatable jobs will not accumulate while the workers are offline.
    • Removal: Use the removeRepeatable method to stop a repeatable job pattern.
    // Repeat every 10 seconds for 100 times.
const myJob = await myqueue.add(
  { foo: 'bar' },
  {
    repeat: {
      every: 10000,
      limit: 100
    }
  }
);

// Repeat payment job once every day at 3:15 (am)
paymentsQueue.add(paymentsData, { repeat: { cron: '15 3 * * *' } });

  4. Listen to queue events

    develop

    Bull provides two types of event listeners:

    1. Local Listeners: Emitted by a specific queue instance. For example, a completed event on a worker instance provides the full job object and the result.
    2. Global Listeners: Emitted for all events across the entire queue (regardless of which instance produced them). Prefix the event name with global:. For performance, global events typically only pass the jobId rather than the full job object.

    Note: A local event will only fire if the instance is acting as a consumer or producer. Use global events for general monitoring.

    // Local completed event (receives job and result)
const myFirstQueue = new Bull('my-first-queue');
myFirstQueue.on('completed', (job, result) => {
  console.log(`Job completed with result ${result}`);
});

// Global completed event (receives only jobId)
const myFirstQueue = new Bull('my-first-queue');
myFirstQueue.on('global:completed', jobId => {
  console.log(`Job with id ${jobId} has been completed`);
});

  5. Understand Bull's 'At Least Once' Strategy and Stalled Jobs

    develop

    Bull aims for an "at least once" delivery strategy. This means jobs might occasionally be processed more than once.

    Stalled Jobs

    When a worker processes a job, it holds a lock. If the lock expires (due to a process crash or a blocked Node.js event loop), the job is considered stalled and Bull will automatically restart it. This leads to double processing.

    Best Practices to Prevent Stalling:

    1. Avoid blocking the event loop: Do not run heavy CPU-intensive code directly in the main process. Use separate processes (sandboxing) instead.
    2. Monitor the stalled event: Always listen for the stalled event and log it to your error monitoring system to detect potential double-processing issues.
    3. Adjust lockDuration: If jobs are naturally long-running, you can increase the lockDuration setting, though this increases the time it takes to detect a real crash.

    Safeguards

    To prevent infinite loops of failing jobs, Bull will recover a job from a stalled state a maximum of maxStalledCount times (default is 1).

  6. Avoid stalled jobs with Sandboxed Processors

    develop
    A job is considered 'stalled' if Bull suspects the process function has hung (e.g., the Node event loop is blocked for too long). To prevent this for CPU-intensive tasks, use Sandboxed Processors. These run the process functions in separate Node processes, ensuring that if a job crashes or blocks the loop, it doesn't affect the main process and can be automatically replaced.

  9. Implement a persistent Message Queue pattern

    develop

    You can use Bull as a robust communication channel between two servers that do not need to be online at the same time. In this pattern, treat add as send and process as receive. Each server maintains its own sendQueue (to send messages to the other) and receiveQueue (to listen for incoming messages).

    // Server A
const Queue = require('bull');

const sendQueue = new Queue('Server B');
const receiveQueue = new Queue('Server A');

receiveQueue.process(function (job, done) {
  console.log('Received message', job.data.msg);
  done();
});

sendQueue.add({ msg: 'Hello' });

  10. Install Bull via npm or yarn

    develop

    Bull is a Node library for fast and robust queue systems based on Redis. You can install it using npm or yarn. Note that a running Redis server is required; Bull connects to localhost:6379 by default.

    $ npm install bull --save
    $ yarn add bull

  11. Make Bull compatible with Redis Cluster

    develop

    Bull requires atomic operations across different keys, which violates Redis Cluster rules. To resolve this, use a queue prefix containing a hash tag (enclosed in brackets). This ensures all keys for a specific queue are placed in the same hash slot.

    To distribute load across the cluster, use different prefixes for different queues.

    const queue = new Queue('cluster', {
  prefix: '{myprefix}'
});

  12. Handle queue readiness in Bull 3.0.0

    develop

    The 'ready' event has been removed in version 3.0.0. If you need to verify that a queue has been initialized, use the Queue#isReady() method instead.

    Note: Most queue methods handle readiness internally, so waiting for a readiness state is often unnecessary.

    // Instead of waiting for the 'ready' event:
if (await myQueue.isReady()) {
  // Proceed with queue operations
}