DOCSEARCHbyAlgolia
Submit Docs

groupmq

repository·main·Indexed 18 days ago

https://github.com/openpanel-dev/groupmq

A fast, reliable Redis-backed per-group FIFO queue for Node.js and TypeScript. GroupMQ guarantees strict sequential job ordering within specific groups while allowing parallel processing across different groups. It features visibility timeouts, retries, repeatable jobs via intervals or cron patterns, and integration with BullBoard for visual monitoring.

Tokens
35.8K
Snippets
110
Records
146
Agent score
63%

What's inside groupmq

  1. What is GroupMQ?

    main

    GroupMQ is a lightweight, high-performance job queue for Node.js backed by Redis. It is designed to provide per-group FIFO (First-In-First-Out) processing guarantees. While any worker in your cluster can process any groupId, GroupMQ ensures that only one job per group is active at any given time, preventing race conditions or out-of-order processing within a specific group context.

    Key features include:

    • Per-group FIFO ordering: Maintains order within a group without creating a global bottleneck.
    • Horizontal Scalability: Workers can be scaled across multiple instances.
    • Job Management: Built-in support for retries, delays, and repeating jobs (cron).
    • BullMQ-inspired API: Familiar ergonomics for developers used to BullMQ.

  3. How GroupMQ handles ordering and concurrency

    main

    GroupMQ is designed for per-group FIFO ordering. This means jobs within the same groupId are processed in strict sequential order, which is ideal for user-specific workflows or sequential data pipelines.

    However, GroupMQ supports cross-group parallelism. By increasing the concurrency setting on a Worker, you can process multiple jobs from different groups simultaneously.

    Key Mental Model:

    • Within a group: Strict FIFO (one job at a time).
    • Across groups: Parallel (multiple groups at a time, up to the concurrency limit).

  4. How GroupMQ works: Architecture and Data Structures

    main

    GroupMQ is a group-based queueing system that uses Redis for storage and Lua scripts for atomic operations. It guarantees per-group FIFO (First-In-First-Out) ordering.

    Redis Data Structures

    All keys are prefixed with groupmq:{namespace}::

    • :g:{groupId}: Sorted set of job IDs in a group, ordered by score.
    • :ready: Sorted set of group IDs that have jobs available.
    • :job:{jobId}: Hash containing job data (id, groupId, data, attempts, status, etc.).
    • :lock:{groupId}: String containing the job ID that currently owns the group lock (with TTL).
    • :processing: Sorted set of active job IDs, ordered by deadline.
    • :processing:{jobId}: Hash containing processing metadata (groupId, deadlineAt).
    • :delayed: Sorted set of delayed jobs, ordered by runAt timestamp.
    • :completed: Sorted set of completed job IDs.
    • :failed: Sorted set of failed job IDs.
    • :repeats: Hash of repeating job definitions (groupId → config).

    Job Lifecycle States

    1. Waiting: Job is in :g:{groupId} and the group is in :ready.
    2. Delayed: Job is in :delayed (scheduled for future).
    3. Active: Job is in :processing and the group is locked.
    4. Completed: Job is in :completed (retention).
    5. Failed: Job exceeded maxAttempts and moved to :failed (retention).

    Ordering and Scoring

    Jobs are ordered using a composite score to ensure stable, sortable ordering: score = (orderMs - baseEpoch) * 1000 + seq

    • orderMs: User-provided timestamp for event ordering.
    • baseEpoch: Fixed epoch timestamp (1704067200000).
    • seq: Auto-incrementing sequence for tiebreaking (resets daily).

  6. How per-group FIFO and concurrency work in GroupMQ

    main

    GroupMQ is designed for workloads where events for a specific entity (e.g., a user) must be processed in order, but events for different entities can be processed in parallel.

    • Per-group FIFO: By providing a groupId in queue.add(), GroupMQ guarantees that exactly one job per groupId is in-flight at any time. This prevents race conditions and ensures no job within a group overtakes another.
    • Concurrency: The concurrency: N setting on a Worker determines how many jobs can be processed simultaneously across different groups. For example, if concurrency is 4, the worker can process 4 jobs at once, provided they belong to 4 different groupIds.
    • Timestamp Ordering: You can use orderMs to specify the intended order of jobs. For producers that might be slightly out of sync, orderingDelayMs can be used to enforce stricter ordering.

  7. How stalled job detection works

    main

    Stalled jobs occur when a worker crashes or loses connection while processing a job, leaving it stuck in the 'processing' state. GroupMQ uses a background mechanism to detect and recover these jobs:

    1. Background Checker: Periodically scans for stalled jobs based on the stalledInterval.
    2. Detection: Identifies jobs in the processing state that have exceeded their deadline plus a grace period.
    3. Recovery: Automatically moves stalled jobs back to the waiting state so they can be retried.
    4. Failure: If a job reaches the maxStalledCount threshold of stalls, it is marked as permanently failed.

  8. How the scheduler works and configuring accuracy

    main

    Repeating jobs are managed by a distributed scheduler that runs during the worker's maintenance cycle. To achieve high-frequency repeats (sub-second), you must tune both the Worker and the Queue settings.

    Warning: Very fast repeats (< 1s) increase Redis load and coordination overhead. Use sparingly.

    Key Configuration Keys

    • Worker: schedulerIntervalMs: How often the worker checks for due repeat jobs. Default: 1000.
    • Queue: schedulerLockTtlMs: The distributed lock TTL. This is the primary bottleneck for fast repeats. Default: 1500.

    To achieve a 100ms repeat, you must lower both values.

    // Example configuration for sub-second repeats
const queue = new Queue({
  redis,
  namespace: 'fast-queue',
  schedulerLockTtlMs: 50, // Low TTL for fast lock acquisition
});

new Worker({
  queue,
  schedulerIntervalMs: 10, // Check every 10ms
  cleanupIntervalMs: 100,  // Run cleanup every 100ms
  async handler(job) { /* ... */ },
}).run();

await queue.add({
  groupId: 'fast-cron',
  data: { task: 'tick' },
  repeat: { every: 100 },
});

  9. Achieve sequential processing with Per-Group FIFO

    main

    GroupMQ guarantees strict FIFO (First-In-First-Out) ordering within a specific groupId. Only one job per groupId is in-flight at a time, ensuring sequential processing for that group. However, jobs belonging to different groupIds can be processed in parallel across multiple workers, allowing for high throughput.

    // Jobs in the same group execute sequentially
await queue.add({ groupId: 'user:1', data: { id: 'a', ms: 100 } });
await queue.add({ groupId: 'user:1', data: { id: 'b', ms: 100 } });

// Jobs in different groups may run in parallel across workers
await queue.add({ groupId: 'user:2', data: { id: 'c', ms: 100 } });

  11. How Worker Concurrency modes work

    main

    GroupMQ supports two concurrency modes for workers. Regardless of the mode, per-group FIFO ordering is strictly maintained: multiple jobs from the same group will never run in parallel.

    Sequential Mode (concurrency = 1)

    • Processes one job at a time.
    • Uses a blocking reserve mechanism (reserveBlocking) which is efficient and prevents wasteful polling.
    • Best for: CPU-intensive jobs or resource-constrained environments.

    Parallel Mode (concurrency > 1)

    • Attempts batch reservation first to reduce Redis round-trips and increase throughput.
    • Processes multiple jobs concurrently, provided they belong to different groups.
    • Falls back to a blocking reserve if the batch is empty.
    • Best for: I/O-bound workloads like network calls or database operations.

    Key Optimizations

    • Batch Reservation: Reduces latency for concurrent workers.
    • Blocking Operations: Prevents wasteful polling.
    • Heartbeat Mechanism: Keeps jobs alive during long processing tasks by extending the group lock.
    • Atomic Operations: Uses Lua scripts for all critical paths (enqueue, reserve, complete, retry, remove) to prevent race conditions.

  12. How GroupMQ differs from BullMQ

    main

    GroupMQ and BullMQ are both Redis-backed job queues for Node.js, but they serve different primary use cases:

    Core Differences

    • Per-group FIFO: GroupMQ guarantees at most one active job per groupId. Different groups run in parallel. BullMQ provides FIFO per queue.
    • Ordering: GroupMQ allows producers to set orderMs for strict ordering and can use orderingDelayMs to wait for late-arriving jobs.
    • Concurrency Model: BullMQ uses an in-process concurrency option. GroupMQ prefers process-level scaling (using Cluster, PM2, or containers), though you can run multiple Worker instances within a single process for I/O-bound work.
    • API & Integration: GroupMQ uses a lean API surface and includes a BullBoard adapter for dashboard integration.

    When to choose GroupMQ

    • You need strict per-group FIFO (exactly one in-flight job per groupId) without using BullMQ Pro.
    • You need correct ordering based on producer timestamps via orderMs.

    When to choose BullMQ

    • You need a broader feature set including flows, priorities, or rate limiting.
    • You prefer managing concurrency within a single process via the concurrency option.