graphile-worker

Graphile Worker is a job queue that uses PostgreSQL to store jobs and executes them on Node.js. It allows you to run background tasks—such as sending emails, performing calculations, or generating PDFs—without blocking your main application code or HTTP responses.

Key characteristics include:

• Postgres-centric: Designed to work seamlessly with jobs created directly inside the database via triggers or stored procedures. It is highly compatible with PostGraphile and PostgREST.
• Reliable: Leverages PostgreSQL's transactional guarantees to ensure jobs are not lost. It guarantees at least once execution, with most jobs executing exactly once. Failed jobs are automatically retried using exponential backoff.
• High Performance: Uses SKIP LOCKED for fast job fetching and LISTEN/NOTIFY for low-latency job discovery (typically under 3ms from schedule to execution).
• Flexible Deployment: Can be run as a standalone process or embedded directly within your existing Node.js server process to reduce infrastructure complexity.

A task executor is an asynchronous function that performs the work associated with a specific task identifier.

When a job is found in the database with a matching task identifier, Graphile Worker calls the executor.

• Success: If the executor returns successfully, the job is considered successful and is deleted from the queue (unless it is a batch job).
• Failure: If the executor throws an error or rejects its promise, the job is considered a failure and is rescheduled using an exponential-backoff algorithm.

Important: Task executors must await all asynchronous work before returning. Do not create "untethered" promises, or Graphile Worker may prematurely mark the job as successful.

Each executor receives two arguments:

1. payload: The JSON payload passed when the job was created.
2. helpers: A collection of utility functions (see helpers).

export default async function task1(payload) {
  await doMyLogicWith(payload);
}

Graphile Worker supports a wide range of events via an EventEmitter. You can consume these events using two primary methods:

1. Via the Runner object: Access the events property on the returned Runner instance. This is the standard way to listen to events during the worker's lifecycle.
2. Via WorkerOptions.events: Provide your own EventEmitter instance in the configuration options. This is useful for capturing events that occur during the startup procedure, before the run() promise has resolved.

Both methods allow you to react to pool lifecycle changes, worker lifecycle changes, and job status changes.

// Method 1: via runner.events
runner.events.on("job:success", ({ worker, job }) => {
  console.log(`Hooray! Worker ${worker.workerId} completed job ${job.id}`);
});

// Method 2: using a custom EventEmitter in options
/** @type {import("graphile-worker").WorkerEvents} */
const events = new EventEmitter();
events.on("job:success", ({ worker, job }) => {
  // ...
});

const runner = await run({ events, ... });

Graphile Worker provides several advanced features for managing background tasks:

• Execution Modes: Supports both standalone and embedded modes.
• Task Management:
  • Parallelism: Runs tasks in parallel by default.
  • Serial Queues: Adding jobs to the same named queue causes them to run in series.
  • De-duplication: Supports task de-duplication via a unique job_key.
  • Batching: Supports 'batch jobs' by appending data to already enqueued jobs.
• Scheduling & Retries:
  • Retries: Automatically re-attempts failed jobs with exponential back-off. The default is 25 attempts over approximately 3 days (customizable).
  • Scheduling: Includes a Crontab-like scheduling feature for recurring tasks, with optional backfill.
• Developer Experience:
  • Modern API: 100% async/await API (no callbacks) written in TypeScript.
  • Testing: Easy to test (the runTaskListOnce utility is recommended).
  • Rate Limiting: Supports flexible runtime controls for complex rate limiting (e.g., via graphile-worker-rate-limiter).

Graphile Worker uses a "Graphile Config preset" to manage settings. A preset is a JavaScript/TypeScript object that can include extends (to merge other presets) and plugins. For Graphile Worker, settings specific to the worker are placed under the worker key.

Using a graphile.config.js (or .ts, .mjs) file is recommended because it allows you to:

• Share configuration between CLI and Library modes.
• Share common options between different instances.
• Use the graphile CLI tool to inspect configuration via graphile config print or graphile config options.

It is recommended to export the preset as the default export of your configuration file.

import { WorkerPreset } from "graphile-worker";

export default {
  extends: [WorkerPreset],
  worker: {
    connectionString: process.env.DATABASE_URL,
    maxPoolSize: 10,
    pollInterval: 2000,
    preparedStatements: true,
    schema: "graphile_worker",
    crontabFile: "crontab",
    concurrentJobs: 1,
    fileExtensions: [".js", ".cjs", ".mjs", ".ts", ".mts"],
  },
};