sqs-consumer

repository·main·Indexed 23 days ago

https://github.com/bbc/sqs-consumer

A Node.js library for building SQS-based applications without boilerplate. It provides a high-level abstraction for polling, error handling, and message acknowledgment, supporting both single message processing via handleMessage and batch processing via handleMessageBatch. The library includes features for graceful shutdown, visibility timeout heartbeats, and specialized error handling through SQSError, TimeoutError, and StandardError.

Tokens
4.2K
Snippets
5
Records
29
Agent score
82%

What's inside sqs-consumer

  1. How message acknowledgment works

    main

    The consumer manages message deletion from the SQS queue based on the return value of your handler and the alwaysAcknowledge option.

    When alwaysAcknowledge is false (default)

    • To acknowledge and delete messages: Return the message object (for single messages) or an array of message objects (for batch processing). Only the IDs returned will be deleted.
    • To NOT delete messages (retry): Return undefined, an empty object {}, or an empty array []. For batch processing, returning undefined or [] prevents acknowledgment of all messages in the batch.
    • Note: Returning void is discouraged and will be deprecated. If strictReturn is true, returning null will throw an error.

    When alwaysAcknowledge is true

    • All messages will be acknowledged and deleted regardless of the handler's return value.
  2. How to handle FIFO queues

    main
    While sqs-consumer does not explicitly test FIFO queues, they can be used with the correct configuration. To maintain FIFO ordering, you should always use the handleMessageBatch method instead of handleMessage. If you are certain your configuration is correct and want to suppress the library's FIFO warning, set suppressFifoWarning: true in your options.
  3. Implement graceful shutdown with consumer.stop()

    main

    To shut down the consumer gracefully, call consumer.stop().

    • Default behavior: stop() prevents new polls but emits stopped immediately, even if in-flight messages are still being processed.
    • Graceful shutdown: Set pollingCompleteWaitTimeMs to a value (in milliseconds) to allow the consumer to wait for the last poll and current message handlers to finish.
    • Abort option: Passing { abort: true } to stop() will cancel the shared AbortController, halting heartbeat extensions and preventing acknowledgments/deletions from finishing. For graceful shutdowns, keep abort: false (the default).

    During the wait period, the consumer emits waiting_for_polling_to_complete every second. If the timeout is reached, waiting_for_polling_to_complete_timeout_exceeded is emitted before stopped.

    const consumer = Consumer.create({
      queueUrl: "https://sqs.eu-west-1.amazonaws.com/account-id/queue-name",
      handleMessage: async (message) => {
        await doWork(message);
        return message;
      },
      pollingCompleteWaitTimeMs: 10_000, // This will allow up to 10s for the last poll + handler
    });
    
    const shutdown = (signal) => {
      console.log(`Received ${signal}, waiting for in-flight work...`);
      consumer.stop();
    
      consumer.once("waiting_for_polling_to_complete", () => {
        console.log("Still processing in-flight messages...");
      });
    
      consumer.once("waiting_for_polling_to_complete_timeout_exceeded", () => {
        console.warn("Graceful shutdown timed out, continuing shutdown anyway.");
      });
    
      consumer.once("stopped", () => {
        console.log("Consumer stopped cleanly");
        process.exit(0);
      });
    };
    
    process.once("SIGINT", shutdown);
    process.once("SIGTERM", shutdown);
    
    consumer.start();
  4. Configure AWS credentials manually

    main

    By default, the consumer uses the standard AWS SDK credential lookup (e.g., environment variables like AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY). To specify credentials manually, pass a pre-configured SQSClient instance via the sqs option.

    import { Consumer } from "sqs-consumer";
    import { SQSClient } from "@aws-sdk/client-sqs";
    
    const app = Consumer.create({
      queueUrl: "https://sqs.eu-west-1.amazonaws.com/account-id/queue-name",
      handleMessage: async (message) => {
        // ...
      },
      sqs: new SQSClient({
        region: "my-region",
        credentials: {
          accessKeyId: "yourAccessKey",
          secretAccessKey: "yourSecret",
        },
      }),
    });
    
    app.start();
  5. How message acknowledgement works in sqs-consumer

    main

    The consumer uses the return value of your handler to decide whether to delete a message from SQS (acknowledgement).

    For handleMessage (Single Message):

    • Return the message object (or an object with the same MessageId): The message is considered successfully processed and will be deleted from SQS.
    • Return undefined or null: The message is NOT deleted. If strictReturn is enabled in options, returning null will throw an error.
    • Return a different object: If the returned object has the same MessageId, it is deleted.
    • alwaysAcknowledge: true: If this option is set, the consumer will delete the message regardless of what the handler returns.

    For handleMessageBatch (Batch):

    • Return an array of Message objects: Only the messages included in this array will be deleted from SQS.
    • Return undefined or null: No messages in the batch will be deleted.
    • alwaysAcknowledge: true: All messages in the batch will be deleted regardless of the return value.
  6. Basic usage of sqs-consumer

    main

    To use sqs-consumer, create a consumer instance using Consumer.create() and provide a queueUrl and an async handleMessage function. You can listen to error and processing_error events to handle issues. By default, messages are processed one at a time using long polling.

    import { Consumer } from "sqs-consumer";
    
    const app = Consumer.create({
      queueUrl: "https://sqs.eu-west-1.amazonaws.com/account-id/queue-name",
      handleMessage: async (message) => {
        // do some work with `message`
      },
    });
    
    app.on("error", (err) => {
      console.error(err.message);
    });
    
    app.on("processing_error", (err) => {
      console.error(err.message);
    });
    
    app.start();
  7. Configure the Consumer with ConsumerOptions

    main

    The ConsumerOptions interface defines the configuration for an SQS consumer. Key configuration areas include:

    Message Handling

    • handleMessage: An async function called for each message.
      • To acknowledge/delete: Return the original Message or a processed Message.
      • To NOT acknowledge: Return undefined.
      • To selectively acknowledge: Return an object containing the MessageId you want to acknowledge.
    • handleMessageBatch: An async function called for a batch of messages. If set, it overrides handleMessage.
      • To acknowledge all: Return the original or processed array of Messages.
      • To NOT acknowledge any: Return undefined or an empty array [].
      • To selectively acknowledge: Return an array containing only the messages you want to acknowledge.
    • alwaysAcknowledge: If true, all messages are acknowledged regardless of the handler's return value. (Default: false)
    • shouldDeleteMessages: If false, the consumer will not delete messages from SQS. (Default: true)
    • strictReturn: If true, handlers returning null will throw an error instead of being treated as "do not acknowledge". (Default: false)

    Polling and Timing

    • batchSize: Number of messages to request per poll (Max 10). (Default: 1)
    • visibilityTimeout: Duration (seconds) messages are hidden after retrieval. (Default: 30 via SQS, but configurable)
    • waitTimeSeconds: Duration (seconds) to wait for a message to arrive (Long Polling). (Default: 20)
    • handleMessageTimeout: Time (ms) to wait for handleMessage to process before timing out. Emits timeout_error.
    • pollingWaitTimeMs: Duration (ms) to wait before repolling. (Default: 0)

    AWS and Client Configuration

    • sqs: An optional SQSClient instance.
    • region: The AWS region. (Default: process.env.AWS_REGION || 'eu-west-1')
    • useQueueUrlAsEndpoint: If false, uses the client's resolved endpoint instead of the queueUrl. (Default: true)

    Error and Visibility Management

    • terminateVisibilityTimeout: Sets visibility timeout to 0 after a processing_error. Can be true, a number, or a function (messages: Message[]) => number for exponential backoff.
    • heartbeatInterval: Interval (seconds) between requests to extend visibility timeout. Must be less than visibilityTimeout.
  8. Configure message visibility and heartbeats

    main

    The consumer manages message visibility to prevent other consumers from picking up the same message while it is being processed.

    • visibilityTimeout: The amount of time (in seconds) the message is hidden from other consumers.
    • heartbeatInterval: If provided, the consumer will automatically extend the visibility timeout of the message at this interval (in seconds) to prevent the message from expiring while the handler is still running.
    • terminateVisibilityTimeout: Defines what happens to the visibility timeout if the message handler fails. It can be a boolean or a number (seconds), or a function (messages: Message[]) => number that calculates the new timeout based on the failed messages.
  9. Consumer Events

    main
    The consumer is an EventEmitter and emits several events including error, processing_error, timeout_error, stopped, waiting_for_polling_to_complete, and waiting_for_polling_to_complete_timeout_exceeded.
  10. Required AWS IAM Permissions

    main

    The consumer requires the following permissions on the target SQS queue to receive and delete messages:

    • sqs:ReceiveMessage
    • sqs:DeleteMessage
    • sqs:DeleteMessageBatch
    • sqs:ChangeMessageVisibility
    • sqs:ChangeMessageVisibilityBatch