sqs-producer

repository·main·Indexed 42 days ago

https://github.com/bbc/sqs-producer

A utility library for enqueuing messages into Amazon SQS queues using the AWS SDK v3. It simplifies the process of sending single or batched messages, handles FIFO queue requirements, and provides a Producer class for managing queue interactions, including retrieving queue size and handling FailedMessagesError.

Tokens
3.6K
Snippets
12
Records
15
Agent score
52%

What's inside sqs-producer

  1. Configure AWS credentials

    main

    The producer follows standard AWS SDK credential resolution.

    1. Environment Variables: The simplest method is to export your credentials in your environment:
      export AWS_SECRET_ACCESS_KEY=...
      export AWS_ACCESS_KEY_ID=...
    2. Manual Configuration: Pass a custom SQSClient instance to Producer.create() using the sqs key.
  2. How batching and error handling work in the Producer

    main

    The Producer is designed to abstract the complexities of SQS batching.

    Batching Logic

    When you call send(), the producer calculates how many messages to include in a single SendMessageBatchCommand based on the batchSize option (which must be between 1 and 10). If the input array is larger than the batchSize, the producer automatically splits the work into multiple sequential batches.

    Error Handling

    The producer tracks both successful and failed message entries across all batches.

    • If all messages are successfully sent across all batches, the method resolves with the array of successful entries.
    • If any messages fail (e.g., due to invalid IDs or internal SQS errors), the producer will collect the IDs of all failed messages and throw a FailedMessagesError. You should catch this error to identify which specific messages failed to be delivered.
  3. Check the current queue size

    main

    Use producer.queueSize() to retrieve the current number of messages in the queue. This method is asynchronous and returns a Promise.

    const size = await producer.queueSize();
    console.log(`There are ${size} messages on the queue.`);
  4. Send messages to an SQS queue

    main

    The producer.send() method allows you to enqueue messages. You can pass an array of strings (where the string is the body) or an array of message objects for more control.

    Supported message object properties:

    • id: A unique identifier (defaults to the body if not provided).
    • body: The message content.
    • delaySeconds: A number between 0 and 900 to delay the message.
    • messageAttributes: An object containing metadata (e.g., String or Binary).
    • groupId: Required for FIFO queues or for implementing fair queue behavior in standard queues.
    • deduplicationId: Required for FIFO queues (unless content-based deduplication is enabled).
    // Send simple strings
    await producer.send(["msg1", "msg2"]);
    
    // Send complex message objects
    await producer.send([
      {
        id: "id1",
        body: "Hello world",
        messageAttributes: {
          attr1: { DataType: "String", StringValue: "stringValue" },
          attr2: { DataType: "Binary", BinaryValue: new Buffer("binaryValue") },
        },
      },
      {
        id: "id2",
        body: "Delayed message",
        delaySeconds: 5,
      },
    ]);
    
    // Send to a FIFO queue
    await producer.send({
      id: "testId",
      body: "FIFO message",
      groupId: "group1234",
      deduplicationId: "abcdef123456",
    });
  5. Create a Producer instance

    main

    Use Producer.create() to initialize a new producer. You must provide a queueUrl and a region. You can optionally provide a pre-configured SQSClient instance to manage credentials and region settings manually.

    import { Producer } from "sqs-producer";
    import { SQSClient } from "@aws-sdk/client-sqs";
    
    // Simple initialization
    const producer = Producer.create({
      queueUrl: "https://sqs.eu-west-1.amazonaws.com/account-id/queue-name",
      region: "eu-west-1",
    });
    
    // Initialization with a custom SQSClient
    const producerWithClient = Producer.create({
      queueUrl: "https://sqs.eu-west-1.amazonaws.com/account-id/queue-name",
      region: "eu-west-1",
      sqs: new SQSClient({
        region: "my-region",
        credentials: {
          accessKeyId: "yourAccessKey",
          secretAccessKey: "yourSecret",
        },
      }),
    });
  6. Configure the Producer with ProducerOptions

    main

    When initializing the producer, you provide a ProducerOptions object to configure the target queue and the AWS client behavior.

    Key configuration options include:

    • queueUrl (required): The URL of the SQS queue.
    • batchSize (optional): The number of messages to send in a single batch.
    • sqs (optional): An existing SQSClient instance. If omitted, a new client is created.
    • region (optional): The AWS region. Defaults to eu-west-1 if not provided or found in the AWS_REGION environment variable.
    • useQueueUrlAsEndpoint (optional): If true (default), the queueUrl is used as the request endpoint. Set to false to use the client's resolved endpoint (useful for custom endpoints).
    import { SQSClient } from "@aws-sdk/client-sqs";
    
    const options: ProducerOptions = {
      queueUrl: "https://sqs.eu-west-1.amazonaws.com/123456789012/my-queue",
      batchSize: 10,
      region: "eu-west-1",
      sqs: new SQSClient({ region: "eu-west-1" })
    };
  7. Initialize the Producer

    main

    To use sqs-producer, instantiate the Producer class or use the Producer.create() static method. You must provide a queueUrl.

    Optional configuration includes:

    • batchSize: The number of messages to send in each SQS batch (defaults to 10). Must be between 1 and 10.
    • sqs: An existing instance of SQSClient from @aws-sdk/client-sqs. If not provided, a new client is created.
    • region: The AWS region (defaults to eu-west-1 or process.env.AWS_REGION).
    • useQueueUrlAsEndpoint: Determines if the queueUrl should be used as the endpoint (defaults to true).
    • Any other valid options for the AWS SDK SQSClient can be passed in the options object.
    import { Producer } from 'sqs-producer';
    
    // Using the static create method
    const producer = Producer.create({
      queueUrl: 'https://sqs.eu-west-1.amazonaws.com/123456789012/my-queue',
      batchSize: 5,
      region: 'eu-west-1'
    });
    
    // Or using the constructor
    const producerAlt = new Producer({
      queueUrl: 'https://sqs.eu-west-1.amazonaws.com/123456789012/my-queue'
    });
  8. Send messages to SQS

    main

    The send() method allows you to dispatch messages to your SQS queue. It accepts a single string, a single Message object, or an array containing both strings and Message objects.

    Internally, the producer automatically handles batching based on the batchSize configured during initialization. If you provide more messages than the batchSize, the producer will recursively send multiple batches until all messages are processed.

    Returns: A promise that resolves to an array of SendMessageBatchResultEntry[] representing the successful messages.

    Errors: If any messages in the batch fail to send, the method throws a FailedMessagesError containing the IDs of the failed messages.

    // Sending a single string message
    await producer.send('hello world');
    
    // Sending an array of mixed strings and Message objects
    await producer.send([
      'message one',
      { id: 'msg-2', body: 'message two' },
      { id: 'msg-3', body: 'message three' }
    ]);
  9. Use the Producer class

    main

    The primary entry point for the library is the Producer class, exported from ./producer.js. You should use this class to instantiate a producer capable of sending messages to an Amazon SQS queue.

    import { Producer } from 'sqs-producer';
    
    // Usage depends on the Producer implementation in ./producer.js
    const producer = new Producer({ /* options */ });
  10. Get the current queue size

    main

    The queueSize() method returns the approximate number of messages currently residing in the SQS queue. This is useful for monitoring queue depth or implementing backpressure logic in your producer.

    const size = await producer.queueSize();
    console.log(`There are approximately ${size} messages in the queue.`);