Sequin Documentation

The Sequin Management API enables programmatic control over Sequin resources. It is designed for developers who need to manage infrastructure from within their own applications, such as automating the setup of sinks and backfills for customers or re-populating cache destinations via automatic backfills.

Key resources managed via this API include:

• Sinks: Streaming data from a Postgres table to a destination.
• HTTP Endpoints: Configuring webhook destinations for webhook sinks.
• Backfills: Backfilling data from Postgres tables to a sink.

Sequin is a high-performance Change Data Capture (CDC) platform for Postgres. It allows you to stream changes from any Postgres database (version 14+) to various streaming platforms, queues, search indexes, and more (e.g., Kafka, GCP Pub/Sub, Redis, etc.).

Key capabilities include:

• Real-time streaming: Stream new changes as they occur.
• Backfilling: Replay existing rows or specific subsets of data to sinks.
• Guaranteed delivery: Ensures 100% delivery of database changes to sinks with strict ordering and exactly-once processing using idempotency keys.
• Data manipulation: Use filters to include/exclude changes, transforms to modify payloads via Elixir functions, and routing to direct messages to specific topics or endpoints.
• Deployment: Runs as a standalone Docker container that can be deployed alongside your Postgres database.

Sequin provides official SDKs for several programming languages, all of which are built on top of the Sequin HTTP API. These SDKs offer a more developer-friendly interface for dynamically creating consumers and interacting with messages compared to raw HTTP calls.

Supported SDKs and integrations include:

• JavaScript: Official SDK for JS environments.
• Elixir: Official SDK for Elixir environments.
• Elixir Broadway: Integration specifically for using Sequin with the Elixir Broadway library.
• Go: Official SDK for Go environments.
• Ruby: Official SDK for Ruby environments.
• Python: Official SDK for Python environments.

Sequin Managed on AWS provides a single-tenant deployment of Sequin within a dedicated sub-account owned by your AWS organization. While Sequin manages the infrastructure (provisioned via Terraform and ECS), the environment is contained within your AWS account. You continue to use the standard Sequin web console and CLI to interact with it.

Note: Sequin Managed is no longer available. Refer to the Sequin Cloud shutdown notice for details.

Sequin is designed for high-performance Change Data Capture (CDC), capable of sustaining 40k operations per second (40 MB/s) with an average latency of 55ms.

To handle high-throughput Postgres databases without falling behind, Sequin can be configured with multiple replication slots. This allows for low-latency streaming even as database load increases.

Sequin provides several advanced features for managing CDC pipelines:

• Database Compatibility: Works with any Postgres database version 14+ without requiring extensions.
• Guaranteed Delivery: Ensures 100% delivery of changes to sinks with strict ordering.
• Filters: Use custom filters to include or exclude specific changes from a sink.
• Transforms: Modify message payloads using low-latency Elixir functions.
• Routing: Route messages to specific topics, endpoints, or indexes.
• Exactly-once processing: Uses idempotency keys to ensure changes are processed exactly once.
• Backfills: Replay historical data or specific rows at any time.
• Observability: Provides a Prometheus metrics endpoint and a Grafana dashboard.
• Management Tools: Manage sinks via the Sequin console, CLI, sequin.yml, or the Management API.

The sink maps Postgres change actions to specific Redis operations. By default, keys follow the pattern sequin:{table_name}:{primary_key} (composite primary keys are joined with colons).

You can attach custom logic to a sink consumer using the following function fields. These functions allow you to manipulate data as it flows from the source to the destination:

• enrichment: An enrichment function to add data to the message.
• filter: A filter function to decide which messages should be passed through.
• transform: A transform function to modify the message content.
• routing: A routing function to determine the destination of the message.

By default, Sequin publishes messages to NATS subjects using a hierarchical pattern based on the source data:

sequin.<database_name>.<schema_name>.<table_name>.<action>

Example: For a table named products in the public schema of a database named shop_prod, the subjects will be:

• sequin.shop_prod.public.products.insert
• sequin.shop_prod.public.products.update
• sequin.shop_prod.public.products.delete

You can override this default behavior by using a routing function to generate dynamic subjects.

To ensure your application can safely reject duplicate messages (due to delivery guarantees), use the metadata.idempotency_key provided in every message.

• For regular changes (non-backfill): The key is derived from the transaction's position in the database log and its position within that transaction.
• For backfill messages: The key is derived from the backfill's unique identifier and the primary key values of the record being processed.

Sequin is designed for high-performance Postgres Change Data Capture (CDC). When streaming from a single Postgres replication slot to Kafka, Sequin achieves the following sustained performance:

• Throughput: 50k ops/s or 40 MB/s (whichever is reached first).
• Average Latency: 55ms.
• 99th-percentile Latency: 253ms.

To handle even higher throughput without falling behind, you can configure Sequin with multiple replication slots, which allows it to scale linearly.

Sequin supports ingesting messages from two primary source types:

1. postgres: Ingests changes from existing Postgres tables via the Write-Ahead Log (WAL). This allows you to treat creates, updates, and deletes from any existing Postgres table as a stream of messages.
2. webhook: Ingests messages from external APIs.