DOCSEARCHbyAlgolia
Submit Docs

Stripe Sync Engine

repository·dev·Indexed 22 days ago

https://github.com/stripe/sync-engine

A service designed to synchronize data between Stripe and external destinations such as Postgres and Google Sheets. It features a Stripe Schema Visualizer for exploring generated schema data via PGlite, support for live WebSocket streaming of real-time events, and a suite of testing utilities including @stripe/sync-test-utils and @stripe/sync-ts-cli for configuration management.

Tokens
139.4K
Snippets
352
Records
549
Agent score
77%

What's inside stripe-sync-engine

  1. How @stripe/sync-test-utils works

    dev

    The @stripe/sync-test-utils package provides a complete testing environment for sync-engine integration tests without requiring an external mock server.

    Key features include:

    • Hono HTTP server: Automatically discovers listable Stripe endpoints from the OpenAPI spec and serves compatible list/retrieve responses.
    • Docker Postgres 18 helper: Automatically spins up a disposable Postgres 18 container with SSL, waits for readiness, and cleans up on exit. If you provide a POSTGRES_URL environment variable, the server will use that instead of starting an internal container.
    • DB seeding: Uses @stripe/sync-openapi to generate objects compliant with OpenAPI schemas and performs bulk-inserts into Postgres.
    • Validation: List query parameters are validated against OpenAPI parameter definitions (including v2 endpoints).
    • Timestamp control: Supports spreading created timestamps across a specific range using applyCreatedTimestampRange.

  2. Use the Stripe Schema Visualizer to explore schema data

    dev

    The Stripe Schema Visualizer is a standalone browser UI used to explore generated Stripe schema data using PGlite. It allows you to run SQL queries directly in your browser against the generated schema.

    The workflow consists of two steps:

    1. Generate artifacts: Build the static data required for the visualizer.
    2. Run the visualizer: Load those artifacts into PGlite to interact with the data.

    Note that this package is strictly for the schema visualizer UI; the deploy/install dashboard is located in packages/dashboard.

    pnpm explorer:build
pnpm visualizer

  3. Important usage warnings for Stripe Sync Engine

    dev

    Development Status

    Active development is currently occurring in the sync-engine-fork. This repository will eventually merge those changes, but currently, the fork is the primary development target.

    Security and Deployment

    Warning: This service lacks tight access controls. It should only be deployed internally and used at your own risk.

    Support and Bug Reporting

    • Bug reports are welcome.
    • Bug bounties are not being considered at this time.

    Legacy Versions

    For the original Supabase Stripe Sync Engine, refer to the og branch.

  4. Understand the Sync Engine monorepo structure

    dev

    The Stripe Sync Engine is organized as a monorepo where packages are separated by isolation boundaries. A fundamental architectural rule is that sources and destinations never depend on each other; they only interact through the core protocol or approved shared utilities.

    Core Packages

    • protocol: The foundation containing message types, interfaces, and Zod schemas.
    • openapi: Handles Stripe OpenAPI spec fetching and parsing.
    • logger: Provides structured logging (via pino) and progress UI (via ink).
    • source-stripe: The connector for the Stripe API source.
    • destination-postgres: The connector for Postgres destinations.
    • destination-google-sheets: The connector for Google Sheets destinations.
    • state-postgres: Manages Postgres state (migration runner and embedded migrations).
    • util-postgres: Shared Postgres utilities like upsert and rate limiting.
    • test-utils: Shared test helpers including servers, seeds, and Postgres fixtures.

    Applications

    • apps/engine: The core sync engine library, stateless CLI, and HTTP API.
    • apps/service: A stateful service for pipeline management using Temporal workflows.
    • apps/dashboard: A React web UI for managing pipelines.
    • apps/visualizer: A Next.js tool for data visualization.
    • apps/supabase: Supabase edge functions running in a Deno runtime.

  5. What is a single sync run in the Engine?

    dev

    A single sync run is a bounded attempt to move state forward. It is characterized by:

    • Parameters: Includes specific sync parameters and the current checkpoint state.
    • Output: Produces a resulting output stream.
    • Boundaries: A run may stop at a safe continuation boundary rather than exhausting the entire source.
    • Liveness: Progress is tracked via state, stream status, logs, and terminal output. Explicit liveness is required for effective orchestration.

  6. Understand `sync-engine` configuration precedence

    dev

    The CLI resolves configuration using a specific hierarchy. If a setting is defined in multiple places, the higher precedence item wins:

    1. CLI flags (Highest precedence)
    2. --config file (Full SyncParams JSON)
    3. Environment variables (e.g., STRIPE_API_KEY, POSTGRES_URL, DATABASE_URL)
    4. Built-in defaults (e.g., schema=stripe)

  7. Understand Streams and Catalogs

    dev

    The Sync Engine distinguishes between what is available and what is actually being synchronized using three levels of abstraction:

    1. Stream: Discovered via discover(). It describes the raw availability: name, primary key, JSON schema, and metadata.
    2. ConfiguredStream: The user's specific selection. It wraps a Stream with synchronization settings like sync_mode, destination_sync_mode, and an optional cursor_field.
    3. ConfiguredCatalog: A collection of ConfiguredStream[]. This is the persisted state of the Sync resource and is passed to both read() and write() methods to define the scope of the work.

  8. Understand the SyncOutput message format

    dev

    The SyncOutput type is the union of all messages yielded by the pipeline_sync operation. It combines destination output with source signals, ensuring that control messages, logs, and traces from the source connector are not lost during the sync process.

    SyncOutput includes:

    • StateMessage
    • TraceMessage
    • LogMessage
    • EofMessage
    • ControlMessage
    export const SyncOutput = z.discriminatedUnion('type', [
  StateMessage,
  TraceMessage,
  LogMessage,
  EofMessage,
  ControlMessage,
])

  9. Understand the Stripe Source Sync Lifecycle

    dev

    The Stripe source manages pagination within a time_range assigned by the engine using an n-ary search algorithm. Instead of upfront density probing, the source discovers the appropriate granularity by paginating and subdividing the range if a single request cannot complete the work.

    Key Concepts:

    • N-ary Search: The source starts with the full range and, if a range is not exhausted in one request, it subdivides the unpaginated portion into $N$ segments (where $N$ is max_segments_per_stream).
    • Constraints: Only resources supporting created[gte] and created[lt] filters are supported. Because the time_range.lt (the time_ceiling) is always in the past, no new objects can appear within a completed range, making subdivision safe.
    • State Management: The source maintains an internal StripeStreamState which includes the accounted_range and a list of remaining sub-ranges to be processed.
    type StripeStreamState = {
  accounted_range: {
    gte: string // ISO 8601 — inclusive lower bound
    lt: string // ISO 8601 — exclusive upper bound
  }
  remaining: Array<{
    gte: string // ISO 8601 — inclusive lower bound
    lt: string // ISO 8601 — exclusive upper bound
    cursor: string | null // Stripe pagination cursor; null = not yet started
  }>
}

  10. Understand the Stripe Sync architecture layers

    dev

    Stripe Sync is organized into three distinct layers that work together to move data from a source to a destination:

    1. Service: The stateful management layer. It handles Pipeline CRUD operations, manages credentials, persists state, and orchestrates long-running sync processes using Temporal workflows. It is exposed via a REST API and a CLI.
    2. Engine: The runtime layer. It is responsible for wiring a Source to a Destination, filtering messages (ensuring only data messages reach the destination), persisting state checkpoints, handling errors, and routing logs.
    3. Source / Destination: The implementation layer. These are the specific components that read from (Source) or write to (Destination) external systems, following the sync engine protocol defined in packages/protocol.

  11. Compare `/pipeline_sync` (backfill) and `/pipeline_handle_events` (events)

    dev

    Understanding the behavioral differences between these two modes is critical for choosing the right endpoint for your use case.

    Feature/pipeline_sync (backfill)/pipeline_handle_events
    Source reads fromUpstream APIProvided events
    Input bodyNone (ignored)Required
    time_limitApplies (may cut mid-page)Not applicable (processes full batch)
    state_limitAppliesOptional (events are typically small batches)
    Typical callerScheduler / cronWebhook receiver / event bus

  12. Handle the EOF terminal message in NDJSON streams

    dev

    Every NDJSON streaming response from the Sync Engine (specifically /read and /sync endpoints) concludes with a terminal message of type eof. This message is sent as the last line of the response to explicitly signal why the stream ended, avoiding the ambiguity of transport-level EOF.

    An eof message will contain a reason field indicating the termination cause.

    {"type":"record","stream":"customer","data":{"id":"cus_1"},"emitted_at":"..."}
{"type":"state","stream":"customer","data":{"cursor":"cus_1"}}
{"type":"eof","reason":"state_limit"}