Namastack Outbox Documentation

The namastack-outbox-examples directory provides several specialized projects to demonstrate library features:

Handler Registration

• Annotation-based: Use @OutboxHandler for registering handlers.

Resilience & Error Handling

• Retry Policies: Demonstrates automatic retry behavior.
• Fallback Handlers: Use @OutboxFallbackHandler to handle permanent failures.

Integration Patterns

• Message Brokers: Externalizing records to Apache Kafka (Java/Kotlin), RabbitMQ (Java/Kotlin), or AWS SNS (Java/Kotlin via LocalStack).
• Spring Modulith: Externalizing payment events to Kafka.
• Spring Events: Using the multicaster example for transaction isolation between Spring events and the outbox.
• Observability: Distributed tracing with OpenTelemetry and Micrometer.

Database & Schema Management

• Database Types: JDBC (no JPA/Hibernate), JPA, MySQL, MariaDB, PostgreSQL, SQL Server, and MongoDB.
• Schema Management: Using Flyway migrations for production-ready manual schema management, or using custom H2 schemas and table prefixes via Hibernate's PhysicalNamingStrategy or manual JDBC configuration.

You can use Namastack Outbox to ensure that Spring Modulith events marked as @Externalized are reliably delivered to Apache Kafka. When Spring Modulith is configured with externalization.mode=outbox, it delegates the delivery of externalized events to Namastack Outbox. This ensures that events are only published to Kafka after the local transaction (e.g., JPA/H2) has successfully committed, preventing data inconsistency between your database and Kafka.

In this pattern:

1. A module publishes an event.
2. The event is marked with @Externalized specifying the Kafka topic and key.
3. Spring Modulith intercepts the event and hands it to Namastack Outbox.
4. Namastack Outbox persists the event in an outbox table/collection.
5. The event is eventually published to the Kafka topic.

spring:
  modulith:
    events:
      externalization:
        enabled: true
        mode: outbox

Namastack Outbox provides several core reliability guarantees to ensure robust event-driven systems. These guarantees are enforced at the library level and apply regardless of your database or messaging broker.

Guaranteed Behaviors

• At-Least-Once Delivery: Every record is delivered to its handler at least once. Records are never silently dropped. If a handler fails or the application crashes, the record remains in the database and is retried.
• Ordering Per Key: Records sharing the same key are processed sequentially in the order they were inserted. This is enforced via partition-based processing where a single key is assigned to a single partition.
• Failure Recovery: System failures (JVM crashes, OOM, etc.) do not result in lost records. Records in PENDING or PROCESSING states are resumed upon restart. Mid-processing records are detected as stale after a configurable timeout and returned to PENDING.
• Transactional Consistency: Outbox records are written within the same database transaction as your business data, solving the "dual-write problem."
• Horizontal Scalability: Multiple instances can process records concurrently using database-level locking to distribute partitions. Adding instances provides linear throughput scaling.
• Automatic Retry & Rebalancing: Failed handlers are retried based on backoff policies. When instances join or leave the cluster, partitions are automatically redistributed during the next polling cycle.

Non-Guaranteed Behaviors (Important for Design)

• NOT Exactly-Once Delivery: Records may be processed more than once (e.g., if a crash occurs after handler success but before the record is marked PROCESSED). Handlers MUST be idempotent.
• NOT Global Ordering: There is no guaranteed order across records with different keys. To achieve global ordering, you must use a single key for all records (which limits parallelism).
• NOT Real-Time Processing: Processing is asynchronous and occurs on polling intervals (default is 2 seconds). It is designed for durability, not sub-second low-latency messaging.

Namastack Outbox uses a Chain of Responsibility pattern to process outbox records through four distinct stages. Each stage handles a specific concern and decides whether to pass the record to the next processor in the sequence.

The Four Processors

1. Primary Handler Processor: Invokes your registered handler for the payload type. If successful, the record is marked COMPLETED. If an exception occurs, it passes to the Retry Processor.
2. Retry Processor: Checks if the exception is retryable and if the retry limit has been reached. If retries are still available, it schedules a retry with a calculated delay. If retries are exhausted, it passes to the Fallback Processor. Note that OutboxRecordMetadata.failureCount tracks previous failed attempts.
3. Fallback Processor: Attempts to invoke a registered fallback handler. If the fallback succeeds, the record is marked COMPLETED. If the fallback fails or no fallback is configured, it passes to the Permanent Failure Processor.
4. Permanent Failure Processor: Marks the record as permanently FAILED. This is the final state, and no further processing occurs.

The Multicaster pattern demonstrates how to achieve isolation between Spring application events and outbox transactions. In this pattern, you can publish Spring application events (which may be processed asynchronously via @Async) alongside scheduling outbox records within the same @Transactional method.

Key behaviors of this isolation include:

• Exception Isolation: Exceptions thrown in @Async event listeners do not trigger a rollback of the database transaction containing the outbox record.
• Transactional Integrity: The outbox record is persisted successfully even if application event listeners fail.
• Independent Processing: The outbox handler processes the scheduled record independently of the application event lifecycle.

@Service
class CustomerService {
    @Transactional
    fun register(...) {
        // 1. Save customer to database
        customerRepository.save(customer)
        
        // 2. Publish Spring application event (async)
        applicationEventPublisher.publish(event)
        
        // 3. Schedule outbox record (transactional)
        outbox.schedule(event)
    }
}