Decaton Documentation

Decaton is a streaming task processing framework built on top of Apache Kafka. Unlike many standard Kafka consumer frameworks, Decaton enables concurrent processing (multi-threaded or asynchronous) of records consumed from a single partition.

Key properties provided by Decaton:

• Concurrent processing: Process records from one partition concurrently.
• Ordering guarantee: Preserves ordering based on record keys.
• At-least-once delivery: Maintains at-least-once semantics regardless of the order in which records are processed.

Because BatchingProcessor performs flushing in a scheduled executor thread rather than the standard processing thread, you must control the parallelism of the flushing operation using ProcessorScope.

Standard decaton.partition.concurrency configuration is not sufficient for controlling batch-flush parallelism. Use the following scopes depending on your requirements:

• ProcessorScope.PARTITION: Parallelize flushing per partition.
• ProcessorScope.THREAD: Parallelize flushing per processor thread.

Task Compaction is a key-based feature used to reduce processing load by deciding which tasks to keep and which to discard when multiple tasks with the same key arrive within a specific time window.

How it works:

1. In-memory windowing: When a task arrives, it is placed in an in-memory window.
2. Key-based compaction: If a new task arrives with a key that already exists in the window, the CompactionProcessor uses a user-defined compactor function to decide which task survives. The 'winner' stays in the window, and the 'loser' is marked as completed (discarded).
3. Downstream propagation: The winning task is passed to the next processor in the pipeline only after it has resided in the window for longer than the configured lingerMillis duration.

To externalize Decaton processor properties and load them dynamically instead of hard-coding them, you must implement the PropertySupplier interface. Once implemented, you can configure the Decaton processor to use your custom supplier during the subscription building process.

import com.linecorp.decaton.processor.runtime.PropertySupplier;

// Implement PropertySupplier to provide dynamic configuration
public class MyCustomSupplier implements PropertySupplier {
    // ... implementation details
}

Decaton is optimized for high-throughput and low-latency processing where the task logic involves I/O-intensive operations against external systems (e.g., Database access, Web API calls).

Use Decaton when:

• Your processing logic contains I/O access that introduces latency per task.
• You need to maximize resource utilization through concurrent partition processing.

Use Kafka Streams when:

• You need complex stream processing or aggregations (e.g., stream joins, windowed processing).
• Your logic does not require frequent access to external storage or web APIs.

Decaton provides several advanced features for managing task flows:

• Retry Queuing: Retries failed tasks with back-off without blocking the flow of other tasks.
• Dynamic Rate Limiting: Allows applying and updating processing rate quotas dynamically.
• Task Compaction: Collapses preceding tasks whose results would be overwritten by a subsequent task.

When ProcessingContext#retry() is invoked, Decaton performs the following steps:

1. Marks the current task as completed.
2. Produces the task to the configured retry topic (e.g., original-topic-retry) with metadata indicating the scheduled retry time.
3. The Decaton consumer subscribes to both the normal topic and the retry topic.
4. When a task is polled from the retry topic, Decaton checks its metadata. If the scheduled retry time has not yet arrived, the consumer thread will block until the task is ready to be processed.

Decaton uses a token bucket algorithm for rate limiting. This implementation provides two key behaviors:

1. Throttling instead of discarding: When the rate limit is reached, the rate limiter slows down processing rather than discarding tasks.
2. Burst support: The rate limiter allows for sudden increases in traffic (burstiness) within the constraints of the token bucket logic.

Decaton uses an internal queuing and worker architecture to parallelize record processing.

Instead of strictly following partition-level ordering, Decaton relaxes the ordering guarantee from partition-based to key-based. It routes each record into an internal queue based on its key. Associated worker threads then consume from these queues and execute the processing logic you provide. This allows multiple threads to work on a single partition simultaneously while still maintaining order for records sharing the same key.

You can provide multiple property suppliers to a single processor. Suppliers are evaluated in the order they are passed to the .properties() method. The first occurrence of a property found in the list is the one that is adopted.

To override dynamic properties with hard-coded values, place a StaticPropertySupplier before the dynamic supplier.

.properties(
    StaticPropertySupplier.of(
        Property.ofStatic(ProcessorProperties.CONFIG_PARTITION_CONCURRENCY, 100)
    ),
    centralDogmaPropertySupplier
);

Decaton uses Semantic Versioning (MAJOR.MINOR.PATCH).

Backward compatibility in Decaton means:

• Applications referring to public APIs can still be compiled without errors.
• Applications using public APIs can expect similar behavior (slight differences that do not affect application behavior are permitted).
• ID-ish strings (property keys, metric names) remain exactly the same.

Version Increment Rules:

• MAJOR version: Incremented when incompatible API changes (breaking changes) are introduced.
• MINOR version: Incremented when new functionality is added in a backward-compatible manner.
• PATCH version: Incremented when backward-compatible bug fixes are made.