SeaStreamer

sea-streamer-redis is a high-level async implementation of the SeaStreamer abstract interface for Redis Streams. It abstracts away low-level Redis commands like XADD, XREAD, and XACK in favor of a type-safe API designed for high throughput (up to 100k messages per second).

Key Features

• RealTime mode: Includes AutoStreamReset behavior.
• Resumable mode: Supports auto-ack and/or auto-commit.
• LoadBalanced mode: Provides failover behavior.
• Time-based operations: Seek or rewind to a specific point in time.
• Stream Sharding: Basic support for splitting a stream into multiple sub-streams.
• Concurrency Model: Uses a mutex-free implementation where read and write loops are separated from your processing loop, allowing them to run in parallel via Rust's async runtime.

The buffered processor is an advanced pattern used when the input stream has a high frequency but the processor has high impedance (e.g., slow database inserts).

Instead of processing messages one by one, it uses an internal queue to decouple the input loop from the processing loop. This allows the processor to handle messages in batches, maximizing throughput and minimizing the impact of sudden bursts. This pattern is ideal for tasks like batch database writes where individual inserts are inefficient.

alias clock='cargo run --package sea-streamer-stdio --features=executables --bin clock'
# Pipe a high-frequency clock into the buffered processor
clock -- --stream clock --interval 100ms | \
cargo run --bin buffered -- --input stdio:///clock --output stdio:///output

The sea-streamer-types crate serves as the foundational layer for the SeaStreamer ecosystem. It defines all the core traits and types required for the SeaStreamer API.

Note: This crate is purely a definition layer; it does not provide any functional implementations. To actually use SeaStreamer features, you must use crates that implement these traits (such as sea-streamer-kafka, sea-streamer-redis, etc.).

The blocking processor pattern is used when you need to perform blocking I/O or heavy CPU-bound computations.

It utilizes a "fan out" pattern where tasks are randomly assigned to multiple worker threads. This allows the system to handle tasks that would otherwise block the main event loop, enabling the processor to catch up with high-speed input streams by parallelizing the workload across available threads.

alias clock='cargo run --package sea-streamer-stdio --features=executables --bin clock'
# Pipe a clock into the blocking processor
clock -- --stream clock --interval 333ms | \
cargo run --bin blocking -- --input stdio:///clock --output stdio:///output

When using the Redis backend, be aware of these fundamental differences compared to Kafka:

1. Sequence Numbers: In Redis, sequence numbers are not contiguous, unlike in Kafka.
2. Message Dispatching: In Redis, messages are dispatched to consumers in a first-ask-first-served manner among group members. This differs from Kafka's 1-to-1 consumer-to-shard mapping in a consumer group.
3. Acknowledgements (ACK): In Redis, ACK must be performed per message. In Kafka, a single Ack (read-up-to) can cover a series of reads.

SeaStreamer is built using a modular architecture of sub-crates that allow for backend-agnostic stream processing. The core components are:

• sea-streamer-types: Defines the fundamental traits and types for the SeaStreamer API. It contains no implementations.
• sea-streamer-socket: Provides a concrete-type API that allows you to select a backend (like Redis, Kafka, or File) at runtime without recompiling your stream processor.
• sea-streamer-runtime: An abstraction layer that aligns type signatures between smol and tokio, enabling you to build applications generic to both async runtimes.
• Backend Crates: Specific implementations for different streaming servers (e.g., sea-streamer-kafka, sea-streamer-redis, sea-streamer-file).

All crates share the same major version (e.g., 0.1 of sea-streamer depends on 0.1 of sea-streamer-socket).

The sea-streamer-socket crate provides a concrete-type API that allows stream processors to work with different SeaStreamer backends (like Redis or Kafka) selected at runtime.

Unlike the trait-based abstraction in sea-streamer-types, this API enables you to switch backends without recompiling your program. This is useful for scenarios such as generating data locally and streaming it to a production server, or sinking data from a server to a local environment for processing.

Note: If your application is strictly tied to a single backend, you should depend directly on the specific backend crate (e.g., sea-streamer-redis or sea-streamer-kafka) instead of using the socket abstraction.

Add sea-streamer to your Cargo.toml. To use Kafka, Redis, and the Tokio runtime, enable the corresponding features:

sea-streamer = { version = "0", features = ["kafka", "redis", "socket", "runtime-tokio"] }