Oura

Oura is a Rust-native pipeline designed to observe the Cardano blockchain and react to specific event patterns. It connects to a Cardano node (via Unix socket or TCP), filters events based on patterns, and submits payloads to pluggable "sinks".

Unlike tools designed for full blockchain exploration, Oura is optimized for real-time observation and integration with data-streaming pipelines. It has a small memory and CPU footprint, making it suitable for resource-constrained environments like Raspberry Pis.

Key Components:

• Sources: Pull data from the blockchain (e.g., Node-to-client, S3, Hydra, Mithril, Utxorpc) and map raw blocks into granular events.
• Filters: Transform or select events (e.g., Parse CBOR, Select, Wasm Plugin, Rollback Buffer).
• Sinks: Submit final events to external systems (e.g., Kafka, AWS Lambda, Webhooks, SQL Databases, Stdout).

Oura is a Rust-native pipeline designed to "observe" the Cardano blockchain in real-time. It connects to a Cardano node via Ouroboros mini-protocols (using a Unix socket or TCP), filters events based on user-defined patterns, and forwards the resulting payloads to pluggable sinks (e.g., terminal, file, message broker, database).

Key characteristics:

• Low Footprint: Designed to run in resource-constrained environments like Raspberry Pis.
• Pipeline Model: Uses multi-threading and mpsc channels to keep each stage (sources, filters, sinks) independent.
• Extensible: Can be used as a standalone CLI tool, a daemon for data streaming, or as a Rust library for custom implementations.

The PostgreSQL sink allows you to split blocks into individual transactions and persist their raw CBOR data into a PostgreSQL database using templated SQL statements.

Pipeline Workflow:

1. Source (N2N): Consumes from the mainnet relay, starting from a specific Point in [intersect].
2. Filter (SplitBlock): Breaks each block down into individual transactions.
3. Sink (SqlDb): Executes apply_template, undo_template, and reset_template SQL statements against a postgres:// connection. It stores each transaction's slot and CBOR. Rollbacks and resets are managed via the undo and reset templates respectively.

flowchart LR
  src[N2N source] --> f1[SplitBlock] --> sink[SqlDb sink]

The GCP Pub/Sub sink is a pipeline component that decodes transactions and publishes each one as a message to a Google Cloud Pub/Sub topic.

In a typical pipeline, the flow is:

1. Source (N2N): Fetches data from the mainnet relay starting from the chain tip.
2. Filter (SplitBlock): Breaks blocks into individual transactions.
3. Filter (ParseCbor): Decodes raw transaction CBOR into structured records.
4. Sink (GcpPubSub): Publishes the resulting events to a specified topic.

The RollbackBuffer filter is a mechanism designed to mitigate the impact of blockchain rollbacks on downstream stages. It holds blocks in memory until they reach a specific depth (min_depth) before forwarding them.

This is particularly useful when your sink (the final destination of your data) writes to storage systems where deleting or undoing orphaned data caused by a rollback is expensive or difficult.

How it handles rollbacks:

• Rollbacks within the buffer: The rollback is resolved in memory. The sink never sees the orphaned data, avoiding expensive cleanup operations.
• Rollbacks outside the buffer: Oura reverts to standard behavior and emits a rollback event, requiring the sink to handle the rollback manually.

A cursor persists the pipeline's position so that it survives restarts. Without a cursor, Oura restarts from its configured intersect point, which can lead to gaps or require re-processing large amounts of data if you start from a fixed point in the past.

With a cursor enabled, the sink stage reports its position as it works, and that position is periodically saved to a backend. Upon restart, Oura loads the saved position and resumes the source from exactly where it left off.

Oura is a Rust-native pipeline designed to "observe" the Cardano blockchain by reacting to specific event patterns.

The Pipeline Flow:

1. Source: Connects to the tip of a Cardano node via the Ouroboros mini-protocol (using a Unix socket or TCP bearer).
2. Filter: Processes the stream to find events that match a specific pattern.
3. Sink: Submits a succinct, self-contained payload to pluggable observers (sinks).

Oura is built using the Pallas library for Ouroboros multiplexer and state-machine implementations (ChainSync and LocalState).

The Fingerprint filter computes a unique identifier for each event using the murmur3 non-cryptographic hash algorithm. This identifier is appended to the event data, allowing downstream sinks to identify and discard duplicate records by using the fingerprint property as a primary key.

When enabled, the filter sets a fingerprint property on the Event data structure passed through the pipeline stages.

The AwsLambda sink decodes transactions and invokes a specific AWS Lambda function once per event. The pipeline typically follows this flow:

1. Source (N2N): Mainnet relay starting from the chain tip.
2. Filter (SplitBlock): Breaks blocks into individual transactions.
3. Filter (ParseCbor): Decodes raw transaction CBOR into structured records.
4. Sink (AwsLambda): Invokes the configured function_name in a specific region, passing each event as a JSON payload.

To use this sink, the project must be built with the aws feature enabled.

For complex or highly specific requirements, Oura can be imported as a library into your Rust project. You can compose tailor-made pipelines by reusing existing components:

• Sources: Connect to Cardano nodes.
• Filters: Pinpoint specific transaction patterns.
• Sinks: Forward data to custom destinations.

Each component is designed to be self-contained and reusable.