RisingWave — Complete Reference

website·Indexed 35 days ago

https://docs.risingwave.com/

An event streaming platform for incremental data processing from databases, streams, and webhooks. Documentation covers PostgreSQL differences, JSONB operations, Kafka security configurations, and streaming patterns such as Tumble Windows.

Tokens: 7.3K
Snippets: 44
Records: 49
Agent score: 20%

What's inside RisingWave

Understand RisingWave Architecture and Execution Modes
RisingWave consists of four node types:
- Frontend: Stateless proxy for PostgreSQL protocol, parsing, and query planning.
- ComputeNode: Executes streaming and batch query plans.
- MetaServer: Manages cluster metadata, catalogs, and storage metadata.
- Compactor: Handles storage compaction tasks.
Storage is based on object storage (S3, GCS, MinIO, Azure Blob) with local SSD/EBS caching.
Execution Modes:
- Batch mode: Standard SELECT queries for immediate results.
- Streaming mode: CREATE MATERIALIZED VIEW creates a persistent pipeline that incrementally maintains results.
https://docs.risingwave.com/llms-full.txt?llms-doc=risingwave-complete-referencewebsitellms
Quick start RisingWave with Docker
Run a single-node RisingWave instance using Docker. The instance exposes port 4566 for PostgreSQL protocol connections and port 5691 for internal communication.
```
# Start RisingWave
docker run -it --pull=always -p 4566:4566 -p 5691:5691 risingwavelabs/risingwave:latest single_node

# Connect via psql
psql -h localhost -p 4566 -d dev -U root
```
https://docs.risingwave.com/llms-full.txt?llms-doc=risingwave-complete-referencewebsitellms

Emit final window results using EMIT ON WINDOW CLOSE

To produce append-only output that only emits the final result when a time window closes, use the EMIT ON WINDOW CLOSE clause. This requires a watermark to be defined on the source to track event time progress.

-- 1. Define source with watermark
CREATE SOURCE events (
    event_id INT, event_time TIMESTAMPTZ, amount DECIMAL,
    WATERMARK FOR event_time AS event_time - INTERVAL '10' SECOND
) WITH (...) FORMAT PLAIN ENCODE JSON;

-- 2. Create MV that emits only on close
CREATE MATERIALIZED VIEW hourly_totals AS
SELECT
    window_start, SUM(amount) AS total
FROM TUMBLE(events, event_time, INTERVAL '1 HOUR')
GROUP BY window_start
EMIT ON WINDOW CLOSE;

Create and manage Materialized Views
Materialized views in RisingWave are continuously and incrementally maintained; they do not require a REFRESH MATERIALIZED VIEW command. They update automatically within sub-seconds when upstream data changes. You can stack materialized views (MV-on-MV) to create multi-layer pipelines.
To avoid blocking the session during long backfills, use SET BACKGROUND_DDL = true; before creation.
```
-- Enable background DDL for non-blocking creation
SET BACKGROUND_DDL = true;

CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] mv_name
[ WITH ( parameter = value [, ... ] ) ]
AS select_query
[ EMIT ON WINDOW CLOSE ];

-- Check backfill progress
SELECT * FROM rw_catalog.rw_fragment_backfill_progress;
```
WITH Parameters:
- source_rate_limit: Throttle ingestion rate (rows/second) during backfill.
- backfill_order: Control backfill sequence (e.g., FIXED(t1 -> t2, t2 -> t3)).
https://docs.risingwave.com/llms-full.txt?llms-doc=risingwave-complete-referencewebsitellms

Configure Temporal Filters for data retention and late-arrival handling

Temporal filters allow you to automatically clean up old data from storage or delay processing to account for late-arriving events using NOW() and INTERVAL.

-- Data Retention: Keep only last 7 days
CREATE MATERIALIZED VIEW recent_orders AS
SELECT * FROM orders
WHERE order_time > NOW() - INTERVAL '7 DAYS';

-- Late-arrival handling: Delay processing by 5 seconds
CREATE MATERIALIZED VIEW delayed_events AS
SELECT * FROM events
WHERE event_time + INTERVAL '5' SECOND < NOW();

Manage secrets for connector configurations
Store sensitive information using CREATE SECRET and reference them in connector configurations using the secret keyword.
```
-- Create the secret
CREATE SECRET secret_name WITH ( backend = 'meta' ) AS 'secret_value';

-- Use in a connector config
-- WITH ( password = secret secret_name )
```
https://docs.risingwave.com/llms-full.txt?llms-doc=risingwave-complete-referencewebsitellms
Choose between SOURCE and TABLE for data ingestion
RisingWave provides different ways to declare external data streams:
- CREATE SOURCE: Declares an external stream. Data is NOT persisted in RisingWave. It is read-only (no INSERT/UPDATE/DELETE) and is primarily used for streaming ingestion into materialized views.
- CREATE TABLE (with connector): Declares an external stream AND persists the data. Supports primary keys and DML operations (INSERT/UPDATE/DELETE). Use this when you need to query raw ingested data or require primary key deduplication.
- CREATE TABLE (without connector): A standard mutable table for reference or dimension data managed manually.
https://docs.risingwave.com/llms-full.txt?llms-doc=risingwave-complete-referencewebsitellms
Consume real-time changes using Subscriptions
Subscriptions allow you to consume incremental changes from tables or materialized views without a message broker. You create a subscription, declare a cursor, and fetch changes. The result includes the table columns plus op (operation type) and rw_timestamp (Unix timestamp in ms for exactly-once delivery).
```
-- Create subscription with 1 day retention
CREATE SUBSCRIPTION my_sub FROM my_mv WITH (retention = '1D');

-- Declare cursor and fetch changes
DECLARE cur SUBSCRIPTION CURSOR FOR my_sub SINCE NOW();
FETCH NEXT FROM cur WITH (timeout = '5s');

-- Resume from specific timestamp for exactly-once delivery
DECLARE cur SUBSCRIPTION CURSOR FOR my_sub SINCE 1700000000000;
```
https://docs.risingwave.com/llms-full.txt?llms-doc=risingwave-complete-referencewebsitellms
Push notifications using Subscriptions
Subscriptions allow applications to receive a stream of changes from a Materialized View. You create a subscription and then use a SUBSCRIPTION CURSOR to fetch changes.
```
-- Create subscription
CREATE SUBSCRIPTION fraud_alerts_sub FROM fraud_signals WITH (retention = '1D');

-- Fetch changes via cursor
DECLARE cur SUBSCRIPTION CURSOR FOR fraud_alerts_sub;
FETCH NEXT FROM cur WITH (timeout = '5s');
```
https://docs.risingwave.com/llms-full.txt?llms-doc=risingwave-complete-referencewebsitellms
Connect to RisingWave using PostgreSQL drivers
RisingWave is wire-compatible with PostgreSQL. Use any standard PostgreSQL driver to connect. The default connection parameters are: Host localhost, Port 4566, User root, and Database dev.
```
# Using psycopg2
import psycopg2
conn = psycopg2.connect(host="127.0.0.1", port=4566, user="root", dbname="dev")
conn.autocommit = True

# Using asyncpg
import asyncpg
conn = await asyncpg.connect("postgresql://root@localhost:4566/dev")
```
https://docs.risingwave.com/llms-full.txt?llms-doc=risingwave-complete-referencewebsitellms
Configure Kafka SASL/PLAIN security
To connect to a secured Kafka cluster using SASL/PLAIN, specify the security protocol, mechanism, username, and password within the WITH clause properties.
```
WITH (
    connector = 'kafka',
    topic = 'my_topic',
    properties.bootstrap.server = 'broker:9092',
    properties.security.protocol = 'SASL_SSL',
    properties.sasl.mechanism = 'PLAIN',
    properties.sasl.username = 'user',
    properties.sasl.password = 'pass'
)
```
https://docs.risingwave.com/llms-full.txt?llms-doc=risingwave-complete-referencewebsitellms
Configure Watermarks for time-windowed computations
Watermarks track event-time progress and are required for time-windowed computations using EMIT ON WINDOW CLOSE.
```
CREATE SOURCE events (
    event_id INT,
    event_time TIMESTAMPTZ,
    payload VARCHAR,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (...) FORMAT PLAIN ENCODE JSON;
```
https://docs.risingwave.com/llms-full.txt?llms-doc=risingwave-complete-referencewebsitellms