Ursula

Ursula is a self-hosted, distributed server designed for replayable, append-only event timelines. It is used to manage the history behind document edits, agent runs, workflows, and chat sessions.

Key Characteristics

• HTTP-native: Uses PUT to create, POST to append, and GET to replay or tail via long-poll/SSE.
• One timeline per resource: Instead of high-throughput pipelines, Ursula uses a stream per document, session, workflow, room, or agent run.
• Thread-per-core × multi-Raft: Each stream hashes to a specific Raft group and owner core. This architecture ensures that hot-path requests touch only one core, avoiding cross-core synchronization and preventing slow followers in one group from stalling others.
• Hot ring + S3 cold tier: Writes are committed at Raft quorum in an in-memory ring (hot tier). Older segments are flushed to S3 (cold tier) in the background. A single GET request transparently spans both tiers.

The Ursula codebase is undergoing a multi-stage reduction plan to decrease its size by approximately 21-24k lines. This plan is categorized by risk levels, moving from mechanical code deduplication to structural unification, feature retirement, and finally test-surface pruning.

Reduction Stages Overview

1. Stage 1: Mechanical Shrink (~8-10k lines) Focuses on deduplication and refactoring with zero behavior or wire-format changes. This includes merging ursula-sim binaries, consolidating HTTP request helpers, and migrating error handling to thiserror.

2. Stage 2: Structural Unification (~4-5k lines) Changes internal APIs and inter-node wire formats. Key changes include unifying command/response models (e.g., StreamCommand as the canonical type) and using Serde-carried Raft envelopes instead of bespoke proto mirrors. This stage requires full Distributed System Test (DST) corpus validation.

3. Stage 3: Feature Retirement (~2-3k lines) Involves removing specific capabilities based on product decisions. Examples include dropping the WAL persistence backend, removing S2-compatible API routes, and dropping JSON/YAML config formats in favor of TOML.

4. Stage 4: Test-Surface Pruning (~4-5k lines) Reduces test redundancy by deleting integration tests covered by existing madsim DST scenarios and parameterizing corruption/failover seed families.

Ursula is a distributed implementation of the Durable Streams Protocol. It provides a primitive for managing ordered, append-only sequences of changes (timelines) that are replayable, durable, and live-tailable.

Key Protocol Operations

Any standard HTTP client can interact with Ursula using the following methods:

• PUT: Creates a new stream.
• POST: Appends bytes to an existing stream.
• GET: Reads from a specific offset or live-tails the stream using Server-Sent Events (SSE).

Core Characteristics

• URL-addressable: Streams are identified by URLs.
• Opaque Offsets: Offsets are treated as opaque tokens for replay semantics.
• Distributed Durability: Uses Raft-replicated in-memory hot rings for low-latency writes, with a background flusher to an S3-backed cold tier for long-term storage.
• Transparent Scaling: Supports quorum-replicated writes and transparent follower forwarding, meaning any node in the cluster can serve requests.

Because Ursula Record Coordinates v1 does not include a server-side timestamp, you should use an external indexer to enable time-based queries.

The Indexing Contract:

1. Source of Truth: Ursula remains the canonical authority for bytes, offsets, record ordinals, retention, and replay.
2. Extraction: An indexer consumes records in ordinal order and extracts the client timestamp from the application JSON.
3. Ordering: Event timestamps in the index may be out of order; the indexer must not reorder the source stream. Equal timestamps in query results should be ordered by their record ordinal.
4. Progress Tracking: The index publishes an indexed_through_record marker, which is the exclusive ordinal through which all retained source records have been processed.
5. Query Workflow: A time query returns record ordinals or coalesced ordinal ranges. The client then uses these ordinals to read the canonical data directly from Ursula.
6. Resilience: The index is derived state. It can be deleted and rebuilt from retained Ursula records without affecting the stream itself. If the index is being rebuilt and source retention has advanced beyond the rebuild point, the index must report that its answer is incomplete.

Durable Streams supports Kafka-style idempotent producers to prevent duplicate writes during client retries. This requires providing three headers together:

• Producer-Id: <string>: A stable, non-empty identifier for the logical producer (e.g., a UUID).
• Producer-Epoch: <integer>: A non-negative integer (≤ 2^53-1). Increment this on producer restart to fence out old (zombie) sessions.
• Producer-Seq: <integer>: A monotonically increasing sequence number per epoch, applied per-batch (per HTTP request).

Producer Lifecycle

1. Start: Send Producer-Epoch: 0 and Producer-Seq: 0.
2. Retry: If a request fails, retry with the same Producer-Id, Producer-Epoch, and the next Producer-Seq.
3. Restart: If the producer process restarts, increment the Producer-Epoch and reset Producer-Seq to 0.

Error Handling

• 403 Forbidden: Returned if the Producer-Epoch is stale (a newer epoch has already been established). The response includes the current server Producer-Epoch to help the client recover.
• 409 Conflict: Returned if a sequence gap is detected. The response includes Producer-Expected-Seq and Producer-Received-Seq.

POST {stream-url}
Producer-Id: order-service-1
Producer-Epoch: 1
Producer-Seq: 5

[body bytes]

Path affinity (extension token: path-affinity-v1) allows multiple independent streams to share a single Ursula Raft group. This is useful for ensuring related streams are co-located for performance or consistency.

URL Structure

The affinity form of a stream URL is: {base_url}/{bucket_id}/{affinity_key}/{stream_id}

Example:

• /workflow/run-42/journal
• /workflow/run-42/queue

Both streams above share the affinity_key run-42 and will route to the same Raft group. In contrast, a standard two-segment URL like /workflow/run-42-journal has no affinity key and will be hashed normally.

Constraints and Behavior

• affinity_key follows the same UTF-8 and path-safety constraints as stream_id.
• The complete {bucket_id}/{affinity_key}/{stream_id} identity MUST NOT exceed 122 bytes.
• Path affinity only affects physical placement/routing; it does not make separate HTTP operations atomic.
• Certain IDs are reserved under an affinity path: $transaction, append-batch, attrs, bootstrap, retention, and snapshot.
• Servers MUST advertise path-affinity-v1 in the Stream-Extensions header when responding to an affinity-form URL.