ra

Ra uses a multi-layered storage approach to support thousands of Ra servers per Erlang node while minimizing fsync(1) calls. The architecture consists of three main components:

1. Write-Ahead Log (WAL)

All write operations are first accepted by the WAL. It stores data in two parts:

• WAL File: A single file where entries from all Ra servers on the node are appended. It calls fsync(1) after a batch of writes or when the mailbox is empty to optimize throughput.
• In-memory ETS tables: The WAL maintains per-Ra-server ETS tables. These serve as the primary way for Ra servers to read entries once they have been replicated and reached consensus.

2. Segment Writer

To prevent the WAL file from growing indefinitely, the WAL periodically "rolls over" to a new file. The Segment Writer process:

• Takes the old WAL file and the associated ETS tables.
• Flushes the ETS tables to per-Ra-server-specific on-disk segments.
• Deletes the WAL file and ETS tables once flushing is complete.

3. Snapshot Writer

Ra servers offload snapshot persistence to the Snapshot Writer process. This prevents large snapshot writes from blocking the Ra server. Once a snapshot is written, the server can delete segments containing entries with an index lower than the snapshot.

Data Lookup Hierarchy

When a Ra server needs to look up an entry, it checks these three mechanisms in order:

1. Short-lived Cache: The fastest lookup; used for entries that may not be confirmed by the WAL yet.
2. ETS Tables: Used for confirmed entries.
3. Segment Files: Used for entries that have been flushed to disk by the Segment Writer.

Log compaction in Ra is a multi-phase process designed to manage disk space while preserving necessary historical data.

Minor Compaction (Phase 1)

Runs synchronously in the Ra server process immediately after each snapshot. It identifies and deletes entire segments that contain no live entries (based on the live_indexes/1 callback). This is highly efficient as it only involves range checks and file deletions.

Major Compaction (Phase 3)

Runs in the background ra_worker process. It groups adjacent segments that have a low density of live entries (less than 50% live entries OR less than 50% live data) and merges them into a single new segment. This reduces fragmentation and the total number of files.

Log Sections

• Normal log section: The contiguous log following the last snapshot.
• Compacting log section: Contains the live raft indexes that are lower than or equal to the last snapshot index.

Ra provides three distinct mechanisms for capturing persistent state, each serving a different purpose in the lifecycle of a Ra node:

1. Snapshots: The primary mechanism for log compaction. They capture the full state machine state up to a specific index, allowing Ra to safely delete log entries prior to that index. Snapshots are replicated to followers and are essential for recovery after crashes.
2. Checkpoints: Similar to snapshots but not replicated to followers. They capture state at a point in time without triggering log truncation. They can be promoted to full snapshots if necessary.
3. Recovery Checkpoints: A lightweight, local-only optimization used during ordered shutdowns. They are written synchronously to avoid expensive log replay during restart but are not replicated and do not store live indexes (which are always recovered from the last snapshot/checkpoint).

Compaction results are communicated via the #compaction_result{} record. When the Ra server processes a result using handle_compaction_result/2, it performs the following lifecycle steps:

1. Update Segment Refs: Removes unreferenced and linked segments from the segment references and adds the new compacted segments.
2. Cache Eviction: Evicts all open segments from the cache because the underlying files may have changed.
3. Cleanup: Schedules the background deletion of unreferenced files via the worker process.

The #compaction_result{} record structure:

• unreferenced: A list of filenames ([file:filename_all()]) representing segments to be deleted.
• linked: A list of filenames ([file:filename_all()]) representing segments that have been converted to symlinks.
• compacted: A list of new segment references ([segment_ref()]) created during compaction.

-record(compaction_result, {
    unreferenced = [] :: [file:filename_all()], %% segments to delete
    linked = [] :: [file:filename_all()],    %% segments now symlinks
    compacted = [] :: [segment_ref()]
}).

The WAL requires a contiguous sequence of log entries for each writer. If entries arrive out of order, the WAL detects a gap and enters an out_of_seq state.

Gap Detection Logic

A gap is detected if the incoming request fails this check: Expected: PrevIdx (from write request) <= LastIndex (WAL's record)

Resend Protocol

1. Detection: If a gap is found, the WAL marks the writer as {out_of_seq, LastIndex} and refuses further writes.
2. Notification: The WAL sends a {resend_write, MissingIndex} message to the Ra server.
3. Resolution: The Ra server resends the missing index. Once the gap is filled, the WAL returns to the {in_seq, LastIndex} state.

Resend Throttling

To prevent 'resend storms', Ra servers throttle resend attempts. A server will not attempt a resend within a configurable window (default: 20 seconds) unless:

• The window has elapsed.
• The WAL process has restarted (detected via a change in the WAL process PID).

The Write-Ahead Log (WAL) uses the ra_log_snapshot_state ETS table to avoid unnecessary I/O. By querying ra_log_snapshot_state:smallest/2, the segment writer determines the 'floor' for flushing entries. Any entries with an index lower than the SmallestLiveIndex are not written to segments.

ra_log_snapshot_state Table Structure: Stored as a 4-tuple: {UId, SnapshotIndex, SmallestLiveIndex, LiveIndexes}

• SnapshotIndex: The index of the last completed snapshot (-1 if none).
• SmallestLiveIndex: The minimum of (SnapshotIndex + 1) and the first live index.
• LiveIndexes: The ra_seq:state() of live indexes.

WAL Optimizations:

• Sparse Writes: Supports ra_log_wal:write/7 with an explicit PrevIndex to handle sparse sequences.
• Memory Tracking: Tracks sparse sequences in memory via ra_mt using ra_seq:state().
• Segment Writer Communication: The WAL sends the full ra_seq of written entries to the segment writer rather than just simple ranges.