RocksDbSharp

The cluster moves through three distinct phases:

1. Bootstrap (initialization)

• Starts with one designated primary (RaftConfig.BootstrapPrimary = true).
• The primary is forced into the Leader role immediately without an election.
• Other nodes pull a full checkpoint via SyncInitialStateAsync and then follow the WAL stream.
• No elections are allowed during this phase to prevent split-brain before data is replicated.

2. Promotion to Cluster mode

• The leader monitors followers using ReportLastSyncSequenceNumber.
• When a sufficient number of followers (defined by BootstrapPromotionQuorum) are within the BootstrapInSyncLagThreshold of the leader's latestSeq, the leader promotes itself to Cluster mode.
• This mode change is broadcast via heartbeats (HeartbeatPayload.LeaderMode).
• Once a node enters Cluster mode, it cannot return to Bootstrap mode.

3. Steady-state Cluster mode

• Full Raft protocol is enforced.
• Leader: Sends heartbeats every HeartbeatIntervalMs (default 250-300ms).
• Follower: Resets its election timer on every heartbeat. If the randomized timer (default [1500, 3000] ms) expires, the follower becomes a Candidate.

RocksDbSharp clustering follows a specific lifecycle to transition from a standalone node to a replicated cluster. The end-to-end flow involves:

1. Bootstrap (initialization): A primary node is initialized.
2. Promotion to Cluster mode: The primary node synchronizes with a follower via SyncUpdatesAsync (streaming the WAL). The follower uses ReportLastSyncSequenceNumber to advance the matchSeq and update the bootSync count. Once criteria are met, the node enters ModeChanged: Cluster.
3. Steady-state Cluster mode: The leader maintains authority via periodic HeartbeatAsync calls, which communicate the current term, leaderCommit, and leaderMode to followers.
4. Leader Failure & Election: If the leader fails, followers' election timers fire, triggering RequestVoteAsync to peers. Upon reaching a quorum, a node executes BecomeLeader and starts a new heartbeat with an incremented term++.

+--------+          snapshot           +----------+
| primary| ----------------------------> follower |
| (boot) |
+--------+                               +----------+
     |                                       |
     |  SyncUpdatesAsync (WAL stream)        |
     |-------------------------------------->|
     |                                       |
     |    ReportLastSyncSequenceNumber       |
     |<--------------------------------------|
     |  (matchSeq advances, bootSync count  |
     |  updates, promotion check fires)     |
     |                                       |
     +--- ModeChanged: Cluster --------------+
     |                                       |
     |   periodic HeartbeatAsync ------>     |
     |   (term, leaderCommit, leaderMode)   |
     |                                       |
     +-- on leader failure -----------------+
                                       election timer fires
                                              |
                                       RequestVoteAsync to peers
                                              |
                                       quorum reached -> BecomeLeader
                                              |
                                       new heartbeat: term++

When a follower's election timer expires, it initiates an election following these steps:

1. Transition: The node becomes a Candidate, increments currentTerm, votes for itself, and resets its election timer.
2. Request Votes: It sends RequestVote(term, candidateId, lastLogSeq, lastLogTerm) to all peers.
3. Voting Criteria: A peer grants a vote only if:
   • The request's term >= currentTerm.
   • The node is in Cluster mode (not Bootstrap).
   • The node hasn't already voted for someone else in this term.
   • The candidate's log is at least as up-to-date as the receiver's (lexicographic comparison of lastLogTerm and lastLogSeq).
4. Winning: If the candidate receives a Quorum of votes:
   • It calls BecomeLeader().
   • It records a new term-range row.
   • It resets followers' nextSeq to latestSeq + 1 and matchSeq to 0.
   • It writes a no-op entry (RaftClusterNode.LeaderNoOpKey) to allow the commit index to advance.
   • It starts the heartbeat loop.

Persistence: To ensure safety during crashes, currentTerm, votedFor, and the new term-range row are all fsync'd (using atomic temp-file renames) before any network responses are sent.

RocksDbSharp handles log divergence by rebuilding the entire replica from a leader snapshot rather than truncating the RocksDB WAL (which is not possible). Divergence is detected through three layers:

1. Attach-time consistency check: Before starting/restarting a WAL stream, a follower calls CheckLogConsistencyAsync(latestSeq, termAtLatest) on the leader. If the leader's (seq, term) pair doesn't match, a resync is triggered.
2. Per-batch sequence verification: Followers require every incoming batch to have a sequence number exactly equal to localLatestSeq + 1. Gaps or overlaps trigger a resync.
3. Heartbeat prev-term check: HandleAppendEntries compares prevLogTerm against the local term map; a mismatch triggers RaftClusterNode.ResyncRequired.

The Resync Process (ClusterNodeHost.PerformResyncAsync):

• The node enters a BeginResync() state: it pauses election timers, refuses votes, and answers heartbeats with "resyncing".
• The local DB is disposed.
• A per-file delta transfer occurs: The node and leader exchange identity signatures for .sst and .blob files using GetFileHashesAsync. The signature hashes the file length and the last 1 MiB (containing the RocksDB footer/metadata) to ensure byte-identity.
• The leader streams only new or changed files. Verified files are reused.
• The node deletes its journal/WAL directory and all unconfirmed local files, keeping only the raft/ state directory.
• CURRENT is transferred last to ensure a crash mid-restore results in a clean re-pull.
• Once CompleteResync(newDb) is called, the database is swapped in and the election timer is re-armed.