NuRaft Documentation

NuRaft is a C++ Raft implementation derived from the cornerstone project. It provides a lightweight implementation with minimum dependencies, enhanced with several real-world features required for production use cases.

Core Raft Capabilities

• Log replication & compaction
• Leader election
• Snapshotting
• Dynamic membership & configuration changes
• Group commit & pipelined write
• User-defined log store & state machine support

Advanced Features

• Pre-vote protocol: Prevents disruptive elections.
• Leadership expiration: Handles leader timeouts.
• Priority-based semi-deterministic leader election: Influences which nodes become leaders.
• Read-only member (learner): Allows nodes to join the cluster without participating in quorums.
• Object-based logical snapshot: For efficient snapshot transmission.
• Custom quorum size: Separate quorum sizes for commit and leader election.
• Asynchronous replication: Improves performance via async operations.
• SSL/TLS support: Secures communication.
• Parallel Log Appending: Optimizes log write performance.
• Custom Commit Policy: Allows fine-grained control over when entries are committed.
• Streaming Mode: Optimized for high-throughput streaming.

When using asynchronous replication mode, the standard lifecycle of state machine operations changes. You must implement the following logic:

1. Execution: Move the actual execution logic from state_machine::commit() to state_machine::pre_commit(). This ensures the result is available immediately when append_entries() returns.
2. Conflict Resolution: You must correctly implement state_machine::rollback() to revert any changes made during pre_commit() if a conflict is detected during background replication.

Comparison of Modes:

The Pre-Vote protocol is a mechanism used to prevent unnecessary leader elections and term disruptions caused by network partitions.

In standard Raft, a node that loses contact with the leader will increment its term and initiate an election. If this node is partitioned but can still reach a quorum, it can force the current leader to step down by increasing the global term, even if the leader is still healthy and serving the rest of the cluster. This causes continuous disruption.

How Pre-Vote works: Before a node increments its term and starts a formal election, it first sends a pre-vote request to other nodes.

• If voters have recently received heartbeats from the current leader, they will reject the pre-vote request. The initiator will not increment its term, preventing disruption.
• If voters' election timers have expired (indicating the leader is likely dead), they will accept the pre-vote request. Once the initiator receives a majority of acceptances, it proceeds to increment its term and initiate a formal request_vote phase.

Trade-off: When a leader actually fails, the Pre-Vote protocol can slightly increase the time required to elect a new leader, as a majority of servers must first experience an election timeout before the pre-vote phase succeeds.

Initiator   Voter(s)
|           |
X           |   raft_server::handle_election_timeout()
X           |   raft_server::request_prevote()
X---------->|   Send pre-vote request
|           X   raft_server::handle_prevote_req()
|<----------X   Send response
X           |   raft_server::handle_prevote_resp()
X           |   raft_server::initiate_vote()
X           |   raft_server::request_vote()
X---------->|   Send vote request
|           X   raft_server::handle_vote_req()
|<----------X   Send response
X           |   raft_server::handle_vote_resp()
X           |   raft_server::become_leader()
|           |

The buffer class represents a raw memory blob. It reserves a small amount of metadata at the beginning (4 bytes if size < 32KB, 8 bytes otherwise). User data must be written starting from the user section, not the absolute start of the memory blob.

Key operations:

• Allocation: Use buffer::alloc(size) to create a new buffer.
• Accessing User Data: Use data() to get the pointer to the start of the user data section, or data_begin() to get the start of the user section regardless of the current cursor position.
• Position Management: The buffer maintains an internal cursor. Use pos() to get or set the current position.

// Allocate memory
ptr<buffer> b = buffer::alloc( size_to_allocate );

// Get pointer to user data section
void* ptr = (void*)b->data();

// Get/Set internal cursor position
size_t current_position = b->pos();
b->pos( new_position );

// Get start of user section regardless of current position
void* ptr_begin = (void*)b->data_begin();

When parallel_log_appending_ is enabled, the leader's behavior changes regarding how it handles the quorum and disk writes:

• Commit Logic: The leader commits the log as soon as a quorum of servers has received it. Because replication and disk writing happen in parallel, the leader might reach a quorum before its own local disk write is complete.
• Safety: The protocol remains safe because at least a majority of servers have the log at the moment it is committed, even if the leader itself is not yet part of that quorum due to a pending disk write.
• Execution Scenarios:
  • If the disk write completes before replication: Behavior is identical to the standard sequential protocol.
  • If replication completes before the disk write: The leader can apply the log to the state machine immediately after the quorum is reached, potentially before the local disk write finishes.

Note: This optimization applies only to the leader. Followers will always wait for the notify_log_append_completion call before responding to the leader.

NuRaft operations are distributed across two main thread categories. Developers must ensure that APIs called by the User/Asio group are lightweight to avoid blocking the Raft engine.

1. User and Asio Thread Pool

These threads execute active Raft operations. Operations called here must be fast and non-blocking. This group invokes:

• log_store operations.
• state_machine::pre_commit() and state_machine::rollback().
• Snapshot chunk I/O via state_machine::read_logical_snp_obj and state_machine::save_logical_snp_obj.

2. Background Commit Thread

This thread runs continuously to handle long-running or periodic tasks. This group invokes:

• log_store operations.
• state_machine::commit().
• Snapshot creation via state_machine::create_snapshot.
• Log compaction via log_store::compact.

Thread Safety Requirement

Because log_store operations can be called by different threads (User/Asio and Background Commit) in parallel, your implementation of the log store must be thread-safe.

NuRaft is composed of five primary modules. To use the library, you must implement three of these modules (Log Store, State Machine, and State Manager) while leveraging the provided Raft Server and Asio layer.

Provided by NuRaft

• Raft server: Coordinates all incoming requests and responses from users and other nodes.
• Asio layer: Handles network communication, timers, and thread pool management.

User-Implemented Modules

• Log store: Manages read, write, and compact operations of Raft logs.
  • Interface: libnuraft/log_store.hxx
• State machine: Executes commits (including optional pre-commit and rollback) and manages snapshots.
  • Interface: libnuraft/state_machine.hxx
• State manager: Responsible for saving and loading cluster configuration and status.
  • Interface: libnuraft/state_mgr.hxx

Optional Modules

• Debugging logger: Used for system logging.
  • Interface: libnuraft/logger.hxx