DOCSEARCHbyAlgolia
Submit Docs

OpenRaft Documentation

repository·main·Indexed 24 days ago

https://github.com/databendlabs/openraft

An advanced, high-performance implementation of the Raft consensus protocol in Rust. Designed to be runtime-agnostic and fully pluggable, OpenRaft is suitable for distributed data storage systems such as SQL, NoSQL, and KV stores. The library includes support for Multi-Raft architectures, various storage implementations (including in-memory and RocksDB), and network implementations via HTTP for RaftNetwork V1 and V2 APIs.

Tokens
78.7K
Snippets
117
Records
432
Agent score
83%

What's inside OpenRaft

  1. Overview of OpenRaft Jepsen Tests

    main

    The jepsen/ directory contains black-box Jepsen tests for OpenRaft. Unlike deterministic simulation tests that run in a controlled Rust environment, these tests validate externally observable behavior by driving a running KV service (a RocksDB-backed application) through its external HTTP APIs.

    These tests exercise real client requests, process lifecycles, and network behavior to ensure linearizability and correctness under fault conditions like network partitions, process crashes, and membership changes.

  2. Overview of OpenRaft example applications

    main

    The examples/ directory provides various implementation approaches for OpenRaft components. These examples demonstrate how to implement the core interfaces required to build a Raft-based system, including storage, networking, and state machines.

    Key implementation areas covered include:

    • LogStore: For storing Raft logs.
    • StateMachine: For managing application state.
    • RaftNetwork: The transport protocol and client library.
    • Client/Server: HTTP or gRPC interfaces for application requests.

  3. Core Raft Functionality and API Triggers

    main

    Openraft provides several built-in capabilities for managing a Raft cluster. Many of these can be controlled via a Trigger object or specific configuration settings:

    Leader Management

    • Leader Election: Handled by policy or manually using Trigger::elect().
    • Leader Transfer: Move leadership to another node using Trigger::transfer_leader().

    Cluster Membership and Roles

    • Non-voter (Learner) Role: Add nodes that receive logs but do not participate in voting using add_learner().
    • Dynamic Membership: Supports arbitrary membership changes using the joint consensus approach. Note that Openraft does not support single-step configuration changes (changing only one node at a time); it uses the more general joint consensus method to ensure safety during transitions.

    Log and State Management

    • Log Compaction: Create snapshots of the state machine either by policy or manually via Trigger::snapshot().
    • Snapshot Replication: Automatically replicates snapshots to followers.
    • Log Purging: Remove logs by policy or manually using Trigger::purge_log().

    Reliability and Reads

    • Pre-vote: Enable Config::enable_pre_vote to avoid unnecessary term increments during elections.
    • Linearizable Reads: Ensure read consistency using ensure_linearizable().

    Monitoring and Runtime Control

    • Metrics: Access cluster health via Raft::metrics(), Raft::data_metrics(), and Raft::server_metrics().
    • Runtime Configuration: Toggle heartbeats or trigger elections using RuntimeConfigHandle::heartbeat() and RuntimeConfigHandle::elect() (or Trigger::elect()).

  4. Use openraft-memstore-custom-node-id for regression testing

    main

    The openraft-memstore-custom-node-id crate is a specialized, minimal in-memory Raft storage implementation designed to verify that the OpenRaft Suite test suite does not depend on the specific string format of a NodeId's Display implementation.

    It is used as a regression guard to ensure that tests do not accidentally hardcode numeric string representations of IDs. While the standard memstore uses u64 (where Display is just the number), this crate uses a custom newtype where Display produces a formatted string like Node[0], Node[1], etc.

    Use this crate if you are developing storage implementations or testing the Suite and want to ensure your logic is resilient to non-integer NodeId string representations.

  5. Use the dir-transfer protocol for immutable file transfers

    main

    The dir-transfer crate provides a file-level protocol for transferring a flat directory of immutable files to a remote peer as an ordered stream of frames. It is designed to be transport-agnostic; the protocol defines the frame types and the logic for sending and receiving, while the carrier (e.g., TCP, gRPC streaming, HTTP body, or a message queue) is responsible for moving the opaque frames reliably and in order.

    This is particularly useful for shipping RocksDB checkpoint directories as OpenRaft snapshots, but it works for any checkpoint-style directory of immutable files.

  6. Use shared KV request and response types in openraft examples

    main

    The types-kv crate provides standardized request and response types used for Key-Value (KV) store operations in openraft example implementations. These types allow for consistent handling of write operations and their results across different example crates.

    Request Types

    Request handles KV store write operations:

    • Set: A standard write operation.
    • CompareAndSet: A version-based write operation used for optimistic concurrency control.

    Response Types

    Response encapsulates the result of a KV store operation, which may include an optional versioned value.

  7. Use openraft-memstore for testing or demonstrations

    main
    The openraft-memstore crate provides in-memory implementations of RaftLogStorage and RaftStateMachine. It is designed primarily for testing purposes or for demonstrating how OpenRaft works without the overhead of persistent storage or complex networking. It is not intended for production use where data durability is required.

  8. What is Leader Lease and how does it work?

    main

    Leader lease is a mechanism used to maintain consistency and coordination by defining a time period during which nodes believe no other Leader exists in the cluster. It prevents unnecessary leader elections and ensures the current Leader can operate with confidence. The behavior of the lease differs significantly between Follower and Leader nodes:

    For Follower Nodes

    • Election Suppression: During a valid Leader lease period, Follower nodes will not initiate a new Leader election.
    • Lease Refresh: A Follower refreshes its Leader lease upon receiving an AppendEntries request from the Leader.
    • Note: Receiving a RequestVote request does not refresh the lease, as RequestVote does not indicate a Leader has been established.

    For Leader Nodes

    • Confidence: During the valid lease period, the Leader is confident that no other Leaders exist in the cluster.
    • Lease Refresh: The Leader refreshes its own lease only after receiving acknowledgments for an AppendEntries request from a quorum (majority) of Follower nodes.
    • Timing and Latency: To account for network latency and clock differences, the Leader's lease starting time is calculated from the moment the AppendEntries request is sent, not when the responses are received. This ensures the Leader's lease duration aligns with the period Followers consider valid.
    • Initial State: A newly elected Leader starts with a lease of 0 until it successfully sends its first AppendEntries requests and receives quorum acknowledgments.

  9. What is a ReplicationSessionId and how does it ensure consistency?

    main

    In OpenRaft, a ReplicationSessionId is a unique identifier used to distinguish individual replication sessions. It is critical for guaranteeing that replication states (e.g., progress updates from a Leader to target nodes) are correctly handled during leader changes or membership modifications.

    A replication session is initiated when a Leader starts replicating log entries to a set of target nodes. The ReplicationSessionId ensures that updates from an old session (e.g., from a previous Leader or a previous cluster membership configuration) are not mistakenly applied to a new session. This prevents the Raft core from incorrectly believing a node has received or committed logs that it actually hasn't.

  10. Determine if state machine persistence is required

    main

    In OpenRaft, you do not need to persist the state machine to disk separately. Instead, you should rely on the snapshotting mechanism. On startup, the state machine should be rebuilt by loading the latest snapshot.

    Whether you need to replay logs after loading a snapshot depends on your implementation of RaftLogStorage::save_committed():

    1. If save_committed() is implemented: You must re-apply logs starting from the snapshot's last included log up to the saved committed log ID during the startup sequence.
    2. If save_committed() is NOT implemented: No log replay is required; the snapshot is considered to represent the complete committed state.

  11. Understand Openraft's Vote and Leader ID mechanisms

    main

    Openraft departs from standard Raft in how it handles elections and leadership identification:

    • Unified Vote: Instead of tracking currentTerm and votedFor separately, Openraft uses a single partially-ordered Vote value. A node grants a request if the incoming Vote is greater than or equal to the last Vote it has seen.
    • Advanced-mode leader-id: By default, the leader ID is a (term, node_id) tuple. This allows multiple candidates to be granted leadership in the same term (the last one wins), which minimizes election conflicts. This approach embeds the node_id into every LogId.
    • Standard-mode leader-id: If minimizing the size of LogId is a priority, a standard-mode leader-id is available via the leader_id configuration.

  12. Understand the Network Layers in the V1 Example

    main

    The network traffic in this example is split into two distinct layers:

    1. Raft RPC Traffic: Handled by network-v1-http.

      • NetworkFactory: Creates outbound Raft RPC clients (adapted from V1 to V2).
      • Server: Receives inbound /append, /vote, and /snapshot requests.

    2. Application Traffic: Handled by app-http.

      • The application uses add_openraft_routes() to register common OpenRaft application endpoints.
      • Custom application-specific read endpoints are added separately.