DOCSEARCHbyAlgolia
Submit Docs

hashicorp/raft

repository·main·Indexed 27 days ago

https://github.com/hashicorp/raft

A Go library that provides consensus and manages replicated logs to build consistent and partition-tolerant distributed systems using replicated state machines. It includes support for leader and follower operations, FSM management, and a Fuzzy Raft framework for testing protocol correctness under various network and timing conditions.

Tokens
17.6K
Snippets
30
Records
129
Agent score
93%

What's inside hashicorp-raft

  1. Overview of HashiCorp Raft

    main
    raft is a Go library designed to manage a replicated log and provide consensus for distributed systems. It can be used in conjunction with a Finite State Machine (FSM) to manage replicated state machines, enabling the creation of Consistent and Partition Tolerant (CP) systems.

  2. Overview of Fuzzy Raft

    main
    Fuzzy Raft is a framework and a collection of test scenarios designed to test the behavior and correctness of the raft library under various network and timing conditions. It is inspired by fuzzing techniques to uncover edge cases in the Raft protocol implementation.

  3. Understand Raft Threading and FSM Access

    main

    The library uses specific threads to manage concurrency and state:

    FSM Access (runFSM)

    Crucial: The runFSM thread has exclusive access to the FSM. All reads and writes must be sent as messages to this thread. It handles:

    • Applying logs to the FSM.
    • Restoring snapshots to the FSM.
    • Capturing snapshots.

    Snapshotting (runSnapshot)

    Handles the heavy lifting of persisting snapshots via FSMSnapshot.Persist and compacting logs. It is triggered by:

    • SnapshotInterval (periodic).
    • User-initiated snapshots.

    Main Execution (run)

    • Follower: Processes RPCs (AppendEntries, RequestVote, InstallSnapshot, TimeoutNow), handles live bootstrap, and manages periodic heartbeats.
    • Candidate: Starts elections and processes RPCs.
    • Leader: Manages replication, applies Noop logs to commit up to the current index, handles leadership transfers, commits logs, verifies leadership, and dispatches client Apply requests.

  4. Understand Raft Terminology

    main

    To use the hashicorp/raft library, familiarize yourself with these core concepts:

    • Cluster: The set of peers in the Raft configuration.
    • Peer: A node participating in consensus. States include follower, candidate, or leader.
    • Log & Log Entry: The full set of entries used to order operations. Each entry has an index.
      • Committed: An entry is committed once the leader has replicated it to a majority of peers.
      • Applied: An entry has been applied to the Finite State Machine (FSM).
    • Term: Monotonically increasing integers representing periods of time. Each term begins with an election.
    • FSM (Finite State Machine): The component that stores the actual cluster state.
    • Client: The application using the hashicorp/raft library.

  5. Use the Fuzzy Raft framework for custom testing

    main

    The Fuzzy Raft framework provides tools to construct multi-node Raft clusters connected by an instrumented transport. This transport allows you to inject behaviors to simulate real-world network scenarios, such as:

    • Simulating network partitions by failing all transport calls to a specific node.
    • Injecting delays in sending or receiving messages.

    The framework includes helper classes to:

    • Apply known sequences of test data.
    • Examine the final state of the cluster.
    • Inspect individual node FSMs (Finite State Machines).
    • Inspect the Raft log.

  6. Migrate from `armon/go-metrics` to `hashicorp/go-metrics`

    main

    To migrate your application to the newer metrics implementation, follow these steps:

    1. Upgrade libraries that currently use armon/go-metrics to consume hashicorp/go-metrics/compat by updating their import statements.
    2. Update your application's dependencies to versions that use the compatibility layer.
    3. Update your application code:
      • Replace all imports of github.com/armon/go-metrics with github.com/hashicorp/go-metrics.
      • Configure your build system to include the hashicorpmetrics build tag.

  8. Use Leadership Transfer

    main

    Leadership transfer allows a leader to proactively hand off leadership to a follower, avoiding the wait for an election timeout during operations like restarts or upgrades.

    Important Notes:

    • This feature is not enabled by default.
    • It must be explicitly triggered at the application level.
    • All Raft members in the cluster should support leadership transfers before attempting one.
    • To ensure the transfer succeeds despite the library's default behavior of rejecting votes when a leader exists, the LeadershipTransfer flag must be set on the RequestVoteRequest.

  9. Run Fuzzy Raft tests

    main

    Fuzzy Raft tests use the standard Go testing framework. Because these tests target timing-sensitive edge cases, a single successful run is insufficient; tests should be run repeatedly to build confidence in the implementation's correctness.

    You can run the tests from the fuzzy directory using go test . or from the parent directory using make fuzz.

  10. Understand Raft snapshotting and log compaction

    main

    Raft manages snapshots to prevent logs from growing indefinitely.

    Snapshot Triggering Snapshots are triggered based on the SnapshotThreshold configured in Raft. When the delta between the last log index and the last snapshot index exceeds this threshold, a snapshot is taken.

    Log Compaction After a successful snapshot, Raft compacts the logs. It removes logs up to the snapshot index, while respecting the TrailingLogs configuration. TrailingLogs ensures that a specific number of entries remain after the snapshot to assist with potential recovery or edge cases.

    Constraints Raft will not take a snapshot if there is an outstanding configuration change that has not yet been applied to the FSM, as the snapshot must be able to represent the committed configuration state accurately.

  11. Configure Pre-Vote extension

    main
    HashiCorp Raft implements the Pre-Vote extension by default. This optimization allows a candidate to verify if its log index is up-to-date enough to win an election before incrementing its term and triggering a cluster-wide election. If you need to disable this behavior, use the PreVoteDisabled flag in the Config struct.

  12. Configure metrics emission via build tags

    main

    The library supports emitting metrics through two different libraries. You control which one is used via Go build tags. If no tag is specified, armon/go-metrics is used by default.

    Build TagMetrics Library
    armonmetricsgithub.com/armon/go-metrics
    hashicorpmetricsgithub.com/hashicorp/go-metrics

    Note: Emitting to armon/go-metrics is officially deprecated. It is recommended to use the hashicorpmetrics tag to opt into hashicorp/go-metrics.