DOCSEARCHbyAlgolia
Submit Docs

Dragonboat Documentation

repository·master·Indexed 26 days ago

https://github.com/lni/dragonboat

A high-performance, multi-group Raft consensus library implemented in pure Go. It provides fault-tolerant, strongly consistent distributed systems supporting scalable multi-group architectures, both disk-based and memory-based state machines, and a specialized database called Tan for storing Raft logs and metadata. Features include leader election, log replication, snapshotting, and Prometheus-based health metrics.

Tokens
14K
Snippets
13
Records
103
Agent score
90%

What's inside Dragonboat

  1. Overview of Dragonboat features

    master

    Dragonboat is a high-performance, multi-group Raft consensus library written in pure Go. It provides fault-tolerance by allowing a system to continue operating as long as a majority of members are available.

    Key features include:

    • Multi-group Raft: Scalable implementation supporting many concurrent Raft groups.
    • State Machine Support: Supports both disk-based and memory-based state machines.
    • Protocol Support: Implements leader election, log replication, snapshotting, log compaction, membership changes, pre-vote, ReadIndex for linearizable reads, leadership transfer, non-voting members, and witness members.
    • High Performance: Fully pipelined with TLS mutual authentication support, optimized for high-latency environments.
    • Extensibility: Custom Raft log storage and transport support.
    • Observability: Prometheus-based health metrics.
    • Reliability: Extensively tested with Jepsen's Knossos linearizability checker.

  2. Overview of Tan database

    master
    Tan is a high-performance database specifically designed for storing Raft logs and metadata. It was developed to address the inefficiencies of using general-purpose LSM-based Key-Value stores (like RocksDB) in Raft implementations. Tan aims to reduce redundant MemTables, keys, and serializations, while minimizing storage, write, and read amplification and reducing the overhead of concurrent access control.

  3. Understand Dragonboat core concepts

    master

    Dragonboat uses a replicated state machine model to provide high availability and strong consistency. Key concepts include:

    • Raft Group (Shard): An independent entity controlled by the Raft protocol consisting of multiple replicas. Each group is identified by a globally unique 64-bit integer ShardID.
    • Node (Replica): A member of a Raft group, identified by a unique 64-bit integer ReplicaID within that group.
    • Leader: The node responsible for coordinating reads and writes for a Raft group. A group can only process requests once a Leader is established.
    • Proposal: A user-submitted update to the state machine. Once a majority of replicas receive and persist a proposal, it is considered committed.
    • Raft Log: An ordered data structure where committed proposals are recorded with a unique index.
    • Snapshot: A point-in-time capture of the state machine's state used for fast recovery.

    Dragonboat ensures that all replicas apply committed proposals in the same order (by index), resulting in identical state machine states across the group.

  4. Storage engine compatibility and defaults

    master

    Dragonboat uses Pebble as its default storage engine for Raft log data.

    • Pebble: The current default. It is a Go-based Key-Value store that is compatible with RocksDB data file formats.
    • RocksDB: Support for RocksDB was removed in version v3.4. Users should migrate to Pebble.
    • Tan: A next-generation Raft log storage implementation. Future versions of Dragonboat will use Tan by default, while Pebble will remain supported for long-term maintenance.

  6. Manage state machine snapshots

    master

    Snapshots save the state of a node at a specific point in time, including the applied proposal index, membership info, active sessions, and the state machine state.

    Creating Snapshots:

    • Automatic: Set the SnapshotEntries field in the Config object to trigger a snapshot every $N$ updates.
    • Manual: Call RequestSnapshot on a NodeHost for a specific Raft group.

    Snapshot Options:

    • DefaultSnapshotOption: A system-managed snapshot.
    • Exported: true: The snapshot is exported to the directory specified in ExportPath. Users are responsible for managing, backing up, and cleaning up these exported files.
    • OverrideCompactionOverhead and CompactionOverhead: Control how many logs are cleaned up after a snapshot is created.

    State Machine Implementation: Your state machine must implement:

    • SaveSnapshot(io.Writer): To save the state.
    • RecoverFromSnapshot(io.Reader): To restore the state.

    Recovery: If a Raft group loses a majority of its nodes permanently, you can use the ImportSnapshot function from the tools package to attempt a recovery (note: this is a lossy operation).

  8. Use Non-Voting nodes as observers or for scaling

    master

    Dragonboat supports Non-Voting nodes (Observers) through NodeHost. These nodes do not participate in Leader elections or the proposal commitment process. They only execute committed proposals.

    Key use cases:

    • Read-only replicas: Since they maintain a complete and identical state machine, they can serve as additional nodes for consistent reads.
    • Safe scaling: New nodes can join a Raft group as observers to catch up on state machine data before being promoted to full voting members.

    How to check if an observer is ready for promotion: An observer is ready to be upgraded to a normal node once it has acquired the necessary Log Entries. You can verify this by performing a SyncRead or GetShardMembership on the NodeHost where the observer resides. A successful return indicates the ReadIndex protocol has completed a round, confirming the observer is synchronized.

  9. Build and use the checkdisk tool

    master

    The checkdisk tool is a utility used to compare the relative performance of different SSD types. It works by creating 48 Raft groups (each with a single node) and using 10,000 client goroutines to continuously make proposals with 16-byte payloads. It reports the average number of completed proposals per second over a 60-second period.

    To use the tool:

    1. Build the executable using Go.
    2. Copy the generated checkdisk binary to the disk you wish to test.
    3. Run the executable. The process takes approximately 1 minute.
    go build github.com/lni/dragonboat/v4/tools/checkdisk

  10. Use custom storage engines in Dragonboat

    master
    You can extend Dragonboat to use a custom storage engine for Raft log data. To do this, you must implement the ILogDB interface defined in github.com/lni/dragonboat/v4/raftio. Once implemented, provide your implementation via a factory function to the LogDBFactory field of the NodeHostConfig struct.

  11. Disaster Recovery via Snapshots

    master

    In extreme cases where a majority of nodes fail permanently and the Raft group becomes unavailable, you must perform a manual recovery:

    • Prerequisite: Regularly use the ExportSnapshot method of NodeHost to export and back up snapshots for disaster recovery.
    • Recovery Tool: Use the ImportSnapshot tool provided in the github.com/lni/dragonboat/tools package to repair the damaged Raft group.

  12. Start a Dragonboat node

    master

    Nodes are managed and loaded by a NodeHost. Use the following methods to start replicas:

    • StartReplica
    • StartConcurrentReplica
    • StartOnDiskReplica

    Startup Rules:

    1. Initial Members: When starting the very first members of a Raft shard, you must provide the complete list of initial members. All replicas in the initial set must start with the exact same information.
    2. Joining a Shard: If a node is not part of the initial set but is being added later via membership changes (e.g., SyncRequestAddReplica), set the join parameter to true during its first startup.
    3. Restarting: When restarting an existing node (whether it was an initial member or added later), do not provide initial member information and do not set join to true.