HotStuff BFT Consensus Framework

The hotstuff project uses a modular architecture to allow different implementations of consensus protocols, view synchronizers (pacemakers), and other components to interoperate. To handle the complexities of modularity, the system employs two key patterns:

1. Deferred Initialization: To solve circular dependencies (e.g., a consensus module needing a synchronizer that in turn needs the consensus module), modules are first constructed in an uninitialized state and then fully initialized later.
2. Dependency Injection via Module Registry: A modules.Builder collects module instances, and a modules.Core object is used to retrieve them during the initialization phase.

The Module Lifecycle

1. Registration: Modules register themselves with a global registry using modules.RegisterModule.
2. Construction: Modules are instantiated (e.g., via modules.New(name)).
3. Composition: Module instances are added to a modules.Builder.
4. Building: Calling builder.Build() triggers the InitModule method on all modules that implement the Module interface.
5. Retrieval: During InitModule, a module uses the provided *modules.Core to fetch its dependencies using Get, GetAll, or TryGet.

// 1. Register the module
func init() {
  modules.RegisterModule("A1", NewA1)
}

// 2. Implement the Module interface for deferred initialization
type A1 struct{ b B }

func (a *A1) InitModule(mods *modules.Core) {
  mods.Get(&a.b) // Fetch dependency B
}

// 3. Compose and Build
func compose(choiceA string) {
  builder := modules.NewBuilder()
  builder.Add(
    NewB1(),
    modules.New(choiceA),
  )
  mods := builder.Build()

  var (
    a A
    b B
  )
  mods.GetAll(&a, &b) // Retrieve the final composed modules
}

To add a new metric to the hotstuff experimentation framework, follow these four steps:

1. Define a Protobuf message: Create a message in a .proto file (e.g., metrics/types.proto). The message must include a field named Event of type types.Event to ensure compatibility with the plotting package.
2. Emit events from modules: If your metric depends on specific system events, define that event type in the relevant module's package and add it to the event loop using eventLoop.AddEvent(...).
3. Implement the modules.Module interface: Create a struct that implements InitModule. In this function, use mods.Get to retrieve the eventloop.EventLoop and logging.Logger. Register handlers for your custom events and an observer for types.TickEvent to periodically log measurements.
4. Register the metric: In your metric's package, use an init() function to call RegisterReplicaMetric or RegisterClientMetric. This makes the metric available via the --metrics CLI flag.

message LatencyMeasurement {
  Event Event = 1;
  double Latency = 2;
  double Variance = 3;
  uint64 Count = 4;
}

// Example InitModule implementation
func (t *Throughput) InitModule(mods *modules.Core) {
    var (
        eventLoop *eventloop.EventLoop
        logger logging.Logger
    )
    mods.Get(&eventLoop, &logger)

    eventLoop.RegisterHandler(consensus.CommitEvent{}, func(event interface{}) {
        commitEvent := event.(consensus.CommitEvent)
        t.recordCommit(commitEvent.Commands)
    })

    eventLoop.RegisterObserver(types.TickEvent{}, func(event interface{}) {
        t.tick(event.(types.TickEvent))
    })

    logger.Info("Throughput metric enabled")
}

// Example Registration
func init() {
    RegisterReplicaMetric("throughput", func() interface{} {
        return &Throughput{}
    })
}

Use the twins generate subcommand to create JSON scenario files for testing Byzantine consensus protocols. Scenarios can be generated ahead of time and written to files, or generated on the fly during execution.

Configuration Flags

• --scenarios: Total number of scenarios to generate. If omitted, all possible scenarios are generated (can result in very large files).
• --scenarios-per-file: Number of scenarios per output file. If set to 0 (default), --output is the filename. If set to a non-zero value, --output is the directory where multiple files will be created.
• --output: The file path or directory where scenarios are written.
• --replicas: Number of replicas to create.
• --twins: Number of replicas that should have a twin.
• --partitions: Number of network partitions to divide replicas and twins into.
• --views: Number of views to run.
• --shuffle: Randomizes the order of scenarios.
• --seed: Used to reproduce a specific randomized order.

Note: The generator is deterministic; given the same inputs, it produces the same scenarios unless --shuffle is used.

# Generate 100 scenarios, 10 per file, into the 'results' directory
./hotstuff twins generate --scenarios 100 --scenarios-per-file 10 --output ./results --replicas 5 --twins 2 --partitions 3 --views 10

The CLI can deploy and run experiments on remote hosts using SSH. To use this feature, ensure the following requirements are met:

1. Authentication: You must use public key authentication.
2. Host Verification: All target hosts must be present in a known_hosts file.
3. Environment: Hosts must support SFTP and provide a Unix-like environment.

SSH Configuration

You must provide an SSH config file to the CLI. The CLI uses the ssh_config package, which supports key options like User, Port, IdentityFile, and UserKnownHostsFile.

Remote Experiment Flags

In addition to standard experiment flags, use these when targeting remote hosts:

• --exe: Path to the executable to deploy. Use this if the remote host has a different OS or architecture than your local machine. If omitted, the CLI deploys the current binary.
• --ssh-config: Path to your SSH config file. Defaults to ~/.ssh/config if not specified.
• --hosts: A comma-separated list of host names (preferably those defined in your SSH config).
• --worker: Runs a worker on your local machine in addition to the remote hosts.

# Example SSH config
Host 127.0.0.1 localhost
    User root
    Port 2020
    IdentityFile ./scripts/id
    UserKnownHostsFile ./scripts/known_hosts

Use the twins run subcommand to execute scenarios against a consensus implementation. You can either provide a pre-generated JSON file or configure the generator to create scenarios on the fly.

Configuration Flags

• --input: Path to a JSON file containing pre-generated scenarios. If this is not provided, the command uses the generator flags (see Generating Scenarios).
• --consensus: Specifies the consensus implementation to test.
• --output: Where to write results.
• --scenarios-per-file: Controls the grouping of output files.
• --log-all: By default, the executor only writes failed scenarios. Use this flag to log all scenarios.
• --concurrency: Number of concurrent executions. Set to 0 to use all available CPUs for faster execution.

# Run scenarios from a file using all available CPUs
./hotstuff twins run --input ./scenarios.json --consensus my-consensus-impl --concurrency 0

When running remote experiments, you can use a configuration file (passed via the --config flag) to manage how hosts communicate and how roles are distributed.

Internal Addresses

If the controller connects to remote hosts via a global address but the hosts should communicate with each other using local network addresses, specify an internal-address for each host in the [[hosts-config]] section.

Manual Client and Replica Assignment

By default, the controller distributes clients and replicas evenly across all workers. To override this, you can manually specify the number of clients and replicas for specific hosts using [[hosts-config]].

Note: If you manually configure hosts, the sum of all configured clients and replicas must equal the total number requested for the experiment.

# Global settings
clients = 2
replicas = 8

hosts = [
    "hotstuff-worker-1",
    "hotstuff-worker-2",
    "hotstuff-worker-3",
    "hotstuff-worker-4",
]

# Configure specific host behavior
[[hosts-config]]
name = "hotstuff-worker-1"
internal-address = "192.168.10.2"
clients = 2
replicas = 0