DOCSEARCHbyAlgolia
Submit Docs

BlockBench Documentation

repository·master·Indexed 18 days ago

https://github.com/ooibc88/blockbench

A benchmarking framework for analyzing private blockchain systems, providing macro and micro workloads to evaluate performance and architectural layers. It includes support and configuration guides for Ethereum, Hyperledger Fabric v1.4, Hyperledger Fabric v2.2, and general Hyperledger environments, including network initialization, miner/client management, and the use of NodeJS-based helper services for Fabric v2.2.

Tokens
23.5K
Snippets
96
Records
137
Agent score
61%

What's inside BlockBench

  1. Overview of the IOHeavy Micro Benchmark Workload

    master

    The IOHeavy workload is a micro-benchmark designed to evaluate the Input/Output (IO) performance of blockchain systems. It achieves this by invoking a contract that executes a high volume of read and write operations against the contract's state.

    Workload Characteristics:

    • Key Size: 20-byte keys
    • Value Size: 100-byte values
    • Operation Type: Large number of reads and writes to key-value pairs.

  2. Overview of BlockBench benchmark workloads

    master

    BlockBench provides implementations for two types of benchmark workloads: Micro-benchmarks and Macro-benchmarks.

    Micro-benchmarks

    Focus on specific, isolated resource usage patterns. Workloads include:

    • DoNothing
    • IOHeavy
    • CPUHeavy
    • Analytics

    Macro-benchmarks

    Focus on complex, system-level interactions. Workloads include:

    • YCSB (using KVStore)
    • SmallBank

  3. Understand the BlockBench benchmark source structure

    master

    The repository is organized by blockchain platform. Use the following structure to locate benchmark logic and smart contract code:

    • Smart Contracts: All contract source files are located in the contracts directory.
    • Platform-Specific Benchmarks: Instructions and scripts to execute benchmarks are organized by platform in these directories:
      • ethereum
      • hyperledger
      • parity
      • quorum_raft
      • quorum_vote
    • Contract Implementation Details:
      • For Ethereum, Parity, and Quorum: See ethereum/contracts.md.
      • For Hyperledger Chaincode: See hyperledger/contracts.md.

  4. Overview of BlockBench benchmarking workloads

    master

    BlockBench provides two types of workloads to evaluate private blockchain systems:

    Macro-benchmark workloads

    Used for evaluating overall system performance:

    • YCSB (KVStore): Key-Value Store workload.
    • SmallBank (OLTP): Online Transaction Processing workload.

    Micro-benchmark workloads

    Used for evaluating the performance of individual system layers:

    • DoNothing: Evaluates the consensus layer.
    • IOHeavy: Evaluates the data model layer (read/write oriented).
    • Analytics: Evaluates the data model layer (analytical query oriented).
    • CPUHeavy: Evaluates the execution layer.

  5. Simulate network partitions with partition.py and partition.sh

    master

    Network partitions are simulated by dropping TCP connections between nodes.

    Using partition.py

    The Python function partition(node_list, time_out) partitions the nodes in the provided node_list in half and maintains that partition for the specified time_out seconds. It acts as a wrapper for the shell script.

    Using partition.sh

    You can manually invoke the partition shell script to drop the TCP connection between two specific nodes for a set duration.

    Syntax:

    partition.sh <node_i> <node_j> <time_out>

  6. Understand the Analytic Micro Benchmark Workload

    master

    The Analytic workload evaluates a blockchain system's performance when answering analytical queries about historical data, similar to an OLAP benchmark. It specifically measures how the system handles scan-like and aggregate queries based on its data model.

    The workload implements two primary statistical queries:

    • Q1: Compute the total transaction values committed between block i and block j.
    • Q2: Compute the largest transaction value involving a given account between block i and block j.

    Implementation methods vary by system:

    • Ethereum and Parity: Queries are implemented using JSON-RPC via a shared driver.
    • Hyperledger: Custom chaincode is developed to implement the query logic.
    • Hyperledger Fabric v1.4: Chaincode simulates monetary transactions, and the NodeJS SDK is used to pull blocks for analysis.

    Note on Fabric v2.2: Analytical workloads are not provided for Fabric v2.2 because the NodeJS SDK no longer supports pulling blocks due to the removal of the fabric-client module.

  7. Use the CPUHeavy micro-benchmark workload

    master

    The CPUHeavy workload is a micro-benchmark designed to measure the efficiency of a blockchain's execution layer during computationally intensive tasks.

    Mechanism:

    1. The smart contract initializes a large array.
    2. The contract executes a quick sort algorithm over that array.

    This workload is useful for evaluating how different blockchain execution environments handle high-CPU demand within a smart contract context.

  8. Run Parity benchmarks

    master

    Benchmarks are configured via specific Python files. Before running a benchmark, you must copy its specific configuration to config.py.

    Available Benchmark Configs:

    • config_ycsb.py, config_smallbank.py, config_donothing.py: Fixed clients/servers.
    • config_saturation.py: Saturation benchmark (YCSB).
    • config_security.py: Security benchmark (YCSB).
    • config_scale.py: Scale benchmark (YCSB).
    • config_scale_fix_8.py: Scale benchmark with 8 fixed clients (YCSB).
    • config_drop.py: Failure benchmark (YCSB).

    Execution Workflow:

    1. Run All Benchmarks: python run_all.py (uses existing config.py).
    2. Run Individual Benchmark:
      • Copy the desired config: cp config_XXX.py config.py
      • Run the driver: python run.py start [opts]
      • Note: Running without [opts] executes the YCSB workload with settings from config.py.
    # Example: Running the scale benchmark
cp config_scale.py config.py
python run.py start

  10. Initialize the Ethereum network

    master

    To prepare the nodes for an experiment, use the initialization scripts:

    1. init-all.sh <nservers>: Invokes init.sh on each of the nservers nodes.
    2. init.sh <nservers>: Performs the following on a local node:
      • Uses $ETH_HOME/CustomGenesis_<nservers>.json as the genesis block.
      • Creates a new account with an empty password.
      • Sets $ETH_DATA as the geth data directory.

  12. Run Micro benchmark workloads

    master

    Blockbench provides micro-benchmark workloads designed to stress specific layers of a blockchain system (Consensus, Data Model, and Execution) to evaluate individual performance.

    Consensus layer: DoNothing

    The DoNothing workload is used to stress the consensus layer and is integrated into the ycsbc-based driver located in blockbench/src/macro/kvstore. You can execute this workload using the ./driver command by specifying the -wl donothing flag.

    Data model layer: IOHeavy & Analytics

    Detailed descriptions for these workloads can be found in their respective directories:

    • ioheavy/README.md
    • analytic/README.md

    Execution layer: CPUHeavy

    Detailed description for the CPUHeavy workload can be found in:

    • cpuheavy/README.md
    # Hyperledger
./driver -db hyperledger -threads 1 -P workloads/workloada.spec -txrate 5 -endpoint localhost:7050/chaincode -wl donothing

# Ethereum
./driver -db ethereum -threads 1 -P workloads/workloada.spec -txrate 5 -endpoint localhost:8545 -wl donothing -wt 20

# Parity
./driver -db parity -threads 1 -P workloads/workloada.spec -txrate 5 -endpoint localhost:8545 -wl donothing -wt 20