BenchBase Documentation

Each benchmark implementation in BenchBase follows a standardized directory layout consisting of four main components:

1. {Prefix}Benchmark.java: The BenchmarkModule implementation. It is responsible for setting up all the necessary class paths required for the benchmark to run.
2. {Prefix}Loader.java: The Loader implementation. This component is responsible for populating the database with initial data.
3. {Prefix}Worker.java: The Worker implementation. It receives a random TransactionType from the abstract Worker driver code and invokes the corresponding procedure.
4. procedures/ directory: Contains the actual implementations for all transaction types defined within that specific benchmark.

Additionally, benchmark-specific dialects and ddls files are located in the src/main/resources directory.

BenchBase supports two monitoring modes via the -mt flag:

1. Throughput Monitoring (-mt=throughput):

   • This is the default setting if no type is explicitly set.
   • It tracks benchmark progress by logging updated throughput values directly to the system output.

2. Advanced Monitoring (-mt=advanced):

   • Monitors query and system properties via system tables (supported for SQLServer and Postgres).
   • Records findings in CSV files located in the results/monitor/ directory.
   • Collects various data points:
     • One-off query info: e.g., query plans, query text.
     • Repeated query info: e.g., elapsed time per execution, worker time, execution count.
     • Repeated system info: e.g., cache hits, number of transactions.
   • The specific events collected depend on the database system and are handled by corresponding drivers.

Values in templated benchmarks are selected using a Round-Robin access pattern. If you provide multiple <values> blocks, BenchBase will cycle through them sequentially for each query execution.

Example pattern for query 'Q1' with two value sets: V1 --> V2 --> V1 --> V2 ...

If you require more fine-grained control over the access pattern (e.g., specific weights), you should create separate queries for each value and adjust their relative weights in the main configuration file.

<templates>
    <template name="Q1">
        <query><![CDATA[$SQLQuery]]></query>
        <types>
            <type>Type1</type>
        </types>
        <values>
            <value>V1</value>
        </values>
        <values>
            <value>V2</value>
        </values>
    </template>
</templates>

To quickly get started with BenchBase using the postgres profile, clone the repository, build it using the Maven wrapper, and extract the resulting artifact. You can then run a benchmark like tpcc using the generated benchbase.jar.

Steps:

1. Clone and build:

   git clone --depth 1 https://github.com/cmu-db/benchbase.git
   cd benchbase
   ./mvnw clean package -P postgres

2. Extract the distribution:

   cd target
   tar xvzf benchbase-postgres.tgz
   cd benchbase-postgres

3. Run the benchmark:

   java -jar benchbase.jar -b tpcc -c config/postgres/sample_tpcc_config.xml --create=true --load=true --execute=true

To publish images to a container registry, use the provided build scripts and then docker push. You can specify a custom registry by setting the CONTAINER_REGISTRY_NAME environment variable.

export CONTAINER_REGISTRY_NAME='benchbasedev.azurecr.io'
./build-dev-image.sh
./build-full-image.sh

docker push $CONTAINER_REGISTRY_NAME/benchbase-dev
docker push $CONTAINER_REGISTRY_NAME/benchbase

To remove the resources created for the emulator and return your gcloud environment to its original state, follow these steps: delete the Spanner instance, reactivate your default configuration, delete the emulator configuration, and stop the Docker container.

# 1. Delete the Spanner instance
gcloud spanner instances delete benchbase-instance

# 2. Reactivate your default configuration
gcloud config configurations activate default

# 3. Delete the emulator-config
gcloud config configurations delete emulator-config

# 4. Stop the Docker container (e.g., via docker stop or docker-compose down)

You can execute benchmarks directly against the source code without manually building and extracting a distribution by using the Maven exec:java goal. This is useful for rapid development.

Example (TPC-C against PostgreSQL):

mvn clean compile exec:java -P postgres -Dexec.args="-b tpcc -c config/postgres/sample_tpcc_config.xml --create=true --load=true --execute=true"

To use advanced monitoring (-mt=advanced), the underlying database must meet specific requirements:

• Postgres: The pg_stat_statements extension must be enabled.
• SQLServer: The user must have sufficient permissions to access system tables.

Important Considerations:

• Graceful Failure: If monitoring queries fail, BenchBase will log an error but will not interrupt the benchmark execution.
• Performance Impact: Frequent monitoring queries against the DBMS can increase system load and potentially distort benchmarking results by competing with the actual benchmark queries for resources.

BenchBase uses Maven profiles to build distributions tailored for specific databases. Use the -P flag with ./mvnw clean package to specify the target database.

Supported profiles: postgres, mysql, mariadb, sqlite, cockroachdb, phoenix, and spanner.

Output: Build artifacts are placed in the ./target folder as:

• benchbase-<profile name>.tgz
• benchbase-<profile name>.zip

./mvnw clean package -P <profile name>

After unpacking a distribution, run benchbase.jar with the required arguments. Note that you must run the command from the root of the expanded distribution folder to avoid java.lang.NoClassDefFoundError.

Common usage:

• Help: java -jar benchbase.jar -h
• Single benchmark: java -jar benchbase.jar -b tpcc -c config/postgres/sample_tpcc_config.xml --create=true --load=true --execute=true
• Composite benchmarks: For benchmarks requiring multiple schemas (e.g., chbenchmark), provide a comma-separated list to the -b flag: java -jar benchbase.jar -b tpcc,chbenchmark -c config/postgres/sample_chbenchmark_config.xml --create=true --load=true --execute=true