DOCSEARCHbyAlgolia
Submit Docs

PerfKit Benchmarker Documentation

repository·master·Indexed 24 days ago

https://github.com/googlecloudplatform/perfkitbenchmarker

An open-source tool to define and run canonical benchmarks for measuring and comparing performance across cloud providers (GCP, AWS, Azure, IBMCloud, AliCloud, DigitalOcean, Rackspace Cloud, ProfitBricks) and platforms like Kubernetes and Mesos. Includes PerfSpect for system performance profiling on AWS baremetal and on-prem servers, a 5-stage benchmark lifecycle (Provision, Prepare, Run, Cleanup, Teardown), and a provider-specific abstraction layer.

Tokens
35K
Snippets
91
Records
186
Agent score
83%

What's inside PerfKit Benchmarker

  1. What is PerfKitBenchmarker?

    master

    PerfKitBenchmarker (PKB) is an open source benchmarking tool designed to measure and compare cloud offerings using a standardized set of benchmarks. It simplifies the process of running benchmarks across different cloud providers (such as Google Cloud Platform, Amazon Web Services, and Microsoft Azure) using unified commands.

    Key capabilities include:

    • Measuring end-to-end time to provision cloud resources.
    • Reporting standard performance metrics: latency, throughput, time-to-complete, and IOPS.

  2. Overview of Benchmarking Bigtable with PerfKit Benchmarker

    master

    This tutorial demonstrates how to use PerfKit Benchmarker (PKB) to run YCSB (Yahoo! Cloud Serving Benchmark) against Google Cloud Bigtable.

    Key Concepts

    • PerfKit Benchmarker (PKB): An open-source framework that automates the setup and teardown of benchmarking resources, including worker VMs that host the clients.
    • Worker VMs: Virtual Machines provisioned by PKB to run the actual benchmark (e.g., YCSB) against the target service.
    • Runner VM: A dedicated machine where PKB itself runs. In this tutorial, PKB runs inside a Docker container on a runner VM.
    • Workloads: The tutorial uses YCSB workloads, specifically:
      • workloada: 50/50 Read/Write
      • workloadb: 95/5 Read/Write
      • workloadc: Read-only
      • workloadx: Write-Only

    Note: Current workloads perform point read/write only.

  3. Explore PerfKit Benchmarker tutorials

    master

    The tutorials/ directory contains several guided walkthroughs for different PerfKit Benchmarker (PKB) use cases. Depending on your goal, you can follow these specific guides:

  4. Benchmark PostgreSQL on GKE with kubernetes_postgres_sysbench

    master

    The kubernetes_postgres_sysbench benchmark module automates PostgreSQL performance testing on Google Kubernetes Engine (GKE). It deploys a single PostgreSQL instance as a Kubernetes StatefulSet and runs Sysbench OLTP workloads from a client pod within the same cluster.

    Key Features

    • Automated Infrastructure: Automatically creates and tears down GKE infrastructure.
    • Metrics: Measures Transactions Per Second (TPS), Queries Per Second (QPS), and Latency.
    • Architecture: Uses two GKE nodepools (postgres for the server and clients for the Sysbench client) with private networking. No public IPs are used for database traffic.
    • Storage Support: Supports pd-ssd (for N-series) or hyperdisk-balanced (for C4-series).

  6. What is a PKB resource?

    master

    In PerfKit Benchmarker, a resource is a Python object representing a real Cloud resource.

    Key components of a resource implementation:

    • Base Class: All resources inherit from resource.BaseResource.
    • Configuration (Spec): Resources are configured via a spec that inherits from BaseSpec. These specs are encoded in the BENCHMARK_CONFIG of a benchmark and instantiated by benchmark_spec.py.
    • Lifecycle Commands: Resources implement _Create and _Delete methods, which execute the corresponding CLI commands to manage the actual Cloud resource (e.g., GoogleCloudRunJob for GCP Cloud Run Jobs).
    • Timing: PKB automatically tracks "Time to Create", "Time to Ready", and "Time to Delete". To ensure accurate metrics, place operations you want to time inside _Create. Use _CreateDependencies or _PostCreate for setup tasks that should not be included in the timing samples.

  7. Understanding the Providers directory

    master
    The /providers directory contains all logic and code specific to individual cloud providers. This includes implementations for specific GCP products and the execution of gcloud commands used to interact with those products. If you are developing or extending functionality for a specific cloud platform, your provider-specific logic should reside within this directory.

  8. Configure resources using Flags and Specs

    master

    Resources can be configured using both Spec values (defined in code) and Flag values (passed via CLI).

    • Flags: Provide a convenient way to override settings. For example, using --zone=us-central1-a is a shorthand that overrides the more verbose cluster_boot.container_cluster.vm_spec.GCP.zone=us-central1-a path in the spec.
    • Purpose: Flags are primarily used to control different testing scenarios (e.g., comparing Scenario A vs Scenario B by changing a flag value).

    Flag Development Guidelines:

    • Choose reasonable defaults in the spec, but don't feel obligated to make every single setting a flag.
    • Do not map PKB flags 1:1 with underlying cloud provider flags if they are logically coupled. For example, if a feature requires an alpha flag, you can implement a single PKB flag that triggers both the feature and the alpha argument in the underlying CLI.
    • Use benchmark-level flags for values specific to a single benchmark (e.g., amount_of_qps or repeat_count) rather than adding them to a generic resource spec.

  9. Common PKB Benchmark Architectures

    master

    Depending on what you are testing, you will use one of three common setup patterns:

    • One VM Setup: A single VM is provisioned, packages are installed, a benchmark (like SPECCPU) is run, and results are parsed. Best for simple compute benchmarks.
    • Two VM Setup: Consists of a 'client' VM and a 'server' VM. The server is the system under test, and the client sends loads to it (e.g., Netperf or database benchmarks).
    • Resource + VM Setup: The system under test is a managed cloud resource (PaaS) rather than a VM. A client VM is used to send load to that resource (e.g., SparkSQL).

  10. Run benchmarks with Juju

    master

    Juju is a service orchestration tool. To use it, specify the --os_type=juju flag. Supported benchmarks will automatically deploy a Juju-modeled service.

    Note for Developers: Benchmark/Package authors must implement the JujuInstall() method inside their package to handle deployment, configuration, and service relations. When FLAGS.os_type == JUJU, other software installation and configuration steps should be bypassed.

    $ ./pkb.py --cloud=AWS --os_type=juju --benchmarks=cassandra_stress

  12. Generate a runner config

    master

    A runner.yaml file is used by the wrapper script in the Docker image to manage benchmark execution. This file must be placed in the same folder as your benchmark configuration files. It supports two modes of operation:

    1. Unordered Tests (unordered_tests)

    Runs all benchmarks in the folder in an arbitrary order. This mode ensures that tests accessing the same Bigtable instance run one at a time to prevent interference.

    2. Ordered Tests (ordered_tests)

    Runs benchmarks in specific stages. Benchmarks in a later stage start only after all previous stages are finished. Benchmarks within the same stage run concurrently (even if they use the same Bigtable instance).

    You can also specify common flags for all benchmarks using the pkb_flags block. Note that these flags can be overridden by flags defined within individual benchmark configs.

    # Example: Unordered mode
runner:
  unordered_tests:
    num_iterations: <number of iterations per benchmark>
    concurrency: <max number of benchmarks that can be run in parallel>
  pkb_flags:
    <common_flag_1>: <value_1>
    # Example: Ordered mode
runner:
  ordered_tests:
  - <stage_0_benchmarks>  # space-separated benchmark configs for the first stage
  - <stage_1_benchmarks>  # space-separated benchmark configs for the second stage
  pkb_flags:
    <common_flag_1>: <value_1>