SuperBench Documentation

SuperBench is a validation and profiling tool designed for AI infrastructure. It serves two primary purposes:

1. AI Infrastructure Validation and Diagnosis:

   • Automates distributed validation for hundreds or thousands of servers.
   • Validates both raw hardware and end-to-end (E2E) model performance using ML workload patterns.
   • Establishes contracts to identify hardware issues.
   • Provides infrastructure-oriented criteria to act as Performance/Quality Gates for hardware and system releases.
   • Generates detailed performance reports and provides advanced analysis tools.

2. AI Workload Benchmarking and Profiling:

   • Enables comprehensive performance comparisons across different hardware platforms.
   • Provides insights for hardware and software co-design.

SuperBench achieves this through a combination of micro-benchmarks (for primitive computation and communication) and model-benchmarks (for domain-aware, end-to-end deep learning workloads).

SuperBench is built on two primary modular frameworks:

Executor Framework

Designed for large-scale cluster validation, this framework uses a runner (control node) and multiple executors (worker nodes).

• The runner receives commands from the CLI, distributes them to worker nodes, collects data, and summarizes results.
• Each worker runs an executor to perform the specified benchmark tasks.

Benchmark Framework

SuperBench distinguishes between two types of benchmarks:

1. Micro-benchmarks: Focus on primitive computation (e.g., GEMM Flops, Kernel Launch Time) and communication (e.g., Memory, RDMA, NCCL).
2. Model-benchmarks: Measure end-to-end deep learning workloads (e.g., CNN, LSTM, BERT, GPT-2).

All benchmarks are built upon an abstract BenchmarkBase class, which provides common functions and ensures a unified interface and result format. Developers can extend the framework by implementing new benchmarks based on this class.

When defining a rule in the YAML rule file, you must specify one of the following function types:

• variance: Calculates the variance between raw data and the baseline: (raw data - baseline) / baseline. The criteria is then applied to this variance value.
  • Example: criteria: lambda x: x < -0.05 identifies a downgrade of more than 5%.
• value: Checks the raw data directly against the criteria without calculating variance.
  • Example: criteria: lambda x: x > 0 identifies if a value is positive.
• multi_rules: Allows a rule to depend on the results of previously defined rules. The criteria receives a label object containing the results of those rules.
  • Example: criteria: 'lambda label: bool(label["rule4"] + label["rule5"] >= 2)' triggers if the sum of results from rule4 and rule5 is at least 2.
• failure_check: Checks if metrics failed or were missed. Metrics should follow the pattern ${benchmark_name}/.*:return_code.
  • A metric is considered missed if it doesn't match any raw data.
  • A metric is considered failed if its return_code violates the value criteria (i.e., return_code != 0).
  • Note: You should always include a default rule for ${benchmark_name}/return_code to identify failed tests.

The rule file (YAML) defines how metrics are classified, aggregated, and statistically processed. It follows the same convention as the SuperBench Config File.

Rule File Structure

version: string
superbench:
  rules:
    ${rule_name}:
      statistics:
        - ${statistic_name}
      categories: string
      aggregate: (optional)[bool|string]
      metrics:
        - ${benchmark_name}/regex

Rule Elements

• metrics: A list of metrics in the format ${benchmark_name}/regex. The benchmark name must be a literal string, but you can use regex after the first / to match specific metrics.
• categories: A string used to group and organize the metrics in the final report.
• statistics: A list of statistical functions to apply to the metrics. Supported functions include:
  • count
  • max
  • mean
  • min
  • p${value} (where ${value} is 1-99, e.g., p50, p90)
  • std
• aggregate (Optional): Determines if results from multiple devices/ranks should be treated as one collection. It accepts two types of values:
  • bool:
    • False (default): No aggregation.
    • True: Aggregates results from multiple ranks. For microbenchmarks, metric names like metric:1 are aggregated into metric.
  • pattern string (regex): Uses a regex pattern to match metric names. The part of the metric matching the capture group () is replaced with * in the aggregated name, while other parts remain unchanged.

# Example Rule File
version: v0.12
superbench:
  rules:
    kernel_launch:
      statistics:
        - mean
        - p90
        - min
        - max
      aggregate: True
      categories: KernelLaunch
      metrics:
        - kernel-launch/event_time
        - kernel-launch/wall_time
    nccl:
      statistics: mean
      categories: NCCL
      metrics:
        - nccl-bw/allreduce_8388608_busbw
    ib-loopback:
      statistics: mean
      categories: RDMA
      metrics:
        - ib-loopback/IB_write_8388608_Avg_\d+
      aggregate: ib-loopback/IB_write_.*_Avg_(\d+)

SuperBench is a distributed testing framework designed for clusters of 100 to 1000 nodes (bare-metal, on-premises, or cloud). It uses a hierarchical execution model to run benchmarks across multiple nodes, supporting various device vendors (NVIDIA/AMD) and execution modes (local/MPI).

Execution Pipeline

1. Preparation: The user provides a configuration file and a host file (optional, defaults can be used).
2. CLI Invocation: The user runs the SuperBench CLI on the head node.
3. Runner Initialization: The SuperBench Runner parses inputs, checks connectivity, ensures Docker environments are ready, and starts Docker containers on all target nodes.
4. Context Distribution: The Runner distributes necessary code, configs, and SSH keys to all nodes to enable passwordless communication.
5. Execution Loop: The Runner iterates through all benchmarks and modes. For each, it calls the SuperBench Executor inside the Docker container on the target nodes.
6. Executor Execution: The Executor runs the specific benchmark tasks, handles pre/post-processing (health checks, validation), and captures results.
7. Result Collection: The Executor sends return codes and results back to the Runner. The Runner then reduces (merges) results from all compute nodes into a single summary report.
8. Completion: The CLI returns the final summarized results to the user.