DOCSEARCHbyAlgolia
Submit Docs

Celero C++ Benchmarking Library

repository·master·Indexed 21 days ago

https://github.com/digitalinblue/celero

A C++14 benchmarking library for reproducible performance measurements and resource tracking (RAM). It supports Windows, Linux, and OSX, providing automated reporting in Markdown, CSV, and JUnit-formatted XML for CI/CD integration. Key features include test fixtures, fixed-time baselines, User-Defined Measurements (UDM), and historical performance tracking.

Tokens
9.4K
Snippets
21
Records
37
Agent score
73%

What's inside Celero

  1. Overview of Celero C++ Benchmarking Library

    master

    Celero is a C++14 benchmarking library designed to provide consistent, reproducible, and easy-to-share performance measurements. It is intended to be integrated directly into C++ projects, making automated benchmarking a standard part of the development process similar to automated testing (e.g., GoogleTest).

    Key Capabilities:

    • Cross-Platform: Supports Windows, Linux, and OSX.
    • Low Overhead: Timing utilities can be used directly in production code.
    • Resource Tracking: Automatically tracks RAM usage during experiments.
    • Flexible Output: Formats console output as Markdown, supports CSV for graphing, and provides JUnit-formatted XML for CI/CD integration.
    • Advanced Benchmarking: Supports test fixtures, fixed-time baselines, user-defined measurements, and user-defined experiment values to scale results and sample sizes.
    • Historical Tracking: Can archive results to track performance trends (current, best, and worst) over time.

  2. Celero support response times and commercial options

    master

    Standard Support SLAs

    • Acknowledgement: Within 48 hours on business days.
    • Initial triage: Within 5 business days.
    • Feature discussions: Responses typically within 1 week.
    • Bug fixes: Prioritized based on severity.

    Commercial Support

    For enterprise-grade or SLA-backed support, contact the maintainer directly with details regarding your specific needs and volume of use.

  3. Understand the two types of Celero SBOM documents

    master

    Celero provides two distinct types of SBOM documents to answer different auditing questions:

    1. Source SBOM (celero-<version>-source.spdx.json): Answers "What is in the source distribution?". It includes Celero, vcpkg-resolved test and build dependencies, vendored third-party headers under experiments/, and GitHub Actions.
    2. Library SBOM (celero-<version>-<triplet>.spdx.json): Answers "What will I be linking against?". It includes Celero and the required operating-system libraries only.

    Note: Celero has no third-party runtime dependencies. gtest is a vcpkg dependency used only for celero-test (when CELERO_ENABLE_TESTS=ON) and therefore does not appear in the library SBOM.

  4. Understand Celero's warm-up pass behavior

    master
    If you request N iterations, Celero will actually perform N+1 passes. The first pass is an un-measured "warm-up" pass designed to account for CPU caching and other environmental factors that might otherwise skew the measurements of the subsequent N iterations.

  5. Use CELERO_MAIN for quick benchmark execution

    master
    For convenience, Celero provides a single header file and a CELERO_MAIN macro. Using this macro allows you to provide a main() function for your benchmark project that automatically discovers and executes all defined benchmark tests.

  6. Install Celero via CMake, Conan, or Vcpkg

    master

    Celero can be integrated into your project using standard C++ package managers or CMake.

    Requirements:

    • A modern C++ compiler supporting C++14.
    • For building unit tests, you must provide GoogleTest. It is recommended to use a package manager like VCPKG or Conan to provide the latest version of GoogleTest.

    Dependencies: Celero has no third-party runtime dependencies. It only relies on operating-system libraries:

    • Windows: powrprof and psapi.
    • Other (Linux/OSX): pthread.

  7. Automatically compute Iterations and Samples

    master

    If you want Celero to determine an optimal number of iterations and samples for your experiment, set both the Samples and Operations arguments to 0 in your BENCHMARK macro. Celero will then compute a statistically valid number of samples and iterations based on the execution time of the code.

    /// Celero will compute both samples and iterations automatically
BENCHMARK(DemoSimple, Complex1, 0, 0)
{
    celero::DoNotOptimizeAway(static_cast<float>(sin(fmod(UniformDistribution(RandomDevice), 3.14159265))));
}

  8. How to report bugs and request features in Celero

    master

    Celero uses GitHub issues for all support requests. To ensure a timely response, follow these labeling and content guidelines:

    Bug Reports

    Label your issue with bug. You must include:

    • Celero version or commit SHA
    • Platform and compiler details
    • Steps to reproduce and a minimal code sample

    General Questions

    Label your issue with question.

    Feature Requests

    Label your issue with enhancement. Describe:

    • The desired behavior and your specific use case
    • Any API or performance considerations

    Best Practices

    • Search existing issues before opening a new one.
    • Use clear, descriptive titles (e.g., Bug: crash in benchmark runner).
    • Fill out the provided ISSUE_TEMPLATE.md.
    • Provide relevant logs or output.
    • Tag @DigitalInBlue/celero-maintainers if you require specific attention.

  10. Explore Celero Demo and Experiment projects

    master

    Celero includes an experiments folder containing two types of reference projects:

    • Demo projects: Designed to illustrate specific techniques, ideas, and API usage.
    • Experiments: Designed to demonstrate real-world benchmarking questions and use cases.

    Users are encouraged to submit real-world use cases to the Celero development branch to be included in these libraries.

  11. Ensure stable results and prevent optimizations

    master

    To get scientifically sound results with Celero, follow these best practices:

    1. Check Assembly: Always verify the generated assembly to ensure the compiler hasn't optimized away your code under test or critical logic.
    2. Use Release Builds: Never benchmark Debug builds; always use Release builds to allow the compiler to optimize as it would in production.
    3. Stabilize the Baseline: Run your baseline multiple times. The us/Iteration and Iterations/sec should be stable. If they fluctuate, increase the number of iterations to overcome timer resolution limits.
    4. Disable CPU Frequency Scaling: On Linux, use cpupower to set the governor to performance to prevent CPU frequency fluctuations from adding noise.
      sudo cpupower frequency-set --governor performance
./your_benchmark_executable
sudo cpupower frequency-set --governor powersave
    5. Use celero::DoNotOptimizeAway: Use this template to ensure the compiler does not eliminate code that it deems has no side effects.

  12. Maintain the Celero SBOM generator

    master

    When modifying the project's dependency or build structure, you must update the SBOM generator configuration. After making changes, rerun python tools/sbom/generate_sbom.py --pinned and commit the updated documents.

    Required updates to the generator script/config:

    • New dependency in vcpkg.json: Add the dependency to PORT_SCOPE to prevent it from being conservatively filed as a build-only dependency.
    • New vendored source: Add the component to VENDORED_COMPONENTS.
    • System library changes: Update SYSTEM_LIBRARIES in CMakeLists.txt.
    • Platform changes: Update DEFAULT_TRIPLETS if adding or removing a platform from the release matrix.