DOCSEARCHbyAlgolia
Submit Docs

Caliper Performance Instrumentation and Profiling Library

repository·master·Indexed 19 days ago

https://github.com/llnl/caliper

A performance instrumentation and profiling library for High-Performance Computing (HPC) programs. It supports C, C++, Fortran, and Python, allowing developers to measure execution time, trace events, and monitor performance across MPI, OpenMP, and CUDA. The ecosystem includes the caliper-reader Python library (v0.4.1) for parsing .cali files, the GOTCHA library for function wrapping, and the CurIOus abstraction layer for POSIX IO.

Tokens
50.4K
Snippets
157
Records
205
Agent score
65%

What's inside Caliper

  1. Overview of Caliper Performance Analysis

    master

    Caliper is a program instrumentation and performance measurement framework designed as a library. It allows developers to bake performance analysis capabilities directly into C, C++, or Fortran applications and activate them at runtime. It is primarily optimized for High-Performance Computing (HPC) applications on Unix or Linux systems.

    Key capabilities include:

    • Low-overhead annotation API: Instrument source code with minimal performance impact.
    • Flexible Data Model: Uses a key:value model to capture application-specific features.
    • Parallel Model Support: Fully threadsafe implementation supporting MPI, OpenMP, and GPU profiling.
    • Measurement Types: Supports timers, PAPI hardware counters, Linux perf_events, and memory-allocation annotations.
    • Third-party Integration: Connects to tools like NVIDIA Nsight/NVProf and Intel VTune.
    • Data Aggregation: Supports both online and offline trace and profile recording.

  2. Overview of Caliper post-processing tools

    master
    Caliper provides additional tool programs designed for post-processing raw Caliper output data. By default, Caliper records data in .cali files. This output is structured as a series of snapshots, where each snapshot contains the specific information recorded by the enabled services during the measurement period.

  3. Overview of the Caliper Query Language (CalQL)

    master

    The Caliper Query Language (CalQL) is a SQL-like syntax used to filter, aggregate, and generate reports from Caliper data using tools like cali-query or report services. It allows users to transform raw trace data into meaningful profiles, such as function time profiles or time-series data, by selecting attributes, applying mathematical operations, grouping results, and formatting the output.

    SELECT *, count(), sum(time.duration.ns)
WHERE loop=mainloop
GROUP BY function
FORMAT table
ORDER BY time.duration.ns

  4. What is the MeasurementService template?

    master
    The MeasurementService is a template designed for developers building performance measurement services. It demonstrates how to implement a service that captures runtime performance data (such as hardware counters or timestamps) and uses the snapshot callback to inject that data into Caliper's snapshot records for later analysis and reporting.

  5. What is CurIOus and how is it used?

    master
    CurIOus is an abstraction layer designed to wrap I/O functions with Gotcha. Its primary purpose is to provide a consistent interface for I/O operations that can be intercepted or managed by the Gotcha framework. Currently, CurIOus specifically supports POSIX IO operations.

  6. Specify region patterns for filtering

    master

    When using include_regions or exclude_regions, you can specify how patterns are matched. If no pattern type is specified, the default is match.

    Pattern Types

    • match(pattern): An exact match (e.g., match(this_function) matches this_function).
    • startswith(pattern): Matches the start of the region name (e.g., startswith(mylib_) matches any region starting with mylib_).
    • regex(pattern): Matches a regular expression using ECMAScript grammar (e.g., regex(.*loop.*) matches any region with loop in the name).

    Example configuration string: include_regions=my_function,startswith(MPI_,mylib_),regex(.*loop.*)

  7. Define and use Configuration Profiles

    master

    Configuration profiles allow you to store multiple distinct Caliper setups in a single file and switch between them using the CALI_CONFIG_PROFILE variable.

    Profile Syntax

    • Declare a profile using # [profile name].
    • All lines following a declaration belong to that profile until the next profile is declared.
    • Entries defined before any profile declaration are part of the default profile.

    Example Profile File

    # [callstack-trace]
CALI_SERVICES_ENABLE=callpath,event,recorder,timer
CALI_CALLPATH_USE_NAME=true

# [hwc-trace]
CALI_SERVICES_ENABLE=event,papi,recorder,timer
CALI_PAPI_COUNTERS=PAPI_FP_OPS

    To select a profile, set the CALI_CONFIG_PROFILE environment variable to the profile name (e.g., export CALI_CONFIG_PROFILE=hwc-trace).

    # [callstack-trace]
CALI_SERVICES_ENABLE=callpath,event,recorder,timer
CALI_CALLPATH_USE_NAME=true

# [hwc-trace]
CALI_SERVICES_ENABLE=event,papi,recorder,timer
CALI_PAPI_COUNTERS=PAPI_FP_OPS

  8. Configure CalQL output formatters

    master

    Caliper uses the FORMAT statement in CalQL to select how data is presented. The SELECT statement determines which attributes are included in the output and their order.

    Some formatters support additional arguments using a function-call syntax, such as FORMAT tree(attribute_name).

    While most formatters ignore it, the table formatter supports the ORDER BY statement to sort records.

    SELECT event.end#function, count() FORMAT tree(event.end#function)
SELECT attribute1, attribute2 FORMAT table ORDER BY attribute1 DESC

  9. Configure region visibility in multi-threaded programs

    master

    In multi-threaded applications, you must decide if a region is visible only to the thread that created it or shared by all threads. This is controlled by the CALI_CALIPER_ATTRIBUTE_DEFAULT_SCOPE configuration variable.

    • thread (Default): Regions are visible only on the thread that creates them.
    • process: Regions are shared by all threads in the process.

    When to use process scope

    If you mark a region outside of a parallel block (on the master thread) but want to associate measurements taken inside the parallel block with that region, set the scope to process.

    Example:

    #include <caliper/cali.h>

int main() {
    // Set scope to process so parallel regions are visible to the master thread
    cali_config_set("CALI_CALIPER_ATTRIBUTE_DEFAULT_SCOPE", "process");

    CALI_MARK_BEGIN("main");
    CALI_MARK_BEGIN("parallel");

    #pragma omp parallel
    {
        // ...
    }

    CALI_MARK_END("parallel");
    CALI_MARK_END("main");
}

    Note on measurement types:

    • Event-based (e.g., runtime-report): Measurements are taken at the entry/exit of regions. If a region is only entered by the master thread, metrics only reflect the master thread.
    • Sampling-based (e.g., callpath-sample-report): Can take measurements on all threads regardless of markers. Using process scope allows these thread-wide measurements to be associated with the correct parent regions.
    #include <caliper/cali.h>

int main()
{
    cali_config_set("CALI_CALIPER_ATTRIBUTE_DEFAULT_SCOPE", "process");

    CALI_MARK_BEGIN("main");
    CALI_MARK_BEGIN("parallel");

#pragma omp parallel
    {
        // ...
    }

    CALI_MARK_END("parallel");
    CALI_MARK_END("main");
}

  10. Understand Top-down microarchitecture metrics

    master

    Top-down analysis provides a hierarchical view of CPU pipeline utilization. The values represent percentages of pipeline slots used.

    Level 1 Metrics

    The four Level 1 values should sum to approximately 100%:

    • Retiring: Pipeline slots filled with retired instructions.
    • Bad spec: Pipeline slots filled with discarded instructions due to bad speculation.
    • FE bound: Pipeline stalled due to processor front-end bottlenecks.
    • BE bound: Pipeline stalled due to processor back-end bottlenecks.

    Level 2 Metrics

    Level 2 metrics further subdivide the Level 1 categories:

    • Sub-categories of Retiring:
      • Heavy ops: Retired instructions with heavy operations.
      • Light ops: Retired instructions with light operations.
    • Sub-categories of Bad spec:
      • Br mispr: Bad speculation slots due to branch misprediction.
      • Mach clrs: Bad speculation slots due to machine clears.
    • Sub-categories of FE bound:
      • Fetch lat: Front-end bottlenecks due to fetch latency.
      • Fetch BW: Front-end bottlenecks due to fetch bandwidth.
    • Sub-categories of BE bound:
      • Core bound: Core-bound backend bottlenecks.
      • Mem bound: Memory-bound backend bottlenecks.

  11. Understand GPU profiling metrics

    master

    When using GPU profiling, Caliper reports several key metrics for each region:

    • Avg Host Time: Inclusive time (seconds) in the Caliper region on the Host (CPU). Typically the wall-clock time. Average across MPI ranks.
    • Max Host Time: Inclusive time (seconds) in the Caliper region on the Host (CPU). Maximum value among all MPI ranks.
    • Avg GPU Time: Inclusive total time (seconds) of activities executing on the GPU launched from the Caliper region. Average across MPI ranks.
    • Max GPU Time: Inclusive total time (seconds) of activities executing on the GPU launched from the Caliper region. Maximum value among all MPI ranks.
    • GPU %: Fraction of total inclusive GPU time vs. CPU time. Typically represents the GPU utilization in the Caliper region.

  12. Understand the json-split output schema

    master

    The json-split output is a top-level JSON object containing four primary fields:

    • data: A 2D array (array of arrays) representing the records. Each row is a record and each column is an attribute. Missing values are represented as null. Columns contain either direct values or references (indices pointing to the nodes array).
    • columns: An array of strings containing the labels (attribute names) for the data columns.
    • column_metadata: An array of objects corresponding to the columns. Each object contains an is_value boolean:
      • If true, the column contains direct values.
      • If false, the column contains indices referencing the nodes array.
    • nodes: An array of metadata node objects that form a tree or forest. Each node has a label (its value) and an optional parent (the index of its parent node in the nodes array). A parent node is guaranteed to appear in the array before its children.
    {
  "data": [
    [ 1, 3395643, 0 ],
    [ 100, 1280, 2 ]
  ],
  "columns": [ "count", "time.inclusive.duration", "path" ],
  "column_metadata": [ { "is_value": true }, { "is_value": true }, { "is_value": false } ],
  "nodes": [ 
    { "label": "main" }, 
    { "label": "lulesh.cycle", "parent": 0 }, 
    { "label": "TimeIncrement", "parent": 1 } 
  ]
}