ReFrame is a framework designed for writing system regression tests and benchmarks specifically for High-Performance Computing (HPC) systems. It abstracts the complexity of system interactions, allowing developers to separate test logic from low-level system configuration and setup details.

Key Features:

• Declarative Testing: Write portable tests as simple Python classes that describe functionality rather than implementation details.
• Automated Pipeline: ReFrame manages a pipeline that handles programming environment switching, compilation, job submission, status querying, sanity checking, and performance assessment.
• Advanced Abstractions: Provides high-level tools for writing sanity and performance checks without manual parsing of output files or pattern searching.
• Data Management: Supports storing test results in a database for analytics and performance comparisons.
• Efficient Runtime: Offers a powerful runtime for managing test execution and integrates with logging facilities to stream live performance data.

The ReFrame Test Library provides a collection of generic tests for various HPC domains including Data Analytics, Interactive Computing, Machine Learning, Microbenchmarks, Python, Scientific Applications, and System checks.

Users can use these tests in two ways:

1. Run out-of-the-box: Specialize the generic tests for your specific system using the -S option.
2. Build site-specific tests: Use the library as a foundation to create custom tests tailored to your environment.

Available domains include:

• Data Analytics: Spark checks (hpctestlib.data_analytics.spark.spark_checks)
• Interactive Computing: Jupyter IPC magic (hpctestlib.interactive.jupyter.ipcmagic)
• Machine Learning: TensorFlow and PyTorch Horovod tests (hpctestlib.ml.tensorflow.horovod, hpctestlib.ml.pytorch.horovod)
• Microbenchmarks: OSU microbenchmarks and GPU benchmarks (gpu_burn)
• Python: NumPy operations (hpctestlib.python.numpy.numpy_ops)
• Scientific Applications: Amber, GROMACS, QEspresso, and Metalwalls benchmarks
• System: Filesystem mount options (hpctestlib.system.fs.mnt_opts)

A deferrable function is a function whose execution is postponed to a later time. Instead of executing the function body immediately, calling a deferrable function returns a reframe.core.deferrable._DeferredExpression object that captures the function and its arguments.

How to define a deferrable function

You can make any function deferrable using the @reframe.utility.sanity.deferrable decorator. If you are working within a class derived from reframe.core.pipeline.RegressionMixin, you can also use the self.deferrable method.

How to evaluate a deferred expression

To trigger the actual execution of the captured function, you must explicitly call reframe.utility.sanity.evaluate() on the expression object, or call the .evaluate() method on the object itself.

Key Characteristics

• Argument Capture: Deferrable functions capture their arguments at the moment the function is called, not when it is evaluated. If you pass a mutable object (like a list) and modify it before evaluation, the function will see the modified state. However, if you rebind a variable name to a new object (common with immutable types like tuples), the function will still operate on the original object it captured.
• Composition: Deferrable functions can accept other deferred expressions as arguments. The evaluation process will recursively traverse the call tree, evaluating all nested deferred expressions.
• Arithmetic Integration: You can use deferred expressions in standard Python arithmetic operations (e.g., +, -, *, /). The result of such an operation will be a new deferred expression representing the entire calculation.

import reframe.utility.sanity as sn

@sn.deferrable
def foo(arg):
    print(arg)

@sn.deferrable
def greetings():
    return 'hello'

# Calling the functions returns a deferred expression, not the result
expr = foo(greetings())

# To get the actual result, you must evaluate it
from reframe.utility.sanity import evaluate
result = evaluate(expr)
# OR
result = expr.evaluate()

While you can manually trigger evaluation using evaluate(), ReFrame's deferred expressions are often evaluated implicitly by the Python interpreter in the following scenarios:

• Boolean contexts: Using a deferred expression in if statements or with logical operators (and, or, not, in). This triggers a call to bool() on the expression.
• Iteration: Calling iter() on a deferred expression (e.g., in a for loop) triggers immediate evaluation.
• String representation: Calling str() on a deferred expression (e.g., via print()) triggers evaluation to show the result.

Note on the in operator: When using x in deferred_expression, the expression is evaluated immediately because Python expands this to bool(deferred_expression.__contains__(x)).

>>> from reframe.utility.sanity import defer
>>> l = defer([1, 2, 3])
>>> l
<reframe.core.deferrable._DeferredExpression object at ...>
>>> evaluate(l)
[1, 2, 3]
>>> 4 in l
False
>>> 3 in l
True

Performance tests (tests defining at least one figure of merit) automatically record data in CSV files.

Default Location: <prefix>/perflogs/<system>/<partition>/<test_base_name>.log (where <prefix> is set via --prefix or --perflogdir, and <test_base_name> is the test class name).

Customizing the CSV Format: Use the config.logging.handlers_perflog section to customize how data is written. The filelog handler is commonly used for file-based logging.

Key configuration keys:

• handlers_perflog.filelog.prefix: An additional directory prefix under the global prefix.
• handlers_perflog.format: A string specifying the log record format using %(placeholder)s. Placeholders starting with check_ refer to test attributes.
• handlers_perflog.format_perfvars: Specifies how performance values (%(check_perfvalues)s) are formatted (e.g., including the value and the unit).

reframe --prefix=/path/to/output/ -C config/custom_perflog.py -c my_test.py -r

Test variables act as knobs to change test behavior on-the-fly from the command line. They are defined using the reframe.core.builtins.variable builtin.

Key Characteristics

• Typing: Variables are typed; assigning a value of a different type raises a TypeError.
• Defaults: You can provide a default value. If no value is provided and no default exists, the variable is considered undefined, and attempting to read it will cause an error.
• Inheritance: Variables are inherited by subclasses.
• Access: Variables are accessed as standard class attributes within the test.

Command Line Usage

Use the -S option to set variables: -S var=value. To avoid setting a variable on all tests (which may trigger warnings if the variable isn't present in all tests), scope the assignment to a specific test using test_name.variable_name.

from reframe.core.builtins import variable

class stream_test(RegressionTest):
    num_threads = variable(int, default=0)

    def setup(self):
        # Accessing the variable
        self.envargs['OMP_NUM_THREADS'] = str(self.num_threads)

Set variable for all tests:

reframe ... -S num_threads=2

Set variable for a specific test only:

reframe ... -S stream_test.num_threads=2

Test fixtures allow you to perform preparation (like compiling code) once per system partition or environment and reuse the results across multiple tests. A fixture is a standard ReFrame test that is associated with a specific scope. When test A is a fixture of test B, A runs before B, and B gains access to all of A's attributes via a fixture handle.

Fixture Scopes

• session: Runs once for the entire run session.
• partition: Runs once per system partition.
• environment: Runs once per system partition and environment combination.
• test: Runs every time the calling test is executed.

Implementation

Use the reframe.core.builtins.fixture builtin. The first argument is the ReFrame test that contains the fixture logic. The function returns a handle used to access the target test's attributes (e.g., stagedir) after its setup stage.

from reframe.core.builtins import fixture

# Example: build_stream is the fixture test
# stream_test uses build_stream as its fixture
@fixture(build_stream)
class stream_test(RegressionTest):
    def setup(self):
        # Access attributes of the fixture via the handle
        executable = self.build_stream.stagedir / 'stream.x'
        self.executable = executable