Tutorial 8: Generating tests programmatically

You can use ReFrame to generate tests programmatically using the special make_test() function. This function creates a new test type as if you have typed it manually using the class keyword. You can create arbitrarily complex tests that use variables, parameters, fixtures and pipeline hooks.

In this tutorial, we will use make_test() to build a simple domain-specific syntax for generating variants of STREAM benchmarks. Our baseline STREAM test is the following:

# Copyright 2016-2023 Swiss National Supercomputing Centre (CSCS/ETH Zurich)
# ReFrame Project Developers. See the top-level LICENSE file for details.
#
# SPDX-License-Identifier: BSD-3-Claus

import os
import reframe as rfm
import reframe.utility.sanity as sn


class stream_build(rfm.CompileOnlyRegressionTest):
    build_system = 'SingleSource'
    sourcepath = 'stream.c'
    array_size = variable(int, value=(1 << 25))
    num_iters = variable(int, value=10)
    elem_type = variable(str, value='double')
    executable = 'stream'

    @run_before('compile')
    def setup_build(self):
        try:
            omp_flag = self.current_environ.extras['ompflag']
        except KeyError:
            envname = self.current_environ.name
            self.skip(f'"ompflag" not defined for enviornment {envname!r}')

        self.build_system.cflags = [omp_flag, '-O3']
        self.build_system.cppflags = [f'-DSTREAM_ARRAY_SIZE={self.array_size}',
                                      f'-DNTIMES={self.num_iters}',
                                      f'-DSTREAM_TYPE={self.elem_type}']

    @sanity_function
    def validate_build(self):
        return True


@rfm.simple_test
class stream_test(rfm.RunOnlyRegressionTest):
    stream_binaries = fixture(stream_build, scope='environment')
    valid_systems = ['*']
    valid_prog_environs = ['+openmp']

    @run_before('run')
    def setup_omp_env(self):
        self.executable = os.path.join(self.stream_binaries.stagedir, 'stream')
        procinfo = self.current_partition.processor
        self.num_cpus_per_task = procinfo.num_cores
        self.env_vars = {
            'OMP_NUM_THREADS': self.num_cpus_per_task,
            'OMP_PLACES': 'cores'
        }

    @sanity_function
    def validate_solution(self):
        return sn.assert_found(r'Solution Validates', self.stdout)

    @performance_function('MB/s')
    def copy_bandwidth(self):
        return sn.extractsingle(r'Copy:\s+(\S+)\s+.*', self.stdout, 1, float)

It is essentially the STREAM benchmark split in two tests: one that builds the binaries based on set of variables and another one that runs it.

For our example, we would like to create a simpler syntax for generating multiple different stream_test versions that could run all at once. Here is an example specification file for those tests:

# Copyright 2016-2023 Swiss National Supercomputing Centre (CSCS/ETH Zurich)
# ReFrame Project Developers. See the top-level LICENSE file for details.
#
# SPDX-License-Identifier: BSD-3-Clause

stream_workflows:
  - elem_type: 'float'
    array_size: 16777216
    num_iters: 10
    num_cpus_per_task: 4
  - elem_type: 'double'
    array_size: 1048576
    num_iters: 100
    num_cpus_per_task: 1
  - elem_type: 'double'
    array_size: 16777216
    num_iters: 10
    thread_scaling: [1, 2, 4, 8]

The thread_scaling configuration parameter for the last workflow will create a parameterised version of the test using different number of threads. In total, we expect six stream_test versions to be generated by this configuration.

The process for generating the actual tests from this spec file comprises three steps and everything happens in a somewhat unconventional, though valid, ReFrame test file:

  1. We load the test configuration from a spec file that is passed through the STREAM_SPEC_FILE environment variable.

  2. Based on the loaded test specs we generate the actual tests using the make_test() function.

  3. We register the generated tests with the framework by applying manually the @simple_test decorator.

The whole code for generating the tests is the following and is only a few lines. Let’s walk through it.

# Copyright 2016-2023 Swiss National Supercomputing Centre (CSCS/ETH Zurich)
# ReFrame Project Developers. See the top-level LICENSE file for details.
#
# SPDX-License-Identifier: BSD-3-Clause

import os
import yaml

import reframe as rfm
import reframe.core.builtins as builtins
from reframe.core.meta import make_test

import stream


def load_specs():
    spec_file = os.getenv('STREAM_SPEC_FILE')
    if spec_file is None:
        raise ValueError('no spec file specified')

    with open(spec_file) as fp:
        try:
            specs = yaml.safe_load(fp)
        except yaml.YAMLError as err:
            raise ValueError(f'could not parse spec file: {err}') from err

    return specs


def generate_tests(specs):
    tests = []
    for i, spec in enumerate(specs['stream_workflows']):
        thread_scaling = spec.pop('thread_scaling', None)
        test_body = {
            'stream_binaries': builtins.fixture(stream.stream_build,
                                                scope='environment',
                                                variables=spec)
        }
        methods = []
        if thread_scaling:
            def _set_num_threads(test):
                test.num_cpus_per_task = test.num_threads

            test_body['num_threads'] = builtins.parameter(thread_scaling)
            methods.append(
                builtins.run_after('init')(_set_num_threads)
            )

        tests.append(make_test(
            f'stream_test_{i}', (stream.stream_test,),
            test_body,
            methods
        ))

    return tests


# Register the tests with the framework
for t in generate_tests(load_specs()):
    rfm.simple_test(t)

The load_specs() function simply loads the test specs from the YAML test spec file and does some simple sanity checking.

The generate_tests() function consumes the test specs and generates a test for each entry. Each test inherits from the base stream_test and redefines its stream_binaries fixture so that it is instantiated with the set of variables specified in the test spec. Remember that all the STREAM test variables in the YAML file refer to its build phase and thus its build fixture. We also treat specially the thread_scaling spec parameter. In this case, we add a num_threads parameter to the test and add a post-init hook that sets the test’s num_cpus_per_task.

Finally, we register the generated tests using the rfm.simple_test() decorator directly; remember that make_test() returns a class.

The equivalent of our test generation for the third spec is exactly the following:

@rfm.simple_test
class stream_test_2(stream_test):
    stream_binaries = fixture(stream_build, scope='environment',
                              variables={'elem_type': 'double',
                                         'array_size': 16777216,
                                         'num_iters': 10})
    num_threads = parameter([1, 2, 4, 8])

    @run_after('init')
    def _set_num_threads(self):
        self.num_cpus_per_task = self.num_threads

And here is the listing of generated tests:

STREAM_SPEC_FILE=stream_config.yaml ./bin/reframe -C tutorials/cscs-webinar-2022/config/mysettings.py -c tutorials/advanced/make_test/stream_workflows.py -l

[List of matched checks]
- stream_test_2 %num_threads=8 %stream_binaries.elem_type=double %stream_binaries.array_size=16777216 %stream_binaries.num_iters=10 /7b20a90a
    ^stream_build %elem_type=double %array_size=16777216 %num_iters=10 ~tresa:default+gnu 'stream_binaries /1dd920e5
- stream_test_2 %num_threads=4 %stream_binaries.elem_type=double %stream_binaries.array_size=16777216 %stream_binaries.num_iters=10 /7cbd26d7
    ^stream_build %elem_type=double %array_size=16777216 %num_iters=10 ~tresa:default+gnu 'stream_binaries /1dd920e5
- stream_test_2 %num_threads=2 %stream_binaries.elem_type=double %stream_binaries.array_size=16777216 %stream_binaries.num_iters=10 /797fb1ed
    ^stream_build %elem_type=double %array_size=16777216 %num_iters=10 ~tresa:default+gnu 'stream_binaries /1dd920e5
- stream_test_2 %num_threads=1 %stream_binaries.elem_type=double %stream_binaries.array_size=16777216 %stream_binaries.num_iters=10 /7a7dcd20
    ^stream_build %elem_type=double %array_size=16777216 %num_iters=10 ~tresa:default+gnu 'stream_binaries /1dd920e5
- stream_test_1 %stream_binaries.elem_type=double %stream_binaries.array_size=1048576 %stream_binaries.num_iters=100 %stream_binaries.num_cpus_per_task=1 /3e3643dd
    ^stream_build %elem_type=double %array_size=1048576 %num_iters=100 %num_cpus_per_task=1 ~tresa:default+gnu 'stream_binaries /3611a49a
- stream_test_0 %stream_binaries.elem_type=float %stream_binaries.array_size=16777216 %stream_binaries.num_iters=10 %stream_binaries.num_cpus_per_task=4 /d99b89f1
    ^stream_build %elem_type=float %array_size=16777216 %num_iters=10 %num_cpus_per_task=4 ~tresa:default+gnu 'stream_binaries /321abb06
Found 6 check(s)

Note

The path passed to STREAM_SPEC_FILE is relative to the test directory. Since version 4.2, ReFrame changes to the test directory before loading a test fil. In prior versions you have to specify the path relative to the current working directory.