Tutorial 2: Customizing Further a Regression Test

In this tutorial we will present common patterns that can come up when writing regression tests with ReFrame. All examples use the configuration file presented in Tutorial 1: Getting Started with ReFrame, which you can find in tutorials/config/settings.py. We also assume that the reader is already familiar with the concepts presented in the basic tutorial. Finally, to avoid specifying the tutorial configuration file each time, make sure to export it here:

export RFM_CONFIG_FILE=$(pwd)/tutorials/config/mysettings.py

Parameterizing a Regression Test

We have briefly looked into parameterized tests in Tutorial 1: Getting Started with ReFrame where we parameterized the “Hello, World!” test based on the programming language. Test parameterization in ReFrame is quite powerful since it allows you to create a multitude of similar tests automatically. In this example, we will parameterize the last version of the STREAM test from the Tutorial 1: Getting Started with ReFrame by changing the array size, so as to check the bandwidth of the different cache levels. Here is the adapted code with the relevant parts highlighted (for simplicity, we are interested only in the “Triad” benchmark):

cat tutorials/advanced/parameterized/stream.py
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class StreamMultiSysTest(rfm.RegressionTest):
    num_bytes = parameter(1 << pow for pow in range(19, 30))
    array_size = variable(int)
    ntimes = variable(int)

    valid_systems = ['*']
    valid_prog_environs = ['cray', 'gnu', 'intel', 'pgi']
    prebuild_cmds = [
        'wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c'  # noqa: E501
    ]
    build_system = 'SingleSource'
    sourcepath = 'stream.c'
    variables = {
        'OMP_NUM_THREADS': '4',
        'OMP_PLACES': 'cores'
    }
    reference = {
        '*': {
            'Triad': (0, None, None, 'MB/s'),
        }
    }

    # Flags per programming environment
    flags = variable(dict, value={
        'cray':  ['-fopenmp', '-O3', '-Wall'],
        'gnu':   ['-fopenmp', '-O3', '-Wall'],
        'intel': ['-qopenmp', '-O3', '-Wall'],
        'pgi':   ['-mp', '-O3']
    })

    # Number of cores for each system
    cores = variable(dict, value={
        'catalina:default': 4,
        'daint:gpu': 12,
        'daint:mc': 36,
        'daint:login': 10
    })

    @run_after('init')
    def set_variables(self):
        self.array_size = (self.num_bytes >> 3) // 3
        self.ntimes = 100*1024*1024 // self.array_size
        self.descr = (
            f'STREAM test (array size: {self.array_size}, '
            f'ntimes: {self.ntimes})'
        )

    @run_before('compile')
    def set_compiler_flags(self):
        self.build_system.cppflags = [f'-DSTREAM_ARRAY_SIZE={self.array_size}',
                                      f'-DNTIMES={self.ntimes}']
        environ = self.current_environ.name
        self.build_system.cflags = self.flags.get(environ, [])

    @run_before('run')
    def set_num_threads(self):
        num_threads = self.cores.get(self.current_partition.fullname, 1)
        self.num_cpus_per_task = num_threads
        self.variables = {
            'OMP_NUM_THREADS': str(num_threads),
            'OMP_PLACES': 'cores'
        }

    @sanity_function
    def validate_solution(self):
        return sn.assert_found(r'Solution Validates', self.stdout)

    @performance_function('MB/s', perf_key='Triad')
    def extract_triad_bw(self):
        return sn.extractsingle(r'Triad:\s+(\S+)\s+.*', self.stdout, 1, float)

Any ordinary ReFrame test becomes a parameterized one if the user defines parameters inside the class body of the test. This is done using the parameter() ReFrame built-in function, which accepts the list of parameter values. For each parameter value ReFrame will instantiate a different regression test by assigning the corresponding value to an attribute named after the parameter. So in this example, ReFrame will generate automatically 11 tests with different values for their num_bytes attribute. From this point on, you can adapt the test based on the parameter values, as we do in this case, where we compute the STREAM array sizes, as well as the number of iterations to be performed on each benchmark, and we also compile the code accordingly.

Let’s try listing the generated tests:

./bin/reframe -c tutorials/advanced/parameterized/stream.py -l
[ReFrame Setup]
  version:           3.6.0-dev.0+2f8e5b3b
  command:           './bin/reframe -c tutorials/advanced/parameterized/stream.py -l'
  launched by:       user@tresa.local
  working directory: '/Users/user/Repositories/reframe'
  settings file:     'tutorials/config/settings.py'
  check search path: '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py'
  stage directory:   '/Users/user/Repositories/reframe/stage'
  output directory:  '/Users/user/Repositories/reframe/output'

[List of matched checks]
- StreamMultiSysTest_2097152 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
- StreamMultiSysTest_67108864 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
- StreamMultiSysTest_1048576 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
- StreamMultiSysTest_536870912 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
- StreamMultiSysTest_4194304 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
- StreamMultiSysTest_33554432 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
- StreamMultiSysTest_8388608 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
- StreamMultiSysTest_268435456 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
- StreamMultiSysTest_16777216 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
- StreamMultiSysTest_524288 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
- StreamMultiSysTest_134217728 (found in '/Users/user/Repositories/reframe/tutorials/advanced/parameterized/stream.py')
Found 11 check(s)

Log file(s) saved in: '/var/folders/h7/k7cgrdl13r996m4dmsvjq7v80000gp/T/rfm-s_ty1l50.log'

ReFrame generates 11 tests from the single parameterized test that we have written and names them by appending a string representation of the parameter value.

Test parameterization in ReFrame is very powerful since you can parameterize your tests on anything and you can create complex parameterization spaces. A common pattern is to parameterize a test on the environment module that loads a software in order to test different versions of it. For this reason, ReFrame offers the find_modules() function, which allows you to parameterize a test on the available modules for a given programming environment and partition combination. The following example will create a test for each GROMACS module found on the software stack associated with a system partition and programming environment (toolchain):

import reframe as rfm
import reframe.utility as util


@rfm.simple_test
class MyTest(rfm.RegressionTest):
    module_info = parameter(util.find_modules('GROMACS'))

    @run_after('init')
    def process_module_info(self):
        s, e, m = self.module_info
        self.valid_systems = [s]
        self.valid_prog_environs = [e]
        self.modules = [m]

More On Building Tests

We have already seen how ReFrame can compile a test with a single source file. However, ReFrame can also build tests that use Make or a configure-Make approach. We are going to demonstrate this through a simple C++ program that computes a dot-product of two vectors and is being compiled through a Makefile. Additionally, we can select the type of elements for the vectors at compilation time. Here is the C++ program:

cat tutorials/advanced/makefiles/src/dotprod.cpp
#include <cassert>
#include <iostream>
#include <random>
#include <vector>

#ifndef ELEM_TYPE
#define ELEM_TYPE double
#endif

using elem_t = ELEM_TYPE;

template<typename T>
T dotprod(const std::vector<T> &x, const std::vector<T> &y)
{
    assert(x.size() == y.size());
    T sum = 0;
    for (std::size_t i = 0; i < x.size(); ++i) {
        sum += x[i] * y[i];
    }

    return sum;
}

template<typename T>
struct type_name {
    static constexpr const char *value = nullptr;
};

template<>
struct type_name<float> {
    static constexpr const char *value = "float";
};

template<>
struct type_name<double> {
    static constexpr const char *value = "double";
};

int main(int argc, char *argv[])
{
    if (argc < 2) {
        std::cerr << argv[0] << ": too few arguments\n";
        std::cerr << "Usage: " << argv[0] << " DIM\n";
        return 1;
    }

    std::size_t N = std::atoi(argv[1]);
    if (N < 0) {
        std::cerr << argv[0]
                  << ": array dimension must a positive integer: " << argv[1]
                  << "\n";
        return 1;
    }

    std::vector<elem_t> x(N), y(N);
    std::random_device seed;
    std::mt19937 rand(seed());
    std::uniform_real_distribution<> dist(-1, 1);
    for (std::size_t i = 0; i < N; ++i) {
        x[i] = dist(rand);
        y[i] = dist(rand);
    }

    std::cout << "Result (" << type_name<elem_t>::value << "): "
              << dotprod(x, y) << "\n";
    return 0;
}

The directory structure for this test is the following:

tutorials/makefiles/
├── maketest.py
└── src
    ├── Makefile
    └── dotprod.cpp

Let’s have a look at the test itself:

cat tutorials/advanced/makefiles/maketest.py
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class MakefileTest(rfm.RegressionTest):
    elem_type = parameter(['float', 'double'])

    descr = 'Test demonstrating use of Makefiles'
    valid_systems = ['*']
    valid_prog_environs = ['clang', 'gnu']
    executable = './dotprod'
    executable_opts = ['100000']
    build_system = 'Make'

    @run_before('compile')
    def set_compiler_flags(self):
        self.build_system.cppflags = [f'-DELEM_TYPE={self.elem_type}']

    @sanity_function
    def validate_test(self):
        return sn.assert_found(rf'Result \({self.elem_type}\):', self.stdout)

First, if you’re using any build system other than SingleSource, you must set the executable attribute of the test, because ReFrame cannot know what is the actual executable to be run. We then set the build system to Make and set the preprocessor flags as we would do with the SingleSource build system.

Let’s inspect the build script generated by ReFrame:

./bin/reframe -c tutorials/advanced/makefiles/maketest.py -r
cat output/catalina/default/clang/MakefileTest_float/rfm_MakefileTest_build.sh
#!/bin/bash

_onerror()
{
    exitcode=$?
    echo "-reframe: command \`$BASH_COMMAND' failed (exit code: $exitcode)"
    exit $exitcode
}

trap _onerror ERR

make -j 1 CC="cc" CXX="CC" FC="ftn" NVCC="nvcc" CPPFLAGS="-DELEM_TYPE=float"

The compiler variables (CC, CXX etc.) are set based on the corresponding values specified in the configuration of the current environment. We can instruct the build system to ignore the default values from the environment by setting its flags_from_environ attribute to false:

self.build_system.flags_from_environ = False

In this case, make will be invoked as follows:

make -j 1 CPPFLAGS="-DELEM_TYPE=float"

Notice that the -j 1 option is always generated. We can increase the build concurrency by setting the max_concurrency attribute. Finally, we may even use a custom Makefile by setting the makefile attribute:

self.build_system.max_concurrency = 4
self.build_system.makefile = 'Makefile_custom'

As a final note, as with the SingleSource build system, it wouldn’t have been necessary to specify one in this test, if we wouldn’t have to set the CPPFLAGS. ReFrame could automatically figure out the correct build system if sourcepath refers to a directory. ReFrame will inspect the directory and it will first try to determine whether this is a CMake or Autotools-based project.

More details on ReFrame’s build systems can be found here.

Retrieving the source code from a Git repository

It might be the case that a regression test needs to clone its source code from a remote repository. This can be achieved in two ways with ReFrame. One way is to set the sourcesdir attribute to None and explicitly clone a repository using the prebuild_cmds:

self.sourcesdir = None
self.prebuild_cmds = ['git clone https://github.com/me/myrepo .']

Alternatively, we can retrieve specifically a Git repository by assigning its URL directly to the sourcesdir attribute:

self.sourcesdir = 'https://github.com/me/myrepo'

ReFrame will attempt to clone this repository inside the stage directory by executing git clone <repo> . and will then proceed with the build procedure as usual.

Note

ReFrame recognizes only URLs in the sourcesdir attribute and requires passwordless access to the repository. This means that the SCP-style repository specification will not be accepted. You will have to specify it as URL using the ssh:// protocol (see Git documentation page).

Adding a configuration step before compiling the code

It is often the case that a configuration step is needed before compiling a code with make. To address this kind of projects, ReFrame aims to offer specific abstractions for “configure-make” style of build systems. It supports CMake-based projects through the CMake build system, as well as Autotools-based projects through the Autotools build system.

For other build systems, you can achieve the same effect using the Make build system and the prebuild_cmds for performing the configuration step. The following code snippet will configure a code with ./custom_configure before invoking make:

self.prebuild_cmds = ['./custom_configure -with-mylib']
self.build_system = 'Make'
self.build_system.cppflags = ['-DHAVE_FOO']
self.build_system.flags_from_environ = False

The generated build script will then have the following lines:

./custom_configure -with-mylib
make -j 1 CPPFLAGS='-DHAVE_FOO'

Writing a Run-Only Regression Test

There are cases when it is desirable to perform regression testing for an already built executable. In the following test we use simply the echo Bash shell command to print a random integer between specific lower and upper bounds. Here is the full regression test:

cat tutorials/advanced/runonly/echorand.py
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class EchoRandTest(rfm.RunOnlyRegressionTest):
    descr = 'A simple test that echoes a random number'
    valid_systems = ['*']
    valid_prog_environs = ['*']
    lower = variable(int, value=90)
    upper = variable(int, value=100)
    executable = 'echo'
    executable_opts = [
        'Random: ',
        f'$((RANDOM%({upper}+1-{lower})+{lower}))'
    ]

    @sanity_function
    def assert_solution(self):
        return sn.assert_bounded(
            sn.extractsingle(
                r'Random: (?P<number>\S+)', self.stdout, 'number', float
            ),
            self.lower, self.upper
        )

There is nothing special for this test compared to those presented so far except that it derives from the RunOnlyRegressionTest class. Note that setting the executable in this type of test is always required. Run-only regression tests may also have resources, as for instance a pre-compiled executable or some input data. These resources may reside under the src/ directory or under any directory specified in the sourcesdir attribute. These resources will be copied to the stage directory at the beginning of the run phase.

Writing a Compile-Only Regression Test

ReFrame provides the option to write compile-only tests which consist only of a compilation phase without a specified executable. This kind of tests must derive from the CompileOnlyRegressionTest class provided by the framework. The following test is a compile-only version of the MakefileTest presented previously which checks that no warnings are issued by the compiler:

cat tutorials/advanced/makefiles/maketest.py
@rfm.simple_test
class MakeOnlyTest(rfm.CompileOnlyRegressionTest):
    elem_type = parameter(['float', 'double'])
    descr = 'Test demonstrating use of Makefiles'
    valid_systems = ['*']
    valid_prog_environs = ['clang', 'gnu']
    build_system = 'Make'

    @run_before('compile')
    def set_compiler_flags(self):
        self.build_system.cppflags = [f'-DELEM_TYPE={self.elem_type}']

    @sanity_function
    def validate_compilation(self):
        return sn.assert_not_found(r'warning', self.stdout)

What is worth noting here is that the standard output and standard error of the test, which are accessible through the stdout and stderr attributes, correspond now to the standard output and error of the compilation command. Therefore sanity checking can be done in exactly the same way as with a normal test.

Grouping parameter packs

New in version 3.4.2.

In the dot product example shown above, we had two independent tests that defined the same elem_type parameter. And the two tests cannot have a parent-child relationship, since one of them is a run-only test and the other is a compile-only one. ReFrame offers the RegressionMixin class that allows you to group parameters and other builtins that are meant to be reused over otherwise unrelated tests. In the example below, we create an ElemTypeParam mixin that holds the definition of the elem_type parameter which is inherited by both the concrete test classes:

import reframe as rfm
import reframe.utility.sanity as sn


class ElemTypeParam(rfm.RegressionMixin):
    elem_type = parameter(['float', 'double'])


@rfm.simple_test
class MakefileTestAlt(rfm.RegressionTest, ElemTypeParam):
    descr = 'Test demonstrating use of Makefiles'
    valid_systems = ['*']
    valid_prog_environs = ['clang', 'gnu']
    executable = './dotprod'
    executable_opts = ['100000']
    build_system = 'Make'

    @run_before('compile')
    def set_compiler_flags(self):
        self.build_system.cppflags = [f'-DELEM_TYPE={self.elem_type}']

    @sanity_function
    def validate_test(self):
        return sn.assert_found(
            rf'Result \({self.elem_type}\):', self.stdout
        )


@rfm.simple_test
class MakeOnlyTestAlt(rfm.CompileOnlyRegressionTest, ElemTypeParam):
    descr = 'Test demonstrating use of Makefiles'
    valid_systems = ['*']
    valid_prog_environs = ['clang', 'gnu']
    build_system = 'Make'

    @run_before('compile')
    def set_compiler_flags(self):
        self.build_system.cppflags = [f'-DELEM_TYPE={self.elem_type}']

    @sanity_function
    def validate_build(self):
        return sn.assert_not_found(r'warning', self.stdout)

Notice how the parameters are expanded in each of the individual tests:

./bin/reframe -c tutorials/advanced/makefiles/maketest_mixin.py -l
[ReFrame Setup]
  version:           3.6.0-dev.0+2f8e5b3b
  command:           './bin/reframe -c tutorials/advanced/makefiles/maketest_mixin.py -l'
  launched by:       user@tresa.local
  working directory: '/Users/user/Repositories/reframe'
  settings file:     'tutorials/config/settings.py'
  check search path: '/Users/user/Repositories/reframe/tutorials/advanced/makefiles/maketest_mixin.py'
  stage directory:   '/Users/user/Repositories/reframe/stage'
  output directory:  '/Users/user/Repositories/reframe/output'

[List of matched checks]
- MakeOnlyTestAlt_double (found in '/Users/user/Repositories/reframe/tutorials/advanced/makefiles/maketest_mixin.py')
- MakeOnlyTestAlt_float (found in '/Users/user/Repositories/reframe/tutorials/advanced/makefiles/maketest_mixin.py')
- MakefileTestAlt_double (found in '/Users/user/Repositories/reframe/tutorials/advanced/makefiles/maketest_mixin.py')
- MakefileTestAlt_float (found in '/Users/user/Repositories/reframe/tutorials/advanced/makefiles/maketest_mixin.py')
Found 4 check(s)

Log file(s) saved in: '/var/folders/h7/k7cgrdl13r996m4dmsvjq7v80000gp/T/rfm-e384bvkd.log'

Applying a Sanity Function Iteratively

It is often the case that a common sanity function has to be applied many times. The following script prints 100 random integers between the limits given by the environment variables LOWER and UPPER.

cat tutorials/advanced/random/src/random_numbers.sh
if [ -z $LOWER ]; then
    export LOWER=90
fi

if [ -z $UPPER ]; then
    export UPPER=100
fi

for i in {1..100}; do
    echo Random: $((RANDOM%($UPPER+1-$LOWER)+$LOWER))
done

In the corresponding regression test we want to check that all the random numbers generated lie between the two limits, which means that a common sanity check has to be applied to all the printed random numbers. Here is the corresponding regression test:

cat tutorials/advanced/random/randint.py
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class DeferredIterationTest(rfm.RunOnlyRegressionTest):
    descr = 'Apply a sanity function iteratively'
    valid_systems = ['*']
    valid_prog_environs = ['*']
    executable = './random_numbers.sh'

    @sanity_function
    def validate_test(self):
        numbers = sn.extractall(
            r'Random: (?P<number>\S+)', self.stdout, 'number', float
        )
        return sn.all([
            sn.assert_eq(sn.count(numbers), 100),
            sn.all(sn.map(lambda x: sn.assert_bounded(x, 90, 100), numbers))
        ])

First, we extract all the generated random numbers from the output. What we want to do is to apply iteratively the assert_bounded() sanity function for each number. The problem here is that we cannot simply iterate over the numbers list, because that would trigger prematurely the evaluation of the extractall(). We want to defer also the iteration. This can be achieved by using the map() ReFrame sanity function, which is a replacement of Python’s built-in map() function and does exactly what we want: it applies a function on all the elements of an iterable and returns another iterable with the transformed elements. Passing the result of the map() function to the all() sanity function ensures that all the elements lie between the desired bounds.

There is still a small complication that needs to be addressed. As a direct replacement of the built-in all() function, ReFrame’s all() sanity function returns True for empty iterables, which is not what we want. So we must make sure that all 100 numbers are generated. This is achieved by the sn.assert_eq(sn.count(numbers), 100) statement, which uses the count() sanity function for counting the generated numbers. Finally, we need to combine these two conditions to a single deferred expression that will be returned by the test’s @sanity_function. We accomplish this by using the all() sanity function.

For more information about how exactly sanity functions work and how their execution is deferred, please refer to Understanding the Mechanism of Deferrable Functions.

Note

New in version 2.13: ReFrame offers also the allx() sanity function which, conversely to the builtin all() function, will return False if its iterable argument is empty.

Customizing the Test Job Script

It is often the case that we need to run some commands before or after the parallel launch of our executable. This can be easily achieved by using the prerun_cmds and postrun_cmds attributes of a ReFrame test.

The following example is a slightly modified version of the random numbers test presented above. The lower and upper limits for the random numbers are now set inside a helper shell script in limits.sh located in the test’s resources, which we need to source before running our tests. Additionally, we want also to print FINISHED after our executable has finished. Here is the modified test file:

cat tutorials/advanced/random/prepostrun.py
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class PrepostRunTest(rfm.RunOnlyRegressionTest):
    descr = 'Pre- and post-run demo test'
    valid_systems = ['*']
    valid_prog_environs = ['*']
    prerun_cmds = ['source limits.sh']
    postrun_cmds = ['echo FINISHED']
    executable = './random_numbers.sh'

    @sanity_function
    def validate_test(self):
        numbers = sn.extractall(
            r'Random: (?P<number>\S+)', self.stdout, 'number', float
        )
        return sn.all([
            sn.assert_eq(sn.count(numbers), 100),
            sn.all(sn.map(lambda x: sn.assert_bounded(x, 90, 100), numbers)),
            sn.assert_found(r'FINISHED', self.stdout)
        ])

The prerun_cmds and postrun_cmds are lists of commands to be emitted in the generated job script before and after the parallel launch of the executable. Obviously, the working directory for these commands is that of the job script itself, which is the stage directory of the test. The generated job script for this test looks like the following:

./bin/reframe -c tutorials/advanced/random/prepostrun.py -r
cat output/catalina/default/gnu/PrepostRunTest/rfm_PrepostRunTest_job.sh
#!/bin/bash
source limits.sh
 ./random_numbers.sh
echo FINISHED

Generally, ReFrame generates the job shell scripts using the following pattern:

#!/bin/bash -l
{job_scheduler_preamble}
{prepare_cmds}
{env_load_cmds}
{prerun_cmds}
{parallel_launcher} {executable} {executable_opts}
{postrun_cmds}

The job_scheduler_preamble contains the backend job scheduler directives that control the job allocation. The prepare_cmds are commands that can be emitted before the test environment commands. These can be specified with the prepare_cmds partition configuration option. The env_load_cmds are the necessary commands for setting up the environment of the test. These include any modules or environment variables set at the system partition level or any modules or environment variables set at the test level. Then the commands specified in prerun_cmds follow, while those specified in the postrun_cmds come after the launch of the parallel job. The parallel launch itself consists of three parts:

  1. The parallel launcher program (e.g., srun, mpirun etc.) with its options,

  2. the regression test executable as specified in the executable attribute and

  3. the options to be passed to the executable as specified in the executable_opts attribute.

Adding job scheduler options per test

Sometimes a test needs to pass additional job scheduler options to the automatically generated job script. This is fairly easy to achieve with ReFrame. In the following test we want to test whether the --mem option of Slurm works as expected. We compiled and ran a program that consumes all the available memory of the node, but we want to restrict the available memory with the --mem option. Here is the test:

cat tutorials/advanced/jobopts/eatmemory.py
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class MemoryLimitTest(rfm.RegressionTest):
    valid_systems = ['daint:gpu', 'daint:mc']
    valid_prog_environs = ['gnu']
    sourcepath = 'eatmemory.c'
    executable_opts = ['2000M']

    @run_before('run')
    def set_memory_limit(self):
        self.job.options = ['--mem=1000']

    @sanity_function
    def validate_test(self):
        return sn.assert_found(
            r'(exceeded memory limit)|(Out Of Memory)', self.stderr
        )

Each ReFrame test has an associated run job descriptor which represents the scheduler job that will be used to run this test. This object has an options attribute, which can be used to pass arbitrary options to the scheduler. The job descriptor is initialized by the framework during the setup pipeline phase. For this reason, we cannot directly set the job options inside the test constructor and we have to use a pipeline hook that runs before running (i.e., submitting the test).

Let’s run the test and inspect the generated job script:

./bin/reframe -c tutorials/advanced/jobopts/eatmemory.py -n MemoryLimitTest -r
cat output/daint/gpu/gnu/MemoryLimitTest/rfm_MemoryLimitTest_job.sh
#!/bin/bash
#SBATCH --job-name="rfm_MemoryLimitTest_job"
#SBATCH --ntasks=1
#SBATCH --output=rfm_MemoryLimitTest_job.out
#SBATCH --error=rfm_MemoryLimitTest_job.err
#SBATCH --time=0:10:0
#SBATCH -A csstaff
#SBATCH --constraint=gpu
#SBATCH --mem=1000
module unload PrgEnv-cray
module load PrgEnv-gnu
srun ./MemoryLimitTest 2000M

The job options specified inside a ReFrame test are always the last to be emitted in the job script preamble and do not affect the options that are passed implicitly through other test attributes or configuration options.

There is a small problem with this test though. What if we change the job scheduler in that partition or what if we want to port the test to a different system that does not use Slurm and another option is needed to achieve the same result. The obvious answer is to adapt the test, but is there a more portable way? The answer is yes and this can be achieved through so-called extra resources. ReFrame gives you the possibility to associate scheduler options to a “resource” managed by the partition scheduler. You can then use those resources transparently from within your test.

To achieve this in our case, we first need to define a memory resource in the configuration:

                # rfmdocstart: gpu-partition
                {
                    'name': 'gpu',
                    'descr': 'Hybrid nodes',
                    'scheduler': 'slurm',
                    'launcher': 'srun',
                    'access': ['-C gpu', '-A csstaff'],
                    'environs': ['gnu', 'intel', 'pgi', 'cray'],
                    'max_jobs': 100,
                    'resources': [
                        {
                            'name': 'memory',
                            'options': ['--mem={size}']
                        }
                    ],
                    'container_platforms': [
                        {
                            'type': 'Sarus',
                            'modules': ['sarus']
                        },
                        {
                            'type': 'Singularity',
                            'modules': ['singularity']
                        }
                    ]
                },
                # rfmdocend: gpu-partition
                {
                    'name': 'mc',
                    'descr': 'Multicore nodes',
                    'scheduler': 'slurm',
                    'launcher': 'srun',
                    'access': ['-C mc', '-A csstaff'],
                    'environs': ['gnu', 'intel', 'pgi', 'cray'],
                    'max_jobs': 100,
                    'resources': [
                        {
                            'name': 'memory',
                            'options': ['--mem={size}']
                        }
                    ]
                }

Notice that we do not define the resource for all the partitions, but only for those that it makes sense. Each resource has a name and a set of scheduler options that will be passed to the scheduler when this resource will be requested by the test. The options specification can contain placeholders, whose value will also be set from the test. Let’s see how we can rewrite the MemoryLimitTest using the memory resource instead of passing the --mem scheduler option explicitly.

cat tutorials/advanced/jobopts/eatmemory.py
@rfm.simple_test
class MemoryLimitWithResourcesTest(rfm.RegressionTest):
    valid_systems = ['daint:gpu', 'daint:mc']
    valid_prog_environs = ['gnu']
    sourcepath = 'eatmemory.c'
    executable_opts = ['2000M']
    extra_resources = {
        'memory': {'size': '1000'}
    }

    @sanity_function
    def validate_test(self):
        return sn.assert_found(
            r'(exceeded memory limit)|(Out Of Memory)', self.stderr
        )

The extra resources that the test needs to obtain through its scheduler are specified in the extra_resources attribute, which is a dictionary with the resource names as its keys and another dictionary assigning values to the resource placeholders as its values. As you can see, this syntax is completely scheduler-agnostic. If the requested resource is not defined for the current partition, it will be simply ignored.

You can now run and verify that the generated job script contains the --mem option:

./bin/reframe -c tutorials/advanced/jobopts/eatmemory.py -n MemoryLimitWithResourcesTest -r
cat output/daint/gpu/gnu/MemoryLimitWithResourcesTest/rfm_MemoryLimitWithResourcesTest_job.sh

Modifying the parallel launcher command

Another relatively common need is to modify the parallel launcher command. ReFrame gives the ability to do that and we will see some examples in this section.

The most common case is to pass arguments to the launcher command that you cannot normally pass as job options. The --cpu-bind of srun is such an example. Inside a ReFrame test, you can access the parallel launcher through the launcher of the job descriptor. This object handles all the details of how the parallel launch command will be emitted. In the following test we run a CPU affinity test using this utility and we will pin the threads using the --cpu-bind option:

cat tutorials/advanced/affinity/affinity.py
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class AffinityTest(rfm.RegressionTest):
    valid_systems = ['daint:gpu', 'daint:mc']
    valid_prog_environs = ['*']
    sourcesdir = 'https://github.com/vkarak/affinity.git'
    build_system = 'Make'
    executable = './affinity'

    @run_before('compile')
    def set_build_system_options(self):
        self.build_system.options = ['OPENMP=1']

    @run_before('run')
    def set_cpu_binding(self):
        self.job.launcher.options = ['--cpu-bind=cores']

    @sanity_function
    def validate_test(self):
        return sn.assert_found(r'CPU affinity', self.stdout)

The approach is identical to the approach we took in the MemoryLimitTest test above, except that we now set the launcher options.

Note

The sanity checking in a real affinity checking test would be much more complex than this.

Another scenario that might often arise when testing parallel debuggers is the need to wrap the launcher command with the debugger command. For example, in order to debug a parallel program with ARM DDT, you would need to invoke the program like this: ddt [OPTIONS] srun [OPTIONS]. ReFrame allows you to wrap the launcher command without the test needing to know which is the actual parallel launcher command for the current partition. This can be achieved with the following pipeline hook:

import reframe as rfm
from reframe.core.launchers import LauncherWrapper

class DebuggerTest(rfm.RunOnlyRegressionTest):
    ...

    @run_before('run')
    def set_launcher(self):
        self.job.launcher = LauncherWrapper(self.job.launcher, 'ddt',
                                            ['--offline'])

The LauncherWrapper is a pseudo-launcher that wraps another one and allows you to prepend anything to it. In this case the resulting parallel launch command, if the current partition uses native Slurm, will be ddt --offline srun [OPTIONS].

Replacing the parallel launcher

Sometimes you might need to replace completely the partition’s launcher command, because the software you are testing might use its own parallel launcher. Examples are ipyparallel, the GREASY high-throughput scheduler, as well as some visualization software. The trick here is to replace the parallel launcher with the local one, which practically does not emit any launch command, and by now you should almost be able to do it all by yourself:

import reframe as rfm
from reframe.core.backends import getlauncher


class CustomLauncherTest(rfm.RunOnlyRegressionTest):
    ...
    executable = 'custom_scheduler'
    executable_opts = [...]

    @run_before('run')
    def replace_launcher(self):
        self.job.launcher = getlauncher('local')()

The getlauncher() function takes the registered name of a launcher and returns the class that implements it. You then instantiate the launcher and assign to the launcher attribute of the job descriptor.

An alternative to this approach would be to define your own custom parallel launcher and register it with the framework. You could then use it as the scheduler of a system partition in the configuration, but this approach is less test-specific.

Adding more parallel launch commands

ReFrame uses a parallel launcher by default for anything defined explicitly or implicitly in the executable test attribute. But what if we want to generate multiple parallel launch commands? One straightforward solution is to hardcode the parallel launch command inside the prerun_cmds or postrun_cmds, but this is not so portable. The best way is to ask ReFrame to emit the parallel launch command for you. The following is a simple test for demonstration purposes that runs the hostname command several times using a parallel launcher. It resembles a scaling test, except that all happens inside a single ReFrame test, instead of launching multiple instances of a parameterized test.

cat tutorials/advanced/multilaunch/multilaunch.py
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class MultiLaunchTest(rfm.RunOnlyRegressionTest):
    valid_systems = ['daint:gpu', 'daint:mc']
    valid_prog_environs = ['builtin']
    executable = 'hostname'
    num_tasks = 4
    num_tasks_per_node = 1

    @run_before('run')
    def pre_launch(self):
        cmd = self.job.launcher.run_command(self.job)
        self.prerun_cmds = [
            f'{cmd} -n {n} {self.executable}'
            for n in range(1, self.num_tasks)
        ]

    @sanity_function
    def validate_test(self):
        return sn.assert_eq(
            sn.count(sn.extractall(r'^nid\d+', self.stdout)), 10
        )

The additional parallel launch commands are inserted in either the prerun_cmds or postrun_cmds lists. To retrieve the actual parallel launch command for the current partition that the test is running on, you can use the run_command() method of the launcher object. Let’s see how the generated job script looks like:

./bin/reframe -c tutorials/advanced/multilaunch/multilaunch.py -r
cat output/daint/gpu/builtin/MultiLaunchTest/rfm_MultiLaunchTest_job.sh
#!/bin/bash
#SBATCH --job-name="rfm_MultiLaunchTest_job"
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=1
#SBATCH --output=rfm_MultiLaunchTest_job.out
#SBATCH --error=rfm_MultiLaunchTest_job.err
#SBATCH --time=0:10:0
#SBATCH -A csstaff
#SBATCH --constraint=gpu
srun -n 1 hostname
srun -n 2 hostname
srun -n 3 hostname
srun hostname

The first three srun commands are emitted through the prerun_cmds whereas the last one comes from the test’s executable attribute.

Flexible Regression Tests

New in version 2.15.

ReFrame can automatically set the number of tasks of a particular test, if its num_tasks attribute is set to a negative value or zero. In ReFrame’s terminology, such tests are called flexible. Negative values indicate the minimum number of tasks that are acceptable for this test (a value of -4 indicates that at least 4 tasks are required). A zero value indicates the default minimum number of tasks which is equal to num_tasks_per_node.

By default, ReFrame will spawn such a test on all the idle nodes of the current system partition, but this behavior can be adjusted with the --flex-alloc-nodes command-line option. Flexible tests are very useful for diagnostics tests, e.g., tests for checking the health of a whole set nodes. In this example, we demonstrate this feature through a simple test that runs hostname. The test will verify that all the nodes print the expected host name:

cat tutorials/advanced/flexnodes/flextest.py
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class HostnameCheck(rfm.RunOnlyRegressionTest):
    valid_systems = ['daint:gpu', 'daint:mc']
    valid_prog_environs = ['cray']
    executable = 'hostname'
    num_tasks = 0
    num_tasks_per_node = 1

    @sanity_function
    def validate_test(self):
        return sn.assert_eq(
            self.num_tasks,
            sn.count(sn.findall(r'^nid\d+$', self.stdout))
        )

The first thing to notice in this test is that num_tasks is set to zero as default, which is a requirement for flexible tests. However, with flexible tests, this value is updated right after the job completes to the actual number of tasks that were used. Consequently, this allows the sanity function of the test to assert that the number host names printed matches num_tasks.

Tip

If you want to run multiple flexible tests at once, it’s better to run them using the serial execution policy, because the first test might take all the available nodes and will cause the rest to fail immediately, since there will be no available nodes for them.

Testing containerized applications

New in version 2.20.

ReFrame can be used also to test applications that run inside a container. First, we need to enable the container platform support in ReFrame’s configuration and, specifically, at the partition configuration level:

                {
                    'name': 'gpu',
                    'descr': 'Hybrid nodes',
                    'scheduler': 'slurm',
                    'launcher': 'srun',
                    'access': ['-C gpu', '-A csstaff'],
                    'environs': ['gnu', 'intel', 'pgi', 'cray'],
                    'max_jobs': 100,
                    'resources': [
                        {
                            'name': 'memory',
                            'options': ['--mem={size}']
                        }
                    ],
                    'container_platforms': [
                        {
                            'type': 'Sarus',
                            'modules': ['sarus']
                        },
                        {
                            'type': 'Singularity',
                            'modules': ['singularity']
                        }
                    ]
                },

For each partition, users can define a list of container platforms supported using the container_platforms configuration parameter. In this case, we define the Sarus platform for which we set the modules parameter in order to instruct ReFrame to load the sarus module, whenever it needs to run with this container platform. Similarly, we add an entry for the Singularity platform.

The following parameterized test, will create two tests, one for each of the supported container platforms:

cat tutorials/advanced/containers/container_test.py
import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class ContainerTest(rfm.RunOnlyRegressionTest):
    platform = parameter(['Sarus', 'Singularity'])
    valid_systems = ['daint:gpu']
    valid_prog_environs = ['builtin']

    @run_before('run')
    def set_container_variables(self):
        self.descr = f'Run commands inside a container using {self.platform}'
        image_prefix = 'docker://' if self.platform == 'Singularity' else ''
        self.container_platform = self.platform
        self.container_platform.image = f'{image_prefix}ubuntu:18.04'
        self.container_platform.command = (
            "bash -c 'cat /etc/os-release | tee /rfm_workdir/release.txt'"
        )

    # rfmdocstart: assert_release
    @sanity_function
    def assert_release(self):
        os_release_pattern = r'18.04.\d+ LTS \(Bionic Beaver\)'
        return sn.assert_found(os_release_pattern, 'release.txt')
    # rfmdocend: assert_release

A container-based test can be written as RunOnlyRegressionTest that sets the container_platform attribute. This attribute accepts a string that corresponds to the name of the container platform that will be used to run the container for this test. If such a platform is not configured for the current system, the test will fail.

As soon as the container platform to be used is defined, you need to specify the container image to use by setting the image. In the Singularity test variant, we add the docker:// prefix to the image name, in order to instruct Singularity to pull the image from DockerHub. The default command that the container runs can be overwritten by setting the command attribute of the container platform.

The image is the only mandatory attribute for container-based checks. It is important to note that the executable and executable_opts attributes of the actual test are ignored in case of container-based tests.

ReFrame will run the container according to the given platform as follows:

# Sarus
sarus run --mount=type=bind,source="/path/to/test/stagedir",destination="/rfm_workdir" ubuntu:18.04 bash -c 'cat /etc/os-release | tee /rfm_workdir/release.txt'

# Singularity
singularity exec -B"/path/to/test/stagedir:/rfm_workdir" docker://ubuntu:18.04 bash -c 'cat /etc/os-release | tee /rfm_workdir/release.txt'

In the Sarus case, ReFrame will prepend the following command in order to pull the container image before running the container:

sarus pull ubuntu:18.04

This is the default behavior of ReFrame, which can be changed if pulling the image is not desired by setting the pull_image attribute to False. By default ReFrame will mount the stage directory of the test under /rfm_workdir inside the container. Once the commands are executed, the container is stopped and ReFrame goes on with the sanity and performance checks. Besides the stage directory, additional mount points can be specified through the mount_points attribute:

self.container_platform.mount_points = [('/path/to/host/dir1', '/path/to/container/mount_point1'),
                                        ('/path/to/host/dir2', '/path/to/container/mount_point2')]

The container filesystem is ephemeral, therefore, ReFrame mounts the stage directory under /rfm_workdir inside the container where the user can copy artifacts as needed. These artifacts will therefore be available inside the stage directory after the container execution finishes. This is very useful if the artifacts are needed for the sanity or performance checks. If the copy is not performed by the default container command, the user can override this command by settings the command attribute such as to include the appropriate copy commands. In the current test, the output of the cat /etc/os-release is available both in the standard output as well as in the release.txt file, since we have used the command:

bash -c 'cat /etc/os-release | tee /rfm_workdir/release.txt'

and /rfm_workdir corresponds to the stage directory on the host system. Therefore, the release.txt file can now be used in the subsequent sanity checks:

    @sanity_function
    def assert_release(self):
        os_release_pattern = r'18.04.\d+ LTS \(Bionic Beaver\)'
        return sn.assert_found(os_release_pattern, 'release.txt')

For a complete list of the available attributes of a specific container platform, please have a look at the Container Platforms section of the Regression Test API guide. On how to configure ReFrame for running containerized tests, please have a look at the Container Platform Configuration section of the Configuration Reference.

Writing reusable tests

New in version 3.5.0.

So far, all the examples shown above were tight to a particular system or configuration, which makes reusing these tests in other systems not straightforward. However, the introduction of the parameter() and variable() ReFrame built-ins solves this problem, eliminating the need to specify any of the test variables in the __init__() method and simplifying code reuse. Hence, readers who are not familiar with these built-in functions are encouraged to read their basic use examples (see parameter() and variable()) before delving any deeper into this tutorial.

In essence, parameters and variables can be treated as simple class attributes, which allows us to leverage Python’s class inheritance and write more modular tests. For simplicity, we illustrate this concept with the above ContainerTest example, where the goal here is to re-write this test as a library that users can simply import from and derive their tests without having to rewrite the bulk of the test. Also, for illustrative purposes, we parameterize this library test on a few different image tags (the above example just used ubuntu:18.04) and throw the container commands into a separate bash script just to create some source files. Thus, removing all the system and configuration specific variables, and moving as many assignments as possible into the class body, the system agnostic library test looks as follows:

cat tutorials/advanced/library/lib/__init__.py
import reframe as rfm
import reframe.utility.sanity as sn


class ContainerBase(rfm.RunOnlyRegressionTest, pin_prefix=True):
    '''Test that asserts the ubuntu version of the image.'''

    # Derived tests must override this parameter
    platform = parameter()
    image_prefix = variable(str, value='')

    # Parametrize the test on two different versions of ubuntu.
    dist = parameter(['18.04', '20.04'])
    dist_name = variable(dict, value={
        '18.04': 'Bionic Beaver',
        '20.04': 'Focal Fossa',
    })

    @run_after('setup')
    def set_description(self):
        self.descr = (
            f'Run commands inside a container using ubuntu {self.dist}'
        )

    @run_before('run')
    def set_container_platform(self):
        self.container_platform = self.platform
        self.container_platform.image = (
            f'{self.image_prefix}ubuntu:{self.dist}'
        )
        self.container_platform.command = (
            "bash -c /rfm_workdir/get_os_release.sh"
        )

    @property
    def os_release_pattern(self):
        name = self.dist_name[self.dist]
        return rf'{self.dist}.\d+ LTS \({name}\)'

    @sanity_function
    def assert_release(self):
        return sn.all([
            sn.assert_found(self.os_release_pattern, 'release.txt'),
            sn.assert_found(self.os_release_pattern, self.stdout)
        ])

Note that the class ContainerBase is not decorated since it does not specify the required variables valid_systems and valid_prog_environs, and it declares the platform parameter without any defined values assigned. Hence, the user can simply derive from this test and specialize it to use the desired container platforms. Since the parameters are defined directly in the class body, the user is also free to override or extend any of the other parameters in a derived test. In this example, we have parameterized the base test to run with the ubuntu:18.04 and ubuntu:20.04 images, but these values from dist (and also the dist_name variable) could be modified by the derived class if needed.

On the other hand, the rest of the test depends on the values from the test parameters, and a parameter is only assigned a specific value after the class has been instantiated. Thus, the rest of the test is expressed as hooks, without the need to write anything in the __init__() method. In fact, writing the test in this way permits having hooks that depend on undefined variables or parameters. This is the case with the set_container_platform() hook, which depends on the undefined parameter platform. Hence, the derived test must define all the required parameters and variables; otherwise ReFrame will notice that the test is not well defined and will raise an error accordingly.

Before moving ahead with the derived test, note that the ContainerBase class takes the additional argument pin_prefix=True, which locks the prefix of all derived tests to this base test. This will allow the retrieval of the sources located in the library by any derived test, regardless of what their containing directory is.

cat tutorials/advanced/library/lib/src/get_os_release.sh
#!/bin/bash
cat /etc/os-release | tee /rfm_workdir/release.txt

Now from the user’s perspective, the only thing to do is to import the above base test and specify the required variables and parameters. For consistency with the above example, we set the platform parameter to use Sarus and Singularity, and we configure the test to run on Piz Daint with the built-in programming environment. Hence, the above ContainerTest is now reduced to the following:

cat tutorials/advanced/library/usr/container_test.py
import reframe as rfm
import tutorials.advanced.library.lib as lib


@rfm.simple_test
class ContainerTest(lib.ContainerBase):
    platform = parameter(['Sarus', 'Singularity'])
    valid_systems = ['daint:gpu']
    valid_prog_environs = ['builtin']

    @run_after('setup')
    def set_image_prefix(self):
        if self.platform == 'Singularity':
            self.image_prefix = 'docker://'

In a similar fashion, any other user could reuse the above ContainerBase class and write the test for their own system with a few lines of code.

Happy test sharing!