Regression Test API¶

This page provides a reference guide of the ReFrame API for writing regression tests covering all the relevant details. Internal data structures and APIs are covered only to the extent that this might be helpful to the final user of the framework.

Test Base Classes¶

class reframe.core.pipeline.CompileOnlyRegressionTest(*args, **kwargs)[source]¶

Bases: reframe.core.pipeline.RegressionTest

Base class for compile-only regression tests.

These tests are by default local and will skip the run phase of the regression test pipeline.

The standard output and standard error of the test will be set to those of the compilation stage.

This class is also directly available under the top-level reframe module.

run()[source]¶

The run stage of the regression test pipeline.

Implemented as no-op.

run_wait()[source]¶

Wait for this test to finish.

Implemented as no-op

setup(partition, environ, **job_opts)[source]¶

The setup stage of the regression test pipeline.

Similar to the RegressionTest.setup(), except that no run job is created for this test.

property stderr¶

The name of the file containing the standard error of the test.

This is set during the setup() phase.

This attribute is evaluated lazily, so it can by used inside sanity expressions.

Type: str or None if a run job has not yet been created.

property stdout¶

The name of the file containing the standard output of the test.

This is set during the setup() phase.

This attribute is evaluated lazily, so it can by used inside sanity expressions.

Type: str or None if a run job has not yet been created.

reframe.core.pipeline.DEPEND_BY_ENV = 2¶: Constant to be passed as the how argument of the RegressionTest.depends_on() method. It denotes that the test cases of the current test will depend only on the corresponding test cases of the target test that use the same programming environment.

This constant is directly available under the reframe module.

Deprecated since version 3.3: Please use a callable as the how argument.

reframe.core.pipeline.DEPEND_EXACT = 1¶: Constant to be passed as the how argument of the depends_on() method. It denotes that test case dependencies will be explicitly specified by the user.

This constant is directly available under the reframe module.

Deprecated since version 3.3: Please use a callable as the how argument.

reframe.core.pipeline.DEPEND_FULLY = 3¶: Constant to be passed as the how argument of the RegressionTest.depends_on() method. It denotes that each test case of this test depends on all the test cases of the target test.

This constant is directly available under the reframe module.

Deprecated since version 3.3: Please use a callable as the how argument.

class reframe.core.pipeline.RegressionMixin(*args, **kwargs)[source]¶

Bases: object

Base mixin class for regression tests.

Multiple inheritance from more than one RegressionTest class is not allowed in ReFrame. Hence, mixin classes provide the flexibility to bundle reusable test add-ons, leveraging the metaclass magic implemented in RegressionTestMeta. Using this metaclass allows mixin classes to use powerful ReFrame features, such as hooks, parameters or variables.

New in version 3.4.2.

class reframe.core.pipeline.RegressionTest(*args, **kwargs)[source]¶

Bases: reframe.core.pipeline.RegressionMixin, reframe.utility.jsonext.JSONSerializable

Base class for regression tests.

All regression tests must eventually inherit from this class. This class provides the implementation of the pipeline phases that the regression test goes through during its lifetime.

This class accepts parameters at the class definition, i.e., the test class can be defined as follows:

class MyTest(RegressionTest, param='foo', ...):

where param is one of the following:

Parameters

pin_prefix – lock the test prefix to the directory where the current class lives.
require_version –
a list of ReFrame version specifications that this test is allowed to run. A version specification string can have one of the following formats:
- VERSION: Specifies a single version.
- {OP}VERSION, where {OP} can be any of >, >=, <, <=, == and !=. For example, the version specification string '>=3.5.0' will allow the following test to be loaded only by ReFrame 3.5.0 and higher. The ==VERSION specification is the equivalent of VERSION.
- V1..V2: Specifies a range of versions.
The test will be selected if any of the versions is satisfied, even if the versions specifications are conflicting.
special – allow pipeline stage methods to be overriden in this class.

Note

Changed in version 2.19: Base constructor takes no arguments.

New in version 3.3: The pin_prefix class definition parameter is added.

New in version 3.7.0: The require_verion class definition parameter is added.

Warning

Changed in version 3.4.2: Multiple inheritance with a shared common ancestor is not allowed.

build_locally = True¶

New in version 3.3.

Always build the source code for this test locally. If set to False, ReFrame will spawn a build job on the partition where the test will run. Setting this to False is useful when cross-compilation is not supported on the system where ReFrame is run. Normally, ReFrame will mark the test as a failure if the spawned job exits with a non-zero exit code. However, certain scheduler backends, such as the squeue do not set it. In such cases, it is the user’s responsibility to check whether the build phase failed by adding an appropriate sanity check.

Type: boolean : :default: True

build_system = None¶

New in version 2.14.

The build system to be used for this test. If not specified, the framework will try to figure it out automatically based on the value of sourcepath.

This field may be set using either a string referring to a concrete build system class name (see build systems) or an instance of reframe.core.buildsystems.BuildSystem. The former is the recommended way.

Type: str or reframe.core.buildsystems.BuildSystem.
Default: None.

build_time_limit = None¶

New in version 3.5.1.

The time limit for the build job of the regression test.

It is specified similarly to the time_limit attribute.

Type: str or float or int
Default: None

check_performance()[source]¶

The performance checking phase of the regression test pipeline.

Raises: reframe.core.exceptions.SanityError – If the performance check fails.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

Changed in version 3.4: Overriding this method directly in no longer allowed. See here for more details.

check_sanity()[source]¶

The sanity checking phase of the regression test pipeline.

Raises

reframe.core.exceptions.SanityError – If the sanity check fails.
reframe.core.exceptions.ReframeSyntaxError – If the sanity function cannot be resolved due to ambiguous syntax.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

Changed in version 3.4: Overriding this method directly in no longer allowed. See here for more details.

cleanup(remove_files=False)[source]¶

The cleanup phase of the regression test pipeline.

Parameters: remove_files – If True, the stage directory associated with this test will be removed.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

Changed in version 3.4: Overriding this method directly in no longer allowed. See here for more details.

compile()[source]¶

The compilation phase of the regression test pipeline.

Raises: reframe.core.exceptions.ReframeError – In case of errors.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

Changed in version 3.4: Overriding this method directly in no longer allowed. See here for more details.

compile_wait()[source]¶: Wait for compilation phase to finish.

New in version 2.13.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

Changed in version 3.4: Overriding this method directly in no longer allowed. See here for more details.

container_platform = None¶

New in version 2.20.

The container platform to be used for launching this test.

If this field is set, the test will run inside a container using the specified container runtime. Container-specific options must be defined additionally after this field is set:

self.container_platform = 'Singularity'
self.container_platform.image = 'docker://ubuntu:18.04'
self.container_platform.commands = ['cat /etc/os-release']

If this field is set, executable and executable_opts attributes are ignored. The container platform’s commands will be used instead.

Type: str or reframe.core.containers.ContainerPlatform.
Default: None.

property current_environ¶

The programming environment that the regression test is currently executing with.

This is set by the framework during the setup() phase.

Type: reframe.core.environments.ProgEnvironment.

property current_partition¶

The system partition the regression test is currently executing on.

This is set by the framework during the setup() phase.

Type: reframe.core.systems.SystemPartition.

property current_system¶

The system the regression test is currently executing on.

This is set by the framework during the initialization phase.

Type: reframe.core.systems.System.

depends_on(target, how=None, *args, **kwargs)[source]¶

Add a dependency to another test.

Parameters

target – The name of the test that this one will depend on.
how –
A callable that defines how the test cases of this test depend on the the test cases of the target test. This callable should accept two arguments:
- The source test case (i.e., a test case of this test) represented as a two-element tuple containing the names of the partition and the environment of the current test case.
- Test destination test case (i.e., a test case of the target test) represented as a two-element tuple containing the names of the partition and the environment of the current target test case.
It should return True if a dependency between the source and destination test cases exists, False otherwise.

This function will be called multiple times by the framework when the test DAG is constructed, in order to determine the connectivity of the two tests.

In the following example, this test depends on T1 when their partitions match, otherwise their test cases are independent.
```
def by_part(src, dst):
    p0, _ = src
    p1, _  = dst
    return p0 == p1

self.depends_on('T0', how=by_part)
```
The framework offers already a set of predefined relations between the test cases of inter-dependent tests. See the reframe.utility.udeps for more details.

The default how function is reframe.utility.udeps.by_case(), where test cases on different partitions and environments are independent.

Test Decorators¶

@reframe.core.decorators.parameterized_test(*inst)[source]¶

Class decorator for registering multiple instantiations of a test class.

The decorated class must derive from reframe.core.pipeline.RegressionTest. This decorator is also available directly under the reframe module.

Parameters: inst – The different instantiations of the test. Each instantiation argument may be either a sequence or a mapping.

New in version 2.13.

Note

This decorator does not instantiate any test. It only registers them. The actual instantiation happens during the loading phase of the test.

Deprecated since version 3.6.0: Please use the parameter() built-in instead.

@reframe.core.decorators.required_version(*versions)[source]¶

Class decorator for specifying the required ReFrame versions for the following test.

If the test is not compatible with the current ReFrame version it will be skipped.

Parameters

versions –

A list of ReFrame version specifications that this test is allowed to run. A version specification string can have one of the following formats:

VERSION: Specifies a single version.
{OP}VERSION, where {OP} can be any of >, >=, <, <=, == and !=. For example, the version specification string '>=3.5.0' will allow the following test to be loaded only by ReFrame 3.5.0 and higher. The ==VERSION specification is the equivalent of VERSION.
V1..V2: Specifies a range of versions.

You can specify multiple versions with this decorator, such as @required_version('3.5.1', '>=3.5.6'), in which case the test will be selected if any of the versions is satisfied, even if the versions specifications are conflicting.

New in version 2.13.

Changed in version 3.5.0: Passing ReFrame version numbers that do not comply with the semantic versioning specification is deprecated. Examples of non-compliant version numbers are 3.5 and 3.5-dev0. These should be written as 3.5.0 and 3.5.0-dev.0.

Deprecated since version 3.5.0: Please set the require_version parameter in the class definition instead.

@reframe.core.decorators.simple_test[source]¶

Class decorator for registering tests with ReFrame.

The decorated class must derive from reframe.core.pipeline.RegressionTest. This decorator is also available directly under the reframe module.

New in version 2.13.

Built-in types¶

New in version 3.4.2.

ReFrame provides built-in types which facilitate the process of writing extensible regression tests (i.e. a test library). These builtins are only available when used directly in the class body of classes derived from any of the Test Base Classes. Through builtins, ReFrame internals are able to pre-process and validate the test input before the actual test creation takes place. This provides the ReFrame internals with further control over the user’s input, making the process of writing regression tests less error-prone. In essence, these builtins exert control over the test creation, and they allow adding and/or modifying certain attributes of the regression test.

Note

The built-in types described below can only be used to declare class variables and must never be part of any container type. Ignoring this restriction will result in undefined behavior.

class MyTest(rfm.RegressionMixin):
    p0 = parameter([1, 2])   # Correct
    p1 = [parameter([1, 2])] # Undefined behavior

RegressionMixin.parameter(values=None, inherit_params=False, filter_params=None)¶

Inserts or modifies a regression test parameter. At the class level, these parameters are stored in a separate namespace referred to as the parameter space. If a parameter with a matching name is already present in the parameter space of a parent class, the existing parameter values will be combined with those provided by this method following the inheritance behavior set by the arguments inherit_params and filter_params. Instead, if no parameter with a matching name exists in any of the parent parameter spaces, a new regression test parameter is created. A regression test can be parameterized as follows:

class Foo(rfm.RegressionTest):
    variant = parameter(['A', 'B'])
    # print(variant) # Error: a parameter may only be accessed from the class instance.

    @run_after('init')
    def do_something(self):
        if self.variant == 'A':
            do_this()
        else:
            do_other()

One of the most powerful features of these built-in functions is that they store their input information at the class level. However, a parameter may only be accessed from the class instance and accessing it directly from the class body is disallowed. With this approach, extending or specializing an existing parameterized regression test becomes straightforward, since the test attribute additions and modifications made through built-in functions in the parent class are automatically inherited by the child test. For instance, continuing with the example above, one could override the do_something() hook in the Foo regression test as follows:

class Bar(Foo):
    @run_after('init')
    def do_something(self):
        if self.variant == 'A':
            override_this()
        else:
            override_other()

Moreover, a derived class may extend, partially extend and/or modify the parameter values provided in the base class as shown below.

class ExtendVariant(Bar):
    # Extend the full set of inherited variant parameter values to ['A', 'B', 'C']
    variant = parameter(['C'], inherit_params=True)

class PartiallyExtendVariant(Bar):
    # Extend a subset of the inherited variant parameter values to ['A', 'D']
    variant = parameter(['D'], inherit_params=True,
                        filter_params=lambda x: x[:1])

class ModifyVariant(Bar):
    # Modify the variant parameter values to ['AA', 'BA']
    variant = parameter(inherit_params=True,
                        filter_params=lambda x: map(lambda y: y+'A', x))

A parameter with no values is referred to as an abstract parameter (i.e. a parameter that is declared but not defined). Therefore, classes with at least one abstract parameter are considered abstract classes.

class AbstractA(Bar):
    variant = parameter()

class AbstractB(Bar):
    variant = parameter(inherit_params=True, filter_params=lambda x: [])

Parameters

values – An iterable containing the parameter values.
inherit_params – If True, the parameter values defined in any base class will be inherited. In this case, the parameter values provided in the current class will extend the set of inherited parameter values. If the parameter does not exist in any of the parent parameter spaces, this option has no effect.
filter_params – Function to filter/modify the inherited parameter values that may have been provided in any of the parent parameter spaces. This function must accept a single iterable argument and return an iterable. It will be called with the inherited parameter values and it must return the filtered set of parameter values. This function will only have an effect if used with inherit_params=True.

RegressionMixin.variable(*types, value=None, field=None, **kwargs)¶

Inserts a new regression test variable. Declaring a test variable through the variable() built-in allows for a more robust test implementation than if the variables were just defined as regular test attributes (e.g. self.a = 10). Using variables declared through the variable() built-in guarantees that these regression test variables will not be redeclared by any child class, while also ensuring that any values that may be assigned to such variables comply with its original declaration. In essence, declaring test variables with the variable() built-in removes any potential test errors that might be caused by accidentally overriding a class attribute. See the example below.

class Foo(rfm.RegressionTest):
    my_var = variable(int, value=8)
    not_a_var = my_var - 4

    @run_after('init')
    def access_vars(self):
        print(self.my_var) # prints 8.
        # self.my_var = 'override' # Error: my_var must be an int!
        self.not_a_var = 'override' # However, this would work. Dangerous!
        self.my_var = 10 # tests may also assign values the standard way

Here, the argument value in the variable() built-in sets the default value for the variable. This value may be accessed directly from the class body, as long as it was assigned before either in the same class body or in the class body of a parent class. This behavior extends the standard Python data model, where a regular class attribute from a parent class is never available in the class body of a child class. Hence, using the variable() built-in enables us to directly use or modify any variables that may have been declared upstream the class inheritance chain, without altering their original value at the parent class level.

class Bar(Foo):
    print(my_var) # prints 8
    # print(not_a_var) # This is standard Python and raises a NameError

    # Since my_var is available, we can also update its value:
    my_var = 4

    # Bar inherits the full declaration of my_var with the original type-checking.
    # my_var = 'override' # Wrong type error again!

    @run_after('init')
    def access_vars(self):
        print(self.my_var) # prints 4
        print(self.not_a_var) # prints 4


print(Foo.my_var) # prints 8
print(Bar.my_var) # prints 4

Here, Bar inherits the variables from Foo and can see that my_var has already been declared in the parent class. Therefore, the value of my_var is updated ensuring that the new value complies to the original variable declaration. However, the value of my_var at Foo remains unchanged.

These examples above assumed that a default value can be provided to the variables in the bases tests, but that might not always be the case. For example, when writing a test library, one might want to leave some variables undefined and force the user to set these when using the test. As shown in the example below, imposing such requirement is as simple as not passing any value to the variable() built-in, which marks the given variable as required.

# Test as written in the library
class EchoBaseTest(rfm.RunOnlyRegressionTest):
  what = variable(str)

  valid_systems = ['*']
  valid_prog_environs = ['*']

  @run_before('run')
  def set_executable(self):
      self.executable = f'echo {self.what}'

  @sanity_function
  def assert_what(self):
      return sn.assert_found(fr'{self.what}')


# Test as written by the user
@rfm.simple_test
class HelloTest(EchoBaseTest):
  what = 'Hello'


# A parameterized test with type-checking
@rfm.simple_test
class FoodTest(EchoBaseTest):
  param = parameter(['Bacon', 'Eggs'])

  @run_after('init')
  def set_vars_with_params(self):
    self.what = self.param

Similarly to a variable with a value already assigned to it, the value of a required variable may be set either directly in the class body, on the __init__() method, or in any other hook before it is referenced. Otherwise an error will be raised indicating that a required variable has not been set. Conversely, a variable with a default value already assigned to it can be made required by assigning it the required keyword. However, this required keyword is only available in the class body.

class MyRequiredTest(HelloTest):
  what = required

Running the above test will cause the set_exec_and_sanity() hook from EchoBaseTest to throw an error indicating that the variable what has not been set.

Parameters

*types – the supported types for the variable.
value – the default value assigned to the variable. If no value is provided, the variable is set as required.
field – the field validator to be used for this variable. If no field argument is provided, it defaults to reframe.core.fields.TypedField. The provided field validator by this argument must derive from reframe.core.fields.Field.
**kwargs – kwargs to be forwarded to the constructor of the field validator.

RegressionMixin.fixture(cls, *, scope='test', action='fork', variants='all', variables=None)¶

Declare a new fixture in the current regression test. A fixture is a regression test that creates, prepares and/or manages a resource for another regression test. Fixtures may contain other fixtures and so on, forming a directed acyclic graph. A parent fixture (or a regular regression test) requires the resources managed by its child fixtures in order to run, and it may only access these fixture resources after its setup pipeline stage. The execution of parent fixtures is postponed until all their respective children have completed execution. However, the destruction of the resources managed by a fixture occurs in reverse order, only after all the parent fixtures have been destroyed. This destruction of resources takes place during the cleanup pipeline stage of the regression test. Fixtures must not define the members valid_systems and valid_prog_environs. These variables are defined based on the values specified in the parent test, ensuring that the fixture runs with a suitable system partition and programming environment combination. A fixture’s name attribute may be internally mangled depending on the arguments passed during the fixture declaration. Hence, manually setting or modifying the name attribute in the fixture class is disallowed, and breaking this restriction will result in undefined behavior.

Warning

The fixture name mangling is considered an internal framework mechanism and it may change in future versions without any notice. Users must not express any logic in their tests that relies on a given fixture name mangling scheme.

By default, the resources managed by a fixture are private to the parent test. However, it is possible to share these resources across different tests by passing the appropriate fixture scope argument. The different scope levels are independent from each other and a fixture only executes once per scope, where all the tests that belong to that same scope may use the same resources managed by a given fixture instance. The available scopes are:

session: This scope encloses all the tests and fixtures that run in the full ReFrame session. This may include tests that use different system partition and programming environment combinations. The fixture class must derive from RunOnlyRegressionTest to avoid any implicit dependencies on the partition or the programming environment used.

partition: This scope spans across a single system partition. This may include different tests that run on the same partition but use different programming environments. Fixtures with this scope must be independent of the programming environment, which restricts the fixture class to derive from RunOnlyRegressionTest.

environment: The extent of this scope covers a single combination of system partition and programming environment. Since the fixture is guaranteed to have the same partition and programming environment as the parent test, the fixture class can be any derived class from RegressionTest.

test: This scope covers a single instance of the parent test, where the resources provided by the fixture are exclusive to each parent test instance. The fixture class can be any derived class from RegressionTest.

Rather than specifying the scope at the fixture class definition, ReFrame fixtures set the scope level from the consumer side (i.e. when used by another test or fixture). A test may declare multiple fixtures using the same class, where fixtures with different scopes are guaranteed to point to different instances of the fixture class. On the other hand, when two or more fixtures use the same fixture class and have the same scope, these different fixtures will point to the same underlying resource if the fixtures refer to the same variant of the fixture class. The example below illustrates the different fixture scope usages:

class MyFixture(rfm.RunOnlyRegressionTest):
   '''Manage some resource'''
   my_var = variable(int, value=1)
   ...


@rfm.simple_test
class TestA(rfm.RegressionTest):
    valid_systems = ['p1', 'p2']
    valid_prog_environs = ['e1', 'e2']
    f1 = fixture(MyFixture, scope='session')     # Shared throughout the full session
    f2 = fixture(MyFixture, scope='partition')   # Shared for each supported partition
    f3 = fixture(MyFixture, scope='environment') # Shared for each supported part+environ
    f4 = fixture(MyFixture, scope='test')        # Private evaluation of MyFixture
    ...


@rfm.simple_test
class TestB(rfm.RegressionTest):
    valid_systems = ['p1']
    valid_prog_environs = ['e1']
    f1 = fixture(MyFixture, scope='test')        # Another private instance of MyFixture
    f2 = fixture(MyFixture, scope='environment') # Same as f3 in TestA for p1 + e1
    f3 = fixture(MyFixture, scope='session')     # Same as f1 in TestA
    ...

    @run_after('setup')
    def access_fixture_resources(self):
        '''Dummy pipeline hook to illustrate fixture resource access.'''
        assert self.f1.my_var is not self.f2.my_var
        assert self.f1.my_var is not self.f3.my_var

TestA supports two different valid systems and another two valid programming environments. Assuming that both environments are supported by each of the system partitions 'p1' and 'p2', this test will execute a total of four times. This test uses the very simple MyFixture fixture multiple times using different scopes, where fixture f1 (session scope) will be shared across the four test instances, and fixture f4 (test scope) will be executed once per test instance. On the other hand, f2 (partition scope) will run once per partition supported by test TestA, and the multiple per-partition executions (i.e. for each programming environment) will share the same underlying resource for f2. Lastly, f3 will run a total of four times, which is once per partition and environment combination. This simple TestA shows how multiple instances from the same test can share resources, but the real power behind fixtures is illustrated with TestB, where this resource sharing is extended across different tests. For simplicity, TestB only supports a single partition 'p1' and programming environment 'e1', and similarly to TestA, f1 (test scope) causes a private evaluation of the fixture MyFixture. However, the resources managed by fixtures f2 (environment scope) and f3 (session scope) are shared with Test1.

Fixtures are treated by ReFrame as first-class ReFrame tests, which means that these classes can use the same built-in functionalities as in regular tests decorated with @rfm.simple_test. This includes the parameter() built-in, where fixtures may have more than one variant. When this occurs, a parent test may select to either treat a parameterized fixture as a test parameter, or instead, to gather all the fixture variants from a single instance of the parent test. In essence, fixtures implement fork-join model whose behavior may be controlled through the action argument. This argument may be set to one of the following options:

fork: This option parameterizes the parent test as a function of the fixture variants. The fixture handle will resolve to a single instance of the fixture.

join: This option gathers all the variants from a fixture into a single instance of the parent test. The fixture handle will point to a list containing all the fixture variants.

A test may declare multiple fixtures with different action options, where the default action option is 'fork'. The example below illustrates the behavior of these two different options.

class ParamFix(rfm.RegressionTest):
    '''Manage some resource'''
    p = parameter(range(5)) # A simple test parameter
    ...


@rfm.simple_test
class TestC(rfm.RegressionTest):
    # Parameterize TestC for each ParamFix variant
    f = fixture(ParamFix, action='fork')
    ...

    @run_after('setup')
    def access_fixture_resources(self):
        print(self.f.p) # Prints the fixture's variant parameter value


@rfm.simple_test
class TestD(rfm.RegressionTest):
    # Gather all fixture variants into a single test
    f = fixture(ParamFix, action='join')
    ...

    @run_after('setup')
    def reduce_range(self):
        '''Sum all the values of p for each fixture variant'''
        res = functools.reduce(lambda x, y: x+y, (fix.p for fix in self.f))
        n = len(self.f)-1
        assert res == (n*n + n)/2

Here ParamFix is a simple fixture class with a single parameter. When the test TestC uses this fixture with a 'fork' action, the test is implicitly parameterized over each variant of ParamFix. Hence, when the access_fixture_resources() post-setup hook accesses the fixture f, it only access a single instance of the ParamFix fixture. On the other hand, when this same fixture is used with a 'join' action by TestD, the test is not parameterized and all the ParamFix instances are gathered into f as a list. Thus, the post-setup pipeline hook reduce_range() can access all the fixture variants and compute a reduction of the different p values.

When declaring a fixture, a parent test may select a subset of the fixture variants through the variants argument. This variant selection can be done by either passing an iterable containing valid variant indices (see Test variants for further information on how the test variants are indexed), or instead, passing a mapping with the parameter name (of the fixture class) as keys and filtering functions as values. These filtering functions are unary functions that return the value of a boolean expression on the values of the specified parameter, and they all must evaluate to True for at least one of the fixture class variants. See the example below for an illustration on how to filter-out fixture variants.

class ComplexFixture(rfm.RegressionTest):
    # A fixture with 400 different variants.
    p0 = parameter(range(100))
    p1 = parameter(['a', 'b', 'c', 'd'])
    ...

@rfm.simple_test
class TestE(rfm.RegressionTest):
    # Select the fixture variants with boolean conditions
    foo = fixture(ComplexFixture,
                  variants={'p0': lambda x: x<10, 'p1': lambda x: x=='d'})

    # Select the fixture variants by index
    bar = fixture(ComplexFixture, variants=range(300,310))
    ...

A parent test may also specify the value of different variables in the fixture class to be set before its instantiation. Each variable must have been declared in the fixture class with the variable() built-in, otherwise it is silently ignored. This variable specification is equivalent to deriving a new class from the fixture class, and setting these variable values in the class body of a newly derived class. Therefore, when fixture declarations use the same fixture class and pass different values to the variables argument, the fixture class is interpreted as a different class for each of these fixture declarations. See the example below.

class Fixture(rfm.RegressionTest):
    v = variable(int, value=1)
    ...

@rfm.simple_test
class TestF(rfm.RegressionTest):
    foo = fixture(Fixture)
    bar = fixture(Fixture, variables={'v':5})
    baz = fixture(Fixture, variables={'v':10})
    ...

    @run_after('setup')
    def print_fixture_variables(self):
        print(self.foo.v) # Prints 1
        print(self.bar.v) # Prints 5
        print(self.baz.v) # Prints 10

The test TestF declares the fixtures foo, bar and baz using the same Fixture class. If no variables were set in bar and baz, this would result into the same fixture being declared multiple times in the same scope (implicitly set to 'test'), which would lead to a single instance of Fixture being referred to by foo, bar and baz. However, in this case ReFrame identifies that the declared fixtures pass different values to the variables argument in the fixture declaration, and executes these three fixtures separately.

Note

Mappings passed to the variables argument that define the same class variables in different order are interpreted as the same value. The two fixture declarations below are equivalent, and both foo and bar will point to the same instance of the fixture class MyResource.

foo = fixture(MyResource, variables={'a':1, 'b':2})
bar = fixture(MyResource, variables={'b':2, 'a':1})

Parameters

cls – A class derived from RegressionTest that manages a given resource. The base from this class may be further restricted to other derived classes of RegressionTest depending on the scope parameter.
scope – Sets the extent to which other regression tests may share the resources managed by a fixture. The available scopes are, from more to less restrictive, 'test', 'environment', 'partition' and 'session'. By default a fixture’s scope is set to 'test', which makes the resource private to the test that uses the fixture. This means that when multiple regression tests use the same fixture class with a 'test' scope, the fixture will run once per regression test. When the scope is set to 'environment', the resources managed by the fixture are shared across all the tests that use the fixture and run on the same system partition and use the same programming environment. When the scope is set to 'partition', the resources managed by the fixture are shared instead across all the tests that use the fixture and run on the same system partition. Lastly, when the scope is set to 'session', the resources managed by the fixture are shared across the full ReFrame session. Fixtures with either 'partition' or 'session' scopes may be shared across different regression tests under different programming environments, and for this reason, when using these two scopes, the fixture class cls is required to derive from RunOnlyRegressionTest.
action – Set the behavior of a parameterized fixture to either 'fork' or 'join'. With a 'fork' action, a parameterized fixture effectively parameterizes the regression test. On the other hand, a 'join' action gathers all the fixture variants into the same instance of the regression test. By default, the action parameter is set to 'fork'.
variants – Filter or sub-select a subset of the variants from a parameterized fixture. This argument can be either an iterable with the indices from the desired variants, or a mapping containing unary functions that return the value of a boolean expression on the values of a given parameter.
variables – Mapping to set the values of fixture’s variables. The variables are set after the fixture class has been created (i.e. after the class body has executed) and before the fixture class is instantiated.

New in version 3.9.0.

Built-in functions¶

ReFrame provides the following built-in functions, which are only available in the class body of classes deriving from RegressionMixin.

@RegressionMixin.sanity_function(func)¶

Decorate a member function as the sanity function of the test.

This decorator will convert the given function into a deferrable() and mark it to be executed during the test’s sanity stage. When this decorator is used, manually assigning a value to sanity_patterns in the test is not allowed.

Decorated functions may be overridden by derived classes, and derived classes may also decorate a different method as the test’s sanity function. Decorating multiple member functions in the same class is not allowed. However, a RegressionTest may inherit from multiple RegressionMixin classes with their own sanity functions. In this case, the derived class will follow Python’s MRO to find a suitable sanity function.

New in version 3.7.0.

@RegressionMixin.performance_function(unit, *, perf_key=None)¶

Decorate a member function as a performance function of the test.

This decorator converts the decorated method into a performance deferrable function (see “Deferrable performance functions” for more details) whose evaluation is deferred to the performance stage of the regression test. The decorated function must take a single argument without a default value (i.e. self) and any number of arguments with default values. A test may decorate multiple member functions as performance functions, where each of the decorated functions must be provided with the units of the performance quantities to be extracted from the test. These performance units must be of type str. Any performance function may be overridden in a derived class and multiple bases may define their own performance functions. In the event of a name conflict, the derived class will follow Python’s MRO to choose the appropriate performance function. However, defining more than one performance function with the same name in the same class is disallowed.

The full set of performance functions of a regression test is stored under perf_variables as key-value pairs, where, by default, the key is the name of the decorated member function, and the value is the deferred performance function itself. Optionally, the key under which a performance function is stored in perf_variables can be customised by passing the desired key as the perf_key argument to this decorator.

New in version 3.8.0.

@RegressionMixin.deferrable(func)¶

Converts the decorated method into a deferrable function.

See Deferrable Functions Reference for further information on deferrable functions.

New in version 3.7.0.

RegressionMixin.bind(func, name=None)¶

Bind a free function to a regression test.

By default, the function is bound with the same name as the free function. However, the function can be bound using a different name with the name argument.

Parameters

func – external function to be bound to a class.
name – bind the function under a different name.

New in version 3.6.2.

@RegressionMixin.require_deps(func)¶

Decorator to denote that a function will use the test dependencies.

The arguments of the decorated function must be named after the dependencies that the function intends to use. The decorator will bind the arguments to a partial realization of the getdep() function, such that conceptually the new function arguments will be the following:

new_arg = functools.partial(getdep, orig_arg_name)

The converted arguments are essentially functions accepting a single argument, which is the target test’s programming environment. Additionally, this decorator will attach the function to run after the test’s setup phase, but before any other “post-setup” pipeline hook.

Warning

Changed in version 3.7.0: Using this function from the reframe or reframe.core.decorators modules is now deprecated. You should use the built-in function described here.

Pipeline Hooks¶

ReFrame provides built-in functions that allow attaching arbitrary functions to run before and/or after a given stage of the execution pipeline. Once attached to a given stage, these functions are referred to as pipeline hooks. A hook may be attached to multiple pipeline stages and multiple hooks may also be attached to the same pipeline stage. Pipeline hooks attached to multiple stages will be executed on each pipeline stage the hook was attached to. Pipeline stages with multiple hooks attached will execute these hooks in the order in which they were attached to the given pipeline stage. A derived class will inherit all the pipeline hooks defined in its bases, except for those whose hook function is overridden by the derived class. A function that overrides a pipeline hook from any of the base classes will not be a pipeline hook unless the overriding function is explicitly reattached to any pipeline stage. In the event of a name clash arising from multiple inheritance, the inherited pipeline hook will be chosen following Python’s MRO.

A function may be attached to any of the following stages (listed in order of execution): init, setup, compile, run, sanity, performance and cleanup. The init stage refers to the test’s instantiation and it runs before entering the execution pipeline. Therefore, a test function cannot be attached to run before the init stage. Hooks attached to any other stage will run exactly before or after this stage executes. So although a “post-init” and a “pre-setup” hook will both run after a test has been initialized and before the test goes through the first pipeline stage, they will execute in different times: the post-init hook will execute right after the test is initialized. The framework will then continue with other activities and it will execute the pre-setup hook just before it schedules the test for executing its setup stage.

Pipeline hooks are executed in reverse MRO order, i.e., the hooks of the least specialized class will be executed first. In the following example, BaseTest.x() will execute before DerivedTest.y():

class BaseTest(rfm.RegressionTest):
    @run_after('setup')
    def x(self):
        '''Hook x'''

class DerivedTest(BaseTeset):
    @run_after('setup')
    def y(self):
        '''Hook y'''

Note

Pipeline hooks do not execute in the test’s stage directory. However, the test’s stagedir can be accessed by explicitly changing the working directory from within the hook function itself (see the change_dir utility for further details):

import reframe.utility.osext as osext

class MyTest(rfm.RegressionTest):
    ...
    @run_after('run')
    def my_post_run_hook(self):
        # Access the stage directory
        with osext.change_dir(self.stagedir):
            ...

Warning

Changed in version 3.7.0: Declaring pipeline hooks using the same name functions from the reframe or reframe.core.decorators modules is now deprecated. You should use the built-in functions described in this section instead.

Warning

Changed in version 3.9.2: Execution of pipeline hooks until this version was implementation-defined. In practice, hooks of a derived class were executed before those of its parents.

This version defines the execution order of hooks, which now follows a strict reverse MRO order, so that parent hooks will execute before those of derived classes. Tests that relied on the execution order of hooks might break with this change.

@RegressionMixin.run_before(stage)¶

Decorator for attaching a function to a given pipeline stage.

The function will run just before the specified pipeline stage and it cannot accept any arguments except self. This decorator can be stacked, in which case the function will be attached to multiple pipeline stages. See above for the valid stage argument values.

@RegressionMixin.run_after(stage)¶

Decorator for attaching a function to a given pipeline stage.

This is analogous to run_before(), except that the hook will execute right after the stage it was attached to. This decorator also supports 'init' as a valid stage argument, where in this case, the hook will execute right after the test is initialized (i.e. after the __init__() method is called) and before entering the test’s pipeline. In essence, a post-init hook is equivalent to defining additional __init__() functions in the test. The following code

class MyTest(rfm.RegressionTest):
  @run_after('init')
  def foo(self):
      self.x = 1

is equivalent to

class MyTest(rfm.RegressionTest):
  def __init__(self):
      self.x = 1

Changed in version 3.5.2: Add support for post-init hooks.

Test variants¶

Through the parameter() and fixture() builtins, a regression test may store multiple versions or variants of a regression test at the class level. During class creation, the test’s parameter and fixture spaces are constructed and combined, assigning a unique index to each of the available test variants. In most cases, the user does not need to be aware of all the internals related to this variant indexing, since ReFrame will run by default all the available variants for each of the registered tests. On the other hand, in more complex use cases such as setting dependencies across different test variants, or when performing some complex variant sub-selection on a fixture declaration, the user may need to access some of this low-level information related to the variant indexing. Therefore, classes that derive from the base RegressionMixin provide classmethods and properties to query these data.

Warning

When selecting test variants through their variant index, no index ordering should ever be assumed, being the user’s responsibility to ensure on each ReFrame run that the selected index corresponds to the desired parameter and/or fixture variants.

RegressionMixin.num_variants¶: Total number of unique test variants in a class.

RegressionMixin.get_variant_info(cls, variant_num, *, recurse=False, max_depth=None)¶

Get the raw variant data for a given variant index. This function returns a dictionary with the variant data such as the parameter values and the fixture variants. The parameter space information is presented in a sub-dictionary under the 'params' key, gathering all the parameter values associated with the given variant number. Similarly, the information on the test’s fixtures is gathered in another sub-dictionary under the 'fixtures' key. By default, this sub-dictionary shows a tuple for each fixture, containing the respective fixture variants associated with the given variant_num. These tuples may only contain more than one fixture variant index if the fixture was declared with a join action (see the fixture() documentation for more information). However, when recurse is set to True, each fixture entry with a single-element tuple will be expanded to show the full fixture variant information. By default, the recursion will traverse the full fixture tree, but this recursion depth can be limited with the max_depth argument. See the example below.

class Foo(rfm.RegressionTest):
    p0 = parameter(range(2))
    ...

class Bar(rfm.RegressionTest):
    p0 = parameter(range(3))
    ...

class MyTest(rfm.RegressionTest):
    p1 = parameter(['a', 'b'])
    f0 = fixture(Foo, action='fork')
    f1 = fixture(Bar, action='join')
    ...

# Get the raw info for variant 0 - without recursion
MyTest.get_variant_info(0, recursive=False)
# {
#     'params': {'p1': 'a'},
#     'fixtures': {
#         'f0': (0,),
#         'f1': (0, 1, 2,)
#     }
# }

# Get the raw info for variant 0 - show the full tree
MyTest.get_variant_info(0, recursive=True)
# {
#     'params': {'p1': 'a'},
#     'fixtures': {
#         'f0': {
#             'params': {'p0': 0},
#             'fixtures': {}
#         },
#         'f1': (0, 1, 2,)
#     }
# }

Parameters

variant_num – An integer in the range of [0, cls.num_variants).
recurse – Flag to control the recursion through the fixture space.
max_depth – Set the recursion limit. When the recurse argument is set to False, this option has no effect.

RegressionMixin.get_variant_nums(cls, **conditions)¶

Get the variant numbers that meet the specified conditions. The given conditions enable filtering the parameter space of the test. These can be specified by passing key-value pairs with the parameter name to filter and an associated callable that returns True when the filtering condition is met. Multiple conditions are supported. However, filtering the fixture space is not allowed.

# Get the variant numbers where my_param is lower than 4
cls.get_variant_nums(my_param=lambda x: x < 4)

Parameters: conditions – keyword arguments where the key is the test parameter name and the value is a unary function that evaluates a bool condition on the parameter value.

RegressionMixin.fullname(cls, variant_num=None)¶

Return the full unique name of a test for a given test variant number. If no variant_num is provided, this function returns the qualified class name.

Parameters: variant_num – An integer in the range of [0, cls.num_variants).

Environments and Systems¶

class reframe.core.environments.Environment(name, modules=None, variables=None, extras=None)[source]¶

Bases: reframe.utility.jsonext.JSONSerializable

This class abstracts away an environment to run regression tests.

It is simply a collection of modules to be loaded and environment variables to be set when this environment is loaded by the framework.

Warning

Users may not create Environment objects directly.

property extras¶

User defined properties defined in the configuration.

New in version 3.9.1.

Type: Dict[str, object]

property modules¶

The modules associated with this environment.

Type: List[str]

property modules_detailed¶

A view of the modules associated with this environment in a detailed format.

Each module is represented as a dictionary with the following attributes:

name: the name of the module.
collection: True if the module name refers to a module collection.

Type: List[Dict[str, object]]

New in version 3.3.

property name¶

The name of this environment.

Type: str

property variables¶

The environment variables associated with this environment.

Type: OrderedDict[str, str]

class reframe.core.environments.ProgEnvironment(name, modules=None, variables=None, extras=None, cc='cc', cxx='CC', ftn='ftn', nvcc='nvcc', cppflags=None, cflags=None, cxxflags=None, fflags=None, ldflags=None, **kwargs)[source]¶

Bases: reframe.core.environments.Environment

A class representing a programming environment.

This type of environment adds also properties for retrieving the compiler and compilation flags.

Warning

Users may not create ProgEnvironment objects directly.

property cc¶

The C compiler of this programming environment.

Type: str

property cflags¶

The C compiler flags of this programming environment.

Type: List[str]

property cppflags¶

The preprocessor flags of this programming environment.

Type: List[str]

property cxx¶

The C++ compiler of this programming environment.

Type: str

property cxxflags¶

The C++ compiler flags of this programming environment.

Type: List[str]

property fflags¶

The Fortran compiler flags of this programming environment.

Type: List[str]

property ftn¶

The Fortran compiler of this programming environment.

Type: str

property ldflags¶

The linker flags of this programming environment.

Type: List[str]

class reframe.core.environments._EnvironmentSnapshot(name='env_snapshot')[source]¶

Bases: reframe.core.environments.Environment

An environment snapshot.

restore()[source]¶: Restore this environment snapshot.

reframe.core.environments.snapshot()[source]¶

Create an environment snapshot

Returns: An instance of _EnvironmentSnapshot.

class reframe.core.systems.DeviceInfo(info)[source]¶

Bases: reframe.core.systems._ReadOnlyInfo, reframe.utility.jsonext.JSONSerializable

A representation of a device inside ReFrame.

You can access all the keys of the device configuration object.

New in version 3.5.0.

Warning

Users may not create DeviceInfo objects directly.

property device_type¶

The type of the device.

Type: str or None

property info¶

All the available information from the configuration.

Type: dict

property num_devices¶

Number of devices of this type.

It will return 1 if it wasn’t set in the configuration.

Type: integral

class reframe.core.systems.ProcessorInfo(info)[source]¶

Bases: reframe.core.systems._ReadOnlyInfo, reframe.utility.jsonext.JSONSerializable

A representation of a processor inside ReFrame.

You can access all the keys of the processor configuration object.

New in version 3.5.0.

Warning

Users may not create ProcessorInfo objects directly.

property info¶

All the available information from the configuration.

Type: dict

property num_cores¶

Total number of cores.

Type: integral or None

property num_cores_per_numa_node¶

Number of cores per NUMA node.

Type: integral or None

property num_cores_per_socket¶

Number of cores per socket.

Type: integral or None

property num_numa_nodes¶

Number of NUMA nodes.

Type: integral or None

class reframe.core.systems.System(name, descr, hostnames, modules_system, preload_env, prefix, outputdir, resourcesdir, stagedir, partitions)[source]¶

Bases: reframe.utility.jsonext.JSONSerializable

A representation of a system inside ReFrame.

Warning

Users may not create System objects directly.

property descr¶

The description of this system.

Type: str

property hostnames¶

The hostname patterns associated with this system.

Type: List[str]

json()[source]¶: Return a JSON object representing this system.

property modules_system¶

The modules system name associated with this system.

Type: reframe.core.modules.ModulesSystem

property name¶

The name of this system.

Type: str

property outputdir¶

The ReFrame output directory prefix associated with this system.

Type: str

property partitions¶

The system partitions associated with this system.

Type: List[SystemPartition]

property prefix¶

The ReFrame prefix associated with this system.

Type: str

property preload_environ¶

The environment to load whenever ReFrame runs on this system.

New in version 2.19.

Type: reframe.core.environments.Environment

property resourcesdir¶

Global resources directory for this system.

This directory may be used for storing large files related to regression tests. The value of this directory is controlled by the resourcesdir configuration parameter.

Type: str

property stagedir¶

The ReFrame stage directory prefix associated with this system.

Type: str

class reframe.core.systems.SystemPartition(parent, name, sched_type, launcher_type, descr, access, container_environs, resources, local_env, environs, max_jobs, prepare_cmds, processor, devices, extras)[source]¶

Bases: reframe.utility.jsonext.JSONSerializable

A representation of a system partition inside ReFrame.

Warning

Users may not create SystemPartition objects directly.

property access¶

The scheduler options for accessing this system partition.

Type: List[str]

property container_environs¶

Environments associated with the different container platforms.

Type: Dict[str, Environment]

property descr¶

The description of this partition.

Type: str

property devices¶

A list of devices in the current partition.

New in version 3.5.0.

Type: List[reframe.core.systems.DeviceInfo]

environment(name)[source]¶: Return the partition environment named name.

property environs¶

The programming environments associated with this system partition.

Type: List[ProgEnvironment]

property extras¶

User defined properties defined in the configuration.

New in version 3.5.0.

Type: Dict[str, object]

property fullname¶

Return the fully-qualified name of this partition.

The fully-qualified name is of the form <parent-system-name>:<partition-name>.

Type: str

json()[source]¶: Return a JSON object representing this system partition.

property launcher¶: See launcher_type.

Deprecated since version 3.2: Please use launcher_type instead.

property launcher_type¶

The type of the backend launcher of this partition.

New in version 3.2.

Type: a subclass of reframe.core.launchers.JobLauncher.

property local_env¶

The local environment associated with this partition.

Type: Environment

property max_jobs¶

The maximum number of concurrent jobs allowed on this partition.

Type: integral

property name¶

The name of this partition.

Type: str

property prepare_cmds¶

Commands to be emitted before loading the modules.

Type: List[str]

property processor¶

Processor information for the current partition.

New in version 3.5.0.

Type: reframe.core.systems.ProcessorInfo

property resources¶

The resources template strings associated with this partition.

This is a dictionary, where the key is the name of a resource and the value is the scheduler options or directives associated with this resource.

Type: Dict[str, List[str]]

property scheduler¶

The backend scheduler of this partition.

Type: reframe.core.schedulers.JobScheduler.

Note

Changed in version 2.8: Prior versions returned a string representing the scheduler and job launcher combination.

Changed in version 3.2: The property now stores a JobScheduler instance.

Job Schedulers and Parallel Launchers¶

class reframe.core.schedulers.Job(name, workdir='.', script_filename=None, stdout=None, stderr=None, max_pending_time=None, sched_flex_alloc_nodes=None, sched_access=[], sched_exclusive_access=None, sched_options=None)[source]¶

Bases: reframe.utility.jsonext.JSONSerializable

A job descriptor.

A job descriptor is created by the framework after the “setup” phase and is associated with the test.

Warning

Users may not create a job descriptor directly.

property completion_time¶

The completion time of this job as a floating point number expressed in seconds since the epoch, in UTC.

This attribute is None if the job hasn’t been finished yet, or if ReFrame runtime hasn’t perceived it yet.

The accuracy of this timestamp depends on the backend scheduler. The slurm scheduler backend relies on job accounting and returns the actual termination time of the job. The rest of the backends report as completion time the moment when the framework realizes that the spawned job has finished. In this case, the accuracy depends on the execution policy used. If tests are executed with the serial execution policy, this is close to the real completion time, but if the asynchronous execution policy is used, it can differ significantly.

Type: float or None

property exitcode¶

The exit code of this job.

This may or may not be set depending on the scheduler backend.

New in version 2.21.

Type: int or None

property jobid¶

The ID of this job.

New in version 2.21.

Changed in version 3.2: Job ID type is now a string.

Type: str or None

launcher¶

The (parallel) program launcher that will be used to launch the (parallel) executable of this job.

Users are allowed to explicitly set the current job launcher, but this is only relevant in rare situations, such as when you want to wrap the current launcher command. For this specific scenario, you may have a look at the reframe.core.launchers.LauncherWrapper class.

The following example shows how you can replace the current partition’s launcher for this test with the “local” launcher:

from reframe.core.backends import getlauncher

@rfm.run_after('setup')
def set_launcher(self):
    self.job.launcher = getlauncher('local')()

Type: reframe.core.launchers.JobLauncher

property nodelist¶

The list of node names assigned to this job.

This attribute is None if no nodes are assigned to the job yet. This attribute is set reliably only for the slurm backend, i.e., Slurm with accounting enabled. The squeue scheduler backend, i.e., Slurm without accounting, might not set this attribute for jobs that finish very quickly. For the local scheduler backend, this returns an one-element list containing the hostname of the current host.

This attribute might be useful in a flexible regression test for determining the actual nodes that were assigned to the test. For more information on flexible node allocation, see the --flex-alloc-nodes command-line option

This attribute is not supported by the pbs scheduler backend.

New in version 2.17.

Type: List[str] or None

options¶

Options to be passed to the backend job scheduler.

Type: List[str]
Default: []

property state¶

The state of this job.

The value of this field is scheduler-specific.

New in version 2.21.

Type: :class`str` or None

property submit_time¶

The submission time of this job as a floating point number expressed in seconds since the epoch, in UTC.

This attribute is None if the job hasn’t been submitted yet.

This attribute is set right after the job is submitted and can vary significantly from the time the jobs starts running, depending on the scheduler.

Type: float or None

class reframe.core.launchers.JobLauncher[source]¶

Bases: abc.ABC

Abstract base class for job launchers.

A job launcher is the executable that actually launches a distributed program to multiple nodes, e.g., mpirun, srun etc.

Warning

Users may not create job launchers directly.

Note

Changed in version 2.8: Job launchers do not get a reference to a job during their initialization.

abstract command(job)[source]¶

The launcher command to be emitted for a specific job.

Launcher backends provide concrete implementations of this method.

Parameters: job – A job descriptor.
Returns: the basic launcher command as a list of tokens.

options¶

List of options to be passed to the job launcher invocation.

Type: List[str]
Default: []

run_command(job)[source]¶

The full launcher command to be emitted for a specific job.

This includes any user options.

Parameters: job – a job descriptor.
Returns: the launcher command as a string.

class reframe.core.launchers.LauncherWrapper(target_launcher, wrapper_command, wrapper_options=[])[source]¶

Bases: reframe.core.launchers.JobLauncher

Wrap a launcher object so as to modify its invocation.

This is useful for parallel debuggers. For example, to launch a regression test using the ARM DDT debugger, you can do the following:

@rfm.run_after('setup')
def set_launcher(self):
    self.job.launcher = LauncherWrapper(self.job.launcher, 'ddt',
                                        ['--offline'])

If the current system partition uses native Slurm for job submission, this setup will generate the following command in the submission script:

ddt --offline srun <test_executable>

If the current partition uses mpirun instead, it will generate

ddt --offline mpirun -np <num_tasks> ... <test_executable>

Parameters

target_launcher – The launcher to wrap.
wrapper_command – The wrapper command.
wrapper_options – List of options to pass to the wrapper command.

command(job)[source]¶

The launcher command to be emitted for a specific job.

Launcher backends provide concrete implementations of this method.

Parameters: job – A job descriptor.
Returns: the basic launcher command as a list of tokens.

reframe.core.backends.getlauncher(name)¶

Retrieve the reframe.core.launchers.JobLauncher concrete implementation for a parallel launcher backend.

Parameters: name – The registered name of the launcher backend.

reframe.core.backends.getscheduler(name)¶

Retrieve the reframe.core.schedulers.JobScheduler concrete implementation for a scheduler backend.

Parameters: name – The registered name of the scheduler backend.

Runtime Services¶

class reframe.core.runtime.RuntimeContext(site_config)[source]¶

Bases: object

The runtime context of the framework.

There is a single instance of this class globally in the framework.

New in version 2.13.

get_option(option)[source]¶

Get a configuration option.

Parameters: option – The option to be retrieved.
Returns: The value of the option.

property modules_system¶

The environment modules system used in the current host.

Type: reframe.core.modules.ModulesSystem.

property output_prefix¶

The output directory prefix.

Type: str

property stage_prefix¶

The stage directory prefix.

Type: str

property system¶

The current host system.

Type: reframe.core.systems.System

reframe.core.runtime.is_env_loaded(environ)[source]¶

Check if environment is loaded.

Parameters: environ (Environment) – Environment to check for.
Returns: True if this environment is loaded, False otherwise.

reframe.core.runtime.loadenv(*environs)[source]¶

Load environments in the current Python context.

Parameters: environs (List[Environment]) – A list of environments to load.
Returns: A tuple containing snapshot of the current environment upon entry to this function and a list of shell commands required to load the environments.
Return type: Tuple[_EnvironmentSnapshot, List[str]]

class reframe.core.runtime.module_use(*paths)[source]¶

Bases: object

Context manager for temporarily modifying the module path.

reframe.core.runtime.runtime()[source]¶

Get the runtime context of the framework.

New in version 2.13.

Returns: A reframe.core.runtime.RuntimeContext object.

class reframe.core.runtime.temp_environment(modules=[], variables=[])[source]¶

Bases: object

Context manager to temporarily change the environment.

Modules Systems¶

class reframe.core.modules.ModulesSystem(backend)[source]¶

Bases: object

A modules system.

available_modules(substr=None)[source]¶

Return a list of available modules that contain substr in their name.

Return type: List[str]

conflicted_modules(name, collection=False, path=None)[source]¶

Return the list of the modules conflicting with module name.

If module name resolves to multiple real modules, then the returned list will be the concatenation of the conflict lists of all the real modules.

Parameters

name – The name of the module.
collection – The module is a “module collection” (TMod4/LMod only).
path – The path where the module resides if not in the default MODULEPATH.

Returns

A list of conflicting module names.

Changed in version 3.3: The collection argument is added.

Changed in version 3.5.0: The path argument is added.

emit_load_commands(name, collection=False, path=None)[source]¶

Return the appropriate shell commands for loading a module.

Module mappings are not taken into account by this function.

Parameters

name – The name of the module to load.
collection – The module is a “module collection” (TMod4/LMod only)
path – The path where the module resides if not in the default MODULEPATH.

Returns

A list of shell commands.

Changed in version 3.3: The collection argument was added and module mappings are no more taken into account by this function.

Changed in version 3.5.0: The path argument is added.

emit_unload_commands(name, collection=False, path=None)[source]¶

Return the appropriate shell commands for unloading a module.

Module mappings are not taken into account by this function.

Parameters

name – The name of the module to unload.
collection – The module is a “module collection” (TMod4/LMod only)
path – The path where the module resides if not in the default MODULEPATH.

Returns

A list of shell commands.

Changed in version 3.3: The collection argument was added and module mappings are no more taken into account by this function.

Changed in version 3.5.0: The path argument is added.

execute(cmd, *args)[source]¶

Execute an arbitrary module command.

Parameters

cmd – The command to execute, e.g., load, restore etc.
args – The arguments to pass to the command.

Returns

The command output.

is_module_loaded(name)[source]¶

Check if module name is loaded.

If module name refers to multiple real modules, this method will return True only if all the referees are loaded.

load_module(name, collection=False, path=None, force=False)[source]¶

Load the module name.

Parameters

collection – The module is a “module collection” (TMod4/Lmod only)
path – The path where the module resides if not in the default MODULEPATH.
force – If set, forces the loading, unloading first any conflicting modules currently loaded. If module name refers to multiple real modules, all of the target modules will be loaded.

Returns

A list of two-element tuples, where each tuple contains the module that was loaded and the list of modules that had to be unloaded first due to conflicts. This list will be normally of size one, but it can be longer if there is mapping that maps module name to multiple other modules.

Changed in version 3.3: - The collection argument is added. - This function now returns a list of tuples.

Changed in version 3.5.0: - The path argument is added. - The force argument is now the last argument.

loaded_modules()[source]¶

Return a list of loaded modules.

Return type: List[str]

property name¶: The name of this module system.

property searchpath¶: The module system search path as a list of directories.

searchpath_add(*dirs)[source]¶: Add dirs to the module system search path.

searchpath_remove(*dirs)[source]¶: Remove dirs from the module system search path.

unload_all()[source]¶: Unload all loaded modules.

unload_module(name, collection=False, path=None)[source]¶

Unload module name.

Parameters

name – The name of the module to unload. If module name is resolved to multiple real modules, all the referred to modules will be unloaded in reverse order.
collection – The module is a “module collection” (TMod4 only)
path – The path where the module resides if not in the default MODULEPATH.

Changed in version 3.3: The collection argument was added.

Changed in version 3.5.0: The path argument is added.

property version¶: The version of this module system.

Build Systems¶

New in version 2.14.

ReFrame delegates the compilation of the regression test to a build system. Build systems in ReFrame are entities that are responsible for generating the necessary shell commands for compiling a code. Each build system defines a set of attributes that users may set in order to customize their compilation. An example usage is the following:

self.build_system = 'SingleSource'
self.build_system.cflags = ['-fopenmp']

Users simply set the build system to use in their regression tests and then they configure it. If no special configuration is needed for the compilation, users may completely ignore the build systems. ReFrame will automatically pick one based on the regression test attributes and will try to compile the code.

All build systems in ReFrame derive from the abstract base class reframe.core.buildsystems.BuildSystem. This class defines a set of common attributes, such us compilers, compilation flags etc. that all subclasses inherit. It is up to the concrete build system implementations on how to use or not these attributes.

class reframe.core.buildsystems.Autotools(*args, **kwargs)[source]¶

Bases: reframe.core.buildsystems.ConfigureBasedBuildSystem

A build system for compiling Autotools-based projects.

This build system will emit the following commands:

Create a build directory if builddir is not None and change to it.
Invoke configure to configure the project by setting the corresponding flags for compilers and compiler flags.
Issue make to compile the code.

class reframe.core.buildsystems.BuildSystem(*args, **kwargs)[source]¶

Bases: object

The abstract base class of any build system.

Concrete build systems inherit from this class and must override the emit_build_commands() abstract function.

cc = ''¶

The C compiler to be used. If empty and flags_from_environ is True, the compiler defined in the current programming environment will be used.

Type: str
Default: ''

cflags = []¶

The C compiler flags to be used. If empty and flags_from_environ is True, the corresponding flags defined in the current programming environment will be used.

Type: List[str]
Default: []

cppflags = []¶

The preprocessor flags to be used. If empty and flags_from_environ is True, the corresponding flags defined in the current programming environment will be used.

Type: List[str]
Default: []

cxx = ''¶

The C++ compiler to be used. If empty and flags_from_environ is True, the compiler defined in the current programming environment will be used.

Type: str
Default: ''

cxxflags = []¶

The C++ compiler flags to be used. If empty and flags_from_environ is True, the corresponding flags defined in the current programming environment will be used.

Type: List[str]
Default: []

fflags = []¶

The Fortran compiler flags to be used. If empty and flags_from_environ is True, the corresponding flags defined in the current programming environment will be used.

Type: List[str]
Default: []

flags_from_environ = True¶

Set compiler and compiler flags from the current programming environment if not specified otherwise.

Type: bool
Default: True

ftn = ''¶

The Fortran compiler to be used. If empty and flags_from_environ is True, the compiler defined in the current programming environment will be used.

Type: str
Default: ''

ldflags = []¶

The linker flags to be used. If empty and flags_from_environ is True, the corresponding flags defined in the current programming environment will be used.

Type: List[str]
Default: []

nvcc = ''¶

The CUDA compiler to be used. If empty and flags_from_environ is True, the compiler defined in the current programming environment will be used.

Type: str
Default: ''

class reframe.core.buildsystems.BuildSystemMeta(name, bases, namespace, **kwargs)[source]¶

Bases: reframe.core.meta.RegressionTestMeta, abc.ABCMeta

Build systems metaclass.

class reframe.core.buildsystems.CMake(*args, **kwargs)[source]¶

Bases: reframe.core.buildsystems.ConfigureBasedBuildSystem

A build system for compiling CMake-based projects.

This build system will emit the following commands:

Create a build directory if builddir is not None and change to it.
Invoke cmake to configure the project by setting the corresponding CMake flags for compilers and compiler flags.
Issue make to compile the code.

class reframe.core.buildsystems.ConfigureBasedBuildSystem(*args, **kwargs)[source]¶

Bases: reframe.core.buildsystems.BuildSystem

Abstract base class for configured-based build systems.

builddir = None¶

The CMake build directory, where all the generated files will be placed.

Type: str
Default: None

config_opts = []¶

Additional configuration options to be passed to the CMake invocation.

Type: List[str]
Default: []

make_opts = []¶

Options to be passed to the subsequent make invocation.

Type: List[str]
Default: []

max_concurrency = 1¶

Same as for the Make build system.

Type: integer
Default: 1

srcdir = None¶

The top-level directory of the code.

This is set automatically by the framework based on the reframe.core.pipeline.RegressionTest.sourcepath attribute.

Type: str
Default: None

class reframe.core.buildsystems.EasyBuild(*args, **kwargs)[source]¶

Bases: reframe.core.buildsystems.BuildSystem

A build system for building test code using EasyBuild.

ReFrame will use EasyBuild to build and install the code in the test’s stage directory by default. ReFrame uses environment variables to configure EasyBuild for running, so users can pass additional options to the eb command and modify the default behaviour.

New in version 3.5.0.

easyconfigs = []¶

The list of easyconfig files to build and install. This field is required.

Type: List[str]
Default: []

emit_package = False¶

Instruct EasyBuild to emit a package for the built software. This will essentially pass the --package option to eb.

Type: bool
Default: False

options = []¶

Options to pass to the eb command.

Type: List[str]
Default: []

package_opts = {}¶

Options controlling the package creation from EasyBuild. For each key/value pair of this dictionary, ReFrame will pass --package-{key}={val} to the EasyBuild invocation.

Type: Dict[str, str]
Default: {}

prefix = 'easybuild'¶

Default prefix for the EasyBuild installation.

Relative paths will be appended to the stage directory of the test. ReFrame will set the following environment variables before running EasyBuild.

export EASYBUILD_BUILDPATH={prefix}/build
export EASYBUILD_INSTALLPATH={prefix}
export EASYBUILD_PREFIX={prefix}
export EASYBUILD_SOURCEPATH={prefix}

Users can change these defaults by passing specific options to the eb command.

Type: str
Default: easybuild

class reframe.core.buildsystems.Make(*args, **kwargs)[source]¶

Bases: reframe.core.buildsystems.BuildSystem

A build system for compiling codes using make.

The generated build command has the following form:

make -j [N] [-f MAKEFILE] [-C SRCDIR] CC="X" CXX="X" FC="X" NVCC="X" CPPFLAGS="X" CFLAGS="X" CXXFLAGS="X" FCFLAGS="X" LDFLAGS="X" OPTIONS

The compiler and compiler flags variables will only be passed if they are not None. Their value is determined by the corresponding attributes of BuildSystem. If you want to completely disable passing these variables to the make invocation, you should make sure not to set any of the correspoding attributes and set also the BuildSystem.flags_from_environ flag to False.

makefile = None¶

Instruct build system to use this Makefile. This option is useful when having non-standard Makefile names.

Type: str
Default: None

max_concurrency = 1¶

Limit concurrency for make jobs. This attribute controls the -j option passed to make. If not None, make will be invoked as make -j max_concurrency. Otherwise, it will invoked as make -j.

Type: integer
Default: 1

Note

Changed in version 2.19: The default value is now 1

options = []¶

Append these options to the make invocation. This variable is also useful for passing variables or targets to make.

Type: List[str]
Default: []

srcdir = None¶

The top-level directory of the code.

This is set automatically by the framework based on the reframe.core.pipeline.RegressionTest.sourcepath attribute.

Type: str
Default: None

class reframe.core.buildsystems.SingleSource(*args, **kwargs)[source]¶

Bases: reframe.core.buildsystems.BuildSystem

A build system for compiling a single source file.

The generated build command will have the following form:

COMP CPPFLAGS XFLAGS SRCFILE -o EXEC LDFLAGS

COMP is the required compiler for compiling SRCFILE. This build system will automatically detect the programming language of the source file and pick the correct compiler. See also the SingleSource.lang attribute.
CPPFLAGS are the preprocessor flags and are passed to any compiler.
XFLAGS is any of CFLAGS, CXXFLAGS or FCFLAGS depending on the programming language of the source file.
SRCFILE is the source file to be compiled. This is set up automatically by the framework. See also the SingleSource.srcfile attribute.
EXEC is the executable to be generated. This is also set automatically by the framework. See also the SingleSource.executable attribute.
LDFLAGS are the linker flags.

For CUDA codes, the language assumed is C++ (for the compilation flags) and the compiler used is BuildSystem.nvcc.

executable = None¶

The executable file to be generated.

This is set automatically by the framework based on the reframe.core.pipeline.RegressionTest.executable attribute.

Type: str or None

include_path = []¶

The include path to be used for this compilation.

All the elements of this list will be appended to the BuildSystem.cppflags, by prepending to each of them the -I option.

Type: List[str]
Default: []

lang = None¶

The programming language of the file that needs to be compiled. If not specified, the build system will try to figure it out automatically based on the extension of the source file. The automatically detected extensions are the following:

C: .c and .upc.

C++: .cc, .cp, .cxx, .cpp, .CPP, .c++ and .C.

Fortran: .f, .for, .ftn, .F, .FOR, .fpp, .FPP, .FTN, .f90, .f95, .f03, .f08, .F90, .F95, .F03 and .F08.

CUDA: .cu.

Type: str or None

srcfile = None¶

The source file to compile. This is automatically set by the framework based on the reframe.core.pipeline.RegressionTest.sourcepath attribute.

Type: str or None

class reframe.core.buildsystems.Spack(*args, **kwargs)[source]¶

Bases: reframe.core.buildsystems.BuildSystem

A build system for building test code using Spack.

ReFrame will use a user-provided Spack environment in order to build and test a set of specs.

New in version 3.6.1.

emit_load_cmds = True¶

Emit the necessary spack load commands before running the test.

Type: bool
Default: True

environment = None¶

The Spack environment to use for building this test.

ReFrame will activate and install this environment. This environment will also be used to run the test.

spack env activate -V -d <environment directory>

ReFrame looks for environments in the test’s sourcesdir.

If this field is None, the default, the environment name will be automatically set to rfm_spack_env.

Type: str or None
Default: None

Note

Changed in version 3.7.3: The field is no longer required and the Spack environment will be automatically created if not provided.

install_opts = []¶

Options to pass to spack install

Type: List[str]
Default: []

install_tree = None¶

The directory where Spack will install the packages requested by this test.

After activating the Spack environment, ReFrame will set the install_tree Spack configuration in the given environment with the following command:

spack config add "config:install_tree:root:<install tree>"

Relative paths are resolved against the test’s stage directory. If this field and the Spack environment are both None, the default, the install directory will be automatically set to opt/spack. If this field None but the Spack environment is not, then install_tree will not be set automatically and the install tree of the given environment will not be overridden.

Type: str or None
Default: None

New in version 3.7.3.

specs = []¶

A list of additional specs to build and install within the given environment.

ReFrame will add the specs to the active environment by emititing the following command:

spack add spec1 spec2 ... specN

If no spec is passed, ReFrame will simply install what is prescribed by the environment.

Type: List[str]
Default: []

Container Platforms¶

New in version 2.20.

class reframe.core.containers.ContainerPlatform[source]¶

Bases: abc.ABC

The abstract base class of any container platform.

command¶

The command to be executed within the container.

If no command is given, then the default command of the corresponding container image is going to be executed.

New in version 3.5.0: Changed the attribute name from commands to command and its type to a string.

Type: str or None
Default: None

commands¶

The commands to be executed within the container.

Deprecated since version 3.5.0: Please use the command field instead.

Type: list[str]
Default: []

image¶

The container image to be used for running the test.

Type: str or None
Default: None

mount_points¶

List of mount point pairs for directories to mount inside the container.

Each mount point is specified as a tuple of (/path/in/host, /path/in/container). The stage directory of the ReFrame test is always mounted under /rfm_workdir inside the container, independelty of this field.

Type: list[tuple[str, str]]
Default: []

options¶

Additional options to be passed to the container runtime when executed.

Type: list[str]
Default: []

pull_image¶

Pull the container image before running.

This does not have any effect for the Singularity container platform.

New in version 3.5.

Type: bool
Default: True

workdir¶

The working directory of ReFrame inside the container.

This is the directory where the test’s stage directory is mounted inside the container. This directory is always mounted regardless if mount_points is set or not.

Deprecated since version 3.5: Please use the options field to set the working directory.

Type: str
Default: /rfm_workdir

class reframe.core.containers.Docker[source]¶

Bases: reframe.core.containers.ContainerPlatform

Container platform backend for running containers with Docker.

class reframe.core.containers.Sarus[source]¶

Bases: reframe.core.containers.ContainerPlatform

Container platform backend for running containers with Sarus.

with_mpi¶

Enable MPI support when launching the container.

Type: boolean
Default: False

class reframe.core.containers.Shifter[source]¶

Bases: reframe.core.containers.Sarus

Container platform backend for running containers with Shifter.

class reframe.core.containers.Singularity[source]¶

Bases: reframe.core.containers.ContainerPlatform

Container platform backend for running containers with Singularity.

with_cuda¶

Enable CUDA support when launching the container.

Type: boolean
Default: False

The `reframe` module¶

The reframe module offers direct access to the basic test classes, constants and decorators.

class reframe.CompileOnlyRegressionTest¶: See reframe.core.pipeline.CompileOnlyRegressionTest.

class reframe.RegressionTest¶: See reframe.core.pipeline.RegressionTest.

class reframe.RunOnlyRegressionTest¶: See reframe.core.pipeline.RunOnlyRegressionTest.

reframe.DEPEND_BY_ENV¶: See reframe.core.pipeline.DEPEND_BY_ENV.

reframe.DEPEND_EXACT¶: See reframe.core.pipeline.DEPEND_EXACT.

reframe.DEPEND_FULLY¶: See reframe.core.pipeline.DEPEND_FULLY.

@reframe.parameterized_test¶: See @reframe.core.decorators.parameterized_test.

@reframe.require_deps¶: Deprecated since version 3.7.0: Please use the require_deps() built-in function

@reframe.required_version¶: See @reframe.core.decorators.required_version.

@reframe.run_after¶: Deprecated since version 3.7.0: Please use the run_after() built-in function

@reframe.run_before¶: Deprecated since version 3.7.0: Please use the run_before() built-in function

@reframe.simple_test¶: See @reframe.core.decorators.simple_test.

Mapping of Test Attributes to Job Scheduler Backends¶

Test attribute	Slurm option	Torque option	PBS option
`num_tasks`	`--ntasks`¹	`-l nodes={num_tasks//num_tasks_per_node}:ppn={num_tasks_per_node*num_cpus_per_task}`	`-l select={num_tasks//num_tasks_per_node}:mpiprocs={num_tasks_per_node}:ncpus={num_tasks_per_node*num_cpus_per_task}`
`num_tasks_per_node`	`--ntasks-per-node`	see `num_tasks`	see `num_tasks`
`num_tasks_per_core`	`--ntasks-per-core`	n/a	n/a
`num_tasks_per_socket`	`--ntasks-per-socket`	n/a	n/a
`num_cpus_per_task`	`--cpus-per-task`	see `num_tasks`	see `num_tasks`
`time_limit`	`--time=hh:mm:ss`	`-l walltime=hh:mm:ss`	`-l walltime=hh:mm::ss`
`exclusive_access`	`--exclusive`	n/a	n/a
`use_smt`	`--hint=[no]multithread`	n/a	n/a

If any of the attributes is set to None it will not be emitted at all in the job script. In cases that the attribute is required, it will be set to 1.

¹ The --nodes option may also be emitted if the use_nodes_option scheduler configuration parameter is set.

Regression Test API¶

Test Base Classes¶

Test Decorators¶

Built-in types¶

Built-in functions¶

Pipeline Hooks¶

Test variants¶

Environments and Systems¶

Job Schedulers and Parallel Launchers¶

Runtime Services¶

Modules Systems¶

Build Systems¶

Container Platforms¶

The reframe module¶

Mapping of Test Attributes to Job Scheduler Backends¶

The `reframe` module¶