ReFrame Programming APIs¶

This page provides a reference guide of the ReFrame API for writing regression tests covering all the relevant details. Internal data structures and APIs are covered only to the extent that this might be helpful to the final user of the framework.

Regression Test Base Classes¶

class reframe.core.pipeline.CompileOnlyRegressionTest(*args, **kwargs)[source]¶

Bases: reframe.core.pipeline.RegressionTest

Base class for compile-only regression tests.

These tests are by default local and will skip the run phase of the regression test pipeline.

The standard output and standard error of the test will be set to those of the compilation stage.

This class is also directly available under the top-level reframe module.

run()[source]¶

The run stage of the regression test pipeline.

Implemented as no-op.

setup(partition, environ, **job_opts)[source]¶

The setup stage of the regression test pipeline.

Similar to the RegressionTest.setup(), except that no job descriptor is set up for this test.

property stderr¶

The name of the file containing the standard error of the test.

This is set during the setup() phase.

This attribute is evaluated lazily, so it can by used inside sanity expressions.

Type: str.

property stdout¶

The name of the file containing the standard output of the test.

This is set during the setup() phase.

This attribute is evaluated lazily, so it can by used inside sanity expressions.

Type: str.

wait()[source]¶

Wait for this test to finish.

Implemented as no-op

reframe.core.pipeline.DEPEND_BY_ENV = 2¶: Constant to be passed as the how argument of the RegressionTest.depends_on() method. It denotes that the test cases of the current test will depend only on the corresponding test cases of the target test that use the same programming environment.

This constant is directly available under the reframe module.

reframe.core.pipeline.DEPEND_EXACT = 1¶: Constant to be passed as the how argument of the RegressionTest.depends_on() method. It denotes that test case dependencies will be explicitly specified by the user.

This constant is directly available under the reframe module.

reframe.core.pipeline.DEPEND_FULLY = 3¶: Constant to be passed as the how argument of the RegressionTest.depends_on() method. It denotes that each test case of this test depends on all the test cases of the target test.

This constant is directly available under the reframe module.

class reframe.core.pipeline.RegressionTest(*args, **kwargs)[source]¶

Bases: object

Base class for regression tests.

All regression tests must eventually inherit from this class. This class provides the implementation of the pipeline phases that the regression test goes through during its lifetime.

Note

Changed in version 2.19: Base constructor takes no arguments.

build_system¶

New in version 2.14.

The build system to be used for this test. If not specified, the framework will try to figure it out automatically based on the value of sourcepath.

This field may be set using either a string referring to a concrete build system class name (see build systems) or an instance of reframe.core.buildsystems.BuildSystem. The former is the recommended way.

Type: str or reframe.core.buildsystems.BuildSystem.
Default: None.

check_performance()[source]¶

The performance checking phase of the regression test pipeline.

Raises: reframe.core.exceptions.SanityError – If the performance check fails.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

check_sanity()[source]¶

The sanity checking phase of the regression test pipeline.

Raises: reframe.core.exceptions.SanityError – If the sanity check fails.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

cleanup(remove_files=False)[source]¶

The cleanup phase of the regression test pipeline.

Parameters: remove_files – If True, the stage directory associated with this test will be removed.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

compile()[source]¶

The compilation phase of the regression test pipeline.

Raises: reframe.core.exceptions.ReframeError – In case of errors.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

compile_wait()[source]¶: Wait for compilation phase to finish.

New in version 2.13.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

container_platform¶

New in version 2.20.

The container platform to be used for launching this test.

If this field is set, the test will run inside a container using the specified container runtime. Container-specific options must be defined additionally after this field is set:

self.container_platform = 'Singularity'
self.container_platform.image = 'docker://ubuntu:18.04'
self.container_platform.commands = ['cat /etc/os-release']

If this field is set, executable and executable_opts attributes are ignored. The container platform’s commands will be used instead.

Type: str or reframe.core.containers.ContainerPlatform.
Default: None.

property current_environ¶

The programming environment that the regression test is currently executing with.

This is set by the framework during the setup() phase.

Type: reframe.core.environments.ProgEnvironment.

property current_partition¶

The system partition the regression test is currently executing on.

This is set by the framework during the setup() phase.

Type: reframe.core.systems.SystemPartition.

property current_system¶

The system the regression test is currently executing on.

This is set by the framework during the initialization phase.

Type: reframe.core.systems.System.

depends_on(target, how=2, subdeps=None)[source]¶

Add a dependency to target in this test.

Parameters

target – The name of the target test.
how – How the dependency should be mapped in the test cases space. This argument can accept any of the three constants DEPEND_EXACT, DEPEND_BY_ENV (default), DEPEND_FULLY.
subdeps –
An adjacency list representation of how this test’s test cases depend on those of the target test. This is only relevant if how == DEPEND_EXACT. The value of this argument is a dictionary having as keys the names of this test’s supported programming environments. The values are lists of the programming environments names of the target test that this test’s test cases will depend on. In the following example, this test’s E0 programming environment case will depend on both E0 and E1 test cases of the target test T0, but its E1 case will depend only on the E1 test case of T0:
```
self.depends_on('T0', how=rfm.DEPEND_EXACT,
                subdeps={'E0': ['E0', 'E1'], 'E1': ['E1']})
```

For more details on how test dependencies work in ReFrame, please refer to How Test Dependencies Work In ReFrame.

New in version 2.21.

descr¶

A detailed description of the test.

Type: str
Default: self.name

exclusive_access¶

Specify whether this test needs exclusive access to nodes.

Type: boolean
Default: False

executable¶

The name of the executable to be launched during the run phase.

Type: str
Default: os.path.join('.', self.name)

executable_opts¶

List of options to be passed to the executable.

Type: List[str]
Default: []

extra_resources¶

New in version 2.8.

Extra resources for this test.

This field is for specifying custom resources needed by this test. These resources are defined in the configuration of a system partition. For example, assume that two additional resources, named gpu and datawarp, are defined in the configuration file as follows:

'resources': {
    'gpu': [
        '--gres=gpu:{num_gpus_per_node}'
    ],
    'datawarp': [
        '#DW jobdw capacity={capacity}',
        '#DW stage_in source={stagein_src}'
    ]
}

A regression test then may instantiate the above resources by setting the extra_resources attribute as follows:

self.extra_resources = {
    'gpu': {'num_gpus_per_node': 2}
    'datawarp': {
        'capacity': '100GB',
        'stagein_src': '/foo'
    }
}

The generated batch script (for Slurm) will then contain the following lines:

#SBATCH --gres=gpu:2
#DW jobdw capacity=100GB
#DW stage_in source=/foo

Notice that if the resource specified in the configuration uses an alternative directive prefix (in this case #DW), this will replace the standard prefix of the backend scheduler (in this case #SBATCH)

If the resource name specified in this variable does not match a resource name in the partition configuration, it will be simply ignored. The num_gpus_per_node attribute translates internally to the _rfm_gpu resource, so that setting self.num_gpus_per_node = 2 is equivalent to the following:

self.extra_resources = {'_rfm_gpu': {'num_gpus_per_node': 2}}

Type: Dict[str, Dict[str, object]]
Default: {}

Note

Changed in version 2.9: A new more powerful syntax was introduced that allows also custom job script directive prefixes.

getdep(target, environ=None)[source]¶

Retrieve the test case of a target dependency.

This is a low-level method. The @require_deps decorators should be preferred.

Parameters

target – The name of the target dependency to be retrieved.
environ – The name of the programming environment that will be used to retrieve the test case of the target test. If None, RegressionTest.current_environ will be used.

New in version 2.21.

info()[source]¶

Provide live information for this test.

This method is used by the front-end to print the status message during the test’s execution. This function is also called to provide the message for the check_info logging attribute. By default, it returns a message reporting the test name, the current partition and the current programming environment that the test is currently executing on.

New in version 2.10.

Returns: a string with an informational message about this test

Note

When overriding this method, you should pay extra attention on how you use the RegressionTest’s attributes, because this method may be called at any point of the test’s lifetime.

is_local()[source]¶

Check if the test will execute locally.

A test executes locally if the local attribute is set or if the current partition’s scheduler does not support job submission.

property job¶

The job descriptor associated with this test.

This is set by the framework during the setup() phase.

Type: reframe.core.schedulers.Job.

keep_files¶

List of files to be kept after the test finishes.

By default, the framework saves the standard output, the standard error and the generated shell script that was used to run this test.

These files will be copied over to the framework’s output directory during the cleanup() phase.

Directories are also accepted in this field.

Relative path names are resolved against the stage directory.

Type: List[str]
Default: []

local¶

Always execute this test locally.

Type: boolean
Default: False

property logger¶

A logger associated with this test.

You can use this logger to log information for your test.

maintainers¶

List of people responsible for this test.

When the test fails, this contact list will be printed out.

Type: List[str]
Default: []

max_pending_time¶

New in version 3.0.

The maximum time a job can be pending before starting running.

Time duration is specified as of the time_limit attribute.

Type: str or datetime.timedelta
Default: None

modules¶

List of modules to be loaded before running this test.

These modules will be loaded during the setup() phase.

Type: List[str]
Default: []

name¶

The name of the test.

Type: string that can contain any character except /

num_cpus_per_task¶

Number of CPUs per task required by this test.

Ignored if None.

Type: integral or None
Default: None

num_gpus_per_node¶

Number of GPUs per node required by this test. This attribute is translated internally to the _rfm_gpu resource. For more information on test resources, have a look at the extra_resources attribute.

Type: integral
Default: 0

num_tasks¶

Number of tasks required by this test.

If the number of tasks is set to a number <=0, ReFrame will try to flexibly allocate the number of tasks, based on the command line option --flex-alloc-nodes. A negative number is used to indicate the minimum number of tasks required for the test. In this case the minimum number of tasks is the absolute value of the number, while Setting num_tasks to 0 is equivalent to setting it to -num_tasks_per_node.

Type: integral
Default: 1

Note

Changed in version 2.15: Added support for flexible allocation of the number of tasks if the number of tasks is set to 0.

Changed in version 2.16: Negative num_tasks is allowed for specifying the minimum number of required tasks by the test.

num_tasks_per_core¶

Number of tasks per core required by this test.

Ignored if None.

Type: integral or None
Default: None

num_tasks_per_node¶

Number of tasks per node required by this test.

Ignored if None.

Type: integral or None
Default: None

num_tasks_per_socket¶

Number of tasks per socket required by this test.

Ignored if None.

Type: integral or None
Default: None

property outputdir¶

The output directory of the test.

This is set during the setup() phase.

New in version 2.13.

Type: str.

perf_patterns¶

Patterns for verifying the performance of this test.

Refer to the ReFrame Tutorials for concrete usage examples.

If set to None, no performance checking will be performed.

Type: A dictionary with keys of type str and deferrable expressions (i.e., the result of a sanity function) as values. None is also allowed.
Default: None

poll()[source]¶

Poll the test’s state.

Returns

True if the associated job has finished, False otherwise.

If no job descriptor is yet associated with this test, True is returned.

Raises

reframe.core.exceptions.ReframeError – In case of errors.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

post_run¶: Deprecated since version 3.0.

Use postrun_cmds instead.

postbuild_cmd¶: Deprecated since version 3.0.

Use postbuild_cmds instead.

postbuild_cmds¶

New in version 3.0.

List of shell commands to be executed after a successful compilation.

These commands are emitted in the script after the actual build commands generated by the selected build system.

Type: List[str]
Default: []

postrun_cmds¶

New in version 3.0.

List of shell commands to execute after launching this job.

See prerun_cmds for a more detailed description of the semantics.

Type: List[str]
Default: []

pre_run¶: Deprecated since version 3.0.

Use prerun_cmds instead.

prebuild_cmd¶: Deprecated since version 3.0.

Use prebuild_cmds instead.

prebuild_cmds¶

New in version 3.0.

List of shell commands to be executed before compiling.

These commands are emitted in the build script before the actual build commands generated by the selected build system.

Type: List[str]
Default: []

property prefix¶

The prefix directory of the test.

Type: str.

prerun_cmds¶

New in version 3.0.

List of shell commands to execute before launching this job.

These commands do not execute in the context of ReFrame. Instead, they are emitted in the generated job script just before the actual job launch command.

Type: List[str]
Default: []

readonly_files¶

List of files or directories (relative to the sourcesdir) that will be symlinked in the stage directory and not copied.

You can use this variable to avoid copying very large files to the stage directory.

Type: List[str]
Default: []

reference¶

The set of reference values for this test.

The reference values are specified as a scoped dictionary keyed on the performance variables defined in perf_patterns and scoped under the system/partition combinations. The reference itself is a four-tuple that contains the reference value, the lower and upper thresholds and the measurement unit.

An example follows:

self.reference = {
    'sys0:part0': {
        'perfvar0': (50, -0.1, 0.1, 'Gflop/s'),
        'perfvar1': (20, -0.1, 0.1, 'GB/s')
    },
    'sys0:part1': {
        'perfvar0': (100, -0.1, 0.1, 'Gflop/s'),
        'perfvar1': (40, -0.1, 0.1, 'GB/s')
    }
}

Type: A scoped dictionary with system names as scopes or None
Default: {}

Note

Changed in version 3.0: The measurement unit is required. The user should explicitly specify None if no unit is available.

run()[source]¶

The run phase of the regression test pipeline.

This call is non-blocking. It simply submits the job associated with this test and returns.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

sanity_patterns¶

Refer to the ReFrame Tutorials for concrete usage examples.

If set to None, a sanity error will be raised during sanity checking.

Type: A deferrable expression (i.e., the result of a sanity function) or None
Default: None

Note

Changed in version 2.9: The default behaviour has changed and it is now considered a sanity failure if this attribute is set to None.

If a test doesn’t care about its output, this must be stated explicitly as follows:

self.sanity_patterns = sn.assert_found(r'.*', self.stdout)

setup(partition, environ, **job_opts)[source]¶

The setup phase of the regression test pipeline.

Parameters

partition – The system partition to set up this test for.
environ – The environment to set up this test for.
job_opts – Options to be passed through to the backend scheduler. When overriding this method users should always pass through job_opts to the base class method.

Raises

reframe.core.exceptions.ReframeError – In case of errors.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

sourcepath¶

The path to the source file or source directory of the test.

It must be a path relative to the sourcesdir, pointing to a subfolder or a file contained in sourcesdir. This applies also in the case where sourcesdir is a Git repository.

If it refers to a regular file, this file will be compiled using the SingleSource build system. If it refers to a directory, ReFrame will try to infer the build system to use for the project and will fall back in using the Make build system, if it cannot find a more specific one.

Type: str
Default: ''

sourcesdir¶

The directory containing the test’s resources.

This directory may be specified with an absolute path or with a path relative to the location of the test. Its contents will always be copied to the stage directory of the test.

This attribute may also accept a URL, in which case ReFrame will treat it as a Git repository and will try to clone its contents in the stage directory of the test.

If set to None, the test has no resources an no action is taken.

Type: str or None
Default: 'src' if such a directory exists at the test level, otherwise None

Note

Changed in version 2.9: Allow None values to be set also in regression tests with a compilation phase

Changed in version 2.10: Support for Git repositories was added.

Changed in version 3.0: Default value is now conditionally set to either 'src' or None.

property stagedir¶

The stage directory of the test.

This is set during the setup() phase.

Type: str.

property stderr¶

The name of the file containing the standard error of the test.

This is set during the setup() phase.

This attribute is evaluated lazily, so it can by used inside sanity expressions.

Type: str.

property stdout¶

The name of the file containing the standard output of the test.

This is set during the setup() phase.

This attribute is evaluated lazily, so it can by used inside sanity expressions.

Type: str.

strict_check¶

Mark this test as a strict performance test.

If a test is marked as non-strict, the performance checking phase will always succeed, unless the --strict command-line option is passed when invoking ReFrame.

Type: boolean
Default: True

tags¶

Set of tags associated with this test.

This test can be selected from the frontend using any of these tags.

Type: Set[str]
Default: an empty set

time_limit¶

Time limit for this test.

Time limit is specified as a string in the form <days>d<hours>h<minutes>m<seconds>s. If set to None, no time limit will be set. The default time limit of the system partition’s scheduler will be used.

The value is internaly kept as a datetime.timedelta object. For example ‘2h30m’ is represented as datetime.timedelta(hours=2, minutes=30)

Type: str or datetime.timedelta
Default: '10m'

Note

Changed in version 2.15: This attribute may be set to None.

Warning

Changed in version 3.0: The old syntax using a (h, m, s) tuple is deprecated.

use_multithreading¶

Specify whether this tests needs simultaneous multithreading enabled.

Ignored if None.

Type: boolean or None
Default: None

valid_prog_environs¶

List of programming environments supported by this test.

If * is in the list then all programming environments are supported by this test.

Type: List[str]
Default: []

Note

Changed in version 2.12: Programming environments can now be specified using wildcards.

Changed in version 2.17: Support for wildcards is dropped.

valid_systems¶

List of systems supported by this test. The general syntax for systems is <sysname>[:<partname>]. Both <sysname> and <partname> accept the value * to mean any value. * is an alias of *:*

Type: List[str]
Default: []

variables¶

Environment variables to be set before running this test.

These variables will be set during the setup() phase.

Type: Dict[str, str]
Default: {}

wait()[source]¶

Wait for this test to finish.

Raises: reframe.core.exceptions.ReframeError – In case of errors.

Warning

Changed in version 3.0: You may not override this method directly unless you are in special test. See here for more details.

class reframe.core.pipeline.RunOnlyRegressionTest(*args, **kwargs)[source]¶

Bases: reframe.core.pipeline.RegressionTest

Base class for run-only regression tests.

This class is also directly available under the top-level reframe module.

compile()[source]¶

The compilation phase of the regression test pipeline.

This is a no-op for this type of test.

compile_wait()[source]¶

Wait for compilation phase to finish.

This is a no-op for this type of test.

run()[source]¶

The run phase of the regression test pipeline.

The resources of the test are copied to the stage directory and the rest of execution is delegated to the RegressionTest.run().

Regression Test Class Decorators¶

@reframe.core.decorators.parameterized_test(*inst)[source]¶

Class decorator for registering multiple instantiations of a test class.

The decorated class must derive from reframe.core.pipeline.RegressionTest. This decorator is also available directly under the reframe module.

Parameters: inst – The different instantiations of the test. Each instantiation argument may be either a sequence or a mapping.

New in version 2.13.

Note

This decorator does not instantiate any test. It only registers them. The actual instantiation happens during the loading phase of the test.

@reframe.core.decorators.required_version(*versions)[source]¶

Class decorator for specifying the required ReFrame versions for the following test.

If the test is not compatible with the current ReFrame version it will be skipped.

New in version 2.13.

Parameters

versions –

A list of ReFrame version specifications that this test is allowed to run. A version specification string can have one of the following formats:

VERSION: Specifies a single version.
{OP}VERSION, where {OP} can be any of >, >=, <, <=, == and !=. For example, the version specification string '>=2.15' will allow the following test to be loaded only by ReFrame 2.15 and higher. The ==VERSION specification is the equivalent of VERSION.
V1..V2: Specifies a range of versions.

You can specify multiple versions with this decorator, such as @required_version('2.13', '>=2.16'), in which case the test will be selected if any of the versions is satisfied, even if the versions specifications are conflicting.

@reframe.core.decorators.simple_test[source]¶

Class decorator for registering parameterless tests with ReFrame.

The decorated class must derive from reframe.core.pipeline.RegressionTest. This decorator is also available directly under the reframe module.

New in version 2.13.

Pipeline Hooks¶

@reframe.core.decorators.run_after(stage)[source]¶

Decorator for attaching a test method to a pipeline stage.

This is completely analogous to the reframe.core.decorators.run_before.

New in version 2.20.

@reframe.core.decorators.run_before(stage)[source]¶

Decorator for attaching a test method to a pipeline stage.

The method will run just before the specified pipeline stage and it should not accept any arguments except self.

This decorator can be stacked, in which case the function will be attached to multiple pipeline stages.

The stage argument can be any of 'setup', 'compile', 'run', 'sanity', 'performance' or 'cleanup'.

New in version 2.20.

@reframe.core.decorators.require_deps[source]¶

Denote that the decorated test method will use the test dependencies.

The arguments of the decorated function must be named after the dependencies that the function intends to use. The decorator will bind the arguments to a partial realization of the reframe.core.pipeline.RegressionTest.getdep() function, such that conceptually the new function arguments will be the following:

new_arg = functools.partial(getdep, orig_arg_name)

The converted arguments are essentially functions accepting a single argument, which is the target test’s programming environment.

Additionally, this decorator will attach the function to run after the test’s setup phase, but before any other “post_setup” pipeline hook.

This decorator is also directly available under the reframe module.

New in version 2.21.

Environments and Systems¶

class reframe.core.environments.Environment(name, modules=None, variables=None)[source]¶

Bases: object

This class abstracts away an environment to run regression tests.

It is simply a collection of modules to be loaded and environment variables to be set when this environment is loaded by the framework.

Warning

Users may not create Environment objects directly.

property modules¶

The modules associated with this environment.

Type: List[str]

property name¶

The name of this environment.

Type: str

property variables¶

The environment variables associated with this environment.

Type: OrderedDict[str, str]

class reframe.core.environments.ProgEnvironment(name, modules=None, variables=None, cc='cc', cxx='CC', ftn='ftn', nvcc='nvcc', cppflags=None, cflags=None, cxxflags=None, fflags=None, ldflags=None, **kwargs)[source]¶

Bases: reframe.core.environments.Environment

A class representing a programming environment.

This type of environment adds also properties for retrieving the compiler and compilation flags.

Warning

Users may not create ProgEnvironment objects directly.

property cc¶

The C compiler of this programming environment.

Type: str

property cflags¶

The C compiler flags of this programming environment.

Type: List[str]

property cppflags¶

The preprocessor flags of this programming environment.

Type: List[str]

property cxx¶

The C++ compiler of this programming environment.

Type: str

property cxxflags¶

The C++ compiler flags of this programming environment.

Type: List[str]

property fflags¶

The Fortran compiler flags of this programming environment.

Type: List[str]

property ftn¶

The Fortran compiler of this programming environment.

Type: str

property ldflags¶

The linker flags of this programming environment.

Type: List[str]

class reframe.core.environments._EnvironmentSnapshot(name='env_snapshot')[source]¶

Bases: reframe.core.environments.Environment

An environment snapshot.

restore()[source]¶: Restore this environment snapshot.

reframe.core.environments.snapshot()[source]¶

Create an environment snapshot

Returns: An instance of _EnvironmentSnapshot.

class reframe.core.systems.System(name, descr, hostnames, modules_system, preload_env, prefix, outputdir, resourcesdir, stagedir, partitions)[source]¶

Bases: object

A representation of a system inside ReFrame.

Warning

Users may not create System objects directly.

property descr¶

The description of this system.

Type: str

property hostnames¶

The hostname patterns associated with this system.

Type: List[str]

json()[source]¶: Return a JSON object representing this system.

property modules_system¶

The modules system name associated with this system.

Type: reframe.core.modules.ModulesSystem

property name¶

The name of this system.

Type: str

property outputdir¶

The ReFrame output directory prefix associated with this system.

Type: str

property partitions¶

The system partitions associated with this system.

Type: List[SystemPartition]

property prefix¶

The ReFrame prefix associated with this system.

Type: str

property preload_environ¶

The environment to load whenever ReFrame runs on this system.

New in version 2.19.

Type: reframe.core.environments.Environment

property resourcesdir¶

Global resources directory for this system.

This directory may be used for storing large files related to regression tests. The value of this directory is controlled by the resourcesdir configuration parameter.

Type: str

property stagedir¶

The ReFrame stage directory prefix associated with this system.

Type: str

class reframe.core.systems.SystemPartition(parent, name, scheduler, launcher, descr, access, container_environs, resources, local_env, environs, max_jobs)[source]¶

Bases: object

A representation of a system partition inside ReFrame.

Warning

Users may not create SystemPartition objects directly.

property access¶

The scheduler options for accessing this system partition.

Type: List[str]

property container_environs¶

Environments associated with the different container platforms.

Type: Dict[str, Environment]

property descr¶

The description of this partition.

Type: str

environment(name)[source]¶: Return the partition environment named name.

property environs¶

The programming environments associated with this system partition.

Type: List[ProgEnvironment]

property fullname¶

Return the fully-qualified name of this partition.

The fully-qualified name is of the form <parent-system-name>:<partition-name>.

Type: str

json()[source]¶: Return a JSON object representing this system partition.

property launcher¶

The type of the backend launcher of this partition.

New in version 2.8.

Returns: a subclass of reframe.core.launchers.JobLauncher.

property local_env¶

The local environment associated with this partition.

Type: Environment

property max_jobs¶

The maximum number of concurrent jobs allowed on this partition.

Type: integral

property name¶

The name of this partition.

Type: str

property resources¶

The resources template strings associated with this partition.

This is a dictionary, where the key is the name of a resource and the value is the scheduler options or directives associated with this resource.

Type: Dict[str, List[str]]

property scheduler¶

The type of the backend scheduler of this partition.

Returns: a subclass of reframe.core.schedulers.JobScheduler.

Note

Changed in version 2.8: Prior versions returned a string representing the scheduler and job launcher combination.

Job Schedulers and Parallel Launchers¶

class reframe.core.schedulers.Job(name, workdir='.', script_filename=None, stdout=None, stderr=None, max_pending_time=None, sched_flex_alloc_nodes=None, sched_access=[], sched_account=None, sched_partition=None, sched_reservation=None, sched_nodelist=None, sched_exclude_nodelist=None, sched_exclusive_access=None, sched_options=None)[source]¶

Bases: object

A job descriptor.

A job descriptor is created by the framework after the “setup” phase and is associated with the test.

Warning

Users may not create a job descriptor directly.

exitcode¶

New in version 2.21.

The exit code of the job.

This may or may not be set depending on the scheduler backend.

Type: int or None.

jobid¶

New in version 2.21.

The ID of the current job.

Type: int or None.

launcher¶

The (parallel) program launcher that will be used to launch the (parallel) executable of this job.

Users are allowed to explicitly set the current job launcher, but this is only relevant in rare situations, such as when you want to wrap the current launcher command. For this specific scenario, you may have a look at the reframe.core.launchers.LauncherWrapper class.

The following example shows how you can replace the current partition’s launcher for this test with the “local” launcher:

from reframe.core.backends import getlauncher

@rfm.run_after('setup')
def set_launcher(self):
    self.job.launcher = getlauncher('local')()

Type: reframe.core.launchers.JobLauncher

nodelist¶

New in version 2.17.

The list of node names assigned to this job.

This attribute is None if no nodes are assigned to the job yet. This attribute is set reliably only for the slurm backend, i.e., Slurm with accounting enabled. The squeue scheduler backend, i.e., Slurm without accounting, might not set this attribute for jobs that finish very quickly. For the local scheduler backend, this returns an one-element list containing the hostname of the current host.

This attribute might be useful in a flexible regression test for determining the actual nodes that were assigned to the test. For more information on flexible node allocation, see the --flex-alloc-nodes command-line option

This attribute is not supported by the pbs scheduler backend.

options¶

Options to be passed to the backend job scheduler.

Type: List[str]
Default: []

state¶

New in version 2.21.

The state of the job.

The value of this field is scheduler-specific.

Type: str or None.

class reframe.core.schedulers.JobScheduler[source]¶

Bases: abc.ABC

Abstract base class for job scheduler backends.

class reframe.core.launchers.JobLauncher[source]¶

Bases: abc.ABC

Abstract base class for job launchers.

A job launcher is the executable that actually launches a distributed program to multiple nodes, e.g., mpirun, srun etc.

Warning

Users may not create job launchers directly.

Note

Changed in version 2.8: Job launchers do not get a reference to a job during their initialization.

options¶

List of options to be passed to the job launcher invocation.

Type: List[str]
Default: []

class reframe.core.launchers.LauncherWrapper(target_launcher, wrapper_command, wrapper_options=[])[source]¶

Bases: reframe.core.launchers.JobLauncher

Wrap a launcher object so as to modify its invocation.

This is useful for parallel debuggers. For example, to launch a regression test using the ARM DDT debugger, you can do the following:

@rfm.run_after('setup')
def set_launcher(self):
    self.job.launcher = LauncherWrapper(self.job.launcher, 'ddt',
                                        ['--offline'])

If the current system partition uses native Slurm for job submission, this setup will generate the following command in the submission script:

ddt --offline srun <test_executable>

If the current partition uses mpirun instead, it will generate

ddt --offline mpirun -np <num_tasks> ... <test_executable>

Parameters

target_launcher – The launcher to wrap.
wrapper_command – The wrapper command.
wrapper_options – List of options to pass to the wrapper command.

reframe.core.backends.getlauncher(name)¶

Retrieve the reframe.core.launchers.JobLauncher concrete implementation for a parallel launcher backend.

Parameters: name – The registered name of the launcher backend.

reframe.core.backends.getscheduler(name)¶

Retrieve the reframe.core.schedulers.JobScheduler concrete implementation for a scheduler backend.

Parameters: name – The registered name of the scheduler backend.

Runtime Services¶

class reframe.core.runtime.RuntimeContext(site_config)[source]¶

Bases: object

The runtime context of the framework.

There is a single instance of this class globally in the framework.

New in version 2.13.

get_option(option)[source]¶

Get a configuration option.

Parameters: option – The option to be retrieved.
Returns: The value of the option.

property modules_system¶

The environment modules system used in the current host.

Type: reframe.core.modules.ModulesSystem.

property output_prefix¶

The output directory prefix.

Type: str

property stage_prefix¶

The stage directory prefix.

Type: str

property system¶

The current host system.

Type: reframe.core.systems.System

reframe.core.runtime.is_env_loaded(environ)[source]¶

Check if environment is loaded.

Parameters: environ (Environment) – Environment to check for.
Returns: True if this environment is loaded, False otherwise.

reframe.core.runtime.loadenv(*environs)[source]¶

Load environments in the current Python context.

Parameters: environs (List[Environment]) – A list of environments to load.
Returns: A tuple containing snapshot of the current environment upon entry to this function and a list of shell commands required to load the environments.
Return type: Tuple[_EnvironmentSnapshot, List[str]]

class reframe.core.runtime.module_use(*paths)[source]¶

Bases: object

Context manager for temporarily modifying the module path.

reframe.core.runtime.runtime()[source]¶

Get the runtime context of the framework.

New in version 2.13.

Returns: A reframe.core.runtime.RuntimeContext object.

class reframe.core.runtime.temp_environment(modules=[], variables=[])[source]¶

Bases: object

Context manager to temporarily change the environment.

Modules Systems¶

class reframe.core.modules.ModulesSystem(backend)[source]¶

Bases: object

A modules system.

conflicted_modules(name)[source]¶

Return the list of the modules conflicting with module name.

If module name resolves to multiple real modules, then the returned list will be the concatenation of the conflict lists of all the real modules.

Return type: List[str]

emit_load_commands(name)[source]¶

Return the appropriate shell command for loading module name.

Return type: List[str]

emit_unload_commands(name)[source]¶

Return the appropriate shell command for unloading module name.

Return type: List[str]

is_module_loaded(name)[source]¶

Check if module name is loaded.

If module name refers to multiple real modules, this method will return True only if all the referees are loaded.

load_module(name, force=False)[source]¶

Load the module name.

If force is set, forces the loading, unloading first any conflicting modules currently loaded. If module name refers to multiple real modules, all of the target modules will be loaded.

Returns: the list of unloaded modules as strings.
Return type: List[str]

loaded_modules()[source]¶

Return a list of loaded modules.

Return type: List[str]

property name¶: The name of this module system.

property searchpath¶: The module system search path as a list of directories.

searchpath_add(*dirs)[source]¶: Add dirs to the module system search path.

searchpath_remove(*dirs)[source]¶: Remove dirs from the module system search path.

unload_all()[source]¶: Unload all loaded modules.

unload_module(name)[source]¶

Unload module name.

If module name refers to multiple real modules, all the referred to modules will be unloaded in reverse order.

property version¶: The version of this module system.

Build Systems¶

New in version 2.14.

ReFrame delegates the compilation of the regression test to a build system. Build systems in ReFrame are entities that are responsible for generating the necessary shell commands for compiling a code. Each build system defines a set of attributes that users may set in order to customize their compilation. An example usage is the following:

self.build_system = 'SingleSource'
self.build_system.cflags = ['-fopenmp']

Users simply set the build system to use in their regression tests and then they configure it. If no special configuration is needed for the compilation, users may completely ignore the build systems. ReFrame will automatically pick one based on the regression test attributes and will try to compile the code.

All build systems in ReFrame derive from the abstract base class reframe.core.buildsystems.BuildSystem. This class defines a set of common attributes, such us compilers, compilation flags etc. that all subclasses inherit. It is up to the concrete build system implementations on how to use or not these attributes.

class reframe.core.buildsystems.Autotools[source]¶

Bases: reframe.core.buildsystems.ConfigureBasedBuildSystem

A build system for compiling Autotools-based projects.

This build system will emit the following commands:

Create a build directory if builddir is not None and change to it.
Invoke configure to configure the project by setting the corresponding flags for compilers and compiler flags.
Issue make to compile the code.

emit_build_commands(environ)[source]¶

Return the list of commands for building using this build system.

The build commands may always assume to be issued from the top-level directory of the code that is to be built.

Parameters: environ (reframe.core.environments.ProgEnvironment) – The programming environment for which to emit the build instructions. The framework passes here the current programming environment.
Raises: BuildSystemError in case of errors when generating the build instructions.

Note

This method is relevant only to developers of new build systems.

class reframe.core.buildsystems.BuildSystem[source]¶

Bases: abc.ABC

The abstract base class of any build system.

Concrete build systems inherit from this class and must override the emit_build_commands() abstract function.

cc¶

The C compiler to be used. If empty and flags_from_environ is True, the compiler defined in the current programming environment will be used.

Type: str
Default: ''

cflags¶

The C compiler flags to be used. If empty and flags_from_environ is True, the corresponding flags defined in the current programming environment will be used.

Type: List[str]
Default: []

cppflags¶

The preprocessor flags to be used. If empty and flags_from_environ is True, the corresponding flags defined in the current programming environment will be used.

Type: List[str]
Default: []

cxx¶

The C++ compiler to be used. If empty and flags_from_environ is True, the compiler defined in the current programming environment will be used.

Type: str
Default: ''

cxxflags¶

The C++ compiler flags to be used. If empty and flags_from_environ is True, the corresponding flags defined in the current programming environment will be used.

Type: List[str]
Default: []

abstract emit_build_commands(environ)[source]¶

Return the list of commands for building using this build system.

The build commands may always assume to be issued from the top-level directory of the code that is to be built.

Parameters: environ (reframe.core.environments.ProgEnvironment) – The programming environment for which to emit the build instructions. The framework passes here the current programming environment.
Raises: BuildSystemError in case of errors when generating the build instructions.

Note

This method is relevant only to developers of new build systems.

fflags¶

The Fortran compiler flags to be used. If empty and flags_from_environ is True, the corresponding flags defined in the current programming environment will be used.

Type: List[str]
Default: []

flags_from_environ¶

Set compiler and compiler flags from the current programming environment if not specified otherwise.

Type: bool
Default: True

ftn¶

The Fortran compiler to be used. If empty and flags_from_environ is True, the compiler defined in the current programming environment will be used.

Type: str
Default: ''

ldflags¶

The linker flags to be used. If empty and flags_from_environ is True, the corresponding flags defined in the current programming environment will be used.

Type: List[str]
Default: []

nvcc¶

The CUDA compiler to be used. If empty and flags_from_environ is True, the compiler defined in the current programming environment will be used.

Type: str
Default: ''

class reframe.core.buildsystems.CMake[source]¶

Bases: reframe.core.buildsystems.ConfigureBasedBuildSystem

A build system for compiling CMake-based projects.

This build system will emit the following commands:

Create a build directory if builddir is not None and change to it.
Invoke cmake to configure the project by setting the corresponding CMake flags for compilers and compiler flags.
Issue make to compile the code.

emit_build_commands(environ)[source]¶

Return the list of commands for building using this build system.

The build commands may always assume to be issued from the top-level directory of the code that is to be built.

Parameters: environ (reframe.core.environments.ProgEnvironment) – The programming environment for which to emit the build instructions. The framework passes here the current programming environment.
Raises: BuildSystemError in case of errors when generating the build instructions.

Note

This method is relevant only to developers of new build systems.

class reframe.core.buildsystems.ConfigureBasedBuildSystem[source]¶

Bases: reframe.core.buildsystems.BuildSystem

Abstract base class for configured-based build systems.

builddir¶

The CMake build directory, where all the generated files will be placed.

Type: str
Default: None

config_opts¶

Additional configuration options to be passed to the CMake invocation.

Type: List[str]
Default: []

make_opts¶

Options to be passed to the subsequent make invocation.

Type: List[str]
Default: []

max_concurrency¶

Same as for the Make build system.

Type: integer
Default: 1

srcdir¶

The top-level directory of the code.

This is set automatically by the framework based on the reframe.core.pipeline.RegressionTest.sourcepath attribute.

Type: str
Default: None

class reframe.core.buildsystems.Make[source]¶

Bases: reframe.core.buildsystems.BuildSystem

A build system for compiling codes using make.

The generated build command has the following form:

make -j [N] [-f MAKEFILE] [-C SRCDIR] CC="X" CXX="X" FC="X" NVCC="X" CPPFLAGS="X" CFLAGS="X" CXXFLAGS="X" FCFLAGS="X" LDFLAGS="X" OPTIONS

The compiler and compiler flags variables will only be passed if they are not None. Their value is determined by the corresponding attributes of BuildSystem. If you want to completely disable passing these variables to the make invocation, you should make sure not to set any of the correspoding attributes and set also the BuildSystem.flags_from_environ flag to False.

emit_build_commands(environ)[source]¶

Return the list of commands for building using this build system.

The build commands may always assume to be issued from the top-level directory of the code that is to be built.

Parameters: environ (reframe.core.environments.ProgEnvironment) – The programming environment for which to emit the build instructions. The framework passes here the current programming environment.
Raises: BuildSystemError in case of errors when generating the build instructions.

Note

This method is relevant only to developers of new build systems.

makefile¶

Instruct build system to use this Makefile. This option is useful when having non-standard Makefile names.

Type: str
Default: None

max_concurrency¶

Limit concurrency for make jobs. This attribute controls the -j option passed to make. If not None, make will be invoked as make -j max_concurrency. Otherwise, it will invoked as make -j.

Type: integer
Default: 1

Note

Changed in version 2.19: The default value is now 1

options¶

Append these options to the make invocation. This variable is also useful for passing variables or targets to make.

Type: List[str]
Default: []

srcdir¶

The top-level directory of the code.

This is set automatically by the framework based on the reframe.core.pipeline.RegressionTest.sourcepath attribute.

Type: str
Default: None

class reframe.core.buildsystems.SingleSource[source]¶

Bases: reframe.core.buildsystems.BuildSystem

A build system for compiling a single source file.

The generated build command will have the following form:

COMP CPPFLAGS XFLAGS SRCFILE -o EXEC LDFLAGS

COMP is the required compiler for compiling SRCFILE. This build system will automatically detect the programming language of the source file and pick the correct compiler. See also the SingleSource.lang attribute.
CPPFLAGS are the preprocessor flags and are passed to any compiler.
XFLAGS is any of CFLAGS, CXXFLAGS or FCFLAGS depending on the programming language of the source file.
SRCFILE is the source file to be compiled. This is set up automatically by the framework. See also the SingleSource.srcfile attribute.
EXEC is the executable to be generated. This is also set automatically by the framework. See also the SingleSource.executable attribute.
LDFLAGS are the linker flags.

For CUDA codes, the language assumed is C++ (for the compilation flags) and the compiler used is BuildSystem.nvcc.

emit_build_commands(environ)[source]¶

Return the list of commands for building using this build system.

The build commands may always assume to be issued from the top-level directory of the code that is to be built.

Parameters: environ (reframe.core.environments.ProgEnvironment) – The programming environment for which to emit the build instructions. The framework passes here the current programming environment.
Raises: BuildSystemError in case of errors when generating the build instructions.

Note

This method is relevant only to developers of new build systems.

executable¶

The executable file to be generated.

This is set automatically by the framework based on the reframe.core.pipeline.RegressionTest.executable attribute.

Type: str or None

include_path¶

The include path to be used for this compilation.

All the elements of this list will be appended to the BuildSystem.cppflags, by prepending to each of them the -I option.

Type: List[str]
Default: []

lang¶

The programming language of the file that needs to be compiled. If not specified, the build system will try to figure it out automatically based on the extension of the source file. The automatically detected extensions are the following:

C: .c and .upc.

C++: .cc, .cp, .cxx, .cpp, .CPP, .c++ and .C.

Fortran: .f, .for, .ftn, .F, .FOR, .fpp, .FPP, .FTN, .f90, .f95, .f03, .f08, .F90, .F95, .F03 and .F08.

CUDA: .cu.

Type: str or None

srcfile¶

The source file to compile. This is automatically set by the framework based on the reframe.core.pipeline.RegressionTest.sourcepath attribute.

Type: str or None

Container Platforms¶

New in version 2.20.

class reframe.core.containers.ContainerPlatform[source]¶

Bases: abc.ABC

The abstract base class of any container platform.

commands¶

The commands to be executed within the container.

Type: list[str]
Default: []

image¶

The container image to be used for running the test.

Type: str or None
Default: None

mount_points¶

List of mount point pairs for directories to mount inside the container.

Each mount point is specified as a tuple of (/path/in/host, /path/in/container).

Type: list[tuple[str, str]]
Default: []

options¶

Additional options to be passed to the container runtime when executed.

Type: list[str]
Default: []

workdir¶

The working directory of ReFrame inside the container.

This is the directory where the test’s stage directory is mounted inside the container. This directory is always mounted regardless if mount_points is set or not.

Type: str
Default: /rfm_workdir

class reframe.core.containers.Docker[source]¶

Bases: reframe.core.containers.ContainerPlatform

Container platform backend for running containers with Docker.

class reframe.core.containers.Sarus[source]¶

Bases: reframe.core.containers.ContainerPlatform

Container platform backend for running containers with Sarus.

with_mpi¶

Enable MPI support when launching the container.

Type: boolean
Default: False

class reframe.core.containers.Shifter[source]¶

Bases: reframe.core.containers.Sarus

Container platform backend for running containers with Shifter.

class reframe.core.containers.Singularity[source]¶

Bases: reframe.core.containers.ContainerPlatform

Container platform backend for running containers with Singularity.

with_cuda¶

Enable CUDA support when launching the container.

Type: boolean
Default: False

The `reframe` module¶

The reframe module offers direct access to the basic test classes, constants and decorators.

class reframe.CompileOnlyRegressionTest¶: See reframe.core.pipeline.CompileOnlyRegressionTest.

class reframe.RegressionTest¶: See reframe.core.pipeline.RegressionTest.

class reframe.RunOnlyRegressionTest¶: See reframe.core.pipeline.RunOnlyRegressionTest.

reframe.DEPEND_BY_ENV¶: See reframe.core.pipeline.DEPEND_BY_ENV.

reframe.DEPEND_EXACT¶: See reframe.core.pipeline.DEPEND_EXACT.

reframe.DEPEND_FULLY¶: See reframe.core.pipeline.DEPEND_FULLY.

@reframe.parameterized_test¶: See @reframe.core.decorators.parameterized_test.

@reframe.require_deps¶: See @reframe.core.decorators.require_deps.

@reframe.required_version¶: See @reframe.core.decorators.required_version.

@reframe.run_after¶: See @reframe.core.decorators.run_after.

@reframe.run_before¶: See @reframe.core.decorators.run_before.

@reframe.simple_test¶: See @reframe.core.decorators.simple_test.

Mapping of Test Attributes to Job Scheduler Backends¶

Test attribute	Slurm option	Torque option	PBS option
`num_tasks`	`--ntasks`¹	`-l nodes={num_tasks//num_tasks_per_node}:ppn={num_tasks_per_node*num_cpus_per_task}`	`-l select={num_tasks//num_tasks_per_node}:mpiprocs={num_tasks_per_node}:ncpus={num_tasks_per_node*num_cpus_per_task}`
`num_tasks_per_node`	`--ntasks-per-node`	see `num_tasks`	see `num_tasks`
`num_tasks_per_core`	`--ntasks-per-core`	n/a	n/a
`num_tasks_per_socket`	`--ntasks-per-socket`	n/a	n/a
`num_cpus_per_task`	`--cpus-per-task`	see `num_tasks`	see `num_tasks`
`time_limit`	`--time=hh:mm:ss`	`-l walltime=hh:mm:ss`	`-l walltime=hh:mm::ss`
`exclusive_access`	`--exclusive`	n/a	n/a
`use_smt`	`--hint=[no]multithread`	n/a	n/a

If any of the attributes is set to None it will not be emitted at all in the job script. In cases that the attribute is required, it will be set to 1.

¹ The --nodes option may also be emitted if the use_nodes_option scheduler configuration parameter is set.

ReFrame Programming APIs¶

Regression Test Base Classes¶

Regression Test Class Decorators¶

Pipeline Hooks¶

Environments and Systems¶

Job Schedulers and Parallel Launchers¶

Runtime Services¶

Modules Systems¶

Build Systems¶

Container Platforms¶

The reframe module¶

Mapping of Test Attributes to Job Scheduler Backends¶

The `reframe` module¶