Reference Guide

class reframe.core.pipeline.CompileOnlyRegressionTest(*args, **kwargs)[source]

Bases: reframe.core.pipeline.RegressionTest

Base class for compile-only regression tests.

These tests are by default local and will skip the run phase of the regression test pipeline.

The standard output and standard error of the test will be set to those of the compilation stage.

compile(**compile_opts)[source]

The compilation stage of the regression test pipeline.

The standard output and standard error of this stage will be used as the standard output and error of the test.

run()[source]

The run stage of the regression test pipeline.

Implemented as no-op.

setup(partition, environ, **job_opts)[source]

The setup stage of the regression test pipeline.

Similar to the RegressionTest.setup(), except that no job descriptor is set up for this test.

wait()[source]

Wait for this test to finish.

Implemented as no-op

class reframe.core.pipeline.RegressionTest(name, prefix, system, resources)[source]

Bases: object

Base class for regression tests.

All regression tests must eventually inherit from this class. This class provides the implementation of the pipeline phases that the regression test goes through during its lifetime.

Parameters:
  • name – The name of the test. This is the only argument that the users may specify freely.
  • prefix – The directory prefix of the test. You should initialize this to the directory containing the file that defines the regression test. You can achieve this by always passing os.path.dirname(__file__).
  • system – The system that this regression test will run on. The framework takes care of initializing and passing correctly this argument.
  • resources – An object managing the framework’s resources. The framework takes care of initializing and passing correctly this argument.

Concrete regression test subclasses should call the base constructor as follows:

class MyTest(RegressionTest):
    def __init__(self, my_test_args, **kwargs):
        super().__init__('mytest', os.path.dirname(__file__), **kwargs)
check_performance()[source]

The performance checking phase of the regression test pipeline.

Raises:reframe.core.exceptions.SanityError – If the performance check fails.
check_sanity()[source]

The sanity checking phase of the regression test pipeline.

Raises:reframe.core.exceptions.SanityError – If the sanity check fails.
cleanup(remove_files=False, unload_env=True)[source]

The cleanup phase of the regression test pipeline.

Parameters:
  • remove_files – If True, the stage directory associated with this test will be removed.
  • unload_env – If True, the environment that was used to run this test will be unloaded.
compile(**compile_opts)[source]

The compilation phase of the regression test pipeline.

Parameters:compile_opts – Extra options to be passed to the programming environment for compiling the source code of the test.
Raises:reframe.core.exceptions.ReframeError – In case of errors.
current_environ

The programming environment that the regression test is currently executing with.

This is set by the framework during the setup() phase.

Type:reframe.core.environments.Environment.
current_partition

The system partition the regression test is currently executing on.

This is set by the framework during the setup() phase.

Type:reframe.core.systems.SystemPartition.
current_system

The system the regression test is currently executing on.

This is set by the framework during the initialization phase.

Type:reframe.core.systems.System.
descr

A detailed description of the test.

Type:str
Default:self.name
exclusive_access

Specify whether this test needs exclusive access to nodes.

Type:boolean
Default:False
executable

The name of the executable to be launched during the run phase.

Type:str
Default:os.path.join('.', self.name)
executable_opts

List of options to be passed to the executable.

Type:list[str]
Default:[]
extra_resources

Extra resources for this test.

This field is for specifying custom resources needed by this test. These resources are defined in the configuration of a system partition. For example, assume that two additional resources, named gpu and datawarp, are defined in the configuration file as follows:

'resources': {
    'gpu': [
        '--gres=gpu:{num_gpus_per_node}'
    ],
    'datawarp': [
        '#DW jobdw capacity={capacity}',
        '#DW stage_in source={stagein_src}'
    ]
}

A regression test then may instantiate the above resources by setting the extra_resources attribute as follows:

self.extra_resources = {
    'gpu': {'num_gpus_per_node': 2}
    'datawarp': {
        'capacity': '100GB',
        'stagein_src': '/foo'
    }
}

The generated batch script (for Slurm) will then contain the following lines:

#SBATCH --gres=gpu:2
#DW jobdw capacity=100GB
#DW stage_in source=/foo

Notice that if the resource specified in the configuration uses an alternative directive prefix (in this case #DW), this will replace the standard prefix of the backend scheduler (in this case #SBATCH)

If the resource name specified in this variable does not match a resource name in the partition configuration, it will be simply ignored. The num_gpus_per_node attribute translates internally to the _rfm_gpu resource, so that setting self.num_gpus_per_node = 2 is equivalent to the following:

self.extra_resources = {'_rfm_gpu': {'num_gpus_per_node': 2}}
Type:dict[str, dict[str, object]]
Default:{}

Note

New in version 2.8.

Changed in version 2.9.

A new more powerful syntax was introduced that allows also custom job script directive prefixes.

info()[source]

Provide live information of a running test.

This method is used by the front-end to print the status message during the test’s execution.

Returns:a string with an informational message containing the test name, the current partition and the current programming environment that the test is currently executing on.
is_local()[source]

Check if the test will execute locally.

A test executes locally if the local attribute is set or if the current partition’s scheduler does not support job submission.

job

The job descriptor associated with this test.

This is set by the framework during the setup() phase.

Type:reframe.core.schedulers.Job.
keep_files

List of files to be kept after the test finishes.

By default, the framework saves the standard output, the standard error and the generated shell script that was used to run this test.

These files will be copied over to the framework’s output directory during the cleanup() phase.

Directories are also accepted in this field.

Relative path names are resolved against the stage directory.

Type:list[str]
Default:[]
local

Always execute this test locally.

Type:boolean
Default:False
logger

A logger associated with the this test.

You can use this logger to log information for your test.

maintainers

List of people responsible for this test.

When the test fails, this contact list will be printed out.

Type:list[str]
Default:[]
modules

List of modules to be loaded before running this test.

These modules will be loaded during the setup() phase.

Type:list[str]
Default:[]
name

The name of the test.

Type:Alphanumeric string.
num_cpus_per_task

Number of CPUs per task required by this test.

Ignored if None.

Type:integral or None
Default:None
num_gpus_per_node

Number of GPUs per node required by this test.

Type:integral
Default:0
num_tasks

Number of tasks required by this test.

If the number of tasks is set to 0, ReFrame will try to use all the available nodes of a reservation. A reservation must be specified through the –reservation command-line option, otherwise the regression test will fail during submission. ReFrame will try to run the test on all the nodes of the reservation that satisfy the selection criteria of the current virtual partition (i.e., constraints and/or partitions).

Type:integral
Default:1

Note

Changed in version 2.9: Added support for running the test using all the nodes of the specified reservation if the number of tasks is set to 0.

num_tasks_per_core

Number of tasks per core required by this test.

Ignored if None.

Type:integral or None
Default:None
num_tasks_per_node

Number of tasks per node required by this test.

Ignored if None.

Type:integral or None
Default:None
num_tasks_per_socket

Number of tasks per socket required by this test.

Ignored if None.

Type:integral or None
Default:None
perf_patterns

Patterns for verifying the performance of this test.

Refer to the ReFrame Tutorial for concrete usage examples.

If set to None, no performance checking will be performed.

Type:A dictionary with keys of type str and deferrable expressions (i.e., the result of a sanity function) as values. None is also allowed.
Default:None
poll()[source]

Poll the test’s state.

Returns:True if the associated job has finished, False otherwise.

If no job descriptor is yet associated with this test, True is returned.

Raises:reframe.core.exceptions.ReframeError – In case of errors.
post_run

List of shell commands to execute after launching this job.

See pre_run for a more detailed description of the semantics.

Type:list of str
Default:[]

Note

New in version 2.10.

postbuild_cmd

List of shell commands to be executed after a successful compilation.

These commands are executed during the compilation phase and from inside the stage directory. Each entry in the list spawns a new shell.

Type:list[str]
Default:[]
pre_run

List of shell commands to execute before launching this job.

These commands do not execute in the context of ReFrame. Instead, they are emitted in the generated job script just before the actual job launch command.

Type:list of str
Default:[]

Note

New in version 2.10.

prebuild_cmd

List of shell commands to be executed before compiling.

These commands are executed during the compilation phase and from inside the stage directory. Each entry in the list spawns a new shell.

Type:list[str]
Default:[]
prefix

The prefix directory of the test.

Type:str.
readonly_files

List of files or directories (relative to the sourcesdir) that will be symlinked in the stage directory and not copied.

You can use this variable to avoid copying very large files to the stage directory.

Type:list[str]
Default:[]
reference

The set of reference values for this test.

Refer to the ReFrame Tutorial for concrete usage examples.

Type:A scoped dictionary with system names as scopes or None
Default:{}
run()[source]

The run phase of the regression test pipeline.

This call is non-blocking. It simply submits the job associated with this test and returns.

sanity_patterns

Refer to the ReFrame Tutorial for concrete usage examples.

If set to None, a sanity error will be raised during sanity checking.

Type:A deferrable expression (i.e., the result of a sanity function) or None
Default:None

Note

Changed in version 2.9: The default behaviour has changed and it is now considered a sanity failure if this attribute is set to None.

If a test doesn’t care about its output, this must be stated explicitly as follows:

self.sanity_patterns = sn.assert_found(r'.*', self.stdout)
setup(partition, environ, **job_opts)[source]

The setup phase of the regression test pipeline.

Parameters:
  • partition – The system partition to set up this test for.
  • environ – The environment to set up this test for.
  • job_opts – Options to be passed through to the backend scheduler. When overriding this method users should always pass through job_opts to the base class method.
Raises:

reframe.core.exceptions.ReframeError – In case of errors.

sourcepath

The path to the source file or source directory of the test.

It must be a path relative to the sourcesdir, pointing to a subfolder or a file contained in sourcesdir. This applies also in the case where sourcesdir is a Git repository.

If it refers to a regular file, this file will be compiled (its language will be automatically recognized). If it refers to a directory, make will be invoked in that directory.

Type:str
Default:''
sourcesdir

The directory containing the test’s resources.

This directory may be specified with an absolute path or with a path relative to the location of the test. Its contents will always be copied to the stage directory of the test.

This attribute may also accept a URL, in which case ReFrame will treat it as a Git repository and will try to clone its contents in the stage directory of the test.

If set to None, the test has no resources an no action is taken.

Type:str or None
Default:'src'

Note

Changed in version 2.9: Allow None values to be set also in regression tests with a compilation phase

Changed in version 2.10: Support for Git repositories was added.

stagedir

The stage directory of the test.

This is set during the setup() phase.

Type:str.
stderr

The name of the file containing the standard error of the test.

This is set during the setup() phase.

This attribute is evaluated lazily, so it can by used inside sanity expressions.

Type:str.
stdout

The name of the file containing the standard output of the test.

This is set during the setup() phase.

This attribute is evaluated lazily, so it can by used inside sanity expressions.

Type:str.
strict_check

Mark this test as a strict performance test.

If a test is marked as non-strict, the performance checking phase will always succeed, unless the --strict command-line option is passed when invoking ReFrame.

Type:boolean
Default:True
tags

Set of tags associated with this test.

This test can be selected from the frontend using any of these tags.

Type:set[str]
Default:an empty set
time_limit

Time limit for this test.

Time limit is specified as a three-tuple in the form (hh, mm, ss), with hh >= 0, 0 <= mm <= 59 and 0 <= ss <= 59.

Type:tuple[int]
Default:(0, 10, 0)
use_multithreading

Specify whether this tests needs simultaneous multithreading enabled.

Ignored if None.

Type:boolean or None
Default:None
valid_prog_environs

List of programming environmets supported by this test.

Type:list[str]
Default:[]
valid_systems

List of systems supported by this test. The general syntax for systems is <sysname>[:<partname].

Type:list[str]
Default:[]
variables

Environment variables to be set before running this test.

These variables will be set during the setup() phase.

Type:dict[str, str]
Default:{}
wait()[source]

Wait for this test to finish.

Raises:reframe.core.exceptions.ReframeError – In case of errors.
class reframe.core.pipeline.RunOnlyRegressionTest(name, prefix, system, resources)[source]

Bases: reframe.core.pipeline.RegressionTest

Base class for run-only regression tests.

compile(**compile_opts)[source]

The compilation phase of the regression test pipeline.

This is a no-op for this type of test.

run()[source]

The run phase of the regression test pipeline.

The resources of the test are copied to the stage directory and the rest of execution is delegated to the RegressionTest.run().

class reframe.core.environments.Environment(name, modules=[], variables={}, **kwargs)[source]

Bases: object

This class abstracts away an environment to run regression tests.

It is simply a collection of modules to be loaded and environment variables to be set when this environment is loaded by the framework. Users may not create or modify directly environments.

is_loaded

True if this environment is loaded, False otherwise.

modules

The modules associated with this environment.

Type:list of str
name

The name of this environment.

Type:str
variables

The environment variables associated with this environment.

Type:dictionary of str keys/values.
class reframe.core.environments.EnvironmentSnapshot(name='env_snapshot')[source]

Bases: reframe.core.environments.Environment

class reframe.core.environments.ProgEnvironment(name, modules=[], variables={}, cc='cc', cxx='CC', ftn='ftn', cppflags=None, cflags=None, cxxflags=None, fflags=None, ldflags=None, **kwargs)[source]

Bases: reframe.core.environments.Environment

A class representing a programming environment.

This type of environment adds also attributes for setting the compiler and compilation flags.

If compilation flags are set to None (the default, if not set otherwise in ReFrame’s configuration), they are not passed to the make invocation.

If you want to disable completely the propagation of the compilation flags to the make invocation, even if they are set, you should set the propagate attribute to False.

cc

The C compiler of this programming environment.

Type:str
cflags

The C compiler flags of this programming environment.

Type:str or None
cppflags

The preprocessor flags of this programming environment.

Type:str or None
cxx

The C++ compiler of this programming environment.

Type:str or None
cxxflags

The C++ compiler flags of this programming environment.

Type:str or None
fflags

The Fortran compiler flags of this programming environment.

Type:str or None
ftn

The Fortran compiler of this programming environment.

Type:str or None
include_search_path

The include search path of this programming environment.

Type:list of str
Default:[]
ldflags

The linker flags of this programming environment.

Type:str or None
propagate

Propagate the compilation flags to the make invocation.

Type:bool
Default:True
class reframe.core.environments.save_environment[source]

Bases: object

A context manager for saving and restoring the current environment.

class reframe.core.systems.System(name, descr=None, hostnames=[], partitions=[], prefix='.', stagedir=None, outputdir=None, logdir=None, resourcesdir='.', modules_system=None)[source]

Bases: object

A representation of a system inside ReFrame.

descr

The description of this system.

name

The name of this system.

partition(name)[source]

Get system partition with name.

Returns:the requested SystemPartition, or None if not found.
partitions

Get all the active partitions of this system.

Returns:a list of SystemPartition.
resourcesdir

Global resources directory for this system

You may use this directory for storing large resource files of your regression tests. See here on how to configure this.

Type:str
class reframe.core.systems.SystemPartition(name, descr=None, scheduler=None, launcher=None, access=[], environs=[], resources={}, local_env=None, max_jobs=1)[source]

Bases: object

A representation of a system partition inside ReFrame.

descr

A detailed description of this partition.

fullname

Return the fully-qualified name of this partition.

The fully-qualified name is of the form <parent-system-name>:<partition-name>.

Type:str
launcher

The type of the backend launcher of this partition.

Returns:a subclass of reframe.core.launchers.JobLauncher.

Note

New in version 2.8.

name

The name of this partition.

Type:str
scheduler

The type of the backend scheduler of this partition.

Returns:a subclass of reframe.core.schedulers.Job.

Note

Changed in version 2.8.

Prior versions returned a string representing the scheduler and job launcher combination.

class reframe.core.schedulers.Job(name, command, launcher, environs=[], workdir='.', num_tasks=1, num_tasks_per_node=None, num_tasks_per_core=None, num_tasks_per_socket=None, num_cpus_per_task=None, use_smt=None, time_limit=(0, 10, 0), script_filename=None, stdout=None, stderr=None, pre_run=[], post_run=[], sched_account=None, sched_partition=None, sched_reservation=None, sched_nodelist=None, sched_exclude_nodelist=None, sched_exclusive_access=None, sched_options=[])[source]

Bases: abc.ABC

A job descriptor.

Note

This is an abstract class. Users may not create jobs directly.

launcher

The parallel program launcher that will be used to launch the parallel executable of this job.

Type:reframe.core.launchers.JobLauncher
options

Options to be passed to the backend job scheduler.

Type:list of str
Default:[]
post_run

List of shell commands to execute after launching this job.

Type:list of str
Default:[]

Note

Deprecated since version 2.10: Please use the reframe.core.pipeline.RegressionTest.post_run field instead.

pre_run

List of shell commands to execute before launching this job.

Type:list of str
Default:[]

Note

Deprecated since version 2.10: Please use the reframe.core.pipeline.RegressionTest.pre_run field instead.

class reframe.core.launchers.JobLauncher(options=[])[source]

Bases: abc.ABC

A job launcher.

A job launcher is the executable that actually launches a distributed program to multiple nodes, e.g., mpirun, srun etc.

Note

This is an abstract class. Regression tests may not instantiate this class directly.

Note

Changed in version 2.8: Job launchers do not get a reference to a job during their initialization.

options

List of options to be passed to the job launcher invocation.

Type:list of str
Default:[]
class reframe.core.launchers.LauncherWrapper(target_launcher, wrapper_command, wrapper_options=[])[source]

Bases: reframe.core.launchers.JobLauncher

Wrap a launcher object so as to modify its invocation.

This is useful for parallel debuggers. For example, to launch a regression test using the DDT debugger, you can do the following:

def setup(self, partition, environ, **job_opts):
    super().setup(partition, environ, **job_opts)
    self.job.launcher = LauncherWrapper(self.job.launcher, 'ddt',
                                        ['--offline'])

If the current system partition uses native Slurm for job submission, this setup will generate the following command in the submission script:

ddt --offline srun <test_executable>

If the current partition uses mpirun instead, it will generate

ddt --offline mpirun -np <num_tasks> ... <test_executable>
Parameters:
  • target_launcher – The launcher to wrap.
  • wrapper_command – The wrapper command.
  • wrapper_options – List of options to pass to the wrapper command.
reframe.core.launchers.registry.getlauncher(name)[source]

Get launcher by its registered name.

The available names are those specified in the configuration file.

This method may become handy in very special situations, e.g., testing an application that needs to replace the system partition launcher or if a different launcher must be used for a different programming environment.

For example, if you want to replace the current partition’s launcher with the local one, here is how you can achieve it:

def setup(self, partition, environ, **job_opts):
    super().setup(partition, environ, **job_opts)
    self.job.launcher = getlauncher('local')()

Note that this method returns a launcher class type and not an instance of that class. You have to instantiate it explicitly before assigning it to the launcher attribute of the job.

Parameters:name – The name of the launcher to retrieve.
Returns:The class of the launcher requested, which is a subclass of reframe.core.launchers.JobLauncher.
Raises:reframe.core.exceptions.ConfigError – if no launcher is registered with that name.

Note

New in version 2.8.

reframe.core.launchers.registry.register_launcher(name, local=False)[source]

Class decorator for registering new job launchers.

Parameters:
  • name – The registration name of this launcher
  • localTrue if launcher may only submit local jobs, False otherwise.
Raises:

ValueError – if a job launcher is already registered with the same name.

Note

New in version 2.8.

This method is only relevant to developers of new job launchers.