Reference Guide¶
This page provides a reference guide of the ReFrame API for writing regression tests covering all the relevant details. Internal data structures and APIs are covered only to the extent that might be helpful to the final user of the framework.
Environments and Systems¶
-
class
reframe.core.environments.
Environment
(name, modules=[], variables=[])[source]¶ Bases:
object
This class abstracts away an environment to run regression tests.
It is simply a collection of modules to be loaded and environment variables to be set when this environment is loaded by the framework. Users may not create or modify directly environments.
-
is_loaded
¶ True
if this environment is loaded,False
otherwise.
-
-
class
reframe.core.environments.
EnvironmentSnapshot
(name='env_snapshot')[source]¶ Bases:
reframe.core.environments.Environment
-
is_loaded
¶ True
if this environment is loaded,False
otherwise.
-
-
class
reframe.core.environments.
ProgEnvironment
(name, modules=[], variables={}, cc='cc', cxx='CC', ftn='ftn', nvcc='nvcc', cppflags=None, cflags=None, cxxflags=None, fflags=None, ldflags=None, **kwargs)[source]¶ Bases:
reframe.core.environments.Environment
A class representing a programming environment.
This type of environment adds also attributes for setting the compiler and compilation flags.
If compilation flags are set to
None
(the default, if not set otherwise in ReFrame’s configuration), they are not passed to themake
invocation.If you want to disable completely the propagation of the compilation flags to the
make
invocation, even if they are set, you should set thepropagate
attribute toFalse
.
-
class
reframe.core.environments.
save_environment
[source]¶ Bases:
object
A context manager for saving and restoring the current environment.
-
class
reframe.core.systems.
System
(name, descr=None, hostnames=[], partitions=[], preload_env=None, prefix='.', stagedir=None, outputdir=None, perflogdir=None, resourcesdir='.', modules_system=None)[source]¶ Bases:
object
A representation of a system inside ReFrame.
-
descr
¶ The description of this system.
-
hostnames
¶ The hostname patterns associated with this system.
-
modules_system
¶ The modules system name associated with this system.
-
name
¶ The name of this system.
-
outputdir
¶ The ReFrame output directory prefix associated with this system.
-
partitions
¶ All the system partitions associated with this system.
-
perflogdir
¶ The ReFrame log directory prefix associated with this system.
-
prefix
¶ The ReFrame prefix associated with this system.
-
preload_environ
¶ The environment to load whenever ReFrame runs on this system.
Note
New in version 2.19.
-
resourcesdir
¶ Global resources directory for this system.
You may use this directory for storing large resource files of your regression tests. See here on how to configure this.
Type: str
-
stagedir
¶ The ReFrame stage directory prefix associated with this system.
-
-
class
reframe.core.systems.
SystemPartition
(name, descr=None, scheduler=None, launcher=None, access=[], environs=[], resources={}, local_env=None, max_jobs=1)[source]¶ Bases:
object
A representation of a system partition inside ReFrame.
This class is immutable.
-
descr
¶ A detailed description of this partition.
-
fullname
¶ Return the fully-qualified name of this partition.
The fully-qualified name is of the form
<parent-system-name>:<partition-name>
.Type: str
-
launcher
¶ The type of the backend launcher of this partition.
Returns: a subclass of reframe.core.launchers.JobLauncher
.Note
New in version 2.8.
-
name
¶ The name of this partition.
Type: str
-
scheduler
¶ The type of the backend scheduler of this partition.
Returns: a subclass of reframe.core.schedulers.Job
.Note
Changed in version 2.8.
Prior versions returned a string representing the scheduler and job launcher combination.
-
Job schedulers and parallel launchers¶
-
class
reframe.core.schedulers.
Job
(name, launcher, workdir='.', num_tasks=1, num_tasks_per_node=None, num_tasks_per_core=None, num_tasks_per_socket=None, num_cpus_per_task=None, use_smt=None, time_limit=None, script_filename=None, stdout=None, stderr=None, pre_run=[], post_run=[], sched_flex_alloc_tasks=None, sched_access=[], sched_account=None, sched_partition=None, sched_reservation=None, sched_nodelist=None, sched_exclude_nodelist=None, sched_exclusive_access=None, sched_options=[])[source]¶ Bases:
abc.ABC
A job descriptor.
Caution
This is an abstract class. Users may not create jobs directly.
-
launcher
¶ The parallel program launcher that will be used to launch the parallel executable of this job.
Type: reframe.core.launchers.JobLauncher
-
nodelist
¶ The list of node names assigned to this job.
This attribute is
None
if no nodes are assigned to the job yet. This attribute is set reliably only for theslurm
backend, i.e., Slurm with accounting enabled. Thesqueue
scheduler backend, i.e., Slurm without accounting, might not set this attribute for jobs that finish very quickly. For thelocal
scheduler backend, this returns an one-element list containing the hostname of the current host.This attribute might be useful in a flexible regression test for determining the actual nodes that were assigned to the test.
For more information on flexible task allocation, please refer to the corresponding section of the tutorial.
This attribute is not supported by the
pbs
scheduler backend.New in version 2.17.
-
num_tasks
¶ The number of tasks assigned to this job.
This attribute is useful in a flexible regression test for determining the actual number of tasks that ReFrame assigned to the test.
For more information on flexible task allocation, please refer to the tutorial.
-
options
¶ Options to be passed to the backend job scheduler.
Type: List[str]
Default: []
-
-
class
reframe.core.launchers.
JobLauncher
(options=[])[source]¶ Bases:
abc.ABC
A job launcher.
A job launcher is the executable that actually launches a distributed program to multiple nodes, e.g.,
mpirun
,srun
etc.Note
This is an abstract class. Regression tests may not instantiate this class directly.
Note
Changed in version 2.8: Job launchers do not get a reference to a job during their initialization.
-
command
(job)[source]¶ The launcher command.
Parameters: job – A reframe.core.schedulers.Job
that will be used by this launcher to properly emit its options. Subclasses may override this method and emit options according the number of tasks associated to the job etc.Returns: a list of command line arguments (including the launcher executable).
-
-
class
reframe.core.launchers.
LauncherWrapper
(target_launcher, wrapper_command, wrapper_options=[])[source]¶ Bases:
reframe.core.launchers.JobLauncher
Wrap a launcher object so as to modify its invocation.
This is useful for parallel debuggers. For example, to launch a regression test using the DDT debugger, you can do the following:
def setup(self, partition, environ, **job_opts): super().setup(partition, environ, **job_opts) self.job.launcher = LauncherWrapper(self.job.launcher, 'ddt', ['--offline'])
If the current system partition uses native Slurm for job submission, this setup will generate the following command in the submission script:
ddt --offline srun <test_executable>
If the current partition uses
mpirun
instead, it will generateddt --offline mpirun -np <num_tasks> ... <test_executable>
Parameters: - target_launcher – The launcher to wrap.
- wrapper_command – The wrapper command.
- wrapper_options – List of options to pass to the wrapper command.
-
command
(job)[source]¶ The launcher command.
Parameters: job – A reframe.core.schedulers.Job
that will be used by this launcher to properly emit its options. Subclasses may override this method and emit options according the number of tasks associated to the job etc.Returns: a list of command line arguments (including the launcher executable).
-
reframe.core.launchers.registry.
getlauncher
(name)[source]¶ Get launcher by its registered name.
The available names are those specified in the configuration file.
This method may become handy in very special situations, e.g., testing an application that needs to replace the system partition launcher or if a different launcher must be used for a different programming environment.
For example, if you want to replace the current partition’s launcher with the local one, here is how you can achieve it:
def setup(self, partition, environ, **job_opts): super().setup(partition, environ, **job_opts) self.job.launcher = getlauncher('local')()
Note that this method returns a launcher class type and not an instance of that class. You have to instantiate it explicitly before assigning it to the
launcher
attribute of the job.Note
New in version 2.8.
Parameters: name – The name of the launcher to retrieve. Returns: The class of the launcher requested, which is a subclass of reframe.core.launchers.JobLauncher
.Raises: reframe.core.exceptions.ConfigError – if no launcher is registered with that name.
-
reframe.core.launchers.registry.
register_launcher
(name, local=False)[source]¶ Class decorator for registering new job launchers.
Caution
This decorator is only relevant to developers of new job launchers.
Note
New in version 2.8.
Parameters: - name – The registration name of this launcher
- local –
True
if launcher may only submit local jobs,False
otherwise.
Raises: ValueError – if a job launcher is already registered with the same name.
Runtime services¶
-
class
reframe.core.runtime.
HostResources
(prefix=None, stagedir=None, outputdir=None, perflogdir=None, timefmt=None)[source]¶ Bases:
object
Resources associated with ReFrame execution on the current host.
Note
New in version 2.13.
-
output_prefix
¶ The output prefix directory of ReFrame.
-
prefix
¶ The prefix directory of ReFrame execution. This is always an absolute path.
Type: str
Caution
Users may not set this field.
-
stage_prefix
¶ The stage prefix directory of ReFrame.
-
-
class
reframe.core.runtime.
HostSystem
(system, partname=None)[source]¶ Bases:
object
The host system of the framework.
The host system is a representation of the system that the framework currently runs on.If the framework is properly configured, the host system is automatically detected. If not, it may be explicitly set by the user.
This class is mainly a proxy of
reframe.core.systems.System
that stores optionally a partition name and provides some additional functionality for manipulating system partitions.All attributes of the
reframe.core.systems.System
may be accessed directly from this proxy.Note
New in version 2.13.
-
partition
(name)[source]¶ Return the system partition
name
.Type: reframe.core.systems.SystemPartition
.
-
partitions
¶ The partitions of this system.
Type: list[reframe.core.systems.SystemPartition]
.
-
-
class
reframe.core.runtime.
RuntimeContext
(dict_config, sysdescr=None)[source]¶ Bases:
object
The runtime context of the framework.
This class essentially groups the current host system and the associated resources of the framework on the current system.
There is a single instance of this class globally in the framework.
Note
New in version 2.13.
-
modules_system
¶ The modules system used by the current host system.
Type: reframe.core.modules.ModulesSystem
.
-
resources
¶ The framework resources.
Type: reframe.core.runtime.HostResources
-
system
¶ The current host system.
Type: reframe.core.runtime.HostSystem
-
-
reframe.core.runtime.
runtime
()[source]¶ Retrieve the framework’s runtime context.
Type: reframe.core.runtime.RuntimeContext
Note
New in version 2.13.
Modules System API¶
-
class
reframe.core.modules.
ModulesSystem
(backend)[source]¶ A modules system abstraction inside ReFrame.
This class interfaces between the framework internals and the actual modules systems implementation.
-
conflicted_modules
(name)[source]¶ Return the list of the modules conflicting with module
name
.If module
name
resolves to multiple real modules, then the returned list will be the concatenation of the conflict lists of all the real modules.This method returns a list of strings.
-
is_module_loaded
(name)[source]¶ Check if module
name
is loaded.If module
name
refers to multiple real modules, this method will returnTrue
only if all the referees are loaded.
-
load_mapping
(mapping)[source]¶ Update the internal module mappings using a single mapping.
Parameters: mapping – a string specifying the module mapping. Example syntax: 'm0: m1 m2'
.
-
load_mapping_from_file
(filename)[source]¶ Update the internal module mappings from mappings read from file.
-
load_module
(name, force=False)[source]¶ Load the module
name
.If
force
is set, forces the loading, unloading first any conflicting modules currently loaded. If modulename
refers to multiple real modules, all of the target modules will be loaded.Returns the list of unloaded modules as strings.
-
name
¶ Return the name of this module system.
-
resolve_module
(name)[source]¶ Resolve module
name
in the registered module map.Returns: the list of real modules names pointed to by name
.Raises: reframe.core.exceptions.ConfigError
if the mapping contains a cycle.
-
searchpath
¶ The module system search path as a list of directories.
-
unload_module
(name)[source]¶ Unload module
name
.If module
name
refers to multiple real modules, all the referred to modules will be unloaded in reverse order.
-
version
¶ Return the version of this module system.
-
Build systems¶
New in version 2.14.
ReFrame delegates the compilation of the regression test to a build system. Build systems in ReFrame are entities that are responsible for generating the necessary shell commands for compiling a code. Each build system defines a set of attributes that users may set in order to customize their compilation. An example usage is the following:
self.build_system = 'SingleSource'
self.build_system.cflags = ['-fopenmp']
Users simply set the build system to use in their regression tests and then they configure it. If no special configuration is needed for the compilation, users may completely ignore the build systems. ReFrame will automatically pick one based on the regression test attributes and will try to compile the code.
All build systems in ReFrame derive from the abstract base class reframe.core.buildsystems.BuildSystem
.
This class defines a set of common attributes, such us compilers, compilation flags etc. that all subclasses inherit.
It is up to the concrete build system implementations on how to use or not these attributes.
-
class
reframe.core.buildsystems.
Autotools
[source]¶ Bases:
reframe.core.buildsystems.ConfigureBasedBuildSystem
A build system for compiling Autotools-based projects.
This build system will emit the following commands:
- Create a build directory if
builddir
is notNone
and change to it. - Invoke
configure
to configure the project by setting the corresponding flags for compilers and compiler flags. - Issue
make
to compile the code.
-
emit_build_commands
(environ)[source]¶ Return the list of commands for building using this build system.
The build commands may always assume to be issued from the top-level directory of the code that is to be built.
Parameters: environ ( reframe.core.environments.ProgEnvironment
) – The programming environment for which to emit the build instructions. The framework passes here the current programming environment.Raises: BuildSystemError
in case of errors when generating the build instructions.Note
This method is relevant only to developers of new build systems.
- Create a build directory if
-
class
reframe.core.buildsystems.
BuildSystem
[source]¶ Bases:
abc.ABC
The abstract base class of any build system.
Concrete build systems inherit from this class and must override the
emit_build_commands()
abstract function.-
cc
¶ The C compiler to be used. If set to
None
andflags_from_environ
isTrue
, the compiler defined in the current programming environment will be used.Type: str
Default: None
-
cflags
¶ The C compiler flags to be used. If set to
None
andflags_from_environ
isTrue
, the corresponding flags defined in the current programming environment will be used.Type: List[str]
Default: None
-
cppflags
¶ The preprocessor flags to be used. If set to
None
andflags_from_environ
isTrue
, the corresponding flags defined in the current programming environment will be used.Type: List[str]
Default: None
-
cxx
¶ The C++ compiler to be used. If set to
None
andflags_from_environ
isTrue
, the compiler defined in the current programming environment will be used.Type: str
Default: None
-
cxxflags
¶ The C++ compiler flags to be used. If set to
None
andflags_from_environ
isTrue
, the corresponding flags defined in the current programming environment will be used.Type: List[str]
Default: None
-
emit_build_commands
(environ)[source]¶ Return the list of commands for building using this build system.
The build commands may always assume to be issued from the top-level directory of the code that is to be built.
Parameters: environ ( reframe.core.environments.ProgEnvironment
) – The programming environment for which to emit the build instructions. The framework passes here the current programming environment.Raises: BuildSystemError
in case of errors when generating the build instructions.Note
This method is relevant only to developers of new build systems.
-
fflags
¶ The Fortran compiler flags to be used. If set to
None
andflags_from_environ
isTrue
, the corresponding flags defined in the current programming environment will be used.Type: List[str]
Default: None
-
flags_from_environ
¶ Set compiler and compiler flags from the current programming environment if not specified otherwise.
Type: bool
Default: True
-
ftn
¶ The Fortran compiler to be used. If set to
None
andflags_from_environ
isTrue
, the compiler defined in the current programming environment will be used.Type: str
Default: None
-
ldflags
¶ The linker flags to be used. If set to
None
andflags_from_environ
isTrue
, the corresponding flags defined in the current programming environment will be used.Type: List[str]
Default: None
-
nvcc
¶ The CUDA compiler to be used. If set to
None
andflags_from_environ
isTrue
, the compiler defined in the current programming environment will be used.Type: str
Default: None
-
-
class
reframe.core.buildsystems.
CMake
[source]¶ Bases:
reframe.core.buildsystems.ConfigureBasedBuildSystem
A build system for compiling CMake-based projects.
This build system will emit the following commands:
- Create a build directory if
builddir
is notNone
and change to it. - Invoke
cmake
to configure the project by setting the corresponding CMake flags for compilers and compiler flags. - Issue
make
to compile the code.
-
emit_build_commands
(environ)[source]¶ Return the list of commands for building using this build system.
The build commands may always assume to be issued from the top-level directory of the code that is to be built.
Parameters: environ ( reframe.core.environments.ProgEnvironment
) – The programming environment for which to emit the build instructions. The framework passes here the current programming environment.Raises: BuildSystemError
in case of errors when generating the build instructions.Note
This method is relevant only to developers of new build systems.
- Create a build directory if
-
class
reframe.core.buildsystems.
ConfigureBasedBuildSystem
[source]¶ Bases:
reframe.core.buildsystems.BuildSystem
Abstract base class for configured-based build systems.
-
builddir
¶ The CMake build directory, where all the generated files will be placed.
Type: str
Default: None
-
config_opts
¶ Additional configuration options to be passed to the CMake invocation.
Type: List[str]
Default: []
-
make_opts
¶ Options to be passed to the subsequent
make
invocation.Type: List[str]
Default: []
-
srcdir
¶ The top-level directory of the code.
This is set automatically by the framework based on the
reframe.core.pipeline.RegressionTest.sourcepath
attribute.Type: str
Default: None
-
-
class
reframe.core.buildsystems.
Make
[source]¶ Bases:
reframe.core.buildsystems.BuildSystem
A build system for compiling codes using
make
.The generated build command has the following form:
make -j [N] [-f MAKEFILE] [-C SRCDIR] CC="X" CXX="X" FC="X" NVCC="X" CPPFLAGS="X" CFLAGS="X" CXXFLAGS="X" FCFLAGS="X" LDFLAGS="X" OPTIONS
The compiler and compiler flags variables will only be passed if they are not
None
. Their value is determined by the corresponding attributes ofBuildSystem
. If you want to completely disable passing these variables to themake
invocation, you should make sure not to set any of the correspoding attributes and set also theBuildSystem.flags_from_environ
flag toFalse
.-
emit_build_commands
(environ)[source]¶ Return the list of commands for building using this build system.
The build commands may always assume to be issued from the top-level directory of the code that is to be built.
Parameters: environ ( reframe.core.environments.ProgEnvironment
) – The programming environment for which to emit the build instructions. The framework passes here the current programming environment.Raises: BuildSystemError
in case of errors when generating the build instructions.Note
This method is relevant only to developers of new build systems.
-
makefile
¶ Instruct build system to use this Makefile. This option is useful when having non-standard Makefile names.
Type: str
Default: None
-
max_concurrency
¶ Limit concurrency for
make
jobs. This attribute controls the-j
option passed tomake
. If notNone
,make
will be invoked asmake -j max_concurrency
. Otherwise, it will invoked asmake -j
.Type: integer Default: 1
Note
Changed in version 2.19: The default value is now
1
-
options
¶ Append these options to the
make
invocation. This variable is also useful for passing variables or targets tomake
.Type: List[str]
Default: []
-
srcdir
¶ The top-level directory of the code.
This is set automatically by the framework based on the
reframe.core.pipeline.RegressionTest.sourcepath
attribute.Type: str
Default: None
-
-
class
reframe.core.buildsystems.
SingleSource
[source]¶ Bases:
reframe.core.buildsystems.BuildSystem
A build system for compiling a single source file.
The generated build command will have the following form:
COMP CPPFLAGS XFLAGS SRCFILE -o EXEC LDFLAGS
COMP
is the required compiler for compilingSRCFILE
. This build system will automatically detect the programming language of the source file and pick the correct compiler. See also theSingleSource.lang
attribute.CPPFLAGS
are the preprocessor flags and are passed to any compiler.XFLAGS
is any ofCFLAGS
,CXXFLAGS
orFCFLAGS
depending on the programming language of the source file.SRCFILE
is the source file to be compiled. This is set up automatically by the framework. See also theSingleSource.srcfile
attribute.EXEC
is the executable to be generated. This is also set automatically by the framework. See also theSingleSource.executable
attribute.LDFLAGS
are the linker flags.
For CUDA codes, the language assumed is C++ (for the compilation flags) and the compiler used is
BuildSystem.nvcc
.-
emit_build_commands
(environ)[source]¶ Return the list of commands for building using this build system.
The build commands may always assume to be issued from the top-level directory of the code that is to be built.
Parameters: environ ( reframe.core.environments.ProgEnvironment
) – The programming environment for which to emit the build instructions. The framework passes here the current programming environment.Raises: BuildSystemError
in case of errors when generating the build instructions.Note
This method is relevant only to developers of new build systems.
-
executable
¶ The executable file to be generated.
This is set automatically by the framework based on the
reframe.core.pipeline.RegressionTest.executable
attribute.Type: str
orNone
-
include_path
¶ The include path to be used for this compilation.
All the elements of this list will be appended to the
BuildSystem.cppflags
, by prepending to each of them the-I
option.Type: List[str]
Default: []
-
lang
¶ The programming language of the file that needs to be compiled. If not specified, the build system will try to figure it out automatically based on the extension of the source file. The automatically detected extensions are the following:
- C: .c.
- C++: .cc, .cp, .cxx, .cpp, .CPP, .c++ and .C.
- Fortran: .f, .for, .ftn, .F, .FOR, .fpp, .FPP, .FTN, .f90, .f95, .f03, .f08, .F90, .F95, .F03 and .F08.
- CUDA: .cu.
Type: str
orNone
-
srcfile
¶ The source file to compile. This is automatically set by the framework based on the
reframe.core.pipeline.RegressionTest.sourcepath
attribute.Type: str
orNone