Configuration Reference

ReFrame’s behavior can be configured through its configuration file (see Configuring ReFrame for Your Site), environment variables and command-line options. An option can be specified via multiple paths (e.g., a configuration file parameter and an environment variable), in which case command-line options precede environment variables, which in turn precede configuration file options. This section provides a complete reference guide of the configuration options of ReFrame that can be set in its configuration file or specified using environment variables.

ReFrame’s configuration is in JSON syntax. The full schema describing it can be found in reframe/schemas/config.json file. Any configuration file given to ReFrame is validated against this schema.

The syntax we use in the following to describe the different configuration object attributes is a valid query string for the jq(1) command-line processor.

Top-level Configuration

The top-level configuration object is essentially the full configuration of ReFrame. It consists of the following properties:

.systems
Required

Yes

A list of system configuration objects.

.environments
Required

Yes

A list of environment configuration objects.

.logging
Required

Yes

A list of logging configuration objects.

.schedulers
Required

No

A list of scheduler configuration objects.

.modes
Required

No

A list of execution mode configuration objects.

.general
Required

No

A list of general configuration objects.

System Configuration

.systems[].name
Required

Yes

The name of this system. Only alphanumeric characters, dashes (-) and underscores (_) are allowed.

.systems[].descr
Required

No

Default

""

The description of this system.

.systems[].hostnames
Required

Yes

A list of hostname regular expression patterns in Python syntax, which will be used by the framework in order to automatically select a system configuration. For the auto-selection process, see here.

.systems[].max_local_jobs

The maximum number of forced local build or run jobs allowed.

Forced local jobs run within the execution context of ReFrame.

Required

No

Default

8

New in version 3.10.0.

.systems[].modules_system
Required

No

Default

"nomod"

The modules system that should be used for loading environment modules on this system. Available values are the following:

  • tmod: The classic Tcl implementation of the environment modules (version 3.2).

  • tmod31: The classic Tcl implementation of the environment modules (version 3.1). A separate backend is required for Tmod 3.1, because Python bindings are different from Tmod 3.2.

  • tmod32: A synonym of tmod.

  • tmod4: The new environment modules implementation (versions older than 4.1 are not supported).

  • lmod: The Lua implementation of the environment modules.

  • spack: Spack’s built-in mechanism for managing modules.

  • nomod: This is to denote that no modules system is used by this system.

New in version 3.4: The spack backend is added.

.systems[].modules
Required

No

Default

[]

A list of environment module objects to be loaded always when running on this system. These modules modify the ReFrame environment. This is useful in cases where a particular module is needed, for example, to submit jobs on a specific system.

.systems[].variables
Required

No

Default

[]

A list of environment variables to be set always when running on this system. These variables modify the ReFrame environment. Each environment variable is specified as a two-element list containing the variable name and its value. You may reference other environment variables when defining an environment variable here. ReFrame will expand its value. Variables are set after the environment modules are loaded.

.systems[].prefix
Required

No

Default

"."

Directory prefix for a ReFrame run on this system. Any directories or files produced by ReFrame will use this prefix, if not specified otherwise.

.systems[].stagedir
Required

No

Default

"${RFM_PREFIX}/stage"

Stage directory prefix for this system. This is the directory prefix, where ReFrame will create the stage directories for each individual test case.

.systems[].outputdir
Required

No

Default

"${RFM_PREFIX}/output"

Output directory prefix for this system. This is the directory prefix, where ReFrame will save information about the successful tests.

.systems[].resourcesdir
Required

No

Default

"."

Directory prefix where external test resources (e.g., large input files) are stored. You may reference this prefix from within a regression test by accessing the reframe.core.systems.System.resourcesdir attribute of the current system.

.systems[].partitions
Required

Yes

A list of system partition configuration objects. This list must have at least one element.

System Partition Configuration

.systems[].partitions[].name
Required

Yes

The name of this partition. Only alphanumeric characters, dashes (-) and underscores (_) are allowed.

.systems[].partitions[].descr
Required

No

Default

""

The description of this partition.

.systems[].partitions[].scheduler
Required

Yes

The job scheduler that will be used to launch jobs on this partition. Supported schedulers are the following:

  • local: Jobs will be launched locally without using any job scheduler.

  • flux: Jobs will be launched using the Flux Framework scheduler.

  • oar: Jobs will be launched using the OAR scheduler.

  • pbs: Jobs will be launched using the PBS Pro scheduler.

  • sge: Jobs will be launched using the Sun Grid Engine scheduler.

  • slurm: Jobs will be launched using the Slurm scheduler. This backend requires job accounting to be enabled in the target system. If not, you should consider using the squeue backend below.

  • squeue: Jobs will be launched using the Slurm scheduler. This backend does not rely on job accounting to retrieve job statuses, but ReFrame does its best to query the job state as reliably as possible.

  • torque: Jobs will be launched using the Torque scheduler.

  • lsf: Jobs will be launched using the LSF scheduler.

New in version 3.7.2: Support for the SGE scheduler is added.

New in version 3.8.2: Support for the OAR scheduler is added.

New in version 3.11.0: Support for the LSF scheduler is added.

Note

The way that multiple node jobs are submitted using the SGE scheduler can be very site-specific. For this reason, the sge scheduler backend does not try to interpret any related arguments, e.g., num_tasks, num_tasks_per_node etc. Users must specify how these resources are to be requested by setting the resources partition configuration parameter and then request them from inside a test using the extra_resources test attribute. Here is an example configuration for a system partition named foo that defines different ways for submitting MPI-only, OpenMP-only and MPI+OpenMP jobs:

{
    'name': 'foo',
    'scheduler': 'sge',
    'resources': [
        {
            'name': 'smp',
            'options': ['-pe smp {num_slots}']
        },
        {
            'name': 'mpi',
            'options': ['-pe mpi {num_slots}']
        },
        {
            'name': 'mpismp',
            'options': ['-pe mpismp {num_slots}']
        }
    ]
}

Each test then can request the different type of slots as follows:

self.extra_resouces = {
    'smp': {'num_slots': self.num_cpus_per_task},
    'mpi': {'num_slots': self.num_tasks},
    'mpismp': {'num_slots': self.num_tasks*self.num_cpus_per_task}
}

Notice that defining extra_resources does not make the test non-portable to other systems that have different schedulers; the extra_resources will be simply ignored in this case and the scheduler backend will interpret the different test fields in the appropriate way.

.systems[].partitions[].launcher
Required

Yes

The parallel job launcher that will be used in this partition to launch parallel programs. Available values are the following:

  • alps: Parallel programs will be launched using the Cray ALPS aprun command.

  • ibrun: Parallel programs will be launched using the ibrun command. This is a custom parallel program launcher used at TACC.

  • local: No parallel program launcher will be used. The program will be launched locally.

  • lrun: Parallel programs will be launched using LC Launcher’s lrun command.

  • lrun-gpu: Parallel programs will be launched using LC Launcher’s lrun -M "-gpu" command that enables the CUDA-aware Spectrum MPI.

  • mpirun: Parallel programs will be launched using the mpirun command.

  • mpiexec: Parallel programs will be launched using the mpiexec command.

  • srun: Parallel programs will be launched using Slurm’s srun command.

  • srunalloc: Parallel programs will be launched using Slurm’s srun command, but job allocation options will also be emitted. This can be useful when combined with the local job scheduler.

  • ssh: Parallel programs will be launched using SSH. This launcher uses the partition’s access property in order to determine the remote host and any additional options to be passed to the SSH client. The ssh command will be launched in “batch mode,” meaning that password-less access to the remote host must be configured. Here is an example configuration for the ssh launcher:

    {
        'name': 'foo'
        'scheduler': 'local',
        'launcher': 'ssh'
        'access': ['-l admin', 'remote.host'],
        'environs': ['builtin'],
    }
    
  • upcrun: Parallel programs will be launched using the UPC upcrun command.

  • upcxx-run: Parallel programs will be launched using the UPC++ upcxx-run command.

.systems[].partitions[].access
Required

No

Default

[]

A list of job scheduler options that will be passed to the generated job script for gaining access to that logical partition.

.systems[].partitions[].environs
required

No

default

[]

A list of environment names that ReFrame will use to run regression tests on this partition. Each environment must be defined in the environments section of the configuration and the definition of the environment must be valid for this partition.

.systems[].partitions[].container_platforms
Required

No

Default

[]

A list for container platform configuration objects. This will allow launching regression tests that use containers on this partition.

.systems[].partitions[].modules
required

No

default

[]

A list of environment module objects to be loaded before running a regression test on this partition.

.systems[].partitions[].time_limit
Required

No

Default

null

The time limit for the jobs submitted on this partition. When the value is null, no time limit is applied.

.systems[].partitions[].variables
Required

No

Default

[]

A list of environment variables to be set before running a regression test on this partition. Each environment variable is specified as a two-element list containing the variable name and its value. You may reference other environment variables when defining an environment variable here. ReFrame will expand its value. Variables are set after the environment modules are loaded.

.systems[].partitions[].max_jobs
Required

No

Default

8

The maximum number of concurrent regression tests that may be active (i.e., not completed) on this partition. This option is relevant only when ReFrame executes with the asynchronous execution policy.

.systems[].partitions[].prepare_cmds
Required

No

Default

[]

List of shell commands to be emitted before any environment loading commands are emitted.

New in version 3.5.0.

.systems[].partitions[].resources
Required

No

Default

[]

A list of job scheduler resource specification objects.

.systems[].partitions[].processor
Required

No

Default

{}

Processor information for this partition stored in a processor info object. If not set, ReFrame will try to auto-detect this information (see Auto-detecting processor information for more information).

New in version 3.5.0.

Changed in version 3.7.0: ReFrame is now able to detect the processor information automatically.

.systems[].partitions[].devices
Required

No

Default

[]

A list with device info objects for this partition.

New in version 3.5.0.

.systems[].partitions[].features
Required

No

Default

[]

User defined features of the partition. These are accessible through the features attribute of the current_partition and can also be selected through the extended syntax of valid_systems. The values of this list must be alphanumeric strings starting with a non-digit character and may also contain a -.

New in version 3.11.0.

.systems[].partitions[].extras
Required

No

Default

{}

User defined attributes of the partition. These are accessible through the extras attribute of the current_partition and can also be selected through the extended syntax of valid_systems. The attributes of this object must be alphanumeric strings starting with a non-digit character and their values can be of any type.

New in version 3.5.0.

Container Platform Configuration

ReFrame can launch containerized applications, but you need to configure properly a system partition in order to do that by defining a container platform configuration.

.systems[].partitions[].container_platforms[].type
Required

Yes

The type of the container platform. Available values are the following:

  • Docker: The Docker container runtime.

  • Sarus: The Sarus container runtime.

  • Shifter: The Shifter container runtime.

  • Singularity: The Singularity container runtime.

.systems[].partitions[].container_platforms[].default
Required

No

If set to true, this is the default container platform of this partition. If not specified, the default container platform is assumed to be the first in the list of container_platforms.

New in version 3.12.0.

.systems[].partitions[].container_platforms[].modules
Required

No

Default

[]

A list of environment module objects to be loaded when running containerized tests using this container platform.

.systems[].partitions[].container_platforms[].variables
Required

No

Default

[]

List of environment variables to be set when running containerized tests using this container platform. Each environment variable is specified as a two-element list containing the variable name and its value. You may reference other environment variables when defining an environment variable here. ReFrame will expand its value. Variables are set after the environment modules are loaded.

Custom Job Scheduler Resources

ReFrame allows you to define custom scheduler resources for each partition that you can then transparently access through the extra_resources attribute of a regression test.

.systems[].partitions[].resources[].name
required

Yes

The name of this resources. This name will be used to request this resource in a regression test’s extra_resources.

.systems[].partitions[].resources[].options
required

No

default

[]

A list of options to be passed to this partition’s job scheduler. The option strings can contain placeholders of the form {placeholder_name}. These placeholders may be replaced with concrete values by a regression test through the extra_resources attribute.

For example, one could define a gpu resource for a multi-GPU system that uses Slurm as follows:

'resources': [
    {
        'name': 'gpu',
        'options': ['--gres=gpu:{num_gpus_per_node}']
    }
]

A regression test then may request this resource as follows:

self.extra_resources = {'gpu': {'num_gpus_per_node': '8'}}

And the generated job script will have the following line in its preamble:

#SBATCH --gres=gpu:8

A resource specification may also start with #PREFIX, in which case #PREFIX will replace the standard job script prefix of the backend scheduler of this partition. This is useful in cases of job schedulers like Slurm, that allow alternative prefixes for certain features. An example is the DataWarp functionality of Slurm which is supported by the #DW prefix. One could then define DataWarp related resources as follows:

'resources': [
    {
        'name': 'datawarp',
        'options': [
            '#DW jobdw capacity={capacity} access_mode={mode} type=scratch',
            '#DW stage_out source={out_src} destination={out_dst} type={stage_filetype}'
        ]
    }
]

A regression test that wants to make use of that resource, it can set its extra_resources as follows:

self.extra_resources = {
    'datawarp': {
        'capacity': '100GB',
        'mode': 'striped',
        'out_src': '$DW_JOB_STRIPED/name',
        'out_dst': '/my/file',
        'stage_filetype': 'file'
    }
}

Note

For the pbs and torque backends, options accepted in the access and resources attributes may either refer to actual qsub options or may be just resources specifications to be passed to the -l option. The backend assumes a qsub option, if the options passed in these attributes start with a -.

Environment Configuration

Environments defined in this section will be used for running regression tests. They are associated with system partitions.

.environments[].name
Required

Yes

The name of this environment.

.environments[].modules
Required

No

Default

[]

A list of environment module objects to be loaded when this environment is loaded.

.environments[].variables
Required

No

Default

[]

A list of environment variables to be set when loading this environment. Each environment variable is specified as a two-element list containing the variable name and its value. You may reference other environment variables when defining an environment variable here. ReFrame will expand its value. Variables are set after the environment modules are loaded.

.environments[].features
Required

No

Default

[]

User defined features of the environment. These are accessible through the features attribute of the current_environ and can also be selected through the extended syntax of valid_prog_environs. The values of this list must be alphanumeric strings starting with a non-digit character and may also contain a -.

New in version 3.11.0.

.environments[].extras
Required

No

Default

{}

User defined attributes of the environment. These are accessible through the extras attribute of the current_environ and can also be selected through the extended syntax of valid_prog_environs. The attributes of this object must be alphanumeric strings starting with a non-digit character and their values can be of any type.

New in version 3.9.1.

.environments[].cc
Required

No

Default

"cc"

The C compiler to be used with this environment.

.environments[].cxx
Required

No

Default

"CC"

The C++ compiler to be used with this environment.

.environments[].ftn
Required

No

Default

"ftn"

The Fortran compiler to be used with this environment.

.environments[].cppflags
Required

No

Default

[]

A list of C preprocessor flags to be used with this environment by default.

.environments[].cflags
Required

No

Default

[]

A list of C flags to be used with this environment by default.

.environments[].cxxflags
Required

No

Default

[]

A list of C++ flags to be used with this environment by default.

.environments[].fflags
Required

No

Default

[]

A list of Fortran flags to be used with this environment by default.

.environments[].ldflags
Required

No

Default

[]

A list of linker flags to be used with this environment by default.

.environments[].target_systems
Required

No

Default

["*"]

A list of systems or system/partitions combinations that this environment definition is valid for. A * entry denotes any system. In case of multiple definitions of an environment, the most specific to the current system partition will be used. For example, if the current system/partition combination is daint:mc, the second definition of the PrgEnv-gnu environment will be used:

'environments': [
    {
        'name': 'PrgEnv-gnu',
        'modules': ['PrgEnv-gnu']
    },
    {
        'name': 'PrgEnv-gnu',
        'modules': ['PrgEnv-gnu', 'openmpi'],
        'cc':  'mpicc',
        'cxx': 'mpicxx',
        'ftn': 'mpif90',
        'target_systems': ['daint:mc']
    }
]

However, if the current system was daint:gpu, the first definition would be selected, despite the fact that the second definition is relevant for another partition of the same system. To better understand this, ReFrame resolves definitions in a hierarchical way. It first looks for definitions for the current partition, then for the containing system and, finally, for global definitions (the * pseudo-system).

Logging Configuration

Logging in ReFrame is handled by logger objects which further delegate message to logging handlers which are eventually responsible for emitting or sending the log records to their destinations. You may define different logger objects per system but not per partition.

.logging[].level
Required

No

Default

"undefined"

The level associated with this logger object. There are the following levels in decreasing severity order:

  • critical: Catastrophic errors; the framework cannot proceed with its execution.

  • error: Normal errors; the framework may or may not proceed with its execution.

  • warning: Warning messages.

  • info: Informational messages.

  • verbose: More informational messages.

  • debug: Debug messages.

  • debug2: Further debug messages.

  • undefined: This is the lowest level; do not filter any message.

If a message is logged by the framework, its severity level will be checked by the logger and if it is higher from the logger’s level, it will be passed down to its handlers.

New in version 3.3: The debug2 and undefined levels are added.

Changed in version 3.3: The default level is now undefined.

.logging[].handlers
Required

Yes

A list of logging handlers responsible for handling normal framework output.

.logging[].handlers_perflog
Required

Yes

A list of logging handlers responsible for handling performance data from tests.

.logging[].target_systems
Required

No

Default

["*"]

A list of systems or system/partitions combinations that this logging configuration is valid for. For a detailed description of this property, you may refer here.

Common logging handler properties

All logging handlers share the following set of common attributes:

.logging[].handlers[].type
.logging[].handlers_perflog[].type
Required

Yes

The type of handler. There are the following types available:

  • file: This handler sends log records to file. See here for more details.

  • filelog: This handler sends performance log records to files. See here for more details.

  • graylog: This handler sends performance log records to Graylog. See here for more details.

  • stream: This handler sends log records to a file stream. See here for more details.

  • syslog: This handler sends log records to a Syslog facility. See here for more details.

  • httpjson: This handler sends log records in JSON format using HTTP post requests. See here for more details.

.logging[].handlers[].level
.logging[].handlers_perflog[].level
Required

No

Default

"info"

The log level associated with this handler.

.logging[].handlers[].format
.logging[].handlers_perflog[].format
Required

No

Default

"%(message)s"

Log record format string. ReFrame accepts all log record attributes from Python’s logging mechanism and adds the following:

Log record attribute

Description

%(check_build_locally)s

The value of the build_locally attribute.

%(check_build_time_limit)s

The value of the build_time_limit attribute.

%(check_descr)s

The value of the descr attribute.

%(check_display_name)s

The value of the display_name attribute.

%(check_environ)s

The name of the test’s current_environ.

%(check_exclusive_access)s

The value of the exclusive_access attribute.

%(check_executable)s

The value of the executable attribute.

%(check_executable_opts)s

The value of the executable_opts attribute.

%(check_extra_resources)s

The value of the extra_resources attribute.

%(check_hashcode)s

The unique hash associated with this test.

%(check_info)s

Various information about this test; essentially the return value of the test’s info() function.

%(check_job_completion_time)s

Same as the (check_job_completion_time_unix)s but formatted according to datefmt.

%(check_job_completion_time_unix)s

The completion time of the associated run job (see reframe.core.schedulers.Job.completion_time).

%(check_job_exitcode)s

The exit code of the associated run job.

%(check_job_nodelist)s

The list of nodes that the associated run job has run on.

%(check_jobid)s

The ID of the associated run job.

%(check_keep_files)s

The value of the keep_files attribute.

%(check_local)s

The value of the local attribute.

%(check_maintainers)s

The value of the maintainers attribute.

%(check_max_pending_time)s

The value of the max_pending_time attribute.

%(check_modules)s

The value of the modules attribute.

%(check_name)s

The value of the name attribute.

%(check_num_cpus_per_task)s

The value of the num_cpus_per_task attribute.

%(check_num_gpus_per_node)s

The value of the num_gpus_per_node attribute.

%(check_num_tasks)s

The value of the num_tasks attribute.

%(check_num_tasks_per_core)s

The value of the num_tasks_per_core attribute.

%(check_num_tasks_per_node)s

The value of the num_tasks_per_node attribute.

%(check_num_tasks_per_socket)s

The value of the num_tasks_per_socket attribute.

%(check_outputdir)s

The value of the outputdir attribute.

%(check_partition)s

The name of the test’s current_partition.

%(check_perf_lower_thres)s

The lower threshold of the logged performance variable.

%(check_perf_ref)s

The reference value of the logged performance variable.

%(check_perf_unit)s

The measurement unit of the logged performance variable.

%(check_perf_upper)s

The upper thresholds of the logged performance variable.

%(check_perf_value)s

The actual value of the logged performance variable.

%(check_perf_var)s

The name of the logged performance variable.

%(check_perfvalues)s

All the performance variables of the test combined along with their values

references and thresholds.

%(check_postbuild_cmds)s

The value of the postbuild_cmds attribute.

%(check_postrun_cmds)s

The value of the postrun_cmds attribute.

%(check_prebuild_cmds)s

The value of the prebuild_cmds attribute.

%(check_prefix)s

The value of the prefix attribute.

%(check_prerun_cmds)s

The value of the prerun_cmds attribute.

%(check_readonly_files)s

The value of the readonly_files attribute.

%(check_short_name)s

The value of the short_name attribute.

%(check_sourcepath)s

The value of the sourcepath attribute.

%(check_sourcesdir)s

The value of the sourcesdir attribute.

%(check_stagedir)s

The value of the stagedir attribute.

%(check_strict_check)s

The value of the strict_check attribute.

%(check_system)s

The name of the test’s current_system.

%(check_tags)s

The value of the tags attribute.

%(check_time_limit)s

The value of the time_limit attribute.

%(check_unique_name)s

The value of the unique_name attribute.

%(check_use_multithreading)s

The value of the use_multithreading attribute.

%(check_valid_prog_environs)s

The value of the valid_prog_environs attribute.

%(check_valid_systems)s

The value of the valid_systems attribute.

%(check_variables)s

The value of the variables attribute.

%(osuser)s

The name of the OS user running ReFrame.

%(osgroup)s

The name of the OS group running ReFrame.

%(version)s

The ReFrame version.

ReFrame allows you to log any test variable, parameter or property if they are marked as “loggable”. The log record attribute will have the form %(check_NAME)s where NAME is the variable name, the parameter name or the property name that is marked as loggable.

New in version 3.3: Allow arbitrary test attributes to be logged.

New in version 3.4.2: Allow arbitrary job attributes to be logged.

Changed in version 3.11.0: Limit the number of attributes that can be logged. User attributes or properties must be explicitly marked as “loggable” in order to be selectable for logging.

.logging[].handlers[].datefmt
.logging[].handlers_perflog[].datefmt
Required

No

Default

"%FT%T"

Time format to be used for printing timestamps fields. There are two timestamp fields available: %(asctime)s and %(check_job_completion_time)s. In addition to the format directives supported by the standard library’s time.strftime() function, ReFrame allows you to use the %:z directive – a GNU date extension – that will print the time zone difference in a RFC3339 compliant way, i.e., +/-HH:MM instead of +/-HHMM.

The file log handler

This log handler handles output to normal files. The additional properties for the file handler are the following:

.logging[].handlers[].name
.logging[].handlers_perflog[].name
Required

No

The name of the file where this handler will write log records. If not specified, ReFrame will create a log file prefixed with rfm- in the system’s temporary directory.

Changed in version 3.3: The name parameter is no more required and the default log file resides in the system’s temporary directory.

.logging[].handlers[].append
.logging[].handlers_perflog[].append
Required

No

Default

false

Controls whether this handler should append to its file or not.

.logging[].handlers[].timestamp
.logging[].handlers_perflog[].timestamp
Required

No

Default

false

Append a timestamp to this handler’s log file. This property may also accept a date format as described in the datefmt property. If the handler’s name property is set to filename.log and this property is set to true or to a specific timestamp format, the resulting log file will be filename_<timestamp>.log.

The filelog log handler

This handler is meant primarily for performance logging and logs the performance of a regression test in one or more files. The additional properties for the filelog handler are the following:

.logging[].handlers[].basedir
.logging[].handlers_perflog[].basedir
Required

No

Default

"./perflogs"

The base directory of performance data log files.

.logging[].handlers[].prefix
.logging[].handlers_perflog[].prefix
Required

Yes

This is a directory prefix (usually dynamic), appended to the basedir, where the performance logs of a test will be stored. This attribute accepts any of the check-specific formatting placeholders. This allows to create dynamic paths based on the current system, partition and/or programming environment a test executes with. For example, a value of %(check_system)s/%(check_partition)s would generate the following structure of performance log files:

{basedir}/
   system1/
       partition1/
           test_short_name.log
       partition2/
           test_short_name.log
       ...
   system2/
   ...
.logging[].handlers[].append
.logging[].handlers_perflog[].append
Required

No

Default

true

Open each log file in append mode.

The graylog log handler

This handler sends log records to a Graylog server. The additional properties for the graylog handler are the following:

.logging[].handlers[].address
.logging[].handlers_perflog[].address
Required

Yes

The address of the Graylog server defined as host:port.

.logging[].handlers[].extras
.logging[].handlers_perflog[].extras
Required

No

Default

{}

A set of optional key/value pairs to be passed with each log record to the server. These may depend on the server configuration.

This log handler uses internally pygelf. If pygelf is not available, this log handler will be ignored. GELF is a format specification for log messages that are sent over the network. The graylog handler sends log messages in JSON format using an HTTP POST request to the specified address. More details on this log format may be found here. An example configuration of this handler for performance logging is shown here:

{
    'type': 'graylog',
    'address': 'graylog-server:12345',
    'level': 'info',
    'format': '%(message)s',
    'extras': {
        'facility': 'reframe',
        'data-version': '1.0'
    }
}

Although the format is defined for this handler, it is not only the log message that will be transmitted the Graylog server. This handler transmits the whole log record, meaning that all the information will be available and indexable at the remote end.

The stream log handler

This handler sends log records to a file stream. The additional properties for the stream handler are the following:

.logging[].handlers[].name
.logging[].handlers_perflog[].name
Required

No

Default

"stdout"

The name of the file stream to send records to. There are only two available streams:

  • stdout: the standard output.

  • stderr: the standard error.

The syslog log handler

This handler sends log records to UNIX syslog. The additional properties for the syslog handler are the following:

.logging[].handlers[].socktype
.logging[].handlers_perflog[].socktype
Required

No

Default

"udp"

The socket type where this handler will send log records to. There are two socket types:

  • udp: A UDP datagram socket.

  • tcp: A TCP stream socket.

.logging[].handlers[].facility
.logging[].handlers_perflog[].facility
Required

No

Default

"user"

The Syslog facility where this handler will send log records to. The list of supported facilities can be found here.

.logging[].handlers[].address
.logging[].handlers_perflog[].address
Required

Yes

The socket address where this handler will connect to. This can either be of the form <host>:<port> or simply a path that refers to a Unix domain socket.

The httpjson log handler

This handler sends log records in JSON format to a server using HTTP POST requests. The additional properties for the httpjson handler are the following:

.logging[].handlers[].url
.logging[].handlers_perflog[].url
Required

Yes

The URL to be used in the HTTP(S) request server.

.logging[].handlers[].extras
.logging[].handlers_perflog[].extras
Required

No

Default

{}

A set of optional key/value pairs to be passed with each log record to the server. These may depend on the server configuration.

The httpjson handler sends log messages in JSON format using an HTTP POST request to the specified URL.

An example configuration of this handler for performance logging is shown here:

{
    'type': 'httpjson',
    'url': 'http://httpjson-server:12345/rfm',
    'level': 'info',
    'extras': {
        'facility': 'reframe',
        'data-version': '1.0'
    }
}

This handler transmits the whole log record, meaning that all the information will be available and indexable at the remote end.

Scheduler Configuration

A scheduler configuration object contains configuration options specific to the scheduler’s behavior.

Common scheduler options

.schedulers[].name
Required

Yes

The name of the scheduler that these options refer to. It can be any of the supported job scheduler backends.

.schedulers[].job_submit_timeout
Required

No

Default

60

Timeout in seconds for the job submission command. If timeout is reached, the regression test issuing that command will be marked as a failure.

.schedulers[].target_systems
Required

No

Default

["*"]

A list of systems or system/partitions combinations that this scheduler configuration is valid for. For a detailed description of this property, you may refer here.

.schedulers[].use_nodes_option
Required

No

Default

false

Always emit the --nodes Slurm option in the preamble of the job script. This option is relevant to Slurm backends only.

.schedulers[].ignore_reqnodenotavail
Required

No

Default

false

This option is relevant to the Slurm backends only.

If a job associated to a test is in pending state with the Slurm reason ReqNodeNotAvail and a list of unavailable nodes is also specified, ReFrame will check the status of the nodes and, if all of them are indeed down, it will cancel the job. Sometimes, however, when Slurm’s backfill algorithm takes too long to compute, Slurm will set the pending reason to ReqNodeNotAvail and mark all system nodes as unavailable, causing ReFrame to kill the job. In such cases, you may set this parameter to true to avoid this.

.schedulers[].resubmit_on_errors
Required

No

Default

[]

This option is relevant to the Slurm backends only.

If any of the listed errors occur, ReFrame will try to resubmit the job after some seconds. As an example, you could have ReFrame trying to resubmit a job in case that the maximum submission limit per user is reached by setting this field to ["QOSMaxSubmitJobPerUserLimit"]. You can ignore multiple errors at the same time if you add more error strings in the list.

New in version 3.4.1.

Warning

Job submission is a synchronous operation in ReFrame. If this option is set, ReFrame’s execution will block until the error conditions specified in this list are resolved. No other test would be able to proceed.

Execution Mode Configuration

ReFrame allows you to define groups of command line options that are collectively called execution modes. An execution mode can then be selected from the command line with the -mode option. The options of an execution mode will be passed to ReFrame as if they were specified in the command line.

.modes[].name
Required

Yes

The name of this execution mode. This can be used with the --mode command line option to invoke this mode.

.modes[].options
Required

No

Default

[]

The command-line options associated with this execution mode.

.modes[].target_systems
Required

No

Default

["*"]

A list of systems only that this execution mode is valid for. For a detailed description of this property, you may refer here.

General Configuration

.general[].check_search_path
Required

No

Default

["${RFM_INSTALL_PREFIX}/checks/"]

A list of paths (files or directories) where ReFrame will look for regression test files. If the search path is set through the environment variable, it should be a colon separated list. If specified from command line, the search path is constructed by specifying multiple times the command line option.

.general[].check_search_recursive
Required

No

Default

false

Search directories in the search path recursively.

.general[].clean_stagedir
Required

No

Default

true

Clean stage directory of tests before populating it.

New in version 3.1.

.general[].colorize
Required

No

Default

true

Use colors in output. The command-line option sets the configuration option to false.

.general[].compress_report
Required

No

Default

false

Compress the generated run report file. See the documentation of the --compress-report option for more information.

New in version 3.12.0.

.general[].git_timeout
Required

No

Default

5

Timeout value in seconds used when checking if a git repository exists.

.general[].dump_pipeline_progress

Dump pipeline progress for the asynchronous execution policy in pipeline-progress.json. This option is meant for debug purposes only.

Required

No

Default

False

New in version 3.10.0.

.general[].pipeline_timeout

Timeout in seconds for advancing the pipeline in the asynchronous execution policy.

ReFrame’s asynchronous execution policy will try to advance as many tests as possible in their pipeline, but some tests may take too long to proceed (e.g., due to copying of large files) blocking the advancement of previously started tests. If this timeout value is exceeded and at least one test has progressed, ReFrame will stop processing new tests and it will try to further advance tests that have already started.

Required

No

Default

10

New in version 3.10.0.

.general[].remote_detect
Required

No

Default

false

Try to auto-detect processor information of remote partitions as well. This may slow down the initialization of the framework, since it involves submitting auto-detection jobs to the remote partitions. For more information on how ReFrame auto-detects processor information, you may refer to Auto-detecting processor information.

New in version 3.7.0.

.general[].remote_workdir
Required

No

Default

"."

The temporary directory prefix that will be used to create a fresh ReFrame clone, in order to auto-detect the processor information of a remote partition.

New in version 3.7.0.

.general[].ignore_check_conflicts
Required

No

Default

false

Ignore test name conflicts when loading tests.

Deprecated since version 3.8.0: This option will be removed in a future version.

.general[].trap_job_errors
Required

No

Default

false

Trap command errors in the generated job scripts and let them exit immediately.

New in version 3.2.

.general[].keep_stage_files
Required

No

Default

false

Keep stage files of tests even if they succeed.

.general[].module_map_file
Required

No

Default

""

File containing module mappings.

.general[].module_mappings
Required

No

Default

[]

A list of module mappings. If specified through the environment variable, the mappings must be separated by commas. If specified from command line, multiple module mappings are defined by passing the command line option multiple times.

.general[].non_default_craype
Required

No

Default

false

Test a non-default Cray Programming Environment. This will emit some special instructions in the generated build and job scripts. See also --non-default-craype for more details.

.general[].purge_environment
Required

No

Default

false

Purge any loaded environment modules before running any tests.

.general[].report_file
Required

No

Default

"${HOME}/.reframe/reports/run-report-{sessionid}.json"

The file where ReFrame will store its report.

New in version 3.1.

Changed in version 3.2: Default value has changed to avoid generating a report file per session.

Changed in version 4.0.0: Default value was reverted back to generate a new file per run.

.general[].report_junit
Required

No

Default

null

The file where ReFrame will store its report in JUnit format. The report adheres to the XSD schema here.

New in version 3.6.0.

.general[].resolve_module_conflicts
Required

No

Default

true

ReFrame by default resolves any module conflicts and emits the right sequence of module unload and module load commands, in order to load the requested modules. This option disables this behavior if set to false.

You should avoid using this option for modules system that cannot handle module conflicts automatically, such as early Tmod verions.

Disabling the automatic module conflict resolution, however, can be useful when modules in a remote system partition are not present on the host where ReFrame runs. In order to resolve any module conflicts and generate the right load sequence of modules, ReFrame loads temporarily the requested modules and tracks any conflicts along the way. By disabling this option, ReFrame will simply emit the requested module load commands without attempting to load any module.

New in version 3.6.0.

.general[].save_log_files
Required

No

Default

false

Save any log files generated by ReFrame to its output directory

.general[].target_systems
Required

No

Default

["*"]

A list of systems or system/partitions combinations that these general options are valid for. For a detailed description of this property, you may refer here.

.general[].timestamp_dirs
Required

No

Default

""

Append a timestamp to ReFrame directory prefixes. Valid formats are those accepted by the time.strftime() function. If specified from the command line without any argument, "%FT%T" will be used as a time format.

.general[].unload_modules
Required

No

Default

[]

A list of environment module objects to unload before executing any test. If specified using an the environment variable, a space separated list of modules is expected. If specified from the command line, multiple modules can be passed by passing the command line option multiple times.

.general[].use_login_shell
Required

No

Default

false

Use a login shell for the generated job scripts. This option will cause ReFrame to emit -l in the shebang of shell scripts. This option, if set to true, may cause ReFrame to fail, if the shell changes permanently to a different directory during its start up.

.general[].user_modules
Required

No

Default

[]

A list of environment module objects to be loaded before executing any test. If specified using an the environment variable, a space separated list of modules is expected. If specified from the command line, multiple modules can be passed by passing the command line option multiple times.

.general[].verbose
Required

No

Default

0

Set the verbosity level of the output. The higher the number, the more verbose the output will be. If set to a negative number, this will decrease the verbosity level.

Module Objects

New in version 3.3.

A module object in ReFrame’s configuration represents an environment module. It can either be a simple string or a JSON object with the following attributes:

.name
Required

Yes

The name of the module.

.collection
Required

No

Default

false

A boolean value indicating whether this module refers to a module collection. Module collections are treated differently from simple modules when loading.

path
Required

No

Default

null

If the module is not present in the default MODULEPATH, the module’s location can be specified here. ReFrame will make sure to set and restore the MODULEPATH accordingly for loading the module.

New in version 3.5.0.

See also

Module collections with Environment Modules and Lmod.

Processor Info

New in version 3.5.0.

A processor info object in ReFrame’s configuration is used to hold information about the processor of a system partition and is made available to the tests through the processor attribute of the current_partition.

.arch
Required

No

Default

None

The microarchitecture of the processor.

.num_cpus
Required

No

Default

None

Number of logical CPUs.

.num_cpus_per_core
Required

No

Default

None

Number of logical CPUs per core.

.num_cpus_per_socket
Required

No

Default

None

Number of logical CPUs per socket.

.num_sockets
Required

No

Default

None

Number of sockets.

.topology
Required

No

Default

None

Processor topology. An example follows:

'topology': {
   'numa_nodes': ['0x000000ff'],
   'sockets': ['0x000000ff'],
   'cores': ['0x00000003', '0x0000000c',
             '0x00000030', '0x000000c0'],
   'caches': [
      {
            'type': 'L3',
            'size': 6291456,
            'linesize': 64,
            'associativity': 0,
            'num_cpus': 8,
            'cpusets': ['0x000000ff']
      },
      {
            'type': 'L2',
            'size': 262144,
            'linesize': 64,
            'associativity': 4,
            'num_cpus': 2,
            'cpusets': ['0x00000003', '0x0000000c',
                        '0x00000030', '0x000000c0']
      },
      {
            'type': 'L1',
            'size': 32768,
            'linesize': 64,
            'associativity': 0,
            'num_cpus': 2,
            'cpusets': ['0x00000003', '0x0000000c',
                        '0x00000030', '0x000000c0']
      }
   ]
}

Device Info

New in version 3.5.0.

A device info object in ReFrame’s configuration is used to hold information about a specific type of devices in a system partition and is made available to the tests through the devices attribute of the current_partition.

.type
Required

No

Default

None

The type of the device, for example "gpu".

.arch
Required

No

Default

None

The microarchitecture of the device.

.num_devices
Required

No

Default

None

Number of devices of this type inside the system partition.