Command Line Reference¶

Synopsis¶

reframe [OPTION]... ACTION¶

Description¶

ReFrame provides both a programming interface for writing regression tests and a command-line interface for managing and running the tests, which is detailed here. The reframe command is part of ReFrame’s frontend. This frontend is responsible for loading and running regression tests written in ReFrame. ReFrame executes tests by sending them down to a well defined pipeline. The implementation of the different stages of this pipeline is part of ReFrame’s core architecture, but the frontend is responsible for driving this pipeline and executing tests through it. There are three basic phases that the frontend goes through, which are described briefly in the following.

Test discovery and test loading¶

This is the very first phase of the frontend. ReFrame will search for tests in its check search path and will load them. When ReFrame loads a test, it actually instantiates it, meaning that it will call its __init__() method unconditionally whether this test is meant to run on the selected system or not. This is something that writers of regression tests should bear in mind.

-c, --checkpath=PATH¶

A filesystem path where ReFrame should search for tests.

PATH can be a directory or a single test file. If it is a directory, ReFrame will search for test files inside this directory load all tests found in them. This option can be specified multiple times, in which case each PATH will be searched in order.

The check search path can also be set using the RFM_CHECK_SEARCH_PATH environment variable or the check_search_path general configuration parameter.

-R, --recursive¶

Search for test files recursively in directories found in the check search path.

This option can also be set using the RFM_CHECK_SEARCH_RECURSIVE environment variable or the check_search_recursive general configuration parameter.

Test filtering¶

After all tests in the search path have been loaded, they are first filtered by the selected system. Any test that is not valid for the current system, it will be filtered out. The current system is either auto-selected or explicitly specified with the --system option. Tests can be filtered by different attributes and there are specific command line options for achieving this. A common characteristic of all test filtering options is that if a test is selected, then all its dependencies will be selected, too, regardless if they match the filtering criteria or not. This happens recursively so that if test T1 depends on T2 and T2 depends on T3, then selecting T1 would also select T2 and T3.

--cpu-only¶

Select tests that do not target GPUs.

These are all tests with num_gpus_per_node equals to zero This option and --gpu-only are mutually exclusive.

The --gpu-only and --cpu-only check only the value of the num_gpus_per_node attribute of tests. The value of this attribute is not required to be non-zero for GPU tests. Tests may or may not make use of it.

Deprecated since version 4.4: Please use -E 'not num_gpus_per_node' instead.

-E, --filter-expr=EXPR¶

Select only tests that satisfy the given expression.

The expression EXPR can be any valid Python expression on the test variables or parameters. For example, -E num_tasks > 10 will select all tests, whose num_tasks exceeds 10. You may use any test variable in expression, even user-defined. Multiple variables can also be included such as -E num_tasks >= my_param, where my_param is user-defined parameter.

New in version 4.4.

--failed¶

Select only the failed test cases for a previous run.

This option can only be used in combination with the --restore-session. To rerun the failed cases from the last run, you can use reframe --restore-session --failed -r.

New in version 3.4.

--gpu-only¶

Select tests that can run on GPUs.

These are all tests with num_gpus_per_node greater than zero. This option and --cpu-only are mutually exclusive.

Deprecated since version 4.4: Please use -E num_gpus_per_node instead.

--maintainer=MAINTAINER¶

Filter tests by maintainer.

MAINTAINER is interpreted as a Python Regular Expression; all tests that have at least a matching maintainer will be selected. MAINTAINER being a regular expression has the implication that --maintainer 'foo' will select also tests that define 'foobar' as a maintainer. To restrict the selection to tests defining only 'foo', you should use --maintainer 'foo$'.

This option may be specified multiple times, in which case only tests defining or matching all maintainers will be selected.

New in version 3.9.1.

Changed in version 4.1.0: The MAINTAINER pattern is matched anywhere in the maintainer’s name and not at its beginning. If you want to match at the beginning of the name, you should prepend ^.

-n, --name=NAME¶

Filter tests by name.

NAME is interpreted as a Python Regular Expression; any test whose display name matches NAME will be selected. The display name of a test encodes also any parameterization information. See Test Naming Scheme for more details on how the tests are automatically named by the framework.

Before matching, any whitespace will be removed from the display name of the test.

This option may be specified multiple times, in which case tests with any of the specified names will be selected: -n NAME1 -n NAME2 is therefore equivalent to -n 'NAME1|NAME2'.

If the special notation <test_name>@<variant_num> is passed as the NAME argument, then an exact match will be performed selecting the variant variant_num of the test test_name.

You may also select a test by its hash code using the notation /<test-hash> for the NAME argument.

Note

Fixtures cannot be selected.

Changed in version 3.10.0: The option’s behaviour was adapted and extended in order to work with the updated test naming scheme.

Changed in version 4.0.0: Support selecting tests by their hash code.

Changed in version 4.1.0: The NAME pattern is matched anywhere in the test name and not at its beginning. If you want to match at the beginning of a test name, you should prepend ^.

-p, --prgenv=NAME¶

Filter tests by programming environment.

NAME is interpreted as a Python Regular Expression; any test for which at least one valid programming environment is matching NAME will be selected.

This option may be specified multiple times, in which case only tests matching all of the specified programming environments will be selected.

--skip-prgenv-check¶

Do not filter tests against programming environments.

Even if the -p option is not specified, ReFrame will filter tests based on the programming environments defined for the currently selected system. This option disables that filter completely.

--skip-system-check¶: Do not filter tests against the selected system.

-T, --exclude-tag=TAG¶

Exclude tests by tags.

TAG is interpreted as a Python Regular Expression; any test with tags matching TAG will be excluded.

This option may be specified multiple times, in which case tests with any of the specified tags will be excluded: -T TAG1 -T TAG2 is therefore equivalent to -T 'TAG1|TAG2'.

Changed in version 4.1.0: The TAG pattern is matched anywhere in the tag name and not at its beginning. If you want to match at the beginning of a tag, you should prepend ^.

-t, --tag=TAG¶

Filter tests by tag.

TAG is interpreted as a Python Regular Expression; all tests that have at least a matching tag will be selected. TAG being a regular expression has the implication that -t 'foo' will select also tests that define 'foobar' as a tag. To restrict the selection to tests defining only 'foo', you should use -t 'foo$'.

This option may be specified multiple times, in which case only tests defining or matching all tags will be selected.

Changed in version 4.1.0: The TAG pattern is matched anywhere in the tag name and not at its beginning. If you want to match at the beginning of a tag, you should prepend ^.

-x, --exclude=NAME¶

Exclude tests by name.

NAME is interpreted as a Python Regular Expression; any test whose name matches NAME will be excluded.

This option may be specified multiple times, in which case tests with any of the specified names will be excluded: -x NAME1 -x NAME2 is therefore equivalent to -x 'NAME1|NAME2'.

Changed in version 4.1.0: The NAME pattern is matched anywhere in the test name and not at its beginning. If you want to match at the beginning of a test name, you should prepend ^.

Test actions¶

ReFrame will finally act upon the selected tests. There are currently two actions that can be performed on tests: (a) list the tests and (b) execute the tests. An action must always be specified.

--ci-generate=FILE¶

Do not run the tests, but generate a Gitlab child pipeline specification in FILE.

You can set up your Gitlab CI to use the generated file to run every test as a separate Gitlab job respecting test dependencies. For more information, have a look in Integrating into a CI pipeline.

Note

This option will not work with the test generation options <#test-generators>.

New in version 3.4.1.

--describe¶: Print a detailed description of the selected tests in JSON format and exit.

Note

The generated test description corresponds to its state after it has been initialized. If any of its attributes are changed or set during its execution, their updated values will not be shown by this listing.

New in version 3.10.0.

--dry-run¶

Dry run the selected tests.

The dry-run mode will try to execute as much of the test pipeline as possible. More specifically, the tests will not be submitted and will not be run for real, but their stage directory will be prepared and the corresponding job script will be emitted. Similarly, the sanity and performance functions will not be evaluated but all the preparation will happen. Tests run in dry-run mode will not fail unless there is a programming error in the test or if the test tries to use a resource that is not produced in dry run mode (e.g., access the standard output or a resource produced by a dependency outside any sanity or performance function). In this case, users can call the is_dry_run() method in their test and take a specific action if the test is run in dry-run mode.

New in version 4.1.

-L, --list-detailed[=T|C]¶

List selected tests providing more details for each test.

The unique id of each test (see also unique_name) as well as the file where each test is defined are printed.

This option accepts optionally a single argument denoting what type of listing is requested. Please refer to -l for an explanation of this argument.

New in version 3.10.0: Support for different types of listing is added.

Changed in version 4.0.5: The variable names to which fixtures are bound are also listed. See Test Naming Scheme for more information.

-l, --list[=T|C]¶

List selected tests and their dependencies.

This option accepts optionally a single argument denoting what type of listing is requested. There are two types of possible listings:

Regular test listing (T, the default): This type of listing lists the tests and their dependencies or fixtures using their display_name. A test that is listed as a dependency of another test will not be listed separately.
Concretized test case listing (C): This type of listing lists the exact test cases and their dependencies as they have been concretized for the current system and environment combinations. This listing shows practically the exact test DAG that will be executed.

New in version 3.10.0: Support for different types of listing is added.

Changed in version 4.0.5: The variable names to which fixtures are bound are also listed. See Test Naming Scheme for more information.

--list-tags¶

List the unique tags of the selected tests.

The tags are printed in alphabetical order.

New in version 3.6.0.

-r, --run¶: Execute the selected tests.

If more than one action options are specified, the precedence order is the following:

--describe > --list-detailed > --list > --list-tags > --ci-generate

Options controlling ReFrame output¶

--compress-report¶

Compress the generated run report (see --report-file). The generated report is a JSON file formatted in a human readable form. If this option is enabled, the generated JSON file will be a single stream of text without additional spaces or new lines.

This option can also be set using the RFM_COMPRESS_REPORT environment variable or the compress_report general configuration parameter.

New in version 3.12.0.

--dont-restage¶

Do not restage a test if its stage directory exists. Normally, if the stage directory of a test exists, ReFrame will remove it and recreate it. This option disables this behavior.

This option can also be set using the RFM_CLEAN_STAGEDIR environment variable or the clean_stagedir general configuration parameter.

New in version 3.1.

--keep-stage-files¶

Keep test stage directories even for tests that finish successfully.

This option can also be set using the RFM_KEEP_STAGE_FILES environment variable or the keep_stage_files general configuration parameter.

-o, --output=DIR¶

Directory prefix for test output files.

When a test finishes successfully, ReFrame copies important output files to a test-specific directory for future reference. This test-specific directory is of the form {output_prefix}/{system}/{partition}/{environment}/{test_name}, where output_prefix is set by this option. The test files saved in this directory are the following:

The ReFrame-generated build script, if not a run-only test.
The standard output and standard error of the build phase, if not a run-only test.
The ReFrame-generated job script, if not a compile-only test.
The standard output and standard error of the run phase, if not a compile-only test.
Any additional files specified by the keep_files regression test attribute.

This option can also be set using the RFM_OUTPUT_DIR environment variable or the outputdir system configuration parameter.

--perflogdir=DIR¶

Directory prefix for logging performance data.

This option is relevant only to the filelog logging handler.

This option can also be set using the RFM_PERFLOG_DIR environment variable or the basedir logging handler configuration parameter.

--prefix=DIR¶

General directory prefix for ReFrame-generated directories.

The base stage and output directories (see below) will be specified relative to this prefix if not specified explicitly.

This option can also be set using the RFM_PREFIX environment variable or the prefix system configuration parameter.

--report-file=FILE¶

The file where ReFrame will store its report.

The FILE argument may contain the special placeholder {sessionid}, in which case ReFrame will generate a new report each time it is run by appending a counter to the report file. If the report is generated in the default location (see the report_file configuration option), a symlink to the latest report named latest.json will also be created.

This option can also be set using the RFM_REPORT_FILE environment variable or the report_file general configuration parameter.

New in version 3.1.

New in version 4.2: Symlink to the latest report is now created.

--report-junit=FILE¶

Instruct ReFrame to generate a JUnit XML report in FILE.

The generated report adheres to the XSD schema here where each retry is treated as an individual testsuite.

This option can also be set using the RFM_REPORT_JUNIT environment variable or the report_junit general configuration parameter.

New in version 3.6.0.

Changed in version 3.6.1: Added support for retries in the JUnit XML report.

-s, --stage=DIR¶

Directory prefix for staging test resources.

ReFrame does not execute tests from their original source directory. Instead it creates a test-specific stage directory and copies all test resources there. It then changes to that directory and executes the test. This test-specific directory is of the form {stage_prefix}/{system}/{partition}/{environment}/{test_name}, where stage_prefix is set by this option. If a test finishes successfully, its stage directory will be removed.

This option can also be set using the RFM_STAGE_DIR environment variable or the stagedir system configuration parameter.

--save-log-files¶

Save ReFrame log files in the output directory before exiting.

Only log files generated by file log handlers will be copied.

This option can also be set using the RFM_SAVE_LOG_FILES environment variable or the save_log_files general configuration parameter.

--timestamp [TIMEFMT]¶

Append a timestamp to the output and stage directory prefixes.

TIMEFMT can be any valid strftime(3) time format. If not specified, TIMEFMT is set to %FT%T.

This option can also be set using the RFM_TIMESTAMP_DIRS environment variable or the timestamp_dirs general configuration parameter.

Options controlling ReFrame execution¶

--disable-hook=HOOK¶

Disable the pipeline hook named HOOK from all the tests that will run.

This feature is useful when you have implemented test workarounds as pipeline hooks, in which case you can quickly disable them from the command line. This option may be specified multiple times in order to disable multiple hooks at the same time.

New in version 3.2.

--duration=TIMEOUT¶

Run the test session repeatedly until the specified timeout expires.

TIMEOUT can be specified in one of the following forms:

<int> or <float>: number of seconds
<days>d<hours>h<minutes>m<seconds>s: a string denoting days, hours, minutes and/or seconds.

At the end, failures from every run will be reported and, similarly, the failure statistics printed by the --failure-stats option will include all runs.

New in version 4.2.

--exec-order=ORDER¶

Impose an execution order for the independent tests. The ORDER argument can take one of the following values:

name: Order tests by their display name.
rname: Order tests by their display name in reverse order.
uid: Order tests by their unique name.
ruid: Order tests by their unique name in reverse order.
random: Randomize the order of execution.

If this option is not specified the order of execution of independent tests is implementation defined. This option can be combined with any of the listing options (-l or -L) to list the tests in the order.

New in version 4.0.0.

--exec-policy=POLICY¶

The execution policy to be used for running tests.

There are two policies defined:

serial: Tests will be executed sequentially.
async: Tests will be executed asynchronously. This is the default policy.

The async execution policy executes the build and run phases of tests asynchronously by submitting their associated jobs in a non-blocking way. ReFrame’s runtime monitors the progress of each test and will resume the pipeline execution of an asynchronously spawned test as soon as its build or run phase have finished. Note that the rest of the pipeline stages are still executed sequentially in this policy.

Concurrency can be controlled by setting the max_jobs system partition configuration parameter. As soon as the concurrency limit is reached, ReFrame will first poll the status of all its pending tests to check if any execution slots have been freed up. If there are tests that have finished their build or run phase, ReFrame will keep pushing tests for execution until the concurrency limit is reached again. If no execution slots are available, ReFrame will throttle job submission.

--max-retries=NUM¶

The maximum number of times a failing test can be retried.

The test stage and output directories will receive a _retry<N> suffix every time the test is retried.

--maxfail=NUM¶

The maximum number of failing test cases before the execution is aborted.

After NUM failed test cases the rest of the test cases will be aborted. The counter of the failed test cases is reset to 0 in every retry.

--mode=MODE¶

ReFrame execution mode to use.

An execution mode is simply a predefined set of options that is set in the modes configuration parameter. Additional options can be passed to the command line, in which case they will be combined with the options defined in the selected execution mode. More specifically, any additional ReFrame options will be appended to the command line options of the selected mode. As a result, if a normal option is specified both inside the execution mode and the in the command line, the command line option will take precedence. On the other hand, if an option that is allowed to be specified multiple times, e.g., the -S option, is passed both inside the execution mode and in the command line, their values will be combined. For example, if the execution mode foo defines -S modules=foo, the invocation --mode=foo -S num_tasks=10 is the equivalent of -S modules=foo -S num_tasks=10.

Changed in version 4.1: Options that can be specified multiple times are now combined between execution modes and the command line.

--reruns=N¶

Rerun the whole test session N times.

In total, the selected tests will run N+1 times as the first time does not count as a rerun.

At the end, failures from every run will be reported and, similarly, the failure statistics printed by the --failure-stats option will include all runs.

Although similar to --repeat, this option behaves differently. This option repeats the whole test session multiple times. All the tests of the session will finish before a new run is started. The --repeat option on the other hand generates clones of the selected tests and schedules them for running in a single session. As a result, all the test clones will run (by default) concurrently.

New in version 4.2.

--restore-session [REPORT1[,REPORT2,...]]¶

Restore a testing session that has run previously.

REPORT1 etc. are a run report files generated by ReFrame. If a report is not given, ReFrame will pick the last report file found in the default location of report files (see the --report-file option). If passed alone, this option will simply rerun all the test cases that have run previously based on the report file data. It is more useful to combine this option with any of the test filtering options, in which case only the selected test cases will be executed. The difference in test selection process when using this option is that the dependencies of the selected tests will not be selected for execution, as they would normally, but they will be restored. For example, if test T1 depends on T2 and T2 depends on T3, then running reframe -n T1 -r would cause both T2 and T3 to run. However, by doing reframe -n T1 --restore-session -r, only T1 would run and its immediate dependence T2 will be restored. This is useful when you have deep test dependencies or some of the tests in the dependency chain are very time consuming.

Multiple reports may be passed as a comma-separated list. ReFrame will try to restore any required test case by looking it up in each report sequentially. If it cannot find it, it will issue an error and exit.

Note

In order for a test case to be restored, its stage directory must be present. This is not a problem when rerunning a failed case, since the stage directories of its dependencies are automatically kept, but if you want to rerun a successful test case, you should make sure to have run with the --keep-stage-files option.

Note

This option will not work with the test generation options <#test-generators>.

New in version 3.4.

Changed in version 3.6.1: Multiple report files are now accepted.

-S, --setvar=[TEST.]VAR=VAL¶

Set variable VAR in all tests or optionally only in test TEST to VAL.

TEST can have the form [TEST.][FIXT.]*, in which case VAR will be set in fixture FIXT of TEST. Note that this syntax is recursive on fixtures, so that a variable can be set in a fixture arbitrarily deep. TEST prefix refers to the test class name, not the test name and FIXT refers to the fixture variable name inside the referenced test, i.e., the test variable to which the fixture is bound. The fixture variable name is referred to as '<varname> when listing tests with the -l and -L options.

Multiple variables can be set at the same time by passing this option multiple times. This option cannot change arbitrary test attributes, but only test variables declared with the variable built-in. If an attempt is made to change an inexistent variable or a test parameter, a warning will be issued.

ReFrame will try to convert VAL to the type of the variable. If it does not succeed, a warning will be issued and the variable will not be set. VAL can take the special value @none to denote that the variable must be set to None. Boolean variables can be set in one of the following ways:

By passing true, yes or 1 to set them to True.
By passing false, no or 0 to set them to False.

Passing any other value will issue an error.

Note

Boolean variables in a test must be declared of type Bool and not of the built-in bool type, in order to adhere to the aforementioned behaviour. If a variable is defined as bool there is no way you can set it to False, since all strings in Python evaluate to True.

Sequence and mapping types can also be set from the command line by using the following syntax:

Sequence types: -S seqvar=1,2,3,4
Mapping types: -S mapvar=a:1,b:2,c:3

Nested mapping types can also be converted using JSON syntax. For example, the extra_resources complex dictionary could be set with -S extra_resources='{"gpu": {"num_gpus_per_node":8}}'.

Conversions to arbitrary objects are also supported. See ConvertibleType for more details.

Variable assignments passed from the command line happen before the test is instantiated and is the exact equivalent of assigning a new value to the variable at the end of the test class body. This has a number of implications that users of this feature should be aware of:

In the following test, num_tasks will have always the value 1 regardless of any command-line assignment of the variable foo:

@rfm.simple_test
class my_test(rfm.RegressionTest):
    foo = variable(int, value=1)
    num_tasks = foo

Tip

In cases where the class body expresses logic as a function of a variable and this variable, as well as its dependent logic, need to be controlled externally, the variable’s default value (i.e. the value set through the value argument) may be modified as follows through an environment variable and not through the -S option:

import os

@rfm.simple_test
class my_test(rfm.RegressionTest):
    max_nodes = variable(int, value=int(os.getenv('MAX_NODES', 1)))
    # Parameterise number of nodes
    num_nodes = parameter((1 << i for i in range(0, int(max_nodes))))

If the variable is set in any pipeline hook, the command line assignment will have an effect until the variable assignment in the pipeline hook is reached. The variable will be then overwritten.
The test filtering happens after a test is instantiated, so the only way to scope a variable assignment is to prefix it with the test class name. However, this has some positive side effects:
- Passing -S valid_systems='*' and -S valid_prog_environs='*' is the equivalent of passing the --skip-system-check and --skip-prgenv-check options.
- Users could alter the behavior of tests based on tag values that they pass from the command line, by changing the behavior of a test in a post-init hook based on the value of the tags attribute.
- Users could force a test with required variables to run if they set these variables from the command line. For example, the following test could only be run if invoked with -S num_tasks=<NUM>:
```
@rfm.simple_test
class my_test(rfm.RegressionTest):
    num_tasks = required
```

New in version 3.8.0.

Changed in version 3.9.3: Proper handling of boolean variables.

Changed in version 3.11.1: Allow setting variables in fixtures.

Changed in version 4.4: Allow setting nested mapping types using JSON syntax.

--skip-performance-check¶

Skip performance checking phase.

The phase is completely skipped, meaning that performance data will not be logged.

--skip-sanity-check¶: Skip sanity checking phase.

Options controlling job submission¶

-J, --job-option=OPTION¶

Pass OPTION directly to the job scheduler backend.

The syntax of OPTION is -J key=value. If OPTION starts with - it will be passed verbatim to the backend job scheduler. If OPTION starts with # it will be emitted verbatim in the job script. Otherwise, ReFrame will pass --key value or -k value (if key is a single character) to the backend scheduler. Any job options specified with this command-line option will be emitted after any job options specified in the access system partition configuration parameter.

Especially for the Slurm backends, constraint options, such as -J constraint=value, -J C=value, -J --constraint=value or -J -C=value, are going to be combined with any constraint options specified in the access system partition configuration parameter. For example, if -C x is specified in the access and -J C=y is passed to the command-line, ReFrame will pass -C x&y as a constraint to the scheduler. Notice, however, that if constraint options are specified through multiple -J options, only the last one will be considered. If you wish to completely overwrite any constraint options passed in access, you should consider passing explicitly the Slurm directive with -J '#SBATCH --constraint=new'.

Changed in version 3.0: This option has become more flexible.

Changed in version 3.1: Use & to combine constraints.

Flexible node allocation¶

ReFrame can automatically set the number of tasks of a test, if its num_tasks attribute is set to a value less than or equal to zero. This scheme is conveniently called flexible node allocation and is valid only for the Slurm backend. When allocating nodes automatically, ReFrame will take into account all node limiting factors, such as partition access options, and any job submission control options described above. Nodes from this pool are allocated according to different policies. If no node can be selected, the test will be marked as a failure with an appropriate message.

--flex-alloc-nodes=POLICY¶

Set the flexible node allocation policy.

Available values are the following:

all: Flexible tests will be assigned as many tasks as needed in order to span over all the nodes of the node pool.
STATE: Flexible tests will be assigned as many tasks as needed in order to span over the nodes that are currently in state STATE. Querying of the node state and submission of the test job are two separate steps not executed atomically. It is therefore possible that the number of tasks assigned does not correspond to the actual nodes in the given state.

If this option is not specified, the default allocation policy for flexible tests is ‘idle’.
Any positive integer: Flexible tests will be assigned as many tasks as needed in order to span over the specified number of nodes from the node pool.

Changed in version 3.1: It is now possible to pass an arbitrary node state as a flexible node allocation parameter.

Options controlling ReFrame environment¶

ReFrame offers the ability to dynamically change its environment as well as the environment of tests. It does so by leveraging the selected system’s environment modules system.

-M, --map-module=MAPPING¶

Apply a module mapping.

ReFrame allows manipulating test modules on-the-fly using module mappings. A module mapping has the form old_module: module1 [module2]... and will cause ReFrame to replace a module with another list of modules upon load time. For example, the mapping foo: foo/1.2 will load module foo/1.2 whenever module foo needs to be loaded. A mapping may also be self-referring, e.g., gnu: gnu gcc/10.1, however cyclic dependencies in module mappings are not allowed and ReFrame will issue an error if it detects one. This option is especially useful for running tests using a newer version of a software or library.

This option may be specified multiple times, in which case multiple mappings will be applied.

This option can also be set using the RFM_MODULE_MAPPINGS environment variable or the module_mappings general configuration parameter.

Changed in version 3.3: If the mapping replaces a module collection, all new names must refer to module collections, too.

Options for generating tests dynamically¶

These options generate new tests dynamically from a set of previously selected tests. The way the tests are generated and how they interact with the test filtering options poses some limitations:

These tests do not have an associated test file and are different from their original tests although the share the same base name. As a result, the --restore-session option cannot be used to restore dynamically generated tests.
Since these tests are generated after the test selection phase, the --ci-generate option cannot be used to generate a child pipeline, as the child pipeline uses the -n option to select the tests for running.

--distribute[=NODESTATE]¶

Distribute the selected tests on all the nodes in state NODESTATE in their respective valid partitions.

ReFrame will parameterize and run the tests on the selected nodes. Effectively, it will dynamically create new tests that inherit from the original tests and add a new parameter named $nid which contains the list of nodes that the test must run on. The new tests are named with the following pattern {orig_test_basename}_{partition_fullname}.

When determining the list of nodes to distribute the selected tests, ReFrame will take into account any job options passed through the -J option.

You can optionally specify the state of the nodes to consider when distributing the test through the NODESTATE argument:

all: Tests will run on all the nodes of their respective valid partitions regardless of the nodes’ state.
idle: Tests will run on all idle nodes of their respective valid partitions.
NODESTATE: Tests will run on all the nodes in state NODESTATE of their respective valid partitions. If NODESTATE is not specified, idle will be assumed.

The state of the nodes will be determined once, before beginning the execution of the tests, so it might be different at the time the tests are actually submitted.

Note

Currently, only single-node jobs can be distributed and only local or the Slurm-based backends support this feature.

Note

Distributing tests with dependencies is not supported, but you can distribute tests that use fixtures.

New in version 3.11.0.

-P, --parameterize=[TEST.]VAR=VAL0,VAL1,...¶

Parameterize a test on an existing variable.

This option will create a new test with a parameter named $VAR with the values given in the comma-separated list VAL0,VAL1,.... The values will be converted based on the type of the target variable VAR. The TEST. prefix will only parameterize the variable VAR of test TEST.

The -P can be specified multiple times in order to parameterize multiple variables.

Note

Conversely to the -S option that can set a variable in an arbitrarily nested fixture, the -P option can only parameterize the leaf test: it cannot be used to parameterize a fixture of the test.

Note

The -P option supports only tests that use fixtures. Tests that use raw dependencies are not supported.

New in version 4.3.

--repeat=N¶: Repeat the selected tests N times. This option can be used in conjunction with the --distribute option in which case the selected tests will be repeated multiple times and distributed on individual nodes of the system’s partitions.

Note

Repeating tests with dependencies is not supported, but you can repeat tests that use fixtures.

New in version 3.12.0.

Miscellaneous options¶

-C, --config-file=FILE¶

Use FILE as configuration file for ReFrame.

This option can be passed multiple times, in which case multiple configuration files will be read and loaded successively. The base of the configuration chain is always the builtin configuration file, namely the ${RFM_INSTALL_PREFIX}/reframe/core/settings.py. At any point, the user can “break” the chain of configuration files by prefixing the configuration file name with a colon as in the following example: -C :/path/to/new_config.py. This will ignore any previously loaded configuration file and will only load the one specified. Note, however, that the builtin configuration file cannot be overriden; It will always be loaded first in the chain.

This option can also be set using the RFM_CONFIG_FILES environment variable.

In order to determine its final configuration, ReFrame first loads the builtin configuration file unconditionally and then starts looking for possible configuration file locations defined in the RFM_CONFIG_PATH environment variable. For each directory defined in the RFM_CONFIG_PATH, ReFrame looks for a file named settings.py or settings.json inside it and loads it. If both a settings.py and a settings.json files are found, the Python configuration will be preferred. ReFrame, finally, processes any configuration files specified in the command line or in the RFM_CONFIG_FILES environment variable.

Changed in version 4.0.0.

--detect-host-topology[=FILE]¶

Detect the local host processor topology, store it to FILE and exit.

If no FILE is specified, the standard output will be used.

New in version 3.7.0.

--failure-stats¶: Print failure statistics at the end of the run.

-h, --help¶: Print a short help message and exit.

--nocolor¶

Disable output coloring.

This option can also be set using the RFM_COLORIZE environment variable or the colorize general configuration parameter.

--performance-report¶

Print a performance report for all the performance tests that have been run.

The report shows the performance values retrieved for the different performance variables defined in the tests.

-q, --quiet¶

Decrease the verbosity level.

This option can be specified multiple times. Every time this option is specified, the verbosity level will be decreased by one. This option can be combined arbitrarily with the -v option, in which case the final verbosity level will be determined by the final combination. For example, specifying -qv will not change the verbosity level, since the two options cancel each other, but -qqv is equivalent to -q. For a list of ReFrame’s verbosity levels, see the description of the -v option.

New in version 3.9.3.

--show-config [PARAM]¶

Show the value of configuration parameter PARAM as this is defined for the currently selected system and exit.

The parameter value is printed in JSON format. If PARAM is not specified or if it set to all, the whole configuration for the currently selected system will be shown. Configuration parameters are formatted as a path navigating from the top-level configuration object to the actual parameter. The / character acts as a selector of configuration object properties or an index in array objects. The @ character acts as a selector by name for configuration objects that have a name property. Here are some example queries:

Retrieve all the partitions of the current system:
```
reframe --show-config=systems/0/partitions
```

Retrieve the job scheduler of the partition named default:

reframe --show-config=systems/0/partitions/@default/scheduler

Retrieve the check search path for system foo:

reframe --system=foo --show-config=general/0/check_search_path

--system=NAME¶

Load the configuration for system NAME.

The NAME must be a valid system name in the configuration file. It may also have the form SYSNAME:PARTNAME, in which case the configuration of system SYSNAME will be loaded, but as if it had PARTNAME as its sole partition. Of course, PARTNAME must be a valid partition of system SYSNAME. If this option is not specified, ReFrame will try to pick the correct configuration entry automatically. It does so by trying to match the hostname of the current machine again the hostname patterns defined in the hostnames system configuration parameter. The system with the first match becomes the current system.

This option can also be set using the RFM_SYSTEM environment variable.

--upgrade-config-file=OLD[:NEW]¶

Convert the old-style configuration file OLD, place it into the new file NEW and exit.

If a new file is not given, a file in the system temporary directory will be created.

-V, --version¶: Print version and exit.

-v, --verbose¶

Increase verbosity level of output.

This option can be specified multiple times. Every time this option is specified, the verbosity level will be increased by one. There are the following message levels in ReFrame listed in increasing verbosity order: critical, error, warning, info, verbose and debug. The base verbosity level of the output is defined by the level stream logging handler configuration parameter.

This option can also be set using the RFM_VERBOSE environment variable or the verbose general configuration parameter.

Test Naming Scheme¶

New in version 3.10.0.

This section describes the test naming scheme. This scheme has superseded the old one in ReFrame 4.0.

Each ReFrame test is assigned a unique name, which will be used internally by the framework to reference the test. Any test-specific path component will use that name, too. It is formed as follows for the various types of tests:

Regular tests: The unique name is simply the test class name. This implies that you cannot load two tests with the same class name within the same run session even if these tests reside in separate directories.
Parameterized tests: The unique name is formed by the test class name followed by an _ and the variant number of the test. Each point in the parameter space of the test is assigned a unique variant number.
Fixtures: The unique name is formed by the test class name followed by an _ and a hash. The hash is constructed by combining the information of the fixture variant (if the fixture is parameterized), the fixture’s scope and any fixture variables that were explicitly set.

Since unique names can be cryptic, they are not listed by the -l option, but are listed when a detailed listing is requested by using the -L option.

A human readable version of the test name, which is called the display name, is also constructed for each test. This name encodes all the parameterization information as well as the fixture-specific information (scopes, variables). The format of the display name is the following in BNF notation:

<display_name> ::= <test_class_name> (<params>)* (<scope> ("'"<fixtvar>)+)?
<params> ::= "%" <parametrization> "=" <pvalue>
<parametrization> ::= (<fname> ".")* <pname>
<scope> ::= "~" <scope_descr>
<scope_descr> ::= <first> ("+" <second>)*

<test_class_name> ::= (* as in Python *)
<fname> ::= (* string *)
<pname> ::= (* string *)
<pvalue> ::= (* string *)
<first> ::= (* string *)
<second> ::= (* string *)
<fixtvar> ::= (* string *)

The following is an example of a fictitious complex test that is itself parameterized and depends on parameterized fixtures as well.

import reframe as rfm


class MyFixture(rfm.RunOnlyRegressionTest):
    p = parameter([1, 2])


class X(rfm.RunOnlyRegressionTest):
    foo = variable(int, value=1)


@rfm.simple_test
class TestA(rfm.RunOnlyRegressionTest):
    f = fixture(MyFixture, scope='test', action='join')
    x = parameter([3, 4])
    t = fixture(MyFixture, scope='test')
    l = fixture(X, scope='environment', variables={'foo': 10})
    valid_systems = ['*']
    valid_prog_environs = ['*']

Here is how this test is listed where the various components of the display name can be seen:

- TestA %x=4 %l.foo=10 %t.p=2 /8804be5d
    ^MyFixture %p=1 ~TestA_3 't 'f /f027ee75
    ^MyFixture %p=2 ~TestA_3 't 'f /830323a4
    ^X %foo=10 ~generic:default+builtin 'l /7dae3cc5
- TestA %x=3 %l.foo=10 %t.p=2 /89f6f5d1
    ^MyFixture %p=1 ~TestA_2 't 'f /02368516
    ^MyFixture %p=2 ~TestA_2 't 'f /854b99b5
    ^X %foo=10 ~generic:default+builtin 'l /7dae3cc5
- TestA %x=4 %l.foo=10 %t.p=1 /af9b2941
    ^MyFixture %p=2 ~TestA_1 't 'f /f0383f7f
    ^MyFixture %p=1 ~TestA_1 't 'f /d07f4281
    ^X %foo=10 ~generic:default+builtin 'l /7dae3cc5
- TestA %x=3 %l.foo=10 %t.p=1 /a9e50aa3
    ^MyFixture %p=2 ~TestA_0 't 'f /b894ab05
    ^MyFixture %p=1 ~TestA_0 't 'f /ca376ca8
    ^X %foo=10 ~generic:default+builtin 'l /7dae3cc5
Found 4 check(s)

Notice that the variable name to which every fixture is bound in its parent test is also listed as '<varname>. This is useful for setting variables down the fixture hierarchy using the -S option.

Display names may not always be unique. Assume the following test:

class MyTest(RegressionTest):
    p = parameter([1, 1, 1])

This generates three different tests with different unique names, but their display name is the same for all: MyTest %p=1. Notice that this example leads to a name conflict with the old naming scheme, since all tests would be named MyTest_1.

Each test is also associated with a hash code that is derived from the test name, its parameters and their values. As in the example listing above, the hash code of each test is printed with the -l option and individual tests can be selected by their hash using the -n option, e.g., -n /1c51609b. The stage and output directories, as well as the performance log file of the filelog performance log handler will use the hash code for the test-specific directories and files. This might lead to conflicts for tests as the one above when executing them with the asynchronous execution policy, but ensures consistency of performance record files when parameter values are added to or deleted from a test parameter. More specifically, the test’s hash will not change if a new parameter value is added or deleted or even if the parameter values are shuffled. Test variants on the other side are more volatile and can change with such changes. Also users should not rely on how the variant numbers are assigned to a test, as this is an implementation detail.

Changed in version 4.0.0: A hash code is associated with each test.

Differences from the old naming scheme¶

Prior to version 3.10, ReFrame used to encode the parameter values of an instance of parameterized test in its name. It did so by taking the string representation of the value and replacing any non-alphanumeric character with an underscore. This could lead to very large and hard to read names when a test defined multiple parameters or the parameter type was more complex. Very large test names meant also very large path names which could also lead to problems and random failures. Fixtures followed a similar naming pattern making them hard to debug.

Environment¶

Several aspects of ReFrame can be controlled through environment variables. Usually environment variables have counterparts in command line options or configuration parameters. In such cases, command-line options take precedence over environment variables, which in turn precede configuration parameters. Boolean environment variables can have any value of true, yes, y (case insensitive) or 1 to denote true and any value of false, no, n (case insensitive) or 0 to denote false.

Changed in version 3.9.2: Values 1 and 0 are now valid for boolean environment variables.

Here is an alphabetical list of the environment variables recognized by ReFrame. Whenever an environment variable is associated with a configuration option, its default value is omitted as it is the same.

RFM_AUTODETECT_FQDN¶

Use the fully qualified domain name as the hostname. This is a boolean variable and defaults to 0.

Associated command line option	N/A
Associated configuration parameter	N/A

New in version 3.11.0.

Changed in version 4.0.0: This variable now defaults to 0.

Deprecated since version 4.3: Please use RFM_AUTODETECT_METHODS=py::fqdn in the future.

RFM_AUTODETECT_METHOD¶

Method to use for detecting the current system and pick the right configuration. The following values can be used:

hostname: The hostname command will be used to detect the current system. This is the default value, if not specified.

Associated command line option	N/A
Associated configuration parameter	N/A

New in version 3.11.0.

Deprecated since version 4.3: This has no effect. For setting multiple auto-detection methods, please use the RFM_AUTODETECT_METHODS.

RFM_AUTODETECT_METHODS¶: A comma-separated list of system auto-detection methods. Please refer to the autodetect_methods configuration parameter for more information on how to set this variable.

New in version 4.3.

RFM_AUTODETECT_XTHOSTNAME¶

Use /etc/xthostname file, if present, to retrieve the current system’s name. If the file cannot be found, the hostname will be retrieved using the hostname command. This is a boolean variable and defaults to 0.

This option meaningful for Cray systems.

Associated command line option	N/A
Associated configuration parameter	N/A

New in version 3.11.0.

Changed in version 4.0.0: This variable now defaults to 0.

Deprecated since version 4.3: Please use RFM_AUTODETECT_METHODS='cat /etc/xthostname,hostname' in the future.

RFM_CHECK_SEARCH_PATH¶

A colon-separated list of filesystem paths where ReFrame should search for tests.

Associated command line option	`-c`
Associated configuration parameter	`check_search_path`

RFM_CHECK_SEARCH_RECURSIVE¶

Search for test files recursively in directories found in the check search path.

Associated command line option	`-R`
Associated configuration parameter	`check_search_recursive`

RFM_CLEAN_STAGEDIR¶

Clean stage directory of tests before populating it.

New in version 3.1.

Associated command line option	`--dont-restage`
Associated configuration parameter	`clean_stagedir`

RFM_COLORIZE¶

Enable output coloring.

Associated command line option	`--nocolor`
Associated configuration parameter	`colorize`

RFM_COMPRESS_REPORT¶

Compress the generated run report file.

Associated command line option	`--compress-report`
Associated configuration parameter	`compress_report`

New in version 3.12.0.

RFM_CONFIG_FILE¶

Set the configuration file for ReFrame.

Associated command line option	`-C`
Associated configuration parameter	N/A

Deprecated since version 4.0.0: Please use the RFM_CONFIG_FILES instead.

RFM_CONFIG_FILES¶

A colon-separated list of configuration files to load. Refer to the documentation of the --config-file option for a detailed description on how ReFrame loads its configuration.

Associated command line option	`-C`
Associated configuration parameter	N/A

New in version 4.0.0.

RFM_CONFIG_PATH¶

A colon-separated list of directories that contain ReFrame configuration files. Refer to the documentation of the --config-file option for a detailed description on how ReFrame loads its configuration.

Associated command line option	N/A
Associated configuration parameter	N/A

New in version 4.0.0.

RFM_GIT_TIMEOUT¶

Timeout value in seconds used when checking if a git repository exists.

Associated command line option	N/A
Associated configuration parameter	`git_timeout`

New in version 3.9.0.

RFM_GRAYLOG_ADDRESS¶

The address of the Graylog server to send performance logs. The address is specified in host:port format.

Associated command line option	N/A
Associated configuration parameter	`address`

New in version 3.1.

RFM_HTTPJSON_URL¶

The URL of the server to send performance logs in JSON format. The URL is specified in scheme://host:port/path format.

Associated command line option	N/A
Associated configuration parameter	`url`

New in version 3.6.1.

RFM_IGNORE_REQNODENOTAVAIL¶

Do not treat specially jobs in pending state with the reason ReqNodeNotAvail (Slurm only).

Associated command line option	N/A
Associated configuration parameter	`ignore_reqnodenotavail`

RFM_KEEP_STAGE_FILES¶

Keep test stage directories even for tests that finish successfully.

Associated command line option	`--keep-stage-files`
Associated configuration parameter	`keep_stage_files`

RFM_MODULE_MAP_FILE¶

A file containing module mappings.

Associated command line option	`--module-mappings`
Associated configuration parameter	`module_map_file`

RFM_MODULE_MAPPINGS¶

A comma-separated list of module mappings.

Associated command line option	`-M`
Associated configuration parameter	`module_mappings`

RFM_NON_DEFAULT_CRAYPE¶

Test a non-default Cray Programming Environment.

Associated command line option	`--non-default-craype`
Associated configuration parameter	`non_default_craype`

RFM_OUTPUT_DIR¶

Directory prefix for test output files.

Associated command line option	`-o`
Associated configuration parameter	`outputdir`

RFM_PERF_INFO_LEVEL¶

Logging level at which the immediate performance information is logged.

Associated command line option	n/a
Associated configuration parameter	`perf_info_level`

RFM_PERFLOG_DIR¶

Directory prefix for logging performance data.

Associated command line option	`--perflogdir`
Associated configuration parameter	`basedir`

RFM_PIPELINE_TIMEOUT¶

Timeout in seconds for advancing the pipeline in the asynchronous execution policy. See Tweaking the throughput and interactivity of test jobs in the asynchronous execution policy for more guidance on how to set this.

Associated command line option	N/A
Associated configuration parameter	`pipeline_timeout`

New in version 3.10.0.

RFM_PREFIX¶

General directory prefix for ReFrame-generated directories.

Associated command line option	`--prefix`
Associated configuration parameter	`prefix`

RFM_PURGE_ENVIRONMENT¶

Unload all environment modules before acting on any tests.

Associated command line option	`--purge-env`
Associated configuration parameter	`purge_environment`

RFM_REMOTE_DETECT¶

Auto-detect processor information of remote partitions as well.

Associated command line option	N/A
Associated configuration parameter	`remote_detect`

New in version 3.7.0.

RFM_REMOTE_WORKDIR¶

The temporary directory prefix that will be used to create a fresh ReFrame clone, in order to auto-detect the processor information of a remote partition.

Associated command line option	N/A
Associated configuration parameter	`remote_workdir`

New in version 3.7.0.

RFM_REPORT_FILE¶

The file where ReFrame will store its report.

New in version 3.1.

Associated command line option	`--report-file`
Associated configuration parameter	`report_file`

RFM_REPORT_JUNIT¶

The file where ReFrame will generate a JUnit XML report.

New in version 3.6.0.

Associated command line option	`--report-junit`
Associated configuration parameter	`report_junit`

RFM_RESOLVE_MODULE_CONFLICTS¶

Resolve module conflicts automatically.

New in version 3.6.0.

Associated command line option	N/A
Associated configuration parameter	`resolve_module_conflicts`

RFM_SAVE_LOG_FILES¶

Save ReFrame log files in the output directory before exiting.

Associated command line option	`--save-log-files`
Associated configuration parameter	`save_log_files`

RFM_STAGE_DIR¶

Directory prefix for staging test resources.

Associated command line option	`-s`
Associated configuration parameter	`stagedir`

RFM_SYSLOG_ADDRESS¶

The address of the Syslog server to send performance logs. The address is specified in host:port format. If no port is specified, the address refers to a UNIX socket.

Associated command line option	N/A
Associated configuration parameter	`address`

New in version 3.1.

RFM_SYSTEM¶

Set the current system.

Associated command line option	`--system`
Associated configuration parameter	N/A

RFM_TIMESTAMP_DIRS¶

Append a timestamp to the output and stage directory prefixes.

Associated command line option	`--timestamp`
Associated configuration parameter	`timestamp_dirs`

RFM_TRAP_JOB_ERRORS¶

Trap job errors in submitted scripts and fail tests automatically.

Associated configuration parameter

trap_job_errors

New in version 3.9.0.

RFM_UNLOAD_MODULES¶

A comma-separated list of environment modules to be unloaded before acting on any tests.

Associated command line option	`-u`
Associated configuration parameter	`unload_modules`

RFM_USE_LOGIN_SHELL¶

Use a login shell for the generated job scripts.

Associated command line option	N/A
Associated configuration parameter	`use_login_shell`

RFM_USER_MODULES¶

A comma-separated list of environment modules to be loaded before acting on any tests.

Associated command line option	`-m`
Associated configuration parameter	`user_modules`

RFM_VERBOSE¶

Set the verbosity level of output.

Associated command line option	`-v`
Associated configuration parameter	`verbose`

Configuration File¶

The configuration file of ReFrame defines the systems and environments to test as well as parameters controlling its behavior. Upon start up ReFrame checks for configuration files in the following locations in that order:

$HOME/.reframe/settings.{py,json}
$RFM_INSTALL_PREFIX/settings.{py,json}
/etc/reframe.d/settings.{py,json}

ReFrame accepts configuration files either in Python or JSON syntax. If both are found in the same location, the Python file will be preferred.

The RFM_INSTALL_PREFIX environment variable refers to the installation directory of ReFrame. Users have no control over this variable. It is always set by the framework upon startup.

If no configuration file can be found in any of the predefined locations, ReFrame will fall back to a generic configuration that allows it to run on any system. This configuration file is located in reframe/core/settings.py. Users may not modify this file.

For a complete reference of the configuration, please refer to reframe.settings(8) man page.

Reporting Bugs¶

For bugs, feature request, help, please open an issue on Github: <https://github.com/reframe-hpc/reframe>

Command Line Reference¶

Synopsis¶

Description¶

Test discovery and test loading¶

Test filtering¶

Test actions¶

Options controlling ReFrame output¶

Options controlling ReFrame execution¶

Options controlling job submission¶

Flexible node allocation¶

Options controlling ReFrame environment¶

Options for generating tests dynamically¶

Miscellaneous options¶

Test Naming Scheme¶

Differences from the old naming scheme¶

Environment¶

Configuration File¶

Reporting Bugs¶

See Also¶