Command Line Reference

Synopsis

reframe [OPTION]... ACTION

Description

ReFrame provides both a programming interface for writing regression tests and a command-line interface for managing and running the tests, which is detailed here. The reframe command is part of ReFrame’s frontend. This frontend is responsible for loading and running regression tests written in ReFrame. ReFrame executes tests by sending them down to a well defined pipeline. The implementation of the different stages of this pipeline is part of ReFrame’s core architecture, but the frontend is responsible for driving this pipeline and executing tests through it. There are three basic phases that the frontend goes through, which are described briefly in the following.

Test discovery and test loading

This is the very first phase of the frontend. ReFrame will search for tests in its check search path and will load them. When ReFrame loads a test, it actually instantiates it, meaning that it will call its __init__() method unconditionally whether this test is meant to run on the selected system or not. This is something that writers of regression tests should bear in mind.

-c, --checkpath=PATH

A filesystem path where ReFrame should search for tests.

PATH can be a directory or a single test file. If it is a directory, ReFrame will search for test files inside this directory load all tests found in them. This option can be specified multiple times, in which case each PATH will be searched in order.

The check search path can also be set using the RFM_CHECK_SEARCH_PATH environment variable or the check_search_path general configuration parameter.

--ignore-check-conflicts

Ignore tests with conflicting names when loading.

ReFrame requires test names to be unique. Test names are used as components of the stage and output directory prefixes of tests, as well as for referencing target test dependencies. This option should generally be avoided unless there is a specific reason.

This option can also be set using the RFM_IGNORE_CHECK_CONFLICTS environment variable or the ignore_check_conflicts general configuration parameter.

Deprecated since version 3.8.0: This option will be removed in a future version.

-R, --recursive

Search for test files recursively in directories found in the check search path.

This option can also be set using the RFM_CHECK_SEARCH_RECURSIVE environment variable or the check_search_recursive general configuration parameter.

Test filtering

After all tests in the search path have been loaded, they are first filtered by the selected system. Any test that is not valid for the current system, it will be filtered out. The current system is either auto-selected or explicitly specified with the --system option. Tests can be filtered by different attributes and there are specific command line options for achieving this. A common characteristic of all test filtering options is that if a test is selected, then all its dependencies will be selected, too, regardless if they match the filtering criteria or not. This happens recursively so that if test T1 depends on T2 and T2 depends on T3, then selecting T1 would also select T2 and T3.

--cpu-only

Select tests that do not target GPUs.

These are all tests with num_gpus_per_node equals to zero This option and --gpu-only are mutually exclusive.

The --gpu-only and --cpu-only check only the value of the num_gpus_per_node attribute of tests. The value of this attribute is not required to be non-zero for GPU tests. Tests may or may not make use of it.

--failed

Select only the failed test cases for a previous run.

This option can only be used in combination with the --restore-session. To rerun the failed cases from the last run, you can use reframe --restore-session --failed -r.

New in version 3.4.

--gpu-only

Select tests that can run on GPUs.

These are all tests with num_gpus_per_node greater than zero. This option and --cpu-only are mutually exclusive.

--maintainer=MAINTAINER

Filter tests by maintainer.

MAINTAINER is interpreted as a Python Regular Expression; all tests that have at least a matching maintainer will be selected. MAINTAINER being a regular expression has the implication that --maintainer 'foo' will select also tests that define 'foobar' as a maintainer. To restrict the selection to tests defining only 'foo', you should use --maintainer 'foo$'.

This option may be specified multiple times, in which case only tests defining or matching all maintainers will be selected.

New in version 3.9.1.

-n, --name=NAME

Filter tests by name.

NAME is interpreted as a Python Regular Expression; any test whose display name matches NAME will be selected. The display name of a test encodes also any parameterization information. See Test Naming Scheme for more details on how the tests are automatically named by the framework.

Before matching, any whitespace will be removed from the display name of the test.

This option may be specified multiple times, in which case tests with any of the specified names will be selected: -n NAME1 -n NAME2 is therefore equivalent to -n 'NAME1|NAME2'.

If the special notation <test_name>@<variant_num> is passed as the NAME argument, then an exact match will be performed selecting the variant variant_num of the test test_name.

You may also select a test by its hash code using the notation /<test-hash> for the NAME argument.

Note

Fixtures cannot be selected.

Changed in version 3.10.0: The option’s behaviour was adapted and extended in order to work with the updated test naming scheme.

Changed in version 4.0.0: Support selecting tests by their hash code.

-p, --prgenv=NAME

Filter tests by programming environment.

NAME is interpreted as a Python Regular Expression; any test for which at least one valid programming environment is matching NAME will be selected.

This option may be specified multiple times, in which case only tests matching all of the specified programming environments will be selected.

--skip-prgenv-check

Do not filter tests against programming environments.

Even if the -p option is not specified, ReFrame will filter tests based on the programming environments defined for the currently selected system. This option disables that filter completely.

--skip-system-check

Do not filter tests against the selected system.

-T, --exclude-tag=TAG

Exclude tests by tags.

TAG is interpreted as a Python Regular Expression; any test with tags matching TAG will be excluded.

This option may be specified multiple times, in which case tests with any of the specified tags will be excluded: -T TAG1 -T TAG2 is therefore equivalent to -T 'TAG1|TAG2'.

-t, --tag=TAG

Filter tests by tag.

TAG is interpreted as a Python Regular Expression; all tests that have at least a matching tag will be selected. TAG being a regular expression has the implication that -t 'foo' will select also tests that define 'foobar' as a tag. To restrict the selection to tests defining only 'foo', you should use -t 'foo$'.

This option may be specified multiple times, in which case only tests defining or matching all tags will be selected.

-x, --exclude=NAME

Exclude tests by name.

NAME is interpreted as a Python Regular Expression; any test whose name matches NAME will be excluded.

This option may be specified multiple times, in which case tests with any of the specified names will be excluded: -x NAME1 -x NAME2 is therefore equivalent to -x 'NAME1|NAME2'.

Test actions

ReFrame will finally act upon the selected tests. There are currently two actions that can be performed on tests: (a) list the tests and (b) execute the tests. An action must always be specified.

--ci-generate=FILE

Do not run the tests, but generate a Gitlab child pipeline specification in FILE.

You can set up your Gitlab CI to use the generated file to run every test as a separate Gitlab job respecting test dependencies. For more information, have a look in Integrating into a CI pipeline.

New in version 3.4.1.

--describe

Print a detailed description of the selected tests in JSON format and exit.

Note

The generated test description corresponds to its state after it has been initialized. If any of its attributes are changed or set during its execution, their updated values will not be shown by this listing.

New in version 3.10.0.

-L, --list-detailed[=T|C]

List selected tests providing more details for each test.

The unique id of each test (see also unique_name) as well as the file where each test is defined are printed.

This option accepts optionally a single argument denoting what type of listing is requested. Please refer to -l for an explanation of this argument.

New in version 3.10.0: Support for different types of listing is added.

-l, --list[=T|C]

List selected tests and their dependencies.

This option accepts optionally a single argument denoting what type of listing is requested. There are two types of possible listings:

  • Regular test listing (T, the default): This type of listing lists the tests and their dependencies or fixtures using their display_name. A test that is listed as a dependency of another test will not be listed separately.

  • Concretized test case listing (C): This type of listing lists the exact test cases and their dependencies as they have been concretized for the current system and environment combinations. This listing shows practically the exact test DAG that will be executed.

New in version 3.10.0: Support for different types of listing is added.

--list-tags

List the unique tags of the selected tests.

The tags are printed in alphabetical order.

New in version 3.6.0.

-r, --run

Execute the selected tests.

If more than one action options are specified, the precedence order is the following:

--describe > --list-detailed > --list > --list-tags > --ci-generate

Options controlling ReFrame output

--compress-report

Compress the generated run report (see --report-file). The generated report is a JSON file formatted in a human readable form. If this option is enabled, the generated JSON file will be a single stream of text without additional spaces or new lines.

This option can also be set using the RFM_COMPRESS_REPORT environment variable or the compress_report general configuration parameter.

New in version 3.12.0.

--dont-restage

Do not restage a test if its stage directory exists. Normally, if the stage directory of a test exists, ReFrame will remove it and recreate it. This option disables this behavior.

This option can also be set using the RFM_CLEAN_STAGEDIR environment variable or the clean_stagedir general configuration parameter.

New in version 3.1.

--keep-stage-files

Keep test stage directories even for tests that finish successfully.

This option can also be set using the RFM_KEEP_STAGE_FILES environment variable or the keep_stage_files general configuration parameter.

-o, --output=DIR

Directory prefix for test output files.

When a test finishes successfully, ReFrame copies important output files to a test-specific directory for future reference. This test-specific directory is of the form {output_prefix}/{system}/{partition}/{environment}/{test_name}, where output_prefix is set by this option. The test files saved in this directory are the following:

  • The ReFrame-generated build script, if not a run-only test.

  • The standard output and standard error of the build phase, if not a run-only test.

  • The ReFrame-generated job script, if not a compile-only test.

  • The standard output and standard error of the run phase, if not a compile-only test.

  • Any additional files specified by the keep_files regression test attribute.

This option can also be set using the RFM_OUTPUT_DIR environment variable or the outputdir system configuration parameter.

--perflogdir=DIR

Directory prefix for logging performance data.

This option is relevant only to the filelog logging handler.

This option can also be set using the RFM_PERFLOG_DIR environment variable or the basedir logging handler configuration parameter.

--prefix=DIR

General directory prefix for ReFrame-generated directories.

The base stage and output directories (see below) will be specified relative to this prefix if not specified explicitly.

This option can also be set using the RFM_PREFIX environment variable or the prefix system configuration parameter.

--report-file=FILE

The file where ReFrame will store its report.

The FILE argument may contain the special placeholder {sessionid}, in which case ReFrame will generate a new report each time it is run by appending a counter to the report file.

This option can also be set using the RFM_REPORT_FILE environment variable or the report_file general configuration parameter.

New in version 3.1.

--report-junit=FILE

Instruct ReFrame to generate a JUnit XML report in FILE.

The generated report adheres to the XSD schema here where each retry is treated as an individual testsuite.

This option can also be set using the RFM_REPORT_JUNIT environment variable or the report_junit general configuration parameter.

New in version 3.6.0.

Changed in version 3.6.1: Added support for retries in the JUnit XML report.

-s, --stage=DIR

Directory prefix for staging test resources.

ReFrame does not execute tests from their original source directory. Instead it creates a test-specific stage directory and copies all test resources there. It then changes to that directory and executes the test. This test-specific directory is of the form {stage_prefix}/{system}/{partition}/{environment}/{test_name}, where stage_prefix is set by this option. If a test finishes successfully, its stage directory will be removed.

This option can also be set using the RFM_STAGE_DIR environment variable or the stagedir system configuration parameter.

--save-log-files

Save ReFrame log files in the output directory before exiting.

Only log files generated by file log handlers will be copied.

This option can also be set using the RFM_SAVE_LOG_FILES environment variable or the save_log_files general configuration parameter.

--timestamp [TIMEFMT]

Append a timestamp to the output and stage directory prefixes.

TIMEFMT can be any valid strftime(3) time format. If not specified, TIMEFMT is set to %FT%T.

This option can also be set using the RFM_TIMESTAMP_DIRS environment variable or the timestamp_dirs general configuration parameter.

Options controlling ReFrame execution

--disable-hook=HOOK

Disable the pipeline hook named HOOK from all the tests that will run.

This feature is useful when you have implemented test workarounds as pipeline hooks, in which case you can quickly disable them from the command line. This option may be specified multiple times in order to disable multiple hooks at the same time.

New in version 3.2.

--distribute[=NODESTATE]

Distribute the selected tests on all the nodes in state NODESTATE in their respective valid partitions.

ReFrame will parameterize and run the tests on the selected nodes. Effectively, it will dynamically create new tests that inherit from the original tests and add a new parameter named $nid which contains the list of nodes that the test must run on. The new tests are named with the following pattern {orig_test_basename}_{partition_fullname}.

When determining the list of nodes to distribute the selected tests, ReFrame will take into account any job options passed through the -J option.

You can optionally specify the state of the nodes to consider when distributing the test through the NODESTATE argument:

  • all: Tests will run on all the nodes of their respective valid partitions regardless of the nodes’ state.

  • idle: Tests will run on all idle nodes of their respective valid partitions.

  • NODESTATE: Tests will run on all the nodes in state NODESTATE of their respective valid partitions. If NODESTATE is not specified, idle will be assumed.

The state of the nodes will be determined once, before beginning the execution of the tests, so it might be different at the time the tests are actually submitted.

Note

Currently, only single-node jobs can be distributed and only local or the Slurm-based backends support this feature.

Note

Distributing tests with dependencies is not supported, but you can distribute tests that use fixtures.

New in version 3.11.0.

--exec-order=ORDER

Impose an execution order for the independent tests. The ORDER argument can take one of the following values:

  • name: Order tests by their display name.

  • rname: Order tests by their display name in reverse order.

  • uid: Order tests by their unique name.

  • ruid: Order tests by their unique name in reverse order.

  • random: Randomize the order of execution.

If this option is not specified the order of execution of independent tests is implementation defined. This option can be combined with any of the listing options (-l or -L) to list the tests in the order.

New in version 4.0.0.

--exec-policy=POLICY

The execution policy to be used for running tests.

There are two policies defined:

  • serial: Tests will be executed sequentially.

  • async: Tests will be executed asynchronously. This is the default policy.

    The async execution policy executes the build and run phases of tests asynchronously by submitting their associated jobs in a non-blocking way. ReFrame’s runtime monitors the progress of each test and will resume the pipeline execution of an asynchronously spawned test as soon as its build or run phase have finished. Note that the rest of the pipeline stages are still executed sequentially in this policy.

    Concurrency can be controlled by setting the max_jobs system partition configuration parameter. As soon as the concurrency limit is reached, ReFrame will first poll the status of all its pending tests to check if any execution slots have been freed up. If there are tests that have finished their build or run phase, ReFrame will keep pushing tests for execution until the concurrency limit is reached again. If no execution slots are available, ReFrame will throttle job submission.

--force-local

Force local execution of tests.

Execute tests as if all partitions of the currently selected system had a local scheduler.

--max-retries=NUM

The maximum number of times a failing test can be retried.

The test stage and output directories will receive a _retry<N> suffix every time the test is retried.

--maxfail=NUM

The maximum number of failing test cases before the execution is aborted.

After NUM failed test cases the rest of the test cases will be aborted. The counter of the failed test cases is reset to 0 in every retry.

--mode=MODE

ReFrame execution mode to use.

An execution mode is simply a predefined invocation of ReFrame that is set with the modes configuration parameter. If an option is specified both in an execution mode and in the command-line, then command-line takes precedence.

--repeat=N

Repeat the selected tests N times. This option can be used in conjunction with the --distribute option in which case the selected tests will be repeated multiple times and distributed on individual nodes of the system’s partitions.

Note

Repeating tests with dependencies is not supported, but you can repeat tests that use fixtures.

New in version 3.12.0.

--restore-session [REPORT1[,REPORT2,...]]

Restore a testing session that has run previously.

REPORT1 etc. are a run report files generated by ReFrame. If a report is not given, ReFrame will pick the last report file found in the default location of report files (see the --report-file option). If passed alone, this option will simply rerun all the test cases that have run previously based on the report file data. It is more useful to combine this option with any of the test filtering options, in which case only the selected test cases will be executed. The difference in test selection process when using this option is that the dependencies of the selected tests will not be selected for execution, as they would normally, but they will be restored. For example, if test T1 depends on T2 and T2 depends on T3, then running reframe -n T1 -r would cause both T2 and T3 to run. However, by doing reframe -n T1 --restore-session -r, only T1 would run and its immediate dependence T2 will be restored. This is useful when you have deep test dependencies or some of the tests in the dependency chain are very time consuming.

Multiple reports may be passed as a comma-separated list. ReFrame will try to restore any required test case by looking it up in each report sequentially. If it cannot find it, it will issue an error and exit.

Note

In order for a test case to be restored, its stage directory must be present. This is not a problem when rerunning a failed case, since the stage directories of its dependencies are automatically kept, but if you want to rerun a successful test case, you should make sure to have run with the --keep-stage-files option.

New in version 3.4.

Changed in version 3.6.1: Multiple report files are now accepted.

-S, --setvar=[TEST.]VAR=VAL

Set variable VAR in all tests or optionally only in test TEST to VAL.

TEST can have the form [TEST.][FIXT.]*, in which case VAR will be set in fixture FIXT of TEST. Note that this syntax is recursive on fixtures, so that a variable can be set in a fixture arbitrarily deep. TEST prefix refers to the test class name, not the test name, but FIXT refers to the fixture name inside the referenced test.

Multiple variables can be set at the same time by passing this option multiple times. This option cannot change arbitrary test attributes, but only test variables declared with the variable built-in. If an attempt is made to change an inexistent variable or a test parameter, a warning will be issued.

ReFrame will try to convert VAL to the type of the variable. If it does not succeed, a warning will be issued and the variable will not be set. VAL can take the special value @none to denote that the variable must be set to None. Boolean variables can be set in one of the following ways:

  • By passing true, yes or 1 to set them to True.

  • By passing false, no or 0 to set them to False.

Passing any other value will issue an error.

Note

Boolean variables in a test must be declared of type Bool and not of the built-in bool type, in order to adhere to the aforementioned behaviour. If a variable is defined as bool there is no way you can set it to False, since all strings in Python evaluate to True.

Sequence and mapping types can also be set from the command line by using the following syntax:

  • Sequence types: -S seqvar=1,2,3,4

  • Mapping types: -S mapvar=a:1,b:2,c:3

Conversions to arbitrary objects are also supported. See ConvertibleType for more details.

Variable assignments passed from the command line happen before the test is instantiated and is the exact equivalent of assigning a new value to the variable at the end of the test class body. This has a number of implications that users of this feature should be aware of:

  • In the following test, num_tasks will have always the value 1 regardless of any command-line assignment of the variable foo:

@rfm.simple_test
class my_test(rfm.RegressionTest):
    foo = variable(int, value=1)
    num_tasks = foo

Tip

In cases where the class body expresses logic as a function of a variable and this variable, as well as its dependent logic, need to be controlled externally, the variable’s default value (i.e. the value set through the value argument) may be modified as follows through an environment variable and not through the -S option:

import os

@rfm.simple_test
class my_test(rfm.RegressionTest):
    max_nodes = variable(int, value=int(os.getenv('MAX_NODES', 1)))
    # Parameterise number of nodes
    num_nodes = parameter((1 << i for i in range(0, int(max_nodes))))
  • If the variable is set in any pipeline hook, the command line assignment will have an effect until the variable assignment in the pipeline hook is reached. The variable will be then overwritten.

  • The test filtering happens after a test is instantiated, so the only way to scope a variable assignment is to prefix it with the test class name. However, this has some positive side effects:

    • Passing -S valid_systems='*' and -S valid_prog_environs='*' is the equivalent of passing the --skip-system-check and --skip-prgenv-check options.

    • Users could alter the behavior of tests based on tag values that they pass from the command line, by changing the behavior of a test in a post-init hook based on the value of the tags attribute.

    • Users could force a test with required variables to run if they set these variables from the command line. For example, the following test could only be run if invoked with -S num_tasks=<NUM>:

    @rfm.simple_test
    class my_test(rfm.RegressionTest):
        num_tasks = required
    

New in version 3.8.0.

Changed in version 3.9.3: Proper handling of boolean variables.

Changed in version 3.11.1: Allow setting variables in fixtures.

--skip-performance-check

Skip performance checking phase.

The phase is completely skipped, meaning that performance data will not be logged.

--skip-sanity-check

Skip sanity checking phase.

--strict

Enforce strict performance checking, even if a performance test is marked as not performance critical by having set its strict_check attribute to False.

Options controlling job submission

-J, --job-option=OPTION

Pass OPTION directly to the job scheduler backend.

The syntax of OPTION is -J key=value. If OPTION starts with - it will be passed verbatim to the backend job scheduler. If OPTION starts with # it will be emitted verbatim in the job script. Otherwise, ReFrame will pass --key value or -k value (if key is a single character) to the backend scheduler. Any job options specified with this command-line option will be emitted after any job options specified in the access system partition configuration parameter.

Especially for the Slurm backends, constraint options, such as -J constraint=value, -J C=value, -J --constraint=value or -J -C=value, are going to be combined with any constraint options specified in the access system partition configuration parameter. For example, if -C x is specified in the access and -J C=y is passed to the command-line, ReFrame will pass -C x&y as a constraint to the scheduler. Notice, however, that if constraint options are specified through multiple -J options, only the last one will be considered. If you wish to completely overwrite any constraint options passed in access, you should consider passing explicitly the Slurm directive with -J '#SBATCH --constraint=new'.

Changed in version 3.0: This option has become more flexible.

Changed in version 3.1: Use & to combine constraints.

Flexible node allocation

ReFrame can automatically set the number of tasks of a test, if its num_tasks attribute is set to a value less than or equal to zero. This scheme is conveniently called flexible node allocation and is valid only for the Slurm backend. When allocating nodes automatically, ReFrame will take into account all node limiting factors, such as partition access options, and any job submission control options described above. Nodes from this pool are allocated according to different policies. If no node can be selected, the test will be marked as a failure with an appropriate message.

--flex-alloc-nodes=POLICY

Set the flexible node allocation policy.

Available values are the following:

  • all: Flexible tests will be assigned as many tasks as needed in order to span over all the nodes of the node pool.

  • STATE: Flexible tests will be assigned as many tasks as needed in order to span over the nodes that are currently in state STATE. Querying of the node state and submission of the test job are two separate steps not executed atomically. It is therefore possible that the number of tasks assigned does not correspond to the actual nodes in the given state.

    If this option is not specified, the default allocation policy for flexible tests is ‘idle’.

  • Any positive integer: Flexible tests will be assigned as many tasks as needed in order to span over the specified number of nodes from the node pool.

Changed in version 3.1: It is now possible to pass an arbitrary node state as a flexible node allocation parameter.

Options controlling ReFrame environment

ReFrame offers the ability to dynamically change its environment as well as the environment of tests. It does so by leveraging the selected system’s environment modules system.

-M, --map-module=MAPPING

Apply a module mapping.

ReFrame allows manipulating test modules on-the-fly using module mappings. A module mapping has the form old_module: module1 [module2]... and will cause ReFrame to replace a module with another list of modules upon load time. For example, the mapping foo: foo/1.2 will load module foo/1.2 whenever module foo needs to be loaded. A mapping may also be self-referring, e.g., gnu: gnu gcc/10.1, however cyclic dependencies in module mappings are not allowed and ReFrame will issue an error if it detects one. This option is especially useful for running tests using a newer version of a software or library.

This option may be specified multiple times, in which case multiple mappings will be applied.

This option can also be set using the RFM_MODULE_MAPPINGS environment variable or the module_mappings general configuration parameter.

Changed in version 3.3: If the mapping replaces a module collection, all new names must refer to module collections, too.

See also

Module collections with Environment Modules and Lmod.

-m, --module=NAME

Load environment module NAME before acting on any tests.

This option may be specified multiple times, in which case all specified modules will be loaded in order. ReFrame will not perform any automatic conflict resolution.

This option can also be set using the RFM_USER_MODULES environment variable or the user_modules general configuration parameter.

--module-mappings=FILE

A file containing module mappings.

Each line of the file contains a module mapping in the form described in the -M option. This option may be combined with the -M option, in which case module mappings specified will be applied additionally.

This option can also be set using the RFM_MODULE_MAP_FILE environment variable or the module_map_file general configuration parameter.

--module-path=PATH

Manipulate the MODULEPATH environment variable before acting on any tests.

If PATH starts with the - character, it will be removed from the MODULEPATH, whereas if it starts with the + character, it will be added to it. In all other cases, PATH will completely override MODULEPATH. This option may be specified multiple times, in which case all the paths specified will be added or removed in order.

New in version 3.3.

--non-default-craype

Test a non-default Cray Programming Environment.

Since CDT 19.11, this option can be used in conjunction with -m, which will load the target CDT. For example:

reframe -m cdt/20.03 --non-default-craype -r

This option causes ReFrame to properly set the LD_LIBRARY_PATH for such cases. It will emit the following code after all the environment modules of a test have been loaded:

export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH

This option can also be set using the RFM_NON_DEFAULT_CRAYPE environment variable or the non_default_craype general configuration parameter.

--purge-env

Unload all environment modules before acting on any tests.

This will unload also sticky Lmod modules.

This option can also be set using the RFM_PURGE_ENVIRONMENT environment variable or the purge_environment general configuration parameter.

-u, --unload-module=NAME

Unload environment module NAME before acting on any tests.

This option may be specified multiple times, in which case all specified modules will be unloaded in order.

This option can also be set using the RFM_UNLOAD_MODULES environment variable or the unload_modules general configuration parameter.

Miscellaneous options

-C --config-file=FILE

Use FILE as configuration file for ReFrame.

This option can also be set using the RFM_CONFIG_FILE environment variable.

--detect-host-topology[=FILE]

Detect the local host processor topology, store it to FILE and exit.

If no FILE is specified, the standard output will be used.

New in version 3.7.0.

--failure-stats

Print failure statistics at the end of the run.

-h, --help

Print a short help message and exit.

--nocolor

Disable output coloring.

This option can also be set using the RFM_COLORIZE environment variable or the colorize general configuration parameter.

--performance-report

Print a performance report for all the performance tests that have been run.

The report shows the performance values retrieved for the different performance variables defined in the tests.

-q, --quiet

Decrease the verbosity level.

This option can be specified multiple times. Every time this option is specified, the verbosity level will be decreased by one. This option can be combined arbitrarily with the -v option, in which case the final verbosity level will be determined by the final combination. For example, specifying -qv will not change the verbosity level, since the two options cancel each other, but -qqv is equivalent to -q. For a list of ReFrame’s verbosity levels, see the description of the -v option.

New in version 3.9.3.

--show-config [PARAM]

Show the value of configuration parameter PARAM as this is defined for the currently selected system and exit.

The parameter value is printed in JSON format. If PARAM is not specified or if it set to all, the whole configuration for the currently selected system will be shown. Configuration parameters are formatted as a path navigating from the top-level configuration object to the actual parameter. The / character acts as a selector of configuration object properties or an index in array objects. The @ character acts as a selector by name for configuration objects that have a name property. Here are some example queries:

  • Retrieve all the partitions of the current system:

    reframe --show-config=systems/0/partitions
    
  • Retrieve the job scheduler of the partition named default:

    reframe --show-config=systems/0/partitions/@default/scheduler
    
  • Retrieve the check search path for system foo:

    reframe --system=foo --show-config=general/0/check_search_path
    
--system=NAME

Load the configuration for system NAME.

The NAME must be a valid system name in the configuration file. It may also have the form SYSNAME:PARTNAME, in which case the configuration of system SYSNAME will be loaded, but as if it had PARTNAME as its sole partition. Of course, PARTNAME must be a valid partition of system SYSNAME. If this option is not specified, ReFrame will try to pick the correct configuration entry automatically. It does so by trying to match the hostname of the current machine again the hostname patterns defined in the hostnames system configuration parameter. The system with the first match becomes the current system.

This option can also be set using the RFM_SYSTEM environment variable.

--upgrade-config-file=OLD[:NEW]

Convert the old-style configuration file OLD, place it into the new file NEW and exit.

If a new file is not given, a file in the system temporary directory will be created.

-V, --version

Print version and exit.

-v, --verbose

Increase verbosity level of output.

This option can be specified multiple times. Every time this option is specified, the verbosity level will be increased by one. There are the following message levels in ReFrame listed in increasing verbosity order: critical, error, warning, info, verbose and debug. The base verbosity level of the output is defined by the level stream logging handler configuration parameter.

This option can also be set using the RFM_VERBOSE environment variable or the verbose general configuration parameter.

Test Naming Scheme

New in version 3.10.0.

This section describes the test naming scheme. This scheme has superseded the old one in ReFrame 4.0.

Each ReFrame test is assigned a unique name, which will be used internally by the framework to reference the test. Any test-specific path component will use that name, too. It is formed as follows for the various types of tests:

  • Regular tests: The unique name is simply the test class name. This implies that you cannot load two tests with the same class name within the same run session even if these tests reside in separate directories.

  • Parameterized tests: The unique name is formed by the test class name followed by an _ and the variant number of the test. Each point in the parameter space of the test is assigned a unique variant number.

  • Fixtures: The unique name is formed by the test class name followed by an _ and a hash. The hash is constructed by combining the information of the fixture variant (if the fixture is parameterized), the fixture’s scope and any fixture variables that were explicitly set.

Since unique names can be cryptic, they are not listed by the -l option, but are listed when a detailed listing is requested by using the -L option.

A human readable version of the test name, which is called the display name, is also constructed for each test. This name encodes all the parameterization information as well as the fixture-specific information (scopes, variables). The format of the display name is the following in BNF notation:

<display_name> ::= <test_class_name> (<params>)* (<scope>)?
<params> ::= "%" <parametrization> "=" <pvalue>
<parametrization> ::= (<fname> ".")* <pname>
<scope> ::= "~" <scope_descr>
<scope_descr> ::= <first> ("+" <second>)*

<test_class_name> ::= (* as in Python *)
<fname> ::= (* string *)
<pname> ::= (* string *)
<pvalue> ::= (* string *)
<first> ::= (* string *)
<second> ::= (* string *)

The following is an example of a fictitious complex test that is itself parameterized and depends on parameterized fixtures as well.

import reframe as rfm


class MyFixture(rfm.RunOnlyRegressionTest):
    p = parameter([1, 2])


class X(rfm.RunOnlyRegressionTest):
    foo = variable(int, value=1)


@rfm.simple_test
class TestA(rfm.RunOnlyRegressionTest):
    f = fixture(MyFixture, scope='test', action='join')
    x = parameter([3, 4])
    t = fixture(MyFixture, scope='test')
    l = fixture(X, scope='environment', variables={'foo': 10})
    valid_systems = ['*']
    valid_prog_environs = ['*']

Here is how this test is listed where the various components of the display name can be seen:

- TestA %x=4 %l.foo=10 %t.p=2 /1c51609b
    ^Myfixture %p=1 ~TestA_3 /f027ee75
    ^MyFixture %p=2 ~TestA_3 /830323a4
    ^X %foo=10 ~generic:default+builtin /7dae3cc5
- TestA %x=3 %l.foo=10 %t.p=2 /707b752c
    ^MyFixture %p=1 ~TestA_2 /02368516
    ^MyFixture %p=2 ~TestA_2 /854b99b5
    ^X %foo=10 ~generic:default+builtin /7dae3cc5
- TestA %x=4 %l.foo=10 %t.p=1 /c65657d5
    ^MyFixture %p=2 ~TestA_1 /f0383f7f
    ^MyFixture %p=1 ~TestA_1 /d07f4281
    ^X %foo=10 ~generic:default+builtin /7dae3cc5
- TestA %x=3 %l.foo=10 %t.p=1 /1b9f44df
    ^MyFixture %p=2 ~TestA_0 /b894ab05
    ^MyFixture %p=1 ~TestA_0 /ca376ca8
    ^X %foo=10 ~generic:default+builtin /7dae3cc5
Found 4 check(s)

Display names may not always be unique. Assume the following test:

class MyTest(RegressionTest):
    p = parameter([1, 1, 1])

This generates three different tests with different unique names, but their display name is the same for all: MyTest %p=1. Notice that this example leads to a name conflict with the old naming scheme, since all tests would be named MyTest_1.

Each test is also associated with a hash code that is derived from the test name, its parameters and their values. As in the example listing above, the hash code of each test is printed with the -l option and individual tests can be selected by their hash using the -n option, e.g., -n /1c51609b. The stage and output directories, as well as the performance log file of the filelog performance log handler will use the hash code for the test-specific directories and files. This might lead to conflicts for tests as the one above when executing them with the asynchronous execution policy, but ensures consistency of performance record files when parameter values are added to or deleted from a test parameter. More specifically, the test’s hash will not change if a new parameter value is added or deleted or even if the parameter values are shuffled. Test variants on the other side are more volatile and can change with such changes. Also users should not rely on how the variant numbers are assigned to a test, as this is an implementation detail.

Changed in version 4.0.0: A hash code is associated with each test.

Differences from the old naming scheme

Prior to version 3.10, ReFrame used to encode the parameter values of an instance of parameterized test in its name. It did so by taking the string representation of the value and replacing any non-alphanumeric character with an underscore. This could lead to very large and hard to read names when a test defined multiple parameters or the parameter type was more complex. Very large test names meant also very large path names which could also lead to problems and random failures. Fixtures followed a similar naming pattern making them hard to debug.

Environment

Several aspects of ReFrame can be controlled through environment variables. Usually environment variables have counterparts in command line options or configuration parameters. In such cases, command-line options take precedence over environment variables, which in turn precede configuration parameters. Boolean environment variables can have any value of true, yes, y (case insensitive) or 1 to denote true and any value of false, no, n (case insensitive) or 0 to denote false.

Changed in version 3.9.2: Values 1 and 0 are now valid for boolean environment variables.

Here is an alphabetical list of the environment variables recognized by ReFrame:

RFM_AUTODETECT_FQDN

Use the fully qualified domain name as the hostname. This is a boolean variable and defaults to 0.

Associated command line option

N/A

Associated configuration parameter

N/A

New in version 3.11.0.

Changed in version 4.0.0: This variable now defaults to 0.

RFM_AUTODETECT_METHOD

Method to use for detecting the current system and pick the right configuration. The following values can be used:

  • hostname: The hostname command will be used to detect the current system. This is the default value, if not specified.

Associated command line option

N/A

Associated configuration parameter

N/A

New in version 3.11.0.

RFM_AUTODETECT_XTHOSTNAME

Use /etc/xthostname file, if present, to retrieve the current system’s name. If the file cannot be found, the hostname will be retrieved using the hostname command. This is a boolean variable and defaults to 0.

This option meaningful for Cray systems.

Associated command line option

N/A

Associated configuration parameter

N/A

New in version 3.11.0.

Changed in version 4.0.0: This variable now defaults to 0.

RFM_CHECK_SEARCH_PATH

A colon-separated list of filesystem paths where ReFrame should search for tests.

Associated command line option

-c

Associated configuration parameter

check_search_path general configuration parameter

RFM_CHECK_SEARCH_RECURSIVE

Search for test files recursively in directories found in the check search path.

Associated command line option

-R

Associated configuration parameter

check_search_recursive general configuration parameter

RFM_CLEAN_STAGEDIR

Clean stage directory of tests before populating it.

New in version 3.1.

Associated command line option

--dont-restage

Associated configuration parameter

clean_stagedir general configuration parameter

RFM_COLORIZE

Enable output coloring.

Associated command line option

--nocolor

Associated configuration parameter

colorize general configuration parameter

RFM_COMPRESS_REPORT

Compress the generated run report file.

Associated command line option

--compress-report

Associated configuration parameter

compress_report general configuration parameter

New in version 3.12.0.

RFM_CONFIG_FILE

Set the configuration file for ReFrame.

Associated command line option

-C

Associated configuration parameter

N/A

RFM_GIT_TIMEOUT

Timeout value in seconds used when checking if a git repository exists.

Associated command line option

N/A

Associated configuration parameter

git_timeout general configuration parameter.

New in version 3.9.0.

RFM_GRAYLOG_ADDRESS

The address of the Graylog server to send performance logs. The address is specified in host:port format.

Associated command line option

N/A

Associated configuration parameter

address graylog log handler configuration parameter

New in version 3.1.

RFM_GRAYLOG_SERVER

Deprecated since version 3.1: Please RFM_GRAYLOG_ADDRESS instead.

RFM_HTTPJSON_URL

The URL of the server to send performance logs in JSON format. The URL is specified in scheme://host:port/path format.

Associated command line option

N/A

Associated configuration parameter

url httpjson log handler configuration parameter

New in version 3.6.1.

RFM_IGNORE_CHECK_CONFLICTS

Ignore tests with conflicting names when loading.

Associated command line option

--ignore-check-conflicts

Associated configuration parameter

ignore_check_conflicts general configuration parameter

Deprecated since version 3.8.0: This environment variable will be removed in a future version.

RFM_IGNORE_REQNODENOTAVAIL

Do not treat specially jobs in pending state with the reason ReqNodeNotAvail (Slurm only).

Associated command line option

N/A

Associated configuration parameter

ignore_reqnodenotavail scheduler configuration parameter

RFM_KEEP_STAGE_FILES

Keep test stage directories even for tests that finish successfully.

Associated command line option

--keep-stage-files

Associated configuration parameter

keep_stage_files general configuration parameter

RFM_MODULE_MAP_FILE

A file containing module mappings.

Associated command line option

--module-mappings

Associated configuration parameter

module_map_file general configuration parameter

RFM_MODULE_MAPPINGS

A comma-separated list of module mappings.

Associated command line option

-M

Associated configuration parameter

module_mappings general configuration parameter

RFM_NON_DEFAULT_CRAYPE

Test a non-default Cray Programming Environment.

Associated command line option

--non-default-craype

Associated configuration parameter

non_default_craype general configuration parameter

RFM_OUTPUT_DIR

Directory prefix for test output files.

Associated command line option

-o

Associated configuration parameter

outputdir system configuration parameter

RFM_PERFLOG_DIR

Directory prefix for logging performance data.

Associated command line option

--perflogdir

Associated configuration parameter

basedir logging handler configuration parameter

RFM_PREFIX

General directory prefix for ReFrame-generated directories.

Associated command line option

--prefix

Associated configuration parameter

prefix system configuration parameter

RFM_PURGE_ENVIRONMENT

Unload all environment modules before acting on any tests.

Associated command line option

--purge-env

Associated configuration parameter

purge_environment general configuration parameter

RFM_REMOTE_DETECT

Auto-detect processor information of remote partitions as well.

Associated command line option

N/A

Associated configuration parameter

remote_detect general configuration parameter

New in version 3.7.0.

RFM_REMOTE_WORKDIR

The temporary directory prefix that will be used to create a fresh ReFrame clone, in order to auto-detect the processor information of a remote partition.

Associated command line option

N/A

Associated configuration parameter

remote_workdir general configuration parameter

New in version 3.7.0.

RFM_REPORT_FILE

The file where ReFrame will store its report.

New in version 3.1.

Associated command line option

--report-file

Associated configuration parameter

report_file general configuration parameter

RFM_REPORT_JUNIT

The file where ReFrame will generate a JUnit XML report.

New in version 3.6.0.

Associated command line option

--report-junit

Associated configuration parameter

report_junit general configuration parameter

RFM_RESOLVE_MODULE_CONFLICTS

Resolve module conflicts automatically.

New in version 3.6.0.

Associated command line option

N/A

Associated configuration parameter

resolve_module_conflicts general configuration parameter

RFM_SAVE_LOG_FILES

Save ReFrame log files in the output directory before exiting.

Associated command line option

--save-log-files

Associated configuration parameter

save_log_files general configuration parameter

RFM_STAGE_DIR

Directory prefix for staging test resources.

Associated command line option

-s

Associated configuration parameter

stagedir system configuration parameter

RFM_SYSLOG_ADDRESS

The address of the Syslog server to send performance logs. The address is specified in host:port format. If no port is specified, the address refers to a UNIX socket.

Associated command line option

N/A

Associated configuration parameter

address syslog log handler configuration parameter

New in version 3.1.

RFM_SYSTEM

Set the current system.

Associated command line option

--system

Associated configuration parameter

N/A

RFM_TIMESTAMP_DIRS

Append a timestamp to the output and stage directory prefixes.

Associated command line option

--timestamp

Associated configuration parameter

timestamp_dirs general configuration parameter.

RFM_TRAP_JOB_ERRORS

Trap job errors in submitted scripts and fail tests automatically.

Associated configuration parameter

trap_job_errors general configuration parameter

New in version 3.9.0.

RFM_UNLOAD_MODULES

A comma-separated list of environment modules to be unloaded before acting on any tests.

Associated command line option

-u

Associated configuration parameter

unload_modules general configuration parameter

RFM_USE_LOGIN_SHELL

Use a login shell for the generated job scripts.

Associated command line option

N/A

Associated configuration parameter

use_login_shell general configuration parameter

RFM_USER_MODULES

A comma-separated list of environment modules to be loaded before acting on any tests.

Associated command line option

-m

Associated configuration parameter

user_modules general configuration parameter

RFM_VERBOSE

Set the verbosity level of output.

Associated command line option

-v

Associated configuration parameter

verbose general configuration parameter

Configuration File

The configuration file of ReFrame defines the systems and environments to test as well as parameters controlling its behavior. Upon start up ReFrame checks for configuration files in the following locations in that order:

  1. $HOME/.reframe/settings.{py,json}

  2. $RFM_INSTALL_PREFIX/settings.{py,json}

  3. /etc/reframe.d/settings.{py,json}

ReFrame accepts configuration files either in Python or JSON syntax. If both are found in the same location, the Python file will be preferred.

The RFM_INSTALL_PREFIX environment variable refers to the installation directory of ReFrame. Users have no control over this variable. It is always set by the framework upon startup.

If no configuration file can be found in any of the predefined locations, ReFrame will fall back to a generic configuration that allows it to run on any system. This configuration file is located in reframe/core/settings.py. Users may not modify this file.

For a complete reference of the configuration, please refer to reframe.settings(8) man page.

Reporting Bugs

For bugs, feature request, help, please open an issue on Github: <https://github.com/reframe-hpc/reframe>

See Also

See full documentation online: <https://reframe-hpc.readthedocs.io/>