What’s New in ReFrame 4.0¶

ReFrame 4.0 introduces some important new features and removes all features, configuration options and interfaces that were deprecated in the 3.x versions. It also introduces a couple of new deprecations.

ReFrame 4.0 maintains backward compatibility as much as possible. Existing 3.x configurations and 3.x tests are expected to run out-of-the-box, despite any warnings issued. The framework’s behavior with respect to performance logging has also changed, but configuration options are offered so that users can switch to the old behavior.

This page summarizes the key changes in ReFrame 4.0 and what users should pay attention to.

For a complete list of changes, please refer to the Release Notes.

New Features and Enchancements¶

Chaining Configuration Files¶

There is no need anymore to keep a huge configuration file with all your system and environment definitions and it is no more required to carry on the generic system configuration as well as any part of the builtin configuration. ReFrame 4.0 allows you to split your configuration in multiple files. This allows you to create minimal configuration files that contain only the necessary parts. For example, if you want to define a general configuration parameter, you don’t need to copy the builtin configuration file and add it, but you simply add it in a single general section. This can also be very useful if you maintain a ReFrame installation used by others, as you can update your settings (systems, environments and other options) and any of your users’ custom configuration will automatically inherit your settings if it is properly chained. To assist with system-wide installation the RFM_CONFIG_PATH environment variable is introduced that allows you to specify a path where ReFrame will look for configuration files to load.

Now that systems and environments definitions can be distributed over multiple configuration files, it can become easy to accidentally redefine a system or environment without a notice. For this reason, ReFrame warns you if a system or an environment is redefined in the same scope. Since in the past all configuration files where extended copies of the builtin configuration, you will get warnings that the generic system and the builtin environment are redefined, as ReFrame finds them in the builtin configuration, which is always loaded. You can safely ignore these warnings and use the definitions in your configuration file. If you want to eliminate them, though, you should remove the conflicting definitions from your configuration file.

Although ReFrame will not warn you for redefining other configuration sections, you are also advised to tidy up your configuration file and remove any parts that were copied unchanged from the builtin configuration.

For more information on how ReFrame 4.0 builds and loads its configuration, please refer to the documentation of the --config-file option.

Performance Reporting and Logging¶

ReFrame 4.0 improves on how performance values are logged and reported. This is a breaking change, but you can easily revert to the old behavior.

ReFrame now logs performance after the test has finished and not during the performance stage. You can now log the result of the test by including %(check_result)s in your log handler format string. However, now, by default, ReFrame will log all the performance variables in a single record; in the past, a new record was logged for each performance variable. Also, the %(check_perf_*)s format placeholders are valid only in the format_perfvars configuration parameter and will be used to format the performance values if the %(check_perfvalues)s placeholder is present in the handler’s format parameter. This change in behavior will likely break your log processing, especially if you are using the graylog or httpjson handlers or any handler that sends the full record to a log server. You can revert to the old behavior by setting the perflog_compat configuration parameter. This will send a separate record for each performance variable that will include all the individual %(check_perf_*)s attributes. For more information, check the documentation of the format_perfvars configuration parameter.

The behavior of the filelog is also substantially improved. The log file is printed by default in CSV format and a header is always printed at the beginning of each log file. If the log format changes or the performance variables logged by the test change, a new log file will be created with an adapted header. This way, every log file is consistent with the data in contains. For more information, please refer to the filelog handler documentation.

When you run a performance test, ReFrame will now print immediately after the test has finished a short summary of its performance. You can suppress this output by setting the log level at which this information is printed to verbose by setting the perf_info_level general configuration parameter.

Finally, the performance report printed at the end of the run using the --performance-report is revised providing more information in more compact form.

New Test Naming Scheme¶

ReFrame 4.0 makes default the new test naming scheme introduced in 3.10.0 and drops support of the old naming scheme. The new naming scheme does not affect normal tests, but it changes how parameterized tests and fixtures are named. Each test is now also associated with a unique hash code. For parameterized tests and fixtures this hash code is appended to the test’s or fixture’s base name when creating any test-specific directories and files, such as the test stage and output directories. The -n option can match a test either by its display name (the default), or by its unique internal name or by its unique hash code. Check the documentation of the -n for more information. For the details of the new naming scheme, please refer to the Test Naming Scheme section.

Note that any tests that used the old naming scheme to depend on parameterized tests will break with this change. Check the article Depending on parameterized tests on how to create dependencies on parameterized tests in a portable way.

Custom parallel launchers¶

By relaxing the configuration schema, users can now define custom parallel launchers inside their Python configuration file. Check the article Implementing a parallel launcher backend to find out how this can be achieved.

Unique run reports¶

ReFrame now generates a unique report for each run inside the $HOME/.reframe/reports directory. If you want to revert to the old behavior, where a single file was generated and was overwritten in every run, you should set the report_file configuration option or the RFM_REPORT_FILE environment variable.

New Backends¶

ReFrame 4.0 adds support for the Apptainer container platform and the Flux framework.

Dropped Features and Deprecations¶

ReFrame 4.0 drops support for all the deprecated features and behaviors of ReFrame 3.x versions. More specifically, the following deprecated features are dropped:

The @parameterized_test decorator is dropped in favor of the parameter builtin.
The name of the test is now read-only.
The decorators @final, @require_deps, @run_after and @run_before are no more accesible via the reframe module. They are directly available in the RegressionTest namespace without the need of importing anything.
The @reframe.utility.sanity.sanity_function decorator is dropped in favor of the @deferrable builtin.
The commands attribute of the ContainerPlatform is dropped in favor of the command attribute.
The launcher attribute of the System is dropped in favor of the launcher_type attribute.
The @required_version decorator is dropped in favor of the require_version builtin. Also, automatically converting version strings that do not comply with the semantic versioning scheme is no more supported.
The DEPEND_EXACT, DEPEND_BY_ENV and DEPEND_FULLY integer constants that were passed as the how argument of the depends_on() method are no more supported and a callable should be used instead. The subdeps argument is also dropped.
The low-level poll() and wait() RegressionTest methods are dropped in favor of the run_complete() and run_wait(), respectively.
The schedulers configuration section is dropped in favor of the partition-specific sched_options. Users should move any options set in the old section to the corresponding partition options.
The --ignore-check-conflicts command line option and the corresponding RFM_IGNORE_CHECK_CONFLICTS environment variable are dropped.
The --force-local and --strict command line options are removed. Please use instead -S local=1 and -S strict_check=1, respectively.
The RFM_GRAYLOG_SERVER environment variable is dropped in favor of the RFM_GRAYLOG_ADDRESS.

New Deprecations¶

All occurrences of the variables name are deprecated in favor of env_vars. This includes the variables test attribute and the homonym systems, partitions and environments configuration parameters as well as the variables of the Environment base class.
Although perf_patterns attribute is not deprecated, users are recommended to migrate to using the new @performance_function builtin. Please refer to Writing your first test of the tutorial for a starting point.