What’s New in ReFrame 4.0¶
ReFrame 4.0 introduces some important new features and removes all features, configuration options and interfaces that were deprecated in the 3.x versions. It also introduces a couple of new deprecations.
ReFrame 4.0 maintains backward compatibility as much as possible. Existing 3.x configurations and 3.x tests are expected to run out-of-the-box, despite any warnings issued. The framework’s behavior with respect to performance logging has also changed, but configuration options are offered so that users can switch to the old behavior.
This page summarizes the key changes in ReFrame 4.0 and what users should pay attention to.
For a complete list of changes, please refer to the Release Notes.
New Features and Enchancements¶
Chaining Configuration Files¶
There is no need anymore to keep a huge configuration file with all your system and environment definitions and it is no more required to carry on the generic system configuration as well as any part of the builtin configuration.
ReFrame 4.0 allows you to split your configuration in multiple files.
This allows you to create minimal configuration files that contain only the necessary parts.
For example, if you want to define a general configuration parameter, you don’t need to copy the builtin configuration file and add it, but you simply add it in a single
This can also be very useful if you maintain a ReFrame installation used by others, as you can update your settings (systems, environments and other options) and any of your users’ custom configuration will automatically inherit your settings if it is properly chained.
To assist with system-wide installation the
RFM_CONFIG_PATH environment variable is introduced that allows you to specify a path where ReFrame will look for configuration files to load.
Now that systems and environments definitions can be distributed over multiple configuration files, it can become easy to accidentally redefine a system or environment without a notice.
For this reason, ReFrame warns you if a system or an environment is redefined in the same scope.
Since in the past all configuration files where extended copies of the builtin configuration, you will get warnings that the
generic system and the
builtin environment are redefined, as ReFrame finds them in the builtin configuration, which is always loaded.
You can safely ignore these warnings and use the definitions in your configuration file.
If you want to eliminate them, though, you should remove the conflicting definitions from your configuration file.
Although ReFrame will not warn you for redefining other configuration sections, you are also advised to tidy up your configuration file and remove any parts that were copied unchanged from the builtin configuration.
Performance Reporting and Logging¶
ReFrame 4.0 improves on how performance values are logged and reported. This is a breaking change, but you can easily revert to the old behavior.
ReFrame now logs performance after the test has finished and not during the performance stage.
You can now log the result of the test by including
%(check_result)s in your log handler format string.
However, now, by default, ReFrame will log all the performance variables in a single record;
in the past, a new record was logged for each performance variable.
%(check_perf_*)s format placeholders are valid only in the
format_perfvars configuration parameter and will be used to format the performance values if the
%(check_perfvalues)s placeholder is present in the handler’s
This change in behavior will likely break your log processing, especially if you are using the
httpjson handlers or any handler that sends the full record to a log server.
You can revert to the old behavior by setting the
perflog_compat configuration parameter.
This will send a separate record for each performance variable that will include all the individual
For more information, check the documentation of the
format_perfvars configuration parameter.
The behavior of the
filelog is also substantially improved.
The log file is printed by default in CSV format and a header is always printed at the beginning of each log file.
If the log format changes or the performance variables logged by the test change, a new log file will be created with an adapted header.
This way, every log file is consistent with the data in contains.
For more information, please refer to the
filelog handler documentation.
When you run a performance test, ReFrame will now print immediately after the test has finished a short summary of its performance.
You can suppress this output by setting the log level at which this information is printed to
verbose by setting the
perf_info_level general configuration parameter.
Finally, the performance report printed at the end of the run using the
--performance-report is revised providing more information in more compact form.
New Test Naming Scheme¶
ReFrame 4.0 makes default the new test naming scheme introduced in 3.10.0 and drops support of the old naming scheme.
The new naming scheme does not affect normal tests, but it changes how parameterized tests and fixtures are named.
Each test is now also associated with a unique hash code.
For parameterized tests and fixtures this hash code is appended to the test’s or fixture’s base name when creating any test-specific directories and files, such as the test stage and output directories.
-n option can match a test either by its display name (the default), or by its unique internal name or by its unique hash code.
Check the documentation of the
-n for more information.
For the details of the new naming scheme, please refer to the Test Naming Scheme section.
Note that any tests that used the old naming scheme to depend on parameterized tests will break with this change. Check the tutorial Depending on Parameterized Tests on how to create dependencies on parameterized tests in a portable way.
Custom parallel launchers¶
By relaxing the configuration schema, users can now define custom parallel launchers inside their Python configuration file. Check the tutorial Adding a custom launcher to a partition to find out how this can be achieved.
Unique run reports¶
ReFrame now generates a unique report for each run inside the
If you want to revert to the old behavior, where a single file was generated and was overwritten in every run, you should set the
report_file configuration option or the
RFM_REPORT_FILE environment variable.
Dropped Features and Deprecations¶
ReFrame 4.0 drops support for all the deprecated features and behaviors of ReFrame 3.x versions. More specifically, the following deprecated features are dropped:
@parameterized_testdecorator is dropped in favor of the
nameof the test is now read-only.
@run_beforeare no more accesible via the
reframemodule. They are directly available in the
RegressionTestnamespace without the need of importing anything.
@reframe.utility.sanity.sanity_functiondecorator is dropped in favor of the
launcherattribute of the
Systemis dropped in favor of the
@required_versiondecorator is dropped in favor of the
require_versionbuiltin. Also, automatically converting version strings that do not comply with the semantic versioning scheme is no more supported.
DEPEND_FULLYinteger constants that were passed as the
howargument of the
depends_on()method are no more supported and a callable should be used instead. The
subdepsargument is also dropped.
schedulersconfiguration section is dropped in favor of the partition-specific
sched_options. Users should move any options set in the old section to the corresponding partition options.
--ignore-check-conflictscommand line option and the corresponding
RFM_IGNORE_CHECK_CONFLICTSenvironment variable are dropped.
--strictcommand line options are removed. Please use instead
-S strict_check=1, respectively.
RFM_GRAYLOG_SERVERenvironment variable is dropped in favor of the
All occurrences of the
variablesname are deprecated in favor of
env_vars. This includes the
variablestest attribute and the homonym systems, partitions and environments configuration parameters as well as the
perf_patternsattribute is not deprecated, users are recommended to migrate to using the new
@performance_functionbuiltin. Please refer to Writing A Performance Test tutorial for a starting point.