Welcome to ReFrame¶
ReFrame is a powerful framework for writing system regression tests and benchmarks, specifically targeted to HPC systems. The goal of the framework is to abstract away the complexity of the interactions with the system, separating the logic of a test from the low-level details, which pertain to the system configuration and setup. This allows users to write portable tests in a declarative way that describes only the test’s functionality.
Tests in ReFrame are simple Python classes that specify the basic variables and parameters of the test. ReFrame offers an intuitive and very powerful syntax that allows users to create test libraries, test factories, as well as complete test workflows using other tests as fixtures. ReFrame will load the tests and send them down a well-defined pipeline that will execute them in parallel. The stages of this pipeline take care of all the system interaction details, such as programming environment switching, compilation, job submission, job status query, sanity checking and performance assessment.
ReFrame also offers a high-level and flexible abstraction for writing sanity and performance checks for your regression tests, without having to care about the details of parsing output files, searching for patterns and testing against reference values for different systems.
Finally, ReFrame offers a powerful and efficient runtime for running and managing the execution of tests, as well as integration with common logging facilities, where ReFrame can send live data from currently running performance tests.
Publications¶
Slides [part 1][part 2][talk] @ 8th EasyBuild User Meeting 2023.
Slides [pdf] @ 7th EasyBuild User Meeting 2022.
Slides [pdf] @ 6th EasyBuild User Meeting 2021.
Slides [pdf] @ 5th EasyBuild User Meeting 2020.
Slides [pdf] @ HPC System Testing BoF, SC’19.
Slides [pdf] @ HPC Knowledge Meeting ‘19.
Slides [pdf] @ 4th EasyBuild User Meeting.
Slides [pdf] @ CSCS User Lab Day 2018.
Slides [pdf] @ HPC Advisory Council 2018.
Webinars and Tutorials¶
“ReFrame – Efficient System and Application Performance Testing,” CSCS Webinar, Aug. 29, 2022 [slides] [recording] [demo run].
Tutorial at 6th EasyBuild User Meeting 2021 [YouTube]
- Getting Started
- ReFrame Tutorial
- Requirements
- Writing your first test
- Running a test
- Systems and environments
- Compiling the test code
- Test fixtures
- Test variables
- Test parameterization
- Mastering sanity and performance checking
- Interacting with workload managers
- Accessing CPU topology information
- Multi-node tests
- Managing the run session
- Managing the configuration
- Logging
- ReFrame How Tos
- Working with build systems
- Working with environment modules
- Working with low-level dependencies
- Integrating into a CI pipeline
- Flexible tests
- Testing containerized applications
- Generating tests programmatically
- Using the Flux framework scheduler
- Building test libraries and utilities
- Debugging
- Extending the framework
- Advanced Topics
- ReFrame Manuals
- Command Line Reference
- Configuration Reference
- Top-level Configuration
- System Configuration
- System Partition Configuration
systems.partitions.name
systems.partitions.descr
systems.partitions.scheduler
systems.partitions.sched_options
systems.partitions.launcher
systems.partitions.access
systems.partitions.environs
systems.partitions.container_platforms
systems.partitions.modules
systems.partitions.time_limit
systems.partitions.env_vars
systems.partitions.variables
systems.partitions.max_jobs
systems.partitions.prepare_cmds
systems.partitions.resources
systems.partitions.processor
systems.partitions.devices
systems.partitions.features
systems.partitions.extras
- Container Platform Configuration
- Custom Job Scheduler Resources
- Environment Configuration
environments.name
environments.modules
environments.env_vars
environments.variables
environments.features
environments.extras
environments.prepare_cmds
environments.cc
environments.cxx
environments.ftn
environments.cppflags
environments.cflags
environments.cxxflags
environments.fflags
environments.ldflags
environments.nvcc
environments.target_systems
environments.resources
- Logging Configuration
logging.level
logging.handlers
logging.handlers_perflog
logging.perflog_compat
logging.target_systems
- Common logging handler properties
- The
file
log handler - The
filelog
log handler - The
graylog
log handler - The
stream
log handler - The
syslog
log handler - The
httpjson
log handlerlogging.handlers_perflog..httpjson..url
logging.handlers_perflog..httpjson..extra_headers
logging.handlers_perflog..httpjson..extras
logging.handlers_perflog..httpjson..ignore_keys
logging.handlers_perflog..httpjson..debug
logging.handlers_perflog..httpjson..json_formatter
logging.handlers_perflog..httpjson.json_formatter()
- Execution Mode Configuration
- General Configuration
general.check_search_path
general.check_search_recursive
general.clean_stagedir
general.colorize
general.compress_report
general.git_timeout
general.dump_pipeline_progress
general.pipeline_timeout
general.perf_info_level
general.remote_detect
general.remote_workdir
general.ignore_check_conflicts
general.trap_job_errors
general.keep_stage_files
general.module_map_file
general.module_mappings
general.non_default_craype
general.purge_environment
general.report_file
general.report_junit
general.resolve_module_conflicts
general.save_log_files
general.target_systems
general.timestamp_dirs
general.unload_modules
general.use_login_shell
general.user_modules
general.verbose
- Module Objects
environments.modules.name
systems.modules.name
systems.partitions.modules.name
systems.partitions.container_platforms.modules.name
environments.modules.collection
systems.modules.collection
systems.partitions.modules.collection
systems.partitions.container_platforms.modules.collection
environments.modules.path
systems.modules.path
systems.partitions.modules.path
systems.partitions.container_platforms.modules.path
- Processor Info
systems.partitions.processor.arch
systems.partitions.processor.model
systems.partitions.processor.platform
systems.partitions.processor.num_cpus
systems.partitions.processor.num_cpus_per_core
systems.partitions.processor.num_cpus_per_socket
systems.partitions.processor.num_sockets
systems.partitions.processor.topology
- Device Info
- Programming APIs
- Test API Reference
- Test Base Classes
CompileOnlyRegressionTest
RegressionMixin
RegressionTest
RegressionTest.build_locally
RegressionTest.build_system
RegressionTest.build_time_limit
RegressionTest.check_performance()
RegressionTest.check_sanity()
RegressionTest.ci_extras
RegressionTest.cleanup()
RegressionTest.compile()
RegressionTest.compile_complete()
RegressionTest.compile_wait()
RegressionTest.container_platform
RegressionTest.current_environ
RegressionTest.current_partition
RegressionTest.current_system
RegressionTest.depends_on()
RegressionTest.descr
RegressionTest.display_name
RegressionTest.env_vars
RegressionTest.exclusive_access
RegressionTest.executable
RegressionTest.executable_opts
RegressionTest.extra_resources
RegressionTest.fixture_variant
RegressionTest.getdep()
RegressionTest.info()
RegressionTest.is_dry_run()
RegressionTest.is_fixture()
RegressionTest.is_local()
RegressionTest.is_performance_check()
RegressionTest.job
RegressionTest.keep_files
RegressionTest.local
RegressionTest.logger
RegressionTest.maintainers
RegressionTest.max_pending_time
RegressionTest.modules
RegressionTest.name
RegressionTest.num_cpus_per_task
RegressionTest.num_gpus_per_node
RegressionTest.num_tasks
RegressionTest.num_tasks_per_core
RegressionTest.num_tasks_per_node
RegressionTest.num_tasks_per_socket
RegressionTest.outputdir
RegressionTest.param_variant
RegressionTest.perf_patterns
RegressionTest.perf_variables
RegressionTest.postbuild_cmds
RegressionTest.postrun_cmds
RegressionTest.prebuild_cmds
RegressionTest.prefix
RegressionTest.prerun_cmds
RegressionTest.readonly_files
RegressionTest.reference
RegressionTest.require_reference
RegressionTest.run()
RegressionTest.run_complete()
RegressionTest.run_wait()
RegressionTest.sanity_patterns
RegressionTest.set_var_default()
RegressionTest.setup()
RegressionTest.short_name
RegressionTest.skip()
RegressionTest.skip_if()
RegressionTest.skip_if_no_procinfo()
RegressionTest.sourcepath
RegressionTest.sourcesdir
RegressionTest.stagedir
RegressionTest.stderr
RegressionTest.stdout
RegressionTest.strict_check
RegressionTest.tags
RegressionTest.time_limit
RegressionTest.unique_name
RegressionTest.use_multithreading
RegressionTest.valid_prog_environs
RegressionTest.valid_systems
RegressionTest.variables
RegressionTest.variant_num
RunOnlyRegressionTest
- Test Decorators
- Builtins
- Pipeline Hooks
- Test variants
- Dynamic Creation of Tests
- Environments and Systems
Environment
ProgEnvironment
_EnvironmentSnapshot
snapshot()
DeviceInfo
ProcessorInfo
System
SystemPartition
SystemPartition.access
SystemPartition.container_environs
SystemPartition.container_runtime
SystemPartition.descr
SystemPartition.devices
SystemPartition.environment()
SystemPartition.environs
SystemPartition.extras
SystemPartition.features
SystemPartition.fullname
SystemPartition.json()
SystemPartition.launcher_type
SystemPartition.local_env
SystemPartition.max_jobs
SystemPartition.name
SystemPartition.prepare_cmds
SystemPartition.processor
SystemPartition.resources
SystemPartition.scheduler
SystemPartition.select_devices()
SystemPartition.time_limit
- Jobs and Parallel Launchers
Job
Job.cli_options
Job.completion_time
Job.exception
Job.exclusive_access
Job.exitcode
Job.jobid
Job.launcher
Job.max_pending_time
Job.name
Job.nodelist
Job.num_cpus_per_task
Job.num_tasks
Job.num_tasks_per_core
Job.num_tasks_per_node
Job.num_tasks_per_socket
Job.options
Job.pin_nodes
Job.sched_access
Job.sched_flex_alloc_nodes
Job.scheduler
Job.script_filename
Job.state
Job.stderr
Job.stdout
Job.submit_time
Job.time_limit
Job.use_smt
Job.workdir
filter_nodes_by_state()
JobLauncher
LauncherWrapper
getlauncher()
getscheduler()
- Runtime Services
- Modules Systems
ModulesSystem
ModulesSystem.available_modules()
ModulesSystem.conflicted_modules()
ModulesSystem.emit_load_commands()
ModulesSystem.emit_unload_commands()
ModulesSystem.execute()
ModulesSystem.is_module_loaded()
ModulesSystem.load_module()
ModulesSystem.loaded_modules()
ModulesSystem.name
ModulesSystem.searchpath
ModulesSystem.searchpath_add()
ModulesSystem.searchpath_remove()
ModulesSystem.unload_all()
ModulesSystem.unload_module()
ModulesSystem.version
- Build Systems
Autotools
Autotools.builddir
Autotools.cc
Autotools.cflags
Autotools.config_opts
Autotools.configuredir
Autotools.cppflags
Autotools.cxx
Autotools.cxxflags
Autotools.fflags
Autotools.flags_from_environ
Autotools.ftn
Autotools.ldflags
Autotools.make_opts
Autotools.max_concurrency
Autotools.nvcc
Autotools.srcdir
BuildSystem
BuildSystemMeta
CMake
ConfigureBasedBuildSystem
CustomBuild
EasyBuild
Make
SingleSource
Spack
- Container Platforms
- The
reframe
module - Mapping of Test Attributes to Job Scheduler Backends
- Test Base Classes
- Deferrable Functions Reference
- Explicit evaluation of deferrable functions
- Implicit evaluation of deferrable functions
- Categories of deferrable functions
- List of deferrable functions and utilities
reframe.utility.sanity.deferrable()
abs()
all()
allx()
and_()
any()
assert_bounded()
assert_eq()
assert_false()
assert_found()
assert_found_s()
assert_ge()
assert_gt()
assert_in()
assert_le()
assert_lt()
assert_ne()
assert_not_found()
assert_not_found_s()
assert_not_in()
assert_reference()
assert_true()
avg()
chain()
contains()
count()
count_uniq()
defer()
enumerate()
evaluate()
extractall()
extractall_s()
extractiter()
extractiter_s()
extractsingle()
extractsingle_s()
filter()
findall()
findall_s()
finditer()
finditer_s()
getattr()
getitem()
glob()
hasattr()
iglob()
len()
make_performance_function()
map()
max()
min()
not_()
or_()
path_exists()
path_isdir()
path_isfile()
path_islink()
print()
reversed()
round()
setattr()
sorted()
sum()
zip()
- Utility Functions
- General Utilities
MappingView
OrderedSet
ScopedDict
SequenceView
allx()
attr_validator()
cache_return_value()
decamelize()
find_modules()
import_from_module()
import_module()
import_module_from_file()
is_copyable()
is_picklable()
is_trivially_callable()
longest()
nodelist_abbrev()
nodelist_expand()
ppretty()
repr()
shortest()
temp_setattr
temp_sys_path
- System Utilities
UnstartedProcError
_ProcFuture
_ProcFuture.add_done_callback()
_ProcFuture.cancel()
_ProcFuture.cancelled()
_ProcFuture.done()
_ProcFuture.exception()
_ProcFuture.exitcode
_ProcFuture.is_session()
_ProcFuture.kill()
_ProcFuture.pid
_ProcFuture.signal
_ProcFuture.start()
_ProcFuture.started()
_ProcFuture.stderr()
_ProcFuture.stdout()
_ProcFuture.terminate()
_ProcFuture.then()
_ProcFuture.wait()
change_dir
concat_files()
copytree()
copytree_virtual()
cray_cdt_version()
cray_cle_info()
expandvars()
follow_link()
force_remove_file()
git_clone()
git_repo_exists()
git_repo_hash()
inpath()
is_interactive()
is_url()
mkstemp_path()
osgroup()
osuser()
rmtree()
run_command()
run_command_async()
run_command_async2()
samefile()
subdirs()
unique_abs_paths()
- Type Checking Utilities
- Test Case Dependencies Management
- General Utilities
- ReFrame Errors
AbortTaskError
BuildError
BuildSystemError
CommandLineError
ConfigError
ContainerError
DependencyError
EnvironError
FailureLimitError
ForceExitError
JobBlockedError
JobError
JobNotStartedError
JobSchedulerError
LoggingError
NameConflictError
PerformanceError
PipelineError
ReframeBaseError
ReframeError
ReframeFatalError
ReframeSyntaxError
RegressionTestLoadError
RunSessionTimeout
SanityError
SkipTestError
SpawnedProcessError
SpawnedProcessTimeout
StatisticsError
TaskDependencyError
TaskExit
is_exit_request()
is_severe()
is_user_error()
is_warning()
user_frame()
what()
- Test API Reference
- What’s New in ReFrame 4.0
- ReFrame Test Library (experimental)
- Data Analytics
- Interactive Computing
- Machine Learning
- Microbenchmarks
- Python
- Scientific Applications
amber_nve_check
gromacs_check
QEspressoPWCheck
QEspressoPWCheck.assert_job_finished()
QEspressoPWCheck.convert_timings()
QEspressoPWCheck.ecut
QEspressoPWCheck.extract_report_time()
QEspressoPWCheck.extractsingle_or_val()
QEspressoPWCheck.input_name
QEspressoPWCheck.nbnd
QEspressoPWCheck.pp_name
QEspressoPWCheck.prepare_test()
QEspressoPWCheck.set_perf_variables()
QEspressoPWCheck.write_input()
- System