Welcome to ReFrame¶

ReFrame is a powerful framework for writing system regression tests and benchmarks, specifically targeted to HPC systems. The goal of the framework is to abstract away the complexity of the interactions with the system, separating the logic of a test from the low-level details, which pertain to the system configuration and setup. This allows users to write portable tests in a declarative way that describes only the test’s functionality.

Tests in ReFrame are simple Python classes that specify the basic variables and parameters of the test. ReFrame offers an intuitive and very powerful syntax that allows users to create test libraries, test factories, as well as complete test workflows using other tests as fixtures. ReFrame will load the tests and send them down a well-defined pipeline that will execute them in parallel. The stages of this pipeline take care of all the system interaction details, such as programming environment switching, compilation, job submission, job status query, sanity checking and performance assessment.

ReFrame also offers a high-level and flexible abstraction for writing sanity and performance checks for your regression tests, without having to care about the details of parsing output files, searching for patterns and testing against reference values for different systems.

Finally, ReFrame offers a powerful and efficient runtime for running and managing the execution of tests, as well as integration with common logging facilities, where ReFrame can send live data from currently running performance tests.

Publications¶

Presentations & Talks¶

[slides] “Introduction to ReFrame,” CINECA visit, Jun 2024.
[slides][recording] “Recent Advances in ReFrame,” 9th EasyBuild User Meeting 2024.
[slides][recording] “Recent Advances in ReFrame,” 8th EasyBuild User Meeting 2023.
[slides][recording] “Embracing ReFrame Programmable Configurations,” 8th EasyBuild User Meeting 2023.
[slides] “ReFrame Update,” 7th EasyBuild User Meeting 2022.
[slides] “Writing powerful HPC regression tests with ReFrame,” 6th EasyBuild User Meeting 2021
[slides] “ReFrame: A Framework for Writing Regression Tests for HPC Systems,” 5th EasyBuild User Meeting 2020.
[slides] “Enabling Continuous Testing of HPC Systems using ReFrame,” HPC System Testing BoF, SC’19.
[slides] “Enabling Continuous Testing of HPC Systems using ReFrame,” HUST 2019, SC’19.
[slides] “ReFrame: A Tool for Enabling Regression Testing and Continuous Integration for HPC Systems,” HPC Knowledge Meeting ‘19.
[slides][recording] “ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems,” FOSDEM’19.
[slides] “ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems,” 4th EasyBuild User Meeting.
[slides] “ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems,” HUST 2018, SC’18.
[slides] “Regression Testing and Continuous Integration with ReFrame,” CSCS User Lab Day 2018.
[slides] “ReFrame: A Regression Testing Framework Enabling Continuous Integration of Large HPC Systems,” HPC Advisory Council 2018.
[slides] “ReFrame: A Regression Testing Tool for HPC Systems,” Regression testing BoF, SC17.
[slides] “ReFrame: A regression framework for checking the health of large HPC systems” CUG 2017.

Webinars & Tutorials¶

[slides][recording][demo run] “ReFrame – Efficient System and Application Performance Testing,” CSCS Webinar, Aug. 29, 2022.
[recording] “ReFrame Tutorial,” 6th EasyBuild User Meeting 2021.

Papers¶

Vasileios Karakasis et al. “A regression framework for checking the health of large HPC systems”. In: Cray User Group 2017 (Redmond, Washington, USA, May 8–11, 2017). [pdf]
Vasileios Karakasis et al. “Enabling Continuous Testing of HPC Systems Using ReFrame”. In: Tools and Techniques for High Performance Computing. HUST – Annual Workshop on HPC User Support Tools (Denver, Colorado, USA, Nov. 17–18, 2019). Ed. by Guido Juckeland and Sunita Chandrasekaran. Vol. 1190. Communications in Computer and Information Science. Cham, Switzerland: Springer International Publishing, Mar. 2020, pp. 49–68. isbn: 978-3-030-44728-1. doi: 10.1007/978-3-030-44728-1_3.

Table of Contents