Tutorial 5: Using Build Automation Tools As a Build System

In this tutorial we will present how to use Easybuild and Spack as a build system for a ReFrame test. The example uses the configuration file presented in Tutorial 1: Getting Started with ReFrame, which you can find in tutorials/config/settings.py. We also assume that the reader is already familiar with the concepts presented in the basic tutorial and has a working knowledge of EasyBuild and Spack.

Using EasyBuild to Build the Test Code

New in version 3.5.0.

Let’s consider a simple ReFrame test that installs bzip2-1.0.6 given the easyconfig bzip2-1.0.6.eb and checks that the installed version is correct. The following code block shows the check, highlighting the lines specific to this tutorial:

import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class BZip2EBCheck(rfm.RegressionTest):
    descr = 'Demo test using EasyBuild to build the test code'
    valid_systems = ['*']
    valid_prog_environs = ['builtin']
    executable = 'bzip2'
    executable_opts = ['--help']
    build_system = 'EasyBuild'

    @run_before('compile')
    def setup_build_system(self):
        self.build_system.easyconfigs = ['bzip2-1.0.6.eb']
        self.build_system.options = ['-f']

    @run_before('run')
    def prepare_run(self):
        self.modules = self.build_system.generated_modules

    @sanity_function
    def assert_version(self):
        return sn.assert_found(r'Version 1.0.6', self.stderr)

The test looks pretty standard except for the highlighted blocks. Let’s have a look first to the block in the BZip2Check class.

The first thing is to specify that the EasyBuild build system will be used. This is done by setting build_system to 'EasyBuild'. Then, the software to be installed is passed as a list to easyconfigs. Here only one easyconfig is given, but more than one can be passed. Finally, through options, command line options can be passed to the eb executable. In this test we pass -f to make sure that bzip2 will be built even if the module already exists externally.

For this test, ReFrame generates the following command to build and install the easyconfig:

export EASYBUILD_BUILDPATH={stagedir}/easybuild/build
export EASYBUILD_INSTALLPATH={stagedir}/easybuild
export EASYBUILD_PREFIX={stagedir}/easybuild
export EASYBUILD_SOURCEPATH={stagedir}/easybuild
eb bzip2-1.0.6.eb -f

ReFrame will keep all the files generated by EasyBuild (sources, temporary files, installed software and the corresponding modules) under the test’s stage directory. For this reason it sets the relevant EasyBuild environment variables.

Tip

Users may set the EasyBuild prefix to a different location by setting the prefix attribute of the build system. This allows you to have the built software installed upon successful completion of the build phase, but if the test fails in a later stage (sanity, performance), the installed software will not be cleaned up automatically.

Note

ReFrame assumes that the eb executable is available on the system where the compilation is run (typically the local host where ReFrame is executed).

Now that we know everything related to building and installing the code, we can move to the part dealing with running it. To run the code, the generated modules need to be loaded in order to make the software available. The modules can be accessed through generated_modules, however, they are available only after EasyBuild completes the installation. This means that modules can be set only after the build phase finishes. For that, we can set modules in a class method wrapped by the run_before() built-in, specifying the run phase. This test will then run the following commands:

module load bzip/1.0.6
bzip2 --help

Packaging the installation

The EasyBuild build system offers a way of packaging the installation via EasyBuild’s packaging support. To use this feature, the FPM package manager must be available. By setting the dictionary package_opts in the test, ReFrame will pass --package-{key}={val} to the EasyBuild invocation. For instance, the following can be set to package the installations as an rpm file:

self.keep_files = ['easybuild/packages']
self.build_system.package_opts = {
    'type': 'rpm',
}

The packages are generated by EasyBuild in the stage directory. To retain them after the test succeeds, keep_files needs to be set.

Using Spack to Build the Test Code

New in version 3.6.1.

This example is the equivalent to the previous one, except that it uses Spack to build bzip2. Here is the test’s code:

import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class BZip2SpackCheck(rfm.RegressionTest):
    descr = 'Demo test using Spack to build the test code'
    valid_systems = ['*']
    valid_prog_environs = ['builtin']
    executable = 'bzip2'
    executable_opts = ['--help']
    build_system = 'Spack'

    @run_before('compile')
    def setup_build_system(self):
        self.build_system.specs = ['bzip2@1.0.6']

    @sanity_function
    def assert_version(self):
        return sn.assert_found(r'Version 1.0.6', self.stderr)

When build_system is set to 'Spack', ReFrame will leverage Spack environments in order to build the test code. By default, ReFrame will create a new Spack environment in the test’s stage directory and add the requested specs to it. Users may also specify an existing Spack environment by setting the environment attribute. In this case, ReFrame treats the environment as a test resource so it expects to find it under the test’s sourcesdir, which defaults to 'src'.

As with every other test, ReFrame will copy the test’s resources to its stage directory before building it. ReFrame will then activate the generated environment (either the one provided by the user or the one generated by ReFrame), add the given specs using the spack add command and, finally, install the packages in the environment. Here is what ReFrame generates as a build script for this example:

. "$(spack location --spack-root)/share/spack/setup-env.sh"
spack env create -d rfm_spack_env
spack env activate -V -d rfm_spack_env
spack config add "config:install_tree:root:opt/spack"
spack add bzip2@1.0.6
spack install

As you might have noticed ReFrame expects that Spack is already installed on the system. The packages specified in the environment and the tests will be installed in the test’s stage directory, where the environment is copied before building. Here is the stage directory structure:

stage/generic/default/builtin/BZip2SpackCheck/
├── rfm_spack_env
│   ├── spack
│   │   └── opt
│   │       └── spack
│   │           ├── bin
│   │           └── darwin-catalina-skylake
│   ├── spack.lock
│   └── spack.yaml
├── rfm_BZip2SpackCheck_build.err
├── rfm_BZip2SpackCheck_build.out
├── rfm_BZip2SpackCheck_build.sh
├── rfm_BZip2SpackCheck_job.err
├── rfm_BZip2SpackCheck_job.out
└── rfm_BZip2SpackCheck_job.sh

Finally, here is the generated run script that ReFrame uses to run the test, once its build has succeeded:

#!/bin/bash
. "$(spack location --spack-root)/share/spack/setup-env.sh"
spack env create -d rfm_spack_env
spack env activate -V -d rfm_spack_env
spack load bzip2@1.0.6
bzip2 --help

From this point on, sanity and performance checking are exactly identical to any other ReFrame test.

Tip

While developing a test using Spack or EasyBuild as a build system, it can be useful to run ReFrame with the --keep-stage-files and --dont-restage options to prevent ReFrame from removing the test’s stage directory upon successful completion of the test. For this particular type of test, these options will avoid having to rebuild the required package dependencies every time the test is retried.