ReFrame Test Library (experimental)

This is a collection of generic tests that you can either run out-of-the-box by specializing them for your system using the -S option or create your site-specific tests by building upon them.

Scientific Applications

class hpctestlib.sciapps.amber.nve.amber_nve_check(*args, **kwargs)[source]

Bases: reframe.core.pipeline.RunOnlyRegressionTest

Amber NVE test.

Amber is a suite of biomolecular simulation programs. It began in the late 1970’s, and is maintained by an active development community.

This test is parametrized over the benchmark type (see benchmark_info) and the variant of the code (see variant). Each test instance executes the benchmark, validates numerically its output and extracts and reports a performance metric.

assert_energy_readout()[source]

Assert that the obtained energy meets the required tolerance.

benchmark

The name of the benchmark that this test encodes.

This is set from the corresponding value in the benchmark_info parameter pack during initialization.

Type

str

Required

Yes

benchmark_info = (('Cellulose_production_NVE', -443246.0, 5e-05), ('FactorIX_production_NVE', -234188.0, 0.0001), ('JAC_production_NVE_4fs', -44810.0, 0.001), ('JAC_production_NVE', -58138.0, 0.0005))

Parameter pack encoding the benchmark information.

The first element of the tuple refers to the benchmark name, the second is the energy reference and the third is the tolerance threshold.

Type

Tuple[str, float, float]

Values
[
    ('Cellulose_production_NVE', -443246.0, 5.0E-05),
    ('FactorIX_production_NVE', -234188.0, 1.0E-04),
    ('JAC_production_NVE_4fs', -44810.0, 1.0E-03),
    ('JAC_production_NVE', -58138.0, 5.0E-04)
]
energy_ref

Energy value reference.

This is set from the corresponding value in the benchmark_info parameter pack during initialization.

Type

float

Required

Yes

energy_tol

Energy value tolerance.

This is set from the corresponding value in the benchmark_info parameter pack during initialization.

Type

float

Required

Yes

input_file

The input file to use.

This is set to mdin.CPU or mdin.GPU depending on the test variant during initialization.

Type

str

Required

Yes

num_tasks

See num_tasks.

The mpi variant of the test requires num_tasks > 1.

Required

Yes

output_file = 'amber.out'

The output file to pass to the Amber executable.

Type

str

Required

No

Default

'amber.out'

perf()[source]

The performance of the benchmark expressed in ns/day.

class hpctestlib.sciapps.gromacs.benchmarks.gromacs_check(*args, **kwargs)[source]

Bases: reframe.core.pipeline.RunOnlyRegressionTest

GROMACS benchmark test.

GROMACS is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles.

The benchmarks consist on a set of different inputs files that vary in the number of atoms and can be found in the following repository, which is also versioned: https://github.com/victorusu/GROMACS_Benchmark_Suite/.

Each test instance validates numerically its output and extracts and reports a performance metric.

assert_energy_readout()[source]

Assert that the obtained energy meets the benchmark tolerances.

property bench_name

The benchmark name.

Type

str

benchmark_info = (('HECBioSim/Crambin', -204107.0, 0.001), ('HECBioSim/Glutamine-Binding-Protein', -724598.0, 0.001), ('HECBioSim/hEGFRDimer', -3328920.0, 0.001), ('HECBioSim/hEGFRDimerSmallerPL', -3270800.0, 0.001), ('HECBioSim/hEGFRDimerPair', -12073300.0, 0.001), ('HECBioSim/hEGFRtetramerPair', -20983100.0, 0.001))

Parameter pack encoding the benchmark information.

The first element of the tuple refers to the benchmark name, the second is the energy reference and the third is the tolerance threshold.

Type

Tuple[str, float, float]

Values

benchmark_version = '1.0.0'

The version of the benchmark suite to use.

Type

str

Default

'1.0.0'

property energy_ref

The energy reference value for this benchmark.

Type

str

property energy_tol

The energy tolerance value for this benchmark.

Type

str

nb_impl = ('cpu', 'gpu')

Parameter encoding the implementation of the non-bonded calculations

Type

str

Values

['cpu', 'gpu']

Data Analytics

class hpctestlib.data_analytics.spark.spark_checks.compute_pi_check(*args, **kwargs)[source]

Bases: reframe.core.pipeline.RunOnlyRegressionTest

Test Apache Spark by computing PI.

Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for incremental computation and stream processing (see spark.apache.org).

This test checks that Spark is functioning correctly. To do this, it is necessary to define the tolerance of acceptable deviation. The tolerance is used to check that the computations are executed correctly, by comparing the value of pi calculated to the one obtained from the math library. The default assumption is that Spark is already installed on the system under test.

assert_pi_readout()[source]

Assert that the obtained pi value meets the specified tolerances.

exec_cores = 1

The number of cores per each Spark executor

Type

int

Required

No

Default

1

executor_memory

Amount of memory to use per executor process, following the JVM memory strings convention, i.e a number with a size unit suffix (“k”, “m”, “g” or “t”) (e.g. 512m, 2g)

Type

str

Required

Yes

num_workers = 1

The number of Spark workers per node

Type

int

Required

No

Default

1

spark_local_dirs = '/tmp'

The local directories used by Spark

Type

str

Required

No

Default

‘/tmp’

spark_prefix

The Spark installation prefix path

Type

str

Required

Yes

tolerance = 0.01

The absolute tolerance of the computed value of PI

Type

float

Required

No

Default

0.01

variant = ('spark', 'pyspark')

Parameter encoding the variant of the test.

Type

str

Values

['spark', 'pyspark']

Python

class hpctestlib.python.numpy.numpy_ops.numpy_ops_check(*args, **kwargs)[source]

Bases: reframe.core.pipeline.RunOnlyRegressionTest

NumPy basic operations test.

NumPy is the fundamental package for scientific computing in Python. It provides a multidimensional array object, various derived objects (such as masked arrays and matrices), and an assortment of routines for fast operations on arrays, including mathematical, logical, shape manipulation, sorting, selecting, I/O, discrete Fourier transforms, basic linear algebra, basic statistical operations, random simulation and much more.

This test test performs some fundamental NumPy linear algebra operations (matrix product, SVD, Cholesky decomposition, eigendecomposition, and inverse matrix calculation) and users the execution time as a performance metric. The default assumption is that NumPy is already installed on the currest system.

time_cholesky()[source]

Time of the cholesky kernel in seconds.

time_dot()[source]

Time of the dot kernel in seconds.

time_eigendec()[source]

Time of the eigendec kernel in seconds.

time_inv()[source]

Time of the inv kernel in seconds.

time_svd()[source]

Time of the svd kernel in seconds.

Interactive Computing

class hpctestlib.interactive.jupyter.ipcmagic.ipcmagic_check(*args, **kwargs)[source]

Bases: reframe.core.pipeline.RunOnlyRegressionTest

Test ipcmagic via a distributed TensorFlow training with ipyparallel.

ipcmagic is a Python package and collection of CLI scripts for controlling clusters for Jupyter. For more information, please have a look here.

This test checks the ipcmagic performance. To do this, a single-layer neural network is trained against a noisy linear function. The parameters of the fitted linear function are returned in the end along with the resulting loss function. The default assumption is that ipcmagic is already installed on the system under test.

assert_successful_execution()[source]

Checks that the program is running on 2 different nodes (nids are different), that IPCMagic is configured and returns the correct end-of-program message (returns the slope parameter in the end).

Machine Learning

class hpctestlib.ml.tensorflow.horovod.tensorflow_cnn_check(*args, **kwargs)[source]

Bases: reframe.core.pipeline.RunOnlyRegressionTest

Run a synthetic CNN benchmark with TensorFlow2 and Horovod.

TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML powered applications. For more information, refer to https://www.tensorflow.org/.

Horovod is a distributed deep learning training framework for TensorFlow, Keras, PyTorch, and Apache MXNet. The goal of Horovod is to make distributed deep learning fast and easy to use. For more information refer to https://github.com/horovod/horovod.

This test runs the Horovod tensorflow2_synthentic_benchmark.py example, checks its sanity and extracts the GPU performance.

batch_size = 32

The size of the batch used during the learning of models.

Type

int

Default

32

benchmark_version = 'v0.21.0'

The version of Horovod to use.

Type

str

Default

'v0.21.0'

model = 'InceptionV3'

The name of the model to use for this benchmark.

Type

str

Default

'InceptionV3'

num_batches_per_iter = 5

The number of batches per iteration.

Type

int

Default

5

num_iters = 5

The number of iterations.

Type

int

Default

5

num_warmup_batches = 5

The number of warmup batches

Type

int

Default

5

throughput_iteration()[source]

The average GPU throughput per iteration in images/s.

throughput_total()[source]

The total GPU throughput of the benchmark in images/s.

class hpctestlib.ml.pytorch.horovod.pytorch_cnn_check(*args, **kwargs)[source]

Bases: reframe.core.pipeline.RunOnlyRegressionTest

Run a synthetic CNN benchmark with PyTorch and Horovod.

PyTorch is a Python package that provides tensor computation like NumPy with strong GPU acceleration and deep neural networks built on a tape-based autograd system. For more information, refer to https://pytorch.org/.

Horovod is a distributed deep learning training framework for TensorFlow, Keras, PyTorch, and Apache MXNet. The goal of Horovod is to make distributed deep learning fast and easy to use. For more information refer to https://github.com/horovod/horovod.

This test runs the Horovod pytorch_synthentic_benchmark.py example, checks its sanity and extracts the GPU performance.

batch_size = 64

The size of the batch used during the learning of models.

Type

int

Default

64

benchmark_version = 'v0.21.0'

The version of Horovod to use.

Type

str

Default

'v0.21.0'

model = 'inception_v3'

The name of the model to use for this benchmark.

Type

str

Default

'inception_v3'

num_batches_per_iter = 5

The number of batches per iteration.

Type

int

Default

5

num_iters = 5

The number of iterations.

Type

int

Default

5

num_warmup_batches = 5

The number of warmup batches

Type

int

Default

5

throughput_iteration()[source]

The average GPU throughput per iteration in images/s.

throughput_total()[source]

The total GPU throughput of the benchmark in images/s.