Sanity Functions Reference¶

Sanity functions are the functions used with the sanity_patterns and perf_patterns. The key characteristic of these functions is that they are not executed the time they are called. Instead they are evaluated at a later point by the framework (inside the check_sanity and check_performance methods). Any sanity function may be evaluated either explicitly or implicitly.

Explicit evaluation of sanity functions¶

Sanity functions may be evaluated at any time by calling evaluate() on their return value or by passing the result of a sanity function to the reframe.utility.sanity.evaluate() free function.

Implicit evaluation of sanity functions¶

Sanity functions may also be evaluated implicitly in the following situations:

When you try to get their truthy value by either explicitly or implicitly calling bool on their return value. This implies that when you include the result of a sanity function in an if statement or when you apply the and, or or not operators, this will trigger their immediate evaluation.
When you try to iterate over their result. This implies that including the result of a sanity function in a for statement will trigger its evaluation immediately.
When you try to explicitly or implicitly get its string representation by calling str on its result. This implies that printing the return value of a sanity function will automatically trigger its evaluation.

Categories of sanity functions¶

Currently ReFrame provides three broad categories of sanity functions:

Deferrable replacements of certain Python built-in functions. These functions simply delegate their execution to the actual built-ins.
Assertion functions. These functions are used to assert certain conditions and they either return True or raise reframe.core.exceptions.SanityError with a message describing the error. Users may provide their own formatted messages through the msg argument. For example, in the following call to assert_eq() the {0} and {1} placeholders will obtain the actual arguments passed to the assertion function.
```
assert_eq(a, 1, msg="{0} is not equal to {1}")
```
If in the user provided message more placeholders are used than the arguments of the assert function (except the msg argument), no argument substitution will be performed in the user message.
Utility functions. The are functions that you will normally use when defining sanity_patterns and perf_patterns. They include, but are not limited to, functions to iterate over regex matches in a file, extracting and converting values from regex matches, computing statistical information on series of data etc.

Users can write their own sanity functions as well. The page “Understanding the Mechanism of Sanity Functions” explains in detail how sanity functions work and how users can write their own.

@sanity_function¶

Sanity function decorator.

The evaluation of the decorated function will be deferred and it will become suitable for use in the sanity and performance patterns of a regression test.

@sanity_function
def myfunc(*args):
    do_sth()

reframe.utility.sanity.abs(x)[source]¶: Replacement for the built-in abs() function.

reframe.utility.sanity.all(iterable)[source]¶: Replacement for the built-in all() function.

reframe.utility.sanity.allx(iterable)[source]¶: Same as the built-in all() function, except that it returns False if iterable is empty.

New in version 2.13.

reframe.utility.sanity.and_(a, b)[source]¶

Deferrable version of the and operator.

Returns: a and b.

reframe.utility.sanity.any(iterable)[source]¶: Replacement for the built-in any() function.

reframe.utility.sanity.assert_bounded(val, lower=None, upper=None, msg=None)[source]¶

Assert that lower <= val <= upper.

Parameters

val – The value to check.
lower – The lower bound. If None, it defaults to -inf.
upper – The upper bound. If None, it defaults to inf.
msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.

Returns

True on success.

Raises

reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_eq(a, b, msg=None)[source]¶

Assert that a == b.

Parameters: msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_false(x, msg=None)[source]¶

Assert that x is evaluated to False.

Parameters: msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_found(patt, filename, msg=None, encoding='utf-8')[source]¶

Assert that regex pattern patt is found in the file filename.

Parameters

patt – The regex pattern to search. Any standard Python regular expression is accepted. The re.MULTILINE flag is set for the pattern search.
filename – The name of the file to examine or a file descriptor as in open(). Any OSError raised while processing the file will be propagated as a reframe.core.exceptions.SanityError.
msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
encoding – The name of the encoding used to decode the file.

Returns

True on success.

Raises

reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_found_s(patt, string, msg=None)[source]¶

Assert that regex pattern patt is found in the string string.

Parameters

patt – as in assert_found().
string – The string to examine.
msg – as in assert_found(). You may use {0} … {N} as placeholders for the function arguments.

Returns

True on success.

Raises

reframe.core.exceptions.SanityError – if assertion fails.

New in version 3.4.1.

reframe.utility.sanity.assert_ge(a, b, msg=None)[source]¶

Assert that a >= b.

Parameters: msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_gt(a, b, msg=None)[source]¶

Assert that a > b.

Parameters: msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_in(item, container, msg=None)[source]¶

Assert that item is in container.

Parameters: msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_le(a, b, msg=None)[source]¶

Assert that a <= b.

Parameters: msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_lt(a, b, msg=None)[source]¶

Assert that a < b.

Parameters: msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_ne(a, b, msg=None)[source]¶

Assert that a != b.

Parameters: msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_not_found(patt, filename, msg=None, encoding='utf-8')[source]¶

Assert that regex pattern patt is not found in the file filename.

This is the inverse of assert_found().

Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_not_found_s(patt, string, msg=None)[source]¶

Assert that regex pattern patt is not found in string.

This is the inverse of assert_found_s().

Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

New in version 3.4.1.

reframe.utility.sanity.assert_not_in(item, container, msg=None)[source]¶

Assert that item is not in container.

Parameters: msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.assert_reference(val, ref, lower_thres=None, upper_thres=None, msg=None)[source]¶

Assert that value val respects the reference value ref.

Parameters

val – The value to check.
ref – The reference value.
lower_thres – The lower threshold value expressed as a negative decimal fraction of the reference value. Must be in [-1, 0] for ref >= 0.0 and in [-inf, 0] for ref < 0.0. If None, no lower thresholds is applied.
upper_thres – The upper threshold value expressed as a decimal fraction of the reference value. Must be in [0, inf] for ref >= 0.0 and in [0, 1] for ref < 0.0. If None, no upper thresholds is applied.
msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.

Returns

True on success.

Raises

reframe.core.exceptions.SanityError – if assertion fails or if the lower and upper thresholds do not have appropriate values.

reframe.utility.sanity.assert_true(x, msg=None)[source]¶

Assert that x is evaluated to True.

Parameters: msg – The error message to use if the assertion fails. You may use {0} … {N} as placeholders for the function arguments.
Returns: True on success.
Raises: reframe.core.exceptions.SanityError – if assertion fails.

reframe.utility.sanity.avg(iterable)[source]¶: Return the average of all the elements of iterable.

reframe.utility.sanity.chain(*iterables)[source]¶: Replacement for the itertools.chain() function.

reframe.utility.sanity.contains(seq, key)[source]¶

Deferrable version of the in operator.

Returns: key in seq.

reframe.utility.sanity.count(iterable)[source]¶

Return the element count of iterable.

This is similar to the built-in len(), except that it can also handle any argument that supports iteration, including generators.

reframe.utility.sanity.count_uniq(iterable)[source]¶: Return the unique element count of iterable.

reframe.utility.sanity.defer(x)[source]¶: Defer the evaluation of variable x.

New in version 2.21.

reframe.utility.sanity.enumerate(iterable, start=0)[source]¶: Replacement for the built-in enumerate() function.

reframe.utility.sanity.evaluate(expr)[source]¶

Evaluate a deferred expression.

If expr is not a deferred expression, it will be returned as is.

New in version 2.21.

reframe.utility.sanity.extractall(patt, filename, tag=0, conv=None, encoding='utf-8')[source]¶

Extract all values from the capturing group tag of a matching regex patt in the file filename.

Parameters

patt –
The regex pattern to search. Any standard Python regular expression is accepted. The re.MULTILINE flag is set for the pattern search.
filename – The name of the file to examine or a file descriptor as in open().
encoding – The name of the encoding used to decode the file.
tag – The regex capturing group to be extracted. Group 0 refers always to the whole match. Since the file is processed line by line, this means that group 0 returns the whole line that was matched.
conv – A callable or iterable of callables taking a single argument and returning a new value. If not an iterable, it will be used to convert the extracted values for all the capturing groups specified in tag. Otherwise, each conversion function will be used to convert the value extracted from the corresponding capturing group in tag. If more conversion functions are supplied than the corresponding capturing groups in tag, the last conversion function will be used for the additional capturing groups.

Returns

A list of tuples of converted values extracted from the capturing groups specified in tag, if tag is an iterable. Otherwise, a list of the converted values extracted from the single capturing group specified in tag.

Raises

reframe.core.exceptions.SanityError – In case of errors.

Changed in version 3.1: Multiple regex capturing groups are now supporetd via tag and multiple conversion functions can be used in conv.

reframe.utility.sanity.extractall_s(patt, string, tag=0, conv=None)[source]¶

Extract all values from the capturing group tag of a matching regex

patt in string.

arg patt: as in extractall().

‘ :arg string: The string to examine.

arg tag: as in extractall().
arg conv: as in extractall().
returns: same as extractall().

New in version 3.4.1.

reframe.utility.sanity.extractiter(patt, filename, tag=0, conv=None, encoding='utf-8')[source]¶

Get an iterator over the values extracted from the capturing group tag of a matching regex patt in the file filename.

This function is equivalent to extractall() except that it returns a generator object, instead of a list, which you can use to iterate over the extracted values.

reframe.utility.sanity.extractiter_s(patt, string, tag=0, conv=None)[source]¶

Get an iterator over the values extracted from the capturing group tag of a matching regex patt in string.

This function is equivalent to extractall_s() except that it returns a generator object, instead of a list, which you can use to iterate over the extracted values.

New in version 3.4.1.

reframe.utility.sanity.extractsingle(patt, filename, tag=0, conv=None, item=0, encoding='utf-8')[source]¶

Extract a single value from the capturing group tag of a matching regex patt in the file filename.

This function is equivalent to extractall(patt, filename, tag, conv)[item], except that it raises a SanityError if item is out of bounds.

Parameters

patt – as in extractall().
filename – as in extractall().
encoding – as in extractall().
tag – as in extractall().
conv – as in extractall().
item – the specific element to extract.

Returns

The extracted value.

Raises

reframe.core.exceptions.SanityError – In case of errors.

reframe.utility.sanity.extractsingle_s(patt, string, tag=0, conv=None, item=0)[source]¶

Extract a single value from the capturing group tag of a matching regex patt in string.

This function is equivalent to extractall_s(patt, string, tag, conv)[item], except that it raises a SanityError if item is out of bounds.

Parameters

patt – as in extractall_s().
string – as in extractall_s().
tag – as in extractall_s().
conv – as in extractall_s().
item – the specific element to extract.

Returns

The extracted value.

Raises

reframe.core.exceptions.SanityError – In case of errors.

New in version 3.4.1.

reframe.utility.sanity.filter(function, iterable)[source]¶: Replacement for the built-in filter() function.

reframe.utility.sanity.findall(patt, filename, encoding='utf-8')[source]¶

Get all matches of regex patt in filename.

Parameters

patt –
The regex pattern to search. Any standard Python regular expression is accepted. The re.MULTILINE flag is set for the pattern search.
filename – The name of the file to examine.
encoding – The name of the encoding used to decode the file.

Returns

A list of raw regex match objects.

Raises

reframe.core.exceptions.SanityError – In case an OSError is raised while processing filename.

reframe.utility.sanity.findall_s(patt, string)[source]¶

Get all matches of regex patt in string.

Parameters

patt – as in findall()
string – The string to examine.

Returns

same as finall().

New in version 3.4.1.

reframe.utility.sanity.finditer(patt, filename, encoding='utf-8')[source]¶

Get an iterator over the matches of the regex patt in filename.

This function is equivalent to findall() except that it returns a generator object instead of a list, which you can use to iterate over the raw matches.

reframe.utility.sanity.finditer_s(patt, string)[source]¶

Get an iterator over the matches of the regex patt in string.

This function is equivalent to findall_s() except that it returns a generator object instead of a list, which you can use to iterate over the raw matches.

New in version 3.4.1.

reframe.utility.sanity.getattr(obj, attr, *args)[source]¶: Replacement for the built-in getattr() function.

reframe.utility.sanity.getitem(container, item)[source]¶

Get item from container.

container may refer to any container that can be indexed.

Raises: reframe.core.exceptions.SanityError – In case item cannot be retrieved from container.

reframe.utility.sanity.glob(pathname, *, recursive=False)[source]¶: Replacement for the glob.glob() function.

reframe.utility.sanity.hasattr(obj, name)[source]¶: Replacement for the built-in hasattr() function.

reframe.utility.sanity.iglob(pathname, recursive=False)[source]¶: Replacement for the glob.iglob() function.

reframe.utility.sanity.len(s)[source]¶: Replacement for the built-in len() function.

reframe.utility.sanity.map(function, *iterables)[source]¶: Replacement for the built-in map() function.

reframe.utility.sanity.max(*args)[source]¶: Replacement for the built-in max() function.

reframe.utility.sanity.min(*args)[source]¶: Replacement for the built-in min() function.

reframe.utility.sanity.not_(a)[source]¶

Deferrable version of the not operator.

Returns: not a.

reframe.utility.sanity.or_(a, b)[source]¶

Deferrable version of the or operator.

Returns: a or b.

reframe.utility.sanity.path_exists(path)[source]¶: Replacement for the os.path.exists() function.

New in version 3.4.

reframe.utility.sanity.path_isdir(path)[source]¶: Replacement for the os.path.isdir() function.

New in version 3.4.

reframe.utility.sanity.path_isfile(path)[source]¶: Replacement for the os.path.isfile() function.

New in version 3.4.

reframe.utility.sanity.path_islink(path)[source]¶: Replacement for the os.path.islink() function.

New in version 3.4.

reframe.utility.sanity.print(obj, *, sep=' ', end='\n', file=None, flush=False)[source]¶

Replacement for the built-in print() function.

The only difference is that this function takes a single object argument and it returns that, so that you can use it transparently inside a complex sanity expression. For example, you could write the following to print the matches returned from the extractall() function:

self.sanity_patterns = sn.assert_eq(
    sn.count(sn.print(sn.extract_all(...))), 10
)

If file is None, print() will print its arguments to the standard output. Unlike the builtin print() function, we don’t bind the file argument to sys.stdout by default. This would capture sys.stdout at the time this function is defined and would prevent it from seeing changes to sys.stdout, such as redirects, in the future.

Changed in version 3.4: This function accepts now a single object argument in contrast to the built-in print() function, which accepts multiple.

reframe.utility.sanity.reversed(seq)[source]¶: Replacement for the built-in reversed() function.

reframe.utility.sanity.round(number, *args)[source]¶: Replacement for the built-in round() function.

reframe.utility.sanity.setattr(obj, name, value)[source]¶: Replacement for the built-in setattr() function.

reframe.utility.sanity.sorted(iterable, *args)[source]¶: Replacement for the built-in sorted() function.

reframe.utility.sanity.sum(iterable, *args)[source]¶: Replacement for the built-in sum() function.

reframe.utility.sanity.zip(*iterables)[source]¶: Replacement for the built-in zip() function.