Sanity Functions Reference¶
Sanity functions are the functions used with the sanity_patterns
and perf_patterns
.
The key characteristic of these functions is that they are not executed the time they are called.
Instead they are evaluated at a later point by the framework (inside the check_sanity
and check_performance
methods).
Any sanity function may be evaluated either explicitly or implicitly.
Explicit evaluation of sanity functions¶
Sanity functions may be evaluated at any time by calling evaluate()
on their return value or by passing the result of a sanity function to the reframe.utility.sanity.evaluate()
free function.
Implicit evaluation of sanity functions¶
Sanity functions may also be evaluated implicitly in the following situations:
When you try to get their truthy value by either explicitly or implicitly calling
bool
on their return value. This implies that when you include the result of a sanity function in anif
statement or when you apply theand
,or
ornot
operators, this will trigger their immediate evaluation.When you try to iterate over their result. This implies that including the result of a sanity function in a
for
statement will trigger its evaluation immediately.When you try to explicitly or implicitly get its string representation by calling
str
on its result. This implies that printing the return value of a sanity function will automatically trigger its evaluation.
Categories of sanity functions¶
Currently ReFrame provides three broad categories of sanity functions:
Deferrable replacements of certain Python built-in functions. These functions simply delegate their execution to the actual built-ins.
Assertion functions. These functions are used to assert certain conditions and they either return
True
or raisereframe.core.exceptions.SanityError
with a message describing the error. Users may provide their own formatted messages through themsg
argument. For example, in the following call toassert_eq()
the{0}
and{1}
placeholders will obtain the actual arguments passed to the assertion function.assert_eq(a, 1, msg="{0} is not equal to {1}")
If in the user provided message more placeholders are used than the arguments of the assert function (except the
msg
argument), no argument substitution will be performed in the user message.Utility functions. The are functions that you will normally use when defining
sanity_patterns
andperf_patterns
. They include, but are not limited to, functions to iterate over regex matches in a file, extracting and converting values from regex matches, computing statistical information on series of data etc.
Users can write their own sanity functions as well. The page “Understanding the Mechanism of Sanity Functions” explains in detail how sanity functions work and how users can write their own.
-
@
sanity_function
¶ Sanity function decorator.
The evaluation of the decorated function will be deferred and it will become suitable for use in the sanity and performance patterns of a regression test.
@sanity_function def myfunc(*args): do_sth()
-
reframe.utility.sanity.
allx
(iterable)[source]¶ Same as the built-in
all()
function, except that it returnsFalse
ifiterable
is empty.New in version 2.13.
-
reframe.utility.sanity.
assert_bounded
(val, lower=None, upper=None, msg=None)[source]¶ Assert that
lower <= val <= upper
.- Parameters
val – The value to check.
lower – The lower bound. If
None
, it defaults to-inf
.upper – The upper bound. If
None
, it defaults toinf
.
- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_eq
(a, b, msg=None)[source]¶ Assert that
a == b
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_false
(x, msg=None)[source]¶ Assert that
x
is evaluated toFalse
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_found
(patt, filename, msg=None, encoding='utf-8')[source]¶ Assert that regex pattern
patt
is found in the filefilename
.- Parameters
patt – The regex pattern to search. Any standard Python regular expression is accepted. The re.MULTILINE flag is set for the pattern search.
filename – The name of the file to examine. Any
OSError
raised while processing the file will be propagated as areframe.core.exceptions.SanityError
.encoding – The name of the encoding used to decode the file.
- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_ge
(a, b, msg=None)[source]¶ Assert that
a >= b
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_gt
(a, b, msg=None)[source]¶ Assert that
a > b
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_in
(item, container, msg=None)[source]¶ Assert that
item
is incontainer
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_le
(a, b, msg=None)[source]¶ Assert that
a <= b
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_lt
(a, b, msg=None)[source]¶ Assert that
a < b
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_ne
(a, b, msg=None)[source]¶ Assert that
a != b
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_not_found
(patt, filename, msg=None, encoding='utf-8')[source]¶ Assert that regex pattern
patt
is not found in the filefilename
.This is the inverse of
assert_found()
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_not_in
(item, container, msg=None)[source]¶ Assert that
item
is not incontainer
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
assert_reference
(val, ref, lower_thres=None, upper_thres=None, msg=None)[source]¶ Assert that value
val
respects the reference valueref
.- Parameters
val – The value to check.
ref – The reference value.
lower_thres – The lower threshold value expressed as a negative decimal fraction of the reference value. Must be in [-1, 0] for ref >= 0.0 and in [-inf, 0] for ref < 0.0. If
None
, no lower thresholds is applied.upper_thres – The upper threshold value expressed as a decimal fraction of the reference value. Must be in [0, inf] for ref >= 0.0 and in [0, 1] for ref < 0.0. If
None
, no upper thresholds is applied.
- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails or if the lower and upper thresholds do not have appropriate values.
-
reframe.utility.sanity.
assert_true
(x, msg=None)[source]¶ Assert that
x
is evaluated toTrue
.- Returns
True
on success.- Raises
reframe.core.exceptions.SanityError – if assertion fails.
-
reframe.utility.sanity.
chain
(*iterables)[source]¶ Replacement for the
itertools.chain()
function.
-
reframe.utility.sanity.
contains
(seq, key)[source]¶ Deferrable version of the
in
operator.- Returns
key in seq
.
-
reframe.utility.sanity.
count
(iterable)[source]¶ Return the element count of
iterable
.This is similar to the built-in
len()
, except that it can also handle any argument that supports iteration, including generators.
-
reframe.utility.sanity.
enumerate
(iterable, start=0)[source]¶ Replacement for the built-in
enumerate()
function.
-
reframe.utility.sanity.
evaluate
(expr)[source]¶ Evaluate a deferred expression.
If
expr
is not a deferred expression, it will be returned as is.New in version 2.21.
-
reframe.utility.sanity.
extractall
(patt, filename, tag=0, conv=None, encoding='utf-8')[source]¶ Extract all values from the capturing group
tag
of a matching regexpatt
in the filefilename
.- Parameters
patt –
The regex pattern to search. Any standard Python regular expression is accepted. The re.MULTILINE flag is set for the pattern search.
filename – The name of the file to examine.
encoding – The name of the encoding used to decode the file.
tag – The regex capturing group to be extracted. Group
0
refers always to the whole match. Since the file is processed line by line, this means that group0
returns the whole line that was matched.conv – A callable or iterable of callables taking a single argument and returning a new value. If not an iterable, it will be used to convert the extracted values for all the capturing groups specified in
tag
. Otherwise, each conversion function will be used to convert the value extracted from the corresponding capturing group intag
. If more conversion functions are supplied than the corresponding capturing groups intag
, the last conversion function will be used for the additional capturing groups.
- Returns
A list of tuples of converted values extracted from the capturing groups specified in
tag
, iftag
is an iterable. Otherwise, a list of the converted values extracted from the single capturing group specified intag
.- Raises
reframe.core.exceptions.SanityError – In case of errors.
Changed in version 3.1: Multiple regex capturing groups are now supporetd via
tag
and multiple conversion functions can be used inconv
.
-
reframe.utility.sanity.
extractiter
(patt, filename, tag=0, conv=None, encoding='utf-8')[source]¶ Get an iterator over the values extracted from the capturing group
tag
of a matching regexpatt
in the filefilename
.This function is equivalent to
extractall()
except that it returns a generator object, instead of a list, which you can use to iterate over the extracted values.
-
reframe.utility.sanity.
extractsingle
(patt, filename, tag=0, conv=None, item=0, encoding='utf-8')[source]¶ Extract a single value from the capturing group
tag
of a matching regexpatt
in the filefilename
.This function is equivalent to
extractall(patt, filename, tag, conv)[item]
, except that it raises aSanityError
ifitem
is out of bounds.- Parameters
patt – as in
extractall()
.filename – as in
extractall()
.encoding – as in
extractall()
.tag – as in
extractall()
.conv – as in
extractall()
.item – the specific element to extract.
- Returns
The extracted value.
- Raises
reframe.core.exceptions.SanityError – In case of errors.
-
reframe.utility.sanity.
filter
(function, iterable)[source]¶ Replacement for the built-in
filter()
function.
-
reframe.utility.sanity.
findall
(patt, filename, encoding='utf-8')[source]¶ Get all matches of regex
patt
infilename
.- Parameters
patt –
The regex pattern to search. Any standard Python regular expression is accepted. The re.MULTILINE flag is set for the pattern search.
filename – The name of the file to examine.
encoding – The name of the encoding used to decode the file.
- Returns
A list of raw regex match objects.
- Raises
reframe.core.exceptions.SanityError – In case an
OSError
is raised while processingfilename
.
-
reframe.utility.sanity.
finditer
(patt, filename, encoding='utf-8')[source]¶ Get an iterator over the matches of the regex
patt
infilename
.This function is equivalent to
findall()
except that it returns a generator object instead of a list, which you can use to iterate over the raw matches.
-
reframe.utility.sanity.
getattr
(obj, attr, *args)[source]¶ Replacement for the built-in
getattr()
function.
-
reframe.utility.sanity.
getitem
(container, item)[source]¶ Get
item
fromcontainer
.container
may refer to any container that can be indexed.- Raises
reframe.core.exceptions.SanityError – In case
item
cannot be retrieved fromcontainer
.
-
reframe.utility.sanity.
glob
(pathname, *, recursive=False)[source]¶ Replacement for the
glob.glob()
function.
-
reframe.utility.sanity.
iglob
(pathname, recursive=False)[source]¶ Replacement for the
glob.iglob()
function.
-
reframe.utility.sanity.
map
(function, *iterables)[source]¶ Replacement for the built-in
map()
function.
-
reframe.utility.sanity.
print
(*objects, sep=' ', end='\n', file=None, flush=False)[source]¶ Replacement for the built-in
print()
function.The only difference is that this function returns the
objects
, so that you can use it transparently inside a complex sanity expression. For example, you could write the following to print the matches returned from theextractall()
function:self.sanity_patterns = sn.assert_eq( sn.count(sn.print(sn.extract_all(...))), 10 )
If
file
is None,print()
will print its arguments to the standard output. Unlike the builtinprint()
function, we don’t bind thefile
argument tosys.stdout
by default. This would capturesys.stdout
at the time this function is defined and would prevent it from seeing changes tosys.stdout
, such as redirects, in the future.
-
reframe.utility.sanity.
reversed
(seq)[source]¶ Replacement for the built-in
reversed()
function.
-
reframe.utility.sanity.
setattr
(obj, name, value)[source]¶ Replacement for the built-in
setattr()
function.