* [PATCH nft 1/3] tests/shell: export NFT_TEST_RANDOM_SEED variable for tests
2023-09-13 8:20 [PATCH nft 0/3] add NFT_TEST_RANDOM_SEED and shuffle tests Thomas Haller
@ 2023-09-13 8:20 ` Thomas Haller
2023-09-13 8:20 ` [PATCH nft 2/3] tests/shell: add "random-source.sh" helper for random-source for sort/shuf Thomas Haller
2023-09-13 8:20 ` [PATCH nft 3/3] tests/shell: add option to shuffle execution order of tests Thomas Haller
2 siblings, 0 replies; 4+ messages in thread
From: Thomas Haller @ 2023-09-13 8:20 UTC (permalink / raw)
To: NetFilter; +Cc: Thomas Haller
Let "run-tests.sh" export a NFT_TEST_RANDOM_SEED variable, set to
a decimal, random integer (in the range of 0 to 0x7FFFFFFF).
The purpose is to provide a seed to tests for randomization.
Randomizing tests is very useful to increase the coverage while not
testing all combinations (which might not be practical).
The point of NFT_TEST_RANDOM_SEED is that the user can set the
environment variable so that the same series of random events is used.
That is useful for reproducing an issue, that is known to happen with a
certain seed.
- by default, if the user leaves NFT_TEST_RANDOM_SEED unset or empty,
the script generates a number using $SRANDOM.
- if the user sets NFT_TEST_RANDOM_SEED to an integer it is taken
as is (modulo 0x80000000).
- otherwise, calculate a number by hashing the value of
$NFT_TEST_RANDOM_SEED.
Signed-off-by: Thomas Haller <thaller@redhat.com>
---
tests/shell/run-tests.sh | 46 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)
diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index f20a2bec9e9b..2acea85e7836 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -21,6 +21,31 @@ array_contains() {
return 1
}
+strtonum() {
+ local s="$1"
+ local n
+ local n2
+
+ re='^[[:space:]]*([0-9]+)[[:space:]]*$'
+ if [[ "$s" =~ $re ]] ; then
+ n="${BASH_REMATCH[1]}"
+ if [ "$(( n + 0 ))" = "$n" ] ; then
+ echo "$n"
+ return 0
+ fi
+ fi
+ re='^[[:space:]]*0x([0-9a-fA-F]+)[[:space:]]*$'
+ if [[ "$s" =~ $re ]] ; then
+ n="${BASH_REMATCH[1]}"
+ n2="$(( 16#$n + 0 ))"
+ if [ "$n2" = "$(printf '%d' "0x$n" 2>/dev/null)" ] ; then
+ echo "$n2"
+ return 0
+ fi
+ fi
+ return 1
+}
+
_msg() {
local level="$1"
shift
@@ -170,6 +195,9 @@ usage() {
echo " Setting this to \"0\" means also to perform global cleanups between tests (remove"
echo " kernel modules)."
echo " Parallel jobs requires unshare and are disabled with NFT_TEST_UNSHARE_CMD=\"\"."
+ echo " NFT_TEST_RANDOM_SEED=<SEED>: The test runner will export the environment variable NFT_TEST_RANDOM_SEED"
+ echo " set to a random number. This can be used as a stable seed for tests to randomize behavior."
+ echo " Set this to a fixed value to get reproducible behavior."
echo " TMPDIR=<PATH> : select a different base directory for the result data."
echo
echo " NFT_TEST_HAVE_<FEATURE>=*|y: Some tests requires certain features or will be skipped."
@@ -209,9 +237,26 @@ KMEMLEAK="$(bool_y "$KMEMLEAK")"
NFT_TEST_KEEP_LOGS="$(bool_y "$NFT_TEST_KEEP_LOGS")"
NFT_TEST_HAS_REALROOT="$NFT_TEST_HAS_REALROOT"
NFT_TEST_JOBS="${NFT_TEST_JOBS:-$_NFT_TEST_JOBS_DEFAULT}"
+NFT_TEST_RANDOM_SEED="$NFT_TEST_RANDOM_SEED"
NFT_TEST_SKIP_slow="$(bool_y "$NFT_TEST_SKIP_slow")"
DO_LIST_TESTS=
+if [ -z "$NFT_TEST_RANDOM_SEED" ] ; then
+ # Choose a random value.
+ n="$SRANDOM"
+else
+ # Parse as number.
+ n="$(strtonum "$NFT_TEST_RANDOM_SEED")"
+ if [ -z "$n" ] ; then
+ # If not a number, pick a hash based on the SHA-sum of the seed.
+ n="$(printf "%d" "0x$(sha256sum <<<"NFT_TEST_RANDOM_SEED:$NFT_TEST_RANDOM_SEED" | sed -n '1 { s/^\(........\).*/\1/p }')")"
+ fi
+fi
+# Limit a 31 bit decimal so tests can rely on this being in a certain
+# restricted form.
+NFT_TEST_RANDOM_SEED="$(( $n % 0x80000000 ))"
+export NFT_TEST_RANDOM_SEED
+
TESTS=()
while [ $# -gt 0 ] ; do
@@ -450,6 +495,7 @@ msg_info "conf: NFT_TEST_HAS_UNSHARED=$(printf '%q' "$NFT_TEST_HAS_UNSHARED")"
msg_info "conf: NFT_TEST_HAS_UNSHARED_MOUNT=$(printf '%q' "$NFT_TEST_HAS_UNSHARED_MOUNT")"
msg_info "conf: NFT_TEST_KEEP_LOGS=$(printf '%q' "$NFT_TEST_KEEP_LOGS")"
msg_info "conf: NFT_TEST_JOBS=$NFT_TEST_JOBS"
+msg_info "conf: NFT_TEST_RANDOM_SEED=$NFT_TEST_RANDOM_SEED"
msg_info "conf: TMPDIR=$(printf '%q' "$_TMPDIR")"
echo
for KEY in $(compgen -v | grep '^NFT_TEST_SKIP_' | sort) ; do
--
2.41.0
^ permalink raw reply related [flat|nested] 4+ messages in thread* [PATCH nft 2/3] tests/shell: add "random-source.sh" helper for random-source for sort/shuf
2023-09-13 8:20 [PATCH nft 0/3] add NFT_TEST_RANDOM_SEED and shuffle tests Thomas Haller
2023-09-13 8:20 ` [PATCH nft 1/3] tests/shell: export NFT_TEST_RANDOM_SEED variable for tests Thomas Haller
@ 2023-09-13 8:20 ` Thomas Haller
2023-09-13 8:20 ` [PATCH nft 3/3] tests/shell: add option to shuffle execution order of tests Thomas Haller
2 siblings, 0 replies; 4+ messages in thread
From: Thomas Haller @ 2023-09-13 8:20 UTC (permalink / raw)
To: NetFilter; +Cc: Thomas Haller
Commands `sort` and `shuf` have a "--random-source" argument. That's
useful for generating stable, reproducible "random" output.
However, we want to do this based on a fixed seed, while the
"--random-source" expects a stream of randomness. Add a helper script
for that.
Also, use the stable randomness for shuf in the test
"tests/shell/testcases/sets/automerge_0".
See-also: https://www.gnu.org/software/coreutils/manual/html_node/Random-sources.html#Random-sources
Signed-off-by: Thomas Haller <thaller@redhat.com>
---
tests/shell/helpers/random-source.sh | 40 ++++++++++++++++++++++++++
tests/shell/testcases/sets/automerge_0 | 2 +-
2 files changed, 41 insertions(+), 1 deletion(-)
create mode 100755 tests/shell/helpers/random-source.sh
diff --git a/tests/shell/helpers/random-source.sh b/tests/shell/helpers/random-source.sh
new file mode 100755
index 000000000000..91a8248bea1f
--- /dev/null
+++ b/tests/shell/helpers/random-source.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# Commands like `sort` and `shuf` have a "--random-source" argument, for
+# generating a stable, reproducible output. However, they require an input
+# that provides sufficiently many bytes (depending on the input).
+#
+# This script generates a stream that can be used like
+#
+# shuf --random-source=<($0 "$seed")
+
+seed=""
+for a; do
+ seed="$seed${#a}:$a\n"
+done
+
+if command -v openssl &>/dev/null ; then
+ # We have openssl. Use it.
+ # https://www.gnu.org/software/coreutils/manual/html_node/Random-sources.html#Random-sources
+ #
+ # Note that we don't care that different installations/architectures generate the
+ # same output.
+ openssl enc -aes-256-ctr -pass "pass:$seed" -nosalt </dev/zero 2>/dev/null
+else
+ # Hack something. It's much slower.
+ idx=0
+ while : ; do
+ idx="$((idx++))"
+ seed="$(sha256sum <<<"$idx.$seed")"
+ echo ">>>$seed" >> a
+ seed="${seed%% *}"
+ LANG=C awk -v s="$seed" 'BEGIN{
+ for (i=1; i <= length(s); i+=2) {
+ xchar = substr(s, i, 2);
+ decnum = strtonum("0x"xchar);
+ printf("%c", decnum);
+ }
+ }' || break
+ done
+fi
+exit 0
diff --git a/tests/shell/testcases/sets/automerge_0 b/tests/shell/testcases/sets/automerge_0
index 170c38651de0..1dbac0b7cdbd 100755
--- a/tests/shell/testcases/sets/automerge_0
+++ b/tests/shell/testcases/sets/automerge_0
@@ -44,7 +44,7 @@ do
done
tmpfile3=$(mktemp)
-shuf $tmpfile2 > $tmpfile3
+shuf "$tmpfile2" --random-source=<("$NFT_TEST_BASEDIR/helpers/random-source.sh" "automerge-shuf-tmpfile2" "$NFT_TEST_RANDOM_SEED") > "$tmpfile3"
i=0
cat $tmpfile3 | while read line && [ $i -lt 10 ]
do
--
2.41.0
^ permalink raw reply related [flat|nested] 4+ messages in thread* [PATCH nft 3/3] tests/shell: add option to shuffle execution order of tests
2023-09-13 8:20 [PATCH nft 0/3] add NFT_TEST_RANDOM_SEED and shuffle tests Thomas Haller
2023-09-13 8:20 ` [PATCH nft 1/3] tests/shell: export NFT_TEST_RANDOM_SEED variable for tests Thomas Haller
2023-09-13 8:20 ` [PATCH nft 2/3] tests/shell: add "random-source.sh" helper for random-source for sort/shuf Thomas Haller
@ 2023-09-13 8:20 ` Thomas Haller
2 siblings, 0 replies; 4+ messages in thread
From: Thomas Haller @ 2023-09-13 8:20 UTC (permalink / raw)
To: NetFilter; +Cc: Thomas Haller
The user can set NFT_TEST_SHUFFLE_TESTS=y|n to have the tests shuffled
randomly. The purpose of shuffling is to find tests that depend on each
other, or would break when run in unexpected order.
If unspecified, by default tests are shuffled if no tests are selected
on the command line.
Signed-off-by: Thomas Haller <thaller@redhat.com>
---
tests/shell/run-tests.sh | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 2acea85e7836..a66f32abfc5a 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -143,6 +143,7 @@ usage() {
echo " -U|--no-unshare : Sets NFT_TEST_UNSHARE_CMD=\"\"."
echo " -k|--keep-logs : Sets NFT_TEST_KEEP_LOGS=y."
echo " -s|--sequential : Sets NFT_TEST_JOBS=0, which also enables global cleanups."
+ echo " Also sets NFT_TEST_SHUFFLE_TESTS=n if left unspecified."
echo " -Q|--quick : Sets NFT_TEST_SKIP_slow=y."
echo " -- : Separate options from tests."
echo " [TESTS...] : Other options are treated as test names,"
@@ -198,6 +199,9 @@ usage() {
echo " NFT_TEST_RANDOM_SEED=<SEED>: The test runner will export the environment variable NFT_TEST_RANDOM_SEED"
echo " set to a random number. This can be used as a stable seed for tests to randomize behavior."
echo " Set this to a fixed value to get reproducible behavior."
+ echo " NFT_TEST_SHUFFLE_TESTS=*|n|y: control whether to randomly shuffle the order of tests. By default, if"
+ echo " tests are specified explicitly, they are not shuffled while they are shuffled when"
+ echo " all tests are run. The shuffling is based on NFT_TEST_RANDOM_SEED."
echo " TMPDIR=<PATH> : select a different base directory for the result data."
echo
echo " NFT_TEST_HAVE_<FEATURE>=*|y: Some tests requires certain features or will be skipped."
@@ -238,6 +242,7 @@ NFT_TEST_KEEP_LOGS="$(bool_y "$NFT_TEST_KEEP_LOGS")"
NFT_TEST_HAS_REALROOT="$NFT_TEST_HAS_REALROOT"
NFT_TEST_JOBS="${NFT_TEST_JOBS:-$_NFT_TEST_JOBS_DEFAULT}"
NFT_TEST_RANDOM_SEED="$NFT_TEST_RANDOM_SEED"
+NFT_TEST_SHUFFLE_TESTS="$NFT_TEST_SHUFFLE_TESTS"
NFT_TEST_SKIP_slow="$(bool_y "$NFT_TEST_SKIP_slow")"
DO_LIST_TESTS=
@@ -293,6 +298,9 @@ while [ $# -gt 0 ] ; do
;;
-s|--sequential)
NFT_TEST_JOBS=0
+ if [ -z "$NFT_TEST_SHUFFLE_TESTS" ] ; then
+ NFT_TEST_SHUFFLE_TESTS=n
+ fi
;;
-Q|--quick)
NFT_TEST_SKIP_slow=y
@@ -314,6 +322,9 @@ find_tests() {
if [ "${#TESTS[@]}" -eq 0 ] ; then
TESTS=( $(find_tests "$NFT_TEST_BASEDIR/testcases/") )
test "${#TESTS[@]}" -gt 0 || msg_error "Could not find tests"
+ if [ -z "$NFT_TEST_SHUFFLE_TESTS" ] ; then
+ NFT_TEST_SHUFFLE_TESTS=y
+ fi
fi
TESTSOLD=( "${TESTS[@]}" )
@@ -328,6 +339,8 @@ for t in "${TESTSOLD[@]}" ; do
fi
done
+NFT_TEST_SHUFFLE_TESTS="$(bool_y "$NFT_TEST_SHUFFLE_TESTS")"
+
if [ "$DO_LIST_TESTS" = y ] ; then
printf '%s\n' "${TESTS[@]}"
exit 0
@@ -496,6 +509,7 @@ msg_info "conf: NFT_TEST_HAS_UNSHARED_MOUNT=$(printf '%q' "$NFT_TEST_HAS_UNSHARE
msg_info "conf: NFT_TEST_KEEP_LOGS=$(printf '%q' "$NFT_TEST_KEEP_LOGS")"
msg_info "conf: NFT_TEST_JOBS=$NFT_TEST_JOBS"
msg_info "conf: NFT_TEST_RANDOM_SEED=$NFT_TEST_RANDOM_SEED"
+msg_info "conf: NFT_TEST_SHUFFLE_TESTS=$NFT_TEST_SHUFFLE_TESTS"
msg_info "conf: TMPDIR=$(printf '%q' "$_TMPDIR")"
echo
for KEY in $(compgen -v | grep '^NFT_TEST_SKIP_' | sort) ; do
@@ -709,6 +723,10 @@ job_wait()
done
}
+if [ "$NFT_TEST_SHUFFLE_TESTS" = y ] ; then
+ TESTS=( $(printf '%s\n' "${TESTS[@]}" | shuf --random-source=<("$NFT_TEST_BASEDIR/helpers/random-source.sh" "nft-test-shuffle-tests" "$NFT_TEST_RANDOM_SEED") ) )
+fi
+
TESTIDX=0
JOBS_N_RUNNING=0
for testfile in "${TESTS[@]}" ; do
--
2.41.0
^ permalink raw reply related [flat|nested] 4+ messages in thread