[PATCH nft 0/3] add NFT_TEST_RANDOM_SEED and shuffle tests

[PATCH nft 0/3] add NFT_TEST_RANDOM_SEED and shuffle tests

[Thomas Haller]

- let "run-tests.sh" export a NFT_TEST_RANDOM_SEED, which tests may use
  for generating stable (reproducible) sequences in randomized tests.
- add NFT_TEST_SHUFFLE_TESTS, to randomize the order in which tests are
  run. The purpose is to find issues where tests interfere with each
  other. It's enabled by default, if no tests are explicitly specified
  on the command line.

Thomas Haller (3):
  tests/shell: export NFT_TEST_RANDOM_SEED variable for tests
  tests/shell: add "random-source.sh" helper for random-source for
    sort/shuf
  tests/shell: add option to shuffle execution order of tests

 tests/shell/helpers/random-source.sh   | 40 ++++++++++++++++
 tests/shell/run-tests.sh               | 64 ++++++++++++++++++++++++++
 tests/shell/testcases/sets/automerge_0 |  2 +-
 3 files changed, 105 insertions(+), 1 deletion(-)
 create mode 100755 tests/shell/helpers/random-source.sh

-- 
2.41.0

[PATCH nft 1/3] tests/shell: export NFT_TEST_RANDOM_SEED variable for tests

[Thomas Haller]

Let "run-tests.sh" export a NFT_TEST_RANDOM_SEED variable, set to
a decimal, random integer (in the range of 0 to 0x7FFFFFFF).

The purpose is to provide a seed to tests for randomization.
Randomizing tests is very useful to increase the coverage while not
testing all combinations (which might not be practical).

The point of NFT_TEST_RANDOM_SEED is that the user can set the
environment variable so that the same series of random events is used.
That is useful for reproducing an issue, that is known to happen with a
certain seed.

- by default, if the user leaves NFT_TEST_RANDOM_SEED unset or empty,
  the script generates a number using $SRANDOM.
- if the user sets NFT_TEST_RANDOM_SEED to an integer it is taken
  as is (modulo 0x80000000).
- otherwise, calculate a number by hashing the value of
  $NFT_TEST_RANDOM_SEED.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 46 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index f20a2bec9e9b..2acea85e7836 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -21,6 +21,31 @@ array_contains() {
 	return 1
 }
 
+strtonum() {
+	local s="$1"
+	local n
+	local n2
+
+	re='^[[:space:]]*([0-9]+)[[:space:]]*$'
+	if [[ "$s" =~ $re ]] ; then
+		n="${BASH_REMATCH[1]}"
+		if [ "$(( n + 0 ))" = "$n" ] ; then
+			echo "$n"
+			return 0
+		fi
+	fi
+	re='^[[:space:]]*0x([0-9a-fA-F]+)[[:space:]]*$'
+	if [[ "$s" =~ $re ]] ; then
+		n="${BASH_REMATCH[1]}"
+		n2="$(( 16#$n + 0 ))"
+		if [ "$n2" = "$(printf '%d' "0x$n" 2>/dev/null)" ] ; then
+			echo "$n2"
+			return 0
+		fi
+	fi
+	return 1
+}
+
 _msg() {
 	local level="$1"
 	shift
@@ -170,6 +195,9 @@ usage() {
 	echo "                 Setting this to \"0\" means also to perform global cleanups between tests (remove"
 	echo "                 kernel modules)."
 	echo "                 Parallel jobs requires unshare and are disabled with NFT_TEST_UNSHARE_CMD=\"\"."
+	echo " NFT_TEST_RANDOM_SEED=<SEED>: The test runner will export the environment variable NFT_TEST_RANDOM_SEED"
+	echo "                 set to a random number. This can be used as a stable seed for tests to randomize behavior."
+	echo "                 Set this to a fixed value to get reproducible behavior."
 	echo " TMPDIR=<PATH> : select a different base directory for the result data."
 	echo
 	echo " NFT_TEST_HAVE_<FEATURE>=*|y: Some tests requires certain features or will be skipped."
@@ -209,9 +237,26 @@ KMEMLEAK="$(bool_y "$KMEMLEAK")"
 NFT_TEST_KEEP_LOGS="$(bool_y "$NFT_TEST_KEEP_LOGS")"
 NFT_TEST_HAS_REALROOT="$NFT_TEST_HAS_REALROOT"
 NFT_TEST_JOBS="${NFT_TEST_JOBS:-$_NFT_TEST_JOBS_DEFAULT}"
+NFT_TEST_RANDOM_SEED="$NFT_TEST_RANDOM_SEED"
 NFT_TEST_SKIP_slow="$(bool_y "$NFT_TEST_SKIP_slow")"
 DO_LIST_TESTS=
 
+if [ -z "$NFT_TEST_RANDOM_SEED" ] ; then
+	# Choose a random value.
+	n="$SRANDOM"
+else
+	# Parse as number.
+	n="$(strtonum "$NFT_TEST_RANDOM_SEED")"
+	if [ -z "$n" ] ; then
+		# If not a number, pick a hash based on the SHA-sum of the seed.
+		n="$(printf "%d" "0x$(sha256sum <<<"NFT_TEST_RANDOM_SEED:$NFT_TEST_RANDOM_SEED" | sed -n '1 { s/^\(........\).*/\1/p }')")"
+	fi
+fi
+# Limit a 31 bit decimal so tests can rely on this being in a certain
+# restricted form.
+NFT_TEST_RANDOM_SEED="$(( $n % 0x80000000 ))"
+export NFT_TEST_RANDOM_SEED
+
 TESTS=()
 
 while [ $# -gt 0 ] ; do
@@ -450,6 +495,7 @@ msg_info "conf: NFT_TEST_HAS_UNSHARED=$(printf '%q' "$NFT_TEST_HAS_UNSHARED")"
 msg_info "conf: NFT_TEST_HAS_UNSHARED_MOUNT=$(printf '%q' "$NFT_TEST_HAS_UNSHARED_MOUNT")"
 msg_info "conf: NFT_TEST_KEEP_LOGS=$(printf '%q' "$NFT_TEST_KEEP_LOGS")"
 msg_info "conf: NFT_TEST_JOBS=$NFT_TEST_JOBS"
+msg_info "conf: NFT_TEST_RANDOM_SEED=$NFT_TEST_RANDOM_SEED"
 msg_info "conf: TMPDIR=$(printf '%q' "$_TMPDIR")"
 echo
 for KEY in $(compgen -v | grep '^NFT_TEST_SKIP_' | sort) ; do
-- 
2.41.0

[PATCH nft 2/3] tests/shell: add "random-source.sh" helper for random-source for sort/shuf

[Thomas Haller]

Commands `sort` and `shuf` have a "--random-source" argument. That's
useful for generating stable, reproducible "random" output.

However, we want to do this based on a fixed seed, while the
"--random-source" expects a stream of randomness. Add a helper script
for that.

Also, use the stable randomness for shuf in the test
"tests/shell/testcases/sets/automerge_0".

See-also: https://www.gnu.org/software/coreutils/manual/html_node/Random-sources.html#Random-sources

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/random-source.sh   | 40 ++++++++++++++++++++++++++
 tests/shell/testcases/sets/automerge_0 |  2 +-
 2 files changed, 41 insertions(+), 1 deletion(-)
 create mode 100755 tests/shell/helpers/random-source.sh

diff --git a/tests/shell/helpers/random-source.sh b/tests/shell/helpers/random-source.sh
new file mode 100755
index 000000000000..91a8248bea1f
--- /dev/null
+++ b/tests/shell/helpers/random-source.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# Commands like `sort` and `shuf` have a "--random-source" argument, for
+# generating a stable, reproducible output. However, they require an input
+# that provides sufficiently many bytes (depending on the input).
+#
+# This script generates a stream that can be used like
+#
+#     shuf --random-source=<($0 "$seed")
+
+seed=""
+for a; do
+	seed="$seed${#a}:$a\n"
+done
+
+if command -v openssl &>/dev/null ; then
+	# We have openssl. Use it.
+	# https://www.gnu.org/software/coreutils/manual/html_node/Random-sources.html#Random-sources
+	#
+	# Note that we don't care that different installations/architectures generate the
+	# same output.
+	openssl enc -aes-256-ctr -pass "pass:$seed" -nosalt </dev/zero 2>/dev/null
+else
+	# Hack something. It's much slower.
+	idx=0
+	while : ; do
+		idx="$((idx++))"
+		seed="$(sha256sum <<<"$idx.$seed")"
+		echo ">>>$seed" >> a
+		seed="${seed%% *}"
+		LANG=C awk -v s="$seed" 'BEGIN{
+			for (i=1; i <= length(s); i+=2) {
+				xchar = substr(s, i, 2);
+				decnum = strtonum("0x"xchar);
+				printf("%c", decnum);
+			}
+		}' || break
+	done
+fi
+exit 0
diff --git a/tests/shell/testcases/sets/automerge_0 b/tests/shell/testcases/sets/automerge_0
index 170c38651de0..1dbac0b7cdbd 100755
--- a/tests/shell/testcases/sets/automerge_0
+++ b/tests/shell/testcases/sets/automerge_0
@@ -44,7 +44,7 @@ do
 done
 
 tmpfile3=$(mktemp)
-shuf $tmpfile2 > $tmpfile3
+shuf "$tmpfile2" --random-source=<("$NFT_TEST_BASEDIR/helpers/random-source.sh" "automerge-shuf-tmpfile2" "$NFT_TEST_RANDOM_SEED") > "$tmpfile3"
 i=0
 cat $tmpfile3 | while read line && [ $i -lt 10 ]
 do
-- 
2.41.0

[PATCH nft 3/3] tests/shell: add option to shuffle execution order of tests

[Thomas Haller]

The user can set NFT_TEST_SHUFFLE_TESTS=y|n to have the tests shuffled
randomly. The purpose of shuffling is to find tests that depend on each
other, or would break when run in unexpected order.

If unspecified, by default tests are shuffled if no tests are selected
on the command line.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 2acea85e7836..a66f32abfc5a 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -143,6 +143,7 @@ usage() {
 	echo " -U|--no-unshare : Sets NFT_TEST_UNSHARE_CMD=\"\"."
 	echo " -k|--keep-logs  : Sets NFT_TEST_KEEP_LOGS=y."
 	echo " -s|--sequential : Sets NFT_TEST_JOBS=0, which also enables global cleanups."
+	echo "                   Also sets NFT_TEST_SHUFFLE_TESTS=n if left unspecified."
 	echo " -Q|--quick      : Sets NFT_TEST_SKIP_slow=y."
 	echo " --              : Separate options from tests."
 	echo " [TESTS...]      : Other options are treated as test names,"
@@ -198,6 +199,9 @@ usage() {
 	echo " NFT_TEST_RANDOM_SEED=<SEED>: The test runner will export the environment variable NFT_TEST_RANDOM_SEED"
 	echo "                 set to a random number. This can be used as a stable seed for tests to randomize behavior."
 	echo "                 Set this to a fixed value to get reproducible behavior."
+	echo " NFT_TEST_SHUFFLE_TESTS=*|n|y: control whether to randomly shuffle the order of tests. By default, if"
+	echo "                 tests are specified explicitly, they are not shuffled while they are shuffled when"
+	echo "                 all tests are run. The shuffling is based on NFT_TEST_RANDOM_SEED."
 	echo " TMPDIR=<PATH> : select a different base directory for the result data."
 	echo
 	echo " NFT_TEST_HAVE_<FEATURE>=*|y: Some tests requires certain features or will be skipped."
@@ -238,6 +242,7 @@ NFT_TEST_KEEP_LOGS="$(bool_y "$NFT_TEST_KEEP_LOGS")"
 NFT_TEST_HAS_REALROOT="$NFT_TEST_HAS_REALROOT"
 NFT_TEST_JOBS="${NFT_TEST_JOBS:-$_NFT_TEST_JOBS_DEFAULT}"
 NFT_TEST_RANDOM_SEED="$NFT_TEST_RANDOM_SEED"
+NFT_TEST_SHUFFLE_TESTS="$NFT_TEST_SHUFFLE_TESTS"
 NFT_TEST_SKIP_slow="$(bool_y "$NFT_TEST_SKIP_slow")"
 DO_LIST_TESTS=
 
@@ -293,6 +298,9 @@ while [ $# -gt 0 ] ; do
 			;;
 		-s|--sequential)
 			NFT_TEST_JOBS=0
+			if [ -z "$NFT_TEST_SHUFFLE_TESTS" ] ; then
+				NFT_TEST_SHUFFLE_TESTS=n
+			fi
 			;;
 		-Q|--quick)
 			NFT_TEST_SKIP_slow=y
@@ -314,6 +322,9 @@ find_tests() {
 if [ "${#TESTS[@]}" -eq 0 ] ; then
 	TESTS=( $(find_tests "$NFT_TEST_BASEDIR/testcases/") )
 	test "${#TESTS[@]}" -gt 0 || msg_error "Could not find tests"
+	if [ -z "$NFT_TEST_SHUFFLE_TESTS" ] ; then
+		NFT_TEST_SHUFFLE_TESTS=y
+	fi
 fi
 
 TESTSOLD=( "${TESTS[@]}" )
@@ -328,6 +339,8 @@ for t in "${TESTSOLD[@]}" ; do
 	fi
 done
 
+NFT_TEST_SHUFFLE_TESTS="$(bool_y "$NFT_TEST_SHUFFLE_TESTS")"
+
 if [ "$DO_LIST_TESTS" = y ] ; then
 	printf '%s\n' "${TESTS[@]}"
 	exit 0
@@ -496,6 +509,7 @@ msg_info "conf: NFT_TEST_HAS_UNSHARED_MOUNT=$(printf '%q' "$NFT_TEST_HAS_UNSHARE
 msg_info "conf: NFT_TEST_KEEP_LOGS=$(printf '%q' "$NFT_TEST_KEEP_LOGS")"
 msg_info "conf: NFT_TEST_JOBS=$NFT_TEST_JOBS"
 msg_info "conf: NFT_TEST_RANDOM_SEED=$NFT_TEST_RANDOM_SEED"
+msg_info "conf: NFT_TEST_SHUFFLE_TESTS=$NFT_TEST_SHUFFLE_TESTS"
 msg_info "conf: TMPDIR=$(printf '%q' "$_TMPDIR")"
 echo
 for KEY in $(compgen -v | grep '^NFT_TEST_SKIP_' | sort) ; do
@@ -709,6 +723,10 @@ job_wait()
 	done
 }
 
+if [ "$NFT_TEST_SHUFFLE_TESTS" = y ] ; then
+	TESTS=( $(printf '%s\n' "${TESTS[@]}" | shuf --random-source=<("$NFT_TEST_BASEDIR/helpers/random-source.sh" "nft-test-shuffle-tests" "$NFT_TEST_RANDOM_SEED") ) )
+fi
+
 TESTIDX=0
 JOBS_N_RUNNING=0
 for testfile in "${TESTS[@]}" ; do
-- 
2.41.0