[PATCH nft v4 00/17] tests/shell: allow running tests as

[PATCH nft v4 00/17] tests/shell: allow running tests as

[Thomas Haller]

Changes to v3:

- add "-j" option to run tests in parallel.
- with real root, don't use `unshare -U`. That breaks tests that require
  real root. Even if the user originally is real-root, after unshare, the
  process can no longer increase the socket buffer beyond wmem_max. For
  rootful, we must not unshare the user namespace. And no longer do that by
  default.
- unshare the mount namespace, this allows to bindmount a different /var/run/netns.
  That's useful with rootful for isolation and necessary with rootless
  to have writable /var/run/netns.
- rework the way how unshare is configurable. Basically, you don't need
  to care, but if you wish, you can override with NFT_TEST_UNSHARE_CMD.
- tests that are known to not work in rootless are now automatically
  skipped. On my system, all tests that pass with rootful also pass 
  or are skipped with rootless (I have some tests that fail also with 
  root). 
- support NFT_TEST_HAS_SOCKET_LIMITS=n environment to get tests that
  would be skipped in rootless to run (and pass, if wmem_max is high
  enough).
- many minor improvements.

Changes to v2:

- large rework of all patches.
- we still try to unshare as much as we can, but gracefully fallback to
  only unshare the netns. What we don't do anymore, is accept failure to unshare
  altogether and proceed silently. If you want that, use NFT_TEST_NO_UNSHARE=y or
  NFT_TEST_UNSHARE_CMD=cmd.
- compared to v2, fix `nft flush` to be called inside the target netns.
  It's now done by "test-wrapper.sh"
- add mode to run jobs in parallel.
- move test-specific functionality from "run-tests.sh to "test-wrapper.sh".
- collect test results in a temporary directory for later inspection.

Changes to v1:

- new patch: rework the parsing of command line options
- new patch: add a "--list-tests" option to show the found tests
- call "unshare" for each test individually.
- drop NFT_TEST_ROOTLESS environment variable. You no longer have to
  opt-in to run rootless. However, if any tests fail and we ran
  rootless, then an info is printed at the end.
- the environment variables NFT_TEST_HAVE_REALROOT and
  NFT_TEST_NO_UNSHARE can still be set to configure the script.
  Those are now also configurable via command line options.
  Usually you would not have to set them.

Thomas Haller (17):
  tests/shell: rework command line parsing in "run-tests.sh"
  tests/shell: rework finding tests and add "--list-tests" option
  tests/shell: check test names before start and support directories
  tests/shell: export NFT_TEST_BASEDIR and NFT_TEST_TMPDIR for tests
  tests/shell: normalize boolean configuration in environment variables
  tests/shell: print test configuration
  tests/shell: run each test in separate namespace and allow rootless
  tests/shell: interpret an exit code of 77 from scripts as "skipped"
  tests/shell: support --keep-logs option (NFT_TEST_KEEP_LOGS=y) to
    preserve test output
  tests/shell: move the dump diff handling inside "test-wrapper.sh"
  tests/shell: rework printing of test results
  tests/shell: move taint check to "test-wrapper.sh"
  tests/shell: support running tests in parallel
  tests/shell: bind mount private /var/run/netns in test container
  tests/shell: skip test in rootless that hit socket buffer size limit
  tests/shell: record the test duration for investigation
  tests/shell: set TMPDIR for tests in "test-wrapper.sh"

 tests/shell/helpers/test-wrapper.sh           | 111 ++++
 tests/shell/run-tests.sh                      | 593 ++++++++++++++----
 tests/shell/testcases/nft-f/0011manydefines_0 |  16 +
 .../testcases/sets/0011add_many_elements_0    |  15 +
 .../sets/0012add_delete_many_elements_0       |  14 +
 .../sets/0013add_delete_many_elements_0       |  14 +
 tests/shell/testcases/sets/automerge_0        |  24 +-
 tests/shell/testcases/transactions/30s-stress |   9 +
 8 files changed, 671 insertions(+), 125 deletions(-)
 create mode 100755 tests/shell/helpers/test-wrapper.sh

-- 
2.41.0

[PATCH nft v4 01/17] tests/shell: rework command line parsing in "run-tests.sh"

[Thomas Haller]

Parse the arguments in a loop, so that their order does not matter.
Also, soon more command line arguments will be added, and this way of
parsing seems more maintainable and flexible.

Currently this is still after the is-root check and after unshare. That
will be addressed later.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 95 +++++++++++++++++++++++++++-------------
 1 file changed, 65 insertions(+), 30 deletions(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index b66ef4fa4d1f..ae8c6d934dcf 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -1,10 +1,5 @@
 #!/bin/bash
 
-# Configuration
-TESTDIR="./$(dirname $0)/testcases"
-SRC_NFT="$(dirname $0)/../../src/nft"
-DIFF=$(which diff)
-
 msg_error() {
 	echo "E: $1 ..." >&2
 	exit 1
@@ -18,6 +13,29 @@ msg_info() {
 	echo "I: $1"
 }
 
+usage() {
+	echo " $0 [OPTIONS]"
+	echo
+	echo "OPTIONS:"
+	echo " -h|--help : print usage"
+	echo " -v        : sets VERBOSE=y"
+	echo " -g        : sets DUMPGEN=y"
+	echo " -V        : sets VALGRIND=y"
+	echo " -K        : sets KMEMLEAK=y"
+	echo
+	echo "ENVIRONMENT VARIABLES:"
+	echo " NFT=<PATH>   : Path to nft executable"
+	echo " VERBOSE=*|y  : Enable verbose output"
+	echo " DUMPGEN=*|y  : Regenerate dump files"
+	echo " VALGRIND=*|y : Run \$NFT in valgrind"
+	echo " KMEMLEAK=*|y : Check for kernel memleaks"
+}
+
+# Configuration
+TESTDIR="./$(dirname $0)/testcases"
+SRC_NFT="$(dirname $0)/../../src/nft"
+DIFF=$(which diff)
+
 if [ "$(id -u)" != "0" ] ; then
 	msg_error "this requires root!"
 fi
@@ -31,6 +49,48 @@ if [ "${1}" != "run" ]; then
 fi
 shift
 
+VERBOSE="$VERBOSE"
+DUMPGEN="$DUMPGEN"
+VALGRIND="$VALGRIND"
+KMEMLEAK="$KMEMLEAK"
+
+TESTS=()
+
+while [ $# -gt 0 ] ; do
+	A="$1"
+	shift
+	case "$A" in
+		-v)
+			VERBOSE=y
+			;;
+		-g)
+			DUMPGEN=y
+			;;
+		-V)
+			VALGRIND=y
+			;;
+		-K)
+			KMEMLEAK=y
+			;;
+		-h|--help)
+			usage
+			exit 0
+			;;
+		--)
+			TESTS+=( "$@" )
+			shift $#
+			;;
+		*)
+			# Any unrecognized option is treated as a test name, and also
+			# enable verbose tests.
+			TESTS+=( "$A" )
+			VERBOSE=y
+			;;
+	esac
+done
+
+SINGLE="${TESTS[*]}"
+
 [ -z "$NFT" ] && NFT=$SRC_NFT
 ${NFT} > /dev/null 2>&1
 ret=$?
@@ -59,31 +119,6 @@ if [ ! -x "$DIFF" ] ; then
 	DIFF=true
 fi
 
-if [ "$1" == "-v" ] ; then
-	VERBOSE=y
-	shift
-fi
-
-if [ "$1" == "-g" ] ; then
-	DUMPGEN=y
-	shift
-fi
-
-if [ "$1" == "-V" ] ; then
-	VALGRIND=y
-	shift
-fi
-
-if [ "$1" == "-K" ]; then
-	KMEMLEAK=y
-	shift
-fi
-
-for arg in "$@"; do
-	SINGLE+=" $arg"
-	VERBOSE=y
-done
-
 kernel_cleanup() {
 	$NFT flush ruleset
 	$MODPROBE -raq \
-- 
2.41.0

[PATCH nft v4 02/17] tests/shell: rework finding tests and add "--list-tests" option

[Thomas Haller]

Cleanup finding the test files. Also add a "--list-tests" option to see
which tests are found and would run.

Also get rid of the FIND="$(which find)" detection. Which system doesn't
have a working find? Also, we can just fail when we try to use find, and
don't need a check first.

This is still  after "unshare", which will be addressed next.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 53 ++++++++++++++++++++--------------------
 1 file changed, 26 insertions(+), 27 deletions(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index ae8c6d934dcf..184dd3f38be5 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -17,11 +17,12 @@ usage() {
 	echo " $0 [OPTIONS]"
 	echo
 	echo "OPTIONS:"
-	echo " -h|--help : print usage"
-	echo " -v        : sets VERBOSE=y"
-	echo " -g        : sets DUMPGEN=y"
-	echo " -V        : sets VALGRIND=y"
-	echo " -K        : sets KMEMLEAK=y"
+	echo " -h|--help       : print usage"
+	echo " -L|--list-tests : list test names and quit"
+	echo " -v              : sets VERBOSE=y"
+	echo " -g              : sets DUMPGEN=y"
+	echo " -V              : sets VALGRIND=y"
+	echo " -K              : sets KMEMLEAK=y"
 	echo
 	echo "ENVIRONMENT VARIABLES:"
 	echo " NFT=<PATH>   : Path to nft executable"
@@ -32,8 +33,8 @@ usage() {
 }
 
 # Configuration
-TESTDIR="./$(dirname $0)/testcases"
-SRC_NFT="$(dirname $0)/../../src/nft"
+BASEDIR="$(dirname "$0")"
+SRC_NFT="$BASEDIR/../../src/nft"
 DIFF=$(which diff)
 
 if [ "$(id -u)" != "0" ] ; then
@@ -53,6 +54,7 @@ VERBOSE="$VERBOSE"
 DUMPGEN="$DUMPGEN"
 VALGRIND="$VALGRIND"
 KMEMLEAK="$KMEMLEAK"
+DO_LIST_TESTS=
 
 TESTS=()
 
@@ -76,6 +78,9 @@ while [ $# -gt 0 ] ; do
 			usage
 			exit 0
 			;;
+		-L|--list-tests)
+			DO_LIST_TESTS=y
+			;;
 		--)
 			TESTS+=( "$@" )
 			shift $#
@@ -89,7 +94,19 @@ while [ $# -gt 0 ] ; do
 	esac
 done
 
-SINGLE="${TESTS[*]}"
+find_tests() {
+	find "$1" -type f -executable | sort
+}
+
+if [ "${#TESTS[@]}" -eq 0 ] ; then
+	TESTS=( $(find_tests "$BASEDIR/testcases/") )
+	test "${#TESTS[@]}" -gt 0 || msg_error "Could not find tests"
+fi
+
+if [ "$DO_LIST_TESTS" = y ] ; then
+	printf '%s\n' "${TESTS[@]}"
+	exit 0
+fi
 
 [ -z "$NFT" ] && NFT=$SRC_NFT
 ${NFT} > /dev/null 2>&1
@@ -100,15 +117,6 @@ else
 	msg_info "using nft command: ${NFT}"
 fi
 
-if [ ! -d "$TESTDIR" ] ; then
-	msg_error "missing testdir $TESTDIR"
-fi
-
-FIND="$(which find)"
-if [ ! -x "$FIND" ] ; then
-	msg_error "no find binary found"
-fi
-
 MODPROBE="$(which modprobe)"
 if [ ! -x "$MODPROBE" ] ; then
 	msg_error "no modprobe binary found"
@@ -143,14 +151,6 @@ kernel_cleanup() {
 	nft_xfrm
 }
 
-find_tests() {
-	if [ ! -z "$SINGLE" ] ; then
-		echo $SINGLE
-		return
-	fi
-	${FIND} ${TESTDIR} -type f -executable | sort
-}
-
 printscript() { # (cmd, tmpd)
 	cat <<EOF
 #!/bin/bash
@@ -248,8 +248,7 @@ check_kmemleak()
 
 check_taint
 
-for testfile in $(find_tests)
-do
+for testfile in "${TESTS[@]}" ; do
 	read taint < /proc/sys/kernel/tainted
 	kernel_cleanup
 
-- 
2.41.0

[PATCH nft v4 03/17] tests/shell: check test names before start and support directories

[Thomas Haller]

Check for valid test names early. That's useful because we treat any
unrecognized options as test names. We should detect a mistake early.

While at it, also support specifying directory names instead of
executable files.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 184dd3f38be5..0a2598f10bed 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -103,6 +103,18 @@ if [ "${#TESTS[@]}" -eq 0 ] ; then
 	test "${#TESTS[@]}" -gt 0 || msg_error "Could not find tests"
 fi
 
+TESTSOLD=( "${TESTS[@]}" )
+TESTS=()
+for t in "${TESTSOLD[@]}" ; do
+	if [ -f "$t" -a -x "$t" ] ; then
+		TESTS+=( "$t" )
+	elif [ -d "$t" ] ; then
+		TESTS+=( $(find_tests "$t") )
+	else
+		msg_error "Unknown test \"$t\""
+	fi
+done
+
 if [ "$DO_LIST_TESTS" = y ] ; then
 	printf '%s\n' "${TESTS[@]}"
 	exit 0
-- 
2.41.0

[PATCH nft v4 04/17] tests/shell: export NFT_TEST_BASEDIR and NFT_TEST_TMPDIR for tests

[Thomas Haller]

Let the test wrapper prepare and export two environment variables for
the test:

- "$NFT_TEST_BASEDIR" is just the top directory where the test scripts
  lie.

- "$NFT_TEST_TMPDIR" is a `mktemp` directory created by "run-tests.sh"
  and removed at the end. Tests may use that to leave data there.
  This directory will be used for various things, like the "nft" wrapper
  in valgrind mode, the results of the tests and possibly as cache for
  feature detection.

The "$NFT_TEST_TMPDIR" was already used before with the "VALGRIND=y"
mode. It's only renamed and got an extended purpose.

Also drop the unnecessary first detection of "$DIFF" and the "$SRC_NFT"
variable.

Also, note that the mktemp creates the temporary directory under /tmp.
Which is commonly a tempfs. The user can override that by exporting
TMPDIR.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 57 +++++++++++++++++++++++++---------------
 1 file changed, 36 insertions(+), 21 deletions(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 0a2598f10bed..03b4cd4f5805 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -25,17 +25,18 @@ usage() {
 	echo " -K              : sets KMEMLEAK=y"
 	echo
 	echo "ENVIRONMENT VARIABLES:"
-	echo " NFT=<PATH>   : Path to nft executable"
-	echo " VERBOSE=*|y  : Enable verbose output"
-	echo " DUMPGEN=*|y  : Regenerate dump files"
-	echo " VALGRIND=*|y : Run \$NFT in valgrind"
-	echo " KMEMLEAK=*|y : Check for kernel memleaks"
+	echo " NFT=<PATH>    : Path to nft executable"
+	echo " VERBOSE=*|y   : Enable verbose output"
+	echo " DUMPGEN=*|y   : Regenerate dump files"
+	echo " VALGRIND=*|y  : Run \$NFT in valgrind"
+	echo " KMEMLEAK=*|y  : Check for kernel memleaks"
+	echo " TMPDIR=<PATH> : select a different base directory for the result data"
 }
 
-# Configuration
-BASEDIR="$(dirname "$0")"
-SRC_NFT="$BASEDIR/../../src/nft"
-DIFF=$(which diff)
+NFT_TEST_BASEDIR="$(dirname "$0")"
+
+# Export the base directory. It may be used by tests.
+export NFT_TEST_BASEDIR
 
 if [ "$(id -u)" != "0" ] ; then
 	msg_error "this requires root!"
@@ -99,7 +100,7 @@ find_tests() {
 }
 
 if [ "${#TESTS[@]}" -eq 0 ] ; then
-	TESTS=( $(find_tests "$BASEDIR/testcases/") )
+	TESTS=( $(find_tests "$NFT_TEST_BASEDIR/testcases/") )
 	test "${#TESTS[@]}" -gt 0 || msg_error "Could not find tests"
 fi
 
@@ -120,7 +121,7 @@ if [ "$DO_LIST_TESTS" = y ] ; then
 	exit 0
 fi
 
-[ -z "$NFT" ] && NFT=$SRC_NFT
+[ -z "$NFT" ] && NFT="$NFT_TEST_BASEDIR/../../src/nft"
 ${NFT} > /dev/null 2>&1
 ret=$?
 if [ ${ret} -eq 126 ] || [ ${ret} -eq 127 ]; then
@@ -139,6 +140,26 @@ if [ ! -x "$DIFF" ] ; then
 	DIFF=true
 fi
 
+cleanup_on_exit() {
+	test -z "$NFT_TEST_TMPDIR" || rm -rf "$NFT_TEST_TMPDIR"
+}
+trap cleanup_on_exit EXIT
+
+_TMPDIR="${TMPDIR:-/tmp}"
+
+NFT_TEST_TMPDIR="$(mktemp --tmpdir="$_TMPDIR" -d "nft-test.$(date '+%Y%m%d-%H%M%S.%3N').XXXXXX")" ||
+	msg_error "Failure to create temp directory in \"$_TMPDIR\""
+chmod 755 "$NFT_TEST_TMPDIR"
+
+NFT_TEST_LATEST="$_TMPDIR/nft-test.latest.$USER"
+
+ln -snf "$NFT_TEST_TMPDIR" "$NFT_TEST_LATEST"
+
+# export the tmp directory for tests. They may use it, but create
+# distinct files! It will be deleted on EXIT.
+export NFT_TEST_TMPDIR
+
+
 kernel_cleanup() {
 	$NFT flush ruleset
 	$MODPROBE -raq \
@@ -193,16 +214,10 @@ EOF
 }
 
 if [ "$VALGRIND" == "y" ]; then
-	tmpd=$(mktemp -d)
-	chmod 755 $tmpd
-
-	msg_info "writing valgrind logs to $tmpd"
-
-	printscript "$NFT" "$tmpd" >${tmpd}/nft
-	trap "rm ${tmpd}/nft" EXIT
-	chmod a+x ${tmpd}/nft
-
-	NFT="${tmpd}/nft"
+	msg_info "writing valgrind logs to $NFT_TEST_TMPDIR"
+	printscript "$NFT" "$NFT_TEST_TMPDIR" > "$NFT_TEST_TMPDIR/nft"
+	chmod a+x "$NFT_TEST_TMPDIR/nft"
+	NFT="$NFT_TEST_TMPDIR/nft"
 fi
 
 echo ""
-- 
2.41.0

[PATCH nft v4 05/17] tests/shell: normalize boolean configuration in environment variables

[Thomas Haller]

Previously, we would honor "y" as opt-in, and all other values meant
false.

- accept alternatives to "y", like "1" or "true".

- normalize the value, to either be "y" or "n".

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 03b4cd4f5805..84975a65243f 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -13,6 +13,24 @@ msg_info() {
 	echo "I: $1"
 }
 
+bool_y() {
+	case "$1" in
+		y|Y|yes|Yes|YES|1|true)
+			printf y
+			;;
+		'')
+			if [ $# -ge 2 ] ; then
+				printf "%s" "$2"
+			else
+				printf n
+			fi
+			;;
+		*)
+			printf n
+			;;
+	esac
+}
+
 usage() {
 	echo " $0 [OPTIONS]"
 	echo
@@ -51,10 +69,10 @@ if [ "${1}" != "run" ]; then
 fi
 shift
 
-VERBOSE="$VERBOSE"
-DUMPGEN="$DUMPGEN"
-VALGRIND="$VALGRIND"
-KMEMLEAK="$KMEMLEAK"
+VERBOSE="$(bool_y "$VERBOSE")"
+DUMPGEN="$(bool_y "$DUMPGEN")"
+VALGRIND="$(bool_y "$VALGRIND")"
+KMEMLEAK="$(bool_y "$KMEMLEAK")"
 DO_LIST_TESTS=
 
 TESTS=()
-- 
2.41.0

[PATCH nft v4 06/17] tests/shell: print test configuration

[Thomas Haller]

As the script can be configured via environment variables or command
line option, it's useful to show the environment variables that we
received or set during the test setup.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 84975a65243f..82694e9c69d3 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -139,14 +139,20 @@ if [ "$DO_LIST_TESTS" = y ] ; then
 	exit 0
 fi
 
+_TMPDIR="${TMPDIR:-/tmp}"
+
 [ -z "$NFT" ] && NFT="$NFT_TEST_BASEDIR/../../src/nft"
 ${NFT} > /dev/null 2>&1
 ret=$?
 if [ ${ret} -eq 126 ] || [ ${ret} -eq 127 ]; then
-	msg_error "cannot execute nft command: ${NFT}"
-else
-	msg_info "using nft command: ${NFT}"
+	msg_error "cannot execute nft command: $NFT"
 fi
+msg_info "conf: NFT=$(printf '%q' "$NFT")"
+msg_info "conf: VERBOSE=$(printf '%q' "$VERBOSE")"
+msg_info "conf: DUMPGEN=$(printf '%q' "$DUMPGEN")"
+msg_info "conf: VALGRIND=$(printf '%q' "$VALGRIND")"
+msg_info "conf: KMEMLEAK=$(printf '%q' "$KMEMLEAK")"
+msg_info "conf: TMPDIR=$(printf '%q' "$_TMPDIR")"
 
 MODPROBE="$(which modprobe)"
 if [ ! -x "$MODPROBE" ] ; then
@@ -163,8 +169,6 @@ cleanup_on_exit() {
 }
 trap cleanup_on_exit EXIT
 
-_TMPDIR="${TMPDIR:-/tmp}"
-
 NFT_TEST_TMPDIR="$(mktemp --tmpdir="$_TMPDIR" -d "nft-test.$(date '+%Y%m%d-%H%M%S.%3N').XXXXXX")" ||
 	msg_error "Failure to create temp directory in \"$_TMPDIR\""
 chmod 755 "$NFT_TEST_TMPDIR"
-- 
2.41.0

[PATCH nft v4 07/17] tests/shell: run each test in separate namespace and allow rootless

[Thomas Haller]

Don't unshare the entire shell script. Instead, call unshare each test
separately. That means, all tests use now a different sandbox and will
also allow (with further changes) to run them in parallel.

Also, allow to run rootless/unprivileged.

The script first tries to run a separate PID+USER+NET namespace. If that
fails, it downgrades to USER+NET. If that fails, it downgrades to a
separate NET namespace. If unshare still fails, the script fails
entirely. That differs from before, where the script would proceed
without sandboxing. The script will now always require that unsharing
works, unless the user opts-out.

If the user cannot unshare, they have two options. Either the
NFT_TEST_NO_UNSHARE=y variable (-U/--no-unshare), which disables unshare
done by the script. This allows the user to run the script as before.
Alternatively, the user can set NFT_TEST_UNSHARE_CMD for the command to
unshare, including setting it to empty for no unshare.

If we are able to create a separate USER namespace, then this mode
allows to run the test as rootless/unprivileged. We no longer require
[ `id -u` = 0 ]. Some tests may not work as rootless. For example, the
socket buffers is limited by /proc/sys/net/core/{wmem_max,rmem_max}
which real-root can override, but rootless tests cannot. Such tests
should check for [ "$NFT_TEST_HAS_REALROOT" != y ] and skip gracefully.

Usually, the user doesn't need to tell the script whether they have
real-root. The script will autodetect it via [ `id -u` = 0 ]. But that
won't work when run inside a rootless container already. In that case,
the user would want to tell the script that there is no real-root. They
can do so via the -R/--without-root option or NFT_TEST_HAS_REALROOT=n.

If tests wish, the can know whether they run inside "unshare"
environment by checking for [ "$NFT_TEST_HAS_UNSHARED" = y ].

When setting NFT_TEST_UNSHARE_CMD to override the unshare command, users
may want to also set NFT_TEST_HAS_UNSHARED= and NFT_TEST_HAS_REALROOT=
correctly.

As we run each test in a separate unshare environment, we need a wrapper
"tests/shell/helpers/test-wrapper.sh" around the test, which executes
inside the tested environment. Also, each test gets its own temp
directory prepared in NFT_TEST_TESTTMPDIR. This is also the place, where
test artifacts and results will be collected.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/test-wrapper.sh |  23 +++++
 tests/shell/run-tests.sh            | 147 +++++++++++++++++++++++-----
 2 files changed, 147 insertions(+), 23 deletions(-)
 create mode 100755 tests/shell/helpers/test-wrapper.sh

diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
new file mode 100755
index 000000000000..f811b44aab0d
--- /dev/null
+++ b/tests/shell/helpers/test-wrapper.sh
@@ -0,0 +1,23 @@
+#!/bin/bash -e
+
+# This wrapper wraps the invocation of the test. It is called by run-tests.sh,
+# and already in the unshared namespace.
+#
+# For some printf debugging, you can also patch this file.
+
+TEST="$1"
+
+printf '%s\n' "$TEST" > "$NFT_TEST_TESTTMPDIR/name"
+
+rc_test=0
+"$TEST" |& tee "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$?
+
+if [ "$rc_test" -eq 0 ] ; then
+	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-ok"
+else
+	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-failed"
+fi
+
+$NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after"
+
+exit "$rc_test"
diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 82694e9c69d3..66da48ab893e 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -35,12 +35,14 @@ usage() {
 	echo " $0 [OPTIONS]"
 	echo
 	echo "OPTIONS:"
-	echo " -h|--help       : print usage"
-	echo " -L|--list-tests : list test names and quit"
-	echo " -v              : sets VERBOSE=y"
-	echo " -g              : sets DUMPGEN=y"
-	echo " -V              : sets VALGRIND=y"
-	echo " -K              : sets KMEMLEAK=y"
+	echo " -h|--help        : print usage"
+	echo " -L|--list-tests  : list test names and quit"
+	echo " -v               : sets VERBOSE=y"
+	echo " -g               : sets DUMPGEN=y"
+	echo " -V               : sets VALGRIND=y"
+	echo " -K               : sets KMEMLEAK=y"
+	echo " -R|--without-realroot : sets NFT_TEST_HAS_REALROOT=n"
+	echo " -U|--no-unshare  : sets NFT_TEST_UNSHARE_CMD="
 	echo
 	echo "ENVIRONMENT VARIABLES:"
 	echo " NFT=<PATH>    : Path to nft executable"
@@ -48,6 +50,25 @@ usage() {
 	echo " DUMPGEN=*|y   : Regenerate dump files"
 	echo " VALGRIND=*|y  : Run \$NFT in valgrind"
 	echo " KMEMLEAK=*|y  : Check for kernel memleaks"
+	echo " NFT_TEST_HAS_REALROOT=*|y : To indicate whether the test has real root permissions."
+	echo "                 Usually, you don't need this and it gets autodetected."
+	echo "                 You might want to set it, if you know better than the"
+	echo "                 \`id -u\` check, whether the user is root in the main namespace."
+	echo "                 Note that without real root, certain tests may not work,"
+	echo "                 e.g. due to limited /proc/sys/net/core/{wmem_max,rmem_max}."
+	echo "                 Checks that cannot pass in such environment should check for"
+	echo "                 [ \"\$NFT_TEST_HAS_REALROOT\" != y ] and skip gracefully."
+	echo " NFT_TEST_UNSHARE_CMD=cmd : when set, this is the command line for an unshare"
+	echo "                 command, which is used to sandbox each test invocation. By"
+	echo "                 setting it to empty, no unsharing is done."
+	echo "                 By default it is unset, in which case it's autodetected as"
+	echo "                 \`unshare -f -p\` (for root) or as \`unshare -f -p --mount-proc -U --map-root-user -n\`"
+	echo "                 for non-root."
+	echo "                 When setting this, you may also want to set NFT_TEST_HAS_UNSHARED="
+	echo "                 and NFT_TEST_HAS_REALROOT= accordingly."
+	echo " NFT_TEST_HAS_UNSHARED=*|y : To indicate to the test whether the test run will be unshared."
+	echo "                 Test may consider this."
+	echo "                 This is only honored when \$NFT_TEST_UNSHARE_CMD= is set. Otherwise it's detected."
 	echo " TMPDIR=<PATH> : select a different base directory for the result data"
 }
 
@@ -56,23 +77,12 @@ NFT_TEST_BASEDIR="$(dirname "$0")"
 # Export the base directory. It may be used by tests.
 export NFT_TEST_BASEDIR
 
-if [ "$(id -u)" != "0" ] ; then
-	msg_error "this requires root!"
-fi
-
-if [ "${1}" != "run" ]; then
-	if unshare -f -n true; then
-		unshare -n "${0}" run $@
-		exit $?
-	fi
-	msg_warn "cannot run in own namespace, connectivity might break"
-fi
-shift
-
 VERBOSE="$(bool_y "$VERBOSE")"
 DUMPGEN="$(bool_y "$DUMPGEN")"
 VALGRIND="$(bool_y "$VALGRIND")"
 KMEMLEAK="$(bool_y "$KMEMLEAK")"
+NFT_TEST_HAS_REALROOT="$NFT_TEST_HAS_REALROOT"
+NFT_TEST_NO_UNSHARE="$NFT_TEST_NO_UNSHARE"
 DO_LIST_TESTS=
 
 TESTS=()
@@ -100,6 +110,12 @@ while [ $# -gt 0 ] ; do
 		-L|--list-tests)
 			DO_LIST_TESTS=y
 			;;
+		-R|--without-realroot)
+			NFT_TEST_HAS_REALROOT=n
+			;;
+		-U|--no-unshare)
+			NFT_TEST_NO_UNSHARE=y
+			;;
 		--)
 			TESTS+=( "$@" )
 			shift $#
@@ -141,6 +157,68 @@ fi
 
 _TMPDIR="${TMPDIR:-/tmp}"
 
+if [ "$NFT_TEST_HAS_REALROOT" = "" ] ; then
+	# The caller didn't set NFT_TEST_HAS_REALROOT and didn't specify
+	# -R/--without-root option. Autodetect it based on `id -u`.
+	export NFT_TEST_HAS_REALROOT="$(test "$(id -u)" = "0" && echo y || echo n)"
+else
+	NFT_TEST_HAS_REALROOT="$(bool_y "$NFT_TEST_HAS_REALROOT")"
+fi
+export NFT_TEST_HAS_REALROOT
+
+detect_unshare() {
+	if ! $1 true &>/dev/null ; then
+		return 1
+	fi
+	NFT_TEST_UNSHARE_CMD="$1"
+	return 0
+}
+
+if [ -n "${NFT_TEST_UNSHARE_CMD+x}" ] ; then
+	# User overrides the unshare command. We ignore NFT_TEST_NO_UNSHARE.
+	# It may be set to empty, to not unshare. The user should export NFT_TEST_HAS_UNSHARED=
+	# accordingly.
+	if ! detect_unshare "$NFT_TEST_UNSHARE_CMD" ; then
+		msg_error "Cannot unshare via NFT_TEST_UNSHARE_CMD=$(printf '%q' "$NFT_TEST_UNSHARE_CMD")"
+	fi
+	if [ -z "${NFT_TEST_HAS_UNSHARED+x}" ] ; then
+		# Autodetect NFT_TEST_HAS_UNSHARED based one whether
+		# $NFT_TEST_UNSHARE_CMD is set.
+		if [ -n "$NFT_TEST_UNSHARE_CMD" ] ; then
+			NFT_TEST_HAS_UNSHARED="y"
+		else
+			NFT_TEST_HAS_UNSHARED="n"
+		fi
+	else
+		NFT_TEST_HAS_UNSHARED="$(bool_y "$NFT_TEST_HAS_UNSHARED")"
+	fi
+else
+	NFT_TEST_UNSHARE_CMD=""
+	NFT_TEST_HAS_UNSHARED="n"
+	if [ "$NFT_TEST_NO_UNSHARE" != y ] ; then
+		if [ "$NFT_TEST_HAS_REALROOT" = y ] ; then
+			# We appear to have real root. So try to unshare
+			# without a separate USERNS. CLONE_NEWUSER will break
+			# tests that are limited by
+			# /proc/sys/net/core/{wmem_max,rmem_max}. With real
+			# root, we want to test that.
+			detect_unshare "unshare -f -n -m" ||
+				detect_unshare "unshare -f -n" ||
+				detect_unshare "unshare -f -p -m --mount-proc -U --map-root-user -n" ||
+				detect_unshare "unshare -f -U --map-root-user -n"
+		else
+			detect_unshare "unshare -f -p -m --mount-proc -U --map-root-user -n" ||
+				detect_unshare "unshare -f -U --map-root-user -n"
+		fi
+		if [ -z "$NFT_TEST_UNSHARE_CMD" ] ; then
+			msg_error "Unshare does not work. Run as root with -U/--no-unshare/NFT_TEST_NO_UNSHARE=y or set NFT_TEST_UNSHARE_CMD"
+		fi
+		NFT_TEST_HAS_UNSHARED=y
+	fi
+fi
+# If tests wish, they can know whether they are unshared via this variable.
+export NFT_TEST_HAS_UNSHARED
+
 [ -z "$NFT" ] && NFT="$NFT_TEST_BASEDIR/../../src/nft"
 ${NFT} > /dev/null 2>&1
 ret=$?
@@ -152,6 +230,9 @@ msg_info "conf: VERBOSE=$(printf '%q' "$VERBOSE")"
 msg_info "conf: DUMPGEN=$(printf '%q' "$DUMPGEN")"
 msg_info "conf: VALGRIND=$(printf '%q' "$VALGRIND")"
 msg_info "conf: KMEMLEAK=$(printf '%q' "$KMEMLEAK")"
+msg_info "conf: NFT_TEST_HAS_REALROOT=$(printf '%q' "$NFT_TEST_HAS_REALROOT")"
+msg_info "conf: NFT_TEST_UNSHARE_CMD=$(printf '%q' "$NFT_TEST_UNSHARE_CMD")"
+msg_info "conf: NFT_TEST_HAS_UNSHARED=$(printf '%q' "$NFT_TEST_HAS_UNSHARED")"
 msg_info "conf: TMPDIR=$(printf '%q' "$_TMPDIR")"
 
 MODPROBE="$(which modprobe)"
@@ -181,9 +262,14 @@ ln -snf "$NFT_TEST_TMPDIR" "$NFT_TEST_LATEST"
 # distinct files! It will be deleted on EXIT.
 export NFT_TEST_TMPDIR
 
+echo
+msg_info "info: NFT_TEST_BASEDIR=$(printf '%q' "$NFT_TEST_BASEDIR")"
+msg_info "info: NFT_TEST_TMPDIR=$(printf '%q' "$NFT_TEST_TMPDIR")"
 
 kernel_cleanup() {
-	$NFT flush ruleset
+	if [ "$NFT_TEST_HAS_UNSHARED" != y ] ; then
+		$NFT flush ruleset
+	fi
 	$MODPROBE -raq \
 	nft_reject_ipv4 nft_reject_bridge nft_reject_ipv6 nft_reject \
 	nft_redir_ipv4 nft_redir_ipv6 nft_redir \
@@ -297,22 +383,32 @@ check_kmemleak()
 
 check_taint
 
+TESTIDX=0
 for testfile in "${TESTS[@]}" ; do
 	read taint < /proc/sys/kernel/tainted
 	kernel_cleanup
 
+	((TESTIDX++))
+
+	# We also create and export a test-specific temporary directory.
+	NFT_TEST_TESTTMPDIR="$NFT_TEST_TMPDIR/test-${testfile//\//-}.$TESTIDX"
+	mkdir "$NFT_TEST_TESTTMPDIR"
+	chmod 755 "$NFT_TEST_TESTTMPDIR"
+	export NFT_TEST_TESTTMPDIR
+
 	msg_info "[EXECUTING]	$testfile"
-	test_output=$(NFT="$NFT" DIFF=$DIFF ${testfile} 2>&1)
+	test_output="$(NFT="$NFT" DIFF=$DIFF $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile" 2>&1)"
 	rc_got=$?
 	echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line
 
 	if [ "$rc_got" -eq 0 ] ; then
+		# FIXME: this should move inside test-wrapper.sh.
 		# check nft dump only for positive tests
 		dumppath="$(dirname ${testfile})/dumps"
 		dumpfile="${dumppath}/$(basename ${testfile}).nft"
 		rc_spec=0
 		if [ "$rc_got" -eq 0 ] && [ -f ${dumpfile} ]; then
-			test_output=$(${DIFF} -u ${dumpfile} <($NFT list ruleset) 2>&1)
+			test_output=$(${DIFF} -u ${dumpfile} <(cat "$NFT_TEST_TESTTMPDIR/ruleset-after") 2>&1)
 			rc_spec=$?
 		fi
 
@@ -323,7 +419,7 @@ for testfile in "${TESTS[@]}" ; do
 
 			if [ "$DUMPGEN" == "y" ] && [ "$rc_got" == 0 ] && [ ! -f "${dumpfile}" ]; then
 				mkdir -p "${dumppath}"
-				$NFT list ruleset > "${dumpfile}"
+				cat "$NFT_TEST_TESTTMPDIR/ruleset-after" > "${dumpfile}"
 			fi
 		else
 			((failed++))
@@ -361,4 +457,9 @@ check_kmemleak_force
 msg_info "results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))"
 
 kernel_cleanup
+
+if [ "$failed" -gt 0 -a "$NFT_TEST_HAS_REALROOT" != y ] ; then
+	msg_info "test was not running as real root"
+fi
+
 [ "$failed" -eq 0 ]
-- 
2.41.0

[PATCH nft v4 08/17] tests/shell: interpret an exit code of 77 from scripts as "skipped"

[Thomas Haller]

Allow scripts to indicate that a test could not run by exiting 77.

"77" is chosen as exit code from automake's testsuites ([1]). Compare to
git-bisect which chooses 125 to indicate skipped.

[1] https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/test-wrapper.sh |  2 ++
 tests/shell/run-tests.sh            | 11 ++++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
index f811b44aab0d..0cf37f408003 100755
--- a/tests/shell/helpers/test-wrapper.sh
+++ b/tests/shell/helpers/test-wrapper.sh
@@ -14,6 +14,8 @@ rc_test=0
 
 if [ "$rc_test" -eq 0 ] ; then
 	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-ok"
+elif [ "$rc_test" -eq 77 ] ; then
+	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-skipped"
 else
 	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-failed"
 fi
diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 66da48ab893e..394073024f5f 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -330,6 +330,7 @@ fi
 
 echo ""
 ok=0
+skipped=0
 failed=0
 taint=0
 
@@ -430,6 +431,14 @@ for testfile in "${TESTS[@]}" ; do
 				msg_warn "[DUMP FAIL]	$testfile"
 			fi
 		fi
+	elif [ "$rc_got" -eq 77 ] ; then
+		((skipped++))
+		if [ "$VERBOSE" == "y" ] ; then
+			msg_warn "[SKIPPED]	$testfile"
+			[ ! -z "$test_output" ] && echo "$test_output"
+		else
+			msg_warn "[SKIPPED]	$testfile"
+		fi
 	else
 		((failed++))
 		if [ "$VERBOSE" == "y" ] ; then
@@ -454,7 +463,7 @@ echo ""
 kmemleak_found=0
 check_kmemleak_force
 
-msg_info "results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))"
+msg_info "results: [OK] $ok [SKIPPED] $skipped [FAILED] $failed [TOTAL] $((ok+skipped+failed))"
 
 kernel_cleanup
 
-- 
2.41.0

[PATCH nft v4 09/17] tests/shell: support --keep-logs option (NFT_TEST_KEEP_LOGS=y) to preserve test output

[Thomas Haller]

The test output is now all collected in the temporary directory. On
success, that directory is deleted. Add an option to always preserve
that directory.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 394073024f5f..a71e5e9a9087 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -43,6 +43,7 @@ usage() {
 	echo " -K               : sets KMEMLEAK=y"
 	echo " -R|--without-realroot : sets NFT_TEST_HAS_REALROOT=n"
 	echo " -U|--no-unshare  : sets NFT_TEST_UNSHARE_CMD="
+	echo " -k|--keep-logs   : sets NFT_TEST_KEEP_LOGS=y"
 	echo
 	echo "ENVIRONMENT VARIABLES:"
 	echo " NFT=<PATH>    : Path to nft executable"
@@ -69,6 +70,7 @@ usage() {
 	echo " NFT_TEST_HAS_UNSHARED=*|y : To indicate to the test whether the test run will be unshared."
 	echo "                 Test may consider this."
 	echo "                 This is only honored when \$NFT_TEST_UNSHARE_CMD= is set. Otherwise it's detected."
+	echo " NFT_TEST_KEEP_LOGS=*|y: Keep the temp directory. On success, it will be deleted by default."
 	echo " TMPDIR=<PATH> : select a different base directory for the result data"
 }
 
@@ -81,6 +83,7 @@ VERBOSE="$(bool_y "$VERBOSE")"
 DUMPGEN="$(bool_y "$DUMPGEN")"
 VALGRIND="$(bool_y "$VALGRIND")"
 KMEMLEAK="$(bool_y "$KMEMLEAK")"
+NFT_TEST_KEEP_LOGS="$(bool_y "$NFT_TEST_KEEP_LOGS")"
 NFT_TEST_HAS_REALROOT="$NFT_TEST_HAS_REALROOT"
 NFT_TEST_NO_UNSHARE="$NFT_TEST_NO_UNSHARE"
 DO_LIST_TESTS=
@@ -107,6 +110,9 @@ while [ $# -gt 0 ] ; do
 			usage
 			exit 0
 			;;
+		-k|--keep-logs)
+			NFT_TEST_KEEP_LOGS=y
+			;;
 		-L|--list-tests)
 			DO_LIST_TESTS=y
 			;;
@@ -233,6 +239,7 @@ msg_info "conf: KMEMLEAK=$(printf '%q' "$KMEMLEAK")"
 msg_info "conf: NFT_TEST_HAS_REALROOT=$(printf '%q' "$NFT_TEST_HAS_REALROOT")"
 msg_info "conf: NFT_TEST_UNSHARE_CMD=$(printf '%q' "$NFT_TEST_UNSHARE_CMD")"
 msg_info "conf: NFT_TEST_HAS_UNSHARED=$(printf '%q' "$NFT_TEST_HAS_UNSHARED")"
+msg_info "conf: NFT_TEST_KEEP_LOGS=$(printf '%q' "$NFT_TEST_KEEP_LOGS")"
 msg_info "conf: TMPDIR=$(printf '%q' "$_TMPDIR")"
 
 MODPROBE="$(which modprobe)"
@@ -258,8 +265,8 @@ NFT_TEST_LATEST="$_TMPDIR/nft-test.latest.$USER"
 
 ln -snf "$NFT_TEST_TMPDIR" "$NFT_TEST_LATEST"
 
-# export the tmp directory for tests. They may use it, but create
-# distinct files! It will be deleted on EXIT.
+# export the tmp directory for tests. They may use it, but create distinct
+# files! On success, it will be deleted on EXIT. See also "--keep-logs"
 export NFT_TEST_TMPDIR
 
 echo
@@ -471,4 +478,11 @@ if [ "$failed" -gt 0 -a "$NFT_TEST_HAS_REALROOT" != y ] ; then
 	msg_info "test was not running as real root"
 fi
 
+if [ "$failed" -gt 0 -o "$NFT_TEST_KEEP_LOGS" = y ] ; then
+	msg_info "check the temp directory \"$NFT_TEST_TMPDIR\" (\"$NFT_TEST_LATEST\")"
+	msg_info "   ls -lad \"$NFT_TEST_LATEST\"/*/*"
+	msg_info "   grep -R ^ \"$NFT_TEST_LATEST\"/"
+	NFT_TEST_TMPDIR=
+fi
+
 [ "$failed" -eq 0 ]
-- 
2.41.0

[PATCH nft v4 10/17] tests/shell: move the dump diff handling inside "test-wrapper.sh"

[Thomas Haller]

This fits there better. At this point, we are  still inside the unshared
namespace and right after the test. The test-wrapper.sh should compare
(and generate) the dumps.

Also change behavior for DUMPGEN=y.

- Previously it would only rewrite the dump if the dumpfile didn't
  exist yet. Now instead, always rewrite the file with DUMPGEN=y.
  The mode of operation is anyway, that the developer afterwards
  checks `git diff|status` to pick up the changes. There should be
  no changes to existing files (as existing tests are supposed to
  pass). So a diff there either means something went wrong (and we
  should see it) or it just means the dumps correctly should be
  regenerated.

- also, only generate the file if the "dumps/" directory exists. This
  allows to write tests that don't have a dump file and don't get it
  automatically generated.

The test wrapper will return a special error code 124 to indicate that
the test passed, but the dumps file differed.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/test-wrapper.sh | 56 +++++++++++++++++++++++++----
 tests/shell/run-tests.sh            | 45 +++++++++--------------
 2 files changed, 66 insertions(+), 35 deletions(-)

diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
index 0cf37f408003..dd5ce7ace7ad 100755
--- a/tests/shell/helpers/test-wrapper.sh
+++ b/tests/shell/helpers/test-wrapper.sh
@@ -6,20 +6,62 @@
 # For some printf debugging, you can also patch this file.
 
 TEST="$1"
+TESTBASE="$(basename "$TEST")"
+TESTDIR="$(dirname "$TEST")"
 
 printf '%s\n' "$TEST" > "$NFT_TEST_TESTTMPDIR/name"
 
 rc_test=0
 "$TEST" |& tee "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$?
 
-if [ "$rc_test" -eq 0 ] ; then
-	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-ok"
+$NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after"
+
+DUMPPATH="$TESTDIR/dumps"
+DUMPFILE="$DUMPPATH/$TESTBASE.nft"
+
+dump_written=
+rc_dump=
+
+# The caller can request a re-geneating of the dumps, by setting
+# DUMPGEN=y.
+#
+# This only will happen if the command completed with success.
+#
+# It also will only happen for tests, that have a "$DUMPPATH" directory. There
+# might be tests, that don't want to have dumps created. The existence of the
+# directory controls that.
+if [ "$rc_test" -eq 0 -a "$DUMPGEN" = y -a -d "$DUMPPATH" ] ; then
+	dump_written=y
+	cat "$NFT_TEST_TESTTMPDIR/ruleset-after" > "$DUMPFILE"
+fi
+
+if [ "$rc_test" -ne 77 -a -f "$DUMPFILE" ] ; then
+	rc_dump=0
+	if [ "$dump_written" != y ] ; then
+		$DIFF -u "$DUMPFILE" "$NFT_TEST_TESTTMPDIR/ruleset-after" &> "$NFT_TEST_TESTTMPDIR/ruleset-diff" || rc_dump=$?
+		if [ "$rc_dump" -eq 0 ] ; then
+			rm -f "$NFT_TEST_TESTTMPDIR/ruleset-diff"
+		fi
+	fi
+fi
+
+rc_exit="$rc_test"
+if [ -n "$rc_dump" ] && [ "$rc_dump" -ne 0 ] ; then
+	echo "$DUMPFILE" > "$NFT_TEST_TESTTMPDIR/rc-failed-dump"
+	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-failed"
+	if [ "$rc_exit" -eq 0 ] ; then
+		# Special exit code to indicate dump diff.
+		rc_exit=124
+	fi
+elif [ "$rc_test" -eq 0 ] ; then
+	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-ok"
 elif [ "$rc_test" -eq 77 ] ; then
-	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-skipped"
+	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-skipped"
 else
-	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-failed"
+	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-failed"
+	if [ "$rc_test" -eq 124 ] ; then
+		rc_exit=125
+	fi
 fi
 
-$NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after"
-
-exit "$rc_test"
+exit "$rc_exit"
diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index a71e5e9a9087..dc5295634baa 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -48,7 +48,9 @@ usage() {
 	echo "ENVIRONMENT VARIABLES:"
 	echo " NFT=<PATH>    : Path to nft executable"
 	echo " VERBOSE=*|y   : Enable verbose output"
-	echo " DUMPGEN=*|y   : Regenerate dump files"
+	echo " DUMPGEN=*|y   : Regenerate dump files. Dump files are only recreated if the"
+	echo "                 test completes successfully and the \"dumps\" directory for the"
+	echo "                 test exits."
 	echo " VALGRIND=*|y  : Run \$NFT in valgrind"
 	echo " KMEMLEAK=*|y  : Check for kernel memleaks"
 	echo " NFT_TEST_HAS_REALROOT=*|y : To indicate whether the test has real root permissions."
@@ -405,38 +407,25 @@ for testfile in "${TESTS[@]}" ; do
 	export NFT_TEST_TESTTMPDIR
 
 	msg_info "[EXECUTING]	$testfile"
-	test_output="$(NFT="$NFT" DIFF=$DIFF $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile" 2>&1)"
+	test_output="$(NFT="$NFT" DIFF=$DIFF DUMPGEN="$DUMPGEN" $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile" 2>&1)"
 	rc_got=$?
 	echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line
 
-	if [ "$rc_got" -eq 0 ] ; then
-		# FIXME: this should move inside test-wrapper.sh.
-		# check nft dump only for positive tests
-		dumppath="$(dirname ${testfile})/dumps"
-		dumpfile="${dumppath}/$(basename ${testfile}).nft"
-		rc_spec=0
-		if [ "$rc_got" -eq 0 ] && [ -f ${dumpfile} ]; then
-			test_output=$(${DIFF} -u ${dumpfile} <(cat "$NFT_TEST_TESTTMPDIR/ruleset-after") 2>&1)
-			rc_spec=$?
-		fi
-
-		if [ "$rc_spec" -eq 0 ]; then
-			msg_info "[OK]		$testfile"
-			[ "$VERBOSE" == "y" ] && [ ! -z "$test_output" ] && echo "$test_output"
-			((ok++))
+	if [ -s "$NFT_TEST_TESTTMPDIR/ruleset-diff" ] ; then
+		test_output="$test_output$(cat "$NFT_TEST_TESTTMPDIR/ruleset-diff")"
+	fi
 
-			if [ "$DUMPGEN" == "y" ] && [ "$rc_got" == 0 ] && [ ! -f "${dumpfile}" ]; then
-				mkdir -p "${dumppath}"
-				cat "$NFT_TEST_TESTTMPDIR/ruleset-after" > "${dumpfile}"
-			fi
+	if [ "$rc_got" -eq 0 ] ; then
+		((ok++))
+		msg_info "[OK]		$testfile"
+		[ "$VERBOSE" == "y" ] && [ ! -z "$test_output" ] && echo "$test_output"
+	elif [ "$rc_got" -eq 124 ] ; then
+		((failed++))
+		if [ "$VERBOSE" == "y" ] ; then
+			msg_warn "[DUMP FAIL]	$testfile: dump diff detected"
+			[ ! -z "$test_output" ] && echo "$test_output"
 		else
-			((failed++))
-			if [ "$VERBOSE" == "y" ] ; then
-				msg_warn "[DUMP FAIL]	$testfile: dump diff detected"
-				[ ! -z "$test_output" ] && echo "$test_output"
-			else
-				msg_warn "[DUMP FAIL]	$testfile"
-			fi
+			msg_warn "[DUMP FAIL]	$testfile"
 		fi
 	elif [ "$rc_got" -eq 77 ] ; then
 		((skipped++))
-- 
2.41.0

[PATCH nft v4 11/17] tests/shell: rework printing of test results

[Thomas Haller]

- "test-wrapper.sh" no longer will print the test output to its stdout.
  Instead, it only writes the testout.log file.

- rework the loop "run-tests.sh" for printing the test results. It no
  longer captures the output of the test, as the wrapper is expected to
  be silent. Instead, they get the output from the result directory.
  The benefit is, that there is no duplication in what we print and the
  captured data in the result directory. The verbose mode is only for
  convenience, to safe looking at the test data. It's not essential
  otherwise.

- also move the evaluation of the test result (and printing of the
  information) to a separate function. Later we want to run tests in
  parallel, so the steps need to be clearly separated.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/test-wrapper.sh |   2 +-
 tests/shell/run-tests.sh            | 122 +++++++++++++++++++---------
 2 files changed, 84 insertions(+), 40 deletions(-)

diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
index dd5ce7ace7ad..bdbe2c370c6b 100755
--- a/tests/shell/helpers/test-wrapper.sh
+++ b/tests/shell/helpers/test-wrapper.sh
@@ -12,7 +12,7 @@ TESTDIR="$(dirname "$TEST")"
 printf '%s\n' "$TEST" > "$NFT_TEST_TESTTMPDIR/name"
 
 rc_test=0
-"$TEST" |& tee "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$?
+"$TEST" &> "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$?
 
 $NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after"
 
diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index dc5295634baa..f88e244f9625 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -1,16 +1,31 @@
 #!/bin/bash
 
+_msg() {
+	local level="$1"
+	shift
+	local msg
+
+	msg="$level: $*"
+	if [ "$level" = E -o "$level" = W ] ; then
+		printf '%s\n' "$msg" >&2
+	else
+		printf '%s\n' "$msg"
+	fi
+	if [ "$level" = E ] ; then
+		exit 1
+	fi
+}
+
 msg_error() {
-	echo "E: $1 ..." >&2
-	exit 1
+	_msg E "$@"
 }
 
 msg_warn() {
-	echo "W: $1" >&2
+	_msg W "$@"
 }
 
 msg_info() {
-	echo "I: $1"
+	_msg I "$@"
 }
 
 bool_y() {
@@ -393,6 +408,67 @@ check_kmemleak()
 
 check_taint
 
+print_test_header() {
+	local msglevel="$1"
+	local testfile="$2"
+	local status="$3"
+	local suffix="$4"
+	local text
+
+	if [ -n "$suffix" ] ; then
+		suffix=": $suffix"
+	fi
+	text="[$status]"
+	text="$(printf '%-12s' "$text")"
+	_msg "$msglevel" "$text $testfile$suffix"
+}
+
+print_test_result() {
+	local NFT_TEST_TESTTMPDIR="$1"
+	local testfile="$2"
+	local rc_got="$3"
+	shift 3
+
+	local result_msg_level="I"
+	local result_msg_status="OK"
+	local result_msg_suffix=""
+	local result_msg_files=( "$NFT_TEST_TESTTMPDIR/testout.log" "$NFT_TEST_TESTTMPDIR/ruleset-diff" )
+
+	if [ "$rc_got" -eq 0 ] ; then
+		((ok++))
+	elif [ "$rc_got" -eq 124 ] ; then
+		((failed++))
+		result_msg_level="W"
+		result_msg_status="DUMP FAIL"
+	elif [ "$rc_got" -eq 77 ] ; then
+		((skipped++))
+		result_msg_level="I"
+		result_msg_status="SKIPPED"
+	else
+		((failed++))
+		result_msg_level="W"
+		result_msg_status="FAILED"
+		result_msg_suffix="got $rc_got"
+		result_msg_files=( "$NFT_TEST_TESTTMPDIR/testout.log" )
+	fi
+
+	print_test_header "$result_msg_level" "$testfile" "$result_msg_status" "$result_msg_suffix"
+
+	if [ "$VERBOSE" = "y" ] ; then
+		local f
+
+		for f in "${result_msg_files[@]}"; do
+			if [ -s "$f" ] ; then
+				cat "$f"
+			fi
+		done
+
+		if [ "$rc_got" -ne 0 ] ; then
+			msg_info "check \"$NFT_TEST_TESTTMPDIR\""
+		fi
+	fi
+}
+
 TESTIDX=0
 for testfile in "${TESTS[@]}" ; do
 	read taint < /proc/sys/kernel/tainted
@@ -406,44 +482,12 @@ for testfile in "${TESTS[@]}" ; do
 	chmod 755 "$NFT_TEST_TESTTMPDIR"
 	export NFT_TEST_TESTTMPDIR
 
-	msg_info "[EXECUTING]	$testfile"
-	test_output="$(NFT="$NFT" DIFF=$DIFF DUMPGEN="$DUMPGEN" $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile" 2>&1)"
+	print_test_header I "$testfile" "EXECUTING" ""
+	NFT="$NFT" DIFF="$DIFF" DUMPGEN="$DUMPGEN" $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile"
 	rc_got=$?
 	echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line
 
-	if [ -s "$NFT_TEST_TESTTMPDIR/ruleset-diff" ] ; then
-		test_output="$test_output$(cat "$NFT_TEST_TESTTMPDIR/ruleset-diff")"
-	fi
-
-	if [ "$rc_got" -eq 0 ] ; then
-		((ok++))
-		msg_info "[OK]		$testfile"
-		[ "$VERBOSE" == "y" ] && [ ! -z "$test_output" ] && echo "$test_output"
-	elif [ "$rc_got" -eq 124 ] ; then
-		((failed++))
-		if [ "$VERBOSE" == "y" ] ; then
-			msg_warn "[DUMP FAIL]	$testfile: dump diff detected"
-			[ ! -z "$test_output" ] && echo "$test_output"
-		else
-			msg_warn "[DUMP FAIL]	$testfile"
-		fi
-	elif [ "$rc_got" -eq 77 ] ; then
-		((skipped++))
-		if [ "$VERBOSE" == "y" ] ; then
-			msg_warn "[SKIPPED]	$testfile"
-			[ ! -z "$test_output" ] && echo "$test_output"
-		else
-			msg_warn "[SKIPPED]	$testfile"
-		fi
-	else
-		((failed++))
-		if [ "$VERBOSE" == "y" ] ; then
-			msg_warn "[FAILED]	$testfile: got $rc_got"
-			[ ! -z "$test_output" ] && echo "$test_output"
-		else
-			msg_warn "[FAILED]	$testfile"
-		fi
-	fi
+	print_test_result "$NFT_TEST_TESTTMPDIR" "$testfile" "$rc_got"
 
 	check_taint
 	check_kmemleak
-- 
2.41.0

[PATCH nft v4 12/17] tests/shell: move taint check to "test-wrapper.sh"

[Thomas Haller]

We will run tests in parallel. That means, we have multiple tests data and results
in fly. That becomes simpler, if we move more result data to the
test-wrapper and out of "run-tests.sh".

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/test-wrapper.sh | 18 +++++++++++++++---
 tests/shell/run-tests.sh            | 16 ++++------------
 2 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
index bdbe2c370c6b..1390985c7f32 100755
--- a/tests/shell/helpers/test-wrapper.sh
+++ b/tests/shell/helpers/test-wrapper.sh
@@ -11,11 +11,15 @@ TESTDIR="$(dirname "$TEST")"
 
 printf '%s\n' "$TEST" > "$NFT_TEST_TESTTMPDIR/name"
 
+read tainted_before < /proc/sys/kernel/tainted
+
 rc_test=0
 "$TEST" &> "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$?
 
 $NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after"
 
+read tainted_after < /proc/sys/kernel/tainted
+
 DUMPPATH="$TESTDIR/dumps"
 DUMPFILE="$DUMPPATH/$TESTBASE.nft"
 
@@ -45,6 +49,10 @@ if [ "$rc_test" -ne 77 -a -f "$DUMPFILE" ] ; then
 	fi
 fi
 
+if [ "$tainted_before" != "$tainted_after" ]; then
+	echo "$tainted_after" > "$NFT_TEST_TESTTMPDIR/rc-failed-tainted"
+fi
+
 rc_exit="$rc_test"
 if [ -n "$rc_dump" ] && [ "$rc_dump" -ne 0 ] ; then
 	echo "$DUMPFILE" > "$NFT_TEST_TESTTMPDIR/rc-failed-dump"
@@ -53,13 +61,17 @@ if [ -n "$rc_dump" ] && [ "$rc_dump" -ne 0 ] ; then
 		# Special exit code to indicate dump diff.
 		rc_exit=124
 	fi
-elif [ "$rc_test" -eq 0 ] ; then
-	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-ok"
 elif [ "$rc_test" -eq 77 ] ; then
 	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-skipped"
+elif [ "$rc_test" -eq 0 -a "$tainted_before" = "$tainted_after" ] ; then
+	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-ok"
 else
 	echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-failed"
-	if [ "$rc_test" -eq 124 ] ; then
+	if [ "$rc_test" -eq 0 -a "$tainted_before" != "$tainted_after" ] ; then
+		# Special exit code to indicate tainted.
+		rc_exit=123
+	elif [ "$rc_test" -eq 124 -o "$rc_test" -eq 123 ] ; then
+		# These exit codes are reserved
 		rc_exit=125
 	fi
 fi
diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index f88e244f9625..d9ec22f11830 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -356,15 +356,6 @@ echo ""
 ok=0
 skipped=0
 failed=0
-taint=0
-
-check_taint()
-{
-	read taint_now < /proc/sys/kernel/tainted
-	if [ $taint -ne $taint_now ] ; then
-		msg_warn "[FAILED]	kernel is tainted: $taint  -> $taint_now"
-	fi
-}
 
 kmem_runs=0
 kmemleak_found=0
@@ -406,7 +397,10 @@ check_kmemleak()
 	fi
 }
 
-check_taint
+read kernel_tainted < /proc/sys/kernel/tainted
+if [ "$kernel_tainted" -ne 0 ] ; then
+	msg_warn "[FAILED]	kernel is tainted"
+fi
 
 print_test_header() {
 	local msglevel="$1"
@@ -471,7 +465,6 @@ print_test_result() {
 
 TESTIDX=0
 for testfile in "${TESTS[@]}" ; do
-	read taint < /proc/sys/kernel/tainted
 	kernel_cleanup
 
 	((TESTIDX++))
@@ -489,7 +482,6 @@ for testfile in "${TESTS[@]}" ; do
 
 	print_test_result "$NFT_TEST_TESTTMPDIR" "$testfile" "$rc_got"
 
-	check_taint
 	check_kmemleak
 done
 
-- 
2.41.0

[PATCH nft v4 13/17] tests/shell: support running tests in parallel

[Thomas Haller]

Add option to enable running jobs in parallel. The purpose is to speed
up the run time of the tests.

The global cleanup (removal of kernel modules) interferes with parallel
jobs (or even with, unrelated jobs on the system). By setting
NFT_TEST_JOBS= to a positive number, that cleanup is skipped.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 86 ++++++++++++++++++++++++++++++++++------
 1 file changed, 73 insertions(+), 13 deletions(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index d9ec22f11830..d157f14eb9a5 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -59,6 +59,7 @@ usage() {
 	echo " -R|--without-realroot : sets NFT_TEST_HAS_REALROOT=n"
 	echo " -U|--no-unshare  : sets NFT_TEST_UNSHARE_CMD="
 	echo " -k|--keep-logs   : sets NFT_TEST_KEEP_LOGS=y"
+	echo " -j|--jobs        : sets NFT_TEST_JOBS=12"
 	echo
 	echo "ENVIRONMENT VARIABLES:"
 	echo " NFT=<PATH>    : Path to nft executable"
@@ -88,6 +89,11 @@ usage() {
 	echo "                 Test may consider this."
 	echo "                 This is only honored when \$NFT_TEST_UNSHARE_CMD= is set. Otherwise it's detected."
 	echo " NFT_TEST_KEEP_LOGS=*|y: Keep the temp directory. On success, it will be deleted by default."
+	echo " NFT_TEST_JOBS=<NUM}>: by default, run test sequentially. Set to an integer > 1 to"
+	echo "                 run jobs in parallel. Leaving this unset or at zero means to run jobs sequentially"
+	echo "                 and perform global cleanups between tests (remove kernel modules). Setting this"
+	echo "                 to a positive number (including \"1\") means to disable such global cleanups."
+	echo "                 Parallel jobs requires unshare and does not work with NFT_TEST_NO_UNSHARE=."
 	echo " TMPDIR=<PATH> : select a different base directory for the result data"
 }
 
@@ -103,6 +109,7 @@ KMEMLEAK="$(bool_y "$KMEMLEAK")"
 NFT_TEST_KEEP_LOGS="$(bool_y "$NFT_TEST_KEEP_LOGS")"
 NFT_TEST_HAS_REALROOT="$NFT_TEST_HAS_REALROOT"
 NFT_TEST_NO_UNSHARE="$NFT_TEST_NO_UNSHARE"
+NFT_TEST_JOBS="$NFT_TEST_JOBS"
 DO_LIST_TESTS=
 
 TESTS=()
@@ -139,6 +146,9 @@ while [ $# -gt 0 ] ; do
 		-U|--no-unshare)
 			NFT_TEST_NO_UNSHARE=y
 			;;
+		-j|--jobs)
+			NFT_TEST_JOBS=12
+			;;
 		--)
 			TESTS+=( "$@" )
 			shift $#
@@ -242,6 +252,14 @@ fi
 # If tests wish, they can know whether they are unshared via this variable.
 export NFT_TEST_HAS_UNSHARED
 
+# normalize the jobs number to be an integer.
+case "$NFT_TEST_JOBS" in
+	''|*[!0-9]*) NFT_TEST_JOBS=0 ;;
+esac
+if [ -z "$NFT_TEST_UNSHARE_CMD" -a "$NFT_TEST_JOBS" -gt 1 ] ; then
+	NFT_TEST_JOBS=1
+fi
+
 [ -z "$NFT" ] && NFT="$NFT_TEST_BASEDIR/../../src/nft"
 ${NFT} > /dev/null 2>&1
 ret=$?
@@ -257,11 +275,14 @@ msg_info "conf: NFT_TEST_HAS_REALROOT=$(printf '%q' "$NFT_TEST_HAS_REALROOT")"
 msg_info "conf: NFT_TEST_UNSHARE_CMD=$(printf '%q' "$NFT_TEST_UNSHARE_CMD")"
 msg_info "conf: NFT_TEST_HAS_UNSHARED=$(printf '%q' "$NFT_TEST_HAS_UNSHARED")"
 msg_info "conf: NFT_TEST_KEEP_LOGS=$(printf '%q' "$NFT_TEST_KEEP_LOGS")"
+msg_info "conf: NFT_TEST_JOBS=$NFT_TEST_JOBS"
 msg_info "conf: TMPDIR=$(printf '%q' "$_TMPDIR")"
 
-MODPROBE="$(which modprobe)"
-if [ ! -x "$MODPROBE" ] ; then
-	msg_error "no modprobe binary found"
+if [ "$NFT_TEST_JOBS" -eq 0 ] ; then
+	MODPROBE="$(which modprobe)"
+	if [ ! -x "$MODPROBE" ] ; then
+		msg_error "no modprobe binary found"
+	fi
 fi
 
 DIFF="$(which diff)"
@@ -291,6 +312,11 @@ msg_info "info: NFT_TEST_BASEDIR=$(printf '%q' "$NFT_TEST_BASEDIR")"
 msg_info "info: NFT_TEST_TMPDIR=$(printf '%q' "$NFT_TEST_TMPDIR")"
 
 kernel_cleanup() {
+	if [ "$NFT_TEST_JOBS" -ne 0 ] ; then
+		# When we run jobs in parallel (even with only one "parallel"
+		# job via `NFT_TEST_JOBS=1`), we skip such global cleanups.
+		return
+	fi
 	if [ "$NFT_TEST_HAS_UNSHARED" != y ] ; then
 		$NFT flush ruleset
 	fi
@@ -463,28 +489,62 @@ print_test_result() {
 	fi
 }
 
+declare -A JOBS_TEMPDIR
+declare -A JOBS_PIDLIST
+
+job_start() {
+	local testfile="$1"
+
+	if [ "$NFT_TEST_JOBS" -le 1 ] ; then
+		print_test_header I "$testfile" "EXECUTING" ""
+	fi
+
+	NFT_TEST_TESTTMPDIR="${JOBS_TEMPDIR["$testfile"]}" \
+	NFT="$NFT" DIFF="$DIFF" DUMPGEN="$DUMPGEN" $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile"
+	local rc_got=$?
+
+	if [ "$NFT_TEST_JOBS" -le 1 ] ; then
+		echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line
+	fi
+
+	return "$rc_got"
+}
+
+job_wait()
+{
+	local num_jobs="$1"
+
+	while [ "$JOBS_N_RUNNING" -gt 0 -a "$JOBS_N_RUNNING" -ge "$num_jobs" ] ; do
+		wait -n -p JOBCOMPLETED
+		local rc_got="$?"
+		testfile2="${JOBS_PIDLIST[$JOBCOMPLETED]}"
+		print_test_result "${JOBS_TEMPDIR["$testfile2"]}" "$testfile2" "$rc_got"
+		((JOBS_N_RUNNING--))
+		check_kmemleak
+	done
+}
+
 TESTIDX=0
+JOBS_N_RUNNING=0
 for testfile in "${TESTS[@]}" ; do
+	job_wait "$NFT_TEST_JOBS"
+
 	kernel_cleanup
 
 	((TESTIDX++))
 
-	# We also create and export a test-specific temporary directory.
 	NFT_TEST_TESTTMPDIR="$NFT_TEST_TMPDIR/test-${testfile//\//-}.$TESTIDX"
 	mkdir "$NFT_TEST_TESTTMPDIR"
 	chmod 755 "$NFT_TEST_TESTTMPDIR"
-	export NFT_TEST_TESTTMPDIR
-
-	print_test_header I "$testfile" "EXECUTING" ""
-	NFT="$NFT" DIFF="$DIFF" DUMPGEN="$DUMPGEN" $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile"
-	rc_got=$?
-	echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line
-
-	print_test_result "$NFT_TEST_TESTTMPDIR" "$testfile" "$rc_got"
+	JOBS_TEMPDIR["$testfile"]="$NFT_TEST_TESTTMPDIR"
 
-	check_kmemleak
+	job_start "$testfile" &
+	JOBS_PIDLIST[$!]="$testfile"
+	((JOBS_N_RUNNING++))
 done
 
+job_wait 0
+
 echo ""
 
 # kmemleak may report suspected leaks
-- 
2.41.0

[PATCH nft v4 14/17] tests/shell: bind mount private /var/run/netns in test container

[Thomas Haller]

Some tests want to run `ip netns add`, which requires write permissions
to /var/run/netns. Also, /var/run/netns would be a systemwide mount
path, and shared between the tests. We would want to isolate that.

Fix that by bind mount a tmpfs inside the test wrapper, if we appear to
have a private mount namespace.

Fixes

  $ ./tests/shell/run-tests.sh -- tests/shell/testcases/netns/0001nft-f_0

Optimally, `ip netns add` would allow to specify a private
location for those bind mounts.

It seems that iproute2 is build with /var/run/netns, instead the more
common /run/netns. Hence, handle /var/run instead of /run.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/test-wrapper.sh | 23 +++++++++++++++++++++
 tests/shell/run-tests.sh            | 32 +++++++++++++++++++++++++----
 2 files changed, 51 insertions(+), 4 deletions(-)

diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
index 1390985c7f32..58cf48172fc0 100755
--- a/tests/shell/helpers/test-wrapper.sh
+++ b/tests/shell/helpers/test-wrapper.sh
@@ -9,10 +9,33 @@ TEST="$1"
 TESTBASE="$(basename "$TEST")"
 TESTDIR="$(dirname "$TEST")"
 
+CLEANUP_UMOUNT_RUN_NETNS=n
+
+cleanup() {
+	if [ "$CLEANUP_UMOUNT_RUN_NETNS" = y ] ; then
+		umount "/var/run/netns" || :
+	fi
+}
+
+trap cleanup EXIT
+
 printf '%s\n' "$TEST" > "$NFT_TEST_TESTTMPDIR/name"
 
 read tainted_before < /proc/sys/kernel/tainted
 
+if [ "$NFT_TEST_HAS_UNSHARED_MOUNT" = y ] ; then
+	# We have a private mount namespace. We will mount /run/netns as a tmpfs,
+	# this is useful because `ip netns add` wants to add files there.
+	#
+	# When running as rootless, this is necessary to get such tests to
+	# pass.  When running rootful, it's still useful to not touch the
+	# "real" /var/run/netns of the system.
+	mkdir -p /var/run/netns
+	if mount -t tmpfs --make-private "/var/run/netns" ; then
+		CLEANUP_UMOUNT_RUN_NETNS=y
+	fi
+fi
+
 rc_test=0
 "$TEST" &> "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$?
 
diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index d157f14eb9a5..8564d9a08bcb 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -83,11 +83,14 @@ usage() {
 	echo "                 By default it is unset, in which case it's autodetected as"
 	echo "                 \`unshare -f -p\` (for root) or as \`unshare -f -p --mount-proc -U --map-root-user -n\`"
 	echo "                 for non-root."
-	echo "                 When setting this, you may also want to set NFT_TEST_HAS_UNSHARED="
-	echo "                 and NFT_TEST_HAS_REALROOT= accordingly."
+	echo "                 When setting this, you may also want to set NFT_TEST_HAS_UNSHARED=,"
+	echo "                 NFT_TEST_HAS_REALROOT= and NFT_TEST_HAS_UNSHARED_MOUNT= accordingly."
 	echo " NFT_TEST_HAS_UNSHARED=*|y : To indicate to the test whether the test run will be unshared."
 	echo "                 Test may consider this."
 	echo "                 This is only honored when \$NFT_TEST_UNSHARE_CMD= is set. Otherwise it's detected."
+	echo " NFT_TEST_HAS_UNSHARED_MOUNT=*|y : To indicate to the test whether the test run will have a private"
+	echo "                 mount namespace."
+	echo "                 This is only honored when \$NFT_TEST_UNSHARE_CMD= is set. Otherwise it's detected."
 	echo " NFT_TEST_KEEP_LOGS=*|y: Keep the temp directory. On success, it will be deleted by default."
 	echo " NFT_TEST_JOBS=<NUM}>: by default, run test sequentially. Set to an integer > 1 to"
 	echo "                 run jobs in parallel. Leaving this unset or at zero means to run jobs sequentially"
@@ -225,9 +228,22 @@ if [ -n "${NFT_TEST_UNSHARE_CMD+x}" ] ; then
 	else
 		NFT_TEST_HAS_UNSHARED="$(bool_y "$NFT_TEST_HAS_UNSHARED")"
 	fi
+	if [ -z "${NFT_TEST_HAS_UNSHARED_MOUNT+x}" ] ; then
+		NFT_TEST_HAS_UNSHARED_MOUNT=n
+		if [ "$NFT_TEST_HAS_UNSHARED" == y ] ; then
+			case "$NFT_TEST_UNSHARE_CMD" in
+				unshare*-m*|unshare*--mount-proc*)
+					NFT_TEST_HAS_UNSHARED_MOUNT=y
+					;;
+			esac
+		fi
+	else
+		NFT_TEST_HAS_UNSHARED_MOUNT="$(bool_y "$NFT_TEST_HAS_UNSHARED_MOUNT")"
+	fi
 else
 	NFT_TEST_UNSHARE_CMD=""
 	NFT_TEST_HAS_UNSHARED="n"
+	NFT_TEST_HAS_UNSHARED_MOUNT=n
 	if [ "$NFT_TEST_NO_UNSHARE" != y ] ; then
 		if [ "$NFT_TEST_HAS_REALROOT" = y ] ; then
 			# We appear to have real root. So try to unshare
@@ -235,13 +251,19 @@ else
 			# tests that are limited by
 			# /proc/sys/net/core/{wmem_max,rmem_max}. With real
 			# root, we want to test that.
-			detect_unshare "unshare -f -n -m" ||
+			if detect_unshare "unshare -f -n -m" ; then
+				NFT_TEST_HAS_UNSHARED_MOUNT=y
+			else
 				detect_unshare "unshare -f -n" ||
 				detect_unshare "unshare -f -p -m --mount-proc -U --map-root-user -n" ||
 				detect_unshare "unshare -f -U --map-root-user -n"
+			fi
 		else
-			detect_unshare "unshare -f -p -m --mount-proc -U --map-root-user -n" ||
+			if detect_unshare "unshare -f -p -m --mount-proc -U --map-root-user -n" ; then
+				NFT_TEST_HAS_UNSHARED_MOUNT=y
+			else
 				detect_unshare "unshare -f -U --map-root-user -n"
+			fi
 		fi
 		if [ -z "$NFT_TEST_UNSHARE_CMD" ] ; then
 			msg_error "Unshare does not work. Run as root with -U/--no-unshare/NFT_TEST_NO_UNSHARE=y or set NFT_TEST_UNSHARE_CMD"
@@ -251,6 +273,7 @@ else
 fi
 # If tests wish, they can know whether they are unshared via this variable.
 export NFT_TEST_HAS_UNSHARED
+export NFT_TEST_HAS_UNSHARED_MOUNT
 
 # normalize the jobs number to be an integer.
 case "$NFT_TEST_JOBS" in
@@ -274,6 +297,7 @@ msg_info "conf: KMEMLEAK=$(printf '%q' "$KMEMLEAK")"
 msg_info "conf: NFT_TEST_HAS_REALROOT=$(printf '%q' "$NFT_TEST_HAS_REALROOT")"
 msg_info "conf: NFT_TEST_UNSHARE_CMD=$(printf '%q' "$NFT_TEST_UNSHARE_CMD")"
 msg_info "conf: NFT_TEST_HAS_UNSHARED=$(printf '%q' "$NFT_TEST_HAS_UNSHARED")"
+msg_info "conf: NFT_TEST_HAS_UNSHARED_MOUNT=$(printf '%q' "$NFT_TEST_HAS_UNSHARED_MOUNT")"
 msg_info "conf: NFT_TEST_KEEP_LOGS=$(printf '%q' "$NFT_TEST_KEEP_LOGS")"
 msg_info "conf: NFT_TEST_JOBS=$NFT_TEST_JOBS"
 msg_info "conf: TMPDIR=$(printf '%q' "$_TMPDIR")"
-- 
2.41.0

[PATCH nft v4 15/17] tests/shell: skip test in rootless that hit socket buffer size limit

[Thomas Haller]

The socket buffer limits like /proc/sys/net/core/{rmem_max,wmem_max}
can cause tests to fail, when running rootless. That's because real-root
can override those limits, rootless cannot.

Add an environment variable NFT_TEST_HAS_SOCKET_LIMITS=*|n which is
automatically set by "run-tests.sh".

Certain tests will check for [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] and
skip the test.

The user may manually bump those limits (requires root), and set
NFT_TEST_HAS_SOCKET_LIMITS=n to get the tests to pass even as rootless.

For example, the test passes with root:

  sudo ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0

Without root, it would fail. Skip it instead:

  ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0
  ...
  I: [SKIPPED]    tests/shell/testcases/sets/automerge_0

Or bump the limit:

  $ echo 3000000 | sudo tee /proc/sys/net/core/wmem_max
  $ NFT_TEST_HAS_SOCKET_LIMITS=n ./tests/shell/run-tests.sh -- tests/shell/testcases/sets/automerge_0
  ...
  I: [OK]         tests/shell/testcases/sets/automerge_0

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh                      | 35 +++++++++++++++++++
 tests/shell/testcases/nft-f/0011manydefines_0 | 16 +++++++++
 .../testcases/sets/0011add_many_elements_0    | 15 ++++++++
 .../sets/0012add_delete_many_elements_0       | 14 ++++++++
 .../sets/0013add_delete_many_elements_0       | 14 ++++++++
 tests/shell/testcases/sets/automerge_0        | 24 ++++++++++---
 tests/shell/testcases/transactions/30s-stress |  9 +++++
 7 files changed, 123 insertions(+), 4 deletions(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 8564d9a08bcb..d2ed1886c3fe 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -28,6 +28,24 @@ msg_info() {
 	_msg I "$@"
 }
 
+bool_n() {
+	case "$1" in
+		n|N|no|No|NO|0|false)
+			printf n
+			;;
+		'')
+			if [ $# -ge 2 ] ; then
+				printf "%s" "$2"
+			else
+				printf n
+			fi
+			;;
+		*)
+			printf y
+			;;
+	esac
+}
+
 bool_y() {
 	case "$1" in
 		y|Y|yes|Yes|YES|1|true)
@@ -77,6 +95,11 @@ usage() {
 	echo "                 e.g. due to limited /proc/sys/net/core/{wmem_max,rmem_max}."
 	echo "                 Checks that cannot pass in such environment should check for"
 	echo "                 [ \"\$NFT_TEST_HAS_REALROOT\" != y ] and skip gracefully."
+	echo " NFT_TEST_HAS_SOCKET_LIMITS=*|n : some tests will fail if /proc/sys/net/core/{wmem_max,rmem_max} is"
+	echo "                 too small. When running as real root, then those limits can be overridden but"
+	echo "                 rootless, the test would fail. Tests will check for [ "\$NFT_TEST_HAS_SOCKET_LIMITS" = y ]"
+	echo "                 and skip. You may set NFT_TEST_HAS_SOCKET_LIMITS=n if you ensure those limits are"
+	echo "                 suitable to run the test rootless."
 	echo " NFT_TEST_UNSHARE_CMD=cmd : when set, this is the command line for an unshare"
 	echo "                 command, which is used to sandbox each test invocation. By"
 	echo "                 setting it to empty, no unsharing is done."
@@ -202,6 +225,17 @@ else
 fi
 export NFT_TEST_HAS_REALROOT
 
+if [ "$NFT_TEST_HAS_SOCKET_LIMITS" = "" ] ; then
+	if [ "$NFT_TEST_HAS_REALROOT" = y ] ; then
+		NFT_TEST_HAS_SOCKET_LIMITS=n
+	else
+		NFT_TEST_HAS_SOCKET_LIMITS=y
+	fi
+else
+	NFT_TEST_HAS_SOCKET_LIMITS="$(bool_n "$NFT_TEST_HAS_SOCKET_LIMITS")"
+fi
+export NFT_TEST_HAS_SOCKET_LIMITS
+
 detect_unshare() {
 	if ! $1 true &>/dev/null ; then
 		return 1
@@ -295,6 +329,7 @@ msg_info "conf: DUMPGEN=$(printf '%q' "$DUMPGEN")"
 msg_info "conf: VALGRIND=$(printf '%q' "$VALGRIND")"
 msg_info "conf: KMEMLEAK=$(printf '%q' "$KMEMLEAK")"
 msg_info "conf: NFT_TEST_HAS_REALROOT=$(printf '%q' "$NFT_TEST_HAS_REALROOT")"
+msg_info "conf: NFT_TEST_HAS_SOCKET_LIMITS=$(printf '%q' "$NFT_TEST_HAS_SOCKET_LIMITS")"
 msg_info "conf: NFT_TEST_UNSHARE_CMD=$(printf '%q' "$NFT_TEST_UNSHARE_CMD")"
 msg_info "conf: NFT_TEST_HAS_UNSHARED=$(printf '%q' "$NFT_TEST_HAS_UNSHARED")"
 msg_info "conf: NFT_TEST_HAS_UNSHARED_MOUNT=$(printf '%q' "$NFT_TEST_HAS_UNSHARED_MOUNT")"
diff --git a/tests/shell/testcases/nft-f/0011manydefines_0 b/tests/shell/testcases/nft-f/0011manydefines_0
index 84664f46af50..aac0670602f6 100755
--- a/tests/shell/testcases/nft-f/0011manydefines_0
+++ b/tests/shell/testcases/nft-f/0011manydefines_0
@@ -4,6 +4,15 @@
 
 HOWMANY=20000
 
+if [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] ; then
+	# The socket limit /proc/sys/net/core/wmem_max may be unsuitable for
+	# the test.
+	#
+	# Run only a subset of the test and mark as skipped at the end.
+	HOWMANY=2000
+fi
+
+
 tmpfile=$(mktemp)
 if [ ! -w $tmpfile ] ; then
 	echo "Failed to create tmp file" >&2
@@ -35,3 +44,10 @@ table t {
 
 set -e
 $NFT -f $tmpfile
+
+if [ "$HOWMANY" != 20000 ] ; then
+	echo "NFT_TEST_HAS_SOCKET_LIMITS indicates that the socket limit for"
+	echo "/proc/sys/net/core/wmem_max is too small for this test. Mark as SKIPPED"
+	echo "You may bump the limit and rerun with \`NFT_TEST_HAS_SOCKET_LIMITS=n\`."
+	exit 77
+fi
diff --git a/tests/shell/testcases/sets/0011add_many_elements_0 b/tests/shell/testcases/sets/0011add_many_elements_0
index ba23f90f2328..c37b2f0dfa2e 100755
--- a/tests/shell/testcases/sets/0011add_many_elements_0
+++ b/tests/shell/testcases/sets/0011add_many_elements_0
@@ -3,6 +3,14 @@
 # test adding many sets elements
 
 HOWMANY=255
+if [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] ; then
+	# The socket limit /proc/sys/net/core/wmem_max may be unsuitable for
+	# the test.
+	#
+	# Run only a subset of the test and mark as skipped at the end.
+	HOWMANY=30
+fi
+
 
 tmpfile=$(mktemp)
 if [ ! -w $tmpfile ] ; then
@@ -30,3 +38,10 @@ add element x y $(generate)" > $tmpfile
 
 set -e
 $NFT -f $tmpfile
+
+if [ "$HOWMANY" != 255 ] ; then
+	echo "NFT_TEST_HAS_SOCKET_LIMITS indicates that the socket limit for"
+	echo "/proc/sys/net/core/wmem_max is too small for this test. Mark as SKIPPED"
+	echo "You may bump the limit and rerun with \`NFT_TEST_HAS_SOCKET_LIMITS=n\`."
+	exit 77
+fi
diff --git a/tests/shell/testcases/sets/0012add_delete_many_elements_0 b/tests/shell/testcases/sets/0012add_delete_many_elements_0
index 7e7beebd2073..64451604c3d1 100755
--- a/tests/shell/testcases/sets/0012add_delete_many_elements_0
+++ b/tests/shell/testcases/sets/0012add_delete_many_elements_0
@@ -3,6 +3,13 @@
 # test adding and deleting many sets elements
 
 HOWMANY=255
+if [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] ; then
+	# The socket limit /proc/sys/net/core/wmem_max may be unsuitable for
+	# the test.
+	#
+	# Run only a subset of the test and mark as skipped at the end.
+	HOWMANY=30
+fi
 
 tmpfile=$(mktemp)
 if [ ! -w $tmpfile ] ; then
@@ -31,3 +38,10 @@ delete element x y $(generate)" > $tmpfile
 
 set -e
 $NFT -f $tmpfile
+
+if [ "$HOWMANY" != 255 ] ; then
+	echo "NFT_TEST_HAS_SOCKET_LIMITS indicates that the socket limit for"
+	echo "/proc/sys/net/core/wmem_max is too small for this test. Mark as SKIPPED"
+	echo "You may bump the limit and rerun with \`NFT_TEST_HAS_SOCKET_LIMITS=n\`."
+	exit 77
+fi
diff --git a/tests/shell/testcases/sets/0013add_delete_many_elements_0 b/tests/shell/testcases/sets/0013add_delete_many_elements_0
index 5774317b6b63..c0925dd57b47 100755
--- a/tests/shell/testcases/sets/0013add_delete_many_elements_0
+++ b/tests/shell/testcases/sets/0013add_delete_many_elements_0
@@ -3,6 +3,13 @@
 # test adding and deleting many sets elements in two nft -f runs.
 
 HOWMANY=255
+if [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] ; then
+	# The socket limit /proc/sys/net/core/wmem_max may be unsuitable for
+	# the test.
+	#
+	# Run only a subset of the test and mark as skipped at the end.
+	HOWMANY=30
+fi
 
 tmpfile=$(mktemp)
 if [ ! -w $tmpfile ] ; then
@@ -32,3 +39,10 @@ add element x y $(generate)" > $tmpfile
 $NFT -f $tmpfile
 echo "delete element x y $(generate)" > $tmpfile
 $NFT -f $tmpfile
+
+if [ "$HOWMANY" != 255 ] ; then
+	echo "NFT_TEST_HAS_SOCKET_LIMITS indicates that the socket limit for"
+	echo "/proc/sys/net/core/wmem_max is too small for this test. Mark as SKIPPED"
+	echo "You may bump the limit and rerun with \`NFT_TEST_HAS_SOCKET_LIMITS=n\`."
+	exit 77
+fi
diff --git a/tests/shell/testcases/sets/automerge_0 b/tests/shell/testcases/sets/automerge_0
index 7530b3db7317..fc34f8865fb3 100755
--- a/tests/shell/testcases/sets/automerge_0
+++ b/tests/shell/testcases/sets/automerge_0
@@ -10,14 +10,23 @@ RULESET="table inet x {
 	}
 }"
 
+HOWMANY=65535
+if [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] ; then
+	# The socket limit /proc/sys/net/core/wmem_max may be unsuitable for
+	# the test.
+	#
+	# Run only a subset of the test and mark as skipped at the end.
+	HOWMANY=5000
+fi
+
 $NFT -f - <<< $RULESET
 
 tmpfile=$(mktemp)
 echo -n "add element inet x y { " > $tmpfile
-for ((i=0;i<65535;i+=2))
+for ((i=0;i<$HOWMANY;i+=2))
 do
 	echo -n "$i, " >> $tmpfile
-	if [ $i -eq 65534 ]
+	if [ $i -eq $((HOWMANY-1)) ]
 	then
 		echo -n "$i" >> $tmpfile
 	fi
@@ -27,7 +36,7 @@ echo "}" >> $tmpfile
 $NFT -f $tmpfile
 
 tmpfile2=$(mktemp)
-for ((i=1;i<65535;i+=2))
+for ((i=1;i<$HOWMANY;i+=2))
 do
 	echo "$i" >> $tmpfile2
 done
@@ -48,7 +57,7 @@ done
 
 for ((i=0;i<10;i++))
 do
-	from=$(($RANDOM%65535))
+	from=$(($RANDOM%$HOWMANY))
 	to=$(($from+100))
 	$NFT add element inet x y { $from-$to }
 	if [ $? -ne 0 ]
@@ -111,3 +120,10 @@ done
 rm -f $tmpfile
 rm -f $tmpfile2
 rm -f $tmpfile3
+
+if [ "$HOWMANY" != 65535 ] ; then
+	echo "NFT_TEST_HAS_SOCKET_LIMITS indicates that the socket limit for"
+	echo "/proc/sys/net/core/wmem_max is too small for this test. Mark as SKIPPED"
+	echo "You may bump the limit and rerun with \`NFT_TEST_HAS_SOCKET_LIMITS=n\`."
+	exit 77
+fi
diff --git a/tests/shell/testcases/transactions/30s-stress b/tests/shell/testcases/transactions/30s-stress
index 4d3317e22b0c..757e7639b5e9 100755
--- a/tests/shell/testcases/transactions/30s-stress
+++ b/tests/shell/testcases/transactions/30s-stress
@@ -16,6 +16,15 @@ if [ x = x"$NFT" ] ; then
 	NFT=nft
 fi
 
+if [ "$NFT_TEST_HAS_SOCKET_LIMITS" = y ] ; then
+	# The socket limit /proc/sys/net/core/wmem_max may be unsuitable for
+	# the test.
+	#
+	# Skip it. You may ensure that the limits are suitable and rerun
+	# with NFT_TEST_HAS_SOCKET_LIMITS=n.
+	exit 77
+fi
+
 testns=testns-$(mktemp -u "XXXXXXXX")
 tmp=""
 
-- 
2.41.0

[PATCH nft v4 16/17] tests/shell: record the test duration for investigation

[Thomas Haller]

Runtimes are important. Add a way to find out how long tests took.

  $ ./tests/shell/run-tests.sh -j
  ...
  $ for d in /tmp/nft-test.latest.*/test-*/ ; do \
         printf '%10.2f  %s\n' \
                "$(sed '1!d' "$d/times")" \
                "$(cat "$d/name")" ; \
    done \
    | sort -n \
    | awk '{print $0; s+=$1} END{printf("%10.2f\n", s)}'

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/test-wrapper.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
index 58cf48172fc0..f8aed3241aea 100755
--- a/tests/shell/helpers/test-wrapper.sh
+++ b/tests/shell/helpers/test-wrapper.sh
@@ -9,6 +9,8 @@ TEST="$1"
 TESTBASE="$(basename "$TEST")"
 TESTDIR="$(dirname "$TEST")"
 
+START_TIME="$(cut -d ' ' -f1 /proc/uptime)"
+
 CLEANUP_UMOUNT_RUN_NETNS=n
 
 cleanup() {
@@ -99,4 +101,9 @@ else
 	fi
 fi
 
+END_TIME="$(cut -d ' ' -f1 /proc/uptime)"
+WALL_TIME="$(awk -v start="$START_TIME" -v end="$END_TIME" "BEGIN { print(end - start) }")" ||
+  WALL_TIME="$(echo "scale=4; $END_TIME - $START_TIME" | bc)"
+printf "%s\n" "$WALL_TIME" "$START_TIME" "$END_TIME" > "$NFT_TEST_TESTTMPDIR/times"
+
 exit "$rc_exit"
-- 
2.41.0

[PATCH nft v4 17/17] tests/shell: set TMPDIR for tests in "test-wrapper.sh"

[Thomas Haller]

Various tests create additional temporary files. They really should just
use "$NFT_TEST_TESTTMPDIR" for that. However, they mostly use `mktemp`.

The scripts are supposed to cleanup those files afterwards. However,
often that does not work correctly and /tmp gets full of left over
temporary files.

Export "TMPDIR" so that they use the test-specific temporary directory.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/test-wrapper.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
index f8aed3241aea..9782040ed798 100755
--- a/tests/shell/helpers/test-wrapper.sh
+++ b/tests/shell/helpers/test-wrapper.sh
@@ -11,6 +11,8 @@ TESTDIR="$(dirname "$TEST")"
 
 START_TIME="$(cut -d ' ' -f1 /proc/uptime)"
 
+export TMPDIR="$NFT_TEST_TESTTMPDIR"
+
 CLEANUP_UMOUNT_RUN_NETNS=n
 
 cleanup() {
-- 
2.41.0