[PATCH nft 0/2] tests/shell: add feature probing via "features/*.nft" files

[PATCH nft 0/2] tests/shell: add feature probing via "features/*.nft" files

[Thomas Haller]

This is based on

  Subject:	[PATCH nft 1/5] tests: add feature probing
  Date:	Mon,  4 Sep 2023 11:06:30 +0200

Changes compared to Florian's patch:

- The variables are called "NFT_TEST_HAVE_feat" instead of "NFT_HAVE_feat".

- The code is rewritten to fit into the latest changes of "run-tests.sh".

- The user may set NFT_TEST_HAVE_feat=y|n to force a feature for testing
  and override the detection.

- Print the feature detection in a different style.

Thomas Haller (2):
  tests/shell: add feature probing via "features/*.nft" files
  tests/shell: colorize NFT_TEST_SKIP_/NFT_TEST_HAVE_ in test output

 tests/shell/run-tests.sh | 40 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 38 insertions(+), 2 deletions(-)

-- 
2.41.0

[PATCH nft 1/2] tests/shell: add feature probing via "features/*.nft" files

[Thomas Haller]

Running selftests on older kernels makes some of them fail very early
because some tests use features that are not available on older kernels,
e.g. -stable releases.

Known examples:
- inner header matching
- anonymous chains
- elem delete from packet path

Also, some test cases might fail because a feature isn't compiled in,
such as netdev chains.

This adds a feature-probing mechanism to shell tests.

Simply drop a 'nft -f' compatible file with a .nft suffix into
"tests/shell/features". "run-tests.sh" will load it via `nft --check`
and will export

  NFT_TEST_HAVE_${feature}=y|n

Here ${feature} is the basename of the .nft file without file extension.
It must be all lower-case.

This extends the existing NFT_TEST_HAVE_json= feature detection.
Similarly, NFT_TEST_REQUIRES(NFT_TEST_HAVE_*) tags work to easily skip a
test.

The test script that cannot fully work without the feature should either
skip the test entirely (NFT_TEST_REQUIRES(NFT_TEST_HAVE_*)), or run a
reduced/modified test. If a modified test was run and passes, it is
still a good idea to mark the overall result as skipped (exit 77)
instead of claiming success to the modified test. We want to know when
not the full test was running, while we want to test as much as we can.

This patch is based on Florian's feature probing patch.

Originally-by: Florian Westphal <fw@strlen.de>
Signed-off-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
---
 tests/shell/run-tests.sh | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index c5d6307d067e..01a312d0ee2c 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -222,6 +222,23 @@ NFT_TEST_BASEDIR="$(dirname "$0")"
 export NFT_TEST_BASEDIR
 
 _HAVE_OPTS=( json )
+_HAVE_OPTS_NFT=()
+shopt -s nullglob
+F=( "$NFT_TEST_BASEDIR/features/"*.nft )
+shopt -u nullglob
+for file in "${F[@]}"; do
+	feat="${file##*/}"
+	feat="${feat%.nft}"
+	re="^[a-z_0-9]+$"
+	if [[ "$feat" =~ $re ]] && ! array_contains "$feat" "${_HAVE_OPTS[@]}" ; then
+		_HAVE_OPTS_NFT+=( "$feat" )
+	else
+		msg_warn "Ignore feature file \"$file\""
+	fi
+done
+_HAVE_OPTS+=( "${_HAVE_OPTS_NFT[@]}" )
+_HAVE_OPTS=( $(printf '%s\n' "${_HAVE_OPTS[@]}" | LANG=C sort) )
+
 for KEY in $(compgen -v | grep '^NFT_TEST_HAVE_' | sort) ; do
 	if ! array_contains "${KEY#NFT_TEST_HAVE_}" "${_HAVE_OPTS[@]}" ; then
 		unset "$KEY"
@@ -477,6 +494,17 @@ else
 fi
 export NFT_TEST_HAVE_json
 
+for feat in "${_HAVE_OPTS_NFT[@]}" ; do
+	var="NFT_TEST_HAVE_$feat"
+	if [ -z "${!var+x}" ] ; then
+		val='y'
+		$NFT_TEST_UNSHARE_CMD "$NFT_REAL" --check -f "$NFT_TEST_BASEDIR/features/$feat.nft" &>/dev/null || val='n'
+	else
+		val="$(bool_n "${!var}")"
+	fi
+	eval "export $var=$val"
+done
+
 if [ "$NFT_TEST_JOBS" -eq 0 ] ; then
 	MODPROBE="$(which modprobe)"
 	if [ ! -x "$MODPROBE" ] ; then
-- 
2.41.0

[PATCH nft 2/2] tests/shell: colorize NFT_TEST_SKIP_/NFT_TEST_HAVE_ in test output

[Thomas Haller]

Having a "SKIP" option as "y" or a "HAVE" option as "n", is note worthy
because tests may be skipped based on that.

Colorize, to make it easier to see in the test output.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 01a312d0ee2c..1527b2a6455c 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -571,11 +571,19 @@ msg_info "conf: NFT_TEST_SHUFFLE_TESTS=$NFT_TEST_SHUFFLE_TESTS"
 msg_info "conf: TMPDIR=$(printf '%q' "$_TMPDIR")"
 echo
 for KEY in $(compgen -v | grep '^NFT_TEST_SKIP_' | sort) ; do
-	msg_info "conf: $KEY=$(printf '%q' "${!KEY}")"
+	v="${!KEY}"
+	if [ "$v" = y ] ; then
+		v="$YELLOW$v$RESET"
+	fi
+	msg_info "conf: $KEY=$v"
 	export "$KEY"
 done
 for KEY in $(compgen -v | grep '^NFT_TEST_HAVE_' | sort) ; do
-	msg_info "conf: $KEY=$(printf '%q' "${!KEY}")"
+	v="${!KEY}"
+	if [ "$v" = n ] ; then
+		v="$YELLOW$v$RESET"
+	fi
+	msg_info "conf: $KEY=$v"
 	export "$KEY"
 done
 
-- 
2.41.0