[PATCH nft 0/4] more tests/shell updates to run on 5.4 kernels

[PATCH nft 0/4] more tests/shell updates to run on 5.4 kernels

[Pablo Neira Ayuso]

Hi,

This patchset contains more updates for tests/shell to support 5.4 kernels:

1) Detect if kernel comes with flowtable counter support.
2) Detect if kernel comes with flowtable can be defined with no devices.
3) Skip pipapo tests if transactions/30s-stress.
4) Restore pipapo and chain binding tests in transactions/30s-stress
   when it is run standalone.

I am still dealing with 3 tests that fail in 5.4, one of them is:

sets/sets_with_ifnames

which needs a bit of work to detach pipapo support from it.

The remaining two failing tests are related to the flowtable, I am still
diagnosing these.

Pablo Neira Ayuso (4):
  tests: shell: skip if kernel does not support flowtable counter
  tests: shell: skip if kernel does not support flowtable with no devices
  tests: shell: skip pipapo set backend in transactions/30s-stress
  tests: shell: restore pipapo and chain binding coverage in standalone 30s-stress

 tests/shell/features/flowtable_counter.sh     | 16 +++++
 tests/shell/features/flowtable_no_devices.nft |  8 +++
 .../flowtable/0012flowtable_variable_0        |  2 +
 tests/shell/testcases/listing/0020flowtable_0 |  2 +
 tests/shell/testcases/transactions/30s-stress | 60 ++++++++++++++++---
 5 files changed, 81 insertions(+), 7 deletions(-)
 create mode 100755 tests/shell/features/flowtable_counter.sh
 create mode 100755 tests/shell/features/flowtable_no_devices.nft

-- 
2.30.2

[PATCH nft 1/4] tests: shell: skip if kernel does not support flowtable counter

[Pablo Neira Ayuso]

Check if kernel provides flowtable counter supports which is available
since 53c2b2899af7 ("netfilter: flowtable: add counter support").

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/features/flowtable_counter.sh        | 16 ++++++++++++++++
 .../testcases/flowtable/0012flowtable_variable_0 |  2 ++
 2 files changed, 18 insertions(+)
 create mode 100755 tests/shell/features/flowtable_counter.sh

diff --git a/tests/shell/features/flowtable_counter.sh b/tests/shell/features/flowtable_counter.sh
new file mode 100755
index 000000000000..a4c4c62124b0
--- /dev/null
+++ b/tests/shell/features/flowtable_counter.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# 53c2b2899af7 ("netfilter: flowtable: add counter support")
+# v5.7-rc1~146^2~12^2~16
+
+EXPECTED="table ip filter2 {
+	flowtable main_ft2 {
+		hook ingress priority filter
+		devices = { lo }
+		counter
+	}
+}"
+
+$NFT -f - <<< $EXPECTED
+
+diff -u <($NFT list ruleset) - <<<"$EXPECTED"
diff --git a/tests/shell/testcases/flowtable/0012flowtable_variable_0 b/tests/shell/testcases/flowtable/0012flowtable_variable_0
index 080059d24935..9c03820f128e 100755
--- a/tests/shell/testcases/flowtable/0012flowtable_variable_0
+++ b/tests/shell/testcases/flowtable/0012flowtable_variable_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_flowtable_counter)
+
 set -e
 
 iface_cleanup() {
-- 
2.30.2

[PATCH nft 2/4] tests: shell: skip if kernel does not support flowtable with no devices

[Pablo Neira Ayuso]

Originally, flowtables required devices in place to work, this was later
relaxed to allow flowtable with no initial devices, see 05abe4456fa3
("netfilter: nf_tables: allow to register flowtable with no devices").

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/features/flowtable_no_devices.nft | 8 ++++++++
 tests/shell/testcases/listing/0020flowtable_0 | 2 ++
 2 files changed, 10 insertions(+)
 create mode 100755 tests/shell/features/flowtable_no_devices.nft

diff --git a/tests/shell/features/flowtable_no_devices.nft b/tests/shell/features/flowtable_no_devices.nft
new file mode 100755
index 000000000000..30dd3db8b8dd
--- /dev/null
+++ b/tests/shell/features/flowtable_no_devices.nft
@@ -0,0 +1,8 @@
+# 05abe4456fa3 ("netfilter: nf_tables: allow to register flowtable with no devices")
+# v5.8-rc1~165^2~27^2~1
+table ip filter2 {
+	flowtable main_ft2 {
+		hook ingress priority filter
+		counter
+	}
+}
diff --git a/tests/shell/testcases/listing/0020flowtable_0 b/tests/shell/testcases/listing/0020flowtable_0
index 6eb82cfeabc3..0e89f5dd0139 100755
--- a/tests/shell/testcases/listing/0020flowtable_0
+++ b/tests/shell/testcases/listing/0020flowtable_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_flowtable_no_devices)
+
 # list only the flowtable asked for with table
 
 set -e
-- 
2.30.2

[PATCH nft 3/4] tests: shell: skip pipapo set backend in transactions/30s-stress

[Pablo Neira Ayuso]

Skip tests with concatenations and intervals if kernel does not support it.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/testcases/transactions/30s-stress | 51 ++++++++++++++++---
 1 file changed, 44 insertions(+), 7 deletions(-)

diff --git a/tests/shell/testcases/transactions/30s-stress b/tests/shell/testcases/transactions/30s-stress
index 4c3c6a275941..b6ad06abed32 100755
--- a/tests/shell/testcases/transactions/30s-stress
+++ b/tests/shell/testcases/transactions/30s-stress
@@ -38,6 +38,10 @@ if [ -z "${NFT_TEST_HAVE_chain_binding+x}" ] ; then
 	fi
 fi
 
+if [ "$NFT_TEST_HAVE_pipapo" != y ] ;then
+	echo "Skipping pipapo set backend, kernel does not support it"
+fi
+
 testns=testns-$(mktemp -u "XXXXXXXX")
 tmp=""
 
@@ -264,6 +268,19 @@ randdelns()
 	done
 }
 
+available_flags()
+{
+	local -n available_flags=$1
+	selected_key=$2
+	if [ "$selected_key" == "single" ] ;then
+		available_flags+=("interval")
+	elif [ "$selected_key" == "concat" ] ;then
+		if [ "$NFT_TEST_HAVE_pipapo" = y ] ;then
+			available_flags+=("interval")
+		fi
+	fi
+}
+
 random_element_string=""
 
 # create a random element.  Could cause any of the following:
@@ -295,7 +312,10 @@ random_elem()
 
 			fr=$((RANDOM%2))
 			f=0
-			for flags in "" "interval" ; do
+
+			FLAGS=("")
+			available_flags FLAGS $key
+			for flags in ${FLAGS[@]} ; do
 				cnt=$((cnt+1))
 				if [ $f -ne fkr ] ;then
 					f=$((f+1))
@@ -504,8 +524,10 @@ for table in $tables; do
 	# bitmap 2byte
 	echo "insert rule inet $table $chain tcp dport != { 22, 23, 80 } goto defaultchain" >> "$tmp"
 	echo "insert rule inet $table $chain tcp dport { 1-1024, 8000-8080 } jump defaultchain comment rbtree" >> "$tmp"
-	# pipapo (concat + set), with goto anonymous chain.
-	gen_anon_chain_jump "$table $chain ip saddr . tcp dport { 1.2.3.4 . 1-1024, 1.2.3.6 - 1.2.3.10 . 8000-8080, 1.2.3.4 . 8080, 1.2.3.6 - 1.2.3.10 . 22 }" >> "$tmp"
+	if [ "$NFT_TEST_HAVE_pipapo" = y ] ;then
+		# pipapo (concat + set), with goto anonymous chain.
+		gen_anon_chain_jump "$table $chain ip saddr . tcp dport { 1.2.3.4 . 1-1024, 1.2.3.6 - 1.2.3.10 . 8000-8080, 1.2.3.4 . 8080, 1.2.3.6 - 1.2.3.10 . 22 }" >> "$tmp"
+	fi
 
 	# add a few anonymous sets. rhashtable is convered by named sets below.
 	c=$((RANDOM%$count))
@@ -518,8 +540,10 @@ for table in $tables; do
 	# bitmap 2byte
 	echo "insert rule inet $table $chain tcp dport != { 22, 23, 80 } goto defaultchain" >> "$tmp"
 	echo "insert rule inet $table $chain tcp dport { 1-1024, 8000-8080 } jump defaultchain comment rbtree" >> "$tmp"
-	# pipapo (concat + set), with goto anonymous chain.
-	gen_anon_chain_jump "$table $chain ip saddr . tcp dport { 1.2.3.4 . 1-1024, 1.2.3.6 - 1.2.3.10 . 8000-8080, 1.2.3.4 . 8080, 1.2.3.6 - 1.2.3.10 . 22 }" >> "$tmp"
+	if [ "$NFT_TEST_HAVE_pipapo" = y ] ;then
+		# pipapo (concat + set), with goto anonymous chain.
+		gen_anon_chain_jump "$table $chain ip saddr . tcp dport { 1.2.3.4 . 1-1024, 1.2.3.6 - 1.2.3.10 . 8000-8080, 1.2.3.4 . 8080, 1.2.3.6 - 1.2.3.10 . 22 }" >> "$tmp"
+	fi
 
 	# add constant/immutable sets
 	size=$((RANDOM%5120000))
@@ -533,12 +557,21 @@ for table in $tables; do
 
 	# add named sets with various combinations (plain value, range, concatenated values, concatenated ranges, with timeouts, with data ...)
 	for key in "ip saddr" "ip saddr . tcp dport"; do
-		for flags in "" "flags interval;" ; do
+		FLAGS=("")
+		if [ "$key" == "ip saddr" ] ;then
+			FLAGS+=("flags interval;")
+		elif [ "$key" == "ip saddr . tcp dport" ] ;then
+			if [ "$NFT_TEST_HAVE_pipapo" = y ] ;then
+				FLAGS+=("flags interval;")
+			fi
+		fi
+		for ((i = 0; i < ${#FLAGS[@]}; i++)) ; do
 			timeout=$((RANDOM%10))
 			timeout=$((timeout+1))
 			timeout="timeout ${timeout}s"
 
 			cnt=$((cnt+1))
+			flags=${FLAGS[$i]}
 			echo "add set inet $table set_${cnt}  { typeof ${key} ; ${flags} }" >> "$tmp"
 			echo "add set inet $table sett${cnt} { typeof ${key} ; $timeout; ${flags} }" >> "$tmp"
 			echo "add map inet $table dmap_${cnt} { typeof ${key} : meta mark ; ${flags} }" >> "$tmp"
@@ -550,7 +583,11 @@ for table in $tables; do
 
 	cnt=0
 	for key in "single" "concat"; do
-		for flags in "" "interval" ; do
+		FLAGS=("")
+		available_flags FLAGS $key
+
+		for ((i = 0; i < ${#FLAGS[@]}; i++)) ; do
+			flags=${FLAGS[$i]}
 			want="${key}${flags}"
 			cnt=$((cnt+1))
 			maxip=$((RANDOM%256))
-- 
2.30.2

Re: [PATCH nft 3/4] tests: shell: skip pipapo set backend in transactions/30s-stress

[Thomas Haller]

On Wed, 2023-11-15 at 10:42 +0100, Pablo Neira Ayuso wrote:
> 
>  
> +if [ "$NFT_TEST_HAVE_pipapo" != y ] ;then
> +	echo "Skipping pipapo set backend, kernel does not support
> it"
> +fi

It's good and useful to run a reduced subset of the test, if some
kernel feature is missing.

But in that case, the end of the test should have something like

  if [ "$NFT_TEST_HAVE_pipapo" = n ] ; then
      echo "Ran a partial test only, due to NFT_TEST_HAVE_pipapo=n"
      exit 77
  fi

so that it shows up as skipped. In other words, "partially skipped"
should also show up as skipped.




Btw, I find it better to compare the  $NFT_TEST_HAVE_ variables only
against "n", like:

  if [ "$NFT_TEST_HAVE_pipapo" = n ] ; then
  if [ "$NFT_TEST_HAVE_pipapo" != n ] ; then

so that the SKIPPING only happens after an explicit opt-out.

Then you wouldn't need the workaround

»·······if [ -z "${NFT_TEST_HAVE_pipapo+x}" ]; then
»·······»·······NFT_TEST_HAVE_pipapo=y
»·······fi

above.

Re: [PATCH nft 3/4] tests: shell: skip pipapo set backend in transactions/30s-stress

[Pablo Neira Ayuso]

On Wed, Nov 15, 2023 at 11:25:44AM +0100, Thomas Haller wrote:
> On Wed, 2023-11-15 at 10:42 +0100, Pablo Neira Ayuso wrote:
> > 
> >  
> > +if [ "$NFT_TEST_HAVE_pipapo" != y ] ;then
> > +	echo "Skipping pipapo set backend, kernel does not support
> > it"
> > +fi
> 
> It's good and useful to run a reduced subset of the test, if some
> kernel feature is missing.
> 
> But in that case, the end of the test should have something like
> 
>   if [ "$NFT_TEST_HAVE_pipapo" = n ] ; then
>       echo "Ran a partial test only, due to NFT_TEST_HAVE_pipapo=n"
>       exit 77
>   fi
> 
> so that it shows up as skipped. In other words, "partially skipped"
> should also show up as skipped.

I don't want this.

This test is very useful in older kernels to catch bugs, I don't want
to see a SKIPPED here.

I prefer the tests autoadapts itself to what the kernel provides.

[PATCH nft 4/4] tests: shell: restore pipapo and chain binding coverage in standalone 30s-stress

[Pablo Neira Ayuso]

Do not disable pipapo and chain binding coverage for standalone runs by
default. Instead, turn them on by default and allow users to disable them
through:

 # export NFT_TEST_HAVE_chain_binding=n; bash tests/shell/testcases/transactions/30s-stress 3600
 ...
 running standalone with:
 NFT_TEST_HAVE_chain_binding=n
 NFT_TEST_HAVE_pipapo=y

given feature detection is not available in this case, thus, user has to
provide an explicit hint on what this kernel supports.

Fixes: c5b5b1044fdd ("tests/shell: add feature probing via "features/*.nft" files")
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/testcases/transactions/30s-stress | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/shell/testcases/transactions/30s-stress b/tests/shell/testcases/transactions/30s-stress
index b6ad06abed32..5c0a9465711b 100755
--- a/tests/shell/testcases/transactions/30s-stress
+++ b/tests/shell/testcases/transactions/30s-stress
@@ -6,6 +6,15 @@ runtime=30
 
 # allow stand-alone execution as well, e.g. '$0 3600'
 if [ x"$1" != "x" ] ;then
+	echo "running standalone with:"
+	if [ -z "${NFT_TEST_HAVE_chain_binding+x}" ]; then
+		NFT_TEST_HAVE_chain_binding=y
+	fi
+	if [ -z "${NFT_TEST_HAVE_pipapo+x}" ]; then
+		NFT_TEST_HAVE_pipapo=y
+	fi
+	echo "NFT_TEST_HAVE_chain_binding="$NFT_TEST_HAVE_chain_binding
+	echo "NFT_TEST_HAVE_pipapo="$NFT_TEST_HAVE_pipapo
 	if [ $1 -ge 0 ]; then
 		runtime="$1"
 	else
-- 
2.30.2