[PATCH nft 00/12] update tests/shell for 5.4 kernels

[PATCH nft 00/12] update tests/shell for 5.4 kernels

[Pablo Neira Ayuso]

Hi,

This is still not complete, still 8 tests/shell fail in 5.4 related to
flowtable features that are missing, but this batch is already getting large.

This is the series:

1) export DIFF so it can be used from feature scripts.
2) Detect if pipapo set backend is present.
3) Detect if reject from prerouting chain support is present.
4) Detect if stateful expression in sets is supported.
5) Detect if NAT netmap support is present.
6) Detect if multidevice netdev chain support is present.
7) Split several tests not to lose coverage, usually those containing
   set intervals and set intervals with concatenations (pipapo). It is
   a bit of sledgehammer solution but for tests/shell I think it is fine.

I can follow up with a more complete series contains all feature detection
for 5.4 or apply incrementally, just let me know.

Thanks.

Pablo Neira Ayuso (12):
  tests: shell: export DIFF to use it from feature scripts
  tests: shell: skip pipapo tests if kernel lacks support
  tests: shell: skip prerouting reject tests if kernel lacks support
  tests: shell: skip stateful expression in sets tests if kernel lacks support
  tests: shell: skip NAT netmap tests if kernel lacks support
  tests: shell: skip comment tests if kernel lacks support
  tests: shell: skip multidevice chain tests if kernel lacks support
  tests: shell: skip if kernel does not support bitshift
  tests: shell: split set NAT interval test
  tests: shell: split map test
  tests: shell: split single element in anonymous set
  tests: shell: split merge nat optimization in two tests

 tests/shell/features/comment.sh               | 11 +++
 .../features/netdev_chain_multidevice.sh      | 14 +++
 tests/shell/features/netmap.nft               |  8 ++
 tests/shell/features/pipapo.nft               |  9 ++
 tests/shell/features/prerouting_reject.nft    |  8 ++
 tests/shell/features/set_expr.sh              | 19 ++++
 tests/shell/run-tests.sh                      | 11 +--
 .../testcases/chains/0042chain_variable_0     |  2 +
 tests/shell/testcases/json/0002table_map_0    |  1 +
 tests/shell/testcases/json/0006obj_comment_0  |  1 +
 tests/shell/testcases/maps/0009vmap_0         |  2 +
 tests/shell/testcases/maps/0012map_0          | 19 ----
 tests/shell/testcases/maps/0012map_concat_0   | 24 ++++++
 tests/shell/testcases/maps/0013map_0          |  2 +
 tests/shell/testcases/maps/anon_objmap_concat |  2 +
 .../shell/testcases/maps/dumps/0012map_0.nft  | 13 ---
 .../testcases/maps/dumps/0012map_concat_0.nft | 14 +++
 tests/shell/testcases/maps/typeof_integer_0   |  2 +
 .../shell/testcases/maps/vmap_mark_bitwise_0  |  2 +
 .../optimizations/dumps/merge_nat.nft         |  8 --
 .../optimizations/dumps/merge_nat_concat.nft  |  8 ++
 .../optimizations/dumps/single_anon_set.nft   |  1 -
 .../dumps/single_anon_set_expr.nft            |  5 ++
 tests/shell/testcases/optimizations/merge_nat | 13 ---
 .../testcases/optimizations/merge_nat_concat  | 18 ++++
 .../optimizations/merge_stmts_concat          |  2 +
 .../testcases/optimizations/merge_stmts_vmap  |  2 +
 .../testcases/optimizations/merge_vmap_raw    |  2 +
 tests/shell/testcases/optimizations/ruleset   |  2 +
 .../testcases/optimizations/single_anon_set   |  3 -
 .../optimizations/single_anon_set_expr        | 26 ++++++
 .../testcases/optionals/comments_chain_0      |  2 +
 .../testcases/optionals/comments_objects_0    |  2 +
 .../testcases/optionals/comments_table_0      |  2 +
 tests/shell/testcases/sets/0020comments_0     |  2 +
 tests/shell/testcases/sets/0034get_element_0  |  2 +
 .../testcases/sets/0043concatenated_ranges_0  |  1 +
 .../testcases/sets/0043concatenated_ranges_1  |  2 +
 .../testcases/sets/0044interval_overlap_0     | 12 ++-
 tests/shell/testcases/sets/0046netmap_0       |  2 +
 tests/shell/testcases/sets/0047nat_0          |  2 +
 tests/shell/testcases/sets/0048set_counters_0 |  2 +
 .../testcases/sets/0051set_interval_counter_0 |  2 +
 .../testcases/sets/0067nat_concat_interval_0  | 17 +---
 tests/shell/testcases/sets/0067nat_interval_0 | 18 ++++
 tests/shell/testcases/sets/concat_interval_0  |  2 +
 .../sets/dumps/0067nat_concat_interval_0.nft  |  7 --
 .../sets/dumps/0067nat_interval_0.nft         | 12 +++
 tests/shell/testcases/sets/elem_opts_compat_0 |  2 +
 tests/shell/testcases/sets/typeof_sets_0      | 86 +++++++++----------
 50 files changed, 302 insertions(+), 129 deletions(-)
 create mode 100755 tests/shell/features/comment.sh
 create mode 100755 tests/shell/features/netdev_chain_multidevice.sh
 create mode 100644 tests/shell/features/netmap.nft
 create mode 100644 tests/shell/features/pipapo.nft
 create mode 100644 tests/shell/features/prerouting_reject.nft
 create mode 100755 tests/shell/features/set_expr.sh
 create mode 100755 tests/shell/testcases/maps/0012map_concat_0
 create mode 100644 tests/shell/testcases/maps/dumps/0012map_concat_0.nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/merge_nat_concat.nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/single_anon_set_expr.nft
 create mode 100755 tests/shell/testcases/optimizations/merge_nat_concat
 create mode 100755 tests/shell/testcases/optimizations/single_anon_set_expr
 create mode 100755 tests/shell/testcases/sets/0067nat_interval_0
 create mode 100644 tests/shell/testcases/sets/dumps/0067nat_interval_0.nft

-- 
2.30.2

[PATCH nft 01/12] tests: shell: export DIFF to use it from feature scripts

[Pablo Neira Ayuso]

export DIFF so it can be used from feature scripts to probe the kernel.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/run-tests.sh | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 27a0ec43042a..e51d51c9539b 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -572,6 +572,12 @@ feature_probe()
 	return 1
 }
 
+DIFF="$(which diff)"
+if [ ! -x "$DIFF" ] ; then
+	DIFF=true
+fi
+export DIFF
+
 for feat in "${_HAVE_OPTS[@]}" ; do
 	var="NFT_TEST_HAVE_$feat"
 	if [ -z "${!var+x}" ] ; then
@@ -590,11 +596,6 @@ if [ "$NFT_TEST_JOBS" -eq 0 ] ; then
 	fi
 fi
 
-DIFF="$(which diff)"
-if [ ! -x "$DIFF" ] ; then
-	DIFF=true
-fi
-
 declare -A JOBS_PIDLIST
 
 _NFT_TEST_VALGRIND_VGDB_PREFIX=
-- 
2.30.2

[PATCH nft 02/12] tests: shell: skip pipapo tests if kernel lacks support

[Pablo Neira Ayuso]

Skip tests that require net/netfilter/nft_set_pipapo support.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/features/pipapo.nft                      |  9 +++++++++
 tests/shell/testcases/maps/0013map_0                 |  2 ++
 tests/shell/testcases/maps/anon_objmap_concat        |  2 ++
 tests/shell/testcases/maps/typeof_integer_0          |  2 ++
 .../shell/testcases/optimizations/merge_stmts_concat |  2 ++
 tests/shell/testcases/optimizations/merge_vmap_raw   |  2 ++
 tests/shell/testcases/sets/0034get_element_0         |  2 ++
 tests/shell/testcases/sets/0043concatenated_ranges_0 |  1 +
 tests/shell/testcases/sets/0043concatenated_ranges_1 |  2 ++
 tests/shell/testcases/sets/0044interval_overlap_0    | 12 ++++++++++--
 tests/shell/testcases/sets/0047nat_0                 |  2 ++
 tests/shell/testcases/sets/concat_interval_0         |  2 ++
 12 files changed, 38 insertions(+), 2 deletions(-)
 create mode 100644 tests/shell/features/pipapo.nft

diff --git a/tests/shell/features/pipapo.nft b/tests/shell/features/pipapo.nft
new file mode 100644
index 000000000000..17b56f2210d4
--- /dev/null
+++ b/tests/shell/features/pipapo.nft
@@ -0,0 +1,9 @@
+# aaa31047a6d2 ("netfilter: nftables: add catch-all set element support")
+# v5.13-rc1~94^2~10^2~2
+table t {
+	set s {
+		type ipv4_addr . inet_service
+		flags interval
+		elements = { 1.1.1.1-2.2.2.2 . 80-90 }
+	}
+}
diff --git a/tests/shell/testcases/maps/0013map_0 b/tests/shell/testcases/maps/0013map_0
index 70d7fd3b002f..c8d20cee7ca7 100755
--- a/tests/shell/testcases/maps/0013map_0
+++ b/tests/shell/testcases/maps/0013map_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
 set -e
 
 RULESET="
diff --git a/tests/shell/testcases/maps/anon_objmap_concat b/tests/shell/testcases/maps/anon_objmap_concat
index 07820b7c4fdd..34465f1da0be 100755
--- a/tests/shell/testcases/maps/anon_objmap_concat
+++ b/tests/shell/testcases/maps/anon_objmap_concat
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
 set -e
 dumpfile=$(dirname $0)/dumps/$(basename $0).nft
 
diff --git a/tests/shell/testcases/maps/typeof_integer_0 b/tests/shell/testcases/maps/typeof_integer_0
index 0deff5eef67b..e93604e849c7 100755
--- a/tests/shell/testcases/maps/typeof_integer_0
+++ b/tests/shell/testcases/maps/typeof_integer_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
 EXPECTED="table inet t {
 	map m1 {
 		typeof udp length . @ih,32,32 : verdict
diff --git a/tests/shell/testcases/optimizations/merge_stmts_concat b/tests/shell/testcases/optimizations/merge_stmts_concat
index 9679d86223fd..4db4a6f90944 100755
--- a/tests/shell/testcases/optimizations/merge_stmts_concat
+++ b/tests/shell/testcases/optimizations/merge_stmts_concat
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
 set -e
 
 RULESET="table ip x {
diff --git a/tests/shell/testcases/optimizations/merge_vmap_raw b/tests/shell/testcases/optimizations/merge_vmap_raw
index f3dc0721b94f..eb04bec3ae69 100755
--- a/tests/shell/testcases/optimizations/merge_vmap_raw
+++ b/tests/shell/testcases/optimizations/merge_vmap_raw
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
 set -e
 
 RULESET="table inet x {
diff --git a/tests/shell/testcases/sets/0034get_element_0 b/tests/shell/testcases/sets/0034get_element_0
index 3343529b8ffa..32375b9f50c2 100755
--- a/tests/shell/testcases/sets/0034get_element_0
+++ b/tests/shell/testcases/sets/0034get_element_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
 RC=0
 
 check() { # (set, elems, expected)
diff --git a/tests/shell/testcases/sets/0043concatenated_ranges_0 b/tests/shell/testcases/sets/0043concatenated_ranges_0
index 83d743503c7b..a3dbf5bf28ba 100755
--- a/tests/shell/testcases/sets/0043concatenated_ranges_0
+++ b/tests/shell/testcases/sets/0043concatenated_ranges_0
@@ -1,5 +1,6 @@
 #!/bin/bash -e
 #
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
 # NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
 #
 # 0043concatenated_ranges_0 - Add, get, list, timeout for concatenated ranges
diff --git a/tests/shell/testcases/sets/0043concatenated_ranges_1 b/tests/shell/testcases/sets/0043concatenated_ranges_1
index 1be2889352c9..bb3bf6b27ea7 100755
--- a/tests/shell/testcases/sets/0043concatenated_ranges_1
+++ b/tests/shell/testcases/sets/0043concatenated_ranges_1
@@ -2,6 +2,8 @@
 #
 # 0043concatenated_ranges_1 - Insert and list subnets of different sizes
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
 check() {
 	$NFT add element "${1}" t s "{ ${2} . ${3} }"
 	[ "$( $NFT list set "${1}" t s | grep -c "${2} . ${3}" )" = 1 ]
diff --git a/tests/shell/testcases/sets/0044interval_overlap_0 b/tests/shell/testcases/sets/0044interval_overlap_0
index 71bf3345a558..b0f51cc8873b 100755
--- a/tests/shell/testcases/sets/0044interval_overlap_0
+++ b/tests/shell/testcases/sets/0044interval_overlap_0
@@ -117,7 +117,11 @@ add_elements() {
 	IFS='	
 '
 	for t in ${intervals_simple} switch ${intervals_concat}; do
+if [ "$NFT_TEST_HAVE_pipapo" = y ] ; then
 		[ "${t}" = "switch" ] && set="c"         && continue
+else
+		break
+fi
 		[ -z "${pass}" ]      && pass="${t}"     && continue
 		[ -z "${interval}" ]  && interval="${t}" && continue
 		unset IFS
@@ -148,7 +152,9 @@ add_elements() {
 
 $NFT add table t
 $NFT add set t s '{ type inet_service ; flags interval ; }'
-$NFT add set t c '{ type inet_service . inet_service ; flags interval ; }'
+if [ "$NFT_TEST_HAVE_pipapo" = y ] ; then
+	$NFT add set t c '{ type inet_service . inet_service ; flags interval ; }'
+fi
 add_elements
 
 $NFT flush ruleset
@@ -157,7 +163,9 @@ estimate_timeout
 $NFT flush ruleset
 $NFT add table t
 $NFT add set t s "{ type inet_service ; flags interval,timeout; timeout ${timeout}s; gc-interval ${timeout}s; }"
-$NFT add set t c "{ type inet_service . inet_service ; flags interval,timeout ; timeout ${timeout}s; gc-interval ${timeout}s; }"
+if [ "$NFT_TEST_HAVE_pipapo" = y ] ; then
+	$NFT add set t c "{ type inet_service . inet_service ; flags interval,timeout ; timeout ${timeout}s; gc-interval ${timeout}s; }"
+fi
 add_elements
 
 sleep $((timeout * 3 / 2))
diff --git a/tests/shell/testcases/sets/0047nat_0 b/tests/shell/testcases/sets/0047nat_0
index 4e53b7b8e8c8..757605ee3492 100755
--- a/tests/shell/testcases/sets/0047nat_0
+++ b/tests/shell/testcases/sets/0047nat_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
 EXPECTED="table ip x {
             map y {
                     type ipv4_addr : interval ipv4_addr
diff --git a/tests/shell/testcases/sets/concat_interval_0 b/tests/shell/testcases/sets/concat_interval_0
index 4d90af9a6557..36138ae0de78 100755
--- a/tests/shell/testcases/sets/concat_interval_0
+++ b/tests/shell/testcases/sets/concat_interval_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
 set -e
 
 RULESET="table ip t {
-- 
2.30.2

[PATCH nft 03/12] tests: shell: skip prerouting reject tests if kernel lacks support

[Pablo Neira Ayuso]

Skip tests that require reject at prerouting hook.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/features/prerouting_reject.nft  | 8 ++++++++
 tests/shell/testcases/optimizations/ruleset | 2 ++
 2 files changed, 10 insertions(+)
 create mode 100644 tests/shell/features/prerouting_reject.nft

diff --git a/tests/shell/features/prerouting_reject.nft b/tests/shell/features/prerouting_reject.nft
new file mode 100644
index 000000000000..26098bb54534
--- /dev/null
+++ b/tests/shell/features/prerouting_reject.nft
@@ -0,0 +1,8 @@
+# f53b9b0bdc59 netfilter: introduce support for reject at prerouting stage
+# v5.13-rc1~94^2~10^2~2
+table inet t {
+	chain nat_filter {
+		type filter hook prerouting priority 0; policy accept;
+		reject with icmpx type host-unreachable
+	}
+}
diff --git a/tests/shell/testcases/optimizations/ruleset b/tests/shell/testcases/optimizations/ruleset
index ef2652dbeae8..2b2d80ffc009 100755
--- a/tests/shell/testcases/optimizations/ruleset
+++ b/tests/shell/testcases/optimizations/ruleset
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_prerouting_reject)
+
 RULESET="table inet uni {
 	chain gtfo {
 		reject with icmpx type host-unreachable
-- 
2.30.2

[PATCH nft 04/12] tests: shell: skip stateful expression in sets tests if kernel lacks support

[Pablo Neira Ayuso]

Skip tests that require stateful expressions in sets.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/features/set_expr.sh              | 19 +++++++++++++++++++
 tests/shell/testcases/json/0002table_map_0    |  1 +
 tests/shell/testcases/maps/0009vmap_0         |  2 ++
 .../testcases/optimizations/merge_stmts_vmap  |  2 ++
 tests/shell/testcases/sets/0048set_counters_0 |  2 ++
 .../testcases/sets/0051set_interval_counter_0 |  2 ++
 tests/shell/testcases/sets/elem_opts_compat_0 |  2 ++
 7 files changed, 30 insertions(+)
 create mode 100755 tests/shell/features/set_expr.sh

diff --git a/tests/shell/features/set_expr.sh b/tests/shell/features/set_expr.sh
new file mode 100755
index 000000000000..c323d59e8920
--- /dev/null
+++ b/tests/shell/features/set_expr.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+# 65038428b2c6 ("netfilter: nf_tables: allow to specify stateful expression in set definition")
+# v5.6-rc5-1736-g65038428b2c6
+
+# NFT_SET_EXPR to detect kernel feature only available since
+# b4e70d8dd9ea ("netfilter: nftables: add set expression flags")
+# v5.10-11680-gb4e70d8dd9ea
+
+EXPECTED="table ip x {
+	set y {
+		typeof ip saddr
+		counter
+	}
+}"
+
+$NFT -f - <<< $EXPECTED
+
+$DIFF -u <($NFT list ruleset) - <<<"$EXPECTED"
diff --git a/tests/shell/testcases/json/0002table_map_0 b/tests/shell/testcases/json/0002table_map_0
index b375e9969608..a1e9f2634978 100755
--- a/tests/shell/testcases/json/0002table_map_0
+++ b/tests/shell/testcases/json/0002table_map_0
@@ -1,6 +1,7 @@
 #!/bin/bash
 
 # NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_set_expr)
 
 set -e
 
diff --git a/tests/shell/testcases/maps/0009vmap_0 b/tests/shell/testcases/maps/0009vmap_0
index d31e1608f792..4e133b72f6ef 100755
--- a/tests/shell/testcases/maps/0009vmap_0
+++ b/tests/shell/testcases/maps/0009vmap_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_set_expr)
+
 set -e
 
 EXPECTED="table inet filter {
diff --git a/tests/shell/testcases/optimizations/merge_stmts_vmap b/tests/shell/testcases/optimizations/merge_stmts_vmap
index 6e0f0762b7bb..e5357c0f66b6 100755
--- a/tests/shell/testcases/optimizations/merge_stmts_vmap
+++ b/tests/shell/testcases/optimizations/merge_stmts_vmap
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_set_expr)
+
 set -e
 
 RULESET="table ip x {
diff --git a/tests/shell/testcases/sets/0048set_counters_0 b/tests/shell/testcases/sets/0048set_counters_0
index e62d25df799c..95babdc9ca5f 100755
--- a/tests/shell/testcases/sets/0048set_counters_0
+++ b/tests/shell/testcases/sets/0048set_counters_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_set_expr)
+
 set -e
 
 EXPECTED="table ip x {
diff --git a/tests/shell/testcases/sets/0051set_interval_counter_0 b/tests/shell/testcases/sets/0051set_interval_counter_0
index ea90e264bfcc..6e67a43c577a 100755
--- a/tests/shell/testcases/sets/0051set_interval_counter_0
+++ b/tests/shell/testcases/sets/0051set_interval_counter_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_set_expr)
+
 set -e
 
 EXPECTED="table ip x {
diff --git a/tests/shell/testcases/sets/elem_opts_compat_0 b/tests/shell/testcases/sets/elem_opts_compat_0
index 3467cc07e646..7563773e626f 100755
--- a/tests/shell/testcases/sets/elem_opts_compat_0
+++ b/tests/shell/testcases/sets/elem_opts_compat_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_set_expr)
+
 # ordering of element options and expressions has changed, make sure parser
 # accepts both ways
 
-- 
2.30.2

[PATCH nft 05/12] tests: shell: skip NAT netmap tests if kernel lacks support

[Pablo Neira Ayuso]

Skip tests that require NAT netmap support

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/features/netmap.nft         | 8 ++++++++
 tests/shell/testcases/sets/0046netmap_0 | 2 ++
 2 files changed, 10 insertions(+)
 create mode 100644 tests/shell/features/netmap.nft

diff --git a/tests/shell/features/netmap.nft b/tests/shell/features/netmap.nft
new file mode 100644
index 000000000000..129ee243e2eb
--- /dev/null
+++ b/tests/shell/features/netmap.nft
@@ -0,0 +1,8 @@
+# 3ff7ddb1353d ("netfilter: nft_nat: add netmap support")
+# v5.7-rc2-635-g3ff7ddb1353d
+table ip x {
+        chain y {
+              type nat hook postrouting priority srcnat; policy accept;
+              snat ip prefix to ip saddr map { 10.141.11.0/24 : 192.168.2.0/24 }
+        }
+}
diff --git a/tests/shell/testcases/sets/0046netmap_0 b/tests/shell/testcases/sets/0046netmap_0
index 60bda4017c59..7533623e7f7b 100755
--- a/tests/shell/testcases/sets/0046netmap_0
+++ b/tests/shell/testcases/sets/0046netmap_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_netmap)
+
 EXPECTED="table ip x {
             chain y {
                     type nat hook postrouting priority srcnat; policy accept;
-- 
2.30.2

[PATCH nft 06/12] tests: shell: skip comment tests if kernel lacks support

[Pablo Neira Ayuso]

Skip tests that require comment support

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/features/comment.sh                    | 11 +++++++++++
 tests/shell/testcases/json/0006obj_comment_0       |  1 +
 tests/shell/testcases/optionals/comments_chain_0   |  2 ++
 tests/shell/testcases/optionals/comments_objects_0 |  2 ++
 tests/shell/testcases/optionals/comments_table_0   |  2 ++
 tests/shell/testcases/sets/0020comments_0          |  2 ++
 6 files changed, 20 insertions(+)
 create mode 100755 tests/shell/features/comment.sh

diff --git a/tests/shell/features/comment.sh b/tests/shell/features/comment.sh
new file mode 100755
index 000000000000..516cf5e183ad
--- /dev/null
+++ b/tests/shell/features/comment.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+EXPECTED="table ip x {
+	chain y {
+		comment \"test\"
+	}
+}"
+
+$NFT -f - <<< $EXPECTED
+
+$DIFF -u <($NFT list ruleset) - <<<"$EXPECTED"
diff --git a/tests/shell/testcases/json/0006obj_comment_0 b/tests/shell/testcases/json/0006obj_comment_0
index 4c2a0e8c0880..7ce859d2529f 100755
--- a/tests/shell/testcases/json/0006obj_comment_0
+++ b/tests/shell/testcases/json/0006obj_comment_0
@@ -1,6 +1,7 @@
 #!/bin/bash
 
 # NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_comment)
 
 set -e
 
diff --git a/tests/shell/testcases/optionals/comments_chain_0 b/tests/shell/testcases/optionals/comments_chain_0
index fba961c76841..1a84cfa67a2a 100755
--- a/tests/shell/testcases/optionals/comments_chain_0
+++ b/tests/shell/testcases/optionals/comments_chain_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_comment)
+
 EXPECTED='table ip test_table {
 	chain test_chain {
 		comment "test"
diff --git a/tests/shell/testcases/optionals/comments_objects_0 b/tests/shell/testcases/optionals/comments_objects_0
index 301f5518fb80..28041ebd2a43 100755
--- a/tests/shell/testcases/optionals/comments_objects_0
+++ b/tests/shell/testcases/optionals/comments_objects_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_comment)
+
 set -e
 
 COMMENT128="12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678"
diff --git a/tests/shell/testcases/optionals/comments_table_0 b/tests/shell/testcases/optionals/comments_table_0
index a0dfd7494661..56bb206bddcf 100755
--- a/tests/shell/testcases/optionals/comments_table_0
+++ b/tests/shell/testcases/optionals/comments_table_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_comment)
+
 # comments are shown
 
 $NFT add table test { comment \"test_comment\"\; }
diff --git a/tests/shell/testcases/sets/0020comments_0 b/tests/shell/testcases/sets/0020comments_0
index 44d451a8ad3a..1df38326ab57 100755
--- a/tests/shell/testcases/sets/0020comments_0
+++ b/tests/shell/testcases/sets/0020comments_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_comment)
+
 # Test that comments are added to set elements in standard sets.
 # Explicitly test bitmap backend set implementation.
 
-- 
2.30.2

[PATCH nft 07/12] tests: shell: skip multidevice chain tests if kernel lacks support

[Pablo Neira Ayuso]

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/features/netdev_chain_multidevice.sh  | 14 ++++++++++++++
 tests/shell/testcases/chains/0042chain_variable_0 |  2 ++
 2 files changed, 16 insertions(+)
 create mode 100755 tests/shell/features/netdev_chain_multidevice.sh

diff --git a/tests/shell/features/netdev_chain_multidevice.sh b/tests/shell/features/netdev_chain_multidevice.sh
new file mode 100755
index 000000000000..07f892035301
--- /dev/null
+++ b/tests/shell/features/netdev_chain_multidevice.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+
+trap "ip link del d0; ip link del d1" EXIT
+
+ip link add d0 type dummy
+ip link add d1 type dummy
+
+EXPECTED="table netdev filter2 {
+        chain Main_Ingress2 {
+                type filter hook ingress devices = { \"d0\", \"d1\" } priority -500; policy accept;
+        }
+}"
+
+$NFT -f - <<< $EXPECTED
diff --git a/tests/shell/testcases/chains/0042chain_variable_0 b/tests/shell/testcases/chains/0042chain_variable_0
index a4b929f7344c..c5de495ef074 100755
--- a/tests/shell/testcases/chains/0042chain_variable_0
+++ b/tests/shell/testcases/chains/0042chain_variable_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_netdev_chain_multidevice)
+
 set -e
 
 ip link add name d23456789012345 type dummy
-- 
2.30.2

[PATCH nft 08/12] tests: shell: skip if kernel does not support bitshift

[Pablo Neira Ayuso]

A few tests are missing bitshift checks that has been added to
885845468408 ("tests/shell: skip bitshift tests if kernel lacks
support").

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 .../shell/testcases/maps/vmap_mark_bitwise_0  |  2 +
 tests/shell/testcases/sets/typeof_sets_0      | 86 +++++++++----------
 2 files changed, 45 insertions(+), 43 deletions(-)

diff --git a/tests/shell/testcases/maps/vmap_mark_bitwise_0 b/tests/shell/testcases/maps/vmap_mark_bitwise_0
index 0d933553e6b8..2f305b27bc33 100755
--- a/tests/shell/testcases/maps/vmap_mark_bitwise_0
+++ b/tests/shell/testcases/maps/vmap_mark_bitwise_0
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_bitshift)
+
 set -e
 
 RULESET="table ip x {
diff --git a/tests/shell/testcases/sets/typeof_sets_0 b/tests/shell/testcases/sets/typeof_sets_0
index 35c572c1e537..92555a1f923e 100755
--- a/tests/shell/testcases/sets/typeof_sets_0
+++ b/tests/shell/testcases/sets/typeof_sets_0
@@ -17,22 +17,53 @@ INPUT_OSF_SET="
 		elements = { \"Linux\" }
 	}
 "
+
+INPUT_FRAG_SET="
+	set s4 {
+		typeof frag frag-off
+		elements = { 1, 1024 }
+	}
+"
+
+INPUT_VERSION_SET="
+	set s8 {
+		typeof ip version
+		elements = { 4, 6 }
+	}
+"
+
 INPUT_OSF_CHAIN="
 	chain c1 {
 		osf name @s1 accept
 	}
 "
 
+INPUT_FRAG_CHAIN="
+	chain c4 {
+		frag frag-off @s4 accept
+	}
+"
+
 INPUT_SCTP_CHAIN="
 	chain c7 {
 		sctp chunk init num-inbound-streams @s7 accept
 	}
 "
+INPUT_VERSION_CHAIN="
+	chain c8 {
+		ip version @s8 accept
+	}
+"
 
 if [ "$NFT_TEST_HAVE_sctp_chunks" = n ] ; then
 	INPUT_SCTP_CHAIN=
 fi
 
+if [ "$NFT_TEST_HAVE_bitshift" = n ] ; then
+	INPUT_FRAG_CHAIN=
+	INPUT_VERSION_CHAIN=
+fi
+
 if [ "$NFT_TEST_HAVE_osf" = n ] ; then
 	if [ "$((RANDOM % 2))" -eq 1 ] ; then
 		# Regardless of $NFT_TEST_HAVE_osf, we can define the set.
@@ -51,12 +82,7 @@ INPUT="table inet t {$INPUT_OSF_SET
 	set s3 {
 		typeof meta ibrpvid
 		elements = { 2, 3, 103 }
-	}
-
-	set s4 {
-		typeof frag frag-off
-		elements = { 1, 1024 }
-	}
+	}$INPUT_FRAG_SET
 
 	set s5 {
 		typeof ip option ra value
@@ -71,12 +97,7 @@ INPUT="table inet t {$INPUT_OSF_SET
 	set s7 {
 		typeof sctp chunk init num-inbound-streams
 		elements = { 1, 4 }
-	}
-
-	set s8 {
-		typeof ip version
-		elements = { 4, 6 }
-	}
+	}$INPUT_VERSION_SET
 
 	set s9 {
 		typeof ip hdrlength
@@ -96,11 +117,7 @@ $INPUT_OSF_CHAIN
 	chain c2 {
 		ether type vlan vlan id @s2 accept
 	}
-
-	chain c4 {
-		frag frag-off @s4 accept
-	}
-
+$INPUT_FRAG_CHAIN
 	chain c5 {
 		ip option ra value @s5 accept
 	}
@@ -109,10 +126,7 @@ $INPUT_OSF_CHAIN
 		tcp option maxseg size @s6 accept
 	}
 $INPUT_SCTP_CHAIN
-	chain c8 {
-		ip version @s8 accept
-	}
-
+$INPUT_VERSION_CHAIN
 	chain c9 {
 		ip hdrlength @s9 accept
 	}
@@ -136,12 +150,7 @@ EXPECTED="table inet t {$INPUT_OSF_SET
 		typeof meta ibrpvid
 		elements = { 2, 3, 103 }
 	}
-
-	set s4 {
-		typeof frag frag-off
-		elements = { 1, 1024 }
-	}
-
+$INPUT_FRAG_SET
 	set s5 {
 		typeof ip option ra value
 		elements = { 1, 1024 }
@@ -156,12 +165,7 @@ EXPECTED="table inet t {$INPUT_OSF_SET
 		typeof sctp chunk init num-inbound-streams
 		elements = { 1, 4 }
 	}
-
-	set s8 {
-		typeof ip version
-		elements = { 4, 6 }
-	}
-
+$INPUT_VERSION_SET
 	set s9 {
 		typeof ip hdrlength
 		elements = { 0, 1, 2, 3, 4,
@@ -181,11 +185,7 @@ $INPUT_OSF_CHAIN
 	chain c2 {
 		vlan id @s2 accept
 	}
-
-	chain c4 {
-		frag frag-off @s4 accept
-	}
-
+$INPUT_FRAG_CHAIN
 	chain c5 {
 		ip option ra value @s5 accept
 	}
@@ -193,11 +193,7 @@ $INPUT_OSF_CHAIN
 	chain c6 {
 		tcp option maxseg size @s6 accept
 	}
-$INPUT_SCTP_CHAIN
-	chain c8 {
-		ip version @s8 accept
-	}
-
+$INPUT_SCTP_CHAIN$INPUT_VERSION_CHAIN
 	chain c9 {
 		ip hdrlength @s9 accept
 	}
@@ -216,6 +212,10 @@ $NFT -f - <<< "$INPUT" || die $'nft command failed to process input:\n'">$INPUT<
 
 $DIFF -u <($NFT list ruleset) - <<<"$EXPECTED" || die $'diff failed between ruleset and expected data.\nExpected:\n'">$EXPECTED<"
 
+if [ "$NFT_TEST_HAVE_bitshift" = n ] ; then
+	echo "Partial test due to NFT_TEST_HAVE_bitshift=n. Skip"
+	exit 77
+fi
 if [ "$NFT_TEST_HAVE_osf" = n ] ; then
 	echo "Partial test due to NFT_TEST_HAVE_osf=n. Skip"
 	exit 77
-- 
2.30.2

[PATCH nft 09/12] tests: shell: split set NAT interval test

[Pablo Neira Ayuso]

Split test in two, one for interval sets and another with concatenation
+ intervals, so at least intervals are tested in older kernels with no
pipapo support.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 .../testcases/sets/0067nat_concat_interval_0   | 17 ++---------------
 tests/shell/testcases/sets/0067nat_interval_0  | 18 ++++++++++++++++++
 .../sets/dumps/0067nat_concat_interval_0.nft   |  7 -------
 .../sets/dumps/0067nat_interval_0.nft          | 12 ++++++++++++
 4 files changed, 32 insertions(+), 22 deletions(-)
 create mode 100755 tests/shell/testcases/sets/0067nat_interval_0
 create mode 100644 tests/shell/testcases/sets/dumps/0067nat_interval_0.nft

diff --git a/tests/shell/testcases/sets/0067nat_concat_interval_0 b/tests/shell/testcases/sets/0067nat_concat_interval_0
index 55cc0d4b43df..816219573870 100755
--- a/tests/shell/testcases/sets/0067nat_concat_interval_0
+++ b/tests/shell/testcases/sets/0067nat_concat_interval_0
@@ -1,21 +1,8 @@
 #!/bin/bash
 
-set -e
-
-EXPECTED="table ip nat {
-       map ipportmap {
-                type ipv4_addr : interval ipv4_addr . inet_service
-                flags interval
-                elements = { 192.168.1.2 : 10.141.10.1-10.141.10.3 . 8888-8999 }
-       }
-       chain prerouting {
-                type nat hook prerouting priority dstnat; policy accept;
-                ip protocol tcp dnat ip to ip saddr map @ipportmap
-       }
-}"
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
 
-$NFT -f - <<< $EXPECTED
-$NFT add element ip nat ipportmap { 192.168.2.0/24 : 10.141.11.5-10.141.11.20 . 8888-8999 }
+set -e
 
 EXPECTED="table ip nat {
         map ipportmap2 {
diff --git a/tests/shell/testcases/sets/0067nat_interval_0 b/tests/shell/testcases/sets/0067nat_interval_0
new file mode 100755
index 000000000000..c90203d0d648
--- /dev/null
+++ b/tests/shell/testcases/sets/0067nat_interval_0
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+set -e
+
+EXPECTED="table ip nat {
+       map ipportmap {
+                type ipv4_addr : interval ipv4_addr . inet_service
+                flags interval
+                elements = { 192.168.1.2 : 10.141.10.1-10.141.10.3 . 8888-8999 }
+       }
+       chain prerouting {
+                type nat hook prerouting priority dstnat; policy accept;
+                ip protocol tcp dnat ip to ip saddr map @ipportmap
+       }
+}"
+
+$NFT -f - <<< $EXPECTED
+$NFT add element ip nat ipportmap { 192.168.2.0/24 : 10.141.11.5-10.141.11.20 . 8888-8999 }
diff --git a/tests/shell/testcases/sets/dumps/0067nat_concat_interval_0.nft b/tests/shell/testcases/sets/dumps/0067nat_concat_interval_0.nft
index 0215691e28ee..9ac3774a7222 100644
--- a/tests/shell/testcases/sets/dumps/0067nat_concat_interval_0.nft
+++ b/tests/shell/testcases/sets/dumps/0067nat_concat_interval_0.nft
@@ -1,10 +1,4 @@
 table ip nat {
-	map ipportmap {
-		type ipv4_addr : interval ipv4_addr . inet_service
-		flags interval
-		elements = { 192.168.1.2 : 10.141.10.1-10.141.10.3 . 8888-8999, 192.168.2.0/24 : 10.141.11.5-10.141.11.20 . 8888-8999 }
-	}
-
 	map ipportmap2 {
 		type ipv4_addr . ipv4_addr : interval ipv4_addr . inet_service
 		flags interval
@@ -33,7 +27,6 @@ table ip nat {
 
 	chain prerouting {
 		type nat hook prerouting priority dstnat; policy accept;
-		ip protocol tcp dnat ip to ip saddr map @ipportmap
 		ip protocol tcp dnat ip to ip saddr . ip daddr map @ipportmap2
 		meta l4proto { tcp, udp } dnat ip to ip daddr . th dport map @fwdtoip_th
 		dnat ip to iifname . ip saddr map @ipportmap4
diff --git a/tests/shell/testcases/sets/dumps/0067nat_interval_0.nft b/tests/shell/testcases/sets/dumps/0067nat_interval_0.nft
new file mode 100644
index 000000000000..b6d07fcdc248
--- /dev/null
+++ b/tests/shell/testcases/sets/dumps/0067nat_interval_0.nft
@@ -0,0 +1,12 @@
+table ip nat {
+	map ipportmap {
+		type ipv4_addr : interval ipv4_addr . inet_service
+		flags interval
+		elements = { 192.168.1.2 : 10.141.10.1-10.141.10.3 . 8888-8999, 192.168.2.0/24 : 10.141.11.5-10.141.11.20 . 8888-8999 }
+	}
+
+	chain prerouting {
+		type nat hook prerouting priority dstnat; policy accept;
+		ip protocol tcp dnat ip to ip saddr map @ipportmap
+	}
+}
-- 
2.30.2

[PATCH nft 10/12] tests: shell: split map test

[Pablo Neira Ayuso]

Split interval + concatenation into a separated file, so older kernels
with no pipapo can still run what it is supported.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 tests/shell/testcases/maps/0012map_0          | 19 ---------------
 tests/shell/testcases/maps/0012map_concat_0   | 24 +++++++++++++++++++
 .../shell/testcases/maps/dumps/0012map_0.nft  | 13 ----------
 .../testcases/maps/dumps/0012map_concat_0.nft | 14 +++++++++++
 4 files changed, 38 insertions(+), 32 deletions(-)
 create mode 100755 tests/shell/testcases/maps/0012map_concat_0
 create mode 100644 tests/shell/testcases/maps/dumps/0012map_concat_0.nft

diff --git a/tests/shell/testcases/maps/0012map_0 b/tests/shell/testcases/maps/0012map_0
index 49e51b755b0f..dd93c482f441 100755
--- a/tests/shell/testcases/maps/0012map_0
+++ b/tests/shell/testcases/maps/0012map_0
@@ -15,22 +15,3 @@ table ip x {
 }"
 
 $NFT -f - <<< "$EXPECTED"
-
-EXPECTED="table ip x {
-	map w {
-		typeof ip saddr . meta mark : verdict
-		flags interval
-		counter
-		elements = {
-			127.0.0.1-127.0.0.4 . 0x123434-0xb00122 : accept,
-		}
-	}
-
-	chain k {
-		type filter hook input priority filter + 1; policy accept;
-		meta mark set 0x123434
-		ip saddr . meta mark vmap @w
-	}
-}"
-
-$NFT -f - <<< "$EXPECTED"
diff --git a/tests/shell/testcases/maps/0012map_concat_0 b/tests/shell/testcases/maps/0012map_concat_0
new file mode 100755
index 000000000000..d18c7a73c844
--- /dev/null
+++ b/tests/shell/testcases/maps/0012map_concat_0
@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
+set -e
+
+EXPECTED="table ip x {
+        map w {
+                typeof ip saddr . meta mark : verdict
+                flags interval
+                counter
+                elements = {
+                        127.0.0.1-127.0.0.4 . 0x123434-0xb00122 : accept,
+                }
+        }
+
+        chain k {
+                type filter hook input priority filter + 1; policy accept;
+                meta mark set 0x123434
+                ip saddr . meta mark vmap @w
+        }
+}"
+
+$NFT -f - <<< "$EXPECTED"
diff --git a/tests/shell/testcases/maps/dumps/0012map_0.nft b/tests/shell/testcases/maps/dumps/0012map_0.nft
index 895490cffa8c..e734fc1c70b9 100644
--- a/tests/shell/testcases/maps/dumps/0012map_0.nft
+++ b/tests/shell/testcases/maps/dumps/0012map_0.nft
@@ -6,20 +6,7 @@ table ip x {
 			     "eth1" : drop }
 	}
 
-	map w {
-		typeof ip saddr . meta mark : verdict
-		flags interval
-		counter
-		elements = { 127.0.0.1-127.0.0.4 . 0x00123434-0x00b00122 counter packets 0 bytes 0 : accept }
-	}
-
 	chain y {
 		iifname vmap { "lo" : accept, "eth0" : drop, "eth1" : drop }
 	}
-
-	chain k {
-		type filter hook input priority filter + 1; policy accept;
-		meta mark set 0x00123434
-		ip saddr . meta mark vmap @w
-	}
 }
diff --git a/tests/shell/testcases/maps/dumps/0012map_concat_0.nft b/tests/shell/testcases/maps/dumps/0012map_concat_0.nft
new file mode 100644
index 000000000000..6649d0342a28
--- /dev/null
+++ b/tests/shell/testcases/maps/dumps/0012map_concat_0.nft
@@ -0,0 +1,14 @@
+table ip x {
+	map w {
+		typeof ip saddr . meta mark : verdict
+		flags interval
+		counter
+		elements = { 127.0.0.1-127.0.0.4 . 0x00123434-0x00b00122 counter packets 0 bytes 0 : accept }
+	}
+
+	chain k {
+		type filter hook input priority filter + 1; policy accept;
+		meta mark set 0x00123434
+		ip saddr . meta mark vmap @w
+	}
+}
-- 
2.30.2

[PATCH nft 11/12] tests: shell: split single element in anonymous set

[Pablo Neira Ayuso]

Split this to move set stateful expression support into a separated test
not to harm existing coverage.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 .../optimizations/dumps/single_anon_set.nft   |  1 -
 .../dumps/single_anon_set_expr.nft            |  5 ++++
 .../testcases/optimizations/single_anon_set   |  3 ---
 .../optimizations/single_anon_set_expr        | 26 +++++++++++++++++++
 4 files changed, 31 insertions(+), 4 deletions(-)
 create mode 100644 tests/shell/testcases/optimizations/dumps/single_anon_set_expr.nft
 create mode 100755 tests/shell/testcases/optimizations/single_anon_set_expr

diff --git a/tests/shell/testcases/optimizations/dumps/single_anon_set.nft b/tests/shell/testcases/optimizations/dumps/single_anon_set.nft
index 3f703034d80f..35e3f36e1a54 100644
--- a/tests/shell/testcases/optimizations/dumps/single_anon_set.nft
+++ b/tests/shell/testcases/optimizations/dumps/single_anon_set.nft
@@ -11,6 +11,5 @@ table ip test {
 		ip daddr . tcp dport { 192.168.0.1 . 22 } accept
 		meta mark set ip daddr map { 192.168.0.1 : 0x00000001 }
 		ct state { established, related } accept
-		meta mark { 0x0000000a counter packets 0 bytes 0 }
 	}
 }
diff --git a/tests/shell/testcases/optimizations/dumps/single_anon_set_expr.nft b/tests/shell/testcases/optimizations/dumps/single_anon_set_expr.nft
new file mode 100644
index 000000000000..54880b927250
--- /dev/null
+++ b/tests/shell/testcases/optimizations/dumps/single_anon_set_expr.nft
@@ -0,0 +1,5 @@
+table ip test {
+	chain test {
+		meta mark { 0x0000000a counter packets 0 bytes 0 }
+	}
+}
diff --git a/tests/shell/testcases/optimizations/single_anon_set b/tests/shell/testcases/optimizations/single_anon_set
index 84fc2a7f03a8..632e965f37a7 100755
--- a/tests/shell/testcases/optimizations/single_anon_set
+++ b/tests/shell/testcases/optimizations/single_anon_set
@@ -46,9 +46,6 @@ table ip test {
 		# ct state cannot be both established and related
 		# at the same time, but this needs extra work.
 		ct state { established, related } accept
-
-		# with stateful statement
-		meta mark { 0x0000000a counter }
 	}
 }
 EOF
diff --git a/tests/shell/testcases/optimizations/single_anon_set_expr b/tests/shell/testcases/optimizations/single_anon_set_expr
new file mode 100755
index 000000000000..81b7cebadd5d
--- /dev/null
+++ b/tests/shell/testcases/optimizations/single_anon_set_expr
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_set_expr)
+
+set -e
+
+test -d "$NFT_TEST_TESTTMPDIR"
+
+# Input file contains rules with anon sets that contain
+# one element, plus extra rule with two elements (that should be
+# left alone).
+
+# Dump file has the simplified rules where anon sets have been
+# replaced by equality tests where possible.
+file_input1="$NFT_TEST_TESTTMPDIR/input1.nft"
+
+cat <<EOF > "$file_input1"
+table ip test {
+	chain test {
+		# with stateful statement
+		meta mark { 0x0000000a counter }
+	}
+}
+EOF
+
+$NFT -f "$file_input1"
-- 
2.30.2

[PATCH nft 12/12] tests: shell: split merge nat optimization in two tests

[Pablo Neira Ayuso]

One without pipapo support and another with not to harm existing
coverage.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 .../optimizations/dumps/merge_nat.nft          |  8 --------
 .../optimizations/dumps/merge_nat_concat.nft   |  8 ++++++++
 tests/shell/testcases/optimizations/merge_nat  | 13 -------------
 .../testcases/optimizations/merge_nat_concat   | 18 ++++++++++++++++++
 4 files changed, 26 insertions(+), 21 deletions(-)
 create mode 100644 tests/shell/testcases/optimizations/dumps/merge_nat_concat.nft
 create mode 100755 tests/shell/testcases/optimizations/merge_nat_concat

diff --git a/tests/shell/testcases/optimizations/dumps/merge_nat.nft b/tests/shell/testcases/optimizations/dumps/merge_nat.nft
index 48d18a676ee0..61feb278d5e6 100644
--- a/tests/shell/testcases/optimizations/dumps/merge_nat.nft
+++ b/tests/shell/testcases/optimizations/dumps/merge_nat.nft
@@ -11,14 +11,6 @@ table ip test2 {
 		ip saddr { 10.141.11.0/24, 10.141.13.0/24 } masquerade
 	}
 }
-table ip test3 {
-	chain y {
-		oif "lo" accept
-		snat to ip saddr . tcp sport map { 1.1.1.1 . 1024-65535 : 3.3.3.3, 2.2.2.2 . 1024-65535 : 4.4.4.4 }
-		oifname "enp2s0" snat ip to ip saddr map { 10.1.1.0/24 : 72.2.3.66-72.2.3.78 }
-		tcp dport { 8888, 9999 } redirect
-	}
-}
 table ip test4 {
 	chain y {
 		oif "lo" accept
diff --git a/tests/shell/testcases/optimizations/dumps/merge_nat_concat.nft b/tests/shell/testcases/optimizations/dumps/merge_nat_concat.nft
new file mode 100644
index 000000000000..0faddfd14fb8
--- /dev/null
+++ b/tests/shell/testcases/optimizations/dumps/merge_nat_concat.nft
@@ -0,0 +1,8 @@
+table ip test3 {
+	chain y {
+		oif "lo" accept
+		snat to ip saddr . tcp sport map { 1.1.1.1 . 1024-65535 : 3.3.3.3, 2.2.2.2 . 1024-65535 : 4.4.4.4 }
+		oifname "enp2s0" snat ip to ip saddr map { 10.1.1.0/24 : 72.2.3.66-72.2.3.78 }
+		tcp dport { 8888, 9999 } redirect
+	}
+}
diff --git a/tests/shell/testcases/optimizations/merge_nat b/tests/shell/testcases/optimizations/merge_nat
index 3a57d9402301..bfe978701b90 100755
--- a/tests/shell/testcases/optimizations/merge_nat
+++ b/tests/shell/testcases/optimizations/merge_nat
@@ -24,19 +24,6 @@ RULESET="table ip test2 {
 
 $NFT -o -f - <<< $RULESET
 
-RULESET="table ip test3 {
-        chain y {
-                oif lo accept
-                ip saddr 1.1.1.1 tcp sport 1024-65535 snat to 3.3.3.3
-                ip saddr 2.2.2.2 tcp sport 1024-65535 snat to 4.4.4.4
-                oifname enp2s0 snat ip to ip saddr map { 10.1.1.0/24 : 72.2.3.66-72.2.3.78 }
-                tcp dport 8888 redirect
-                tcp dport 9999 redirect
-        }
-}"
-
-$NFT -o -f - <<< $RULESET
-
 RULESET="table ip test4 {
         chain y {
                 oif lo accept
diff --git a/tests/shell/testcases/optimizations/merge_nat_concat b/tests/shell/testcases/optimizations/merge_nat_concat
new file mode 100755
index 000000000000..2e0a91a35bd6
--- /dev/null
+++ b/tests/shell/testcases/optimizations/merge_nat_concat
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+# NFT_TEST_REQUIRES(NFT_TEST_HAVE_pipapo)
+
+set -e
+
+RULESET="table ip test3 {
+        chain y {
+                oif lo accept
+                ip saddr 1.1.1.1 tcp sport 1024-65535 snat to 3.3.3.3
+                ip saddr 2.2.2.2 tcp sport 1024-65535 snat to 4.4.4.4
+                oifname enp2s0 snat ip to ip saddr map { 10.1.1.0/24 : 72.2.3.66-72.2.3.78 }
+                tcp dport 8888 redirect
+                tcp dport 9999 redirect
+        }
+}"
+
+$NFT -o -f - <<< $RULESET
-- 
2.30.2

Re: [PATCH nft 01/12] tests: shell: export DIFF to use it from feature scripts

[Thomas Haller]

On Thu, 2023-11-09 at 17:22 +0100, Pablo Neira Ayuso wrote:
> export DIFF so it can be used from feature scripts to probe the
> kernel.
> 
> +DIFF="$(which diff)"
> +if [ ! -x "$DIFF" ] ; then
> +	DIFF=true
> +fi
> +export DIFF


what is the purpose of having $DIFF variable at all?
Why not require to have `diff` installed?


Maybe that justification is somewhere in the history of the project. If
so, could you drop one line in the commit message what the point is?


Thomas

Re: [PATCH nft 01/12] tests: shell: export DIFF to use it from feature scripts

[Pablo Neira Ayuso]

On Thu, Nov 09, 2023 at 06:49:21PM +0100, Thomas Haller wrote:
> On Thu, 2023-11-09 at 17:22 +0100, Pablo Neira Ayuso wrote:
> > export DIFF so it can be used from feature scripts to probe the
> > kernel.
> > 
> > +DIFF="$(which diff)"
> > +if [ ! -x "$DIFF" ] ; then
> > +	DIFF=true
> > +fi
> > +export DIFF
> 
> 
> what is the purpose of having $DIFF variable at all?
> Why not require to have `diff` installed?
> 
> Maybe that justification is somewhere in the history of the project. If
> so, could you drop one line in the commit message what the point is?

It is all available in git annotate:

68310ba0f9c2 ("tests: shell: Search diff tool once and for all")
7d93e2c2fbc7 ("tests: shell: autogenerate dump verification")

I just need to move it around so I can use it from feature scripts.
If you prefer I can just use 'diff' instead from the feature scripts.

Re: [PATCH nft 01/12] tests: shell: export DIFF to use it from feature scripts

[Thomas Haller]

On Thu, 2023-11-09 at 20:14 +0100, Pablo Neira Ayuso wrote:
> On Thu, Nov 09, 2023 at 06:49:21PM +0100, Thomas Haller wrote:
> > On Thu, 2023-11-09 at 17:22 +0100, Pablo Neira Ayuso wrote:
> > > export DIFF so it can be used from feature scripts to probe the
> > > kernel.
> > > 
> > > +DIFF="$(which diff)"
> > > +if [ ! -x "$DIFF" ] ; then
> > > +	DIFF=true
> > > +fi
> > > +export DIFF
> > 
> > 
> > what is the purpose of having $DIFF variable at all?
> > Why not require to have `diff` installed?
> > 
> > Maybe that justification is somewhere in the history of the
> > project. If
> > so, could you drop one line in the commit message what the point
> > is?
> 
> It is all available in git annotate:
> 
> 68310ba0f9c2 ("tests: shell: Search diff tool once and for all")
> 7d93e2c2fbc7 ("tests: shell: autogenerate dump verification")

First use of $DIFF comes from 

  3fb3bb603374 ('tests/listing: add some listing tests')

which says:

    In order to ease debug in case of failure, if the diff tool is in the system,
    then a textual diff is printed.


With the `true` fallback, checks are skipped. It would be possible, to
use a fallback that still checks for equality (albeit without fancy
diff output).

But really. Just require everybody to install a diff program.


> 
> I just need to move it around so I can use it from feature scripts.
> If you prefer I can just use 'diff' instead from the feature scripts.

Sure. The patch is fine.


I think one day,

   sed 's/\$DIFF\>/diff/g' -i $(git grep -l DIFF tests/shell/)

should be done.



Thomas

Re: [PATCH nft 01/12] tests: shell: export DIFF to use it from feature scripts

[Florian Westphal]

Thomas Haller <thaller@redhat.com> wrote:
> I think one day,
> 
>    sed 's/\$DIFF\>/diff/g' -i $(git grep -l DIFF tests/shell/)
> 
> should be done.

FWIW I agree with this.

Re: [PATCH nft 02/12] tests: shell: skip pipapo tests if kernel lacks support

[Florian Westphal]

Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> Skip tests that require net/netfilter/nft_set_pipapo support.
> 
> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
> ---
>  tests/shell/features/pipapo.nft                      |  9 +++++++++
>  tests/shell/testcases/maps/0013map_0                 |  2 ++
>  tests/shell/testcases/maps/anon_objmap_concat        |  2 ++
>  tests/shell/testcases/maps/typeof_integer_0          |  2 ++
>  .../shell/testcases/optimizations/merge_stmts_concat |  2 ++
>  tests/shell/testcases/optimizations/merge_vmap_raw   |  2 ++
>  tests/shell/testcases/sets/0034get_element_0         |  2 ++
>  tests/shell/testcases/sets/0043concatenated_ranges_0 |  1 +
>  tests/shell/testcases/sets/0043concatenated_ranges_1 |  2 ++
>  tests/shell/testcases/sets/0044interval_overlap_0    | 12 ++++++++++--
>  tests/shell/testcases/sets/0047nat_0                 |  2 ++
>  tests/shell/testcases/sets/concat_interval_0         |  2 ++
>  12 files changed, 38 insertions(+), 2 deletions(-)
>  create mode 100644 tests/shell/features/pipapo.nft
> 
> diff --git a/tests/shell/features/pipapo.nft b/tests/shell/features/pipapo.nft
> new file mode 100644
> index 000000000000..17b56f2210d4
> --- /dev/null
> +++ b/tests/shell/features/pipapo.nft
> @@ -0,0 +1,9 @@
> +# aaa31047a6d2 ("netfilter: nftables: add catch-all set element support")
> +# v5.13-rc1~94^2~10^2~2

# 3c4287f62044 ("nf_tables: Add set type for arbitrary concatenation of ranges")
# v5.6-rc1~151^2~28^2~1

Re: [PATCH nft 01/12] tests: shell: export DIFF to use it from feature scripts

[Florian Westphal]

Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> I just need to move it around so I can use it from feature scripts.
> If you prefer I can just use 'diff' instead from the feature scripts.

Seems better to just use 'diff'.

Re: [PATCH nft 03/12] tests: shell: skip prerouting reject tests if kernel lacks support

[Florian Westphal]

Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> Skip tests that require reject at prerouting hook.
> 
> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
> ---
>  tests/shell/features/prerouting_reject.nft  | 8 ++++++++
>  tests/shell/testcases/optimizations/ruleset | 2 ++
>  2 files changed, 10 insertions(+)
>  create mode 100644 tests/shell/features/prerouting_reject.nft
> 
> diff --git a/tests/shell/features/prerouting_reject.nft b/tests/shell/features/prerouting_reject.nft
> new file mode 100644
> index 000000000000..26098bb54534
> --- /dev/null
> +++ b/tests/shell/features/prerouting_reject.nft
> @@ -0,0 +1,8 @@
> +# f53b9b0bdc59 netfilter: introduce support for reject at prerouting stage
> +# v5.13-rc1~94^2~10^2~2

# v5.9-rc1~133^2~302^2~11

Re: [PATCH nft 04/12] tests: shell: skip stateful expression in sets tests if kernel lacks support

[Florian Westphal]

Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> Skip tests that require stateful expressions in sets.
> 
> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
> ---
>  tests/shell/features/set_expr.sh              | 19 +++++++++++++++++++
>  tests/shell/testcases/json/0002table_map_0    |  1 +
>  tests/shell/testcases/maps/0009vmap_0         |  2 ++
>  .../testcases/optimizations/merge_stmts_vmap  |  2 ++
>  tests/shell/testcases/sets/0048set_counters_0 |  2 ++
>  .../testcases/sets/0051set_interval_counter_0 |  2 ++
>  tests/shell/testcases/sets/elem_opts_compat_0 |  2 ++
>  7 files changed, 30 insertions(+)
>  create mode 100755 tests/shell/features/set_expr.sh
> 
> diff --git a/tests/shell/features/set_expr.sh b/tests/shell/features/set_expr.sh
> new file mode 100755
> index 000000000000..c323d59e8920
> --- /dev/null
> +++ b/tests/shell/features/set_expr.sh
> @@ -0,0 +1,19 @@
> +#!/bin/bash
> +
> +# 65038428b2c6 ("netfilter: nf_tables: allow to specify stateful expression in set definition")
> +# v5.6-rc5-1736-g65038428b2c6

v5.7-rc1~146^2~12^2~25

(git-describe --contains).

Re: [PATCH nft 00/12] update tests/shell for 5.4 kernels

[Florian Westphal]

Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> Hi,
> 
> This is still not complete, still 8 tests/shell fail in 5.4 related to
> flowtable features that are missing, but this batch is already getting large.

Thats fine, you can push what you have.