[RFC nft PATCH 0/3] new test suite

netfilter-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* [RFC nft PATCH 0/3] new test suite
@ 2015-10-09 12:38 Arturo Borrero Gonzalez
  2015-10-09 12:38 ` [RFC nft PATCH 1/3] tests: add operations test-suite Arturo Borrero Gonzalez
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Arturo Borrero Gonzalez @ 2015-10-09 12:38 UTC (permalink / raw)
  To: netfilter-devel; +Cc: fw, kaber, pablo

Hi,

I suggest a testsuite like this, which can test higher level operations,
so more tests can be performed to the code apart of the regression tests:
 * listings/filterings
 * deletions/flushes
 * monitor operations
 * ruleset loadings with -f
 * interactive interface with -i (perhaps)

In patch 1/3 there is a description of this simple testbed.

If you like the idea I would be happy to add more tests cases.

---

Arturo Borrero Gonzalez (3):
      tests: add operations test-suite
      tests/operations: add maps tests cases
      tests/operations: add tests for handles and comments


 tests/operations/README                            |   23 ++++++
 tests/operations/run-operations-tests.sh           |   72 ++++++++++++++++++++
 .../operations/testcases/maps/anonymous_snat_map_0 |    7 ++
 tests/operations/testcases/maps/named_snat_map_0   |    9 +++
 tests/operations/testcases/optionals/comments_0    |    9 +++
 .../testcases/optionals/comments_handles_0         |   11 +++
 .../testcases/optionals/comments_handles_monitor_0 |   21 ++++++
 tests/operations/testcases/optionals/handles_0     |    9 +++
 tests/operations/testcases/optionals/handles_1     |    9 +++
 9 files changed, 170 insertions(+)
 create mode 100644 tests/operations/README
 create mode 100755 tests/operations/run-operations-tests.sh
 create mode 100755 tests/operations/testcases/maps/anonymous_snat_map_0
 create mode 100755 tests/operations/testcases/maps/named_snat_map_0
 create mode 100755 tests/operations/testcases/optionals/comments_0
 create mode 100755 tests/operations/testcases/optionals/comments_handles_0
 create mode 100755 tests/operations/testcases/optionals/comments_handles_monitor_0
 create mode 100755 tests/operations/testcases/optionals/handles_0
 create mode 100755 tests/operations/testcases/optionals/handles_1

--

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [RFC nft PATCH 1/3] tests: add operations test-suite
  2015-10-09 12:38 [RFC nft PATCH 0/3] new test suite Arturo Borrero Gonzalez
@ 2015-10-09 12:38 ` Arturo Borrero Gonzalez
  2015-10-09 12:38 ` [RFC nft PATCH 2/3] tests/operations: add maps tests cases Arturo Borrero Gonzalez
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 5+ messages in thread
From: Arturo Borrero Gonzalez @ 2015-10-09 12:38 UTC (permalink / raw)
  To: netfilter-devel; +Cc: fw, kaber, pablo

This new test-suite is intended to perform tests of higher level than
the other reggresion test-suite.

It can run arbitrary executables which can perform any test apart of testing
the nft syntax or netlink code (which is what the regression tests does).

To run the test suite (as root):
 % cd tests/operations
 % ./run-operations-tests.sh

Test files are executables files with the pattern <<name_N>>, where N is the
expected return code of the executable. Since they are located with `find',
test-files can be spreaded in any sub-directories.

You can turn on a verbose execution by calling:
 % ./run-operations-tests.sh -v

Before each call to the test-files, `nft flush ruleset' will be called.
Also, test-files will receive the environment variable $NFT which contains the
path to the nftables binary being tested.

You can pass an arbitrary $NFT value as well:
 % NFT=../../src/nft ./run-operations-tests.sh

Signed-off-by: Arturo Borrero Gonzalez <arturo.borrero.glez@gmail.com>
---
 tests/operations/README                  |   23 ++++++++++
 tests/operations/run-operations-tests.sh |   72 ++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100644 tests/operations/README
 create mode 100755 tests/operations/run-operations-tests.sh

diff --git a/tests/operations/README b/tests/operations/README
new file mode 100644
index 0000000..85084c9
--- /dev/null
+++ b/tests/operations/README
@@ -0,0 +1,23 @@
+This test-suite is intended to perform tests of higher level than
+the other reggresion test-suite.
+
+It can run arbitrary executables which can perform any test apart of testing
+the nft syntax or netlink code (which is what the regression tests does).
+
+To run the test suite (as root):
+ % cd tests/operations
+ % ./run-operations-tests.sh
+
+Test files are executables files with the pattern <<name_N>>, where N is the
+expected return code of the executable. Since they are located with `find',
+test-files can be spreaded in any sub-directories.
+
+You can turn on a verbose execution by calling:
+ % ./run-operations-tests.sh -v
+
+Before each call to the test-files, `nft flush ruleset' will be called.
+Also, test-files will receive the environment variable $NFT which contains the
+path to the nftables binary being tested.
+
+You can pass an arbitrary $NFT value as well:
+ % NFT=../../src/nft ./run-operations-tests.sh
diff --git a/tests/operations/run-operations-tests.sh b/tests/operations/run-operations-tests.sh
new file mode 100755
index 0000000..df2670b
--- /dev/null
+++ b/tests/operations/run-operations-tests.sh
@@ -0,0 +1,72 @@
+#!/bin/bash
+
+# Configuration
+TESTDIR="./"
+RETURNCODE_SEPARATOR="_"
+
+msg_error() {
+	echo "E: $1 ..." >&2
+	exit 1
+}
+
+msg_warn() {
+	echo "W: $1" >&2
+}
+
+msg_info() {
+	echo "I: $1"
+}
+
+if [ "$(id -u)" != "0" ] ; then
+	msg_error "this requires root!"
+fi
+
+[ -z "$NFT" ] && NFT="$(which nft)"
+if [ ! -x "$NFT" ] ; then
+	msg_error "no nft binary!"
+else
+	msg_info "using nft binary $NFT"
+fi
+
+if [ ! -d "$TESTDIR" ] ; then
+	msg_error "missing testdir $TESTDIR"
+fi
+
+FIND="$(which find)"
+if [ ! -x "$FIND" ] ; then
+	msg_error "no find binary found"
+fi
+
+if [ "$1" == "-v" ] ; then
+	VERBOSE=y
+fi
+
+echo ""
+ok=0
+failed=0
+for testfile in $(${FIND} ${TESTDIR} -executable -regex .*${RETURNCODE_SEPARATOR}[0-9]+)
+do
+	$NFT flush ruleset
+
+	rc_spec=$(awk -F${RETURNCODE_SEPARATOR} '{print $NF}' <<< $testfile)
+	test_output=$(NFT=$NFT ${testfile} ${TESTS_OUTPUT} 2>&1)
+	rc_got=$?
+	if [ "$rc_got" == "$rc_spec" ] ; then
+		msg_info "[OK]		$testfile"
+		[ "$VERBOSE" == "y" ] && [ ! -z "$test_output" ] && echo "$test_output"
+		((ok++))
+	else
+		((failed++))
+		if [ "$VERBOSE" == "y" ] ; then
+			msg_warn "[FAILED]	$testfile: expected $rc_spec but got $rc_got"
+			[ ! -z "$test_output" ] && echo "$test_output"
+		else
+			msg_warn "[FAILED]	$testfile"
+		fi
+	fi
+done
+
+echo ""
+msg_info "results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))"
+
+$NFT flush ruleset


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [RFC nft PATCH 2/3] tests/operations: add maps tests cases
  2015-10-09 12:38 [RFC nft PATCH 0/3] new test suite Arturo Borrero Gonzalez
  2015-10-09 12:38 ` [RFC nft PATCH 1/3] tests: add operations test-suite Arturo Borrero Gonzalez
@ 2015-10-09 12:38 ` Arturo Borrero Gonzalez
  2015-10-09 12:38 ` [RFC nft PATCH 3/3] tests/operations: add tests for handles and comments Arturo Borrero Gonzalez
  2015-10-09 12:52 ` [RFC nft PATCH 0/3] new test suite Pablo Neira Ayuso
  3 siblings, 0 replies; 5+ messages in thread
From: Arturo Borrero Gonzalez @ 2015-10-09 12:38 UTC (permalink / raw)
  To: netfilter-devel; +Cc: fw, kaber, pablo

Let's add some tests cases for maps.

Signed-off-by: Arturo Borrero Gonzalez <arturo.borrero.glez@gmail.com>
---
 .../operations/testcases/maps/anonymous_snat_map_0 |    7 +++++++
 tests/operations/testcases/maps/named_snat_map_0   |    9 +++++++++
 2 files changed, 16 insertions(+)
 create mode 100755 tests/operations/testcases/maps/anonymous_snat_map_0
 create mode 100755 tests/operations/testcases/maps/named_snat_map_0

diff --git a/tests/operations/testcases/maps/anonymous_snat_map_0 b/tests/operations/testcases/maps/anonymous_snat_map_0
new file mode 100755
index 0000000..67698ec
--- /dev/null
+++ b/tests/operations/testcases/maps/anonymous_snat_map_0
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# anonymous map can be added to a snat rule
+
+$NFT add table nat
+$NFT add chain nat postrouting
+$NFT add rule nat postrouting snat ip saddr map {1.1.1.1 : 2.2.2.2}
diff --git a/tests/operations/testcases/maps/named_snat_map_0 b/tests/operations/testcases/maps/named_snat_map_0
new file mode 100755
index 0000000..6e11051
--- /dev/null
+++ b/tests/operations/testcases/maps/named_snat_map_0
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# nameds map can be addedd to a snat rule
+
+$NFT add table nat
+$NFT add chain nat postrouting
+$NFT add map nat m { type ipv4_addr : ipv4_addr\; }
+$NFT add element nat m {1.1.1.1: 2.2.2.2}
+$NFT add rule nat postrouting snat ip saddr map @m


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [RFC nft PATCH 3/3] tests/operations: add tests for handles and comments
  2015-10-09 12:38 [RFC nft PATCH 0/3] new test suite Arturo Borrero Gonzalez
  2015-10-09 12:38 ` [RFC nft PATCH 1/3] tests: add operations test-suite Arturo Borrero Gonzalez
  2015-10-09 12:38 ` [RFC nft PATCH 2/3] tests/operations: add maps tests cases Arturo Borrero Gonzalez
@ 2015-10-09 12:38 ` Arturo Borrero Gonzalez
  2015-10-09 12:52 ` [RFC nft PATCH 0/3] new test suite Pablo Neira Ayuso
  3 siblings, 0 replies; 5+ messages in thread
From: Arturo Borrero Gonzalez @ 2015-10-09 12:38 UTC (permalink / raw)
  To: netfilter-devel; +Cc: fw, kaber, pablo

Here some tests for optional things like rule handles and comments.

Signed-off-by: Arturo Borrero Gonzalez <arturo.borrero.glez@gmail.com>
---
 tests/operations/testcases/optionals/comments_0    |    9 +++++++++
 .../testcases/optionals/comments_handles_0         |   11 ++++++++++
 .../testcases/optionals/comments_handles_monitor_0 |   21 ++++++++++++++++++++
 tests/operations/testcases/optionals/handles_0     |    9 +++++++++
 tests/operations/testcases/optionals/handles_1     |    9 +++++++++
 5 files changed, 59 insertions(+)
 create mode 100755 tests/operations/testcases/optionals/comments_0
 create mode 100755 tests/operations/testcases/optionals/comments_handles_0
 create mode 100755 tests/operations/testcases/optionals/comments_handles_monitor_0
 create mode 100755 tests/operations/testcases/optionals/handles_0
 create mode 100755 tests/operations/testcases/optionals/handles_1

diff --git a/tests/operations/testcases/optionals/comments_0 b/tests/operations/testcases/optionals/comments_0
new file mode 100755
index 0000000..51024c8
--- /dev/null
+++ b/tests/operations/testcases/optionals/comments_0
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# comments are shown
+
+$NFT flush ruleset
+$NFT add table test
+$NFT add chain test test
+$NFT add rule test test tcp dport 22 counter accept comment test_comment
+$NFT list table test -a | grep 'accept comment \"test_comment\"' >/dev/null
diff --git a/tests/operations/testcases/optionals/comments_handles_0 b/tests/operations/testcases/optionals/comments_handles_0
new file mode 100755
index 0000000..85048e8
--- /dev/null
+++ b/tests/operations/testcases/optionals/comments_handles_0
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+# handles and comments mix well
+
+$NFT flush ruleset
+$NFT add table test
+$NFT add chain test test
+$NFT add rule test test tcp dport 22 counter accept comment test_comment
+set -e
+$NFT list table test -a | grep 'accept comment \"test_comment\" # handle '[[:digit:]]$ >/dev/null
+$NFT list table test | grep 'accept comment \"test_comment\"' | grep -v '# handle '[[:digit:]]$ >/dev/null
diff --git a/tests/operations/testcases/optionals/comments_handles_monitor_0 b/tests/operations/testcases/optionals/comments_handles_monitor_0
new file mode 100755
index 0000000..81fe85f
--- /dev/null
+++ b/tests/operations/testcases/optionals/comments_handles_monitor_0
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# handles and comments mix well in monitor
+
+MKTEMP=$(which mktemp)
+[ ! -x $MKTEMP ] && exit 0 # :-( no mktemp
+
+tmpfile=$(${MKTEMP})
+trap "rm -rf $tmpfile" EXIT # cleanup if aborted
+
+$NFT flush ruleset
+$NFT add table test
+$NFT add chain test test
+
+$NFT monitor -a > $tmpfile &
+$NFT add rule test test tcp dport 22 counter accept comment test_comment
+kill $!
+set -e
+grep 'accept comment \"test_comment\" # handle '[[:digit:]]$ $tmpfile >/dev/null
+set +e
+rm -rf $tmpfile
diff --git a/tests/operations/testcases/optionals/handles_0 b/tests/operations/testcases/optionals/handles_0
new file mode 100755
index 0000000..b082eca
--- /dev/null
+++ b/tests/operations/testcases/optionals/handles_0
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# handles are shown last
+
+$NFT flush ruleset
+$NFT add table test
+$NFT add chain test test
+$NFT add rule test test tcp dport 22 counter accept
+$NFT list table test -a | grep 'accept # handle '[[:digit:]]$ >/dev/null
diff --git a/tests/operations/testcases/optionals/handles_1 b/tests/operations/testcases/optionals/handles_1
new file mode 100755
index 0000000..517637c
--- /dev/null
+++ b/tests/operations/testcases/optionals/handles_1
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# handles are not shown if not asked for them
+
+$NFT flush ruleset
+$NFT add table test
+$NFT add chain test test
+$NFT add rule test test tcp dport 22 counter accept
+$NFT list table test | grep 'accept # handle '[[:digit:]]$ >/dev/null


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [RFC nft PATCH 0/3] new test suite
  2015-10-09 12:38 [RFC nft PATCH 0/3] new test suite Arturo Borrero Gonzalez
                   ` (2 preceding siblings ...)
  2015-10-09 12:38 ` [RFC nft PATCH 3/3] tests/operations: add tests for handles and comments Arturo Borrero Gonzalez
@ 2015-10-09 12:52 ` Pablo Neira Ayuso
  3 siblings, 0 replies; 5+ messages in thread
From: Pablo Neira Ayuso @ 2015-10-09 12:52 UTC (permalink / raw)
  To: Arturo Borrero Gonzalez; +Cc: netfilter-devel, fw, kaber

On Fri, Oct 09, 2015 at 02:38:06PM +0200, Arturo Borrero Gonzalez wrote:
> Hi,
> 
> I suggest a testsuite like this, which can test higher level operations,
> so more tests can be performed to the code apart of the regression tests:
>  * listings/filterings
>  * deletions/flushes
>  * monitor operations
>  * ruleset loadings with -f
>  * interactive interface with -i (perhaps)

I started something similar here locally.

We can probably rearrange the directory to something like:

tests
  |
  .--- py
  .--- shell
  .--- files

So under tests/py/ we get the python-based infrastructure, then shell
includes shell scripts for your test cases. The files directory should
contain files that we can load via 'nft -f'.

That should be pretty much covering all paths of the command line
interface.

> In patch 1/3 there is a description of this simple testbed.
> 
> If you like the idea I would be happy to add more tests cases.

I think it's good idea to enhance the test infrastructure to catch
regressions on as many fronts as possible.

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2015-10-09 13:48 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-10-09 12:38 [RFC nft PATCH 0/3] new test suite Arturo Borrero Gonzalez
2015-10-09 12:38 ` [RFC nft PATCH 1/3] tests: add operations test-suite Arturo Borrero Gonzalez
2015-10-09 12:38 ` [RFC nft PATCH 2/3] tests/operations: add maps tests cases Arturo Borrero Gonzalez
2015-10-09 12:38 ` [RFC nft PATCH 3/3] tests/operations: add tests for handles and comments Arturo Borrero Gonzalez
2015-10-09 12:52 ` [RFC nft PATCH 0/3] new test suite Pablo Neira Ayuso

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).