Re: [PATCH iptables] tests: iptables-test: extend coverage for ip6tables

[Phil Sutter <phil@nwl.cc>]

On Tue, Oct 22, 2024 at 05:07:25PM +0200, Pablo Neira Ayuso wrote:
> On Tue, Oct 22, 2024 at 04:55:33PM +0200, Phil Sutter wrote:
> > On Tue, Oct 22, 2024 at 03:48:12PM +0200, Pablo Neira Ayuso wrote:
> > > On Tue, Oct 22, 2024 at 03:08:01PM +0200, Phil Sutter wrote:
> > > > On Tue, Oct 22, 2024 at 02:30:58PM +0200, Phil Sutter wrote:
> > > > [...]
> > > > > - With your patch applied, 20 rules fail (in both variants). Is this
> > > > >   expected or a bug on my side?
> > > > 
> > > > OK, so most failures are caused by my test kernel not having
> > > > CONFIG_IP_VS_IPV6 enabled.
> > > > 
> > > > Apart from that, there is a minor bug in introduced libip6t_recent.t in
> > > > that it undoes commit d859b91e6f3ed ("extensions: recent: New kernels
> > > > support 999 hits") by accident. More interesting though, it's reported
> > > > twice, once for fast mode and once for normal mode. I'll see how I can
> > > > turn off error reporting in fast mode, failing tests are repeated
> > > > anyway.
> > > 
> > > Would you point me to the relevant line in the libip6t_recent.t?
> > 
> > It is in line 7, I had changed the supposed-to-fail --hitcount value of
> > 999 to 65536.
> 
> This was already fixed in v2, correct?

Ah, you're right. I didn't notice your v2.

If you're OK with it, I'll apply your v3 with the following changes:
- Describe 'iptables' param in _run_test_file()
- Drop duplicate 'endswith' test from _run_test_file()
- Print results with command name suffixed for libxt tests (it is more
  consistent wrt. tests count)

Thanks, Phil

diff --git a/iptables-test.py b/iptables-test.py
index 521c11d7bbc05..0d2f30dfb0d7c 100755
--- a/iptables-test.py
+++ b/iptables-test.py
@@ -385,24 +385,20 @@ STDERR_IS_TTY = sys.stderr.isatty()
 
     return tests
 
-def _run_test_file(iptables, filename, netns, print_result):
+def _run_test_file(iptables, filename, netns, suffix):
     '''
     Runs a test file
 
+    :param iptables: string with the iptables command to execute
     :param filename: name of the file with the test rules
     :param netns: network namespace to perform test run in
     '''
-    #
-    # if this is not a test file, skip.
-    #
-    if not filename.endswith(".t"):
-        return 0, 0
 
     fast_failed = False
     if fast_run_possible(filename):
         tests = run_test_file_fast(iptables, filename, netns)
-        if tests > 0 and print_result:
-            print(filename + ": " + maybe_colored('green', "OK", STDOUT_IS_TTY))
+        if tests > 0:
+            print(filename + ": " + maybe_colored('green', "OK", STDOUT_IS_TTY) + suffix)
             return tests, tests
         fast_failed = True
 
@@ -482,10 +478,9 @@ STDERR_IS_TTY = sys.stderr.isatty()
 
     if netns:
         execute_cmd("ip netns del " + netns, filename)
-    if total_test_passed and print_result:
-        suffix = ""
+    if total_test_passed:
         if fast_failed:
-            suffix = maybe_colored('red', " but fast mode failed!", STDOUT_IS_TTY)
+            suffix += maybe_colored('red', " but fast mode failed!", STDOUT_IS_TTY)
         print(filename + ": " + maybe_colored('green', "OK", STDOUT_IS_TTY) + suffix)
 
     f.close()
@@ -527,11 +522,12 @@ STDERR_IS_TTY = sys.stderr.isatty()
     tests = 0
     passed = 0
     print_result = False
-    for index, iptables in enumerate(xtables):
-        if index == len(xtables) - 1:
-            print_result = True
+    suffix = ""
+    for iptables in xtables:
+        if len(xtables) > 1:
+            suffix = "({})".format(iptables)
 
-        file_tests, file_passed = _run_test_file(iptables, filename, netns, print_result)
+        file_tests, file_passed = _run_test_file(iptables, filename, netns, suffix)
         if file_tests:
             tests += file_tests
             passed += file_passed