[PATCH v2] tests: xlate: print total no. of testfiles, tests and tests passed

[PATCH v2] tests: xlate: print total no. of testfiles, tests and tests passed

[Harsha Sharma]

Print errors and total no of tests and tests passed for testfile argument.
Print total no. of testfiles, total no. of tests and total no. of tests
passed for "all" argument.
For e.g. :
sudo ./xlate-test.py --all (adds this line in end with whole output)
64 test file, 246 tests, 242 tests passed

sudo ./xlate-test.py libxt_ipcomp.txlate
1 test file, 2 tests, 2 tests passed

Signed-off-by: Harsha Sharma <harshasharmaiitr@gmail.com>
---
Changes in v2:
 -Change log message 
 -Remove changes for testfile argument

 xlate-test.py | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/xlate-test.py b/xlate-test.py
index 43c4be19..97b3a903 100755
--- a/xlate-test.py
+++ b/xlate-test.py
@@ -9,7 +9,6 @@ from subprocess import Popen, PIPE
 
 keywords = ("iptables-translate", "ip6tables-translate")
 
-
 if sys.stdout.isatty():
     colors = {"magenta": "\033[95m", "green": "\033[92m", "yellow": "\033[93m",
               "red": "\033[91m", "end": "\033[0m"}
@@ -35,11 +34,14 @@ def green(string):
 
 def run_test(name, payload):
     test_passed = True
+    tests = 0
+    passed = 0
     result = []
     result.append(yellow("## " + name.replace(".txlate", "")))
 
     for line in payload:
         if line.startswith(keywords):
+            tests += 1
             process = Popen(shlex.split(line), stdout=PIPE, stderr=PIPE)
             (output, error) = process.communicate()
             if process.returncode == 0:
@@ -52,9 +54,12 @@ def run_test(name, payload):
                     result.append(magenta("res: ") + translation + "\n")
                     test_passed = False
                 elif args.all:
+                    passed += 1
                     result.append(green("Ok"))
                     result.append(magenta("src: ") + line.rstrip(" \n"))
                     result.append(magenta("res: ") + translation + "\n")
+                elif args.test:
+                    passed += 1
             else:
                 test_passed = False
                 result.append(red("Error: ") + "iptables-translate failure")
@@ -62,14 +67,25 @@ def run_test(name, payload):
 
     if not test_passed or args.all:
         print("\n".join(result))
+    if args.test:
+        print("1 test file, %d tests, %d tests passed" % (tests, passed))
+    else:
+        return tests, passed
 
 
 def load_test_files():
+    test_files = 0
+    total_tests = 0
+    total_passed = 0
     for test in sorted(os.listdir("extensions")):
         if test.endswith(".txlate"):
             with open("extensions/" + test, "r") as payload:
-                run_test(test, payload)
-
+                tests, passed = run_test(test, payload)
+                test_files += 1
+                total_tests += tests
+                total_passed += passed
+    if args.all:
+        print("%d test file, %d tests, %d tests passed" % (test_files, total_tests, total_passed))
 
 def main():
     if os.getuid() != 0:
-- 
2.11.0

Re: [PATCH v2] tests: xlate: print total no. of testfiles, tests and tests passed

[Pablo Neira Ayuso]

On Thu, Oct 19, 2017 at 06:10:02PM +0530, Harsha Sharma wrote:
> Print errors and total no of tests and tests passed for testfile argument.
> Print total no. of testfiles, total no. of tests and total no. of tests
> passed for "all" argument.
> For e.g. :
> sudo ./xlate-test.py --all (adds this line in end with whole output)

Why we need --all?

Don't we print a line for each test that we run to report that it is
OK? I mean:

# python nft-test.py
any/fwd.t: OK
any/rt.t: OK
...

What I would like to see is that xlate-test.py behaves just like
nft-tests.py in what it prints.

Let me know, thanks!

> 64 test file, 246 tests, 242 tests passed
> 
> sudo ./xlate-test.py libxt_ipcomp.txlate
> 1 test file, 2 tests, 2 tests passed

Re: [PATCH v2] tests: xlate: print total no. of testfiles, tests and tests passed

[Pablo Neira Ayuso]

On Fri, Oct 20, 2017 at 02:33:56PM +0200, Pablo Neira Ayuso wrote:
> On Thu, Oct 19, 2017 at 06:10:02PM +0530, Harsha Sharma wrote:
> > Print errors and total no of tests and tests passed for testfile argument.
> > Print total no. of testfiles, total no. of tests and total no. of tests
> > passed for "all" argument.
> > For e.g. :
> > sudo ./xlate-test.py --all (adds this line in end with whole output)
> 
> Why we need --all?

Oh, I forgot. This is like this, right?

http://patchwork.ozlabs.org/patch/825100/

Hm, but --all here should probably be --test-file-path or such.

I may be missing anything, but I think we don't need a --all for
xlate-test.py.

Thanks!