[PATCH net-next] selftests: net: report output format as TAP 13 in Python tests

[PATCH net-next] selftests: net: report output format as TAP 13 in Python tests

[Jakub Kicinski]

The Python lib based tests report that they are producing
"KTAP version 1", but really we aren't making use of any
KTAP features, like subtests. Our output is plain TAP.

Report TAP 13 instead of KTAP 1, this is what mptcp tests do,
and what NIPA knows how to parse best. For HW testing we need
precise subtest result tracking.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
CC: shuah@kernel.org
CC: petrm@nvidia.com
CC: matttbe@kernel.org
CC: willemb@google.com
CC: linux-kselftest@vger.kernel.org
---
 tools/testing/selftests/drivers/net/README.rst | 4 ++--
 tools/testing/selftests/net/lib/py/ksft.py     | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/drivers/net/README.rst b/tools/testing/selftests/drivers/net/README.rst
index 3b6a29e6564b..eb838ae94844 100644
--- a/tools/testing/selftests/drivers/net/README.rst
+++ b/tools/testing/selftests/drivers/net/README.rst
@@ -107,7 +107,7 @@ Example
   1..1
   # timeout set to 45
   # selftests: drivers/net: ping.py
-  # KTAP version 1
+  # TAP version 13
   # 1..3
   # ok 1 ping.test_v4
   # ok 2 ping.test_v6
@@ -128,7 +128,7 @@ Example
 Run the test::
 
   [/root] # ./ksft-net-drv/drivers/net/ping.py
-  KTAP version 1
+  TAP version 13
   1..3
   ok 1 ping.test_v4
   ok 2 ping.test_v6 # SKIP Test requires IPv6 connectivity
diff --git a/tools/testing/selftests/net/lib/py/ksft.py b/tools/testing/selftests/net/lib/py/ksft.py
index fd23349fa8ca..3cfad0fd4570 100644
--- a/tools/testing/selftests/net/lib/py/ksft.py
+++ b/tools/testing/selftests/net/lib/py/ksft.py
@@ -207,7 +207,7 @@ KSFT_DISRUPTIVE = True
 
     totals = {"pass": 0, "fail": 0, "skip": 0, "xfail": 0}
 
-    print("KTAP version 1")
+    print("TAP version 13")
     print("1.." + str(len(cases)))
 
     global KSFT_RESULT
-- 
2.48.1

Re: [PATCH net-next] selftests: net: report output format as TAP 13 in Python tests

[Matthieu Baerts]

Hi Jakub,

On 28/02/2025 19:00, Jakub Kicinski wrote:
> The Python lib based tests report that they are producing
> "KTAP version 1", but really we aren't making use of any
> KTAP features, like subtests. Our output is plain TAP.
> 
> Report TAP 13 instead of KTAP 1, this is what mptcp tests do,

Indeed, and also TC tests, and all the ones using kselftest_harness.h I
think.

> and what NIPA knows how to parse best. For HW testing we need
> precise subtest result tracking.

I guess the best would be to have the kselftests infrastructure fully
supporting KTAP, and then have a way for each test to print subtests
correctly, not nested in the comments like it is done for the moment.
But that looks harder to put in place, and the current solution is used
in a few areas.

Good idea to switch to 'TAP version 13' for the moment, and re-use the
existing parsing in NIPA!

Acked-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [PATCH net-next] selftests: net: report output format as TAP 13 in Python tests

[Jakub Kicinski]

On Sat, 1 Mar 2025 11:52:06 +0100 Matthieu Baerts wrote:
> On 28/02/2025 19:00, Jakub Kicinski wrote:
> > The Python lib based tests report that they are producing
> > "KTAP version 1", but really we aren't making use of any
> > KTAP features, like subtests. Our output is plain TAP.
> > 
> > Report TAP 13 instead of KTAP 1, this is what mptcp tests do,  
> 
> Indeed, and also TC tests, and all the ones using kselftest_harness.h I
> think.
> 
> > and what NIPA knows how to parse best. For HW testing we need
> > precise subtest result tracking.  
> 
> I guess the best would be to have the kselftests infrastructure fully
> supporting KTAP, and then have a way for each test to print subtests
> correctly, not nested in the comments like it is done for the moment.
> But that looks harder to put in place, and the current solution is used
> in a few areas.

Right. So for additional context for folks less familiar with selftests
the two minor inconveniences which made "supporting KTAP" a less
exciting option are: (a) we run make run_tests which wraps things
in an outer TAP, so we'd end up with a hybrid, anyway; (b) NIPA
only supports one level of subtests.

Thanks for the review!

Re: [PATCH net-next] selftests: net: report output format as TAP 13 in Python tests

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net-next.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Fri, 28 Feb 2025 10:00:07 -0800 you wrote:
> The Python lib based tests report that they are producing
> "KTAP version 1", but really we aren't making use of any
> KTAP features, like subtests. Our output is plain TAP.
> 
> Report TAP 13 instead of KTAP 1, this is what mptcp tests do,
> and what NIPA knows how to parse best. For HW testing we need
> precise subtest result tracking.
> 
> [...]

Here is the summary with links:
  - [net-next] selftests: net: report output format as TAP 13 in Python tests
    https://git.kernel.org/netdev/net-next/c/d110dbf1490b

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html