Re: [PATCH net-next RFC] selftests/net: integrate packetdrill with ksft

[Jakub Kicinski <kuba@kernel.org>]

On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:
> That could work.
> 
> Is reporting one KTAP and exitcode per directory vs per packetdrill
> invocation good/bad/neither?

To me "neither", hopefully and unhelpfully I won't tell you anything
that's not common sense :) The balance is between:

 - having test cases which don't take too long (assuming debug kernel):
     <15min is good, 
     >1h I will start complaining,
     >1h30m is bad because we can't retry and still make the 3h
            deadline that NIPA has,
     >3h the test can't run in NIPA at all.

vs

 - flip side is having so many cases it's hard to keep track and render
   in the UI. JSON is relatively slow to process. If you have 150 cases,
   that's 300 per branch (debug and non-debug kernels), times 8
   branches a day => 2.4k results / day
   I think that's still fine-ish, but on the larger side for sure. For
   reference net and forwarding have ~100 tests each. FWIW we do have
   the ability to collect and display nested KTAP so the information is
   not lost (but it makes queries slower so we don't fetch it by default).

> Three other issues if this is calling packetdrill directly is
> - passing the non-trivial IP specific flags
> - running twice, for IPv4 and IPv6
> - chdir into the directory of the pkt file
> 
> That can be addressed by instead calling a small wrapper shell script.
> 
> That would do the test_func_builder part of packetdrill_ksft.py.
> But without the need to handle netns, popen/cmd, etc, and thus the
> ksft dependencies.

Right!