Re: [PATCH net-next RFC] selftests/net: integrate packetdrill with ksft

[Willem de Bruijn <willemdebruijn.kernel@gmail.com>]

Jakub Kicinski wrote:
> On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:
> > That could work.
> > 
> > Is reporting one KTAP and exitcode per directory vs per packetdrill
> > invocation good/bad/neither?
> 
> To me "neither", hopefully and unhelpfully I won't tell you anything
> that's not common sense :) The balance is between:
> 
>  - having test cases which don't take too long (assuming debug kernel):
>      <15min is good, 
>      >1h I will start complaining,
>      >1h30m is bad because we can't retry and still make the 3h
>             deadline that NIPA has,
>      >3h the test can't run in NIPA at all.
> 
> vs
> 
>  - flip side is having so many cases it's hard to keep track and render
>    in the UI. JSON is relatively slow to process. If you have 150 cases,
>    that's 300 per branch (debug and non-debug kernels), times 8
>    branches a day => 2.4k results / day
>    I think that's still fine-ish, but on the larger side for sure. For
>    reference net and forwarding have ~100 tests each. FWIW we do have
>    the ability to collect and display nested KTAP so the information is
>    not lost (but it makes queries slower so we don't fetch it by default).
> 
> > Three other issues if this is calling packetdrill directly is
> > - passing the non-trivial IP specific flags
> > - running twice, for IPv4 and IPv6
> > - chdir into the directory of the pkt file
> > 
> > That can be addressed by instead calling a small wrapper shell script.
> > 
> > That would do the test_func_builder part of packetdrill_ksft.py.
> > But without the need to handle netns, popen/cmd, etc, and thus the
> > ksft dependencies.
> 
> Right!

Implemented this, but hit a snag

Kselftest install does not preserve directories.

So all .pkt files are copied into net/packetdrill root. This is messy.
More fundamentally it breaks the includes in the files (e..g, `source
../common/defaults.sh`).

The output of having each test separate is also quite unreadable.

An alternative is to add each subdirectory as a separate TARGET.
But that seems similarly impractical. And does nothing to group
KTAP output.

In practice, most directories have a handful of .pkt scripts, and
each script is fast, so serialization is not a huge issue.

I found tools/testing/selftests/net/kselftest/ktap_helpers.sh,
which has helpful KTAP boilerplate for shell tests, including
KSFT exit codes. Am using that instead of python, to reduce the
external dependencies.