Re: [PATCH net-next 3/4] selftests: kselftest_harness: support using xfail

[Kees Cook <keescook@chromium.org>]

On Wed, Feb 14, 2024 at 04:25:14PM -0800, Jakub Kicinski wrote:
> On Wed, 14 Feb 2024 22:46:46 +0100 Jakub Sitnicki wrote:
> > > On second thought, if I can suggest a follow up change so this:
> > >
> > > ok 17 # XFAIL SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> > >
> > > ... becomes this
> > >
> > > ok 17 ip_local_port_range.ip4_stcp.late_bind # XFAIL SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> > >
> > > You see, we parse test results if they are in TAP format. Lack of test
> > > name for xfail'ed and skip'ed tests makes it difficult to report in CI
> > > which subtest was it. Happy to contribute it, once this series gets
> > > applied.  
> > 
> > Should have said "harder", not "difficult". That was an overstatement.
> > 
> > Test name can be extracted from diagnostic lines preceeding the status.
> > 
> > #  RUN           ip_local_port_range.ip4_stcp.late_bind ...
> > #      XFAIL      SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> > #            OK  ip_local_port_range.ip4_stcp.late_bind
> > ok 17 ip_local_port_range.ip4_stcp.late_bind # XFAIL SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> > 
> > It just makes the TAP parser easier if the test name is included on the
> > status line. That would be the motivation here. Let me know what you
> > think.
> 
> Good catch, I just copied what we do for skip and completely missed
> this. As you said we'd report:
> 
> ok 17 # XFAIL SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> 
> and I think that's sort of closer to valid TAP than to valid KTAP
> which always mentions test/test_case_name:
> 
> https://docs.kernel.org/dev-tools/ktap.html
> 
> We currently do the same thing for SKIP, e.g.:
> 
> #  RUN           ip_local_port_range.ip4_stcp.late_bind ...
> #      SKIP      SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> #            OK  ip_local_port_range.ip4_stcp.late_bind
> ok 17 # SKIP SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> 
> I'm not sure if we can realistically do surgery on the existing print
> helpers to add the test_name, because:
> 
> $ git grep 'ksft_test_result_*' | wc -l
> 915
> 
> That'd be a cruel patch to send.
> 
> But I do agree that adding the test_name to the prototype is a good
> move, to avoid others making the same mistake. Should we introduce
> a new set of helpers which take the extra arg and call them
> ksft_test_report_*() instead of ksft_test_result_*() ?
> 
> Maybe we're overthinking and a local fix in the harness is enough.
> 
> Kees, WDYT?

Yeah, let's separate this fix-up from the addition of the XFAIL logic.

-- 
Kees Cook