Re: [PATCH net] selftests: netdevsim: Fix ethtool-features.sh fail

[Sabrina Dubroca <sd@queasysnail.net>]

2025-11-03, 16:01:33 -0800, Jakub Kicinski wrote:
> On Mon, 3 Nov 2025 11:13:08 +0100 Sabrina Dubroca wrote:
> > 2025-10-30, 17:02:17 -0700, Jakub Kicinski wrote:
> > > On Fri, 31 Oct 2025 00:13:59 +0100 Sabrina Dubroca wrote:  
> > > > I guess it's improving the situation, but I've got a system with an
> > > > ethtool that accepts the --json argument, but silently ignores it for
> > > >  -k (ie `ethtool --json -k $DEV` succeeds but doesn't produce a json
> > > > output), which will still cause the test to fail later.  
> > > 
> > > And --json was added to -k in Jan 2022, that's pretty long ago.
> > > I'm not sure we need this aspect of the patch at all..  
> > 
> > Ok.  Then maybe a silly idea: for the tests that currently have some
> > form of "$TOOL is too old" check, do we want to remove those after a
> > while? If so, how long after the feature was introduced in $TOOL?
> > 
> > Or should we leave them, but not accept new checks to exclude
> > really-old versions of tools?  Do we need to document the cut-off ("we
> > don't support tool versions older than 2 years for networking
> > selftests" [or similar]) somewhere in Documentation/ ?
> 
> FWIW my current thinking is to prioritize test development and kernel
> needs over the ability to run ksft on random old set of tools and have
> clean skips. IOW avoid complicating writing tests by making the author
> also responsible for testing versions of all tools.

I see. I liked Andrew's idea ("embed the date the requirement was
added into the test"), but it goes completely in the opposite
direction.

Figuring out why exactly a test failed in case of an old tool
(unexpected output passed to some pipe/parsing, exit with a non-zero
code, maybe other issues) is not always obvious. So without version
checks on the tools, I think we have to assume that the test requires
the latest version of all tools it calls (or at least a very recent
one). Which I guess is reasonable for upstream kernel development.

> The list of tools which need to be updated or installed for all
> networking tests to pass is rather long. My uneducated guess
> is all these one off SKIP patches don't amount to much. Here for
> example author is fixing one test, I'm pretty sure that far more
> tests depend on -k --json.

A quick grep found only a few more (in python scripts under
drivers/net) for -k. But (also from a quick grep) many tests seem to
use jq without checking that the command is present.

So I guess you would lean toward not accepting any such patch, not
requiring new tests to have SKIP checks, but leaving any existing
checks in? (and I suspect removing all the existing ones wouldn't
actually reduce the flow of "add check for too old $tool" patches, so
it probably doesn't make sense to do that)

-- 
Sabrina