Re: [PATCH net-next V10 2/6] selftest: netdevsim: Add devlink rate tc-bw test

[Jakub Kicinski <kuba@kernel.org>]

On Wed, 21 May 2025 10:05:13 +0300 Tariq Toukan wrote:
> On 21/05/2025 1:59, Jakub Kicinski wrote:
> > On Tue, 20 May 2025 21:38:03 +0300 Tariq Toukan wrote:  
> >> Test verifies that netdevsim correctly implements devlink ops callbacks
> >> that set tc-bw on leaf or node rate object.  
> > 
> > Please add a test that can actually validate a NIC HW.
> > The test probably needs to be in Python to use a remote endpoint,
> > and should live under tools/testing/../drivers/net/hw
> > 
> > We had a long conversation about what we expect from the API
> > vs how your HW works. One of the test cases should confirm
> > the expected behavior, IOW fail on mlx5. Which is fine,
> > unlikely that any NIC will have 100% compliance. But at
> > least we will be documenting the expectations.
> 
> No problem with that, we'll add.
> 
> We could've saved this extra cycle if my questions [1] exactly about 
> this topic weren't ignored.
> Area is vague and not well defined. We can continue with the iterative 
> guess and fix cycles, or alternatively get it clearly and formally defined.

I started a couple of times on answering but my hands go a little limb
when I have to explain things so obvious like "testing is a crucial part
of software development" :S  I mean.. nvidia certainly tests their code,
so I'm not sure where the disconnect is. I had a short conversation with
Gal at some conference where he, AFAIU, was doubting that device testing
can be part of an open source project.

It certainly is not advantageous to companies to have to share their
test code. So when you ask me for details on the rules what I hear is
"how can we make sure we do as little as possible".

Broadly, any new uAPI should come with tests which exercise the
functionality. We started a decade or so ago with netdevsim tests
which just validate the API layer itself. That did not provide
sufficient validation of the real implementations, crucially it did 
not check whether shallow APIs (devlink) actually behave the same
when implemented by multiple vendors. So two years ago we built 
the Python harness to be able to write tests for NIC functionality.
That is the level of testing we expect now. Obviously there will always
be exceptions. For instance I was pushing for common tests for the time
sync code (DPLL etc.) but I was convinced by the experts that it's hard
and that they generally test with $x0,000 measurement equipment.
So fair, I guess that's too hard. But for BW shaping tests? 
IIRC mlxsw has qdisc offload tests for BW shaping upstream.