Re: [PATCH v5 1/2] selftests: net: Create veth pair for testing in networkless kernel

[Abhinav Jain <jain.abhinav177@gmail.com>]

On Fri, 9 Aug 2024 21:19:11 -0700, Jakub Kicinski wrote:

> > On Thu, 8 Aug 2024 09:23:09 -0700, Jakub Kicinski wrote:
> > > A number of checks now return SKIP because veth doesn't support all
> > > ethtool APIs.
> > >
> > > In netdev selftests we try to make sure SKIP is only used when test
> > > cannot be performed because of limitations of the environment.
> > > For example some tool is not installed, kernel doesn't have a config.
> > > Something that the person running the test is able to fix by fixing
> > > how the test is run.
> > >
> > > Running this test on veth will always SKIP, nothing CI system can do.
> > > Please make the test use the keyword XFAIL instead of SKIP when
> > > functionality is not supported by the underlying driver.  
> > 
> > Ack, understood. I will do that, one clarification though.
> > Currently, the tests are using either PASS or FAIL and no SKIP. Based on
> > the above suggestion, it seems that I have replace FAIL with XFAIL for all
> > the tests that fail due to functionality not being supported by the
> > underlying driver.
>
> Right, sorry for lack of clarity.
>
> Our CI doesn't fully trust the exit codes, so even though the test
> exits with zero the CI parses the output and finds the "SKIP: ..."
> lines. You need to replace those "SKIP"s in the output with "XFAIL".

I re-tested and found that currently only two APIs are tested, "dump" and
"stats". For veth pair, the only test that fails currently with a SKIP is
the dump operation.

```
# Cannot get register dump: Operation not supported
# SKIP: veth1: ethtool dump not supported
```

This is present in kci_netdev_ethtool_test function, please confirm if this
is the one that I need to change to XFAIL. The logic to incorporate
the failure exit code for dump operation (74) is already in place.

```
kci_netdev_ethtool_test 74 'dump' "ethtool -d $netdev"
```

I just need to change the SKIP in this function to XFAIL. In case, if you
were referring to any other tests that are failing (features on/off),
please let me know. Thank you for prompt feedbacks, really appreciated.