Re: [PATCH nft 1/3] tests/shell: skip "table_onoff" test if kernel patch is missing

[Florian Westphal <fw@strlen.de>]

Thomas Haller <thaller@redhat.com> wrote:
> As you probably run a self-built kernel, wouldn't you just `export
> NFT_TEST_FAIL_ON_SKIP=y` and reject all skips as failures? What's the
> problem with that? That exists exactly for your use case.

No, its not my use case.

The use case is to make sure that the frankenkernels that I am in charge
of do not miss any important bug fixes.

This is the reason for the feature probing, "skip" should tell me that
I can safely ignore it because the feature is not present.

I could built a list of "expected failures", but that will mask real
actual regressions.

> > This is a bug, and it tells me that I might have to do something
> > about it.
> 
> OK, do you intend to fix this bug in a very timely manner on Fedora 38
> (and other popular kernels)? Then maybe hold back the test until that
> happend? (or let it skip for now, and in a few weeks, upgrade to hard
> failure -- the only problem is not to forget about that).

I did keep the test back until I saw that -stable had picked it up.

I can wait longer, sure.

> Ah right. "tests/shell/testcases/transactions/table_onoff" is fixed on
> 6.5.6-200.fc38.x86_64. There still is a general problem. For example
> what about tests/shell/testcases/packetpath/vlan_8021ad_tag ?

Its also a bug that needs to be fixed in the kernel.
I applied it after stable had picked it up for 6.5.7.

> 1) the test would exit 78 instead of 77. And run-test.sh would treat 78
> either as failure or as skip, based on NFT_TEST_FAIL_ON_SKIP
> 
> 2) the test itself could look at NFT_TEST_FAIL_ON_SKIP and decide
> whether to exit with 77 or 1.
> 
> 
> Or how about adding a mechanism, that compares the kernel version and
> decides whether to skip? For example

I don't think that kernel versions work or are something that we can
realistically handle.  Even just RHEL would be a nightmare if one
considers all the different release streams.

I think even just handling upstream -stable is too much work.

That said, I hope that these kinds of tests will happen less frequently
over time.