Re: [OE-core] [PATCH v2 1/2] oeqa/sdk: update cryptodev to a revision that builds on kernel 6.18

[Trevor Woerner <twoerner@gmail.com>]

On Mon 2026-08-17 @ 01:48:55 PM, Richard Purdie wrote:
> On Mon, 2026-08-17 at 12:51 +0200, Alexander Kanavin via lists.openembedded.org wrote:
> > On Sun, 16 Aug 2026 at 13:12, Paul Barker via lists.openembedded.org
> > <paul=pbarker.dev@lists.openembedded.org> wrote:
> > 
> > > I'm wondering why we haven't spotted this issue previously, we've been
> > > using Linux 6.18 for months now. Are we missing a test case?
> > 
> > I suspect it's because the execution of useful cases for sdk and image
> > runtimes tends to regress due to 'soft-skip', e.g. if a needed package
> > isn't present, the test isn't run. I don't think this is quite right,
> > and each particular image and sdk should check that particular tests
> > do run, and if they aren't, that's a fail in itself. I'm not yet sure
> > how to technically implement this.
> 
> I've worried about this for a while too.
> 
> We really need to mark up the images with the lists of tests we expect
> to pass. It is tricky as that list can vary depending on distro
> features and package classes but should be possible.

Agreed on the goal. Having gone through what the cases actually do, I
want to be careful about how much of it one mechanism can cover, because
the skips are not all the same shape.

Some are a stated requirement. cmake, gtk3, kmod, maturin, meson, perl
and python each name a package and skip when the SDK's manifest does not 
list it. Those are already machine-readable, and one narrow check on
them looks worth having by itself: if a case names a requirement, the 
manifest satisfies it, and the case skips anyway, that is a bug rather
than a configuration. That is exactly what the twelve above were, and 
catching it needs no per-image expected list.

The rest are not declarations at all. autotools, cmake, gcc, makefile
and meson skip on "SDK doesn't contain a supported C library"; gcc, go
and rust skip on not finding a cross-canadian toolchain; go also skips
when the command raises. Those are runtime probes of the SDK that was 
installed, and marking up an image cannot say in advance what they will
decide. The go and rust cases are among the ones that skip on a default
SDK, so this is not a rare shape.

So I do not think declarations on their own produce the expected-test
list, and I would rather say that now than discover it half way through.
The configuration-dependence you name looks like the hard part, and I do
not have a design for it. 

What I am willing to commit to is the narrow check above, since the 
oeqa/sdk change it builds on is already written. If that seems useful I
will send it as its own patch and we can see whether it earns its keep
before anyone builds anything larger on top.

> The other option is to have resulttool have more checks where it
> compares to a list of tests we should see running. The regression
> report is supposed to partly handle that but if you miss a regression,
> the difference will no longer show it...
> 
> Cheers,
> 
> Richard