Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Thu, 2023-10-19 at 14:28 +0200, Alexis Lothoré wrote:
> Hi Ross,
> On 10/19/23 13:04, Ross Burton wrote:
> > > On 19 Oct 2023, at 09:36, Alexis Lothoré via lists.yoctoproject.org <alexis.lothore=bootlin.com@lists.yoctoproject.org> wrote:
> > I just skimmed your report and have some feedback to hopefully make it easier to read in the future.
> > 
> > I’d suggest sorting the output in order of importance.  For example, this is a section that I really don’t care about:
> > 
> > Match:       sdk_core-image-sato_x86_64_fvp-base_20230910083055
> >              sdk_core-image-sato_x86_64_fvp-base_20231017222150
> > 
> > Put those at the bottom, or even better collate them into a single section where there have been no changes.
> > 
> > Similarly:
> > 
> > Match:       runtime_core-image-sato_qemux86_20230911011430
> >              runtime_core-image-sato_qemux86_20231017223736
> >     Additionally, 1 new test(s) is/are present
> > 
> > I guess marginally more important than identical results, but “there are new tests that passed” isn’t very interesting.
> 
> Indeed, not so useful and hiding the real content at the bottom
> 
> > Regression: oeselftest_ubuntu-22.04_qemux86-64_20230911011940
> >     oeselftest_almalinux-9.2_qemux86-64_20231017221342
> > 
> > Should they have matched? The host distro doesn’t match and this matters for some of the tests, as some distros don’t support some of the selftests. In this case specifically, there are seven regressions and six of them are specific to the host changing, which has the side-effect of hiding the one actual regression.
> 
> Yeah, that's a point I have been struggling with when starting to update those
> tools. The initial assumption I have started working with, after discussing the
> matter with Richard (see [1]), is the following:
> 1. MACHINE _must_ match between base and target
> 2. Different HOSTS _can_ be cross-checked
> 
> But the issue you are pointing tends to show it does not work well in some
> cases. I will have to do some tests to see if dropping 2. reduce this noise
> without loosing valuable data, or if we need to find something smarter

What the autobuilder does is either runs one selftest on a random host
for q-quick, or for a-full it will run five selftests, one for
"centos", one for "arm", one for "ubuntu", one for "fedora" and one for
"debian".

Ideally we'd therefore compare debian to debian if we have any choice.
It is a question of finding the closest matches. That is hard in code
though whilst keeping it generic.

Cheers,

Richard