Re: tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)

[Ingo Molnar <mingo@kernel.org>]

* Guillaume Tucker <guillaume.tucker@collabora.com> wrote:

> Hi Ingo,
> 
> On 07/02/2023 09:53, Ingo Molnar wrote:
> > 
> > * kernelci.org bot <bot@kernelci.org> wrote:
> > 
> >> tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)
> >>
> >> Full Build Summary: https://kernelci.org/build/tip/branch/master/kernel/v6.2-rc7-273-gd67c17ddc899/
> >>
> >> Tree: tip
> >> Branch: master
> > 
> > Is anyone maintaining this CI build bot? Because it's either unmaintained, 
> > or is blatantly lying about regressions.
> > 
> > Let's see one of the 'errors' it reported today:
> 
> Yes, it's being maintained.

Good, and thanks for the quick response. :-)

> However there's no way for the KernelCI core team to manually verify all 
> the errors reported. So kernel developers following up on such errors is 
> the best way to handle them and get them resolved.  Thanks for replying 
> to the email report :)
> 
> >> Errors summary:
> >>
> >>     4    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
> > 
> > ... but this exact same error was 'reported' a year ago on January 22:
> > 
> >>    2    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
> > 
> > So these regression reports are useless in this form and they clutter 
> > people's inboxes. Is any person reading them and acting on them to make 
> > sure these emails are sensible?
> 
> They are not reported as regressions but as build errors.  So it doesn't 
> matter when it started failing or if this has always failed, any build 
> error will show up in the report.  It's definitely something to improve, 
> the runtime test reports differentiate regressions or "new failures" and 
> errors that have always been there.  This will also apply to builds at 
> some point, it's on the roadmap for the next few months.  Please bear 
> with us in the meantime.

At least the title of the email should report the 'delta' status of any 
tree it is testing: the difference to Linus's tree.

For example if Linus's tree is:

   linus/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-xxx-xxxxxxxxxxx)

   [ mockup. ]

Then the tip/master email title should not be:

   tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)

But something like this 'delta' report:

   tip/master build: 205 builds: PASS (v6.2-rc7-273-gd67c17ddc899)

Or if tip/master introduced new build failures/warnings, it should not 
report this:

   tip/master build: 205 builds: 6 failed, 199 passed, 10 errors, 24 warnings (v6.2-rc7-273-gd67c17ddc899)

... but should report something like this:

   tip/master build: 205 builds: FAIL [+1 errors, +2 warnings] (v6.2-rc7-273-gd67c17ddc899)

BTW., if I got such 'delta' emails I could guarantee to handle each and 
every one of them. With the current format it's the occasional spot check 
of these emails and a frustrating excercise of trying to figure out the 
delta to Linus's tree - and failing.

Yes, if tip/master has an older base then these might not be entirely 
accurate deltas, but *that* is something maintainers can fix in their 
routine flow: merge in Linus's latest or fix a bug newly introduced 
upstream.

But prominently displaying years-old bugs as 'errors' in the title is not 
just misleading, it's actively counterproductive. The primary output of 
continuous integration reports must always be 'delta' based.

> About the actual kernel build error, I guess it's a shame nobody has 
> fixed this yet. [...]

The main problem with the reports here is that there's no way to determine 
'at a glance' whether there's any regressions compared to Linus's tree - 
which is really what maintainers of non-Linus trees care about.

The figures the bot sends out as email titles are fine for Linus's tree I 
suspect, but are misleading in this form for any maintainer tree.

Thanks,

	Ingo