Re: [yocto] [yocto-autobuilder-helper][PATCH 0/3] fix test results storage for mickledore

[Alexandre Belloni <alexandre.belloni@bootlin.com>]

On 14/06/2023 14:15:54+0200, Alexis Lothor� wrote:
> On 6/14/23 12:31, Richard Purdie wrote:
> > On Wed, 2023-06-14 at 10:56 +0200, Alexis Lothor� via
> > lists.yoctoproject.org wrote:
> >> From: Alexis Lothor� <alexis.lothore@bootlin.com>
> >>
> >> This series is a follow-up for the 4.3_M1.rc1 regression report issue.
> >>
> >> It has been observed that the report is empty. This issue is linked to
> >> configuration description in yocto-autobuilder-helper, and has been
> >> identified through the following steps:
> >> - empty report is supposed to be a comparison between yocto-4.2 (4.2.rc3)
> >>   and 4.3_M1.rc1
> >> - yocto-4.2 results are almost empty: we only find test results from Intel
> >>   QA (pushed _after_ the AB build) and not the AB test results
> >> - tests results are managed by send-qa-email.send-qa-email uses resulttool
> >>   to systematically gather and store test results in local git directory
> >> - however, it looks for basebranch/comparebranch to know if those results
> >>   can be pushed onto git server, and those variables depend on config.json
> >>   content
> >> - yocto-4.2 (4.2.rc3) has been built on release branch mickledore
> >>   (https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5212)
> >> - since mickledore is not yet described in config.json, send-qa-email
> >>   considers it as a "work" branch (contrary to a "release" branch) and does
> >>   not push test results
> >>
> >> As a consequence:
> >> - first commit brings in python logger
> >> - second commit adds a warning when such case happen, since we are able to
> >>   detect it
> >> - third fix actually adds mickledore as a release branch to properly store
> >>   again test results
> >>
> >> There must be a more robust rework to do (because the issue will likely
> >> happen on each major delivery), but I aimed for the quick and small fix to
> >> quickly bring back tests results storage without breaking other things in
> >> the process
> >>
> >> Alexis Lothor� (3):
> >>   scripts/send-qa-email: use logger instead of raw prints
> >>   scripts/send-qa-email: print warning when test results are not stored
> >>   config.json: add mickledore as direct push branch for test results
> > 
> > Thanks for the analysis. I agree we need to somehow fix this properly.
> > One solution might be to always push for poky if the branch name
> > doesn't end with -next?
> 
> That might work indeed. If we are sure enough that no custom/feature branch will
> be used in poky with send-qa-email (ie, only in poky-contrib), I can do the fix
> this way

I sometimes use a different branch name when testing things out (like 64
bit time) but as long as we all know, we can probably ensure this ends in
-next.

> > 
> > Since we have the release artefacts for the release, could we add the
> > test results after the fact now?>
> > Id' be interested to see the 4.3 M1 to 4.2 comparison rerun with that
> > added.
> 
> I am not sure about where to find those artifacts for yocto-4.2 ? If you are
> referring to https://autobuilder.yocto.io/pub/, yocto-4.2 has already been
> removed from there. And if you are referring to the archived release on main
> site
> (https://downloads.yoctoproject.org/releases/yocto/yocto-4.2/poky-21790e71d55f417f27cd51fae9dd47549758d4a0.tar.bz2),
> it does contain a single, 40 line testresults.json, so that's definitely not the
> full AB tests results.
> 
> > 
> > Cheers,
> > 
> > Richard
> > 
> > 
> > 
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#60297): https://lists.yoctoproject.org/g/yocto/message/60297
> > Mute This Topic: https://lists.yoctoproject.org/mt/99523809/7394887
> > Group Owner: yocto+owner@lists.yoctoproject.org
> > Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/12378809/7394887/1227806781/xyzzy [alexis.lothore@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> > 
> 
> -- 
> Alexis Lothor�, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
> 

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com