Re: Autobuilder reproducibility target changes

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Sun, 2021-02-14 at 13:17 -0600, Joshua Watt wrote:
> On Sun, Feb 14, 2021 at 6:19 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > Regular users of the autobuilder will note that I've split the
> > reproducible builds test out of the main oe-selftest build and into its
> > own target build. This is because that test tends to run for a lot
> > longer time period and it helps to see the result separately.
> > 
> > I've only done this for master. If gatesgarth and dunfell want to
> > follow, that should be straight forward with a change to the branch in
> > autobuilder-helper. Obviously we should ensure this is working ok with
> > master first but so far so good.
> > 
> > It has already highlighted the difference between a successful run:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/115/builds/2
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/119/builds/2
> > (took 3-4 hours)
> > 
> > and failing two failing runs:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/116/builds/2
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/118/builds/2
> > (took 9 hours)
> 
> OK, I read through the code and unfortunately found a bug: when
> attempting to make sure the "B" build doesn't use sstate, I misspelled
> the SSTATE_MIRRORS, which means that the B build could have been
> pulling from the sstate mirror when it was not supposed to. This has a
> few implications:
> 
>  1) It might explain why some of the reproducible results seem intermittent
>  2) It might explain why there is such a time disparity between the tests

The "good" news is that this didn't affect the autobuilder as it sets
SSTATE_DIR to a common directory and doesn't use SSTATE_MIRRORS.

> Unfortunately, while it probably will help the intermittent results,
> it probably means that the tests taking 9 hours is what is "supposed"
> to happen, and they happen to be shorter sometimes because the B build
> is pulling from sstate when it's not supposed to.

I don't think we're to the bottom of this. If its not spending the time
in diffoscope, something seems to cause builds with differences to take
much longer...

Cheers,

Richard