Re: [OE-core] [PATCH 0/5] linux-yocto: consolidated pull request

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Mon, 2020-05-04 at 09:06 -0400, Bruce Ashfield wrote:
> On Mon, May 4, 2020 at 8:56 AM Bruce Ashfield via
> lists.openembedded.org
> <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
> > On Mon, May 4, 2020 at 4:31 AM Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > > Hi Bruce,
> > > 
> > > On Sun, 2020-05-03 at 11:44 -0400, bruce.ashfield@gmail.com wrote:
> > > > From: Bruce Ashfield <bruce.ashfield@gmail.com>
> > > > 
> > > > Hi all,
> > > > 
> > > > Here are the -stable updates I've collected since m3 of dunfell. I
> > > > ran things through the autobuilder, and no new kernel issues were
> > > > picked up.
> > > > 
> > > > The -dev bump is good for master, while all of the 5.4-stable bumps
> > > > are good for both master and dunfell.
> > > 
> > > I pulled these into master-next and saw reproducibility issues. I
> > > wondered why your test build didn't show it, turns out it also did:
> > > 
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1024
> > > 
> > > 2020-05-04 05:18:46,863 - oe-selftest - INFO - ======================================================================
> > > 2020-05-04 05:18:46,863 - oe-selftest - INFO - FAIL: reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase)
> > > 2020-05-04 05:18:46,863 - oe-selftest - INFO - ----------------------------------------------------------------------
> > > 2020-05-04 05:18:46,864 - oe-selftest - INFO - testtools.testresult.real._StringException:
> > > 
> > > AssertionError: The following deb packages are missing or different: /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-36044/reproducibleB/tmp/deploy/deb/./qemux86_64/kernel-module-kheaders-5.4.38-yocto-standard_5.4.38+git0+f405543442_6cb5b11e83-r0_amd64.deb
> > > The following ipk packages are missing or different: /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-36044/reproducibleB/tmp/deploy/ipk/./qemux86_64/kernel-module-kheaders-5.4.38-yocto-standard_5.4.38+git0+f405543442_6cb5b11e83-r0_qemux86_64.ipk
> > > 
> > 
> > I don't see any changes in the kernels that would have triggered it,
> > but they never are obvious.
> 
> I am working on some kernel reproducibility changes that were
> discussed about three weeks ago.
> 
> Is the default for reproducible builds in master-next different from
> what it was before dunfell was releases ? If so, that might explain
> why I'm not seeing a change that would have triggered it (i.e. this is
> unrelated to my -stable changes).

There are no reproducibility changes between master-next and dunfell.

> Like I was mentioning before, if there's a document or pastebin of
> steps I can follow, I can run the tests against my pending changes and
> finish them for master. They of course would not necessarily be
> suitable for dunfell.

The good news is the autobuilder has some debugging built in for this
kind of problem. You can see the diffoscope output here:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200504-vhr0o_t9/packages/diff-html/

which tells us its kheaders.ko in .rodata by the looks of it. Some
changes are cause, some are effect, e.g. the build ID is different as
the sections are different so that isn't the issue but an effect of it.
It looks like diffoscope doesn't know how to decode the .rodata which
could be compressed?

The raw data is also there (the packages that differ), e.g.:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200504-vhr0o_t9/packages/reproducibleA/tmp/deploy/deb/qemux86_64/kernel-module-kheaders-5.4.38-yocto-standard_5.4.38%2Bgit0%2Bf405543442_6cb5b11e83-r0_amd64.deb

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200504-vhr0o_t9/packages/reproducibleB/tmp/deploy/deb/qemux86_64/kernel-module-kheaders-5.4.38-yocto-standard_5.4.38%2Bgit0%2Bf405543442_6cb5b11e83-r0_amd64.debe

which would allow you to look further at them by hand.

You can manually run this test locally with:

oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds

but it will take a while to run as it has to run two builds and compare
them and only uses sstate for one of them. To make it faster since you
know where the issue is, edit
meta/lib/oeqa/selftest/cases/reproducible.py to change images (core-
image-sato, core-image-minimal and friends to simply "virtual/kernel".

Cheers,

Richard