Re: [OE-core][PATCH v5 0/8] Add SPDX 3.0 support

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Fri, 2024-07-05 at 08:17 +0100, Richard Purdie via lists.openembedded.org wrote:
> > On Wed, 2024-07-03 at 07:59 -0600, Joshua Watt via
> > lists.openembedded.org wrote:
> > > > This patch series add support for SPDX 3.0 and sets it as the
> > > > default.
> > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
> > > > the same time
> > > > 
> > > > v2: Added tests and addressed feedback
> > > > v3: Fixed several oe-selftest and build failures
> > > > v4: Fixed silly typo mistake in staging.bbclass
> > > > v5: Reworked to make SPDX 3 output reproducible by default. Variables
> > > >     that introduce non-reproducible output are documented as such.
> > 
> > Thanks for working on this. The patches generally seem to working well
> > in testing but this does look related:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio
> > 
> > Particularly these bits:
> > 
> > AssertionError: 10 != 0 : Missing objects in the cache:
> > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst
> > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst
> > 
> > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst
> > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst
> > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst
> > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst
> > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst
> > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst
> > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst

I think I understand what might have happened here. I suspect we've had
test builds on the new test cluster which can write to the hash
equivalence server but not the sstate. I think this means entries were
added for artefacts which don't exist on sstate. 

In theory those should be created on new build runs but in practise I
suspect they're shadowed by other artefacts so newer builds probably
don't create them. That would at least explain how we end up where we
are.

If I'm correct, if there are changes to the code (as we discussed on
yesterdays tech call), everything will rerun and assuming we run this
only on the main cluster, things should work out ok.

I would be curious to understand how these things aren't being
recreated when missing though?

I did also notice an interesting issue in that if you run a build with
PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the
do_collect_spdx_deps tasks and do_create_spdx tasks change hashes and
rerun. This isn't ideal as it effectively breaks our ability to turn
package backends on/off without rebuilds :(. Can we avoid that or at
least minimise things a bit more?

Cheers,

Richard