Re: [OE-core][PATCH v5 0/8] Add SPDX 3.0 support

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2024-07-10 at 08:03 -0600, Joshua Watt wrote:
> On Wed, Jul 10, 2024 at 6:18 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > On Fri, 2024-07-05 at 08:17 +0100, Richard Purdie via lists.openembedded.org wrote:
> > > > On Wed, 2024-07-03 at 07:59 -0600, Joshua Watt via
> > > > lists.openembedded.org wrote:
> > > > > > This patch series add support for SPDX 3.0 and sets it as the
> > > > > > default.
> > > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
> > > > > > the same time
> > > > > > 
> > > > > > v2: Added tests and addressed feedback
> > > > > > v3: Fixed several oe-selftest and build failures
> > > > > > v4: Fixed silly typo mistake in staging.bbclass
> > > > > > v5: Reworked to make SPDX 3 output reproducible by default. Variables
> > > > > >     that introduce non-reproducible output are documented as such.
> > > > 
> > > > Thanks for working on this. The patches generally seem to working well
> > > > in testing but this does look related:
> > > > 
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio
> > > > 
> > > > Particularly these bits:
> > > > 
> > > > AssertionError: 10 != 0 : Missing objects in the cache:
> > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst
> > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst
> > > > 
> > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst
> > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst
> > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst
> > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst
> > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst
> > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst
> > > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst
> > 
> > I think I understand what might have happened here. I suspect we've had
> > test builds on the new test cluster which can write to the hash
> > equivalence server but not the sstate. I think this means entries were
> > added for artefacts which don't exist on sstate.
> > 
> > In theory those should be created on new build runs but in practise I
> > suspect they're shadowed by other artefacts so newer builds probably
> > don't create them. That would at least explain how we end up where we
> > are.
> 
> There is also https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n933
> which I beleive is still accurate and probably why the images
> themselves show up; we'll need to adjust the exclusions to match the
> new SPDX tasks I added. I think '.*:do_create.*_spdx' would be OK?

If that restriction was just for images I'd be ok with it. It is saying
we'd ignore any missing spdx sstate artefacts anywhere though which
seems wrong, assuming I understand it correctly.

> > If I'm correct, if there are changes to the code (as we discussed on
> > yesterdays tech call), everything will rerun and assuming we run this
> > only on the main cluster, things should work out ok.
> > 
> > I would be curious to understand how these things aren't being
> > recreated when missing though?
> 
> I'm not sure. I can't really think of any reason in the SPDX code that
> it wouldn't recreate a missing sstate object; I'm not doing anything
> particularly strange with sstate

Imagine you have some tasks which depend on do_package like
do_package_write_ipk and friends. If you only ever build the ipk task,
the package task would never run again. I suspect spdx may have similar
tasks.

> > I did also notice an interesting issue in that if you run a build with
> > PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the
> > do_collect_spdx_deps tasks and do_create_spdx tasks change hashes and
> > rerun. This isn't ideal as it effectively breaks our ability to turn
> > package backends on/off without rebuilds :(. Can we avoid that or at
> > least minimise things a bit more?
> 
> Ya, that's a leftover from when I was trying to tie the SPDX packages
> to the produced package files. I had to give up thought because it's
> actually really difficult to figure out what .rpm and .deb files
> correspond to a PACKAGE (at least, I couldn't see an easy way to do
> it). We can just remove the dependency on do_package_write_* for now
> and figure out how to do it later.

Ok, that makes sense. We should be able to map package files to
PACKAGES but we can discuss that one later!

Cheers,

Richard