Re: [OE-core][PATCH v5 0/8] Add SPDX 3.0 support

Joshua Watt
2024-07-10T14:03:41Z

On Wed, Jul 10, 2024 at 6:18 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Fri, 2024-07-05 at 08:17 +0100, Richard Purdie via lists.openembedded.org wrote:
> > > On Wed, 2024-07-03 at 07:59 -0600, Joshua Watt via
> > > lists.openembedded.org wrote:
> > > > > This patch series add support for SPDX 3.0 and sets it as the
> > > > > default.
> > > > > Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
> > > > > the same time
> > > > >
> > > > > v2: Added tests and addressed feedback
> > > > > v3: Fixed several oe-selftest and build failures
> > > > > v4: Fixed silly typo mistake in staging.bbclass
> > > > > v5: Reworked to make SPDX 3 output reproducible by default. Variables
> > > > > that introduce non-reproducible output are documented as such.
> > >
> > > Thanks for working on this. The patches generally seem to working well
> > > in testing but this does look related:
> > >
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio
> > >
> > > Particularly these bits:
> > >
> > > AssertionError: 10 != 0 : Missing objects in the cache:
> > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst
> > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst
> > >
> > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst
> > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst
> > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst
> > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst
> > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst
> > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst
> > > /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst
>
> I think I understand what might have happened here. I suspect we've had
> test builds on the new test cluster which can write to the hash
> equivalence server but not the sstate. I think this means entries were
> added for artefacts which don't exist on sstate.
>
> In theory those should be created on new build runs but in practise I
> suspect they're shadowed by other artefacts so newer builds probably
> don't create them. That would at least explain how we end up where we
> are.

There is also https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n933
which I beleive is still accurate and probably why the images
themselves show up; we'll need to adjust the exclusions to match the
new SPDX tasks I added. I think '.*:do_create.*_spdx' would be OK?

>
> If I'm correct, if there are changes to the code (as we discussed on
> yesterdays tech call), everything will rerun and assuming we run this
> only on the main cluster, things should work out ok.
>
> I would be curious to understand how these things aren't being
> recreated when missing though?

I'm not sure. I can't really think of any reason in the SPDX code that
it wouldn't recreate a missing sstate object; I'm not doing anything
particularly strange with sstate

>
> I did also notice an interesting issue in that if you run a build with
> PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the
> do_collect_spdx_deps tasks and do_create_spdx tasks change hashes and
> rerun. This isn't ideal as it effectively breaks our ability to turn
> package backends on/off without rebuilds :(. Can we avoid that or at
> least minimise things a bit more?

Ya, that's a leftover from when I was trying to tie the SPDX packages
to the produced package files. I had to give up thought because it's
actually really difficult to figure out what .rpm and .deb files
correspond to a PACKAGE (at least, I couldn't see an easy way to do
it). We can just remove the dependency on do_package_write_* for now
and figure out how to do it later.

>
> Cheers,
>
> Richard
>
>
>