[RFC] spdx30: name, version and purl for dependency downloads

[Stefano Tondo <stondo@gmail.com>]

Hi all,

I would like your opinion on a direction for the SPDX output of
recipes with language dependencies. I have working patches for it,
described below, but I would rather agree on the approach before
sending them.

Yocto already knows each language dependency at fetch time for the
recipes that list them in SRC_URI (crate://, gomod://, gomodgit://, and
npmsw:// through its shrinkwrap file). The SPDX output loses most of
that: the downloads are recorded, but without a name, version or
package URL that a scanner can use, and dependencies fetched
implicitly (git submodules, shrinkwrap dependencies) are not recorded
at all.

Measured on current master (oe-core 3a3eef7afb, bitbake 0f33b68cb,
meta-openembedded 33f74f5dc5), source downloads in the build SPDX of
each recipe, today and with the patches:

                  today                   with the patches
  python3-bcrypt  42 crates, none with    all 42 with name,
                  version or purl         version and purl
  crucible        36 Go modules, none     all 36 with name,
                  with version or purl    version and purl
  p11-kit         gitsm, submodule        submodule recorded with
                  pkcs11-json missing     its own revision
  webserver-oob   2 downloads, the 70     72 downloads, 71 with
                  shrinkwrap packages     a purl
                  missing

webserver-oob is from TI's meta-tisdk, built on qemux86-64 without its
am335x and meta-ti runtime dependencies and without one git.ti.com
source that failed to clone, since neither oe-core nor
meta-openembedded has an npmsw recipe. crucible also fetches 67
go.mod-only entries (;mod=1), which are not compiled in and stay
without an identity.

Today a crate shows up as name "cipher-0.4.4.crate" with download
location https://static.crates.io/crates/cipher/0.4.4/download, and a
Go module as gomod://github.com/felixge/httpsnoop, with the version
only in the file name "github.com.felixge.httpsnoop@v1.0.4.zip".
Consumers end up parsing URLs and file names per ecosystem to recover
what the fetcher already knew.

I am aware of two earlier threads and would like to build on them, not
reopen them:

- Joshua's "lib: spdx30_tasks: Use Fetch.expanded_urldata()" (March
  2025), which stalled on an intermittent edk2-firmware gitsm failure
  on the autobuilder. I think I found the cause: the patch passes each
  expanded URL to oe.patch.patch_path(), which calls
  Fetch.localpath(url). For a URL that is not in SRC_URI, that creates
  a new FetchData with the recipe datastore, where the
  SRCREV_<submodule> that the gitsm fetcher sets in its own datastore
  copy does not exist. A recipe with only named SRCREVs, like
  edk2-firmware, then fails with "Please set a valid SRCREV". It
  reproduces every time with meta-selftest's git-submodule-test given
  a named SRCREV. My guess for the intermittency is that gitsm only
  lists the submodules when the clone is in DL_DIR.
- Stefan's "Concept for tightly coupled package manager" RFC (December
  2024). My reading of that thread is that explicit SRC_URI lists
  generated in layers (go-mod-update-modules,
  cargo-update-recipe-crates) are the preferred model and that
  fetchers should stay thin. This proposal assumes that model and does
  not add any dependency resolution to bitbake.

The patches have two parts.

1. Record implicit downloads

   add_download_files() also records what implicit_urldata() returns
   for each SRC_URI entry, but keeps the patch check on the SRC_URI
   entries, so implicit downloads are never set up again with the
   recipe datastore. The npmsw fetcher gets an implicit_urldata(),
   since it already builds the list of URLs internally (ud.proxy). The
   gitsm fetcher warns when the repository is not in DL_DIR, instead
   of silently returning no submodules.

2. Let the fetcher state the identity of what it downloads

   FetchData gets a package_identity attribute, None by default and
   set in urldata_init() by the fetchers that know it:

     crate     ("cargo", "cipher", "0.4.4", {})
     gomod     ("golang", "github.com/felixge/httpsnoop", "v1.0.4", {})
     gomodgit  same as gomod
     npm       ("npm", "@scope/name", "1.2.3", {})
     npmsw     the same for each registry dependency

   The last element holds purl qualifiers, so a crate or npm package
   from another registry gets a repository_url. go.mod-only entries
   get None. create-spdx-3.0 sets the name, software_packageVersion and
   software_packageUrl from it and builds the purl in oe-core, so
   bitbake does not need to know the purl specification.

   I first had a FetchMethod method in mind, but npmsw fetches its
   dependencies through plain https URLs, so it has to attach the
   identity to each of them, and an attribute is enough for that. No
   URL syntax changes, unlike the dn=/dv= parameters in Stefan's
   series.

Both parts are small, about 35 lines in bitbake and 40 in oe-core plus
tests. I tested them on master with new bitbake fetcher unit tests and
new oe-selftest cases (submodules, including the named SRCREV case,
and crate identity), which fail without the patches, and with the
existing fetcher tests and the SPDX selftests closest to the change.
Two bugs I found in the existing Git handling while doing this (gitsm
skipped, changes not reaching the task signature) are fixed in a
separate series, which the oe-core part builds on:
https://lore.kernel.org/openembedded-core/20261104112257.866914-1-stondo@gmail.com/

Questions:

a) Is the fetcher the right place for the identity, or would you
   prefer oe-core to derive it from each fetcher's FetchData? The
   latter avoids touching bitbake, but for npmsw it means reading the
   internals of its proxy fetcher.

b) Should bitbake keep returning the parts and leave the purl to
   oe-core, as above, or return a purl directly?

c) For ecosystems without a fetcher (.NET/NuGet, Maven), would a
   fetcher plus an update class following go-mod-update-modules be
   welcome, or is that better kept in a layer?

d) Dependency scope. Identity alone does not give a correct SBOM: a
   lock file lists what may be needed on any platform. python3-bcrypt
   fetches wasi, wasip2 and wit-bindgen (WebAssembly only) and r-efi
   (UEFI), and with the patches they show up with a clean purl as if
   they were part of the product. As a follow-up I tried running
   cargo metadata --filter-platform ${RUST_HOST_SYS} in cargo.bbclass:
   of the 42 crates, 25 are runtime, 12 only run at build time (build
   scripts, proc-macros) and 5 are not used on the target. I recorded
   them as LifecycleScopedRelationship dependsOn with scope runtime or
   build, and an annotation on the unused ones. Go (go list -deps) and
   npm (dev, optional, os and cpu in the shrinkwrap) could work the
   same way. Does that split sound right: declared scope from the
   fetcher where the lock file has it, observed scope from the
   language classes? And is there a preferred SPDX representation for
   "fetched but not built"?

Thanks,
Stefano