Re: [RFC] spdx30: name, version and purl for dependency downloads

[Stefano Tondo <stondo@gmail.com>]

Hi Richard, hi Alex,

Thank you both for the feedback. Your replies point in the same
direction, so I answer them together here.

On Fri, 6 Nov 2026 at 12:04, Alexander Kanavin wrote:
> - do not make fetchers any more complex than they already are.
> [...] Rather try to do as much as possible in oe-core through a
> clear API.

On Fri, 6 Nov 2026 at 12:17, Richard Purdie wrote:
> I think that the fetcher is the right place to do this as I would
> be very much against poking fetcher internals from OE-Core. We need
> to declare and document the stage at which this is expected to work
> (i.e. before/after fetch, before/after unpack).
> [...]
> It should probably return structured data including the purl from
> some new API.

I think these two points fit together. The fetchers already parse the
name and the version of each dependency for their own use. The new
API only returns that data as a documented structure, with the purl.
It does not change the fetch or the unpack. oe-core then writes
the SPDX from it and does not read any fetcher internals.

About the stage: crate, gomod and npmsw know the identity when they
create the FetchData. That is before fetch, and it needs no network
access. npmsw only reads its shrinkwrap file, which is a local file
in the layer. gitsm is the exception, because it can list the
submodules only from the clone. When the submodules are in SRC_URI,
that exception goes away. So I propose to document "before fetch".

> I am thinking we should change the fetcher so all the submodule urls
> are included in SRC_URI in future and expanded_urldata() goes away.
> We need to update the other backends to match. I have no knowledge
> of npm so that one will be hard for me.

Alex wrote:
> - gitsm and npmsw are historical mistakes, however 'convenient' they
> may be. [...] I would want to replace them with flat, explicit
> implementations.

I agree, and I would like to help with npm. My idea is to follow
cargo-update-recipe-crates and go-mod-update-modules: an update class
writes one SRC_URI entry with a checksum for each package in the
shrinkwrap file. Then bitbake checks that the list matches the
shrinkwrap file, as you said in your reply. Is
that what you have in mind? If yes, I can prepare patches.

This also makes my RFC smaller. Its first part records implicit
downloads. With flat SRC_URI lists, there are no implicit downloads
left to record.

> We should revisit that and fix it as I think it exposes general
> bugs.

I can investigate this too. I have a reproducer that fails every time:
meta-selftest's git-submodule-test with a named SRCREV. The bug I
found is in Fetch.localpath() for a URL that is not in SRC_URI. It
creates a new FetchData with the recipe datastore, which does not
have the SRCREV that gitsm set for the submodule. Is this the bug you
mean, or do you know of others in this area? I can start from any
cases you have.

> If the fetcher can conform to our APIs and roughly follow how
> rust/crates are working today, I think those would be fine,
> particularly if we have good tests too.

Thanks, that is clear. I will come back to new fetchers only after
we agree on the API.

Joshua, Richard deferred question d) to you. In short: a lock file
lists packages for all platforms, so the fetcher downloads some
dependencies that are never built for the target. I proposed to take
the declared scope from the lock file in the fetcher, and the
observed scope from the language classes (for example cargo metadata
--filter-platform). Does that split work for you? And how do you want
"fetched but not built" to look in SPDX 3.0?

For the next step: crate and gomod already list every dependency in
SRC_URI, so the identity API can work for them now. Do you want me to
send patches for these two, with tests, and add npm after the
flattening? Or do you prefer that I wait for Joshua's answer first?

Thanks,
Stefano