* [PATCH v2 0/2] fetch: package identity of crate, gomod and npm URLs
@ 2026-11-19 20:48 Stefano Tondo
2026-11-19 20:48 ` [PATCH v2 1/2] fetch: Add package_identity() to the fetchers Stefano Tondo
2026-11-19 20:48 ` [PATCH v2 2/2] fetch: Return the package identity of crate, gomod and npm URLs Stefano Tondo
0 siblings, 2 replies; 3+ messages in thread
From: Stefano Tondo @ 2026-11-19 20:48 UTC (permalink / raw)
To: bitbake-devel
Cc: Joshua Watt, Alexander Kanavin, Stefan Herbrechtsmeier,
Paul Barker, Mathieu Dubois-Briand, Peter Marko,
stefano.tondo.ext
Hi,
This series adds the fetcher API from the RFC "spdx30: name, version
and purl for dependency downloads" [1]. Richard asked for a fetcher
API that returns structured data with the purl, so that OE-Core does
not read fetcher internals.
Patch 1 adds FetchMethod.package_identity(). It returns a
PackageIdentity dataclass with the name, the version and the purl of
the downloaded package, or None. It uses only the URL and the
datastore, so it works before the fetch and without network access.
Patch 2 implements it for the crate, gomod, gomodgit and npm fetchers,
with unit tests.
The RFC also covered the downloads that gitsm and npmsw add by
themselves. That part is not in this series. Richard and Alexander
prefer to list those downloads in SRC_URI. I will send that change
for npm in a separate series.
Changes in v2:
- package_identity() returns a PackageIdentity dataclass instead of a
dict, as Paul proposed [2]. The dataclass is frozen.
- Rebased on master.
Tests: bitbake-selftest bb.tests.fetch.PackageIdentityTest passes.
bb.tests.fetch without the network tests gives the same results as on
master.
v3 of the OE-Core series that uses the API follows on the OE-Core
list.
[1] https://lore.kernel.org/openembedded-core/20261105122145.543282-1-stondo@gmail.com/
[2] https://lore.kernel.org/bitbake-devel/DMZ3ERL47CEB.3KE9TOG6YIZPD@pbarker.dev/
Stefano Tondo (2):
fetch: Add package_identity() to the fetchers
fetch: Return the package identity of crate, gomod and npm URLs
lib/bb/fetch/__init__.py | 38 ++++++++++++++++++++++++
lib/bb/fetch/crate.py | 7 +++++
lib/bb/fetch/gomod.py | 16 ++++++++++
lib/bb/fetch/npm.py | 7 +++++
lib/bb/tests/fetch.py | 64 ++++++++++++++++++++++++++++++++++++++++
5 files changed, 132 insertions(+)
--
2.56.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH v2 1/2] fetch: Add package_identity() to the fetchers
2026-11-19 20:48 [PATCH v2 0/2] fetch: package identity of crate, gomod and npm URLs Stefano Tondo
@ 2026-11-19 20:48 ` Stefano Tondo
2026-11-19 20:48 ` [PATCH v2 2/2] fetch: Return the package identity of crate, gomod and npm URLs Stefano Tondo
1 sibling, 0 replies; 3+ messages in thread
From: Stefano Tondo @ 2026-11-19 20:48 UTC (permalink / raw)
To: bitbake-devel
Cc: Joshua Watt, Alexander Kanavin, Stefan Herbrechtsmeier,
Paul Barker, Mathieu Dubois-Briand, Peter Marko,
stefano.tondo.ext
From: Stefano Tondo <stefano.tondo.ext@siemens.com>
Fetchers of package registries know which package a URL downloads:
its name and its version are part of the URL. SBOM generation needs
this data, but it can only get it today by parsing fetcher internals.
Add FetchMethod.package_identity(). It returns a PackageIdentity
dataclass with the name, the version and the Package URL (purl) of
the downloaded package, or None when the URL does not name a package.
It uses only the URL and the datastore, so it works before the fetch
and without network access. Add bb.fetch.package_url() to build the
purl with the correct encoding.
Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
lib/bb/fetch/__init__.py | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)
diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index ce2bd62ad..41ce55921 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -20,6 +20,7 @@ if 'git' not in urllib.parse.uses_netloc:
urllib.parse.uses_netloc.append('git')
import operator
import collections
+import dataclasses
import subprocess
import pickle
import errno
@@ -1765,6 +1766,43 @@ class FetchMethod(object):
"""
return []
+ def package_identity(self, ud, d):
+ """
+ Return the identity of the package that the URL downloads, for
+ fetchers of package registries that know it, as a PackageIdentity.
+ Return None when the URL does not name a package.
+
+ The result depends only on the URL and the datastore. It is
+ available as soon as the FetchData exists, before the fetch and
+ without network access. It is meant for SBOM generation.
+ """
+ return None
+
+
+@dataclasses.dataclass(frozen=True)
+class PackageIdentity:
+ """
+ The identity of a package that a fetcher downloads from a package
+ registry: its name, its version and its Package URL (purl).
+ """
+ name: str
+ version: str
+ purl: str
+
+
+def package_url(purl_type, name, version, qualifiers=None):
+ """
+ Return a Package URL (https://github.com/package-url/purl-spec) for a
+ package. name can contain "/": the segments before the last one are
+ the namespace. Each segment, the version and the qualifier values are
+ percent-encoded. qualifiers is a dict, written sorted by key.
+ """
+ quote = lambda s: urllib.parse.quote(s, safe="")
+ purl = "pkg:%s/%s@%s" % (purl_type, "/".join(quote(s) for s in name.split("/")), quote(version))
+ if qualifiers:
+ purl += "?" + "&".join("%s=%s" % (k, quote(v)) for k, v in sorted(qualifiers.items()))
+ return purl
+
class DummyUnpackTracer(object):
"""
--
2.56.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* [PATCH v2 2/2] fetch: Return the package identity of crate, gomod and npm URLs
2026-11-19 20:48 [PATCH v2 0/2] fetch: package identity of crate, gomod and npm URLs Stefano Tondo
2026-11-19 20:48 ` [PATCH v2 1/2] fetch: Add package_identity() to the fetchers Stefano Tondo
@ 2026-11-19 20:48 ` Stefano Tondo
1 sibling, 0 replies; 3+ messages in thread
From: Stefano Tondo @ 2026-11-19 20:48 UTC (permalink / raw)
To: bitbake-devel
Cc: Joshua Watt, Alexander Kanavin, Stefan Herbrechtsmeier,
Paul Barker, Mathieu Dubois-Briand, Peter Marko,
stefano.tondo.ext
From: Stefano Tondo <stefano.tondo.ext@siemens.com>
Implement package_identity() for the crate, gomod, gomodgit and npm
fetchers. Packages from a registry other than the default one get a
repository_url qualifier in the purl. A gomod URL with mod=1 downloads
only the go.mod file, which is not built into anything, so it has no
identity.
Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
lib/bb/fetch/crate.py | 7 +++++
lib/bb/fetch/gomod.py | 16 +++++++++++
lib/bb/fetch/npm.py | 7 +++++
lib/bb/tests/fetch.py | 64 +++++++++++++++++++++++++++++++++++++++++++
4 files changed, 94 insertions(+)
diff --git a/lib/bb/fetch/crate.py b/lib/bb/fetch/crate.py
index 16ea02d8a..bf38fa45b 100644
--- a/lib/bb/fetch/crate.py
+++ b/lib/bb/fetch/crate.py
@@ -103,11 +103,18 @@ class Crate(Wget):
ud.versionsurl = "%s://%s/%s/versions" % (proto, host, name)
ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version)
+ ud.crate = (name, version, None if host == 'crates.io' else "%s://%s" % (proto, host))
if 'name' not in ud.parm:
ud.parm['name'] = '%s-%s' % (name, version)
logger.debug2("Fetching %s to %s" % (ud.url, ud.parm['downloadfilename']))
+ def package_identity(self, ud, d):
+ name, version, registry = ud.crate
+ qualifiers = {"repository_url": registry} if registry else None
+ purl = bb.fetch.package_url("cargo", name, version, qualifiers)
+ return bb.fetch.PackageIdentity(name, version, purl)
+
def unpack(self, ud, rootdir, d):
"""
Uses the crate to build the necessary paths for cargo to utilize it
diff --git a/lib/bb/fetch/gomod.py b/lib/bb/fetch/gomod.py
index a62d643fb..1950429d8 100644
--- a/lib/bb/fetch/gomod.py
+++ b/lib/bb/fetch/gomod.py
@@ -80,6 +80,13 @@ def escape(path):
return re.sub(r'([A-Z])', lambda m: '!' + m.group(1).lower(), path)
+def go_package_identity(ud):
+ module = ud.parm['module']
+ version = ud.parm['version']
+ return bb.fetch.PackageIdentity(module, version,
+ bb.fetch.package_url("golang", module, version))
+
+
class GoMod(Wget):
"""Class to fetch Go modules from a Go module proxy via wget"""
@@ -127,6 +134,12 @@ class GoMod(Wget):
super().urldata_init(ud, d)
+ def package_identity(self, ud, d):
+ # A go.mod file alone is not built into anything
+ if ud.parm.get('mod', '0') == '1':
+ return None
+ return go_package_identity(ud)
+
def unpack(self, ud, rootdir, d):
"""Unpack the module in the module cache."""
@@ -218,6 +231,9 @@ class GoModGit(Git):
super().urldata_init(ud, d)
+ def package_identity(self, ud, d):
+ return go_package_identity(ud)
+
def unpack(self, ud, rootdir, d):
"""Unpack the module in the module cache."""
diff --git a/lib/bb/fetch/npm.py b/lib/bb/fetch/npm.py
index c8995ba6b..be01591e6 100644
--- a/lib/bb/fetch/npm.py
+++ b/lib/bb/fetch/npm.py
@@ -225,6 +225,13 @@ class Npm(FetchMethod):
ud.needdonestamp = False
ud.resolvefile = self.localpath(ud, d) + ".resolved"
+ def package_identity(self, ud, d):
+ qualifiers = None
+ if ud.registry.rstrip("/") != "https://registry.npmjs.org":
+ qualifiers = {"repository_url": ud.registry}
+ purl = bb.fetch.package_url("npm", ud.package, ud.version, qualifiers)
+ return bb.fetch.PackageIdentity(ud.package, ud.version, purl)
+
def _resolve_proxy_url(self, ud, d):
"""Resolve the tarball URL from the registry and cache it without any checksum.
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 61dc06660..104ee1f68 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -4731,3 +4731,67 @@ class GitUnpackUpdateTest(FetcherTest):
with open(os.path.join(unpack_path, 'testfile.txt')) as f:
self.assertEqual(f.read(), 'updated content\n')
+
+
+class PackageIdentityTest(FetcherTest):
+ """package_identity() works from the URL alone, without network access."""
+
+ def identity(self, url):
+ fetcher = bb.fetch.Fetch([url], self.d)
+ ud = fetcher.ud[url]
+ return ud.method.package_identity(ud, self.d)
+
+ def test_crate(self):
+ self.assertEqual(self.identity("crate://crates.io/glob/0.2.11"), bb.fetch.PackageIdentity(
+ name="glob", version="0.2.11",
+ purl="pkg:cargo/glob@0.2.11"))
+
+ def test_crate_alternate_registry(self):
+ self.assertEqual(self.identity("crate://crates.example.com/glob/0.2.11"), bb.fetch.PackageIdentity(
+ name="glob", version="0.2.11",
+ purl="pkg:cargo/glob@0.2.11?repository_url=https%3A%2F%2Fcrates.example.com"))
+
+ def test_gomod(self):
+ url = ('gomod://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;'
+ 'sha256sum=9bb69aea32f1d59711701f9562d66432c9c0374205e5009d1d1a62f03fb4fdad')
+ self.assertEqual(self.identity(url), bb.fetch.PackageIdentity(
+ name="github.com/Azure/azure-sdk-for-go/sdk/storage/azblob", version="v1.0.0",
+ purl="pkg:golang/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0"))
+
+ def test_gomod_go_mod_only(self):
+ # A go.mod file alone is not built into anything
+ url = ('gomod://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;mod=1;'
+ 'sha256sum=7873b8544842329b4f385a3aa6cf82cc2bc8defb41a04fa5291c35fd5900e873')
+ self.assertIsNone(self.identity(url))
+
+ def test_gomodgit(self):
+ url = ('gomodgit://golang.org/x/net;version=v0.9.0;repo=go.googlesource.com/net;'
+ 'srcrev=694cff8668bac64e0864b552bffc280cd27f21b1')
+ self.assertEqual(self.identity(url), bb.fetch.PackageIdentity(
+ name="golang.org/x/net", version="v0.9.0",
+ purl="pkg:golang/golang.org/x/net@v0.9.0"))
+
+ def test_npm(self):
+ url = ('npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223')
+ self.assertEqual(self.identity(url), bb.fetch.PackageIdentity(
+ name="@savoirfairelinux/node-server-example", version="1.0.0",
+ purl="pkg:npm/%40savoirfairelinux/node-server-example@1.0.0"))
+
+ def test_npm_alternate_registry(self):
+ url = ('npm://npm.example.com/registry;package=foo;version=1.2.3'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223')
+ self.assertEqual(self.identity(url), bb.fetch.PackageIdentity(
+ name="foo", version="1.2.3",
+ purl="pkg:npm/foo@1.2.3?repository_url=https%3A%2F%2Fnpm.example.com%2Fregistry"))
+
+ def test_git_has_none(self):
+ url = ("git://git.yoctoproject.org/git-submodule-test;branch=master;protocol=https;"
+ "rev=f280847494763cdcf71197557a81ba7d8a6bce42")
+ self.assertIsNone(self.identity(url))
+
+ def test_package_url_encoding(self):
+ self.assertEqual(bb.fetch.package_url("golang", "example.com/m", "v1.0.0+incompatible"),
+ "pkg:golang/example.com/m@v1.0.0%2Bincompatible")
--
2.56.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-19 20:49 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19 20:48 [PATCH v2 0/2] fetch: package identity of crate, gomod and npm URLs Stefano Tondo
2026-11-19 20:48 ` [PATCH v2 1/2] fetch: Add package_identity() to the fetchers Stefano Tondo
2026-11-19 20:48 ` [PATCH v2 2/2] fetch: Return the package identity of crate, gomod and npm URLs Stefano Tondo
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.