From: Stefano Tondo <stondo@gmail.com>
To: openembedded-core@lists.openembedded.org
Cc: Alexander Kanavin <alex.kanavin@gmail.com>,
Joshua Watt <JPEWhacker@gmail.com>,
Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>,
Peter Marko <peter.marko@siemens.com>,
stefano.tondo.ext@siemens.com
Subject: [PATCH 06/10] lib/oe/npm: Add SRC_URI entries for the dependencies of a shrinkwrap file
Date: Fri, 20 Nov 2026 14:53:37 +0100 [thread overview]
Message-ID: <20261120135342.2628228-7-stondo@gmail.com> (raw)
In-Reply-To: <20261120135342.2628228-1-stondo@gmail.com>
From: Stefano Tondo <stefano.tondo.ext@siemens.com>
The npmsw fetcher reads npm-shrinkwrap.json to find the dependencies of
an npm package. So SRC_URI does not show these dependencies, and the
fetcher holds logic that belongs in the metadata.
Add shrinkwrap_src_uri(), which gives one SRC_URI entry for each
dependency of a shrinkwrap file:
- A package from a registry gets an npm:// entry with its name,
version and checksum.
- Another tarball gets its URL with the checksum.
- A git dependency gets a git:// entry with its revision.
Each entry unpacks to the location of the dependency in the source
tree, where npm.bbclass expects it. Add src_uri_differences(), which
compares these entries with SRC_URI.
Add tests for both functions.
Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
meta/lib/oe/npm.py | 99 ++++++++++++++++++
meta/lib/oeqa/selftest/cases/oelib/npm.py | 121 ++++++++++++++++++++++
2 files changed, 220 insertions(+)
create mode 100644 meta/lib/oe/npm.py
create mode 100644 meta/lib/oeqa/selftest/cases/oelib/npm.py
diff --git a/meta/lib/oe/npm.py b/meta/lib/oe/npm.py
new file mode 100644
index 0000000000..19f547839e
--- /dev/null
+++ b/meta/lib/oe/npm.py
@@ -0,0 +1,99 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+"""Explicit SRC_URI entries for the dependencies of an npm shrinkwrap file."""
+
+import os
+import re
+
+import bb.fetch
+import bb.utils
+from bb.fetch.npm import npm_integrity, npm_localfile
+from bb.fetch.npmsw import foreach_dependencies
+
+GIT_RESOLVED_RE = re.compile(r"^git\+(?P<protocol>[a-z]+)://(?P<url>[^#]+)#(?P<rev>[0-9a-f]+)$")
+
+
+def shrinkwrap_src_uri(shrinkwrap, destdir, dev=False):
+ """Return one SRC_URI entry for each dependency of a shrinkwrap file.
+
+ shrinkwrap is the content of npm-shrinkwrap.json (lockfileVersion 2 or
+ later). destdir is the directory of the main package, relative to
+ UNPACKDIR. Each dependency unpacks to its location in destdir (for example
+ destdir/node_modules/foo), where npm.bbclass expects it. dev also adds the
+ devDependencies. The entries are sorted by location.
+
+ A package from a registry gets an npm:// entry, so that the fetcher knows
+ its name and version. Other tarballs get a plain URL and git dependencies
+ a git:// URL. Each entry has the checksum or revision of the shrinkwrap
+ file.
+ """
+ entries = {}
+
+ def _add(name, data, location):
+ package = data.get("name", name)
+ version = data.get("version")
+ resolved = data.get("resolved")
+ integrity = data.get("integrity")
+ destsuffix = os.path.join(destdir, location)
+
+ if not resolved:
+ # npm leaves out "resolved" for some duplicate packages. npm.bbclass
+ # does not need these, as npmsw already skipped them.
+ if version and integrity:
+ return
+ raise ValueError("%s: no 'resolved' field" % location)
+
+ if data.get("link") or resolved.startswith("file:"):
+ raise ValueError("%s: local dependencies are not supported" % location)
+
+ match = GIT_RESOLVED_RE.match(resolved)
+ if match:
+ entries[location] = "git://%s;protocol=%s;rev=%s;nobranch=1;destsuffix=%s" % (
+ match["url"], match["protocol"], match["rev"], destsuffix)
+ return
+
+ if not integrity or not resolved.startswith(("https://", "http://")):
+ raise ValueError("%s: unsupported dependency %s" % (location, resolved))
+
+ checksum_name, checksum = npm_integrity(integrity)
+ basename = package.split("/")[-1]
+ suffix = "/%s/-/%s-%s.tgz" % (package, basename, version)
+ if resolved.startswith("https://") and resolved.endswith(suffix) and bb.utils.is_semver(version):
+ registry = resolved[len("https://"):-len(suffix)]
+ entries[location] = "npm://%s/;package=%s;version=%s;%s=%s;destsuffix=%s" % (
+ registry, package, version, checksum_name, checksum, destsuffix)
+ else:
+ entries[location] = "%s;downloadfilename=%s;%s=%s;subdir=%s;striplevel=1" % (
+ resolved, npm_localfile(os.path.basename(resolved)), checksum_name, checksum, destsuffix)
+
+ foreach_dependencies(shrinkwrap, _add, dev)
+ return [entries[location] for location in sorted(entries)]
+
+
+def src_uri_differences(expected, src_uri):
+ """Compare the expected entries with SRC_URI.
+
+ Returns (missing, extra): the expected entries that SRC_URI does not have,
+ and the SRC_URI entries that unpack into a node_modules directory but are
+ not expected. Entries are compared after parsing, so the order of the URL
+ parameters does not matter.
+ """
+ def _key(url):
+ scheme, host, path, user, password, parm = bb.fetch.decodeurl(url)
+ return (scheme, host, path, user, password, tuple(sorted(parm.items())))
+
+ def _unpacks_to_node_modules(url):
+ parm = bb.fetch.decodeurl(url)[5]
+ dest = parm.get("destsuffix") or parm.get("subdir") or ""
+ return "node_modules/" in dest + "/"
+
+ expected_keys = {_key(url): url for url in expected}
+ actual_keys = {_key(url): url for url in src_uri}
+ missing = [url for key, url in expected_keys.items() if key not in actual_keys]
+ extra = [url for key, url in actual_keys.items()
+ if key not in expected_keys and _unpacks_to_node_modules(url)]
+ return missing, extra
diff --git a/meta/lib/oeqa/selftest/cases/oelib/npm.py b/meta/lib/oeqa/selftest/cases/oelib/npm.py
new file mode 100644
index 0000000000..df4f6aff5b
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/oelib/npm.py
@@ -0,0 +1,121 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import base64
+from unittest.case import TestCase
+
+import oe.npm
+
+CHECKSUM = bytes(range(64)).hex()
+INTEGRITY = "sha512-" + base64.b64encode(bytes(range(64))).decode()
+
+
+def shrinkwrap(packages):
+ """Return a shrinkwrap file with the given dependencies"""
+ return {"lockfileVersion": 3, "packages": {"": {"name": "main"}, **packages}}
+
+
+def registry(package, version, host="registry.npmjs.org"):
+ """Return a dependency from a registry"""
+ basename = package.split("/")[-1]
+ return {
+ "version": version,
+ "resolved": "https://%s/%s/-/%s-%s.tgz" % (host, package, basename, version),
+ "integrity": INTEGRITY,
+ }
+
+
+class TestShrinkwrapSrcUri(TestCase):
+ def test_registry(self):
+ entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({
+ "node_modules/send": registry("send", "0.19.0"),
+ "node_modules/send/node_modules/ms": registry("ms", "2.1.3"),
+ "node_modules/@scope/pkg": registry("@scope/pkg", "1.0.0", "npm.example.com/path"),
+ }), "npm")
+ self.assertEqual(entries, [
+ "npm://npm.example.com/path/;package=@scope/pkg;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/@scope/pkg" % CHECKSUM,
+ "npm://registry.npmjs.org/;package=send;version=0.19.0;sha512sum=%s;destsuffix=npm/node_modules/send" % CHECKSUM,
+ "npm://registry.npmjs.org/;package=ms;version=2.1.3;sha512sum=%s;destsuffix=npm/node_modules/send/node_modules/ms" % CHECKSUM,
+ ])
+
+ def test_alias(self):
+ # An alias installs a package under another name: the "name" field
+ # gives the real package
+ dep = registry("real", "1.0.0")
+ dep["name"] = "real"
+ entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({"node_modules/alias": dep}), "npm")
+ self.assertEqual(entries, [
+ "npm://registry.npmjs.org/;package=real;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/alias" % CHECKSUM,
+ ])
+
+ def test_tarball(self):
+ entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({
+ "node_modules/foo": {
+ "version": "1.0.0",
+ "resolved": "https://example.com/downloads/foo.tgz",
+ "integrity": INTEGRITY,
+ },
+ }), "npm")
+ self.assertEqual(entries, [
+ "https://example.com/downloads/foo.tgz;downloadfilename=npm2/foo.tgz;sha512sum=%s;subdir=npm/node_modules/foo;striplevel=1" % CHECKSUM,
+ ])
+
+ def test_git(self):
+ entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({
+ "node_modules/foo": {
+ "version": "1.0.0",
+ "resolved": "git+ssh://git@github.com/example/foo.git#0123456789abcdef0123456789abcdef01234567",
+ },
+ }), "npm")
+ self.assertEqual(entries, [
+ "git://git@github.com/example/foo.git;protocol=ssh;rev=0123456789abcdef0123456789abcdef01234567;nobranch=1;destsuffix=npm/node_modules/foo",
+ ])
+
+ def test_skipped(self):
+ dev = registry("dev", "1.0.0")
+ dev["dev"] = True
+ bundled = registry("bundled", "1.0.0")
+ bundled["inBundle"] = True
+ no_resolved = registry("dup", "1.0.0")
+ del no_resolved["resolved"]
+ packages = {
+ "node_modules/dev": dev,
+ "node_modules/bundled": bundled,
+ "node_modules/dup": no_resolved,
+ }
+ self.assertEqual(oe.npm.shrinkwrap_src_uri(shrinkwrap(packages), "npm"), [])
+ self.assertEqual(len(oe.npm.shrinkwrap_src_uri(shrinkwrap(packages), "npm", dev=True)), 1)
+
+ def test_unsupported(self):
+ for dep in ({"version": "1.0.0", "resolved": "file:../foo"},
+ {"link": True, "resolved": "../foo"},
+ {"version": "1.0.0"}):
+ with self.assertRaises(ValueError):
+ oe.npm.shrinkwrap_src_uri(shrinkwrap({"node_modules/foo": dep}), "npm")
+
+
+class TestSrcUriDifferences(TestCase):
+ EXPECTED = [
+ "npm://registry.npmjs.org/;package=foo;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/foo" % CHECKSUM,
+ "npm://registry.npmjs.org/;package=bar;version=2.0.0;sha512sum=%s;destsuffix=npm/node_modules/bar" % CHECKSUM,
+ ]
+
+ def test_match(self):
+ # The order of the entries and of the URL parameters does not matter.
+ # Entries that do not unpack into node_modules are not extra.
+ src_uri = [
+ "npm://registry.npmjs.org/;package=main;version=1.0.0;sha512sum=%s" % CHECKSUM,
+ "file://npm-shrinkwrap.json;subdir=npm",
+ self.EXPECTED[1],
+ "npm://registry.npmjs.org/;destsuffix=npm/node_modules/foo;sha512sum=%s;version=1.0.0;package=foo" % CHECKSUM,
+ ]
+ self.assertEqual(oe.npm.src_uri_differences(self.EXPECTED, src_uri), ([], []))
+
+ def test_missing_and_extra(self):
+ old_bar = "npm://registry.npmjs.org/;package=bar;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/bar" % CHECKSUM
+ missing, extra = oe.npm.src_uri_differences(self.EXPECTED, [self.EXPECTED[0], old_bar])
+ self.assertEqual(missing, [self.EXPECTED[1]])
+ self.assertEqual(extra, [old_bar])
--
2.56.1
next prev parent reply other threads:[~2026-11-20 13:53 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
2026-11-20 13:53 ` [PATCH 01/10] recipetool: create_npm: Give the fallback licenses as a string Stefano Tondo
2026-11-20 13:53 ` [PATCH 02/10] recipetool: create: Get the checksum of npm packages from the registry Stefano Tondo
2026-11-20 13:53 ` [PATCH 03/10] recipetool: create_npm: Give the package fallback licenses as strings Stefano Tondo
2026-11-20 13:53 ` [PATCH 04/10] recipetool: create_npm: Remove the "@" of a scope from generated names Stefano Tondo
2026-11-20 14:04 ` Patchtest results for " patchtest
2026-11-20 13:53 ` [PATCH 05/10] oeqa/selftest: recipetool: Expect the npm source in UNPACKDIR Stefano Tondo
2026-11-20 13:53 ` Stefano Tondo [this message]
2026-11-20 13:53 ` [PATCH 07/10] npm-update-dependencies: Add a class to list the npm dependencies Stefano Tondo
2026-11-20 13:53 ` [PATCH 08/10] npm: Check that SRC_URI lists all dependencies of the shrinkwrap file Stefano Tondo
2026-11-20 13:53 ` [PATCH 09/10] recipetool: create: Clean the recipe after its tasks Stefano Tondo
2026-11-20 13:53 ` [PATCH 10/10] recipetool: create_npm: List the npm dependencies in SRC_URI Stefano Tondo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120135342.2628228-7-stondo@gmail.com \
--to=stondo@gmail.com \
--cc=JPEWhacker@gmail.com \
--cc=alex.kanavin@gmail.com \
--cc=openembedded-core@lists.openembedded.org \
--cc=peter.marko@siemens.com \
--cc=stefan.herbrechtsmeier@weidmueller.com \
--cc=stefano.tondo.ext@siemens.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox