From: Stefano Tondo <stondo@gmail.com>
To: openembedded-core@lists.openembedded.org
Cc: Alexander Kanavin <alex.kanavin@gmail.com>,
Joshua Watt <JPEWhacker@gmail.com>,
Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>,
Peter Marko <peter.marko@siemens.com>,
stefano.tondo.ext@siemens.com
Subject: [PATCH 06/10] lib/oe/npm: Add SRC_URI entries for the dependencies of a shrinkwrap file
Date: Fri, 20 Nov 2026 14:53:37 +0100 [thread overview]
Message-ID: <20261120135342.2628228-7-stondo@gmail.com> (raw)
In-Reply-To: <20261120135342.2628228-1-stondo@gmail.com>
From: Stefano Tondo <stefano.tondo.ext@siemens.com>
The npmsw fetcher reads npm-shrinkwrap.json to find the dependencies of
an npm package. So SRC_URI does not show these dependencies, and the
fetcher holds logic that belongs in the metadata.
Add shrinkwrap_src_uri(), which gives one SRC_URI entry for each
dependency of a shrinkwrap file:
- A package from a registry gets an npm:// entry with its name,
version and checksum.
- Another tarball gets its URL with the checksum.
- A git dependency gets a git:// entry with its revision.
Each entry unpacks to the location of the dependency in the source
tree, where npm.bbclass expects it. Add src_uri_differences(), which
compares these entries with SRC_URI.
Add tests for both functions.
Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
meta/lib/oe/npm.py | 99 ++++++++++++++++++
meta/lib/oeqa/selftest/cases/oelib/npm.py | 121 ++++++++++++++++++++++
2 files changed, 220 insertions(+)
create mode 100644 meta/lib/oe/npm.py
create mode 100644 meta/lib/oeqa/selftest/cases/oelib/npm.py
diff --git a/meta/lib/oe/npm.py b/meta/lib/oe/npm.py
new file mode 100644
index 0000000000..19f547839e
--- /dev/null
+++ b/meta/lib/oe/npm.py
@@ -0,0 +1,99 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+"""Explicit SRC_URI entries for the dependencies of an npm shrinkwrap file."""
+
+import os
+import re
+
+import bb.fetch
+import bb.utils
+from bb.fetch.npm import npm_integrity, npm_localfile
+from bb.fetch.npmsw import foreach_dependencies
+
+GIT_RESOLVED_RE = re.compile(r"^git\+(?P<protocol>[a-z]+)://(?P<url>[^#]+)#(?P<rev>[0-9a-f]+)$")
+
+
+def shrinkwrap_src_uri(shrinkwrap, destdir, dev=False):
+ """Return one SRC_URI entry for each dependency of a shrinkwrap file.
+
+ shrinkwrap is the content of npm-shrinkwrap.json (lockfileVersion 2 or
+ later). destdir is the directory of the main package, relative to
+ UNPACKDIR. Each dependency unpacks to its location in destdir (for example
+ destdir/node_modules/foo), where npm.bbclass expects it. dev also adds the
+ devDependencies. The entries are sorted by location.
+
+ A package from a registry gets an npm:// entry, so that the fetcher knows
+ its name and version. Other tarballs get a plain URL and git dependencies
+ a git:// URL. Each entry has the checksum or revision of the shrinkwrap
+ file.
+ """
+ entries = {}
+
+ def _add(name, data, location):
+ package = data.get("name", name)
+ version = data.get("version")
+ resolved = data.get("resolved")
+ integrity = data.get("integrity")
+ destsuffix = os.path.join(destdir, location)
+
+ if not resolved:
+ # npm leaves out "resolved" for some duplicate packages. npm.bbclass
+ # does not need these, as npmsw already skipped them.
+ if version and integrity:
+ return
+ raise ValueError("%s: no 'resolved' field" % location)
+
+ if data.get("link") or resolved.startswith("file:"):
+ raise ValueError("%s: local dependencies are not supported" % location)
+
+ match = GIT_RESOLVED_RE.match(resolved)
+ if match:
+ entries[location] = "git://%s;protocol=%s;rev=%s;nobranch=1;destsuffix=%s" % (
+ match["url"], match["protocol"], match["rev"], destsuffix)
+ return
+
+ if not integrity or not resolved.startswith(("https://", "http://")):
+ raise ValueError("%s: unsupported dependency %s" % (location, resolved))
+
+ checksum_name, checksum = npm_integrity(integrity)
+ basename = package.split("/")[-1]
+ suffix = "/%s/-/%s-%s.tgz" % (package, basename, version)
+ if resolved.startswith("https://") and resolved.endswith(suffix) and bb.utils.is_semver(version):
+ registry = resolved[len("https://"):-len(suffix)]
+ entries[location] = "npm://%s/;package=%s;version=%s;%s=%s;destsuffix=%s" % (
+ registry, package, version, checksum_name, checksum, destsuffix)
+ else:
+ entries[location] = "%s;downloadfilename=%s;%s=%s;subdir=%s;striplevel=1" % (
+ resolved, npm_localfile(os.path.basename(resolved)), checksum_name, checksum, destsuffix)
+
+ foreach_dependencies(shrinkwrap, _add, dev)
+ return [entries[location] for location in sorted(entries)]
+
+
+def src_uri_differences(expected, src_uri):
+ """Compare the expected entries with SRC_URI.
+
+ Returns (missing, extra): the expected entries that SRC_URI does not have,
+ and the SRC_URI entries that unpack into a node_modules directory but are
+ not expected. Entries are compared after parsing, so the order of the URL
+ parameters does not matter.
+ """
+ def _key(url):
+ scheme, host, path, user, password, parm = bb.fetch.decodeurl(url)
+ return (scheme, host, path, user, password, tuple(sorted(parm.items())))
+
+ def _unpacks_to_node_modules(url):
+ parm = bb.fetch.decodeurl(url)[5]
+ dest = parm.get("destsuffix") or parm.get("subdir") or ""
+ return "node_modules/" in dest + "/"
+
+ expected_keys = {_key(url): url for url in expected}
+ actual_keys = {_key(url): url for url in src_uri}
+ missing = [url for key, url in expected_keys.items() if key not in actual_keys]
+ extra = [url for key, url in actual_keys.items()
+ if key not in expected_keys and _unpacks_to_node_modules(url)]
+ return missing, extra
diff --git a/meta/lib/oeqa/selftest/cases/oelib/npm.py b/meta/lib/oeqa/selftest/cases/oelib/npm.py
new file mode 100644
index 0000000000..df4f6aff5b
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/oelib/npm.py
@@ -0,0 +1,121 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import base64
+from unittest.case import TestCase
+
+import oe.npm
+
+CHECKSUM = bytes(range(64)).hex()
+INTEGRITY = "sha512-" + base64.b64encode(bytes(range(64))).decode()
+
+
+def shrinkwrap(packages):
+ """Return a shrinkwrap file with the given dependencies"""
+ return {"lockfileVersion": 3, "packages": {"": {"name": "main"}, **packages}}
+
+
+def registry(package, version, host="registry.npmjs.org"):
+ """Return a dependency from a registry"""
+ basename = package.split("/")[-1]
+ return {
+ "version": version,
+ "resolved": "https://%s/%s/-/%s-%s.tgz" % (host, package, basename, version),
+ "integrity": INTEGRITY,
+ }
+
+
+class TestShrinkwrapSrcUri(TestCase):
+ def test_registry(self):
+ entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({
+ "node_modules/send": registry("send", "0.19.0"),
+ "node_modules/send/node_modules/ms": registry("ms", "2.1.3"),
+ "node_modules/@scope/pkg": registry("@scope/pkg", "1.0.0", "npm.example.com/path"),
+ }), "npm")
+ self.assertEqual(entries, [
+ "npm://npm.example.com/path/;package=@scope/pkg;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/@scope/pkg" % CHECKSUM,
+ "npm://registry.npmjs.org/;package=send;version=0.19.0;sha512sum=%s;destsuffix=npm/node_modules/send" % CHECKSUM,
+ "npm://registry.npmjs.org/;package=ms;version=2.1.3;sha512sum=%s;destsuffix=npm/node_modules/send/node_modules/ms" % CHECKSUM,
+ ])
+
+ def test_alias(self):
+ # An alias installs a package under another name: the "name" field
+ # gives the real package
+ dep = registry("real", "1.0.0")
+ dep["name"] = "real"
+ entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({"node_modules/alias": dep}), "npm")
+ self.assertEqual(entries, [
+ "npm://registry.npmjs.org/;package=real;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/alias" % CHECKSUM,
+ ])
+
+ def test_tarball(self):
+ entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({
+ "node_modules/foo": {
+ "version": "1.0.0",
+ "resolved": "https://example.com/downloads/foo.tgz",
+ "integrity": INTEGRITY,
+ },
+ }), "npm")
+ self.assertEqual(entries, [
+ "https://example.com/downloads/foo.tgz;downloadfilename=npm2/foo.tgz;sha512sum=%s;subdir=npm/node_modules/foo;striplevel=1" % CHECKSUM,
+ ])
+
+ def test_git(self):
+ entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({
+ "node_modules/foo": {
+ "version": "1.0.0",
+ "resolved": "git+ssh://git@github.com/example/foo.git#0123456789abcdef0123456789abcdef01234567",
+ },
+ }), "npm")
+ self.assertEqual(entries, [
+ "git://git@github.com/example/foo.git;protocol=ssh;rev=0123456789abcdef0123456789abcdef01234567;nobranch=1;destsuffix=npm/node_modules/foo",
+ ])
+
+ def test_skipped(self):
+ dev = registry("dev", "1.0.0")
+ dev["dev"] = True
+ bundled = registry("bundled", "1.0.0")
+ bundled["inBundle"] = True
+ no_resolved = registry("dup", "1.0.0")
+ del no_resolved["resolved"]
+ packages = {
+ "node_modules/dev": dev,
+ "node_modules/bundled": bundled,
+ "node_modules/dup": no_resolved,
+ }
+ self.assertEqual(oe.npm.shrinkwrap_src_uri(shrinkwrap(packages), "npm"), [])
+ self.assertEqual(len(oe.npm.shrinkwrap_src_uri(shrinkwrap(packages), "npm", dev=True)), 1)
+
+ def test_unsupported(self):
+ for dep in ({"version": "1.0.0", "resolved": "file:../foo"},
+ {"link": True, "resolved": "../foo"},
+ {"version": "1.0.0"}):
+ with self.assertRaises(ValueError):
+ oe.npm.shrinkwrap_src_uri(shrinkwrap({"node_modules/foo": dep}), "npm")
+
+
+class TestSrcUriDifferences(TestCase):
+ EXPECTED = [
+ "npm://registry.npmjs.org/;package=foo;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/foo" % CHECKSUM,
+ "npm://registry.npmjs.org/;package=bar;version=2.0.0;sha512sum=%s;destsuffix=npm/node_modules/bar" % CHECKSUM,
+ ]
+
+ def test_match(self):
+ # The order of the entries and of the URL parameters does not matter.
+ # Entries that do not unpack into node_modules are not extra.
+ src_uri = [
+ "npm://registry.npmjs.org/;package=main;version=1.0.0;sha512sum=%s" % CHECKSUM,
+ "file://npm-shrinkwrap.json;subdir=npm",
+ self.EXPECTED[1],
+ "npm://registry.npmjs.org/;destsuffix=npm/node_modules/foo;sha512sum=%s;version=1.0.0;package=foo" % CHECKSUM,
+ ]
+ self.assertEqual(oe.npm.src_uri_differences(self.EXPECTED, src_uri), ([], []))
+
+ def test_missing_and_extra(self):
+ old_bar = "npm://registry.npmjs.org/;package=bar;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/bar" % CHECKSUM
+ missing, extra = oe.npm.src_uri_differences(self.EXPECTED, [self.EXPECTED[0], old_bar])
+ self.assertEqual(missing, [self.EXPECTED[1]])
+ self.assertEqual(extra, [old_bar])
--
2.56.1
next prev parent reply other threads:[~2026-11-20 13:53 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
2026-11-20 13:53 ` [PATCH 01/10] recipetool: create_npm: Give the fallback licenses as a string Stefano Tondo
2026-11-20 13:53 ` [PATCH 02/10] recipetool: create: Get the checksum of npm packages from the registry Stefano Tondo
2026-11-20 13:53 ` [PATCH 03/10] recipetool: create_npm: Give the package fallback licenses as strings Stefano Tondo
2026-11-20 13:53 ` [PATCH 04/10] recipetool: create_npm: Remove the "@" of a scope from generated names Stefano Tondo
2026-11-20 14:04 ` Patchtest results for " patchtest
2026-11-20 13:53 ` [PATCH 05/10] oeqa/selftest: recipetool: Expect the npm source in UNPACKDIR Stefano Tondo
2026-11-20 13:53 ` Stefano Tondo [this message]
2026-11-20 13:53 ` [PATCH 07/10] npm-update-dependencies: Add a class to list the npm dependencies Stefano Tondo
2026-11-20 13:53 ` [PATCH 08/10] npm: Check that SRC_URI lists all dependencies of the shrinkwrap file Stefano Tondo
2026-11-20 13:53 ` [PATCH 09/10] recipetool: create: Clean the recipe after its tasks Stefano Tondo
2026-11-20 13:53 ` [PATCH 10/10] recipetool: create_npm: List the npm dependencies in SRC_URI Stefano Tondo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120135342.2628228-7-stondo@gmail.com \
--to=stondo@gmail.com \
--cc=JPEWhacker@gmail.com \
--cc=alex.kanavin@gmail.com \
--cc=openembedded-core@lists.openembedded.org \
--cc=peter.marko@siemens.com \
--cc=stefan.herbrechtsmeier@weidmueller.com \
--cc=stefano.tondo.ext@siemens.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.