Openembedded Core Discussions
 help / color / mirror / Atom feed
* [PATCH 00/10] npm: list the dependencies in SRC_URI
@ 2026-11-20 13:53 Stefano Tondo
  2026-11-20 13:53 ` [PATCH 01/10] recipetool: create_npm: Give the fallback licenses as a string Stefano Tondo
                   ` (9 more replies)
  0 siblings, 10 replies; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

Hi,

This series lets an npm recipe list its dependencies in SRC_URI. Each
package gets one entry with its checksum. We agreed on this in the RFC
thread [1]. npmsw:// still works, but recipetool no longer uses it.

Today, a recipe that recipetool creates builds without its
dependencies. recipetool writes an npmsw:// entry without destsuffix.
Since bitbake commit 84f102954, npmsw unpacks into UNPACKDIR and not
into ${S}. So npm.bbclass does not find npm-shrinkwrap.json, packs no
dependencies, and gives no error. The package has no node_modules.

1. Patches 1 to 5 make "recipetool create" for npm work again on
   master. It failed on the missing checksum, on license values given
   as lists, and on the "@" of scoped names. The selftest expected an
   old value of S.
2. lib/oe/npm gives the SRC_URI entries for the dependencies of a
   shrinkwrap file, with unit tests.
3. The new class npm-update-dependencies writes these entries to
   ${BPN}-npm-deps.inc, like the cargo and Go update classes.
4. npm.bbclass checks that SRC_URI matches the shrinkwrap file, as
   Richard asked. It fails with a hint to run the update task.
5. recipetool runs do_clean after the tasks that it runs for the new
   recipe. Else devtool leaves stale stamps of do_unpack behind.
6. recipetool creates the recipe with the explicit list.

This series needs the two patches of the bitbake series "fetch/npm: no
npm executable, and an event log fix" [2]. Without "fetch/npm: Build
the tarball URL without the npm executable", the npm fetcher still
calls the npm executable of the host. Without "cooker: Write variables
that are not JSON types to the event log", the recipetool selftest
fails.

Two points for review:

- The check also applies to a recipe without npmsw:// that has
  node_modules in its source tree. I do not know of such a recipe.
  If there is one, an opt-out variable is simple to add.
- This series does not deprecate npmsw://. That can follow if you
  want it.

Tests on master with the two bitbake patches: oe-selftest -r oelib.npm
recipetool.RecipetoolCreateTests.test_recipetool_create_npm
devtool.DevtoolAddTests.test_devtool_add_npm pass. A build of
node-server-example with the explicit list installs all 72
dependencies. With one entry removed, do_configure fails and names it.

[1] https://lore.kernel.org/openembedded-core/20261111115845.1940227-1-stondo@gmail.com/T/
[2] https://lore.kernel.org/bitbake-devel/20261120135323.2627478-1-stondo@gmail.com/

Stefano Tondo (10):
  recipetool: create_npm: Give the fallback licenses as a string
  recipetool: create: Get the checksum of npm packages from the registry
  recipetool: create_npm: Give the package fallback licenses as strings
  recipetool: create_npm: Remove the "@" of a scope from generated names
  oeqa/selftest: recipetool: Expect the npm source in UNPACKDIR
  lib/oe/npm: Add SRC_URI entries for the dependencies of a shrinkwrap
    file
  npm-update-dependencies: Add a class to list the npm dependencies
  npm: Check that SRC_URI lists all dependencies of the shrinkwrap file
  recipetool: create: Clean the recipe after its tasks
  recipetool: create_npm: List the npm dependencies in SRC_URI

 .../npm-update-dependencies.bbclass           |  48 +++++++
 meta/classes-recipe/npm.bbclass               |  24 ++++
 meta/lib/oe/npm.py                            |  99 ++++++++++++++
 meta/lib/oeqa/selftest/cases/oelib/npm.py     | 121 ++++++++++++++++++
 meta/lib/oeqa/selftest/cases/recipetool.py    |  22 +++-
 scripts/lib/recipetool/create.py              |  31 +++++
 scripts/lib/recipetool/create_npm.py          |  62 ++++++---
 7 files changed, 384 insertions(+), 23 deletions(-)
 create mode 100644 meta/classes-recipe/npm-update-dependencies.bbclass
 create mode 100644 meta/lib/oe/npm.py
 create mode 100644 meta/lib/oeqa/selftest/cases/oelib/npm.py

-- 
2.56.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH 01/10] recipetool: create_npm: Give the fallback licenses as a string
  2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
@ 2026-11-20 13:53 ` Stefano Tondo
  2026-11-20 13:53 ` [PATCH 02/10] recipetool: create: Get the checksum of npm packages from the registry Stefano Tondo
                   ` (8 subsequent siblings)
  9 siblings, 0 replies; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

Since 727ea2769c ("license: Remove tidy_licenses()"),
handle_license_vars() reads LICENSE from the extra values as a license
string. The npm handler still gives a set, so recipetool create stops
with "AttributeError: 'set' object has no attribute 'strip'" for each
npm package that has dependencies.

Join the sorted fallback licenses with " AND ". Also remove a line that
sorted a copy of the set and did not use the result.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 scripts/lib/recipetool/create_npm.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 10eca3c63e..5edf647176 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -283,10 +283,9 @@ class NpmRecipeHandler(RecipeHandler):
         (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev)
         licvalues = match_licenses(licfiles, srctree, d)
         split_pkg_licenses(licvalues, packages, lines_after, fallback_licenses)
-        fallback_licenses_flat = set(license for sublist in fallback_licenses.values() for license in sublist)
-        fallback_licesens_flat = sorted(fallback_licenses_flat).sort()
+        fallback_licenses_flat = sorted(set(license for sublist in fallback_licenses.values() for license in sublist))
         extravalues["LIC_FILES_CHKSUM"] = generate_common_licenses_chksums(fallback_licenses_flat, d)
-        extravalues["LICENSE"] = fallback_licenses_flat
+        extravalues["LICENSE"] = " AND ".join(fallback_licenses_flat)
 
         classes.append("npm")
         handled.append("buildsystem")
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 02/10] recipetool: create: Get the checksum of npm packages from the registry
  2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
  2026-11-20 13:53 ` [PATCH 01/10] recipetool: create_npm: Give the fallback licenses as a string Stefano Tondo
@ 2026-11-20 13:53 ` Stefano Tondo
  2026-11-20 13:53 ` [PATCH 03/10] recipetool: create_npm: Give the package fallback licenses as strings Stefano Tondo
                   ` (7 subsequent siblings)
  9 siblings, 0 replies; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

Since bitbake 9e45e00a2 ("fetch/{npm,npmsw}: re-enable fetchers now
that checksums come from SRC_URI"), an npm:// URL needs a checksum.
recipetool create gets an npm:// URL without a checksum, so it fails
before the fetch:

  MissingParameterError: ... Missing checksum for npm package

The usual way, a fetch with BB_STRICT_CHECKSUM = "ignore", does not
work for npm on purpose. So get the checksum from the registry
("dist.integrity") before the fetch, and add it to the URL in the
recipe. For a new recipe, the registry is the only source of this
checksum.

Update the selftest for the checksum in SRC_URI.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/lib/oeqa/selftest/cases/recipetool.py |  4 +++-
 scripts/lib/recipetool/create.py           | 22 ++++++++++++++++++++++
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index 0aaa090ff5..c6aa77399f 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -449,7 +449,9 @@ class RecipetoolCreateTests(RecipetoolBase):
         checkvars['HOMEPAGE'] = 'https://github.com/savoirfairelinux/node-server-example#readme'
         checkvars['LICENSE'] = 'BSD-3-Clause AND ISC AND MIT AND Unknown'
         urls = []
-        urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}')
+        urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}'
+                    ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+                    '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223')
         urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
         checkvars['SRC_URI'] = set(urls)
         checkvars['S'] = '${WORKDIR}/npm'
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 662e7a6232..2df17dd6e4 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -413,6 +413,18 @@ def is_package(url):
         return True
     return False
 
+def get_npm_checksum(registry, package, version):
+    """Get the checksum of an npm package from the registry.
+
+    The npm fetcher needs a checksum before the download. When a new recipe
+    is created, the registry is the only source of this checksum.
+    """
+    import urllib.request
+    from bb.fetch.npm import npm_integrity
+    url = "https://%s/%s/%s" % (registry.strip("/"), package, version)
+    with urllib.request.urlopen(url, timeout=60) as response:
+        return npm_integrity(json.load(response)["dist"]["integrity"])
+
 def create_recipe(args):
     import bb.process
     import tempfile
@@ -523,6 +535,15 @@ def create_recipe(args):
         # Assume 'master' branch if not set
         if scheme in ['git', 'gitsm'] and 'branch' not in params and 'nobranch' not in params:
             params['branch'] = 'master'
+        npm_checksum = {}
+        if scheme == 'npm' and not any(p in params for p in bb.fetch.npm.CHECKSUM_PARMS):
+            try:
+                checksum_name, checksum = get_npm_checksum(network + path, params.get('package', ''), params.get('version', ''))
+            except Exception as e:
+                logger.error('Unable to get the checksum of %s from the registry: %s' % (srcuri, e))
+                sys.exit(1)
+            npm_checksum[checksum_name] = checksum
+            params.update(npm_checksum)
         fetchuri = bb.fetch.encodeurl((scheme, network, path, user, passwd, params))
 
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
@@ -587,6 +608,7 @@ def create_recipe(args):
         # Since we might have a value in srcbranch, we need to
         # recontruct the srcuri to include 'branch' in params.
         scheme, network, path, user, passwd, params = bb.fetch.decodeurl(srcuri)
+        params.update(npm_checksum)
         if scheme in ['git', 'gitsm']:
             params['branch'] = srcbranch or 'master'
 
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 03/10] recipetool: create_npm: Give the package fallback licenses as strings
  2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
  2026-11-20 13:53 ` [PATCH 01/10] recipetool: create_npm: Give the fallback licenses as a string Stefano Tondo
  2026-11-20 13:53 ` [PATCH 02/10] recipetool: create: Get the checksum of npm packages from the registry Stefano Tondo
@ 2026-11-20 13:53 ` Stefano Tondo
  2026-11-20 13:53 ` [PATCH 04/10] recipetool: create_npm: Remove the "@" of a scope from generated names Stefano Tondo
                   ` (6 subsequent siblings)
  9 siblings, 0 replies; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

The npm handler finds the licenses of a dependency in its package.json
when it has no license file. Since 727ea2769c ("license: Remove
tidy_licenses()"), split_pkg_licenses() writes such a fallback value
as it is. The npm handler gives a list, so the recipe gets lines like:

  LICENSE:${PN}-cookie-signature = "['MIT']"

Join the licenses of each package with " AND ", as before that change.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 scripts/lib/recipetool/create_npm.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 5edf647176..437cb91ad4 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -282,7 +282,8 @@ class NpmRecipeHandler(RecipeHandler):
         bb.note("Handling licences ...")
         (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev)
         licvalues = match_licenses(licfiles, srctree, d)
-        split_pkg_licenses(licvalues, packages, lines_after, fallback_licenses)
+        split_pkg_licenses(licvalues, packages, lines_after,
+                           {pkg: " AND ".join(lics) for pkg, lics in fallback_licenses.items()})
         fallback_licenses_flat = sorted(set(license for sublist in fallback_licenses.values() for license in sublist))
         extravalues["LIC_FILES_CHKSUM"] = generate_common_licenses_chksums(fallback_licenses_flat, d)
         extravalues["LICENSE"] = " AND ".join(fallback_licenses_flat)
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 04/10] recipetool: create_npm: Remove the "@" of a scope from generated names
  2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
                   ` (2 preceding siblings ...)
  2026-11-20 13:53 ` [PATCH 03/10] recipetool: create_npm: Give the package fallback licenses as strings Stefano Tondo
@ 2026-11-20 13:53 ` Stefano Tondo
  2026-11-20 14:04   ` Patchtest results for " patchtest
  2026-11-20 13:53 ` [PATCH 05/10] oeqa/selftest: recipetool: Expect the npm source in UNPACKDIR Stefano Tondo
                   ` (5 subsequent siblings)
  9 siblings, 1 reply; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

Since bitbake commit 7c9573cb6, npm_package() keeps the "@" of a
scoped package. This gives unique download file names. recipetool also
uses npm_package() for the recipe name and the package names. So it
creates "@savoirfairelinux-node-server-example_1.0.0.bb", and for a
scoped dependency it writes overrides that bitbake cannot parse:

  LICENSE:${PN}-@-execute = "MIT"

Remove the "@" from these names. The selftest expects the recipe name
without it.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 scripts/lib/recipetool/create_npm.py | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 437cb91ad4..50dd45c7d2 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -23,6 +23,12 @@ logger = logging.getLogger('recipetool')
 
 TINFOIL = None
 
+def npm_recipe_name(package):
+    """Get a recipe or package name from an npm package name"""
+    # npm_package() keeps the "@" of a scope for unique download file names.
+    # An override cannot have it.
+    return npm_package(package).lstrip("@").strip("-")
+
 def tinfoil_init(instance):
     """Initialize tinfoil"""
     global TINFOIL
@@ -147,7 +153,7 @@ class NpmRecipeHandler(RecipeHandler):
         # Handle the dependencies
         def _handle_dependency(name, params, destdir):
             deptree = destdir.split('node_modules/')
-            suffix = "-".join([npm_package(dep) for dep in deptree])
+            suffix = "-".join([npm_recipe_name(dep) for dep in deptree])
             packages["${PN}" + suffix] = destdir
             _get_package_licenses(destdir, "${PN}" + suffix)
 
@@ -171,7 +177,7 @@ class NpmRecipeHandler(RecipeHandler):
         peer_deps = packages.get("", {}).get("peerDependencies", {})
 
         for peer_dep in peer_deps:
-            peer_dep_yocto_name = npm_package(peer_dep)
+            peer_dep_yocto_name = npm_recipe_name(peer_dep)
             bb.warn(peer_dep + " is a peer dependencie of the actual package. " +
             "Please add this peer dependencie to the RDEPENDS variable as %s and generate its recipe with devtool"
             % peer_dep_yocto_name)
@@ -195,7 +201,7 @@ class NpmRecipeHandler(RecipeHandler):
         if "name" not in data or "version" not in data:
             return False
 
-        extravalues["PN"] = npm_package(data["name"])
+        extravalues["PN"] = npm_recipe_name(data["name"])
         extravalues["PV"] = data["version"]
 
         if "description" in data:
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 05/10] oeqa/selftest: recipetool: Expect the npm source in UNPACKDIR
  2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
                   ` (3 preceding siblings ...)
  2026-11-20 13:53 ` [PATCH 04/10] recipetool: create_npm: Remove the "@" of a scope from generated names Stefano Tondo
@ 2026-11-20 13:53 ` Stefano Tondo
  2026-11-20 13:53 ` [PATCH 06/10] lib/oe/npm: Add SRC_URI entries for the dependencies of a shrinkwrap file Stefano Tondo
                   ` (4 subsequent siblings)
  9 siblings, 0 replies; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

recipetool writes S = "${UNPACKDIR}/npm" for an npm package, because
the fetcher unpacks into UNPACKDIR. The selftest still expected
"${WORKDIR}/npm".

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index c6aa77399f..ba4f674b4c 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -454,7 +454,7 @@ class RecipetoolCreateTests(RecipetoolBase):
                     '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223')
         urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
         checkvars['SRC_URI'] = set(urls)
-        checkvars['S'] = '${WORKDIR}/npm'
+        checkvars['S'] = '${UNPACKDIR}/npm'
         checkvars['LICENSE:${PN}'] = 'MIT'
         checkvars['LICENSE:${PN}-base64'] = 'Unknown'
         checkvars['LICENSE:${PN}-accepts'] = 'MIT'
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 06/10] lib/oe/npm: Add SRC_URI entries for the dependencies of a shrinkwrap file
  2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
                   ` (4 preceding siblings ...)
  2026-11-20 13:53 ` [PATCH 05/10] oeqa/selftest: recipetool: Expect the npm source in UNPACKDIR Stefano Tondo
@ 2026-11-20 13:53 ` Stefano Tondo
  2026-11-20 13:53 ` [PATCH 07/10] npm-update-dependencies: Add a class to list the npm dependencies Stefano Tondo
                   ` (3 subsequent siblings)
  9 siblings, 0 replies; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

The npmsw fetcher reads npm-shrinkwrap.json to find the dependencies of
an npm package. So SRC_URI does not show these dependencies, and the
fetcher holds logic that belongs in the metadata.

Add shrinkwrap_src_uri(), which gives one SRC_URI entry for each
dependency of a shrinkwrap file:

- A package from a registry gets an npm:// entry with its name,
  version and checksum.
- Another tarball gets its URL with the checksum.
- A git dependency gets a git:// entry with its revision.

Each entry unpacks to the location of the dependency in the source
tree, where npm.bbclass expects it. Add src_uri_differences(), which
compares these entries with SRC_URI.

Add tests for both functions.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/lib/oe/npm.py                        |  99 ++++++++++++++++++
 meta/lib/oeqa/selftest/cases/oelib/npm.py | 121 ++++++++++++++++++++++
 2 files changed, 220 insertions(+)
 create mode 100644 meta/lib/oe/npm.py
 create mode 100644 meta/lib/oeqa/selftest/cases/oelib/npm.py

diff --git a/meta/lib/oe/npm.py b/meta/lib/oe/npm.py
new file mode 100644
index 0000000000..19f547839e
--- /dev/null
+++ b/meta/lib/oe/npm.py
@@ -0,0 +1,99 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+"""Explicit SRC_URI entries for the dependencies of an npm shrinkwrap file."""
+
+import os
+import re
+
+import bb.fetch
+import bb.utils
+from bb.fetch.npm import npm_integrity, npm_localfile
+from bb.fetch.npmsw import foreach_dependencies
+
+GIT_RESOLVED_RE = re.compile(r"^git\+(?P<protocol>[a-z]+)://(?P<url>[^#]+)#(?P<rev>[0-9a-f]+)$")
+
+
+def shrinkwrap_src_uri(shrinkwrap, destdir, dev=False):
+    """Return one SRC_URI entry for each dependency of a shrinkwrap file.
+
+    shrinkwrap is the content of npm-shrinkwrap.json (lockfileVersion 2 or
+    later). destdir is the directory of the main package, relative to
+    UNPACKDIR. Each dependency unpacks to its location in destdir (for example
+    destdir/node_modules/foo), where npm.bbclass expects it. dev also adds the
+    devDependencies. The entries are sorted by location.
+
+    A package from a registry gets an npm:// entry, so that the fetcher knows
+    its name and version. Other tarballs get a plain URL and git dependencies
+    a git:// URL. Each entry has the checksum or revision of the shrinkwrap
+    file.
+    """
+    entries = {}
+
+    def _add(name, data, location):
+        package = data.get("name", name)
+        version = data.get("version")
+        resolved = data.get("resolved")
+        integrity = data.get("integrity")
+        destsuffix = os.path.join(destdir, location)
+
+        if not resolved:
+            # npm leaves out "resolved" for some duplicate packages. npm.bbclass
+            # does not need these, as npmsw already skipped them.
+            if version and integrity:
+                return
+            raise ValueError("%s: no 'resolved' field" % location)
+
+        if data.get("link") or resolved.startswith("file:"):
+            raise ValueError("%s: local dependencies are not supported" % location)
+
+        match = GIT_RESOLVED_RE.match(resolved)
+        if match:
+            entries[location] = "git://%s;protocol=%s;rev=%s;nobranch=1;destsuffix=%s" % (
+                match["url"], match["protocol"], match["rev"], destsuffix)
+            return
+
+        if not integrity or not resolved.startswith(("https://", "http://")):
+            raise ValueError("%s: unsupported dependency %s" % (location, resolved))
+
+        checksum_name, checksum = npm_integrity(integrity)
+        basename = package.split("/")[-1]
+        suffix = "/%s/-/%s-%s.tgz" % (package, basename, version)
+        if resolved.startswith("https://") and resolved.endswith(suffix) and bb.utils.is_semver(version):
+            registry = resolved[len("https://"):-len(suffix)]
+            entries[location] = "npm://%s/;package=%s;version=%s;%s=%s;destsuffix=%s" % (
+                registry, package, version, checksum_name, checksum, destsuffix)
+        else:
+            entries[location] = "%s;downloadfilename=%s;%s=%s;subdir=%s;striplevel=1" % (
+                resolved, npm_localfile(os.path.basename(resolved)), checksum_name, checksum, destsuffix)
+
+    foreach_dependencies(shrinkwrap, _add, dev)
+    return [entries[location] for location in sorted(entries)]
+
+
+def src_uri_differences(expected, src_uri):
+    """Compare the expected entries with SRC_URI.
+
+    Returns (missing, extra): the expected entries that SRC_URI does not have,
+    and the SRC_URI entries that unpack into a node_modules directory but are
+    not expected. Entries are compared after parsing, so the order of the URL
+    parameters does not matter.
+    """
+    def _key(url):
+        scheme, host, path, user, password, parm = bb.fetch.decodeurl(url)
+        return (scheme, host, path, user, password, tuple(sorted(parm.items())))
+
+    def _unpacks_to_node_modules(url):
+        parm = bb.fetch.decodeurl(url)[5]
+        dest = parm.get("destsuffix") or parm.get("subdir") or ""
+        return "node_modules/" in dest + "/"
+
+    expected_keys = {_key(url): url for url in expected}
+    actual_keys = {_key(url): url for url in src_uri}
+    missing = [url for key, url in expected_keys.items() if key not in actual_keys]
+    extra = [url for key, url in actual_keys.items()
+             if key not in expected_keys and _unpacks_to_node_modules(url)]
+    return missing, extra
diff --git a/meta/lib/oeqa/selftest/cases/oelib/npm.py b/meta/lib/oeqa/selftest/cases/oelib/npm.py
new file mode 100644
index 0000000000..df4f6aff5b
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/oelib/npm.py
@@ -0,0 +1,121 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import base64
+from unittest.case import TestCase
+
+import oe.npm
+
+CHECKSUM = bytes(range(64)).hex()
+INTEGRITY = "sha512-" + base64.b64encode(bytes(range(64))).decode()
+
+
+def shrinkwrap(packages):
+    """Return a shrinkwrap file with the given dependencies"""
+    return {"lockfileVersion": 3, "packages": {"": {"name": "main"}, **packages}}
+
+
+def registry(package, version, host="registry.npmjs.org"):
+    """Return a dependency from a registry"""
+    basename = package.split("/")[-1]
+    return {
+        "version": version,
+        "resolved": "https://%s/%s/-/%s-%s.tgz" % (host, package, basename, version),
+        "integrity": INTEGRITY,
+    }
+
+
+class TestShrinkwrapSrcUri(TestCase):
+    def test_registry(self):
+        entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({
+            "node_modules/send": registry("send", "0.19.0"),
+            "node_modules/send/node_modules/ms": registry("ms", "2.1.3"),
+            "node_modules/@scope/pkg": registry("@scope/pkg", "1.0.0", "npm.example.com/path"),
+        }), "npm")
+        self.assertEqual(entries, [
+            "npm://npm.example.com/path/;package=@scope/pkg;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/@scope/pkg" % CHECKSUM,
+            "npm://registry.npmjs.org/;package=send;version=0.19.0;sha512sum=%s;destsuffix=npm/node_modules/send" % CHECKSUM,
+            "npm://registry.npmjs.org/;package=ms;version=2.1.3;sha512sum=%s;destsuffix=npm/node_modules/send/node_modules/ms" % CHECKSUM,
+        ])
+
+    def test_alias(self):
+        # An alias installs a package under another name: the "name" field
+        # gives the real package
+        dep = registry("real", "1.0.0")
+        dep["name"] = "real"
+        entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({"node_modules/alias": dep}), "npm")
+        self.assertEqual(entries, [
+            "npm://registry.npmjs.org/;package=real;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/alias" % CHECKSUM,
+        ])
+
+    def test_tarball(self):
+        entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({
+            "node_modules/foo": {
+                "version": "1.0.0",
+                "resolved": "https://example.com/downloads/foo.tgz",
+                "integrity": INTEGRITY,
+            },
+        }), "npm")
+        self.assertEqual(entries, [
+            "https://example.com/downloads/foo.tgz;downloadfilename=npm2/foo.tgz;sha512sum=%s;subdir=npm/node_modules/foo;striplevel=1" % CHECKSUM,
+        ])
+
+    def test_git(self):
+        entries = oe.npm.shrinkwrap_src_uri(shrinkwrap({
+            "node_modules/foo": {
+                "version": "1.0.0",
+                "resolved": "git+ssh://git@github.com/example/foo.git#0123456789abcdef0123456789abcdef01234567",
+            },
+        }), "npm")
+        self.assertEqual(entries, [
+            "git://git@github.com/example/foo.git;protocol=ssh;rev=0123456789abcdef0123456789abcdef01234567;nobranch=1;destsuffix=npm/node_modules/foo",
+        ])
+
+    def test_skipped(self):
+        dev = registry("dev", "1.0.0")
+        dev["dev"] = True
+        bundled = registry("bundled", "1.0.0")
+        bundled["inBundle"] = True
+        no_resolved = registry("dup", "1.0.0")
+        del no_resolved["resolved"]
+        packages = {
+            "node_modules/dev": dev,
+            "node_modules/bundled": bundled,
+            "node_modules/dup": no_resolved,
+        }
+        self.assertEqual(oe.npm.shrinkwrap_src_uri(shrinkwrap(packages), "npm"), [])
+        self.assertEqual(len(oe.npm.shrinkwrap_src_uri(shrinkwrap(packages), "npm", dev=True)), 1)
+
+    def test_unsupported(self):
+        for dep in ({"version": "1.0.0", "resolved": "file:../foo"},
+                    {"link": True, "resolved": "../foo"},
+                    {"version": "1.0.0"}):
+            with self.assertRaises(ValueError):
+                oe.npm.shrinkwrap_src_uri(shrinkwrap({"node_modules/foo": dep}), "npm")
+
+
+class TestSrcUriDifferences(TestCase):
+    EXPECTED = [
+        "npm://registry.npmjs.org/;package=foo;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/foo" % CHECKSUM,
+        "npm://registry.npmjs.org/;package=bar;version=2.0.0;sha512sum=%s;destsuffix=npm/node_modules/bar" % CHECKSUM,
+    ]
+
+    def test_match(self):
+        # The order of the entries and of the URL parameters does not matter.
+        # Entries that do not unpack into node_modules are not extra.
+        src_uri = [
+            "npm://registry.npmjs.org/;package=main;version=1.0.0;sha512sum=%s" % CHECKSUM,
+            "file://npm-shrinkwrap.json;subdir=npm",
+            self.EXPECTED[1],
+            "npm://registry.npmjs.org/;destsuffix=npm/node_modules/foo;sha512sum=%s;version=1.0.0;package=foo" % CHECKSUM,
+        ]
+        self.assertEqual(oe.npm.src_uri_differences(self.EXPECTED, src_uri), ([], []))
+
+    def test_missing_and_extra(self):
+        old_bar = "npm://registry.npmjs.org/;package=bar;version=1.0.0;sha512sum=%s;destsuffix=npm/node_modules/bar" % CHECKSUM
+        missing, extra = oe.npm.src_uri_differences(self.EXPECTED, [self.EXPECTED[0], old_bar])
+        self.assertEqual(missing, [self.EXPECTED[1]])
+        self.assertEqual(extra, [old_bar])
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 07/10] npm-update-dependencies: Add a class to list the npm dependencies
  2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
                   ` (5 preceding siblings ...)
  2026-11-20 13:53 ` [PATCH 06/10] lib/oe/npm: Add SRC_URI entries for the dependencies of a shrinkwrap file Stefano Tondo
@ 2026-11-20 13:53 ` Stefano Tondo
  2026-11-20 13:53 ` [PATCH 08/10] npm: Check that SRC_URI lists all dependencies of the shrinkwrap file Stefano Tondo
                   ` (2 subsequent siblings)
  9 siblings, 0 replies; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

Add a class with the task do_update_npm_deps. The task reads
npm-shrinkwrap.json in ${S} and writes one SRC_URI entry for each
dependency to ${THISDIR}/${BPN}-npm-deps.inc. The recipe requires this
file. This is the same model as the cargo and Go update tasks.

Then the recipe does not need npmsw://. The fetchers see each
dependency as a normal SRC_URI entry with its own checksum.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 .../npm-update-dependencies.bbclass           | 48 +++++++++++++++++++
 1 file changed, 48 insertions(+)
 create mode 100644 meta/classes-recipe/npm-update-dependencies.bbclass

diff --git a/meta/classes-recipe/npm-update-dependencies.bbclass b/meta/classes-recipe/npm-update-dependencies.bbclass
new file mode 100644
index 0000000000..43b00b9b1b
--- /dev/null
+++ b/meta/classes-recipe/npm-update-dependencies.bbclass
@@ -0,0 +1,48 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+##
+## Purpose:
+## This class is used to update the list of npm dependencies in SRC_URI
+## by reading npm-shrinkwrap.json in the source tree. It writes one entry
+## with a checksum for each dependency, so that no fetcher has to read the
+## shrinkwrap file. npm.bbclass checks that SRC_URI matches the shrinkwrap
+## file.
+##
+## To perform the update: bitbake -c update_npm_deps recipe-name
+
+addtask do_update_npm_deps after do_patch
+do_update_npm_deps[nostamp] = "1"
+do_update_npm_deps[doc] = "Update the recipe by reading npm-shrinkwrap.json and write in ${THISDIR}/${BPN}-npm-deps.inc"
+
+RECIPE_UPGRADE_EXTRA_TASKS += "do_update_npm_deps"
+
+python do_update_npm_deps() {
+    import json
+    import oe.npm
+
+    shrinkwrap_file = d.expand("${S}/npm-shrinkwrap.json")
+    with open(shrinkwrap_file) as f:
+        shrinkwrap = json.load(f)
+
+    destdir = os.path.relpath(d.getVar("S"), d.getVar("UNPACKDIR"))
+    dev = bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False)
+    try:
+        entries = oe.npm.shrinkwrap_src_uri(shrinkwrap, destdir, dev)
+    except ValueError as e:
+        bb.fatal("%s: %s" % (shrinkwrap_file, e))
+
+    lines = ["# Autogenerated with 'bitbake -c update_npm_deps %s'" % d.getVar("PN"), ""]
+    lines.append('SRC_URI += " \\')
+    lines.extend("    %s \\" % entry for entry in entries)
+    lines.append('"')
+
+    target_file = d.expand("${THISDIR}/${BPN}-npm-deps.inc")
+    with open(target_file, "w") as f:
+        f.write("\n".join(lines) + "\n")
+
+    bb.note("Successfully updated %d npm dependencies inside %s" % (len(entries), target_file))
+}
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 08/10] npm: Check that SRC_URI lists all dependencies of the shrinkwrap file
  2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
                   ` (6 preceding siblings ...)
  2026-11-20 13:53 ` [PATCH 07/10] npm-update-dependencies: Add a class to list the npm dependencies Stefano Tondo
@ 2026-11-20 13:53 ` Stefano Tondo
  2026-11-20 13:53 ` [PATCH 09/10] recipetool: create: Clean the recipe after its tasks Stefano Tondo
  2026-11-20 13:53 ` [PATCH 10/10] recipetool: create_npm: List the npm dependencies in SRC_URI Stefano Tondo
  9 siblings, 0 replies; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

When a recipe lists the npm dependencies in SRC_URI, the list can become
different from npm-shrinkwrap.json, for example after an upgrade. Then
npm.bbclass packs a wrong set of dependencies.

If SRC_URI has no npmsw:// entry, compare SRC_URI with the shrinkwrap
file in do_configure. Fail if a dependency is missing, or if an entry
unpacks into node_modules but the shrinkwrap file does not have it. The
error tells the user to run the update task.

Do not check with externalsrc, which removes the dependencies from
SRC_URI. There the external source tree must contain them.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/classes-recipe/npm.bbclass | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/meta/classes-recipe/npm.bbclass b/meta/classes-recipe/npm.bbclass
index 6f3261238d..e0d3d88d2b 100644
--- a/meta/classes-recipe/npm.bbclass
+++ b/meta/classes-recipe/npm.bbclass
@@ -111,6 +111,24 @@ def npm_pack(env, srcdir, workdir, excludes=None):
 
     return (tarball, j)
 
+def npm_check_src_uri(d, shrinkwrap):
+    """Check that SRC_URI has one entry for each dependency of the shrinkwrap file"""
+    import oe.npm
+
+    destdir = os.path.relpath(d.getVar("S"), d.getVar("UNPACKDIR"))
+    dev = bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False)
+    try:
+        expected = oe.npm.shrinkwrap_src_uri(shrinkwrap, destdir, dev)
+    except ValueError as e:
+        bb.fatal("npm-shrinkwrap.json: %s" % e)
+
+    missing, extra = oe.npm.src_uri_differences(expected, d.getVar("SRC_URI").split())
+    if missing or extra:
+        bb.fatal("SRC_URI does not match npm-shrinkwrap.json (%d missing, %d extra entries). "
+                 "Inherit npm-update-dependencies and run 'bitbake -c update_npm_deps %s'.\n%s"
+                 % (len(missing), len(extra), d.getVar("PN"),
+                    "\n".join(["missing: " + url for url in missing] + ["extra: " + url for url in extra])))
+
 python npm_do_configure() {
     """
     Step one: configure the npm cache and the main npm package
@@ -177,6 +195,12 @@ python npm_do_configure() {
        if int(orig_shrinkwrap.get("lockfileVersion", 0)) < 2:
            bb.fatal("%s: lockfileVersion version 2 or later is required" % orig_shrinkwrap_file)
 
+       # externalsrc removes the dependencies from SRC_URI: the external
+       # source tree must contain them
+       if not d.getVar("EXTERNALSRC") and \
+               not any(url.startswith("npmsw://") for url in d.getVar("SRC_URI").split()):
+           npm_check_src_uri(d, orig_shrinkwrap)
+
        cached_shrinkwrap = copy.deepcopy(orig_shrinkwrap)
        for package in orig_shrinkwrap["packages"]:
             if package != "":
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 09/10] recipetool: create: Clean the recipe after its tasks
  2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
                   ` (7 preceding siblings ...)
  2026-11-20 13:53 ` [PATCH 08/10] npm: Check that SRC_URI lists all dependencies of the shrinkwrap file Stefano Tondo
@ 2026-11-20 13:53 ` Stefano Tondo
  2026-11-20 13:53 ` [PATCH 10/10] recipetool: create_npm: List the npm dependencies in SRC_URI Stefano Tondo
  9 siblings, 0 replies; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

recipetool can run tasks of the new recipe, as it does for Go. These
tasks leave stamps, for example for do_unpack and do_patch. devtool
then builds the recipe with externalsrc, which uses other stamps, and
its do_clean removes the work directory. The first stamps stay.

A later build of the recipe with the same signatures trusts these
stamps and does not unpack again, so the source is missing. The next
commit runs do_update_npm_deps for npm recipes. Then
test_devtool_add_npm followed by test_recipetool_create_npm fails:
do_update_npm_deps cannot find npm-shrinkwrap.json.

Run do_clean after the tasks.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 scripts/lib/recipetool/create.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 2df17dd6e4..b2cc92c90b 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -955,6 +955,11 @@ def create_recipe(args):
     for task in run_tasks:
         logger.info("Running task %s" % task)
         tinfoil.build_file_sync(outfile, task)
+    if run_tasks:
+        # Remove the work directory and the stamps of these tasks. Else a
+        # later build of the recipe can trust the stamps, also when devtool
+        # has removed the work directory in the meantime.
+        tinfoil.build_file_sync(outfile, "clean")
 
     if tempsrc:
         if args.keep_temp:
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 10/10] recipetool: create_npm: List the npm dependencies in SRC_URI
  2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
                   ` (8 preceding siblings ...)
  2026-11-20 13:53 ` [PATCH 09/10] recipetool: create: Clean the recipe after its tasks Stefano Tondo
@ 2026-11-20 13:53 ` Stefano Tondo
  9 siblings, 0 replies; 12+ messages in thread
From: Stefano Tondo @ 2026-11-20 13:53 UTC (permalink / raw)
  To: openembedded-core
  Cc: Alexander Kanavin, Joshua Watt, Stefan Herbrechtsmeier,
	Peter Marko, stefano.tondo.ext

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

recipetool adds an npmsw:// entry without destsuffix to the recipe.
Since bitbake commit 84f102954, the npmsw fetcher unpacks into the root
of UNPACKDIR and not into ${S}. So npm-shrinkwrap.json is not in ${S},
and npm.bbclass silently packs no dependencies. The package then has no
node_modules.

Create the recipe with the explicit list of dependencies instead:

- Add npm-shrinkwrap.json to SRC_URI, so that it unpacks into ${S}.
- Inherit npm-update-dependencies and require ${BPN}-npm-deps.inc.
- Run the update task, which writes the dependencies to the .inc file.

create_recipe() gives the source subdirectory to the handlers in
extravalues, so that the npm handler knows where ${S} is.

Fetch the dependencies from the same list for the license scan.

The selftest now checks the .inc file.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 16 +++++++--
 scripts/lib/recipetool/create.py           |  4 +++
 scripts/lib/recipetool/create_npm.py       | 42 +++++++++++++++-------
 3 files changed, 48 insertions(+), 14 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index ba4f674b4c..181d95804f 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -5,11 +5,13 @@
 #
 
 import errno
+import json
 import os
 import shutil
 import tempfile
 import urllib.parse
 
+import oe.npm
 import oe.spdx_license
 
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var
@@ -452,16 +454,26 @@ class RecipetoolCreateTests(RecipetoolBase):
         urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}'
                     ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
                     '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223')
-        urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
+        urls.append('file://npm-shrinkwrap.json;subdir=npm')
         checkvars['SRC_URI'] = set(urls)
         checkvars['S'] = '${UNPACKDIR}/npm'
         checkvars['LICENSE:${PN}'] = 'MIT'
         checkvars['LICENSE:${PN}-base64'] = 'Unknown'
         checkvars['LICENSE:${PN}-accepts'] = 'MIT'
         checkvars['LICENSE:${PN}-inherits'] = 'ISC'
-        inherits = ['npm']
+        inherits = ['npm', 'npm-update-dependencies']
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
+        # The update task writes one SRC_URI entry for each dependency
+        with open(shrinkwrap) as f:
+            expected = oe.npm.shrinkwrap_src_uri(json.load(f), 'npm')
+        deps_require_file = os.path.join(temprecipe, 'savoirfairelinux-node-server-example-npm-deps.inc')
+        self.assertFileExists(deps_require_file)
+        checkvars = {'SRC_URI': set(expected)}
+        self._test_recipe_contents(deps_require_file, checkvars, [])
+        self.assertTrue(any(url.startswith('npm://registry.npmjs.org/;package=accepts;')
+                            and url.endswith(';destsuffix=npm/node_modules/accepts') for url in expected))
+
     def test_recipetool_create_github(self):
         # Basic test to see if github URL mangling works. Deliberately use an
         # older release of Meson at present so we don't need a toml parser.
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index b2cc92c90b..46a321a866 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -784,8 +784,12 @@ def create_recipe(args):
         classes.append('bin_package')
         handled.append('buildsystem')
 
+    # Handlers can read the source subdirectory (S relative to UNPACKDIR),
+    # when S is set to it above
+    extravalues['srcsubdir'] = srcsubdir if not args.binary else ''
     for handler in handlers:
         handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
+    extravalues.pop('srcsubdir', None)
 
     # native and nativesdk classes are special and must be inherited last
     # If present, put them at the end of the classes list
diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 50dd45c7d2..1b03aef612 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -12,6 +12,7 @@ import re
 import sys
 import tempfile
 import bb
+import oe.npm
 from bb.fetch.npm import NpmEnvironment
 from bb.fetch.npm import npm_package
 from bb.fetch.npmsw import foreach_dependencies
@@ -257,33 +258,39 @@ class NpmRecipeHandler(RecipeHandler):
         extravalues.setdefault("extrafiles", {})
         extravalues["extrafiles"]["npm-shrinkwrap.json"] = shrinkwrap_copy
 
-        url_local = "npmsw://%s" % shrinkwrap_file
-        url_recipe= "npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json"
+        has_dependencies = "dependencies" in shrinkwrap.get("packages", {}).get("", {})
 
-        if dev:
-            url_local += ";dev=1"
-            url_recipe += ";dev=1"
+        # The shrinkwrap file must be in ${S}, where npm.bbclass reads it
+        srcsubdir = extravalues.get("srcsubdir") or "${BP}"
+        url_recipe = "file://npm-shrinkwrap.json;subdir=%s" % srcsubdir
 
-        # Add the npmsw url in the SRC_URI of the generated recipe
+        # Add the shrinkwrap file to the SRC_URI of the generated recipe
         def _handle_srcuri(varname, origvalue, op, newlines):
-            """Update the version value and add the 'npmsw://' url"""
+            """Update the version value and add the shrinkwrap file"""
             value = origvalue.replace("version=" + data["version"], "version=${PV}")
             value = value.replace("version=latest", "version=${PV}")
             values = [line.strip() for line in value.strip('\n').splitlines()]
-            if "dependencies" in shrinkwrap.get("packages", {}).get("", {}):
+            if has_dependencies:
                 values.append(url_recipe)
             return values, None, 4, False
 
         (_, newlines) = bb.utils.edit_metadata(lines_before, ["SRC_URI"], _handle_srcuri)
         lines_before[:] = [line.rstrip('\n') for line in newlines]
 
+        try:
+            dependency_urls = oe.npm.shrinkwrap_src_uri(shrinkwrap, "", dev)
+        except ValueError as e:
+            bb.error("%s: %s" % (shrinkwrap_file, e))
+            sys.exit(14)
+
         # In order to generate correct licence checksums in the recipe the
-        # dependencies have to be fetched again using the npmsw url
+        # dependencies have to be fetched again
         bb.note("Fetching npm dependencies ...")
         bb.utils.remove(os.path.join(srctree, "node_modules"), recurse=True)
-        fetcher = bb.fetch.Fetch([url_local], d)
-        fetcher.download()
-        fetcher.unpack(srctree)
+        if dependency_urls:
+            fetcher = bb.fetch.Fetch(dependency_urls, d)
+            fetcher.download()
+            fetcher.unpack(srctree)
 
         bb.note("Handling licences ...")
         (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev)
@@ -297,6 +304,17 @@ class NpmRecipeHandler(RecipeHandler):
         classes.append("npm")
         handled.append("buildsystem")
 
+        if has_dependencies:
+            # npm-update-dependencies writes the dependencies to
+            # ${BPN}-npm-deps.inc. Write a stub file, which the task replaces.
+            classes.append("npm-update-dependencies")
+            extravalues["run_tasks"] = "update_npm_deps"
+            stub = os.path.join(tempfile.mkdtemp(prefix="recipetool-npm-"), "npm-deps.inc")
+            with open(stub, "w") as f:
+                f.write("# FROM RECIPETOOL\n")
+            extravalues["extrafiles"]["../{pn}-npm-deps.inc"] = stub
+            lines_before.append("require ${BPN}-npm-deps.inc")
+
         # Check if package has peer dependencies and inform the user
         self._handle_peer_dependency(shrinkwrap_file)
 
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Patchtest results for [PATCH 04/10] recipetool: create_npm: Remove the "@" of a scope from generated names
  2026-11-20 13:53 ` [PATCH 04/10] recipetool: create_npm: Remove the "@" of a scope from generated names Stefano Tondo
@ 2026-11-20 14:04   ` patchtest
  0 siblings, 0 replies; 12+ messages in thread
From: patchtest @ 2026-11-20 14:04 UTC (permalink / raw)
  To: Stefano Tondo; +Cc: openembedded-core

[-- Attachment #1: Type: text/plain, Size: 3269 bytes --]

Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/04-10-recipetool-create_npm-Remove-the-of-a-scope-from-generated-names.patch

FAIL: test commit message user tags: Mbox includes one or more GitHub-style username tags. Ensure that any "@" symbols are stripped out of usernames (test_mbox.TestMbox.test_commit_message_user_tags)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test auh changelog truncation notice (test_mbox.TestMbox.test_auh_changelog_truncation_notice)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new source patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new source patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new source patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping test (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2026-11-20 14:04 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
2026-11-20 13:53 ` [PATCH 01/10] recipetool: create_npm: Give the fallback licenses as a string Stefano Tondo
2026-11-20 13:53 ` [PATCH 02/10] recipetool: create: Get the checksum of npm packages from the registry Stefano Tondo
2026-11-20 13:53 ` [PATCH 03/10] recipetool: create_npm: Give the package fallback licenses as strings Stefano Tondo
2026-11-20 13:53 ` [PATCH 04/10] recipetool: create_npm: Remove the "@" of a scope from generated names Stefano Tondo
2026-11-20 14:04   ` Patchtest results for " patchtest
2026-11-20 13:53 ` [PATCH 05/10] oeqa/selftest: recipetool: Expect the npm source in UNPACKDIR Stefano Tondo
2026-11-20 13:53 ` [PATCH 06/10] lib/oe/npm: Add SRC_URI entries for the dependencies of a shrinkwrap file Stefano Tondo
2026-11-20 13:53 ` [PATCH 07/10] npm-update-dependencies: Add a class to list the npm dependencies Stefano Tondo
2026-11-20 13:53 ` [PATCH 08/10] npm: Check that SRC_URI lists all dependencies of the shrinkwrap file Stefano Tondo
2026-11-20 13:53 ` [PATCH 09/10] recipetool: create: Clean the recipe after its tasks Stefano Tondo
2026-11-20 13:53 ` [PATCH 10/10] recipetool: create_npm: List the npm dependencies in SRC_URI Stefano Tondo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox