Openembedded Core Discussions
 help / color / mirror / Atom feed
From: Stefano Tondo <stondo@gmail.com>
To: openembedded-core@lists.openembedded.org
Cc: Alexander Kanavin <alex.kanavin@gmail.com>,
	Joshua Watt <JPEWhacker@gmail.com>,
	Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>,
	Peter Marko <peter.marko@siemens.com>,
	stefano.tondo.ext@siemens.com
Subject: [PATCH 10/10] recipetool: create_npm: List the npm dependencies in SRC_URI
Date: Fri, 20 Nov 2026 14:53:41 +0100	[thread overview]
Message-ID: <20261120135342.2628228-11-stondo@gmail.com> (raw)
In-Reply-To: <20261120135342.2628228-1-stondo@gmail.com>

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

recipetool adds an npmsw:// entry without destsuffix to the recipe.
Since bitbake commit 84f102954, the npmsw fetcher unpacks into the root
of UNPACKDIR and not into ${S}. So npm-shrinkwrap.json is not in ${S},
and npm.bbclass silently packs no dependencies. The package then has no
node_modules.

Create the recipe with the explicit list of dependencies instead:

- Add npm-shrinkwrap.json to SRC_URI, so that it unpacks into ${S}.
- Inherit npm-update-dependencies and require ${BPN}-npm-deps.inc.
- Run the update task, which writes the dependencies to the .inc file.

create_recipe() gives the source subdirectory to the handlers in
extravalues, so that the npm handler knows where ${S} is.

Fetch the dependencies from the same list for the license scan.

The selftest now checks the .inc file.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 16 +++++++--
 scripts/lib/recipetool/create.py           |  4 +++
 scripts/lib/recipetool/create_npm.py       | 42 +++++++++++++++-------
 3 files changed, 48 insertions(+), 14 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index ba4f674b4c..181d95804f 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -5,11 +5,13 @@
 #
 
 import errno
+import json
 import os
 import shutil
 import tempfile
 import urllib.parse
 
+import oe.npm
 import oe.spdx_license
 
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var
@@ -452,16 +454,26 @@ class RecipetoolCreateTests(RecipetoolBase):
         urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}'
                     ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
                     '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223')
-        urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
+        urls.append('file://npm-shrinkwrap.json;subdir=npm')
         checkvars['SRC_URI'] = set(urls)
         checkvars['S'] = '${UNPACKDIR}/npm'
         checkvars['LICENSE:${PN}'] = 'MIT'
         checkvars['LICENSE:${PN}-base64'] = 'Unknown'
         checkvars['LICENSE:${PN}-accepts'] = 'MIT'
         checkvars['LICENSE:${PN}-inherits'] = 'ISC'
-        inherits = ['npm']
+        inherits = ['npm', 'npm-update-dependencies']
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
+        # The update task writes one SRC_URI entry for each dependency
+        with open(shrinkwrap) as f:
+            expected = oe.npm.shrinkwrap_src_uri(json.load(f), 'npm')
+        deps_require_file = os.path.join(temprecipe, 'savoirfairelinux-node-server-example-npm-deps.inc')
+        self.assertFileExists(deps_require_file)
+        checkvars = {'SRC_URI': set(expected)}
+        self._test_recipe_contents(deps_require_file, checkvars, [])
+        self.assertTrue(any(url.startswith('npm://registry.npmjs.org/;package=accepts;')
+                            and url.endswith(';destsuffix=npm/node_modules/accepts') for url in expected))
+
     def test_recipetool_create_github(self):
         # Basic test to see if github URL mangling works. Deliberately use an
         # older release of Meson at present so we don't need a toml parser.
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index b2cc92c90b..46a321a866 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -784,8 +784,12 @@ def create_recipe(args):
         classes.append('bin_package')
         handled.append('buildsystem')
 
+    # Handlers can read the source subdirectory (S relative to UNPACKDIR),
+    # when S is set to it above
+    extravalues['srcsubdir'] = srcsubdir if not args.binary else ''
     for handler in handlers:
         handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
+    extravalues.pop('srcsubdir', None)
 
     # native and nativesdk classes are special and must be inherited last
     # If present, put them at the end of the classes list
diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 50dd45c7d2..1b03aef612 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -12,6 +12,7 @@ import re
 import sys
 import tempfile
 import bb
+import oe.npm
 from bb.fetch.npm import NpmEnvironment
 from bb.fetch.npm import npm_package
 from bb.fetch.npmsw import foreach_dependencies
@@ -257,33 +258,39 @@ class NpmRecipeHandler(RecipeHandler):
         extravalues.setdefault("extrafiles", {})
         extravalues["extrafiles"]["npm-shrinkwrap.json"] = shrinkwrap_copy
 
-        url_local = "npmsw://%s" % shrinkwrap_file
-        url_recipe= "npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json"
+        has_dependencies = "dependencies" in shrinkwrap.get("packages", {}).get("", {})
 
-        if dev:
-            url_local += ";dev=1"
-            url_recipe += ";dev=1"
+        # The shrinkwrap file must be in ${S}, where npm.bbclass reads it
+        srcsubdir = extravalues.get("srcsubdir") or "${BP}"
+        url_recipe = "file://npm-shrinkwrap.json;subdir=%s" % srcsubdir
 
-        # Add the npmsw url in the SRC_URI of the generated recipe
+        # Add the shrinkwrap file to the SRC_URI of the generated recipe
         def _handle_srcuri(varname, origvalue, op, newlines):
-            """Update the version value and add the 'npmsw://' url"""
+            """Update the version value and add the shrinkwrap file"""
             value = origvalue.replace("version=" + data["version"], "version=${PV}")
             value = value.replace("version=latest", "version=${PV}")
             values = [line.strip() for line in value.strip('\n').splitlines()]
-            if "dependencies" in shrinkwrap.get("packages", {}).get("", {}):
+            if has_dependencies:
                 values.append(url_recipe)
             return values, None, 4, False
 
         (_, newlines) = bb.utils.edit_metadata(lines_before, ["SRC_URI"], _handle_srcuri)
         lines_before[:] = [line.rstrip('\n') for line in newlines]
 
+        try:
+            dependency_urls = oe.npm.shrinkwrap_src_uri(shrinkwrap, "", dev)
+        except ValueError as e:
+            bb.error("%s: %s" % (shrinkwrap_file, e))
+            sys.exit(14)
+
         # In order to generate correct licence checksums in the recipe the
-        # dependencies have to be fetched again using the npmsw url
+        # dependencies have to be fetched again
         bb.note("Fetching npm dependencies ...")
         bb.utils.remove(os.path.join(srctree, "node_modules"), recurse=True)
-        fetcher = bb.fetch.Fetch([url_local], d)
-        fetcher.download()
-        fetcher.unpack(srctree)
+        if dependency_urls:
+            fetcher = bb.fetch.Fetch(dependency_urls, d)
+            fetcher.download()
+            fetcher.unpack(srctree)
 
         bb.note("Handling licences ...")
         (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev)
@@ -297,6 +304,17 @@ class NpmRecipeHandler(RecipeHandler):
         classes.append("npm")
         handled.append("buildsystem")
 
+        if has_dependencies:
+            # npm-update-dependencies writes the dependencies to
+            # ${BPN}-npm-deps.inc. Write a stub file, which the task replaces.
+            classes.append("npm-update-dependencies")
+            extravalues["run_tasks"] = "update_npm_deps"
+            stub = os.path.join(tempfile.mkdtemp(prefix="recipetool-npm-"), "npm-deps.inc")
+            with open(stub, "w") as f:
+                f.write("# FROM RECIPETOOL\n")
+            extravalues["extrafiles"]["../{pn}-npm-deps.inc"] = stub
+            lines_before.append("require ${BPN}-npm-deps.inc")
+
         # Check if package has peer dependencies and inform the user
         self._handle_peer_dependency(shrinkwrap_file)
 
-- 
2.56.1


      parent reply	other threads:[~2026-11-20 13:54 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 13:53 [PATCH 00/10] npm: list the dependencies in SRC_URI Stefano Tondo
2026-11-20 13:53 ` [PATCH 01/10] recipetool: create_npm: Give the fallback licenses as a string Stefano Tondo
2026-11-20 13:53 ` [PATCH 02/10] recipetool: create: Get the checksum of npm packages from the registry Stefano Tondo
2026-11-20 13:53 ` [PATCH 03/10] recipetool: create_npm: Give the package fallback licenses as strings Stefano Tondo
2026-11-20 13:53 ` [PATCH 04/10] recipetool: create_npm: Remove the "@" of a scope from generated names Stefano Tondo
2026-11-20 14:04   ` Patchtest results for " patchtest
2026-11-20 13:53 ` [PATCH 05/10] oeqa/selftest: recipetool: Expect the npm source in UNPACKDIR Stefano Tondo
2026-11-20 13:53 ` [PATCH 06/10] lib/oe/npm: Add SRC_URI entries for the dependencies of a shrinkwrap file Stefano Tondo
2026-11-20 13:53 ` [PATCH 07/10] npm-update-dependencies: Add a class to list the npm dependencies Stefano Tondo
2026-11-20 13:53 ` [PATCH 08/10] npm: Check that SRC_URI lists all dependencies of the shrinkwrap file Stefano Tondo
2026-11-20 13:53 ` [PATCH 09/10] recipetool: create: Clean the recipe after its tasks Stefano Tondo
2026-11-20 13:53 ` Stefano Tondo [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120135342.2628228-11-stondo@gmail.com \
    --to=stondo@gmail.com \
    --cc=JPEWhacker@gmail.com \
    --cc=alex.kanavin@gmail.com \
    --cc=openembedded-core@lists.openembedded.org \
    --cc=peter.marko@siemens.com \
    --cc=stefan.herbrechtsmeier@weidmueller.com \
    --cc=stefano.tondo.ext@siemens.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox