* [kirkstone][PATCH 2/5] classes: npm: Handle peer dependencies for npm packages
2023-05-27 6:25 [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts belouargamohamed
@ 2023-05-27 6:25 ` belouargamohamed
2023-05-27 6:25 ` [kirkstone][PATCH 3/5] recipetool: create: npm: Add support for the new format of the shrinkwrap file belouargamohamed
` (3 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: belouargamohamed @ 2023-05-27 6:25 UTC (permalink / raw)
To: openembedded-core; +Cc: e.aubineau, f.lahoudere, BELOUARGA Mohamed
From: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
NPM changed its manner to handle peer dependencies over its versions:
- Before NPM 3: NPM installs automatically peer dependencies
- Between NPM 3 and 7: NPM shows a warning about peer dependencies
- After NPM 3: NPM reworked its manner how to handle peer dependencies
The shrinkwrap doesn't have the parameters of the peer dependencies, so we cannot
fetch them. in the same time peer dependencies are not direct dependencies, they should
be installed as run time dependencies.
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
---
meta/classes/npm.bbclass | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 8379c7b988..abc4c6f92b 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -108,6 +108,7 @@ python npm_do_configure() {
import tempfile
from bb.fetch2.npm import NpmEnvironment
from bb.fetch2.npm import npm_unpack
+ from bb.fetch2.npm import npm_package
from bb.fetch2.npmsw import foreach_dependencies
from bb.progress import OutOfProgressHandler
from oe.npm_registry import NpmRegistry
@@ -168,6 +169,7 @@ python npm_do_configure() {
if has_shrinkwrap_file:
cached_shrinkwrap = copy.deepcopy(orig_shrinkwrap)
cached_shrinkwrap.pop("dependencies", None)
+ cached_shrinkwrap["packages"][""].pop("peerDependencies", None)
# Manage the dependencies
progress = OutOfProgressHandler(d, r"^(\d+)/(\d+)$")
@@ -202,6 +204,19 @@ python npm_do_configure() {
if has_shrinkwrap_file:
foreach_dependencies(orig_shrinkwrap, _count_dependency, dev)
foreach_dependencies(orig_shrinkwrap, _cache_dependency, dev)
+
+ # Manage Peer Dependencies
+ if has_shrinkwrap_file:
+ packages = orig_shrinkwrap.get("packages", {})
+ peer_deps = packages.get("", {}).get("peerDependencies", {})
+ package_runtime_dependencies = d.getVar("RDEPENDS:%s" % d.getVar("PN"))
+
+ for peer_dep in peer_deps:
+ peer_dep_yocto_name = npm_package(peer_dep)
+ if peer_dep_yocto_name not in package_runtime_dependencies:
+ bb.warn(peer_dep + " is a peer dependencie that is not in RDEPENDS variable. " +
+ "Please add this peer dependencie to the RDEPENDS variable as %s and generate its recipe with devtool"
+ % peer_dep_yocto_name)
# Configure the main package
with tempfile.TemporaryDirectory() as tmpdir:
@@ -277,6 +292,9 @@ python npm_do_compile() {
args.append(("target_arch", d.getVar("NPM_ARCH")))
args.append(("build-from-source", "true"))
+ # Don't install peer dependencies as they should be in RDEPENDS variable
+ args.append(("legacy-peer-deps", "true"))
+
# Pack and install the main package
(tarball, _) = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
cmd = "npm install %s %s" % (shlex.quote(tarball), d.getVar("EXTRA_OENPM"))
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [kirkstone][PATCH 3/5] recipetool: create: npm: Add support for the new format of the shrinkwrap file
2023-05-27 6:25 [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts belouargamohamed
2023-05-27 6:25 ` [kirkstone][PATCH 2/5] classes: npm: Handle peer dependencies for npm packages belouargamohamed
@ 2023-05-27 6:25 ` belouargamohamed
2023-05-27 6:25 ` [kirkstone][PATCH 4/5] recipetool: create: npm: Add support to handle peer dependencies belouargamohamed
` (2 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: belouargamohamed @ 2023-05-27 6:25 UTC (permalink / raw)
To: openembedded-core; +Cc: e.aubineau, f.lahoudere, BELOUARGA Mohamed
From: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
The shrinkwrap file changed its format, but npm does not version this file. So we can use it properly.
The actual changes make the script check if the npm package has dependencies in the actual shrinkwrap format.
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
---
scripts/lib/recipetool/create_npm.py | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index e667a4d19b..25e7ddb472 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -134,11 +134,10 @@ class NpmRecipeHandler(RecipeHandler):
licfiles.append(os.path.relpath(readme, srctree))
# Handle the dependencies
- def _handle_dependency(name, params, deptree):
+ def _handle_dependency(name, params, destdir):
+ deptree = destdir.split('node_modules/')
suffix = "-".join([npm_package(dep) for dep in deptree])
- destdirs = [os.path.join("node_modules", dep) for dep in deptree]
- destdir = os.path.join(*destdirs)
- packages["${PN}-" + suffix] = destdir
+ packages["${PN}" + suffix] = destdir
_licfiles_append_fallback_readme_files(destdir)
with open(shrinkwrap_file, "r") as f:
@@ -234,7 +233,7 @@ class NpmRecipeHandler(RecipeHandler):
value = origvalue.replace("version=" + data["version"], "version=${PV}")
value = value.replace("version=latest", "version=${PV}")
values = [line.strip() for line in value.strip('\n').splitlines()]
- if "dependencies" in shrinkwrap:
+ if "dependencies" in shrinkwrap.get("packages", {}).get("", {}):
values.append(url_recipe)
return values, None, 4, False
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [kirkstone][PATCH 4/5] recipetool: create: npm: Add support to handle peer dependencies
2023-05-27 6:25 [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts belouargamohamed
2023-05-27 6:25 ` [kirkstone][PATCH 2/5] classes: npm: Handle peer dependencies for npm packages belouargamohamed
2023-05-27 6:25 ` [kirkstone][PATCH 3/5] recipetool: create: npm: Add support for the new format of the shrinkwrap file belouargamohamed
@ 2023-05-27 6:25 ` belouargamohamed
2023-05-27 6:25 ` [kirkstone][PATCH 5/5] classes: npm: Add support for the new format of the shrinkwrap file belouargamohamed
2023-05-29 15:13 ` [OE-core] [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts Steve Sakoman
4 siblings, 0 replies; 10+ messages in thread
From: belouargamohamed @ 2023-05-27 6:25 UTC (permalink / raw)
To: openembedded-core; +Cc: e.aubineau, f.lahoudere, BELOUARGA Mohamed
From: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
NPM changed its manner to handle peer dependencies over its versions:
- Before NPM 3: NPM installs automatically peer dependencies
- Between NPM 3 and 7: NPM shows a warning about peer dependencies
- After NPM 3: NPM reworked its manner how to handle peer dependencies
The shrinkwrap doesn't have the parameters of the peer dependencies, so we cannot
fetch them. in the same time peer dependencies are not direct dependencies, they should
be installed as run time dependencies.
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
---
scripts/lib/recipetool/create_npm.py | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 25e7ddb472..113a89f6a6 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -146,6 +146,23 @@ class NpmRecipeHandler(RecipeHandler):
foreach_dependencies(shrinkwrap, _handle_dependency, dev)
return licfiles, packages
+
+ # Handle the peer dependencies
+ def _handle_peer_dependency(self, shrinkwrap_file):
+ """Check if package has peer dependencies and show warning if it is the case"""
+ with open(shrinkwrap_file, "r") as f:
+ shrinkwrap = json.load(f)
+
+ packages = shrinkwrap.get("packages", {})
+ peer_deps = packages.get("", {}).get("peerDependencies", {})
+
+ for peer_dep in peer_deps:
+ peer_dep_yocto_name = npm_package(peer_dep)
+ bb.warn(peer_dep + " is a peer dependencie of the actual package. " +
+ "Please add this peer dependencie to the RDEPENDS variable as %s and generate its recipe with devtool"
+ % peer_dep_yocto_name)
+
+
def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
"""Handle the npm recipe creation"""
@@ -283,6 +300,9 @@ class NpmRecipeHandler(RecipeHandler):
classes.append("npm")
handled.append("buildsystem")
+ # Check if package has peer dependencies and inform the user
+ self._handle_peer_dependency(shrinkwrap_file)
+
return True
def register_recipe_handlers(handlers):
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [kirkstone][PATCH 5/5] classes: npm: Add support for the new format of the shrinkwrap file
2023-05-27 6:25 [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts belouargamohamed
` (2 preceding siblings ...)
2023-05-27 6:25 ` [kirkstone][PATCH 4/5] recipetool: create: npm: Add support to handle peer dependencies belouargamohamed
@ 2023-05-27 6:25 ` belouargamohamed
2023-05-29 15:13 ` [OE-core] [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts Steve Sakoman
4 siblings, 0 replies; 10+ messages in thread
From: belouargamohamed @ 2023-05-27 6:25 UTC (permalink / raw)
To: openembedded-core; +Cc: e.aubineau, f.lahoudere, BELOUARGA Mohamed
From: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
1 - Adapt do_configure to the new format of the shrinkwrap
2 - Remove useless function _npmsw_dependency_dict because the dictionnary
is already given by npmsw:foreach_dependencies
3 - Rename arguments of callback functions
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
---
meta/classes/npm.bbclass | 27 ++++++---------------------
1 file changed, 6 insertions(+), 21 deletions(-)
diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index abc4c6f92b..6899a64909 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -129,22 +129,6 @@ python npm_do_configure() {
sha512 = bb.utils.sha512_file(tarball)
return "sha512-" + base64.b64encode(bytes.fromhex(sha512)).decode()
- def _npmsw_dependency_dict(orig, deptree):
- """
- Return the sub dictionary in the 'orig' dictionary corresponding to the
- 'deptree' dependency tree. This function follows the shrinkwrap file
- format.
- """
- ptr = orig
- for dep in deptree:
- if "dependencies" not in ptr:
- ptr["dependencies"] = {}
- ptr = ptr["dependencies"]
- if dep not in ptr:
- ptr[dep] = {}
- ptr = ptr[dep]
- return ptr
-
# Manage the manifest file and shrinkwrap files
orig_manifest_file = d.expand("${S}/package.json")
orig_shrinkwrap_file = d.expand("${S}/npm-shrinkwrap.json")
@@ -176,24 +160,25 @@ python npm_do_configure() {
progress_total = 1 # also count the main package
progress_done = 0
- def _count_dependency(name, params, deptree):
+ def _count_dependency(name, params, destsuffix):
nonlocal progress_total
progress_total += 1
- def _cache_dependency(name, params, deptree):
- destsubdirs = [os.path.join("node_modules", dep) for dep in deptree]
- destsuffix = os.path.join(*destsubdirs)
+ def _cache_dependency(name, params, destsuffix):
with tempfile.TemporaryDirectory() as tmpdir:
# Add the dependency to the npm cache
destdir = os.path.join(d.getVar("S"), destsuffix)
(tarball, pkg) = npm_pack(env, destdir, tmpdir)
_npm_cache_add(tarball, pkg)
# Add its signature to the cached shrinkwrap
- dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
+ dep = params
dep["version"] = pkg['version']
dep["integrity"] = _npm_integrity(tarball)
if params.get("dev", False):
dep["dev"] = True
+ if "dependencies" not in cached_shrinkwrap:
+ cached_shrinkwrap["dependencies"] = {}
+ cached_shrinkwrap["dependencies"][name] = dep
# Display progress
nonlocal progress_done
progress_done += 1
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [OE-core] [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts
2023-05-27 6:25 [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts belouargamohamed
` (3 preceding siblings ...)
2023-05-27 6:25 ` [kirkstone][PATCH 5/5] classes: npm: Add support for the new format of the shrinkwrap file belouargamohamed
@ 2023-05-29 15:13 ` Steve Sakoman
2023-05-29 20:16 ` belouargamohamed
4 siblings, 1 reply; 10+ messages in thread
From: Steve Sakoman @ 2023-05-29 15:13 UTC (permalink / raw)
To: belouargamohamed
Cc: openembedded-core, e.aubineau, f.lahoudere, BELOUARGA Mohamed
Are these same issues present in the master branch? If so, this
series should be submitted for the master branch first, and I will
then backport to kirkstone (and mickledore).
Steve
On Fri, May 26, 2023 at 8:25 PM <belouargamohamed@gmail.com> wrote:
>
> From: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
>
> Npm packages do not have yocto friendly names. fore instance we can have names like
> "@example/npmPackage"
>
> npm fetcher has a function that convert these names to yocto friendly names.
> But in recipe tool we have an other function (duplicate).
>
> Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
> ---
> scripts/lib/recipetool/create_npm.py | 14 +++-----------
> 1 file changed, 3 insertions(+), 11 deletions(-)
>
> diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
> index 3394a89970..e667a4d19b 100644
> --- a/scripts/lib/recipetool/create_npm.py
> +++ b/scripts/lib/recipetool/create_npm.py
> @@ -13,6 +13,7 @@ import sys
> import tempfile
> import bb
> from bb.fetch2.npm import NpmEnvironment
> +from bb.fetch2.npm import npm_package
> from bb.fetch2.npmsw import foreach_dependencies
> from recipetool.create import RecipeHandler
> from recipetool.create import get_license_md5sums
> @@ -30,15 +31,6 @@ def tinfoil_init(instance):
> class NpmRecipeHandler(RecipeHandler):
> """Class to handle the npm recipe creation"""
>
> - @staticmethod
> - def _npm_name(name):
> - """Generate a Yocto friendly npm name"""
> - name = re.sub("/", "-", name)
> - name = name.lower()
> - name = re.sub(r"[^\-a-z0-9]", "", name)
> - name = name.strip("-")
> - return name
> -
> @staticmethod
> def _get_registry(lines):
> """Get the registry value from the 'npm://registry' url"""
> @@ -143,7 +135,7 @@ class NpmRecipeHandler(RecipeHandler):
>
> # Handle the dependencies
> def _handle_dependency(name, params, deptree):
> - suffix = "-".join([self._npm_name(dep) for dep in deptree])
> + suffix = "-".join([npm_package(dep) for dep in deptree])
> destdirs = [os.path.join("node_modules", dep) for dep in deptree]
> destdir = os.path.join(*destdirs)
> packages["${PN}-" + suffix] = destdir
> @@ -173,7 +165,7 @@ class NpmRecipeHandler(RecipeHandler):
> if "name" not in data or "version" not in data:
> return False
>
> - extravalues["PN"] = self._npm_name(data["name"])
> + extravalues["PN"] = npm_package(data["name"])
> extravalues["PV"] = data["version"]
>
> if "description" in data:
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#181791): https://lists.openembedded.org/g/openembedded-core/message/181791
> Mute This Topic: https://lists.openembedded.org/mt/99163838/3620601
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [steve@sakoman.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts
2023-05-29 15:13 ` [OE-core] [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts Steve Sakoman
@ 2023-05-29 20:16 ` belouargamohamed
[not found] ` <CAOSpxda3AWOQfskRpCWB0zQMDHs+SMKz0RGqE5z2Nd80DRKp7g@mail.gmail.com>
0 siblings, 1 reply; 10+ messages in thread
From: belouargamohamed @ 2023-05-29 20:16 UTC (permalink / raw)
To: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 729 bytes --]
Yes, the same issues are present in master branch.
But this series of patches won't work on master, it needs some adaptation.
Because in kirkstone we have Nodejs 16.19.1 and in master we have Nodejs 18.16, and they generate shrinkwrap files that are slightly different.
For Mickledore, I didn't test yet but I think that it will work like in the master branch.
Note: these changes work only if my patch for bitbake is accepted: https://lists.openembedded.org/g/bitbake-devel/topic/kirkstone_patch_fetch2/99163598?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,0,99163598,previd%3D1685367798112436212,nextid%3D1683629533018906083&previd=1685367798112436212&nextid=1683629533018906083
Best regards,
BELOUARGA Mohamed
[-- Attachment #2: Type: text/html, Size: 1265 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread