Fun with npm shrinkwrap

Fun with npm shrinkwrap

[Jan Kiszka]

Hi all,

devtool add / recipetool create do not work properly /wrt shrinkwrap
when it comes to node-red-node-serialport. npm shrinkwrap finds a number
of extraneous packages and refuses to generate an output.

There are various workarounds discussed in [1]. For me this seems to
work:

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index cb8f338b8b..8ce952bb4c 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -68,6 +68,8 @@ class NpmRecipeHandler(RecipeHandler):
     def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before, d):
         try:
             runenv = dict(os.environ, PATH=d.getVar('PATH'))
+            bb.process.run('rm -rf node_modules', cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
+            bb.process.run('npm install', cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
             bb.process.run('npm shrinkwrap', cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
         except bb.process.ExecutionError as e:
             logger.warn('npm shrinkwrap failed:\n%s' % e.stdout)


But it may have unwanted side effects. So, what would be the proper way
to address this issue?

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Fun with npm shrinkwrap

[Alexander Kanavin]

On 06/06/2017 01:17 PM, Jan Kiszka wrote:
> devtool add / recipetool create do not work properly /wrt shrinkwrap
> when it comes to node-red-node-serialport. npm shrinkwrap finds a number
> of extraneous packages and refuses to generate an output.
>
> There are various workarounds discussed in [1]. For me this seems to
> work:

You forgot to include the actual [1] :)

> But it may have unwanted side effects. So, what would be the proper way
> to address this issue?

A few months ago we had a long, big discussion about how to support 
these kinds of language environments in a way that is sustainable (I 
personally do not think the current approach with npm is viable in the 
long term). Here's the initial couple of emails:

http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html

http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000481.html

Alex

Re: Fun with npm shrinkwrap

[Jan Kiszka]

On 2017-06-06 16:12, Alexander Kanavin wrote:
> On 06/06/2017 01:17 PM, Jan Kiszka wrote:
>> devtool add / recipetool create do not work properly /wrt shrinkwrap
>> when it comes to node-red-node-serialport. npm shrinkwrap finds a number
>> of extraneous packages and refuses to generate an output.
>>
>> There are various workarounds discussed in [1]. For me this seems to
>> work:
> 
> You forgot to include the actual [1] :)

Ah, sorry: https://github.com/npm/npm/issues/4435

> 
>> But it may have unwanted side effects. So, what would be the proper way
>> to address this issue?
> 
> A few months ago we had a long, big discussion about how to support
> these kinds of language environments in a way that is sustainable (I
> personally do not think the current approach with npm is viable in the
> long term). Here's the initial couple of emails:
> 
> http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html
> 
> 
> http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000481.html
> 

Yeah, good points. Probably this problem is also related to OE
reimplementing parts of the npm logic.

But I suppose those concepts won't materialize very soon, will they? I'm
also looking for a short-term solution, at least some reasonable tactic
to generate such recipes in the meantime. Is there something in that
thread I should dig into?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Fun with npm shrinkwrap

[Alexander Kanavin]

On 06/06/2017 06:14 PM, Jan Kiszka wrote:

> Yeah, good points. Probably this problem is also related to OE
> reimplementing parts of the npm logic.
>
> But I suppose those concepts won't materialize very soon, will they? I'm
> also looking for a short-term solution, at least some reasonable tactic
> to generate such recipes in the meantime. Is there something in that
> thread I should dig into?

What I am getting at is that it would be awesome if you convince your 
management that npm support in Yocto is a problem that needs a better 
solution and work on it. Yes, this means you can't work on product 
development for quite some time; this is the price for using open source 
projects without a commercial support contract :)

My apologies if this is not the answer you were seeking.

Alex

Re: Fun with npm shrinkwrap

[Jan Kiszka]

On 2017-06-07 13:43, Alexander Kanavin wrote:
> On 06/06/2017 06:14 PM, Jan Kiszka wrote:
> 
>> Yeah, good points. Probably this problem is also related to OE
>> reimplementing parts of the npm logic.
>>
>> But I suppose those concepts won't materialize very soon, will they? I'm
>> also looking for a short-term solution, at least some reasonable tactic
>> to generate such recipes in the meantime. Is there something in that
>> thread I should dig into?
> 
> What I am getting at is that it would be awesome if you convince your
> management that npm support in Yocto is a problem that needs a better
> solution and work on it. Yes, this means you can't work on product
> development for quite some time; this is the price for using open source
> projects without a commercial support contract :)

We are such a instance support :) - but we also need to get paid / pay
our contractors.

> 
> My apologies if this is not the answer you were seeking.
> 

I'm getting your point, even more now as I ran into a package that does
not even install at all (node-red-contrib-opcua). Can't promise anything
at this point, but I will carry this forward, in all directions.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Fun with npm shrinkwrap

[Davis, Michael]

If offline builds, license checking, or stable dependency versions are not important to you then you can use the meta-nodejs layer.  It just uses the yarn tool directly https://github.com/imyller/meta-nodejs.


On a hijacked side note has yarn been considered as a solution to some of our npm issues?
It seems to be npm with built in licnese checking and better version determination.
Might not cover 100% of our use cases, but then npm only seems to work 50% of the time without tweaking something anyways.



-----Original Message-----
From: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of Jan Kiszka
Sent: Wednesday, June 07, 2017 7:04 AM
To: Alexander Kanavin; openembedded-core@lists.openembedded.org
Cc: Paul Eggleton
Subject: Re: [OE-core] Fun with npm shrinkwrap

On 2017-06-07 13:43, Alexander Kanavin wrote:
> On 06/06/2017 06:14 PM, Jan Kiszka wrote:
> 
>> Yeah, good points. Probably this problem is also related to OE
>> reimplementing parts of the npm logic.
>>
>> But I suppose those concepts won't materialize very soon, will they? I'm
>> also looking for a short-term solution, at least some reasonable tactic
>> to generate such recipes in the meantime. Is there something in that
>> thread I should dig into?
> 
> What I am getting at is that it would be awesome if you convince your
> management that npm support in Yocto is a problem that needs a better
> solution and work on it. Yes, this means you can't work on product
> development for quite some time; this is the price for using open source
> projects without a commercial support contract :)

We are such a instance support :) - but we also need to get paid / pay
our contractors.

> 
> My apologies if this is not the answer you were seeking.
> 

I'm getting your point, even more now as I ran into a package that does
not even install at all (node-red-contrib-opcua). Can't promise anything
at this point, but I will carry this forward, in all directions.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Fun with npm shrinkwrap

[Jan Kiszka]

On 2017-06-07 16:09, Davis, Michael wrote:
> If offline builds, license checking, or stable dependency versions are not important to you then you can use the meta-nodejs layer.  It just uses the yarn tool directly https://github.com/imyller/meta-nodejs.
> 

In fact, meta-nodejs is already in use here, and I've been told to give
"inherit npm-base" a try. Is that what you mean?

How "unreproducible" will builds become? The license checks are not that
critical as I wouldn't trust them right now anyway. Offline builds
correlate with reproduciblilty - we need to archive the artifacts and
generate images with changes later on, independent of what happened
outside meanwhile. Not a killer right now, but not an option on the long
run.

> 
> On a hijacked side note has yarn been considered as a solution to some of our npm issues?
> It seems to be npm with built in licnese checking and better version determination.
> Might not cover 100% of our use cases, but then npm only seems to work 50% of the time without tweaking something anyways.
> 

Can't comment on yarn, but another colleague pointed out that the
machinery of arch-linux may also be worth a look, not only for the
nodejs/npm galaxy.

Thanks,
Jan

PS: Top-posting is considered evil.

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Fun with npm shrinkwrap

[Davis, Michael]

> In fact, meta-nodejs is already in use here, and I've been told to give
> "inherit npm-base" a try. Is that what you mean?

Yes inherit npm-base should give you what you need. With the before mentioned limitations.

> How "unreproducible" will builds become? The license checks are not that
> critical as I wouldn't trust them right now anyway. Offline builds
> correlate with reproduciblilty - we need to archive the artifacts and
> generate images with changes later on, independent of what happened
> outside meanwhile. Not a killer right now, but not an option on the long
> run.

Mostly it just means you may get slightly different versions of dependencies over time.
Depending mostly on how locked down your package.json and those of your deps are.
https://docs.npmjs.com/misc/semver


> PS: Top-posting is considered evil.
The sad shame of having to use Outlook.  No IMAP / POP and davmail hates me.
Bottom-posting is dying a slow death these days.  Only see it in Linux mailing lists for the most part.

--
Mike