All of lore.kernel.org
 help / color / mirror / Atom feed
* Fun with npm shrinkwrap
@ 2017-06-06 10:17 Jan Kiszka
  2017-06-06 14:12 ` Alexander Kanavin
  0 siblings, 1 reply; 8+ messages in thread
From: Jan Kiszka @ 2017-06-06 10:17 UTC (permalink / raw)
  To: openembedded-core; +Cc: Paul Eggleton

Hi all,

devtool add / recipetool create do not work properly /wrt shrinkwrap
when it comes to node-red-node-serialport. npm shrinkwrap finds a number
of extraneous packages and refuses to generate an output.

There are various workarounds discussed in [1]. For me this seems to
work:

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index cb8f338b8b..8ce952bb4c 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -68,6 +68,8 @@ class NpmRecipeHandler(RecipeHandler):
     def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before, d):
         try:
             runenv = dict(os.environ, PATH=d.getVar('PATH'))
+            bb.process.run('rm -rf node_modules', cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
+            bb.process.run('npm install', cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
             bb.process.run('npm shrinkwrap', cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
         except bb.process.ExecutionError as e:
             logger.warn('npm shrinkwrap failed:\n%s' % e.stdout)


But it may have unwanted side effects. So, what would be the proper way
to address this issue?

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: Fun with npm shrinkwrap
  2017-06-06 10:17 Fun with npm shrinkwrap Jan Kiszka
@ 2017-06-06 14:12 ` Alexander Kanavin
  2017-06-06 15:14   ` Jan Kiszka
  0 siblings, 1 reply; 8+ messages in thread
From: Alexander Kanavin @ 2017-06-06 14:12 UTC (permalink / raw)
  To: Jan Kiszka, openembedded-core; +Cc: Paul Eggleton

On 06/06/2017 01:17 PM, Jan Kiszka wrote:
> devtool add / recipetool create do not work properly /wrt shrinkwrap
> when it comes to node-red-node-serialport. npm shrinkwrap finds a number
> of extraneous packages and refuses to generate an output.
>
> There are various workarounds discussed in [1]. For me this seems to
> work:

You forgot to include the actual [1] :)

> But it may have unwanted side effects. So, what would be the proper way
> to address this issue?

A few months ago we had a long, big discussion about how to support 
these kinds of language environments in a way that is sustainable (I 
personally do not think the current approach with npm is viable in the 
long term). Here's the initial couple of emails:

http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html

http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000481.html

Alex


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Fun with npm shrinkwrap
  2017-06-06 14:12 ` Alexander Kanavin
@ 2017-06-06 15:14   ` Jan Kiszka
  2017-06-07 11:43     ` Alexander Kanavin
  0 siblings, 1 reply; 8+ messages in thread
From: Jan Kiszka @ 2017-06-06 15:14 UTC (permalink / raw)
  To: Alexander Kanavin, openembedded-core; +Cc: Paul Eggleton

On 2017-06-06 16:12, Alexander Kanavin wrote:
> On 06/06/2017 01:17 PM, Jan Kiszka wrote:
>> devtool add / recipetool create do not work properly /wrt shrinkwrap
>> when it comes to node-red-node-serialport. npm shrinkwrap finds a number
>> of extraneous packages and refuses to generate an output.
>>
>> There are various workarounds discussed in [1]. For me this seems to
>> work:
> 
> You forgot to include the actual [1] :)

Ah, sorry: https://github.com/npm/npm/issues/4435

> 
>> But it may have unwanted side effects. So, what would be the proper way
>> to address this issue?
> 
> A few months ago we had a long, big discussion about how to support
> these kinds of language environments in a way that is sustainable (I
> personally do not think the current approach with npm is viable in the
> long term). Here's the initial couple of emails:
> 
> http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html
> 
> 
> http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000481.html
> 

Yeah, good points. Probably this problem is also related to OE
reimplementing parts of the npm logic.

But I suppose those concepts won't materialize very soon, will they? I'm
also looking for a short-term solution, at least some reasonable tactic
to generate such recipes in the meantime. Is there something in that
thread I should dig into?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Fun with npm shrinkwrap
  2017-06-06 15:14   ` Jan Kiszka
@ 2017-06-07 11:43     ` Alexander Kanavin
  2017-06-07 12:04       ` Jan Kiszka
  0 siblings, 1 reply; 8+ messages in thread
From: Alexander Kanavin @ 2017-06-07 11:43 UTC (permalink / raw)
  To: Jan Kiszka, openembedded-core; +Cc: Paul Eggleton

On 06/06/2017 06:14 PM, Jan Kiszka wrote:

> Yeah, good points. Probably this problem is also related to OE
> reimplementing parts of the npm logic.
>
> But I suppose those concepts won't materialize very soon, will they? I'm
> also looking for a short-term solution, at least some reasonable tactic
> to generate such recipes in the meantime. Is there something in that
> thread I should dig into?

What I am getting at is that it would be awesome if you convince your 
management that npm support in Yocto is a problem that needs a better 
solution and work on it. Yes, this means you can't work on product 
development for quite some time; this is the price for using open source 
projects without a commercial support contract :)

My apologies if this is not the answer you were seeking.

Alex


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Fun with npm shrinkwrap
  2017-06-07 11:43     ` Alexander Kanavin
@ 2017-06-07 12:04       ` Jan Kiszka
  2017-06-07 14:09         ` Davis, Michael
  0 siblings, 1 reply; 8+ messages in thread
From: Jan Kiszka @ 2017-06-07 12:04 UTC (permalink / raw)
  To: Alexander Kanavin, openembedded-core; +Cc: Paul Eggleton

On 2017-06-07 13:43, Alexander Kanavin wrote:
> On 06/06/2017 06:14 PM, Jan Kiszka wrote:
> 
>> Yeah, good points. Probably this problem is also related to OE
>> reimplementing parts of the npm logic.
>>
>> But I suppose those concepts won't materialize very soon, will they? I'm
>> also looking for a short-term solution, at least some reasonable tactic
>> to generate such recipes in the meantime. Is there something in that
>> thread I should dig into?
> 
> What I am getting at is that it would be awesome if you convince your
> management that npm support in Yocto is a problem that needs a better
> solution and work on it. Yes, this means you can't work on product
> development for quite some time; this is the price for using open source
> projects without a commercial support contract :)

We are such a instance support :) - but we also need to get paid / pay
our contractors.

> 
> My apologies if this is not the answer you were seeking.
> 

I'm getting your point, even more now as I ran into a package that does
not even install at all (node-red-contrib-opcua). Can't promise anything
at this point, but I will carry this forward, in all directions.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Fun with npm shrinkwrap
  2017-06-07 12:04       ` Jan Kiszka
@ 2017-06-07 14:09         ` Davis, Michael
  2017-06-07 18:32           ` Jan Kiszka
  0 siblings, 1 reply; 8+ messages in thread
From: Davis, Michael @ 2017-06-07 14:09 UTC (permalink / raw)
  To: Jan Kiszka, Alexander Kanavin,
	openembedded-core@lists.openembedded.org
  Cc: Paul Eggleton

If offline builds, license checking, or stable dependency versions are not important to you then you can use the meta-nodejs layer.  It just uses the yarn tool directly https://github.com/imyller/meta-nodejs.


On a hijacked side note has yarn been considered as a solution to some of our npm issues?
It seems to be npm with built in licnese checking and better version determination.
Might not cover 100% of our use cases, but then npm only seems to work 50% of the time without tweaking something anyways.



-----Original Message-----
From: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of Jan Kiszka
Sent: Wednesday, June 07, 2017 7:04 AM
To: Alexander Kanavin; openembedded-core@lists.openembedded.org
Cc: Paul Eggleton
Subject: Re: [OE-core] Fun with npm shrinkwrap

On 2017-06-07 13:43, Alexander Kanavin wrote:
> On 06/06/2017 06:14 PM, Jan Kiszka wrote:
> 
>> Yeah, good points. Probably this problem is also related to OE
>> reimplementing parts of the npm logic.
>>
>> But I suppose those concepts won't materialize very soon, will they? I'm
>> also looking for a short-term solution, at least some reasonable tactic
>> to generate such recipes in the meantime. Is there something in that
>> thread I should dig into?
> 
> What I am getting at is that it would be awesome if you convince your
> management that npm support in Yocto is a problem that needs a better
> solution and work on it. Yes, this means you can't work on product
> development for quite some time; this is the price for using open source
> projects without a commercial support contract :)

We are such a instance support :) - but we also need to get paid / pay
our contractors.

> 
> My apologies if this is not the answer you were seeking.
> 

I'm getting your point, even more now as I ran into a package that does
not even install at all (node-red-contrib-opcua). Can't promise anything
at this point, but I will carry this forward, in all directions.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Fun with npm shrinkwrap
  2017-06-07 14:09         ` Davis, Michael
@ 2017-06-07 18:32           ` Jan Kiszka
  2017-06-07 19:51             ` Davis, Michael
  0 siblings, 1 reply; 8+ messages in thread
From: Jan Kiszka @ 2017-06-07 18:32 UTC (permalink / raw)
  To: Davis, Michael, Alexander Kanavin,
	openembedded-core@lists.openembedded.org
  Cc: Paul Eggleton

On 2017-06-07 16:09, Davis, Michael wrote:
> If offline builds, license checking, or stable dependency versions are not important to you then you can use the meta-nodejs layer.  It just uses the yarn tool directly https://github.com/imyller/meta-nodejs.
> 

In fact, meta-nodejs is already in use here, and I've been told to give
"inherit npm-base" a try. Is that what you mean?

How "unreproducible" will builds become? The license checks are not that
critical as I wouldn't trust them right now anyway. Offline builds
correlate with reproduciblilty - we need to archive the artifacts and
generate images with changes later on, independent of what happened
outside meanwhile. Not a killer right now, but not an option on the long
run.

> 
> On a hijacked side note has yarn been considered as a solution to some of our npm issues?
> It seems to be npm with built in licnese checking and better version determination.
> Might not cover 100% of our use cases, but then npm only seems to work 50% of the time without tweaking something anyways.
> 

Can't comment on yarn, but another colleague pointed out that the
machinery of arch-linux may also be worth a look, not only for the
nodejs/npm galaxy.

Thanks,
Jan

PS: Top-posting is considered evil.

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Fun with npm shrinkwrap
  2017-06-07 18:32           ` Jan Kiszka
@ 2017-06-07 19:51             ` Davis, Michael
  0 siblings, 0 replies; 8+ messages in thread
From: Davis, Michael @ 2017-06-07 19:51 UTC (permalink / raw)
  To: Jan Kiszka, Alexander Kanavin,
	openembedded-core@lists.openembedded.org
  Cc: Paul Eggleton

> In fact, meta-nodejs is already in use here, and I've been told to give
> "inherit npm-base" a try. Is that what you mean?

Yes inherit npm-base should give you what you need. With the before mentioned limitations.

> How "unreproducible" will builds become? The license checks are not that
> critical as I wouldn't trust them right now anyway. Offline builds
> correlate with reproduciblilty - we need to archive the artifacts and
> generate images with changes later on, independent of what happened
> outside meanwhile. Not a killer right now, but not an option on the long
> run.

Mostly it just means you may get slightly different versions of dependencies over time.
Depending mostly on how locked down your package.json and those of your deps are.
https://docs.npmjs.com/misc/semver


> PS: Top-posting is considered evil.
The sad shame of having to use Outlook.  No IMAP / POP and davmail hates me.
Bottom-posting is dying a slow death these days.  Only see it in Linux mailing lists for the most part.

--
Mike


^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2017-06-07 19:51 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-06-06 10:17 Fun with npm shrinkwrap Jan Kiszka
2017-06-06 14:12 ` Alexander Kanavin
2017-06-06 15:14   ` Jan Kiszka
2017-06-07 11:43     ` Alexander Kanavin
2017-06-07 12:04       ` Jan Kiszka
2017-06-07 14:09         ` Davis, Michael
2017-06-07 18:32           ` Jan Kiszka
2017-06-07 19:51             ` Davis, Michael

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.