npm.bbclass

npm.bbclass

[Christopher Lord]

[-- Attachment #1: Type: text/plain, Size: 1493 bytes --]

I've been using npm.bbclass with various node projects we're working on in
the Connected Devices team at Mozilla, and had some questions/comments
about behaviour (note that I'm basing this off of ostro master, which I
unfortunately have to use as no other Intel Edison bsp is adequate);

- Is there a good reason the npm fetcher only works with a registry? It
seems it could get most (all?) pertinent information from a package.json in
the root of a repository.

- Is there a reason to split the package like it does? Node projects tend
to have huge dependency trees, it makes updating and distributing
node-based applications a bit of a chore if they end up split into 20
packages, most of which have no use separately. It would be great if there
was at least a way to disable this.

- The information about packaging non-registry software here:
https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM is incomplete - you
also need to generate a shrinkwrap and install that into the srcdir, or
you're very likely to get failures doing dependency resolution. Not an OE
issue, but I guess Yocto folks also read this list and it's related :)

- Any patches end up getting packaged because they get put in the srcdir.
I'm guessing this isn't intentional (or maybe it is?)

Just wanted to provide some feedback. It's fantastic that OE has the
ability to package node software, and despite the teething difficulties,
I've appreciated its availability!

Cheers,

--Chris

[-- Attachment #2: Type: text/html, Size: 1763 bytes --]

Re: npm.bbclass

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 366 bytes --]

On 16 August 2016 at 17:41, Christopher Lord <clord@mozilla.com> wrote:

> - Any patches end up getting packaged because they get put in the srcdir.
> I'm guessing this isn't intentional (or maybe it is?)
>

This isn't intentional - I've love to move the patches out of ${S} but
that's non-trivial - so npm.bbclass should be skipping that directory.

Ross

[-- Attachment #2: Type: text/html, Size: 740 bytes --]

Re: npm.bbclass

[Brendan Le Foll]

On Tue, Aug 16, 2016 at 05:41:28PM +0100, Christopher Lord wrote:
>    I've been using npm.bbclass with various node projects we're working on
>    in the Connected Devices team at Mozilla, and had some
>    questions/comments about behaviour (note that I'm basing this off of
>    ostro master, which I unfortunately have to use as no other Intel
>    Edison bsp is adequate);

I'm sorry :/

>    - Is there a good reason the npm fetcher only works with a registry? It
>    seems it could get most (all?) pertinent information from a
>    package.json in the root of a repository.

Not really a good reason, I'm trying to make it so that you can use it
with a git fetched source but I haven't had the time to finish it off.
It's on the todo list :)

>    - Is there a reason to split the package like it does? Node projects
>    tend to have huge dependency trees, it makes updating and distributing
>    node-based applications a bit of a chore if they end up split into 20
>    packages, most of which have no use separately. It would be great if
>    there was at least a way to disable this.

Paul added this, the worry was that we wanted to make sure all the
licenses where tracked properly of the package etc... I'm not a huge
fan either to be honest. Maybe we can have a npm-no-split.bbclass
would that be ok - Paul? It's in python populate_packages_prepend in
npm.bbclass.

>    - The information about packaging non-registry software here:
>    [1]https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM is incomplete -
>    you also need to generate a shrinkwrap and install that into the
>    srcdir, or you're very likely to get failures doing dependency
>    resolution. Not an OE issue, but I guess Yocto folks also read this
>    list and it's related :)

Agree 100%, that is still WIP, Henry can you make sure you add a how
to use the npm lockdown file too in the tutorial? Note - If you use
recipetool It does this by default :). And recipetool with node.js
pkgs is quite cool imho.

>    - Any patches end up getting packaged because they get put in the
>    srcdir. I'm guessing this isn't intentional (or maybe it is?)
>    Just wanted to provide some feedback. It's fantastic that OE has the
>    ability to package node software, and despite the teething
>    difficulties, I've appreciated its availability!

That is a good point, didn't think about it tbh. in npm.bbclass we
could maybe delete everything that looks like a patch before
compilation, little bit worried there might be nasty side effects but
I can try :)

Thanks for the comments!
Brendan

Re: npm.bbclass

[Paul Eggleton]

Hey Chris, long time!

On Wed, 17 Aug 2016 11:11:13 Brendan Le Foll wrote:
> >    - Is there a reason to split the package like it does? Node projects
> >    tend to have huge dependency trees, it makes updating and distributing
> >    node-based applications a bit of a chore if they end up split into 20
> >    packages, most of which have no use separately. It would be great if
> >    there was at least a way to disable this.
> 
> Paul added this, the worry was that we wanted to make sure all the
> licenses where tracked properly of the package etc...

Right, that is why we do the splitting - if you want the licenses and actual 
packages used to be fully represented in the image license manifest then they 
really do have to be split this way. Honestly I view this ugliness more as an 
artifact of how npm works (i.e. every dependency is handled separately instead 
of trying to ensure only one instance of a particular package).

> I'm not a huge fan either to be honest. Maybe we can have a npm-no-
> split.bbclass would that be ok - Paul? It's in python
> populate_packages_prepend in npm.bbclass.

Sure, but rather than a separate class I would just add a variable such as 
NPM_SPLIT_PACKAGES that defaults to "1" and then you can set it to "0" either 
globally or in each recipe.

> >    - Any patches end up getting packaged because they get put in the
> >    srcdir. I'm guessing this isn't intentional (or maybe it is?)
> >    Just wanted to provide some feedback. It's fantastic that OE has the
> >    ability to package node software, and despite the teething
> >    difficulties, I've appreciated its availability!
> 
> That is a good point, didn't think about it tbh. in npm.bbclass we
> could maybe delete everything that looks like a patch before
> compilation, little bit worried there might be nasty side effects but
> I can try :)

This is definitely not intentional, but the way we have been dealing with npm 
it makes it easy for stuff like this to leak into the output packages. I guess 
we just need to delete them under ${D} after the files are installed there.

Cheers,
Paul

Re: npm.bbclass

[Christopher Lord]

[-- Attachment #1: Type: text/plain, Size: 3457 bytes --]

On 17 August 2016 at 13:40, Paul Eggleton <paul.eggleton@intel.com> wrote:

> Hey Chris, long time!
>

Hey :)


> On Wed, 17 Aug 2016 11:11:13 Brendan Le Foll wrote:
> > >    - Is there a reason to split the package like it does? Node projects
> > >    tend to have huge dependency trees, it makes updating and
> distributing
> > >    node-based applications a bit of a chore if they end up split into
> 20
> > >    packages, most of which have no use separately. It would be great if
> > >    there was at least a way to disable this.
> >
> > Paul added this, the worry was that we wanted to make sure all the
> > licenses where tracked properly of the package etc...
>
> Right, that is why we do the splitting - if you want the licenses and
> actual
> packages used to be fully represented in the image license manifest then
> they
> really do have to be split this way. Honestly I view this ugliness more as
> an
> artifact of how npm works (i.e. every dependency is handled separately
> instead
> of trying to ensure only one instance of a particular package).
>
> > I'm not a huge fan either to be honest. Maybe we can have a npm-no-
> > split.bbclass would that be ok - Paul? It's in python
> > populate_packages_prepend in npm.bbclass.
>
> Sure, but rather than a separate class I would just add a variable such as
> NPM_SPLIT_PACKAGES that defaults to "1" and then you can set it to "0"
> either
> globally or in each recipe.
>

This sounds good. A further bug I just encountered (after finally getting a
particularly gnarly project packaged), the RDEPENDS that gets generated
doesn't seem to be complete. Dependencies of dependencies don't get added
it appears. So for example, you package appA that has a dependency on
moduleX, which itself has a dependency on moduleY. It all gets packaged,
but moduleY is missed from the RDEPENDS of appA (or of moduleX, which I
guess is where it would be ideally). At least, I've included appA in an
image, and appA is installed, moduleX is installed, but moduleY is missing.
I've assumed this is the error, but I could be assuming incorrectly.


> > >    - Any patches end up getting packaged because they get put in the
> > >    srcdir. I'm guessing this isn't intentional (or maybe it is?)
> > >    Just wanted to provide some feedback. It's fantastic that OE has the
> > >    ability to package node software, and despite the teething
> > >    difficulties, I've appreciated its availability!
> >
> > That is a good point, didn't think about it tbh. in npm.bbclass we
> > could maybe delete everything that looks like a patch before
> > compilation, little bit worried there might be nasty side effects but
> > I can try :)
>
> This is definitely not intentional, but the way we have been dealing with
> npm
> it makes it easy for stuff like this to leak into the output packages. I
> guess
> we just need to delete them under ${D} after the files are installed there.
>

On further thought about this, I actually don't think it's a terrible idea
to do this anyway. Unless your patches are huge, it's not much overhead,
and it's nice to include to hint at anyone inspecting the filesystem that
the distributed files have been modified. It'd definitely be better to
package them than to delete them and possibly end up deleting things that
shouldn't be deleted. Maybe another thing that could be made optional?

Cheers,

--Chris

[-- Attachment #2: Type: text/html, Size: 4482 bytes --]