npm.bbclass support for deep native modules?

npm.bbclass support for deep native modules?

[Peter A. Bigot]

I'm using the current head of morty and trying to get a handle on the
new nodejs support in OE.

I'm failing to build a recipe for statsd.  Starting with this:

     devtool add 'npm://registry.npmjs.org;name=statsd;version=0.8.0'
     bitbake statsd

produces an error related to the modern-syslog dependency:

  DEBUG: Executing shell function do_compile
| npm ERR! Linux 4.4.0-47-generic
| npm ERR! argv 
"/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin/node" 
"/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin/npm" 
"--arch=arm" "--target_arch=arm" "--production" "--no-registry" "install"
| npm ERR! node v4.6.1
| npm ERR! npm  v2.15.9
|
| npm ERR! Registry not defined and registry files not found: 
"/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/statsd/0.8.0-r0/npm_cache/noregistry/modern-syslog/.cache.json", 
"/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/statsd/0.8.0-r0/npm_cache/modern-syslog/.cache.json".

modern-syslog 1.1.2 needs node-gyp to build a native component and
https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM notes that devtool
can't detect such things.  Doing this works fine to build that package:

     devtool add 'npm://registry.npmjs.org;name=modern-syslog;version=1.1.2'
     bitbake modern-syslog

but I'm having no luck getting "bitbake statsd" to find the result.
I've added:

     DEPENDS = "modern-syslog"

to statsd_0.8.0.bb but that isn't helping.  It looks like I need some
way to have the recipe install the prepared modern-syslog into the cache
(or globally?) before baking statsd, but since the cache gets cleared in
npm_do_compile() it's not clear how to make that happen.

I'm very rusty with OE (two years away), so am I missing something or is
this just beyond what the bitbake infrastructure can currently handle?
If so, can somebody suggest a way to hand-patch the recipe, or outline
how npm.bbclass might be extended to support this?

Thanks.

Peter

Re: npm.bbclass support for deep native modules?

[Paul Eggleton]

Hi Peter,

On Sat, 26 Nov 2016 18:17:48 Peter A. Bigot wrote:
> I'm using the current head of morty and trying to get a handle on the
> new nodejs support in OE.
> 
> I'm failing to build a recipe for statsd.  Starting with this:
> 
>      devtool add 'npm://registry.npmjs.org;name=statsd;version=0.8.0'
>      bitbake statsd
> 
> produces an error related to the modern-syslog dependency:
> 
>   DEBUG: Executing shell function do_compile
> 
> | npm ERR! Linux 4.4.0-47-generic
> | npm ERR! argv
> 
> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin/
> node"
> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin
> /npm" "--arch=arm" "--target_arch=arm" "--production" "--no-registry"
> "install"
> | npm ERR! node v4.6.1
> | npm ERR! npm  v2.15.9
> 
> | npm ERR! Registry not defined and registry files not found:
> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-linu
> x-gnueabi/statsd/0.8.0-r0/npm_cache/noregistry/modern-syslog/.cache.json",
> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-lin
> ux-gnueabi/statsd/0.8.0-r0/npm_cache/modern-syslog/.cache.json".
> 
> modern-syslog 1.1.2 needs node-gyp to build a native component and
> https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM notes that devtool
> can't detect such things.  Doing this works fine to build that package:
> 
>      devtool add 'npm://registry.npmjs.org;name=modern-syslog;version=1.1.2'
> bitbake modern-syslog
> 
> but I'm having no luck getting "bitbake statsd" to find the result.
> I've added:
> 
>      DEPENDS = "modern-syslog"
> 
> to statsd_0.8.0.bb but that isn't helping.  It looks like I need some
> way to have the recipe install the prepared modern-syslog into the cache
> (or globally?) before baking statsd, but since the cache gets cleared in
> npm_do_compile() it's not clear how to make that happen.
> 
> I'm very rusty with OE (two years away), so am I missing something or is
> this just beyond what the bitbake infrastructure can currently handle?
> If so, can somebody suggest a way to hand-patch the recipe, or outline
> how npm.bbclass might be extended to support this?

Disclaimer - I'm the one who has been doing most of the recent work with npm 
support (aside from the node.js recipe and the original npm fetcher plugin, 
which were the work of others) however my knowledge of node.js is pretty 
limited - most of it has been picked up along the way. So unfortunately I 
can't immediately see why this isn't working. The thing that puzzles me in 
particular about the error you're seeing though is that we're explicitly 
telling npm not to look for a registry, so why is it complaining about the 
lack of a registry?

Henry / Brendan - any suggestions?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: npm.bbclass support for deep native modules?

[Peter A. Bigot]

On 11/28/2016 04:11 AM, Paul Eggleton wrote:
> Hi Peter,
>
> On Sat, 26 Nov 2016 18:17:48 Peter A. Bigot wrote:
>> I'm using the current head of morty and trying to get a handle on the
>> new nodejs support in OE.
>>
>> I'm failing to build a recipe for statsd.  Starting with this:
>>
>>       devtool add 'npm://registry.npmjs.org;name=statsd;version=0.8.0'
>>       bitbake statsd
>>
>> produces an error related to the modern-syslog dependency:
>>
>>    DEBUG: Executing shell function do_compile
>>
>> | npm ERR! Linux 4.4.0-47-generic
>> | npm ERR! argv
>>
>> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin/
>> node"
>> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin
>> /npm" "--arch=arm" "--target_arch=arm" "--production" "--no-registry"
>> "install"
>> | npm ERR! node v4.6.1
>> | npm ERR! npm  v2.15.9
>>
>> | npm ERR! Registry not defined and registry files not found:
>> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-linu
>> x-gnueabi/statsd/0.8.0-r0/npm_cache/noregistry/modern-syslog/.cache.json",
>> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-lin
>> ux-gnueabi/statsd/0.8.0-r0/npm_cache/modern-syslog/.cache.json".
>>
>> modern-syslog 1.1.2 needs node-gyp to build a native component and
>> https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM notes that devtool
>> can't detect such things.  Doing this works fine to build that package:
>>
>>       devtool add 'npm://registry.npmjs.org;name=modern-syslog;version=1.1.2'
>> bitbake modern-syslog
>>
>> but I'm having no luck getting "bitbake statsd" to find the result.
>> I've added:
>>
>>       DEPENDS = "modern-syslog"
>>
>> to statsd_0.8.0.bb but that isn't helping.  It looks like I need some
>> way to have the recipe install the prepared modern-syslog into the cache
>> (or globally?) before baking statsd, but since the cache gets cleared in
>> npm_do_compile() it's not clear how to make that happen.
>>
>> I'm very rusty with OE (two years away), so am I missing something or is
>> this just beyond what the bitbake infrastructure can currently handle?
>> If so, can somebody suggest a way to hand-patch the recipe, or outline
>> how npm.bbclass might be extended to support this?
> Disclaimer - I'm the one who has been doing most of the recent work with npm
> support (aside from the node.js recipe and the original npm fetcher plugin,
> which were the work of others) however my knowledge of node.js is pretty
> limited - most of it has been picked up along the way. So unfortunately I
> can't immediately see why this isn't working. The thing that puzzles me in
> particular about the error you're seeing though is that we're explicitly
> telling npm not to look for a registry, so why is it complaining about the
> lack of a registry?

Sorry, that wasn't clear.  statsd depends on modern-syslog but the 
lockdown and shrinkwrap files generated by devtool don't include it.  
 From the Wiki:

"Devtool cannot detect native libraries in module dependencies, you 
you'll need to manually add packages to recipe"

The Wiki doesn't go into detail of how that's supposed to be done. Is 
the existing infrastructure supposed to be able to find 
globally-installed modules?

I'm wondering whether https://yarnpkg.com/ or one of the other nodejs 
dependency managers might be an alternative, as I believe npm's approach 
to dependencies is not suited to level of lockdown needed by Yocto and 
many other production systems.

Peter

Re: npm.bbclass support for deep native modules?

[Jack Mitchell]

On 28/11/16 10:35, Peter A. Bigot wrote:
> On 11/28/2016 04:11 AM, Paul Eggleton wrote:
>> Hi Peter,
>>
>> On Sat, 26 Nov 2016 18:17:48 Peter A. Bigot wrote:
>>> I'm using the current head of morty and trying to get a handle on the
>>> new nodejs support in OE.
>>>
>>> I'm failing to build a recipe for statsd.  Starting with this:
>>>
>>>       devtool add 'npm://registry.npmjs.org;name=statsd;version=0.8.0'
>>>       bitbake statsd
>>>
>>> produces an error related to the modern-syslog dependency:
>>>
>>>    DEBUG: Executing shell function do_compile
>>>
>>> | npm ERR! Linux 4.4.0-47-generic
>>> | npm ERR! argv
>>>
>>> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin/
>>>
>>> node"
>>> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/sysroots/x86_64-linux/usr/bin
>>>
>>> /npm" "--arch=arm" "--target_arch=arm" "--production" "--no-registry"
>>> "install"
>>> | npm ERR! node v4.6.1
>>> | npm ERR! npm  v2.15.9
>>>
>>> | npm ERR! Registry not defined and registry files not found:
>>> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-linu
>>>
>>> x-gnueabi/statsd/0.8.0-r0/npm_cache/noregistry/modern-syslog/.cache.json",
>>>
>>> "/mnt/devel/oe/omap/build-bb-morty-master/tmp/work/cortexa8hf-neon-poky-lin
>>>
>>> ux-gnueabi/statsd/0.8.0-r0/npm_cache/modern-syslog/.cache.json".
>>>
>>> modern-syslog 1.1.2 needs node-gyp to build a native component and
>>> https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM notes that devtool
>>> can't detect such things.  Doing this works fine to build that package:
>>>
>>>       devtool add
>>> 'npm://registry.npmjs.org;name=modern-syslog;version=1.1.2'
>>> bitbake modern-syslog
>>>
>>> but I'm having no luck getting "bitbake statsd" to find the result.
>>> I've added:
>>>
>>>       DEPENDS = "modern-syslog"
>>>
>>> to statsd_0.8.0.bb but that isn't helping.  It looks like I need some
>>> way to have the recipe install the prepared modern-syslog into the cache
>>> (or globally?) before baking statsd, but since the cache gets cleared in
>>> npm_do_compile() it's not clear how to make that happen.
>>>
>>> I'm very rusty with OE (two years away), so am I missing something or is
>>> this just beyond what the bitbake infrastructure can currently handle?
>>> If so, can somebody suggest a way to hand-patch the recipe, or outline
>>> how npm.bbclass might be extended to support this?
>> Disclaimer - I'm the one who has been doing most of the recent work
>> with npm
>> support (aside from the node.js recipe and the original npm fetcher
>> plugin,
>> which were the work of others) however my knowledge of node.js is pretty
>> limited - most of it has been picked up along the way. So unfortunately I
>> can't immediately see why this isn't working. The thing that puzzles
>> me in
>> particular about the error you're seeing though is that we're explicitly
>> telling npm not to look for a registry, so why is it complaining about
>> the
>> lack of a registry?
>
> Sorry, that wasn't clear.  statsd depends on modern-syslog but the
> lockdown and shrinkwrap files generated by devtool don't include it.
> From the Wiki:
>
> "Devtool cannot detect native libraries in module dependencies, you
> you'll need to manually add packages to recipe"
>
> The Wiki doesn't go into detail of how that's supposed to be done. Is
> the existing infrastructure supposed to be able to find
> globally-installed modules?
>
> I'm wondering whether https://yarnpkg.com/ or one of the other nodejs
> dependency managers might be an alternative, as I believe npm's approach
> to dependencies is not suited to level of lockdown needed by Yocto and
> many other production systems.
>
> Peter

Hi Peter,

I'm in a similar boat packaging a custom project with a very large 
dependency tree. After looking at the available options and our current 
struggles with npm, yarn was our next point of call. We haven't done 
anything with it yet, but probably plan to in the near future.

Not very helpful, but just a heads up that you're not the only one 
fighting npm ;)

Cheers,
Jack.

Re: npm.bbclass support for deep native modules?

[Bruce, Henry]

On Mon, 2016-11-28 at 04:35 -0600, Peter A. Bigot wrote:

Hi Peter,

I'm helping Paul with this. Sadly, I'm not much of an npm expert
either, but want to improve support for node.js development. 

> 
> Sorry, that wasn't clear.  statsd depends on modern-syslog but the 
> lockdown and shrinkwrap files generated by devtool don't include it.
> From the Wiki:
> 
> "Devtool cannot detect native libraries in module dependencies, you 
> you'll need to manually add packages to recipe"
> 
> The Wiki doesn't go into detail of how that's supposed to be done. Is
> the existing infrastructure supposed to be able to find 
> globally-installed modules?

I believe that this refers to native code depending on another native
library (e.g. libfoo). In this case, a package for libfoo would have to
be available and added to DEPENDS. This is not applicable for modern-
syslog. Paul - correct me if I've got this wrong.

> I'm wondering whether https://yarnpkg.com/ or one of the other nodejs
> dependency managers might be an alternative, as I believe npm's
> approach to dependencies is not suited to level of lockdown needed by
> Yocto and many other production systems.

Good idea, but we need to balance the complexity of adding another tool
vs. figuring this out with npm.

Back to the problem. I have re-created the statsd build failure, and
agree with your diagnosis, but don't have am immediate solution.  

I have opened a bug #10760, and added you to CC list. Let's use this
bug to communicate from hereon.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10760


Henry