[Buildroot] [PATCH 1/1] support/scripts: add node modules to check-bin-arch's IGNORES list

[Buildroot] [PATCH 1/1] support/scripts: add node modules to check-bin-arch's IGNORES list

[Valerio Crini]

From: Valerio Crini <valerio.crini@docomodigital.com>

Some node modules as stf ship compiled binaries causing check-bin-arch script
to return errors.

Signed-off-by: Valerio Crini <valerio.crini@docomodigital.com>
---
 support/scripts/check-bin-arch | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/support/scripts/check-bin-arch b/support/scripts/check-bin-arch
index 3449bd1aeb..a276b75b92 100755
--- a/support/scripts/check-bin-arch
+++ b/support/scripts/check-bin-arch
@@ -25,6 +25,11 @@ declare -a IGNORES=(
 	# it for a different architecture (e.g. i386 grub on x86_64).
 	"/lib/grub"
 	"/usr/lib/grub"
+
+	# Skip files in /usr/lib/node_modules, since it is possible to have it
+	# for a different architecture (e.g. stf module has some packages already
+	# compiled for several architectures).
+	"/usr/lib/node_modules"
 )
 
 while getopts p:l:r:a:i: OPT ; do
-- 
2.20.1

[Buildroot] [PATCH 1/1] support/scripts: add node modules to check-bin-arch's IGNORES list

[Thomas Petazzoni]

Hello Valerio,

+Martin Bark in Cc.

On Fri, 31 May 2019 09:58:00 +0200
Valerio Crini <vcrini@gmail.com> wrote:

> From: Valerio Crini <valerio.crini@docomodigital.com>
> 
> Some node modules as stf ship compiled binaries causing check-bin-arch script
> to return errors.
> 
> Signed-off-by: Valerio Crini <valerio.crini@docomodigital.com>
> ---
>  support/scripts/check-bin-arch | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/support/scripts/check-bin-arch b/support/scripts/check-bin-arch
> index 3449bd1aeb..a276b75b92 100755
> --- a/support/scripts/check-bin-arch
> +++ b/support/scripts/check-bin-arch
> @@ -25,6 +25,11 @@ declare -a IGNORES=(
>  	# it for a different architecture (e.g. i386 grub on x86_64).
>  	"/lib/grub"
>  	"/usr/lib/grub"
> +
> +	# Skip files in /usr/lib/node_modules, since it is possible to have it
> +	# for a different architecture (e.g. stf module has some packages already
> +	# compiled for several architectures).
> +	"/usr/lib/node_modules"

Thanks for providing this patch. Could you give a bit more details
about which libraries get installed, what is their name, etc. ?

Indeed, rather than ignoring files from /usr/lib/node_modules, it would
be a lot better to remove the libraries that are anyway not useful
because they don't match the CPU architecture of the target. This would
both reduce the filesystem size *and* solve the check-bin-arch issue.

If really this isn't possible, then we could define
NODEJS_BIN_ARCH_EXCLUDE in nodejs.mk, to exclude /usr/lib/node_modules/
from the check-bin-arch verification. It's not as good as preventing
the problem, but it's better than a global exclusion.

In order to investigate this, I tried to do a build with the stf NPM
package, but it failed on bufferutil. I tried the following Buildroot
defconfig:

BR2_arm=y
BR2_cortex_a8=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_LINARO_ARM=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set
BR2_PACKAGE_NODEJS=y
BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="stf"
# BR2_TARGET_ROOTFS_TAR is not set

And it resulted in this:

[...]
.../home/thomas/projets/buildroot/output/host/bin/npm install -g stf
[...]
> node-gyp rebuild || node suppress-error.js

  TOUCH Release/obj.target/DTraceProviderStub.stamp

> bufferutil at 1.3.0 install /home/thomas/projets/buildroot/output/target/usr/lib/node_modules/stf/node_modules/bufferutil
> node-gyp rebuild

  CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
In file included from ../../nan/nan.h:190:0,
                 from ../src/bufferutil.cc:16:
../../nan/nan_maybe_43_inl.h: In function ?Nan::Maybe<bool> Nan::ForceSet(v8::Local<v8::Object>, v8::Local<v8::Value>, v8::Local<v8::Value>, v8::PropertyAttribute)?:
../../nan/nan_maybe_43_inl.h:88:15: error: ?class v8::Object? has no member named ?ForceSet?
   return obj->ForceSet(GetCurrentContext(), key, value, attribs);
               ^~~~~~~~

Do you know what is going on ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] support/scripts: add node modules to check-bin-arch's IGNORES list

[Martin Bark]

Hi,

On Sun, 4 Aug 2019 at 16:52, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Valerio,
>
> +Martin Bark in Cc.
>
> On Fri, 31 May 2019 09:58:00 +0200
> Valerio Crini <vcrini@gmail.com> wrote:
>
> > From: Valerio Crini <valerio.crini@docomodigital.com>
> >
> > Some node modules as stf ship compiled binaries causing check-bin-arch script
> > to return errors.
> >
> > Signed-off-by: Valerio Crini <valerio.crini@docomodigital.com>
> > ---
> >  support/scripts/check-bin-arch | 5 +++++
> >  1 file changed, 5 insertions(+)
> >
> > diff --git a/support/scripts/check-bin-arch b/support/scripts/check-bin-arch
> > index 3449bd1aeb..a276b75b92 100755
> > --- a/support/scripts/check-bin-arch
> > +++ b/support/scripts/check-bin-arch
> > @@ -25,6 +25,11 @@ declare -a IGNORES=(
> >       # it for a different architecture (e.g. i386 grub on x86_64).
> >       "/lib/grub"
> >       "/usr/lib/grub"
> > +
> > +     # Skip files in /usr/lib/node_modules, since it is possible to have it
> > +     # for a different architecture (e.g. stf module has some packages already
> > +     # compiled for several architectures).
> > +     "/usr/lib/node_modules"
>
> Thanks for providing this patch. Could you give a bit more details
> about which libraries get installed, what is their name, etc. ?
>
> Indeed, rather than ignoring files from /usr/lib/node_modules, it would
> be a lot better to remove the libraries that are anyway not useful
> because they don't match the CPU architecture of the target. This would
> both reduce the filesystem size *and* solve the check-bin-arch issue.

Yes i agree Thomas, this is the correct solution.  I would recommend
you create a custom package for your application and in the BUILD_CMDS
use $(NPM) install to install you node_modules followed by code to
filter out any files that do not match your CPU architecture.  Here is
a script i've used before to do this

# Remove executables which are not the correct target arch
for f in $(find $(@D) -executable -type f) ; do
    a=$(${TARGET_READELF} -h $f 2>&1 | sed -r -e '/^  Machine:
+(.+)/!d; s//\1/;')
    if [ ! -z "$a" -a "$a" != "${BR2_READELF_ARCH_NAME}" ] ; then
        echo "Remove: $f Arch: $a"
        rm -f $f
    fi
done

Thanks

Martin

>
> If really this isn't possible, then we could define
> NODEJS_BIN_ARCH_EXCLUDE in nodejs.mk, to exclude /usr/lib/node_modules/
> from the check-bin-arch verification. It's not as good as preventing
> the problem, but it's better than a global exclusion.
>
> In order to investigate this, I tried to do a build with the stf NPM
> package, but it failed on bufferutil. I tried the following Buildroot
> defconfig:
>
> BR2_arm=y
> BR2_cortex_a8=y
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_TOOLCHAIN_EXTERNAL_LINARO_ARM=y
> BR2_INIT_NONE=y
> BR2_SYSTEM_BIN_SH_NONE=y
> # BR2_PACKAGE_BUSYBOX is not set
> BR2_PACKAGE_NODEJS=y
> BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="stf"
> # BR2_TARGET_ROOTFS_TAR is not set
>
> And it resulted in this:
>
> [...]
> .../home/thomas/projets/buildroot/output/host/bin/npm install -g stf
> [...]
> > node-gyp rebuild || node suppress-error.js
>
>   TOUCH Release/obj.target/DTraceProviderStub.stamp
>
> > bufferutil at 1.3.0 install /home/thomas/projets/buildroot/output/target/usr/lib/node_modules/stf/node_modules/bufferutil
> > node-gyp rebuild
>
>   CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
> In file included from ../../nan/nan.h:190:0,
>                  from ../src/bufferutil.cc:16:
> ../../nan/nan_maybe_43_inl.h: In function ?Nan::Maybe<bool> Nan::ForceSet(v8::Local<v8::Object>, v8::Local<v8::Value>, v8::Local<v8::Value>, v8::PropertyAttribute)?:
> ../../nan/nan_maybe_43_inl.h:88:15: error: ?class v8::Object? has no member named ?ForceSet?
>    return obj->ForceSet(GetCurrentContext(), key, value, attribs);
>                ^~~~~~~~
>
> Do you know what is going on ?
>
> Thanks!
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

[Buildroot] [PATCH 1/1] support/scripts: add node modules to check-bin-arch's IGNORES list

[Yegor Yefremov]

Hi all,

On Sat, Aug 10, 2019 at 5:44 PM Martin Bark <martin@barkynet.com> wrote:
>
> Hi,
>
> On Sun, 4 Aug 2019 at 16:52, Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> >
> > Hello Valerio,
> >
> > +Martin Bark in Cc.
> >
> > On Fri, 31 May 2019 09:58:00 +0200
> > Valerio Crini <vcrini@gmail.com> wrote:
> >
> > > From: Valerio Crini <valerio.crini@docomodigital.com>
> > >
> > > Some node modules as stf ship compiled binaries causing check-bin-arch script
> > > to return errors.
> > >
> > > Signed-off-by: Valerio Crini <valerio.crini@docomodigital.com>
> > > ---
> > >  support/scripts/check-bin-arch | 5 +++++
> > >  1 file changed, 5 insertions(+)
> > >
> > > diff --git a/support/scripts/check-bin-arch b/support/scripts/check-bin-arch
> > > index 3449bd1aeb..a276b75b92 100755
> > > --- a/support/scripts/check-bin-arch
> > > +++ b/support/scripts/check-bin-arch
> > > @@ -25,6 +25,11 @@ declare -a IGNORES=(
> > >       # it for a different architecture (e.g. i386 grub on x86_64).
> > >       "/lib/grub"
> > >       "/usr/lib/grub"
> > > +
> > > +     # Skip files in /usr/lib/node_modules, since it is possible to have it
> > > +     # for a different architecture (e.g. stf module has some packages already
> > > +     # compiled for several architectures).
> > > +     "/usr/lib/node_modules"
> >
> > Thanks for providing this patch. Could you give a bit more details
> > about which libraries get installed, what is their name, etc. ?
> >
> > Indeed, rather than ignoring files from /usr/lib/node_modules, it would
> > be a lot better to remove the libraries that are anyway not useful
> > because they don't match the CPU architecture of the target. This would
> > both reduce the filesystem size *and* solve the check-bin-arch issue.
>
> Yes i agree Thomas, this is the correct solution.  I would recommend
> you create a custom package for your application and in the BUILD_CMDS
> use $(NPM) install to install you node_modules followed by code to
> filter out any files that do not match your CPU architecture.  Here is
> a script i've used before to do this
>
> # Remove executables which are not the correct target arch
> for f in $(find $(@D) -executable -type f) ; do
>     a=$(${TARGET_READELF} -h $f 2>&1 | sed -r -e '/^  Machine:
> +(.+)/!d; s//\1/;')
>     if [ ! -z "$a" -a "$a" != "${BR2_READELF_ARCH_NAME}" ] ; then
>         echo "Remove: $f Arch: $a"
>         rm -f $f
>     fi
> done

Any update on this issue?

I cannot reproduce this issue when installing ref-napi or ffi-napi via
BR Node.js BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL option and "make
nodejs-rebuild".

This issue comes only when I install my own package libonrisc [1] that
uses npm to install the module having these dependencies.Then I get
the following errors:

ERROR: architecture for
"/usr/lib/node_modules/libonrisc/node_modules/ffi-napi/node_modules/ref-napi/prebuilds/linux-arm64/electron.napi.armv8.node"
is "AArch64", should be "ARM"
ERROR: architecture for
"/usr/lib/node_modules/libonrisc/node_modules/ref-napi/prebuilds/linux-arm64/electron.napi.armv8.node"
is "AArch64", should be "ARM"
ERROR: architecture for
"/usr/lib/node_modules/libonrisc/node_modules/ref-array-napi/node_modules/ref-napi/prebuilds/linux-x64/electron.napi.node"
is "Advanced Micro Devices X86-64", should be "ARM"
ERROR: architecture for
"/usr/lib/node_modules/libonrisc/node_modules/ref-struct-napi/node_modules/ref-napi/prebuilds/linux-x64/electron.napi.node"
is "Advanced Micro Devices X86-64", should be "ARM"
ERROR: architecture for
"/usr/lib/node_modules/libonrisc/node_modules/ffi-napi/node_modules/ref-napi/prebuilds/linux-x64/electron.napi.node"
is "Advanced Micro Devices X86-64", should be "ARM"

[1] https://github.com/visionsystemsgmbh/onrisc_br_bsp/blob/master/package/libonrisc/libonrisc.mk#L23

Best regards,
Yegor

> >
> > If really this isn't possible, then we could define
> > NODEJS_BIN_ARCH_EXCLUDE in nodejs.mk, to exclude /usr/lib/node_modules/
> > from the check-bin-arch verification. It's not as good as preventing
> > the problem, but it's better than a global exclusion.
> >
> > In order to investigate this, I tried to do a build with the stf NPM
> > package, but it failed on bufferutil. I tried the following Buildroot
> > defconfig:
> >
> > BR2_arm=y
> > BR2_cortex_a8=y
> > BR2_TOOLCHAIN_EXTERNAL=y
> > BR2_TOOLCHAIN_EXTERNAL_LINARO_ARM=y
> > BR2_INIT_NONE=y
> > BR2_SYSTEM_BIN_SH_NONE=y
> > # BR2_PACKAGE_BUSYBOX is not set
> > BR2_PACKAGE_NODEJS=y
> > BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="stf"
> > # BR2_TARGET_ROOTFS_TAR is not set
> >
> > And it resulted in this:
> >
> > [...]
> > .../home/thomas/projets/buildroot/output/host/bin/npm install -g stf
> > [...]
> > > node-gyp rebuild || node suppress-error.js
> >
> >   TOUCH Release/obj.target/DTraceProviderStub.stamp
> >
> > > bufferutil at 1.3.0 install /home/thomas/projets/buildroot/output/target/usr/lib/node_modules/stf/node_modules/bufferutil
> > > node-gyp rebuild
> >
> >   CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
> > In file included from ../../nan/nan.h:190:0,
> >                  from ../src/bufferutil.cc:16:
> > ../../nan/nan_maybe_43_inl.h: In function ?Nan::Maybe<bool> Nan::ForceSet(v8::Local<v8::Object>, v8::Local<v8::Value>, v8::Local<v8::Value>, v8::PropertyAttribute)?:
> > ../../nan/nan_maybe_43_inl.h:88:15: error: ?class v8::Object? has no member named ?ForceSet?
> >    return obj->ForceSet(GetCurrentContext(), key, value, attribs);
> >                ^~~~~~~~
> >
> > Do you know what is going on ?
> >
> > Thanks!
> >
> > Thomas
> > --
> > Thomas Petazzoni, CTO, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/scripts: add node modules to check-bin-arch's IGNORES list

[Yegor Yefremov via buildroot]

Hi All,

On Wed, Dec 9, 2020 at 7:35 AM Valerio Crini <vcrini@gmail.com> wrote:
>
> Hi Yegor,
> Thanks for trying to fix the issue.
> Unfortunately I do not work for the company that gave me the opportunity to use the code that gave me the original error. So I am not able  to reproduce the issue anymore.
>
>
> Il mar 8 dic 2020, 16:36 Yegor Yefremov <yegorslists@googlemail.com> ha scritto:
>>
>> Hi all,
>>
>> On Sat, Aug 10, 2019 at 5:44 PM Martin Bark <martin@barkynet.com> wrote:
>> >
>> > Hi,
>> >
>> > On Sun, 4 Aug 2019 at 16:52, Thomas Petazzoni
>> > <thomas.petazzoni@bootlin.com> wrote:
>> > >
>> > > Hello Valerio,
>> > >
>> > > +Martin Bark in Cc.
>> > >
>> > > On Fri, 31 May 2019 09:58:00 +0200
>> > > Valerio Crini <vcrini@gmail.com> wrote:
>> > >
>> > > > From: Valerio Crini <valerio.crini@docomodigital.com>
>> > > >
>> > > > Some node modules as stf ship compiled binaries causing check-bin-arch script
>> > > > to return errors.
>> > > >
>> > > > Signed-off-by: Valerio Crini <valerio.crini@docomodigital.com>
>> > > > ---
>> > > >  support/scripts/check-bin-arch | 5 +++++
>> > > >  1 file changed, 5 insertions(+)
>> > > >
>> > > > diff --git a/support/scripts/check-bin-arch b/support/scripts/check-bin-arch
>> > > > index 3449bd1aeb..a276b75b92 100755
>> > > > --- a/support/scripts/check-bin-arch
>> > > > +++ b/support/scripts/check-bin-arch
>> > > > @@ -25,6 +25,11 @@ declare -a IGNORES=(
>> > > >       # it for a different architecture (e.g. i386 grub on x86_64).
>> > > >       "/lib/grub"
>> > > >       "/usr/lib/grub"
>> > > > +
>> > > > +     # Skip files in /usr/lib/node_modules, since it is possible to have it
>> > > > +     # for a different architecture (e.g. stf module has some packages already
>> > > > +     # compiled for several architectures).
>> > > > +     "/usr/lib/node_modules"
>> > >
>> > > Thanks for providing this patch. Could you give a bit more details
>> > > about which libraries get installed, what is their name, etc. ?
>> > >
>> > > Indeed, rather than ignoring files from /usr/lib/node_modules, it would
>> > > be a lot better to remove the libraries that are anyway not useful
>> > > because they don't match the CPU architecture of the target. This would
>> > > both reduce the filesystem size *and* solve the check-bin-arch issue.
>> >
>> > Yes i agree Thomas, this is the correct solution.  I would recommend
>> > you create a custom package for your application and in the BUILD_CMDS
>> > use $(NPM) install to install you node_modules followed by code to
>> > filter out any files that do not match your CPU architecture.  Here is
>> > a script i've used before to do this
>> >
>> > # Remove executables which are not the correct target arch
>> > for f in $(find $(@D) -executable -type f) ; do
>> >     a=$(${TARGET_READELF} -h $f 2>&1 | sed -r -e '/^  Machine:
>> > +(.+)/!d; s//\1/;')
>> >     if [ ! -z "$a" -a "$a" != "${BR2_READELF_ARCH_NAME}" ] ; then
>> >         echo "Remove: $f Arch: $a"
>> >         rm -f $f
>> >     fi
>> > done

I now also have two cases where I get precompiled Node.js module binaries:

- my own BR package
- when installing the modules via Node.js config option

Where should Martin's script reside? In a script before
support/scripts/check-bin-arch and handling only /usr/lib/node_modules
or inside support/scripts/check-bin-arch and handing all folders that
are not in the IGNORE list?

Best regards,
Yegor
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot