[Buildroot] [Bug 12426] New: NodeJS slow startup performance (~30s)

[Buildroot] [Bug 12426] New: NodeJS slow startup performance (~30s)

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=12426

            Bug ID: 12426
           Summary: NodeJS slow startup performance (~30s)
           Product: buildroot
           Version: 2019.11
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: Other
          Assignee: unassigned at buildroot.uclibc.org
          Reporter: erichiggins at gmail.com
                CC: buildroot at uclibc.org
  Target Milestone: ---

The performance of the NodeJS executable, as compiled by Buildroot, has
exceptionally poor startup performance, taking around 30s to run.

The prebuilt binaries from NodeJS running on the same system take less than 2s
to run.

I created a Gist with a comparison using the time command:
https://gist.github.com/erichiggins/ac139ebe34b8d3c6dbb68f3b09ca9008

In both cases, the runtime performance is about equal.

The prebuilt binaries require the buildroot system to use glibc and will not
run if a more space-efficient alternative (uClibc-ng) is selected.

Further, NodeJS v12.x (now the major version used by Buildroot) does not
officially support ARM v6l and is considered experimental.
https://github.com/nodejs/build/issues/1677#issuecomment-486204349

Current workaround is to use glibc and overwrite the NodeJS executables with
the unofficial prebuilts via rootfs overlay.
https://github.com/nodejs/unofficial-builds/

It would be quite helpful to build a performant NodeJS executable without
requiring these workarounds.

Thank you!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 12426] NodeJS slow startup performance (~30s)

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=12426

--- Comment #1 from Thomas Petazzoni <thomas.petazzoni@free-electrons.com> ---
Are you building Buildroot with the default of BR2_OPTIMIZE_S=y (which implies
using the -Os gcc optimization flag), or using BR2_OPTIMIZE_2=y or
BR2_OPTIMIZE_3=y ?

If you're using BR2_OPTIMIZE_S=y, could you try BR2_OPTIMIZE_2=y and
BR2_OPTIMIZE_3=y and see how much it changes things ?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 12426] NodeJS slow startup performance (~30s)

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=12426

--- Comment #2 from Eric Higgins <erichiggins@gmail.com> ---
Yes, I'm using BR2_OPTIMIZE_S=y but I can try BR2_OPTIMIZE_2=y and report back.

I emailed the listed developers for package/nodejs and Daniel emailed back with
a few suggestions specific to that package's build flags, though they were not
successful for me and it started to get a bit more complex than I could commit
to (hence this ticket).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 12426] NodeJS slow startup performance (~30s)

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=12426

--- Comment #3 from D. Price <daniel.price+buildrootbz@gmail.com> ---

Hi Thomas!  Long time since I've helped out with Buildroot but happy to try to
be of assistance.

Eric indicated to me that this largely a startup issue, and not one of node's
actual performance.

One of node's performance tricks is to take a snapshot of its V8 interpreter
"world" and store that as a snapshot.  This is accomplished using the
mksnapshot command during the build.  This has been tricky for them to get
right when cross-compiling.

Eric tried removing the --without-snapshot flag from the build but then the
build failed trying to run mksnapshot (I think because it was running the
target command, not the host version).

Eric Said:
> Maybe there's another problem in my config, but having removed that flag from nodejs.mk, the build failed.
> I reverted my changes to .config, ran make clean and removed the contents from dl/ and ~/.buildroot-ccache before building.
> 
>  touch 5e8f1fa6be42bf232d2ec48bae2a475b98d6f2f6.intermediate
>  LD_LIBRARY_PATH=/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/lib.host:/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/obj.target/v8_snapshot/geni; "/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/mksnapshot" --turbo_instruction_scheduling "--target_os=linux" "--target_arch=arm" --startup_src "/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/obj.target/v8_snapshot/geni/snapshot.cc" --embedded_variant Default --embedded_src "/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/obj.target/v8_snapshot/geni/embedded.S" --no-native-code-counters
> /lib/ld-uClibc.so.0: No such file or directory
> tools/v8_gypfiles/v8_snapshot.target.mk:16: recipe for target 
> '5e8f1fa6be42bf232d2ec48bae2a475b98d6f2f6.intermediate' failed
> make[3]: *** [5e8f1fa6be42bf232d2ec48bae2a475b98d6f2f6.intermediate] Error 255
> ...
> Makefile:101: recipe for target 'node' failed
> make[2]: *** [node] Error 2

I said:

> I haven't worked with this, but to me this looks like the mksnapshot command
> (which has been cross-compiled for you) is now failing to run
> (/home/vagrant/buildroot/output/build/nodejs-12.13.0/out/Release/mksnapshot).
> 
> I found some github bug reports related to this but nothing which really is
> a definitive guide.  I also found this, which might help: 
> https://chrislea.com/2018/08/20/cross-compiling-node-js-for-arm-on-ubuntu/ .
> Several of the sources I found talk about setting CC_host and CXX_host, which
> to the best of my recollection the buildroot recipe does not do.  IIRC
> buildroot uses HOSTCC and HOSTCXX instead
> (see https://buildroot.org/downloads/manual/manual.html, 8.6).  So maybe that
> is important?  I don't know.  Somewhere I also found a thing which said
> that you have to be careful exactly how you set CC_host and CXX_host, to make
> sure -m32 was passed.  I dunno.  
> Buried in https://github.com/nodejs/build/issues/266 is the statement
> 
> "I believe I have this working now and will be able to re-enable snapshots
> when nodejs/node#4117 lands. V8 can handle cross compiling of snapshots if
> the {CC,CXX}_host variables are defined, by compiling the mksnapshot
> executable with the host compiler."
> 
> Which seems useful.

So that is my recommendation:
- Double check that it's strictly a startup issue
- Try to enable the snapshot feature

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 12426] NodeJS slow startup performance (~30s)

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=12426

--- Comment #4 from Eric Higgins <erichiggins@gmail.com> ---
Using BR2_OPTIMIZE_2=y didn't make any discernible difference.

time node --help
...
real    0m 27.46s
user    0m 22.92s
sys     0m 1.90s

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 12426] NodeJS slow startup performance (~30s)

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=12426

--- Comment #5 from Peter Korsgaard <jacmet@uclibc.org> ---
Is this only at first startup or does running nodejs multiple times each time
take ~30s on the RPI? What storage are you using? The prebuilt nodejs
presumably bundles a bunch of dependencies statically, whereas the Buildroot
one uses dynamic linking

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 12426] NodeJS slow startup performance (~30s)

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=12426

--- Comment #6 from Eric Higgins <erichiggins@gmail.com> ---
All subsequent runs of nodejs are just as slow to start.

Storage is a class 10 microSD.

Actually, the prebuilt nodejs binaries are also dynamically linked. I found
this out the hard way because I was using uClibc-ng and the prebuilt expects
glibc.


(prebuilt)
# file /usr/local/lib/nodejs/node-v10.18.0-linux-armv6l/bin/node

/usr/local/lib/nodejs/node-v10.18.0-linux-armv6l/bin/node: ELF 32-bit LSB
executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter
/lib/ld-linux-armhf.so.3, for GNU/Linux 3.16.42, with debug_info, not stripped


(compiled by buildroot)
# file /usr/bin/node 

/usr/bin/node: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV),
dynamically linked, interpreter /lib/ld-uClibc.so.0, stripped


For what it's worth, the flags they use for armv6l are available here:

https://github.com/nodejs/unofficial-builds/tree/master/recipes/armv6l

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 12426] NodeJS slow startup performance (~30s)

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=12426

--- Comment #7 from Peter Korsgaard <jacmet@uclibc.org> ---
(In reply to Eric Higgins from comment #6)

Yes, both builds dynamically link to libc, but the unofficial builds probably
statically link to openssl / libuv / zlib / ..

Thanks for the link to their Dockerfile, but I don't seem to find where they
run ./configure?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 12426] NodeJS slow startup performance (~30s)

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=12426

--- Comment #8 from Eric Higgins <erichiggins@gmail.com> ---
I don't see them running ./configure either. My guess is that they are passing
in flags and options as env variables instead?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 12426] NodeJS slow startup performance (~30s)

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=12426

--- Comment #9 from Eric Higgins <erichiggins@gmail.com> ---
Looks like ./configure is called a few times within the Makefile


https://github.com/nodejs/node/blob/v12.14.0/Makefile

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 12426] NodeJS slow startup performance (~30s)

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=12426

Yann E. MORIN <yann.morin.1998@free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |yann.morin.1998@free.fr
         Resolution|---                         |MOVED
             Status|NEW                         |RESOLVED

--- Comment #10 from Yann E. MORIN <yann.morin.1998@free.fr> ---
Thank you for your report.

The issue tracker for the Buildroot project has been moved to
the Gitlab.com issue tracker:
    https://gitlab.com/buildroot.org/buildroot/-/issues

We are taking this opportunity to close old issues in this old
tracker. If you believe your issue is still relevant, please
open one in the new issue tracker.

Thank you!

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot