[Buildroot] NodeJS, qemu wrapper and ELF interpreter fun

[Thomas Petazzoni via buildroot <buildroot@buildroot.org>]

Hello,

I finally took some time to investigate the NodeJS build issue that
occurs on x86-64:

  http://autobuild.buildroot.net/results/4f2/4f29c5e663b5d01af20d671baf6504332b8c1f7b/build-end.log

The failure goes like this:

cd ../../tools/v8_gypfiles; python ../../deps/v8/tools/run.py ../../out/Release/v8-qemu-wrapper ../../out/Release/bytecode_builtins_list_generator ../../out/Release/obj.host/gen/generate-bytecode-output-root/builtins-generated/bytecodes-builtins-list.h
../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ../../out/Release/bytecode_builtins_list_generator)
../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ../../out/Release/bytecode_builtins_list_generator)
../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ../../out/Release/bytecode_builtins_list_generator)
../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ../../out/Release/bytecode_builtins_list_generator)
Return code is 1

This problem also has a bug report at
https://bugs.busybox.net/show_bug.cgi?id=14366.

As you probably know, v8-qemu-wrapper is a script that uses Qemu
user-mode emulation to run some programs built for the target. We are
then supposed to use the libraries built for the target, but here for
some reason, the C library of the host gets used, and it fails badly
because it is older than the C library that is available for the
target.

So the question is: why the heck does Qemu use the host C library?

To answer this question, we first have to look at how the -L option of
Qemu is implemented. This option is documented as such:

-L path              QEMU_LD_PREFIX    set the elf interpreter prefix to 'path'

And the v8-qemu-wrapper script makes this option point to
$(STAGING_DIR), so that the ELF interpreter used is the one in
$(STAGING_DIR).

However, contrary to what the option documentation says, this option
does much more than setting the ELF interpreter prefix: it is going to
affect how *all* system calls manipulating files (open, etc.) are
going to work.

When this option is passed, the function init_paths() in
https://git.qemu.org/?p=qemu.git;a=blob;f=util/path.c is called at
initialization time, and essentially its sets the global "base"
variable to point to the directory passed as -L argument.

Then, for every single syscall that manipulates a path, this path will
be passed through the path() function in the same file. This function
will first attempt to resolve the path with "base" as a prefix, and if
not, return the unprefixed path.

After adding some traces into this function, I was able to understand
what happens:

(1) -L$(STAGING_DIR) is passed, causing "base" to point to $(STAGING_DIR)

(2) The target ELF interpreter from $(STAGING_DIR) is properly invoked

(3) When this ELF interpreter then resolves the libc.so.6 library, it
    first looks for /etc/ld.so.cache.

(4) Qemu first looks for /etc/ld.so.cache with the -L prefix, i.e
    $(STAGING_DIR)/etc/ld.so.cache, but it does not exist. So, the Qemu
    system call emulation falls back to /etc/ld.so.cache, which means
    the target ELF interpreter reads the /etc/ld.so.cache of the host
    system.

(5) This /etc/ld.so.cache of the host system says that libc.so.6 is in
    /lib/x86_64-linux-gnu/

(6) The target ELF interpreter therefore tries to use
    /lib/x86_64-linux-gnu/libc.so.6. The Qemu system call emulation
    first tries $(STAGING_DIR)/lib/x86_64-linux-gnu/libc.so.6, but
    this library does not exist (it is in
    $(STAGING_DIR)/lib/libc.so.6), so the Qemu system call emulation
    falls back to /lib/x86_64-linux-gnu/libc.so.6 of the host system,
    which exist... but is too old compared to the target C library.
    Indeed, results from ld.so.cache take precedence over the simple
    resolution of library paths in /usr/lib and /lib.

We see 3 possible ideas to resolve this problem:

(A) Change the behavior of Qemu to not fallback to unprefixed paths:
    when -L is passed, all path-related system calls should see the
    paths prefixed by the -L option.

    Issue with this is that this change is unlikely to get accepted by
    Qemu upstream. And there might be some side effects we have not
    really identified.

(B) Create an empty $(STAGING_DIR)/etc/ld.so.cache. We have tested
    this solution and it works: it gets used instead of the host
    /etc/ld.so.cache. Because $(STAGING_DIR)/etc/ld.so.cache is empty,
    there's no libc.so.6 match, so the target ELF interpreter goes
    through its normal library location resolution logic, which falls
    back to trying in /usr/lib and /lib, which works as those paths
    ends up being prefixed with $(STAGING_DIR) by Qemu.

(C) Pass LD_LIBRARY_PATH pointing to $(STAGING_DIR)/lib and
    $(STAGING_DIR)/usr/lib in the Qemu wrapper. This works because
    LD_LIBRARY_PATH paths have precedence over paths given by
    ld.so.cache.

    This is the solution already used by the GOI qemu wrapper in
    package/gobject-introspection/g-ir-scanner-qemuwrapper.in, and
    which was suggested in bug
    https://bugs.busybox.net/show_bug.cgi?id=14366.

Any opinion? Or other ideas?

Thanks!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot