Linker failures trying to build static qemu-user binary

Linker failures trying to build static qemu-user binary

[John Paul Adrian Glaubitz]

Hi!

I usually build a static qemu-user binary from the git source using the following
configure line:

export ARCH=m68k && ./configure --target-list=$ARCH-linux-user --static --disable-rdma \
                    --disable-pie --disable-system --disable-xen --disable-pvrdma \
                    --disable-libssh --disable-werror --disable-glusterfs

However, that stopped working some time ago with the linker error below.

Does anyone know how to fix this?

Adrian

PS: Please keep me CC'ed, I have disabled receiving any mails from the mailing list.

[833/934] Compiling C object libqemu-m68k-linux-user.fa.p/linux-user_syscall.c.o
[834/934] Linking target tests/unit/check-qstring
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libglib-2.0.a(gutils.c.o): in function `g_get_user_database_entry':
(.text+0x277): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version
used for linking
/usr/bin/ld: (.text+0xe0): warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the
glibc version used for linking
/usr/bin/ld: (.text+0x11e): warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the
glibc version used for linking
[835/934] Compiling C object tests/fp/fp-test.p/fp-test.c.o
[836/934] Compiling C object tests/fp/fp-bench.p/fp-bench.c.o
[837/934] Compiling C object tests/fp/fp-test.p/berkeley-testfloat-3_source_slowfloat.c.o
[838/934] Compiling C object tests/unit/check-qnum.p/check-qnum.c.o
[839/934] Compiling C object tests/unit/check-qdict.p/check-qdict.c.o
[840/934] Linking target tests/unit/check-block-qdict
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libglib-2.0.a(gutils.c.o): in function `g_get_user_database_entry':
(.text+0x277): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version
used for linking
/usr/bin/ld: (.text+0xe0): warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the
glibc version used for linking
/usr/bin/ld: (.text+0x11e): warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the
glibc version used for linking
[841/934] Linking static target tests/fp/libsoftfloat.a
[842/934] Linking target qemu-m68k
FAILED: qemu-m68k 
c++ -m64 -mcx16  -o qemu-m68k libcommon.fa.p/hw_core_cpu-common.c.o libcommon.fa.p/hw_core_machine-smp.c.o libcommon.fa.p/cpus-common.c.o
libcommon.fa.p/page-vary-common.c.o libcommon.fa.p/disas_m68k.c.o libcommon.fa.p/disas_capstone.c.o libcommon.fa.p/gdbstub_user.c.o
libcommon.fa.p/accel_accel-user.c.o libcommon.fa.p/common-user_safe-syscall.S.o libcommon.fa.p/common-user_safe-syscall-error.c.o libqemu-
m68k-linux-user.fa.p/linux-user_m68k_signal.c.o libqemu-m68k-linux-user.fa.p/linux-user_m68k_cpu_loop.c.o libqemu-m68k-linux-
user.fa.p/target_m68k_cpu.c.o libqemu-m68k-linux-user.fa.p/target_m68k_fpu_helper.c.o libqemu-m68k-linux-user.fa.p/target_m68k_gdbstub.c.o
libqemu-m68k-linux-user.fa.p/target_m68k_helper.c.o libqemu-m68k-linux-user.fa.p/target_m68k_op_helper.c.o libqemu-m68k-linux-
user.fa.p/target_m68k_softfloat.c.o libqemu-m68k-linux-user.fa.p/target_m68k_translate.c.o libqemu-m68k-linux-user.fa.p/trace_control-
target.c.o libqemu-m68k-linux-user.fa.p/gdbstub_gdbstub.c.o libqemu-m68k-linux-user.fa.p/cpu.c.o libqemu-m68k-linux-user.fa.p/disas.c.o
libqemu-m68k-linux-user.fa.p/page-vary.c.o libqemu-m68k-linux-user.fa.p/tcg_optimize.c.o libqemu-m68k-linux-user.fa.p/tcg_region.c.o
libqemu-m68k-linux-user.fa.p/tcg_tcg.c.o libqemu-m68k-linux-user.fa.p/tcg_tcg-common.c.o libqemu-m68k-linux-user.fa.p/tcg_tcg-op.c.o
libqemu-m68k-linux-user.fa.p/tcg_tcg-op-gvec.c.o libqemu-m68k-linux-user.fa.p/tcg_tcg-op-vec.c.o libqemu-m68k-linux-
user.fa.p/fpu_softfloat.c.o libqemu-m68k-linux-user.fa.p/accel_accel-common.c.o libqemu-m68k-linux-user.fa.p/accel_accel-blocker.c.o
libqemu-m68k-linux-user.fa.p/accel_tcg_tcg-all.c.o libqemu-m68k-linux-user.fa.p/accel_tcg_cpu-exec-common.c.o libqemu-m68k-linux-
user.fa.p/accel_tcg_cpu-exec.c.o libqemu-m68k-linux-user.fa.p/accel_tcg_tb-maint.c.o libqemu-m68k-linux-user.fa.p/accel_tcg_tcg-runtime-
gvec.c.o libqemu-m68k-linux-user.fa.p/accel_tcg_tcg-runtime.c.o libqemu-m68k-linux-user.fa.p/accel_tcg_translate-all.c.o libqemu-m68k-linux-
user.fa.p/accel_tcg_translator.c.o libqemu-m68k-linux-user.fa.p/accel_tcg_user-exec.c.o libqemu-m68k-linux-user.fa.p/accel_tcg_user-exec-
stub.c.o libqemu-m68k-linux-user.fa.p/accel_tcg_debuginfo.c.o libqemu-m68k-linux-user.fa.p/accel_tcg_perf.c.o libqemu-m68k-linux-
user.fa.p/linux-user_elfload.c.o libqemu-m68k-linux-user.fa.p/linux-user_exit.c.o libqemu-m68k-linux-user.fa.p/linux-user_fd-trans.c.o
libqemu-m68k-linux-user.fa.p/linux-user_linuxload.c.o libqemu-m68k-linux-user.fa.p/linux-user_main.c.o libqemu-m68k-linux-user.fa.p/linux-
user_mmap.c.o libqemu-m68k-linux-user.fa.p/linux-user_signal.c.o libqemu-m68k-linux-user.fa.p/linux-user_strace.c.o libqemu-m68k-linux-
user.fa.p/linux-user_syscall.c.o libqemu-m68k-linux-user.fa.p/linux-user_thunk.c.o libqemu-m68k-linux-user.fa.p/linux-user_uaccess.c.o
libqemu-m68k-linux-user.fa.p/linux-user_uname.c.o libqemu-m68k-linux-user.fa.p/linux-user_flatload.c.o libqemu-m68k-linux-user.fa.p/meson-
generated_.._m68k-linux-user-gdbstub-xml.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,--whole-archive libhwcore.fa libqom.fa -Wl,--start-group
libevent-loop-base.a -Wl,--no-whole-archive -fstack-protector-strong -static -Wl,-z,relro -Wl,-z,now -Wl,--warn-common libqemuutil.a
libhwcore.fa libqom.fa /usr/lib/x86_64-linux-gnu/libcapstone.a /usr/lib/x86_64-linux-gnu/libdw.a /usr/lib/x86_64-linux-gnu/libbz2.a
/usr/lib/x86_64-linux-gnu/liblzma.a -pthread -lpthread /usr/lib/x86_64-linux-gnu/libelf.a /usr/lib/x86_64-linux-gnu/libz.a -lrt -lm -
lgthread-2.0 -lglib-2.0 -lpcre -lstdc++ -Wl,--end-group
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libglib-2.0.a(gutils.c.o): in function `g_get_user_database_entry':
(.text+0x277): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version
used for linking
/usr/bin/ld: (.text+0xe0): warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the
glibc version used for linking
/usr/bin/ld: (.text+0x11e): warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the
glibc version used for linking
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libdw.a(debuginfod-client.o): in function `__libdwfl_debuginfod_init':
(.text.startup+0x17): undefined reference to `dlopen'
/usr/bin/ld: (.text.startup+0x32): undefined reference to `dlsym'
/usr/bin/ld: (.text.startup+0x4b): undefined reference to `dlsym'
/usr/bin/ld: (.text.startup+0x64): undefined reference to `dlsym'
/usr/bin/ld: (.text.startup+0x7d): undefined reference to `dlsym'
/usr/bin/ld: (.text.startup+0xdc): undefined reference to `dlclose'
collect2: error: ld returned 1 exit status
[843/934] Linking target tests/unit/check-qdict
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libglib-2.0.a(gutils.c.o): in function `g_get_user_database_entry':
(.text+0x277): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version
used for linking
/usr/bin/ld: (.text+0xe0): warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the
glibc version used for linking
/usr/bin/ld: (.text+0x11e): warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the
glibc version used for linking
[844/934] Linking target tests/unit/check-qnum
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libglib-2.0.a(gutils.c.o): in function `g_get_user_database_entry':
(.text+0x277): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version
used for linking
/usr/bin/ld: (.text+0xe0): warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the
glibc version used for linking
/usr/bin/ld: (.text+0x11e): warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the
glibc version used for linking
[845/934] Compiling C object tests/unit/check-qlist.p/check-qlist.c.o
[846/934] Compiling C object tests/fp/fp-test.p/.._.._fpu_softfloat.c.o
[847/934] Compiling C object tests/fp/fp-bench.p/.._.._fpu_softfloat.c.o
[848/934] Compiling C object tests/fp/fp-test-log2.p/.._.._fpu_softfloat.c.o
[849/934] Generating docs/QEMU manual with a custom command
ninja: build stopped: subcommand failed.
make[1]: *** [Makefile:165: run-ninja] Error 1
make[1]: Leaving directory '/root/qemu/build'
make: *** [GNUmakefile:11: all] Error 2q

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Linker failures trying to build static qemu-user binary

[Peter Maydell]

On Thu, 9 Feb 2023 at 13:54, John Paul Adrian Glaubitz
<glaubitz@physik.fu-berlin.de> wrote:
>
> Hi!
>
> I usually build a static qemu-user binary from the git source using the following
> configure line:
>
> export ARCH=m68k && ./configure --target-list=$ARCH-linux-user --static --disable-rdma \
>                     --disable-pie --disable-system --disable-xen --disable-pvrdma \
>                     --disable-libssh --disable-werror --disable-glusterfs
>
> However, that stopped working some time ago with the linker error below.
>
> Does anyone know how to fix this?

The "Using getpwuid in statically linked applications" etc warnings
are expected, so we can ignore those; this is the key error:

> /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libdw.a(debuginfod-client.o): in function `__libdwfl_debuginfod_init':
> (.text.startup+0x17): undefined reference to `dlopen'
> /usr/bin/ld: (.text.startup+0x32): undefined reference to `dlsym'
> /usr/bin/ld: (.text.startup+0x4b): undefined reference to `dlsym'
> /usr/bin/ld: (.text.startup+0x64): undefined reference to `dlsym'
> /usr/bin/ld: (.text.startup+0x7d): undefined reference to `dlsym'
> /usr/bin/ld: (.text.startup+0xdc): undefined reference to `dlclose'
> collect2: error: ld returned 1 exit status

We use pkg-config to find out what the libdw library needs on
the compiler/linker command line to link successfully, so
maybe your distro's pkg-config info isn't right. What does
"pkg-config --static --libs libdw" say ? If libdw needs libdl
then it ought to list it in that output, I think. IME pkg-config
information is often incorrect for static linking, though.
I guess this one happened to work previously because glibc didn't
actually mandate linking with '-ldl', and now on your system it
apparently does. On my system pkg-config says
-ldw -lbz2 -llzma -pthread -lpthread -lelf -lz
which looks like it's missing -ldl, but the link succeeds anyway,
presumably because the symbols are provided by the main glibc .a.

On the other hand, if libdw wants to use dlopen/dlsym then
I wonder if we should just suppress it for static linking:
on my (Ubuntu 22.04) ld warns:
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libdw.a(debuginfod-client.o):
in function `__libdwfl_debuginfod_init':
(.text.startup+0x1b): warning: Using 'dlopen' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking

so whatever libdw is trying to do will likely not work in most
statically-linked situations anyway.

I've cc'd the author of the commit that added the libdw
dependency.

thanks
-- PMM

Re: Linker failures trying to build static qemu-user binary

[John Paul Adrian Glaubitz]

Hi!

On Thu, 2023-02-09 at 14:14 +0000, Peter Maydell wrote:
> The "Using getpwuid in statically linked applications" etc warnings
> are expected, so we can ignore those; this is the key error:

OK.

> > /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libdw.a(debuginfod-client.o): in function `__libdwfl_debuginfod_init':
> > (.text.startup+0x17): undefined reference to `dlopen'
> > /usr/bin/ld: (.text.startup+0x32): undefined reference to `dlsym'
> > /usr/bin/ld: (.text.startup+0x4b): undefined reference to `dlsym'
> > /usr/bin/ld: (.text.startup+0x64): undefined reference to `dlsym'
> > /usr/bin/ld: (.text.startup+0x7d): undefined reference to `dlsym'
> > /usr/bin/ld: (.text.startup+0xdc): undefined reference to `dlclose'
> > collect2: error: ld returned 1 exit status
> 
> We use pkg-config to find out what the libdw library needs on
> the compiler/linker command line to link successfully, so
> maybe your distro's pkg-config info isn't right. What does
> "pkg-config --static --libs libdw" say ?

glaubitz@nofan:~> pkg-config --static --libs libdw
-ldw -lbz2 -llzma -pthread -lpthread -lelf -lz
glaubitz@nofan:~>

I'm building on Debian stable (Bullseye).

> If libdw needs libdl
> then it ought to list it in that output, I think. IME pkg-config
> information is often incorrect for static linking, though.
> I guess this one happened to work previously because glibc didn't
> actually mandate linking with '-ldl', and now on your system it
> apparently does. On my system pkg-config says
> -ldw -lbz2 -llzma -pthread -lpthread -lelf -lz
> which looks like it's missing -ldl, but the link succeeds anyway,
> presumably because the symbols are provided by the main glibc .a.
> 
> On the other hand, if libdw wants to use dlopen/dlsym then
> I wonder if we should just suppress it for static linking:
> on my (Ubuntu 22.04) ld warns:
> /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libdw.a(debuginfod-client.o):
> in function `__libdwfl_debuginfod_init':
> (.text.startup+0x1b): warning: Using 'dlopen' in statically linked
> applications requires at runtime the shared libraries from the glibc
> version used for linking
> 
> so whatever libdw is trying to do will likely not work in most
> statically-linked situations anyway.

So, just include "-ldl" in LD_FLAGS?

> I've cc'd the author of the commit that added the libdw
> dependency.

Thank you!

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Linker failures trying to build static qemu-user binary

[Peter Maydell]

On Thu, 9 Feb 2023 at 14:21, John Paul Adrian Glaubitz
<glaubitz@physik.fu-berlin.de> wrote:
>
> Hi!
>
> On Thu, 2023-02-09 at 14:14 +0000, Peter Maydell wrote:
> > The "Using getpwuid in statically linked applications" etc warnings
> > are expected, so we can ignore those; this is the key error:
>
> OK.
>
> > > /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libdw.a(debuginfod-client.o): in function `__libdwfl_debuginfod_init':
> > > (.text.startup+0x17): undefined reference to `dlopen'
> > > /usr/bin/ld: (.text.startup+0x32): undefined reference to `dlsym'
> > > /usr/bin/ld: (.text.startup+0x4b): undefined reference to `dlsym'
> > > /usr/bin/ld: (.text.startup+0x64): undefined reference to `dlsym'
> > > /usr/bin/ld: (.text.startup+0x7d): undefined reference to `dlsym'
> > > /usr/bin/ld: (.text.startup+0xdc): undefined reference to `dlclose'
> > > collect2: error: ld returned 1 exit status
> >
> > We use pkg-config to find out what the libdw library needs on
> > the compiler/linker command line to link successfully, so
> > maybe your distro's pkg-config info isn't right. What does
> > "pkg-config --static --libs libdw" say ?
>
> glaubitz@nofan:~> pkg-config --static --libs libdw
> -ldw -lbz2 -llzma -pthread -lpthread -lelf -lz

Right, same as my output.

> I'm building on Debian stable (Bullseye).
>
> > If libdw needs libdl
> > then it ought to list it in that output, I think. IME pkg-config
> > information is often incorrect for static linking, though.
> > I guess this one happened to work previously because glibc didn't
> > actually mandate linking with '-ldl', and now on your system it
> > apparently does. On my system pkg-config says
> > -ldw -lbz2 -llzma -pthread -lpthread -lelf -lz
> > which looks like it's missing -ldl, but the link succeeds anyway,
> > presumably because the symbols are provided by the main glibc .a.
> >
> > On the other hand, if libdw wants to use dlopen/dlsym then
> > I wonder if we should just suppress it for static linking:
> > on my (Ubuntu 22.04) ld warns:
> > /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libdw.a(debuginfod-client.o):
> > in function `__libdwfl_debuginfod_init':
> > (.text.startup+0x1b): warning: Using 'dlopen' in statically linked
> > applications requires at runtime the shared libraries from the glibc
> > version used for linking
> >
> > so whatever libdw is trying to do will likely not work in most
> > statically-linked situations anyway.
>
> So, just include "-ldl" in LD_FLAGS?

If this is necessary, then pkg-config should tell us to do it :-)

But in the usual situation that you put the statically linked
QEMU binary into a chroot, the dlopen() that libdw is going to
try to do won't work anyway...

thanks
-- PMM

Re: Linker failures trying to build static qemu-user binary

[John Paul Adrian Glaubitz]

On Thu, 2023-02-09 at 14:23 +0000, Peter Maydell wrote:
> > So, just include "-ldl" in LD_FLAGS?
> 
> If this is necessary, then pkg-config should tell us to do it :-)
> 
> But in the usual situation that you put the statically linked
> QEMU binary into a chroot, the dlopen() that libdw is going to
> try to do won't work anyway...

FWIW, passing --extra-ldflags="-ldl" fixes the issue for me.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Linker failures trying to build static qemu-user binary

[Ilya Leoshkevich]

On Thu, 2023-02-09 at 14:14 +0000, Peter Maydell wrote:
> On Thu, 9 Feb 2023 at 13:54, John Paul Adrian Glaubitz
> <glaubitz@physik.fu-berlin.de> wrote:
> > 
> > Hi!
> > 
> > I usually build a static qemu-user binary from the git source using
> > the following
> > configure line:
> > 
> > export ARCH=m68k && ./configure --target-list=$ARCH-linux-user --
> > static --disable-rdma \
> >                     --disable-pie --disable-system --disable-xen --
> > disable-pvrdma \
> >                     --disable-libssh --disable-werror --disable-
> > glusterfs
> > 
> > However, that stopped working some time ago with the linker error
> > below.
> > 
> > Does anyone know how to fix this?
> 
> The "Using getpwuid in statically linked applications" etc warnings
> are expected, so we can ignore those; this is the key error:
> 
> > /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libdw.a(debuginfod-
> > client.o): in function `__libdwfl_debuginfod_init':
> > (.text.startup+0x17): undefined reference to `dlopen'
> > /usr/bin/ld: (.text.startup+0x32): undefined reference to `dlsym'
> > /usr/bin/ld: (.text.startup+0x4b): undefined reference to `dlsym'
> > /usr/bin/ld: (.text.startup+0x64): undefined reference to `dlsym'
> > /usr/bin/ld: (.text.startup+0x7d): undefined reference to `dlsym'
> > /usr/bin/ld: (.text.startup+0xdc): undefined reference to `dlclose'
> > collect2: error: ld returned 1 exit status
> 
> We use pkg-config to find out what the libdw library needs on
> the compiler/linker command line to link successfully, so
> maybe your distro's pkg-config info isn't right. What does
> "pkg-config --static --libs libdw" say ? If libdw needs libdl
> then it ought to list it in that output, I think. IME pkg-config
> information is often incorrect for static linking, though.
> I guess this one happened to work previously because glibc didn't
> actually mandate linking with '-ldl', and now on your system it
> apparently does. On my system pkg-config says
> -ldw -lbz2 -llzma -pthread -lpthread -lelf -lz
> which looks like it's missing -ldl, but the link succeeds anyway,
> presumably because the symbols are provided by the main glibc .a.
> 
> On the other hand, if libdw wants to use dlopen/dlsym then
> I wonder if we should just suppress it for static linking:
> on my (Ubuntu 22.04) ld warns:
> /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libdw.a(debuginfod-client.o):
> in function `__libdwfl_debuginfod_init':
> (.text.startup+0x1b): warning: Using 'dlopen' in statically linked
> applications requires at runtime the shared libraries from the glibc
> version used for linking
> 
> so whatever libdw is trying to do will likely not work in most
> statically-linked situations anyway.
> 
> I've cc'd the author of the commit that added the libdw
> dependency.
> 
> thanks
> -- PMM

Hi,

libdw uses dlopen() and friends for debuginfod integration, which QEMU
does not currently use. So in principle --extra-ldflags="-ldl"
should be safe.

That said, the functionality provided by libdw is needed for analyzing
performance of JITed code, which is mostly useful to developers and
researchers. Therefore, in order to avoid unpleasant surprises for
people who don't need this, I will send a patch that disables libdw
for static builds.

Best regards,
Ilya