[Buildroot] [PATCH] package/numactl: point to target libs via LDFLAGS

[Buildroot] [PATCH] package/numactl: point to target libs via LDFLAGS

[Alexey Brodkin via buildroot]

In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc")
we opted-out from unconditional linking with "-latomic" and made it
a conditional based on real availability of __atomic_fetch_and_1()
in the toolchain. I.e. for many targets "-latomic" was not used
any longer.

And in those supposedly rare cases when libatomic was required it was
all OK until there's a LD_LIBRARY_PATH set with something where yet
another libatomic.so instance exists, especially if that's the one from
host (in my case I used a separately built host toolchain). Then we see:
---------------------->8-----------------------
/bin/sh ./libtool  --tag=CC   --mode=link .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os -g0  -Wl,-z,max-page-size=4096 -fPIC -std=gnu99    -o numastat numastat-numastat.o  -latomic

libtool: link: .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z -Wl,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o  .../gcc-5.2.0/lib/../lib64/libatomic.so -pthread -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64
.../buildroot/output/host/lib/gcc/arc32-buildroot-linux-uclibc/12.2.1/../../../../arc32-buildroot-linux-uclibc/bin/ld: .../gcc-5.2.0/lib/../lib64/libatomic.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
---------------------->8-----------------------

That happens because libtool of numactl adds contents of the aforementioned
LD_LIBRARY_PATH to its "$searchdirs" and then the first libatomic.so it
finds becomes the one used for linking. What's more our precios cross-built
libatomic.so never even gets considered because that first found libatomic.so
is used with full path, not as "-latomic" as otherwise it would have been used
without helpful libtool.

But with paths pointing to the staging location, these paths get added to
"$searchdirs" and so the right libatomic.so gets found, which solves our problem.

Signed-off-by: Alexey Brodkin <abrodkin@synopsys.com>
---
 package/numactl/numactl.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/numactl/numactl.mk b/package/numactl/numactl.mk
index cf9c759693..58b304d6e1 100644
--- a/package/numactl/numactl.mk
+++ b/package/numactl/numactl.mk
@@ -10,6 +10,7 @@ NUMACTL_LICENSE = LGPL-2.1 (libnuma), GPL-2.0 (programs)
 NUMACTL_LICENSE_FILES = README.md
 NUMACTL_INSTALL_STAGING = YES
 NUMACTL_AUTORECONF = YES
-NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC"
+NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC" \
+		   LDFLAGS="$(TARGET_LDFLAGS) -L$(STAGING_DIR)/lib"
 
 $(eval $(autotools-package))
-- 
2.16.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/numactl: point to target libs via LDFLAGS

[Thomas Petazzoni via buildroot]

Hello Alexey,

On Mon, 24 Oct 2022 19:49:11 +0200
Alexey Brodkin via buildroot <buildroot@buildroot.org> wrote:

> In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc")
> we opted-out from unconditional linking with "-latomic" and made it
> a conditional based on real availability of __atomic_fetch_and_1()
> in the toolchain. I.e. for many targets "-latomic" was not used
> any longer.
> 
> And in those supposedly rare cases when libatomic was required it was
> all OK until there's a LD_LIBRARY_PATH set with something where yet
> another libatomic.so instance exists, especially if that's the one from
> host (in my case I used a separately built host toolchain). Then we see:
> ---------------------->8-----------------------  
> /bin/sh ./libtool  --tag=CC   --mode=link .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os -g0  -Wl,-z,max-page-size=4096 -fPIC -std=gnu99    -o numastat numastat-numastat.o  -latomic
> 
> libtool: link: .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z -Wl,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o  .../gcc-5.2.0/lib/../lib64/libatomic.so -pthread -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64
> .../buildroot/output/host/lib/gcc/arc32-buildroot-linux-uclibc/12.2.1/../../../../arc32-buildroot-linux-uclibc/bin/ld: .../gcc-5.2.0/lib/../lib64/libatomic.so: error adding symbols: file in wrong format
> collect2: error: ld returned 1 exit status
> ---------------------->8-----------------------  
> 
> That happens because libtool of numactl adds contents of the aforementioned
> LD_LIBRARY_PATH to its "$searchdirs" and then the first libatomic.so it
> finds becomes the one used for linking. What's more our precios cross-built
> libatomic.so never even gets considered because that first found libatomic.so
> is used with full path, not as "-latomic" as otherwise it would have been used
> without helpful libtool.

Thanks for the patch. However, I don't get it. Where is LD_LIBRARY_PATH
defined? Do you have a reproducer for this issue?

> +NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC" \
> +		   LDFLAGS="$(TARGET_LDFLAGS) -L$(STAGING_DIR)/lib"

This is really a red flag, $(STAGING_DIR)/lib is already in the default
search path of the compiler, so that should never be needed.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/numactl: point to target libs via LDFLAGS

[Alexey Brodkin via buildroot]

Hello, Thomas!
 
> On Mon, 24 Oct 2022 19:49:11 +0200
> Alexey Brodkin via buildroot <buildroot@buildroot.org> wrote:
> 
> > In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc")
> > we opted-out from unconditional linking with "-latomic" and made it
> > a conditional based on real availability of __atomic_fetch_and_1()
> > in the toolchain. I.e. for many targets "-latomic" was not used
> > any longer.
> > 
> > And in those supposedly rare cases when libatomic was required it was
> > all OK until there's a LD_LIBRARY_PATH set with something where yet
> > another libatomic.so instance exists, especially if that's the one from
> > host (in my case I used a separately built host toolchain). Then we see:
> > ---------------------->8-----------------------  
> > /bin/sh ./libtool  --tag=CC   --mode=link .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os -g0  -Wl,-z,max-page-size=4096 -fPIC -std=gnu99    -o numastat numastat-numastat.o  -latomic
> > 
> > libtool: link: .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z -Wl,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o  .../gcc-5.2.0/lib/../lib64/libatomic.so -pthread -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64
> > .../buildroot/output/host/lib/gcc/arc32-buildroot-linux-uclibc/12.2.1/../../../../arc32-buildroot-linux-uclibc/bin/ld: .../gcc-5.2.0/lib/../lib64/libatomic.so: error adding symbols: file in wrong format
> > collect2: error: ld returned 1 exit status
> > ---------------------->8-----------------------  
> > 
> > That happens because libtool of numactl adds contents of the aforementioned
> > LD_LIBRARY_PATH to its "$searchdirs" and then the first libatomic.so it
> > finds becomes the one used for linking. What's more our precios cross-built
> > libatomic.so never even gets considered because that first found libatomic.so
> > is used with full path, not as "-latomic" as otherwise it would have been used
> > without helpful libtool.
> 
> Thanks for the patch. However, I don't get it. Where is LD_LIBRARY_PATH
> defined?

I meant LD_LIBRARY_PATH is defined before Buildroot's top-level make invocation,
i.e. "LD_LIBRARY_PATH=_some_location_of_hosts_libatomic.so_ make". See below.

> Do you have a reproducer for this issue?

Yes, I have finally. It took me a bit longer than expected to figure out all
the details, thus answering that late. So there're a couple of prerequsites
for the reproduction even on vanilla "numactl" sources, I used 2.0.14 with
the Buildroot's libatomic patch [1]:

1. For example in my Ubuntu 20.04 on x86_64 I installed ARM cross-tools with:
   ----------------------------->8-----------------------------
   apt install gcc-9-arm-linux-gnueabi
   ----------------------------->8-----------------------------

2. LD_LIBRARY_PATH which points to a location of a foreign (e.g. host or
   some other architecture) "libatomic.so", plus (that's really important)
   right there side-by-side with "libatomic.so" it's important to have
   "libatomic.la" (valid libtool archive, and not an empty file, see below
   my copy which works for the reproduction). I.e.
   ----------------------------->8-----------------------------
   export LD_LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/
   ----------------------------->8-----------------------------
   
3. Configure "numactl" with `lt_cv_sys_lib_search_path_spec=""`, which we exactly
   do in Buildroot, see [2], i.e.:
   ----------------------------->8-----------------------------
   ./configure --host=arm-linux-gnueabi lt_cv_sys_lib_search_path_spec=""
   ----------------------------->8-----------------------------
   
4. Try to build now:
   ----------------------------->8-----------------------------
   make V=1
   make  all-am
   make[1]: Entering directory 'numactl'
   /bin/bash ./libtool  --tag=CC   --mode=link arm-linux-gnueabi-gcc -g -O2  -version-info 1:0:0 -Wl,--version-script,./versions.ldscript -Wl,-init,numa_init -Wl,-fini,numa_fini  -o libnuma.la -rpath /usr/local/lib libnuma.lo syscall.lo distance.lo affinity.lo sysfs.lo rtnetlink.lo  -latomic
   libtool: warning: library '/usr/lib/gcc/x86_64-linux-gnu/9//libatomic.la' was moved.
   libtool: link: arm-linux-gnueabi-gcc -shared  -fPIC -DPIC  .libs/libnuma.o .libs/syscall.o .libs/distance.o .libs/affinity.o .libs/sysfs.o .libs/rtnetlink.o   -Wl,-rpath -Wl,/usr/lib/gcc/x86_64-linux-gnu/9/ -Wl,-rpath -Wl,/usr/lib/gcc/x86_64-linux-gnu/9/ /usr/lib/gcc/x86_64-linux-gnu/9//libatomic.so  -g -O2 -Wl,--version-script -Wl,./versions.ldscript -Wl,-init -Wl,numa_init -Wl,-fini -Wl,numa_fini   -pthread -Wl,-soname -Wl,libnuma.so.1 -o .libs/libnuma.so.1.0.0
   /usr/lib/gcc/x86_64-linux-gnu/9//libatomic.so: file not recognized: file format not recognized
   collect2: error: ld returned 1 exit status
   make[1]: *** [Makefile:889: libnuma.la] Error 1
   make[1]: Leaving directory 'numactl'
   make: *** [Makefile:742: all] Error 2
   ----------------------------->8-----------------------------

So, what happens (3) removes compiler's default paths from libtool's
"sys_lib_dlsearch_path", while (2) makes libtool think that libatomic.so
from LD_LIBRARY_PATH is the right one.

> > +NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC" \
> > +                LDFLAGS="$(TARGET_LDFLAGS) -L$(STAGING_DIR)/lib"
> 
> This is really a red flag, $(STAGING_DIR)/lib is already in the default
> search path of the compiler, so that should never be needed.

Well, if we don't mess with "lt_cv_sys_lib_search_path_spec" in the
configure script options, then all works OK, but I guess Peter's change made in
2011 makes some sense and we don't want to revert it, or do we?

Also, I accept the fact that my setup is a bit unusual at least and it might
be a good enough reason to disregard all that discussion, but first I'd like
to make sure we really don't care about situations like that:
LD_LIBRARY_PATH pointing to a self-built toolchain.

If we assume that situation as one of possible corner-cases, then probably
we may want to troubleshoot such funny things which happen due to "libtool"
smarts.

-Alexey


[1] https://github.com/numactl/numactl/commit/e0de0d9e981ddb53bdeb4a4b9dc43046c9ff4ff9
[2] https://git.buildroot.net/buildroot/commit/?id=6246704008ef219257f30a824903855d0e00cffb

----------------------------->8-----------------------------
 Contents of "/usr/lib/gcc/x86_64-linux-gnu/9/libatomic.la"
----------------------------->8-----------------------------
# libatomic.la - a libtool library file
# Generated by libtool (GNU libtool 1.3134 2009-11-29) 2.2.7a
#
# Please DO NOT delete this file!
# It is necessary for linking the library.

# The name that we can dlopen(3).
dlname='libatomic.so.1'

# Names of this library.
library_names='libatomic.so.1.2.0 libatomic.so.1 libatomic.so'

# The name of the static archive.
old_library='libatomic.a'

# Linker flags that can not go in dependency_libs.
inherited_linker_flags=' -pthread'

# Libraries that this one depends upon.
dependency_libs=''

# Names of additional weak libraries provided by this library
weak_library_names=''

# Version information for libatomic.
current=3
age=2
revision=0

# Is this an already installed library?
installed=yes

# Should we warn about portability when linking against -modules?
shouldnotlink=no

# Files to dlopen/dlpreopen
dlopen=''
dlpreopen=''

# Directory that this library needs to be installed in:
libdir='/usr/lib/x86_64-linux-gnu/'
----------------------------->8-----------------------------
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/numactl: point to target libs via LDFLAGS

[Thomas Petazzoni via buildroot]

Hello Alexey,

On Mon, 24 Oct 2022 19:49:11 +0200
Alexey Brodkin via buildroot <buildroot@buildroot.org> wrote:

> In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc")
> we opted-out from unconditional linking with "-latomic" and made it
> a conditional based on real availability of __atomic_fetch_and_1()
> in the toolchain. I.e. for many targets "-latomic" was not used
> any longer.

We finally looked at this patch today during the on-going Buildroot
hackathon. We understand/recognize the problem, but we don't like the
proposed solution as you can imagine (especially as the proposed
solution is package specific). In addition, and as you have yourself
expressed, your situation is quite a corner case.

Here are the options that we see to resolve this:

(1) Patch libtool so that it doesn't look at LD_LIBRARY_PATH. We are
    already patching libtool to work-around other issues. However, from
    a quick look at the libtool code base (which of course is not the
    most pleasant exercise on earth), it's not trivial to see where
    this logic is taking place.

(2) Unset LD_LIBRARY_PATH entirely in the main Buildroot makefile to
    avoid interference with LD_LIBRARY_PATH being set. This would of
    course prevent people who have special/weird setups that require
    LD_LIBRARY_PATH from using Buildroot.

(3) Warn/error out if LD_LIBRARY_PATH is set, which essentially is a
    variant of (2), but where we are explicit instead of silently
    ignoring LD_LIBRARY_PATH defined in the user's environment.

Regarding the setting of lt_cv_sys_lib_search_path_spec="" that you
have identified, the motivation for this was explained in
6246704008ef219257f30a824903855d0e00cffb:

    package/Makefile.in: ensure libtool doesn't search host dirs for libraries
    
    Commit 7e3e8ec040b (CFLAGS/LDFLAGS: don't add -I / -L args for STAGING_DIR)
    exposed a lingering libtool problem.
    
    Unless instructed otherwise (using -L) libtool will search its built in
    system path for libraries, and use those instead if found. The default
    search path is '/usr/lib, /lib, /usr/local/lib', which is no good for
    cross compilation.
    
    Fix it by setting the system search path to the empty string, effectively
    disabling this feature.
    
    Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>

So this seems to still be relevant. The reason why when you drop this
in your case and it "works" is we believe the following one: due to
-latomic, libtool will search for libatomic. It will find it in your
system in say /lib or /usr/lib, and because that's a standard search
path, libtool will keep -latomic instead of turning into a full
absolute path. So the cross linker gets passed -latomic, which it
properly resolves to the target/sysroot libatomic. When
lt_cv_sys_lib_search_path_spec="" is there, libatomic from /lib or
/usr/lib isn't found, so it finds the one in your LD_LIBRARY_PATH
directory, which isn't a standard path, so our dear libtool replaces
-latomic by the full absolute path to libatomic, and passes that to the
cross-linker, which of course blows up.

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot