[Buildroot] [PATCH 0/1] Build host-libcap with libattr support

[Buildroot] [PATCH 0/1] Build host-libcap with libattr support

[Ben Boeckel]

I've also sent upstream to always define these symbols, so future versions can
drop the hard-coded host-attr dependency if wanted.

--Ben

Ben Boeckel (1):
  libcap: depend on host-attr for the host

 package/attr/attr.mk     | 1 +
 package/libcap/libcap.mk | 8 ++++----
 2 files changed, 5 insertions(+), 4 deletions(-)

-- 
2.6.2

[Buildroot] [PATCH 1/1] libcap: depend on host-attr for the host

[Ben Boeckel]

When a host-libcap is built, it is built without libattr support.
However, libcap does not define some symbols based on this flag. This
becomes a problem when a build-host tool is run (e.g., ls) which uses
these symbols gets the host-libcap library via LD_LIBRARY_PATH.

In order to define all of the symbols, add libattr support on the host.
The RAISE_SETFCAP setting must also be disabled since the required
permissions are not available during install.

Signed-off-by: Ben Boeckel <mathstuf@gmail.com>
---
 package/attr/attr.mk     | 1 +
 package/libcap/libcap.mk | 9 ++++-----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/package/attr/attr.mk b/package/attr/attr.mk
index a66891b..7dac7cf 100644
--- a/package/attr/attr.mk
+++ b/package/attr/attr.mk
@@ -37,3 +37,4 @@ endef
 ATTR_POST_INSTALL_STAGING_HOOKS += ATTR_FIX_LIBTOOL_LA_LIBDIR
 
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))
diff --git a/package/libcap/libcap.mk b/package/libcap/libcap.mk
index 54844f8..65057e5 100644
--- a/package/libcap/libcap.mk
+++ b/package/libcap/libcap.mk
@@ -20,8 +20,7 @@ else
 LIBCAP_HAVE_LIBATTR = no
 endif
 
-# we don't have host-attr
-HOST_LIBCAP_DEPENDENCIES =
+HOST_LIBCAP_DEPENDENCIES = host-attr
 
 ifeq ($(BR2_STATIC_LIBS),y)
 LIBCAP_MAKE_TARGET = libcap.a
@@ -67,12 +66,12 @@ define LIBCAP_INSTALL_TARGET_CMDS
 endef
 
 define HOST_LIBCAP_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D) LIBATTR=no
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)
 endef
 
 define HOST_LIBCAP_INSTALL_CMDS
-	$(HOST_MAKE_ENV) $(MAKE) -C $(@D) LIBATTR=no DESTDIR=$(HOST_DIR) \
-		prefix=/usr lib=lib install
+	$(HOST_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(HOST_DIR) \
+		RAISE_SETFCAP=no prefix=/usr lib=lib install
 endef
 
 $(eval $(generic-package))
-- 
2.6.2

[Buildroot] [PATCH 1/1] libcap: depend on host-attr for the host

[Thomas Petazzoni]

Dear Ben Boeckel,

On Sat, 24 Oct 2015 17:39:14 -0400, Ben Boeckel wrote:
> When a host-libcap is built, it is built without libattr support.
> However, libcap does not define some symbols based on this flag. This
> becomes a problem when a build-host tool is run (e.g., ls) which uses
> these symbols gets the host-libcap library via LD_LIBRARY_PATH.

Ok, so let me try to understand this a bit better. You say that if you
have host-libcap built and installed in $(HOST_DIR), then when you
build another package, if the build process of that package uses your
system 'ls' (i.e /bin/ls), then this ls doesn't work properly because
it uses $(HOST_DIR)/usr/lib/libacl.so instead of /usr/lib/libacl.so.

Is that correct ?

If that is correct, then I think your proposed change is not the right
solution, because it is just a workaround for the specific situation,
on your distribution, where ls gets linked against an acl that has attr
support. A similar problem can appear with other binaries/libraries,
and we don't want to clutter our host package dependencies to solve
such problems IMO.

Instead, I believe we should look at removing $(HOST_DIR)/usr/lib from
LD_LIBRARY_PATH, i.e do something like:

diff --git a/package/Makefile.in b/package/Makefile.in
index 8a592d4..10f3a47 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -298,12 +298,10 @@ HOST_CONFIGURE_OPTS = \
        PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
        PKG_CONFIG_SYSROOT_DIR="/" \
        PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig:$(HOST_DIR)/usr/share/pkgconfig" \
-       LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
        INTLTOOL_PERL=$(PERL)
 
 HOST_MAKE_ENV = \
        PATH=$(BR_PATH) \
-       LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
        PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
        PKG_CONFIG_SYSROOT_DIR="/" \
        PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig"

We do build all host binaries in $(HOST_DIR) with a rpath that points
to $(HOST_DIR)/usr/lib, so this LD_LIBRARY_PATH should not be
necessary. And this LD_LIBRARY_PATH is indeed wrong as it means that
any system binary will first use the Buildroot-built host libraries
rather than the system ones, as you experienced. These LD_LIBRARY_PATH
were introduced back in February 2010, at a time when we were not
building host binaries with a rpath. So I believe they should by now be
unnecessary, and as you found out, potentially harmful.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 1/1] libcap: depend on host-attr for the host

[Ben Boeckel]

On Mon, Oct 26, 2015 at 09:58:27 +0100, Thomas Petazzoni wrote:
> On Sat, 24 Oct 2015 17:39:14 -0400, Ben Boeckel wrote:
> > When a host-libcap is built, it is built without libattr support.
> > However, libcap does not define some symbols based on this flag. This
> > becomes a problem when a build-host tool is run (e.g., ls) which uses
> > these symbols gets the host-libcap library via LD_LIBRARY_PATH.
> 
> Ok, so let me try to understand this a bit better. You say that if you
> have host-libcap built and installed in $(HOST_DIR), then when you
> build another package, if the build process of that package uses your
> system 'ls' (i.e /bin/ls), then this ls doesn't work properly because
> it uses $(HOST_DIR)/usr/lib/libacl.so instead of /usr/lib/libacl.so.
> 
> Is that correct ?

Yes.

> If that is correct, then I think your proposed change is not the right
> solution, because it is just a workaround for the specific situation,
> on your distribution, where ls gets linked against an acl that has attr
> support. A similar problem can appear with other binaries/libraries,
> and we don't want to clutter our host package dependencies to solve
> such problems IMO.
> 
> Instead, I believe we should look at removing $(HOST_DIR)/usr/lib from
> LD_LIBRARY_PATH, i.e do something like:
<snip>

This sounds like a much better solution.

Thanks,

--Ben

[Buildroot] [PATCH 1/1] libcap: depend on host-attr for the host

[Thomas Petazzoni]

Ben,

On Mon, 26 Oct 2015 10:27:25 -0400, Ben Boeckel wrote:

> > If that is correct, then I think your proposed change is not the right
> > solution, because it is just a workaround for the specific situation,
> > on your distribution, where ls gets linked against an acl that has attr
> > support. A similar problem can appear with other binaries/libraries,
> > and we don't want to clutter our host package dependencies to solve
> > such problems IMO.
> > 
> > Instead, I believe we should look at removing $(HOST_DIR)/usr/lib from
> > LD_LIBRARY_PATH, i.e do something like:
> <snip>
> 
> This sounds like a much better solution.

Does this mean you will send a patch for it, after doing some testing ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 1/1] libcap: depend on host-attr for the host

[Ben Boeckel]

On Tue, Oct 27, 2015 at 07:29:54 +0100, Thomas Petazzoni wrote:
> Does this mean you will send a patch for it, after doing some testing ?

Patch sent. Tested locally by building util-linux.

Thanks,

--Ben

[Buildroot] [PATCH] makefile: remove LD_LIBRARY_PATH from the environment

[Ben Boeckel]

If system tools are selected, the host's lib/ directory may shadow
libraries from the system which are configured differently and do not
have all of the symbols required by the system tool.

Since buildroot uses rpath everywhere, LD_LIBRARY_PATH should not be
necessary anyways.

Signed-off-by: Ben Boeckel <mathstuf@gmail.com>
---
 package/Makefile.in | 2 --
 1 file changed, 2 deletions(-)

diff --git a/package/Makefile.in b/package/Makefile.in
index 8a592d4..10f3a47 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -298,12 +298,10 @@ HOST_CONFIGURE_OPTS = \
 	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
 	PKG_CONFIG_SYSROOT_DIR="/" \
 	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig:$(HOST_DIR)/usr/share/pkgconfig" \
-	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
 	INTLTOOL_PERL=$(PERL)
 
 HOST_MAKE_ENV = \
 	PATH=$(BR_PATH) \
-	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
 	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
 	PKG_CONFIG_SYSROOT_DIR="/" \
 	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig"
-- 
2.6.2

[Buildroot] [PATCH] makefile: remove LD_LIBRARY_PATH from the environment

[Ben Boeckel]

On Wed, Oct 28, 2015 at 22:48:15 -0400, Ben Boeckel wrote:
> If system tools are selected, the host's lib/ directory may shadow
> libraries from the system which are configured differently and do not
> have all of the symbols required by the system tool.
> 
> Since buildroot uses rpath everywhere, LD_LIBRARY_PATH should not be
> necessary anyways.
> 
> Signed-off-by: Ben Boeckel <mathstuf@gmail.com>

Bump.

--Ben

[Buildroot] [PATCH] makefile: remove LD_LIBRARY_PATH from the environment

[Arnout Vandecappelle]

On 29-10-15 03:48, Ben Boeckel wrote:
> If system tools are selected, the host's lib/ directory may shadow
> libraries from the system which are configured differently and do not
> have all of the symbols required by the system tool.
> 
> Since buildroot uses rpath everywhere, LD_LIBRARY_PATH should not be
> necessary anyways.
> 
> Signed-off-by: Ben Boeckel <mathstuf@gmail.com>

Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 As I mentioned in a discussion with Yann (don't remember if it was on IRC or on
the list), I think we should apply this now no master to see what the
autobuilders say. We currently break the build on Fedora 23 for any
configuration including libattr, which IMHO is worse than the (probably small)
number of host-tools that are missing rpath.

 If we apply this patch _now_, we still have a few weeks to look at autobuild
results and handle the fallout.

 Note that all the host tools that are mentioned in Config.in.host will have to
be tested manually. Anybody up for that? Also the boot loaders will have to be
tested manually since they're not handled by the autobuilders.

 Regards,
 Arnout

> ---
>  package/Makefile.in | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/package/Makefile.in b/package/Makefile.in
> index 8a592d4..10f3a47 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -298,12 +298,10 @@ HOST_CONFIGURE_OPTS = \
>  	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
>  	PKG_CONFIG_SYSROOT_DIR="/" \
>  	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig:$(HOST_DIR)/usr/share/pkgconfig" \
> -	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
>  	INTLTOOL_PERL=$(PERL)
>  
>  HOST_MAKE_ENV = \
>  	PATH=$(BR_PATH) \
> -	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
>  	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
>  	PKG_CONFIG_SYSROOT_DIR="/" \
>  	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig"
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] makefile: remove LD_LIBRARY_PATH from the environment

[Yann E. MORIN]

On November 13, 2015 7:37:46 AM GMT+01:00, Arnout Vandecappelle <arnout@mind.be> wrote:
>On 29-10-15 03:48, Ben Boeckel wrote:
>> If system tools are selected, the host's lib/ directory may shadow
>> libraries from the system which are configured differently and do not
>> have all of the symbols required by the system tool.
>> 
>> Since buildroot uses rpath everywhere, LD_LIBRARY_PATH should not be
>> necessary anyways.
>> 
>> Signed-off-by: Ben Boeckel <mathstuf@gmail.com>
>
>Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>
>As I mentioned in a discussion with Yann (don't remember if it was on
>IRC or on
>the list), I think we should apply this now no master to see what the
>autobuilders say. We currently break the build on Fedora 23 for any
>configuration including libattr, which IMHO is worse than the (probably
>small)
>number of host-tools that are missing rpath.
>
>If we apply this patch _now_, we still have a few weeks to look at
>autobuild
>results and handle the fallout.
>
>Note that all the host tools that are mentioned in Config.in.host will
>have to
>be tested manually. Anybody up for that? Also the boot loaders will
>have to be
>tested manually since they're not handled by the autobuilders.
>
> Regards,
> Arnout
>
>> ---
>>  package/Makefile.in | 2 --
>>  1 file changed, 2 deletions(-)
>> 
>> diff --git a/package/Makefile.in b/package/Makefile.in
>> index 8a592d4..10f3a47 100644
>> --- a/package/Makefile.in
>> +++ b/package/Makefile.in
>> @@ -298,12 +298,10 @@ HOST_CONFIGURE_OPTS = \
>>  	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
>>  	PKG_CONFIG_SYSROOT_DIR="/" \
>> 
>	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig:$(HOST_DIR)/usr/share/pkgconfig"
>\
>> -	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if
>$(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
>>  	INTLTOOL_PERL=$(PERL)
>>  
>>  HOST_MAKE_ENV = \
>>  	PATH=$(BR_PATH) \
>> -	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if
>$(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
>>  	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
>>  	PKG_CONFIG_SYSROOT_DIR="/" \
>>  	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig"
>> 
>
>
>-- 
>Arnout Vandecappelle                          arnout at mind be
>Senior Embedded Software Architect            +32-16-286500
>Essensium/Mind                                http://www.mind.be
>G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
>LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
>GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
>_______________________________________________
>buildroot mailing list
>buildroot at busybox.net
>http://lists.busybox.net/mailman/listinfo/buildroot

Ben, Arnout, All,

NAK for this one is the current state.

I have been working on it in the past few days (current branch not pushed, I'm away... Will push tonight):
http://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/no-ld-library-path

We still have packages that are missing the rpath, but that's not the only issue.

Look at libcurl: we're fiddling with LD_LIBRARY_PATH. Since we're setting it today, it is not empty. But if we no longer set it, we'd have the current directory in it (see the dependency check we do on it too for info).

So we should at least fix the known issues first, which I'm working on... More on this tonight when I land back home...

Regards,
Yann E. MORIN (on the move).
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151113/270389b1/attachment.html>

[Buildroot] [PATCH] makefile: remove LD_LIBRARY_PATH from the environment

[Yann E. MORIN]

Ben, Arnout, Peter, All,

+Peter on this, because it is important he reads Arnout's comments and
my reply about it (for once, I agree with Arnout! ;-] ).

On 2015-11-13 10:00 +0100, Yann E. MORIN spake thusly:
> On November 13, 2015 7:37:46 AM GMT+01:00, Arnout Vandecappelle <arnout@mind.be> wrote:
>  On 29-10-15 03:48, Ben Boeckel wrote:
>   If system tools are selected, the host's lib/ directory may shadow
>   libraries from the system which are configured differently and do not
>   have all of the symbols required by the system tool.
>  
>   Since buildroot uses rpath everywhere, LD_LIBRARY_PATH should not be
>   necessary anyways.
>  
>   Signed-off-by: Ben Boeckel <mathstuf@gmail.com>
> 
>  Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

I've marked your patch as superseded in Pathwork, since I have now
included it in my series:
    http://lists.busybox.net/pipermail/buildroot/2015-November/144362.html

Thanks! :-)

>   As I mentioned in a discussion with Yann (don't remember if it was on IRC or on
>  the list), I think we should apply this now no master to see what the
>  autobuilders say. We currently break the build on Fedora 23 for any
>  configuration including libattr, which IMHO is worse than the (probably small)
>  number of host-tools that are missing rpath.

And we should normally no longer have any host tool missing an RPATH.
I have built all of them now, and the one that was broken wrt. rpath,
I fixed.

>   If we apply this patch _now_, we still have a few weeks to look at autobuild
>  results and handle the fallout.

And at worst, we can simply revert it. Even if the patch touches a
critical aspect of the build system, it is very well contained, so will
be very easy to revert at the last minute if need be.

>   Note that all the host tools that are mentioned in Config.in.host will have to
>  be tested manually. Anybody up for that? Also the boot loaders will have to be
>  tested manually since they're not handled by the autobuilders.

Ah, I forgot about the bootloaders... ;-/

However, for the host tools, here's what I did;

    $ git grep -l -E '^\$\(eval \$\(host-'    \
    |sed 'magic-to-extract-pkg-name'        \
    >host-pkg-list

    $ vi host-pkg-list  # remove gcc, binutils, gdb

    $ make $(cat host-pkg-list)

And after ~73min, all the host packages in the lit have been built, and
none were installing executables missing an RPATH when they are using
our host libs.

Note: openssl install libraries that are missing an RPATH: libssl.so and
all the engines libraries. But that's in practice not an issue, since
the executables that uses those libraries do have an RPATH (as they were
able to find those libs in the first place!) so they will also find the
dependencies of those libs.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'