[Buildroot] [pull request] Pull request for branch for-2012.05/perl

[Buildroot] [pull request] Pull request for branch for-2012.05/perl

[Thomas Petazzoni]

The following changes since commit b5fd0f2709cab72f48da12a5804dd8670fffbf6e:

  Change the /etc/mtab symlink to use an absolute path (2012-03-19 17:08:13 +0100)

are available in the git repository at:
  git://git.free-electrons.com/users/thomas-petazzoni/buildroot.git for-2012.05/perl

Thomas Petazzoni (2):
      microperl: fix MICROPERL_INSTALL_TARGET_CMDS
      microperl: install host-microperl in $(HOST_DIR)/opt/perl

 package/microperl/microperl.mk |   28 ++++++++++++++++++----------
 1 files changed, 18 insertions(+), 10 deletions(-)

Thanks,
-- 
Thomas Petazzoni

[Buildroot] [PATCH 1/2] microperl: fix MICROPERL_INSTALL_TARGET_CMDS

[Thomas Petazzoni]

The MICROPERL_INSTALL_TARGET_CMDS used the following construct in a
for loop:

		[ -d $(@D)/lib/$$j ] && cp -af $(@D)/lib/$$j \
			$(TARGET_DIR)/$(MICROPERL_MODS_DIR) ; \
		[ -f $(@D)/lib/$$i ] && $(INSTALL) -m 0644 -D $(@D)/lib/$$i \
			$(TARGET_DIR)/$(MICROPERL_MODS_DIR)/$$i; \

The problem is that when at the last iteration, the second test (-f)
fails, then the whole loop ends with a non-zero error code, and makes
aborts the build. This happens for example if the last Perl modules in
the list is Time::Local, because such modules are taken care of by the
first condition (that copies a complete directory).

By moving to full if statements, we ensure that the return code is
zero even if the condition was false.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/microperl/microperl.mk |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/package/microperl/microperl.mk b/package/microperl/microperl.mk
index d5b5a7c..5bf4a2e 100644
--- a/package/microperl/microperl.mk
+++ b/package/microperl/microperl.mk
@@ -132,10 +132,12 @@ define MICROPERL_INSTALL_TARGET_CMDS
 	$(MICROPERL_BUILD_EXTENSIONS)
 	for i in $(MICROPERL_MODS); do \
 		j=`echo $$i|cut -d : -f 1` ; \
-		[ -d $(@D)/lib/$$j ] && cp -af $(@D)/lib/$$j \
-			$(TARGET_DIR)/$(MICROPERL_MODS_DIR) ; \
-		[ -f $(@D)/lib/$$i ] && $(INSTALL) -m 0644 -D $(@D)/lib/$$i \
-			$(TARGET_DIR)/$(MICROPERL_MODS_DIR)/$$i; \
+		if [ -d $(@D)/lib/$$j ] ; then \
+			cp -af $(@D)/lib/$$j $(TARGET_DIR)/$(MICROPERL_MODS_DIR) ; \
+		fi ; \
+		if [ -f $(@D)/lib/$$i ] ; then \
+			$(INSTALL) -m 0644 -D $(@D)/lib/$$i $(TARGET_DIR)/$(MICROPERL_MODS_DIR)/$$i; \
+		fi ; \
 	done
 	# Remove test files
 	find $(TARGET_DIR)/$(MICROPERL_MODS_DIR) -type f -name *.t \
-- 
1.7.4.1

[Buildroot] [PATCH 1/2] microperl: fix MICROPERL_INSTALL_TARGET_CMDS

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 Thomas> The MICROPERL_INSTALL_TARGET_CMDS used the following construct in a
 Thomas> for loop:

 Thomas> 		[ -d $(@D)/lib/$$j ] && cp -af $(@D)/lib/$$j \
 Thomas> 			$(TARGET_DIR)/$(MICROPERL_MODS_DIR) ; \
 Thomas> 		[ -f $(@D)/lib/$$i ] && $(INSTALL) -m 0644 -D $(@D)/lib/$$i \
 Thomas> 			$(TARGET_DIR)/$(MICROPERL_MODS_DIR)/$$i; \

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Thomas Petazzoni]

This commit is a tentative at fixing some Perl-related build problems
we are having nowadays, due to the fact that microperl now depends on
host-microperl. Since the perl binary from host-microperl is installed
in $(HOST_DIR)/usr/bin, it gets picked up when we run host tools like
host-intltool, which were normally meant to be used against the
distribution-provided Perl.

This problem currently causes between a third to half of the failures
in our random build tests.

Another, cleaner, option would be to make host-intltool depend on
host-microperl, but this would require every user needing intltool to
build a host Perl interpreter, even if the real reason for build the
host Perl interpreter is only needed for those who want to build the
Perl interpreter for the target. This commit is really a test at
trying another solution than adding this dependency, to see whether it
turns out to be a reasonable trade-off or not. Later developments and
improvements in the Perl area may require to switch to a different
solution than the one implemented by this patch.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/microperl/microperl.mk |   18 ++++++++++++------
 1 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/package/microperl/microperl.mk b/package/microperl/microperl.mk
index 5bf4a2e..31b7a63 100644
--- a/package/microperl/microperl.mk
+++ b/package/microperl/microperl.mk
@@ -22,10 +22,16 @@ MICROPERL_MODS += constant.pm CGI CGI.pm Carp.pm Exporter.pm overload.pm \
 	vars.pm warnings.pm warnings/register.pm
 endif
 
-# Host microperl is actually full-blown perl
+# Host microperl is actually full-blown perl. Instead of installing it
+# with prefix=$(HOST_DIR)/usr, we install it with
+# prefix=$(HOST_DIR)/opt/perl so that this host-microperl is only used
+# for building the target microperl. If we install the host-microperl
+# in $(HOST_DIR)/usr, it gets picked up by all Perl-based utilities
+# (such as intltool) that Buildroot has installed, creating
+# incompatibilites with the Perl provided by the distribution.
 define HOST_MICROPERL_CONFIGURE_CMDS
 	cd $(@D) ; \
-	./Configure -Dcc="$(HOSTCC)" -Dprefix="$(HOST_DIR)/usr" \
+	./Configure -Dcc="$(HOSTCC)" -Dprefix="$(HOST_DIR)/opt/perl/" \
 		-Dloclibpth='/lib /lib64 /usr/lib /usr/lib64' -des
 endef
 
@@ -95,8 +101,8 @@ define MICROPERL_CONFIGURE_CMDS
 	cp -f $(@D)/uconfig.sh $(@D)/config.sh
 	echo "ccname='$(TARGET_CC)'" >>$(@D)/config.sh
 	echo "PERL_CONFIG_SH=true" >>$(@D)/config.sh
-	cd $(@D) ; $(HOST_DIR)/usr/bin/perl make_patchnum.pl ; \
-	$(HOST_DIR)/usr/bin/perl configpm
+	cd $(@D) ; $(HOST_DIR)/opt/perl/bin/perl make_patchnum.pl ; \
+	$(HOST_DIR)/opt/perl/bin/perl configpm
 endef
 
 define MICROPERL_BUILD_CMDS
@@ -111,9 +117,9 @@ endef
 # Just ignore make_ext.pl warning/errors
 define MICROPERL_BUILD_EXTENSIONS
 	for i in $(MICROPERL_MODS); do \
-	cd $(@D); ln -sf $(HOST_DIR)/usr/bin/perl miniperl; \
+	cd $(@D); ln -sf $(HOST_DIR)/opt/perl/bin/perl miniperl; \
 		PERL5LIB=$(TARGET_DIR)/$(MICROPERL_ARCH_DIR) \
-		$(HOST_DIR)/usr/bin/perl make_ext.pl MAKE="$(MAKE)" --nonxs \
+		$(HOST_DIR)/opt/perl/bin/perl make_ext.pl MAKE="$(MAKE)" --nonxs \
 		`echo $$i|sed -e 's/.pm//'`; \
 	done
 endef
-- 
1.7.4.1

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Thomas Petazzoni]

Le Tue, 27 Mar 2012 09:06:18 +0200,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a ?crit :

> This commit is a tentative at fixing some Perl-related build problems
> we are having nowadays, due to the fact that microperl now depends on
> host-microperl. Since the perl binary from host-microperl is installed
> in $(HOST_DIR)/usr/bin, it gets picked up when we run host tools like
> host-intltool, which were normally meant to be used against the
> distribution-provided Perl.
> 
> This problem currently causes between a third to half of the failures
> in our random build tests.
> 
> Another, cleaner, option would be to make host-intltool depend on
> host-microperl, but this would require every user needing intltool to
> build a host Perl interpreter, even if the real reason for build the
> host Perl interpreter is only needed for those who want to build the
> Perl interpreter for the target. This commit is really a test at
> trying another solution than adding this dependency, to see whether it
> turns out to be a reasonable trade-off or not. Later developments and
> improvements in the Perl area may require to switch to a different
> solution than the one implemented by this patch.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

I have tested
http://autobuild.buildroot.org/results/9980055bf8ebc1ea442f5affc3418eb869c4c90d/defconfig
without the patch, in which case we have the host-intltool problem.
With this patch, this problem disappears and this defconfig builds
until the end with no problem.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 Thomas> This commit is a tentative at fixing some Perl-related build problems
 Thomas> we are having nowadays, due to the fact that microperl now depends on
 Thomas> host-microperl. Since the perl binary from host-microperl is installed
 Thomas> in $(HOST_DIR)/usr/bin, it gets picked up when we run host tools like
 Thomas> host-intltool, which were normally meant to be used against the
 Thomas> distribution-provided Perl.

Thanks, but I have fixed it a bit differently instead:

commit d0e5eb281f0e3b323ecb3446c1b16baf7f3baa69
Author: Peter Korsgaard <jacmet@sunsite.dk>
Date:   Tue Mar 27 17:11:36 2012 +0200

    libxml-parser-perl: fix host-intltool breakage when microperl is enabled
    
    Microperl will build host-microperl and install it into HOST_DIR/usr/bin,
    where other packages will pick it up as _CONFIGURE_OPTS / _MAKE_ENV
    prepends that to the path.
    
    libxml-parser-perl didn't though, so it would still be built against
    the system perl, causing host-intltool to fail when it would use
    host-microperl together with libxml-parser-perl if the system perl
    isn't compatible with host-microperl.
    
    Fix it by using HOST_CONFIGURE_OPTS and ensuring it is built after
    (host-)microperl if enabled.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Thomas Petazzoni]

Hello Peter,

Le Wed, 28 Mar 2012 08:54:32 +0200,
Peter Korsgaard <jacmet@uclibc.org> a ?crit :

> Thanks, but I have fixed it a bit differently instead:
> 
> commit d0e5eb281f0e3b323ecb3446c1b16baf7f3baa69
> Author: Peter Korsgaard <jacmet@sunsite.dk>
> Date:   Tue Mar 27 17:11:36 2012 +0200

Thanks, yes. As discussed on IRC, I haven't tested your change, but I
agree on the principle.

However, I am worried by the case reported by Will Newton in:

From: Will Newton <will.newton@gmail.com>
To: buildroot at busybox.net
Subject: [Buildroot] Host libxml-parser-perl build issue
Date: Mon, 26 Mar 2012 17:35:29 +0100

In his case, the microperl for the target was not selected, so the
host-microperl was not built, and still he was having issues.
Apparently, the problem is that host-libxml-parser-perl didn't pick up
the libexpat from the $(HOST_DIR). Normally, we build all binaries with
a rpath set to $(HOST_DIR)/usr/lib, but in the specific case of Perl
modules that use a native library such as libexpat, I am not sure how
we are supposed to tell Perl to build such modules with an rpath set.
The current workaround of Will is to pass the LD_LIBRARY_PATH
environment variable in TARGET_CONFIGURE_OPTS and TARGET_MAKE_ENV, but
I don't think this would work for all packages (I remember libtool
being confused by a LD_LIBRARY_PATH being set).

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

Hi,

 Thomas> However, I am worried by the case reported by Will Newton in:

 Thomas> From: Will Newton <will.newton@gmail.com>
 Thomas> To: buildroot at busybox.net
 Thomas> Subject: [Buildroot] Host libxml-parser-perl build issue
 Thomas> Date: Mon, 26 Mar 2012 17:35:29 +0100

 Thomas> In his case, the microperl for the target was not selected, so the
 Thomas> host-microperl was not built, and still he was having issues.

Yes, I've seen it. I don't really have a good idea about what we can do
about it though :/

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Will Newton]

On Wed, Mar 28, 2012 at 10:32 AM, Peter Korsgaard <jacmet@uclibc.org> wrote:

Hi Peter,

>>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
>
> Hi,
>
> ?Thomas> However, I am worried by the case reported by Will Newton in:
>
> ?Thomas> From: Will Newton <will.newton@gmail.com>
> ?Thomas> To: buildroot at busybox.net
> ?Thomas> Subject: [Buildroot] Host libxml-parser-perl build issue
> ?Thomas> Date: Mon, 26 Mar 2012 17:35:29 +0100
>
> ?Thomas> In his case, the microperl for the target was not selected, so the
> ?Thomas> host-microperl was not built, and still he was having issues.
>
> Yes, I've seen it. I don't really have a good idea about what we can do
> about it though :/

Do you think that setting LD_LIBRARY_PATH in the way I suggested is
unsafe? I had some concerns that it might affect the search path for
libraries but I believe this is not an issue with a cross linker
(according to the ld man page, I haven't checked the code), and it
does not break any of the test builds I have done.

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Thomas Petazzoni]

Le Wed, 28 Mar 2012 11:40:00 +0100,
Will Newton <will.newton@gmail.com> a ?crit :

> Do you think that setting LD_LIBRARY_PATH in the way I suggested is
> unsafe? I had some concerns that it might affect the search path for
> libraries but I believe this is not an issue with a cross linker
> (according to the ld man page, I haven't checked the code), and it
> does not break any of the test builds I have done.

In the past, we had the problem that when we were building target
packages that required host utilities installed in $(HOST_DIR)/usr/bin,
those utilities were not able to find their libraries in
$(HOST_DIR)/usr/lib. We tried using LD_LIBRARY_PATH, but libtool used
it as the search path for libraries when compiling target stuff. So, we
decided to hardcode the correct rpath in all binaries built and
installed in $(HOST_DIR)/usr/bin.

So, yes, I believe that adding LD_LIBRARY_PATH is going to cause
problems, unless those problems have disappeared for some other reason.

See:

 c1b6242fdcf2cff7ebf09fec4cc1be58963e8427
 0d1830b07db4ebfd14e77a258de6fb391e57e960
 6b939d40f6a29a43277566adc9d4312d49cb3abf

For a history of what we tried on this topic.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Will Newton]

On Wed, Mar 28, 2012 at 11:55 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Le Wed, 28 Mar 2012 11:40:00 +0100,
> Will Newton <will.newton@gmail.com> a ?crit :
>
>> Do you think that setting LD_LIBRARY_PATH in the way I suggested is
>> unsafe? I had some concerns that it might affect the search path for
>> libraries but I believe this is not an issue with a cross linker
>> (according to the ld man page, I haven't checked the code), and it
>> does not break any of the test builds I have done.
>
> In the past, we had the problem that when we were building target
> packages that required host utilities installed in $(HOST_DIR)/usr/bin,
> those utilities were not able to find their libraries in
> $(HOST_DIR)/usr/lib. We tried using LD_LIBRARY_PATH, but libtool used
> it as the search path for libraries when compiling target stuff. So, we
> decided to hardcode the correct rpath in all binaries built and
> installed in $(HOST_DIR)/usr/bin.
>
> So, yes, I believe that adding LD_LIBRARY_PATH is going to cause
> problems, unless those problems have disappeared for some other reason.
>
> See:
>
> ?c1b6242fdcf2cff7ebf09fec4cc1be58963e8427
> ?0d1830b07db4ebfd14e77a258de6fb391e57e960
> ?6b939d40f6a29a43277566adc9d4312d49cb3abf
>
> For a history of what we tried on this topic.

If ld handles this explicitly and correctly then this sounds like a
libtool bug to me. Do you have a testcase that can reproduce it? I
have spent some time trying to make libtool break but without success,
but I am far from an expert in that area...

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Thomas Petazzoni]

Le Wed, 28 Mar 2012 12:55:47 +0200,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a ?crit :

> In the past, we had the problem that when we were building target
> packages that required host utilities installed in $(HOST_DIR)/usr/bin,
> those utilities were not able to find their libraries in
> $(HOST_DIR)/usr/lib. We tried using LD_LIBRARY_PATH, but libtool used
> it as the search path for libraries when compiling target stuff. So, we
> decided to hardcode the correct rpath in all binaries built and
> installed in $(HOST_DIR)/usr/bin.

I had a quick look at this, and here the Perl dynamic module that
depends on libexpat.so.1 is built with the correct RPATH:

$ arm-linux-gnueabi-readelf -a host/usr/lib/perl/auto/XML/Parser/Expat/Expat.so | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/opt/outputs/perl/host/usr/lib]

So normally, when this Expat.so library is dlopen'ed by Perl, it should
use the RPATH of Expat.so to find where libexpat.so.1 is located. And
on my system, it does:

$ PERLLIB=/opt/outputs/perl/host/usr/lib/perl/ LD_DEBUG=libs perl -e 'require XML::Parser;'
      9529:	find library=libperl.so.5.10 [0]; searching
      9529:	 search cache=/etc/ld.so.cache
      9529:	  trying file=/usr/lib/libperl.so.5.10
      9529:	
      9529:	find library=libdl.so.2 [0]; searching
      9529:	 search cache=/etc/ld.so.cache
      9529:	  trying file=/lib/x86_64-linux-gnu/libdl.so.2
      9529:	
      9529:	find library=libm.so.6 [0]; searching
      9529:	 search cache=/etc/ld.so.cache
      9529:	  trying file=/lib/x86_64-linux-gnu/libm.so.6
      9529:	
      9529:	find library=libpthread.so.0 [0]; searching
      9529:	 search cache=/etc/ld.so.cache
      9529:	  trying file=/lib/x86_64-linux-gnu/libpthread.so.0
      9529:	
      9529:	find library=libc.so.6 [0]; searching
      9529:	 search cache=/etc/ld.so.cache
      9529:	  trying file=/lib/x86_64-linux-gnu/libc.so.6
      9529:	
      9529:	find library=libcrypt.so.1 [0]; searching
      9529:	 search cache=/etc/ld.so.cache
      9529:	  trying file=/lib/x86_64-linux-gnu/libcrypt.so.1
      9529:	
      9529:	
      9529:	calling init: /lib/x86_64-linux-gnu/libpthread.so.0
      9529:	calling init: /lib/x86_64-linux-gnu/libc.so.6
      9529:	calling init: /lib/x86_64-linux-gnu/libcrypt.so.1
      9529:	calling init: /lib/x86_64-linux-gnu/libm.so.6
      9529:	calling init: /lib/x86_64-linux-gnu/libdl.so.2
      9529:	calling init: /usr/lib/libperl.so.5.10
      9529:	
      9529:	
      9529:	initialize program: perl
      9529:	transferring control: perl
      9529:	
      9529:	find library=libexpat.so.1 [0]; searching
      9529:	 search path=/opt/outputs/perl/host/usr/lib/tls/x86_64:/opt/outputs/perl/host/usr/lib/tls:/opt/outputs/perl/host/usr/lib/x86_64:/opt/outputs/perl/host/usr/lib		(RPATH from file /opt/outputs/perl/host/usr/lib/perl//auto/XML/Parser/Expat/Expat.so)
      9529:	  trying file=/opt/outputs/perl/host/usr/lib/tls/x86_64/libexpat.so.1
      9529:	  trying file=/opt/outputs/perl/host/usr/lib/tls/libexpat.so.1
      9529:	  trying file=/opt/outputs/perl/host/usr/lib/x86_64/libexpat.so.1
      9529:	  trying file=/opt/outputs/perl/host/usr/lib/libexpat.so.1
      9529:	
      9529:	calling init: /opt/outputs/perl/host/usr/lib/libexpat.so.1
      9529:	calling init: /opt/outputs/perl/host/usr/lib/perl//auto/XML/Parser/Expat/Expat.so
      9529:	calling fini: perl [0]
      9529:	calling fini: /usr/lib/libperl.so.5.10 [0]
      9529:	calling fini: /lib/x86_64-linux-gnu/libdl.so.2 [0]
      9529:	calling fini: /lib/x86_64-linux-gnu/libm.so.6 [0]
      9529:	calling fini: /lib/x86_64-linux-gnu/libpthread.so.0 [0]
      9529:	calling fini: /lib/x86_64-linux-gnu/libcrypt.so.1 [0]
      9529:	calling fini: /opt/outputs/perl/host/usr/lib/perl//auto/XML/Parser/Expat/Expat.so [0]
      9529:	calling fini: /opt/outputs/perl/host/usr/lib/libexpat.so.1 [0]
      9529:	calling fini: /lib/x86_64-linux-gnu/libc.so.6 [0]

As you can see here, it clearly looks in the RPATH encoded into
Expat.so to find where libexpat.so.1 is located. In my
case /opt/outputs/perl/ in the Buildroot output directory, and it
clearly uses the libexpat.so.1 from this output directory, and not the
one from my system.

Are you sure you don't have any value set in your LD_LIBRARY_PATH
environment variable (outside of Buildroot)? Can you try the commands
above (check the RPATH of Expat.so and check the resolution of
libraries when asking Perl to require the XML::Parser module, when
PERLLIB points to the Perl modules built by Buildroot), and post the
results?

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Will Newton]

On Wed, Mar 28, 2012 at 1:52 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Le Wed, 28 Mar 2012 12:55:47 +0200,
> Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a ?crit :
>
>> In the past, we had the problem that when we were building target
>> packages that required host utilities installed in $(HOST_DIR)/usr/bin,
>> those utilities were not able to find their libraries in
>> $(HOST_DIR)/usr/lib. We tried using LD_LIBRARY_PATH, but libtool used
>> it as the search path for libraries when compiling target stuff. So, we
>> decided to hardcode the correct rpath in all binaries built and
>> installed in $(HOST_DIR)/usr/bin.
>
> I had a quick look at this, and here the Perl dynamic module that
> depends on libexpat.so.1 is built with the correct RPATH:
>
> $ arm-linux-gnueabi-readelf -a host/usr/lib/perl/auto/XML/Parser/Expat/Expat.so | grep RPATH
> ?0x000000000000000f (RPATH) ? ? ? ? ? ? ?Library rpath: [/opt/outputs/perl/host/usr/lib]
>
> So normally, when this Expat.so library is dlopen'ed by Perl, it should
> use the RPATH of Expat.so to find where libexpat.so.1 is located. And
> on my system, it does:
>
> $ PERLLIB=/opt/outputs/perl/host/usr/lib/perl/ LD_DEBUG=libs perl -e 'require XML::Parser;'
> ? ? ?9529: ? ? find library=libperl.so.5.10 [0]; searching
> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
> ? ? ?9529: ? ? ? trying file=/usr/lib/libperl.so.5.10
> ? ? ?9529:
> ? ? ?9529: ? ? find library=libdl.so.2 [0]; searching
> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
> ? ? ?9529: ? ? ? trying file=/lib/x86_64-linux-gnu/libdl.so.2
> ? ? ?9529:
> ? ? ?9529: ? ? find library=libm.so.6 [0]; searching
> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
> ? ? ?9529: ? ? ? trying file=/lib/x86_64-linux-gnu/libm.so.6
> ? ? ?9529:
> ? ? ?9529: ? ? find library=libpthread.so.0 [0]; searching
> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
> ? ? ?9529: ? ? ? trying file=/lib/x86_64-linux-gnu/libpthread.so.0
> ? ? ?9529:
> ? ? ?9529: ? ? find library=libc.so.6 [0]; searching
> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
> ? ? ?9529: ? ? ? trying file=/lib/x86_64-linux-gnu/libc.so.6
> ? ? ?9529:
> ? ? ?9529: ? ? find library=libcrypt.so.1 [0]; searching
> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
> ? ? ?9529: ? ? ? trying file=/lib/x86_64-linux-gnu/libcrypt.so.1
> ? ? ?9529:
> ? ? ?9529:
> ? ? ?9529: ? ? calling init: /lib/x86_64-linux-gnu/libpthread.so.0
> ? ? ?9529: ? ? calling init: /lib/x86_64-linux-gnu/libc.so.6
> ? ? ?9529: ? ? calling init: /lib/x86_64-linux-gnu/libcrypt.so.1
> ? ? ?9529: ? ? calling init: /lib/x86_64-linux-gnu/libm.so.6
> ? ? ?9529: ? ? calling init: /lib/x86_64-linux-gnu/libdl.so.2
> ? ? ?9529: ? ? calling init: /usr/lib/libperl.so.5.10
> ? ? ?9529:
> ? ? ?9529:
> ? ? ?9529: ? ? initialize program: perl
> ? ? ?9529: ? ? transferring control: perl
> ? ? ?9529:
> ? ? ?9529: ? ? find library=libexpat.so.1 [0]; searching
> ? ? ?9529: ? ? ?search path=/opt/outputs/perl/host/usr/lib/tls/x86_64:/opt/outputs/perl/host/usr/lib/tls:/opt/outputs/perl/host/usr/lib/x86_64:/opt/outputs/perl/host/usr/lib ? ? ? ? ?(RPATH from file /opt/outputs/perl/host/usr/lib/perl//auto/XML/Parser/Expat/Expat.so)
> ? ? ?9529: ? ? ? trying file=/opt/outputs/perl/host/usr/lib/tls/x86_64/libexpat.so.1
> ? ? ?9529: ? ? ? trying file=/opt/outputs/perl/host/usr/lib/tls/libexpat.so.1
> ? ? ?9529: ? ? ? trying file=/opt/outputs/perl/host/usr/lib/x86_64/libexpat.so.1
> ? ? ?9529: ? ? ? trying file=/opt/outputs/perl/host/usr/lib/libexpat.so.1
> ? ? ?9529:
> ? ? ?9529: ? ? calling init: /opt/outputs/perl/host/usr/lib/libexpat.so.1
> ? ? ?9529: ? ? calling init: /opt/outputs/perl/host/usr/lib/perl//auto/XML/Parser/Expat/Expat.so
> ? ? ?9529: ? ? calling fini: perl [0]
> ? ? ?9529: ? ? calling fini: /usr/lib/libperl.so.5.10 [0]
> ? ? ?9529: ? ? calling fini: /lib/x86_64-linux-gnu/libdl.so.2 [0]
> ? ? ?9529: ? ? calling fini: /lib/x86_64-linux-gnu/libm.so.6 [0]
> ? ? ?9529: ? ? calling fini: /lib/x86_64-linux-gnu/libpthread.so.0 [0]
> ? ? ?9529: ? ? calling fini: /lib/x86_64-linux-gnu/libcrypt.so.1 [0]
> ? ? ?9529: ? ? calling fini: /opt/outputs/perl/host/usr/lib/perl//auto/XML/Parser/Expat/Expat.so [0]
> ? ? ?9529: ? ? calling fini: /opt/outputs/perl/host/usr/lib/libexpat.so.1 [0]
> ? ? ?9529: ? ? calling fini: /lib/x86_64-linux-gnu/libc.so.6 [0]
>
> As you can see here, it clearly looks in the RPATH encoded into
> Expat.so to find where libexpat.so.1 is located. In my
> case /opt/outputs/perl/ in the Buildroot output directory, and it
> clearly uses the libexpat.so.1 from this output directory, and not the
> one from my system.
>
> Are you sure you don't have any value set in your LD_LIBRARY_PATH
> environment variable (outside of Buildroot)? Can you try the commands
> above (check the RPATH of Expat.so and check the resolution of
> libraries when asking Perl to require the XML::Parser module, when
> PERLLIB points to the Perl modules built by Buildroot), and post the
> results?

I haven't got any LD_LIBRARY_PATH set, but it looks like there is no
rpath set. I wonder if this is some perl Makefile issue.

[win at lemeta01 buildroot]$ env | grep LD_LIBRARY
[win at lemeta01 buildroot]$ ./output/host/usr/bin/arm-linux-readelf -a
./output/host/usr/lib/perl/auto/XML/Parser/Expat/Expat.so | grep RPATH
[win at lemeta01 buildroot]$

Although the makefiles do mention rpath, but that one does not seem to
be getting used either:

[win at lemeta01 host-libxml-parser-perl-2.36]$ grep -ri rpath *
Expat/Makefile:CCDLFLAGS = -Wl,-E
-Wl,-rpath,/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE
Makefile:CCDLFLAGS = -Wl,-E
-Wl,-rpath,/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Will Newton]

On Wed, Mar 28, 2012 at 2:01 PM, Will Newton <will.newton@gmail.com> wrote:
> On Wed, Mar 28, 2012 at 1:52 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>> Le Wed, 28 Mar 2012 12:55:47 +0200,
>> Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a ?crit :
>>
>>> In the past, we had the problem that when we were building target
>>> packages that required host utilities installed in $(HOST_DIR)/usr/bin,
>>> those utilities were not able to find their libraries in
>>> $(HOST_DIR)/usr/lib. We tried using LD_LIBRARY_PATH, but libtool used
>>> it as the search path for libraries when compiling target stuff. So, we
>>> decided to hardcode the correct rpath in all binaries built and
>>> installed in $(HOST_DIR)/usr/bin.
>>
>> I had a quick look at this, and here the Perl dynamic module that
>> depends on libexpat.so.1 is built with the correct RPATH:
>>
>> $ arm-linux-gnueabi-readelf -a host/usr/lib/perl/auto/XML/Parser/Expat/Expat.so | grep RPATH
>> ?0x000000000000000f (RPATH) ? ? ? ? ? ? ?Library rpath: [/opt/outputs/perl/host/usr/lib]
>>
>> So normally, when this Expat.so library is dlopen'ed by Perl, it should
>> use the RPATH of Expat.so to find where libexpat.so.1 is located. And
>> on my system, it does:
>>
>> $ PERLLIB=/opt/outputs/perl/host/usr/lib/perl/ LD_DEBUG=libs perl -e 'require XML::Parser;'
>> ? ? ?9529: ? ? find library=libperl.so.5.10 [0]; searching
>> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
>> ? ? ?9529: ? ? ? trying file=/usr/lib/libperl.so.5.10
>> ? ? ?9529:
>> ? ? ?9529: ? ? find library=libdl.so.2 [0]; searching
>> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
>> ? ? ?9529: ? ? ? trying file=/lib/x86_64-linux-gnu/libdl.so.2
>> ? ? ?9529:
>> ? ? ?9529: ? ? find library=libm.so.6 [0]; searching
>> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
>> ? ? ?9529: ? ? ? trying file=/lib/x86_64-linux-gnu/libm.so.6
>> ? ? ?9529:
>> ? ? ?9529: ? ? find library=libpthread.so.0 [0]; searching
>> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
>> ? ? ?9529: ? ? ? trying file=/lib/x86_64-linux-gnu/libpthread.so.0
>> ? ? ?9529:
>> ? ? ?9529: ? ? find library=libc.so.6 [0]; searching
>> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
>> ? ? ?9529: ? ? ? trying file=/lib/x86_64-linux-gnu/libc.so.6
>> ? ? ?9529:
>> ? ? ?9529: ? ? find library=libcrypt.so.1 [0]; searching
>> ? ? ?9529: ? ? ?search cache=/etc/ld.so.cache
>> ? ? ?9529: ? ? ? trying file=/lib/x86_64-linux-gnu/libcrypt.so.1
>> ? ? ?9529:
>> ? ? ?9529:
>> ? ? ?9529: ? ? calling init: /lib/x86_64-linux-gnu/libpthread.so.0
>> ? ? ?9529: ? ? calling init: /lib/x86_64-linux-gnu/libc.so.6
>> ? ? ?9529: ? ? calling init: /lib/x86_64-linux-gnu/libcrypt.so.1
>> ? ? ?9529: ? ? calling init: /lib/x86_64-linux-gnu/libm.so.6
>> ? ? ?9529: ? ? calling init: /lib/x86_64-linux-gnu/libdl.so.2
>> ? ? ?9529: ? ? calling init: /usr/lib/libperl.so.5.10
>> ? ? ?9529:
>> ? ? ?9529:
>> ? ? ?9529: ? ? initialize program: perl
>> ? ? ?9529: ? ? transferring control: perl
>> ? ? ?9529:
>> ? ? ?9529: ? ? find library=libexpat.so.1 [0]; searching
>> ? ? ?9529: ? ? ?search path=/opt/outputs/perl/host/usr/lib/tls/x86_64:/opt/outputs/perl/host/usr/lib/tls:/opt/outputs/perl/host/usr/lib/x86_64:/opt/outputs/perl/host/usr/lib ? ? ? ? ?(RPATH from file /opt/outputs/perl/host/usr/lib/perl//auto/XML/Parser/Expat/Expat.so)
>> ? ? ?9529: ? ? ? trying file=/opt/outputs/perl/host/usr/lib/tls/x86_64/libexpat.so.1
>> ? ? ?9529: ? ? ? trying file=/opt/outputs/perl/host/usr/lib/tls/libexpat.so.1
>> ? ? ?9529: ? ? ? trying file=/opt/outputs/perl/host/usr/lib/x86_64/libexpat.so.1
>> ? ? ?9529: ? ? ? trying file=/opt/outputs/perl/host/usr/lib/libexpat.so.1
>> ? ? ?9529:
>> ? ? ?9529: ? ? calling init: /opt/outputs/perl/host/usr/lib/libexpat.so.1
>> ? ? ?9529: ? ? calling init: /opt/outputs/perl/host/usr/lib/perl//auto/XML/Parser/Expat/Expat.so
>> ? ? ?9529: ? ? calling fini: perl [0]
>> ? ? ?9529: ? ? calling fini: /usr/lib/libperl.so.5.10 [0]
>> ? ? ?9529: ? ? calling fini: /lib/x86_64-linux-gnu/libdl.so.2 [0]
>> ? ? ?9529: ? ? calling fini: /lib/x86_64-linux-gnu/libm.so.6 [0]
>> ? ? ?9529: ? ? calling fini: /lib/x86_64-linux-gnu/libpthread.so.0 [0]
>> ? ? ?9529: ? ? calling fini: /lib/x86_64-linux-gnu/libcrypt.so.1 [0]
>> ? ? ?9529: ? ? calling fini: /opt/outputs/perl/host/usr/lib/perl//auto/XML/Parser/Expat/Expat.so [0]
>> ? ? ?9529: ? ? calling fini: /opt/outputs/perl/host/usr/lib/libexpat.so.1 [0]
>> ? ? ?9529: ? ? calling fini: /lib/x86_64-linux-gnu/libc.so.6 [0]
>>
>> As you can see here, it clearly looks in the RPATH encoded into
>> Expat.so to find where libexpat.so.1 is located. In my
>> case /opt/outputs/perl/ in the Buildroot output directory, and it
>> clearly uses the libexpat.so.1 from this output directory, and not the
>> one from my system.
>>
>> Are you sure you don't have any value set in your LD_LIBRARY_PATH
>> environment variable (outside of Buildroot)? Can you try the commands
>> above (check the RPATH of Expat.so and check the resolution of
>> libraries when asking Perl to require the XML::Parser module, when
>> PERLLIB points to the Perl modules built by Buildroot), and post the
>> results?
>
> I haven't got any LD_LIBRARY_PATH set, but it looks like there is no
> rpath set. I wonder if this is some perl Makefile issue.
>
> [win at lemeta01 buildroot]$ env | grep LD_LIBRARY
> [win at lemeta01 buildroot]$ ./output/host/usr/bin/arm-linux-readelf -a
> ./output/host/usr/lib/perl/auto/XML/Parser/Expat/Expat.so | grep RPATH
> [win at lemeta01 buildroot]$
>
> Although the makefiles do mention rpath, but that one does not seem to
> be getting used either:
>
> [win at lemeta01 host-libxml-parser-perl-2.36]$ grep -ri rpath *
> Expat/Makefile:CCDLFLAGS = -Wl,-E
> -Wl,-rpath,/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE
> Makefile:CCDLFLAGS = -Wl,-E
> -Wl,-rpath,/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE

However it does use LD_RUN_PATH:

[win at lemeta01 host-libxml-parser-perl-2.36]$ grep -r LD_RUN_PATH *
Expat/Makefile:LD_RUN_PATH = /meta/home/win/buildroot/output/host/usr/lib

If I change that line to "export LD_RUN_PATH..." it seems to set the
rpath and work correctly...

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Thomas Petazzoni]

Le Wed, 28 Mar 2012 14:01:56 +0100,
Will Newton <will.newton@gmail.com> a ?crit :

> [win at lemeta01 buildroot]$ env | grep LD_LIBRARY
> [win at lemeta01 buildroot]$ ./output/host/usr/bin/arm-linux-readelf -a
> ./output/host/usr/lib/perl/auto/XML/Parser/Expat/Expat.so | grep RPATH
> [win at lemeta01 buildroot]$

Ah, ah, this is our problem.

> Although the makefiles do mention rpath, but that one does not seem to
> be getting used either:

We have a rpath set in HOST_LDFLAGS, but this variable doesn't seem to
be used when building host-libxml-parser-perl. However, by
Expat/Makefile contains LD_RUN_PATH="/opt/outputs/perl/host/usr/lib" in
the environment when calling ld:

LD_RUN_PATH="/opt/outputs/perl/host/usr/lib" cc  -shared -O2 -g -L/usr/local/lib -fstack-protector Expat.o  -o ../blib/arch/auto/XML/Parser/Expat/Expat.so 	\
	   -L/opt/outputs/perl/host/usr/lib -lexpat

And LD_RUN_PATH is used by ld as the RPATH when no -rpath option is
used. I guess this LD_RUN_PATH is set because we run the Makefile.PL
script with INSTALLSITELIB=/opt/outputs/perl/host/usr/lib/perl.

Could you post the output of:

rm -rf output/build/host-libxml-parser-perl*
make

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Will Newton]

On Wed, Mar 28, 2012 at 2:18 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Le Wed, 28 Mar 2012 14:01:56 +0100,
> Will Newton <will.newton@gmail.com> a ?crit :
>
>> [win at lemeta01 buildroot]$ env | grep LD_LIBRARY
>> [win at lemeta01 buildroot]$ ./output/host/usr/bin/arm-linux-readelf -a
>> ./output/host/usr/lib/perl/auto/XML/Parser/Expat/Expat.so | grep RPATH
>> [win at lemeta01 buildroot]$
>
> Ah, ah, this is our problem.
>
>> Although the makefiles do mention rpath, but that one does not seem to
>> be getting used either:
>
> We have a rpath set in HOST_LDFLAGS, but this variable doesn't seem to
> be used when building host-libxml-parser-perl. However, by
> Expat/Makefile contains LD_RUN_PATH="/opt/outputs/perl/host/usr/lib" in
> the environment when calling ld:
>
> LD_RUN_PATH="/opt/outputs/perl/host/usr/lib" cc ?-shared -O2 -g -L/usr/local/lib -fstack-protector Expat.o ?-o ../blib/arch/auto/XML/Parser/Expat/Expat.so ? ? ?\
> ? ? ? ? ? -L/opt/outputs/perl/host/usr/lib -lexpat
>
> And LD_RUN_PATH is used by ld as the RPATH when no -rpath option is
> used. I guess this LD_RUN_PATH is set because we run the Makefile.PL
> script with INSTALLSITELIB=/opt/outputs/perl/host/usr/lib/perl.
>
> Could you post the output of:
>
> rm -rf output/build/host-libxml-parser-perl*
> make

I think I may have a solution. It looks like Red Hat (and hence
CentOS) remove support for LD_RUN_PATH by default:

https://bugzilla.redhat.com/show_bug.cgi?id=136009

However there is a flag to turn it back on. Setting this seems to get
me a sane runpath. Does this look ok? I assume it will silently do
nothing on non-RH systems.

[Buildroot] [PATCH 2/2] microperl: install host-microperl in $(HOST_DIR)/opt/perl

[Thomas Petazzoni]

Le Wed, 28 Mar 2012 15:00:36 +0100,
Will Newton <will.newton@gmail.com> a ?crit :

> I think I may have a solution. It looks like Red Hat (and hence
> CentOS) remove support for LD_RUN_PATH by default:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=136009
> 
> However there is a flag to turn it back on. Setting this seems to get
> me a sane runpath. Does this look ok? I assume it will silently do
> nothing on non-RH systems.

Ah, ah, it looks like a very good solution indeed.

Acked-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Thanks for the time you have spent investigating this problem!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com