[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Lionel Landwerlin]

Some time ago, an external toolchain improvement patch added
$(HOST_DIR)/bin and $(HOST_DIR)/usr/bin to the TARGET_PATH variable.
This patch also removed $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin
to TARGET_PATH which breaks the build of the WebKit package for
example.

WebKit depends on the icu package, and when configuring the package,
there is a need to run icu-config which is installed inside the
staging directory.

Signed-off-by: Lionel Landwerlin <llandwerlin@gmail.com>
---
 package/Makefile.in |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/package/Makefile.in b/package/Makefile.in
index ccd7480..57bd6f5 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -119,7 +119,7 @@ else
 TOOLCHAIN_EXTERNAL_PREFIX:=$(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_PREFIX))
 TOOLCHAIN_EXTERNAL_PATH:=$(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_PATH))
 TOOLCHAIN_DIR=$(BASE_DIR)/toolchain
-TARGET_PATH="$(HOST_DIR)/bin:$(HOST_DIR)/usr/bin:$(TOOLCHAIN_DIR)/bin:$(TOOLCHAIN_EXTERNAL_PATH)/bin:$(PATH)"
+TARGET_PATH="$(HOST_DIR)/bin:$(HOST_DIR)/usr/bin:$(TOOLCHAIN_DIR)/bin:$(TOOLCHAIN_EXTERNAL_PATH)/bin:$(STAGING_DIR)/bin:$(STAGING_DIR)/usr/bin:$(PATH)"
 #IMAGE:=$(BINARIES_DIR)/$(BR2_ROOTFS_PREFIX).$(TOOLCHAIN_EXTERNAL_PREFIX)$(ROOTFS_SUFFIX)
 IMAGE:=$(BINARIES_DIR)/$(BR2_ROOTFS_PREFIX).$(ARCH)$(ROOTFS_SUFFIX)
 
-- 
1.6.4.3

[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Peter Korsgaard]

>>>>> "Lionel" == Lionel Landwerlin <llandwerlin@gmail.com> writes:

 Lionel> Some time ago, an external toolchain improvement patch added
 Lionel> $(HOST_DIR)/bin and $(HOST_DIR)/usr/bin to the TARGET_PATH variable.
 Lionel> This patch also removed $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin
 Lionel> to TARGET_PATH which breaks the build of the WebKit package for
 Lionel> example.

 Lionel> WebKit depends on the icu package, and when configuring the package,
 Lionel> there is a need to run icu-config which is installed inside the
 Lionel> staging directory.

I don't like this, as it mixes binaries/scripts for the host and the
target - I think it's better for the icu package to install the host
tool into host/usr/bin as well instead.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Lionel Landwerlin]

Le samedi 24 octobre 2009 ? 20:37 +0200, Peter Korsgaard a ?crit :
> >>>>> "Lionel" == Lionel Landwerlin <llandwerlin@gmail.com> writes:
> 
>  Lionel> Some time ago, an external toolchain improvement patch added
>  Lionel> $(HOST_DIR)/bin and $(HOST_DIR)/usr/bin to the TARGET_PATH variable.
>  Lionel> This patch also removed $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin
>  Lionel> to TARGET_PATH which breaks the build of the WebKit package for
>  Lionel> example.
> 
>  Lionel> WebKit depends on the icu package, and when configuring the package,
>  Lionel> there is a need to run icu-config which is installed inside the
>  Lionel> staging directory.
> 
> I don't like this, as it mixes binaries/scripts for the host and the
> target - I think it's better for the icu package to install the host
> tool into host/usr/bin as well instead.
> 

First let me argue that when buildroot is not using an external
toolchain, the $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin paths are a
part of the TARGET_PATH variable.

But anyway, you're right, mixing binaries sucks.

Then, in this case, I don't think we can use the host script.

Let's right icu-config on my system :

djdeath at coalu:~$ icu-config --ldflags
 -lm   -L/usr/lib -licui18n -licuuc -licudata  -lm   

Same thing inside the staging directory :

djdeath at coalu:~/src/buildroot/buildroot_rebase$ ./output/staging/usr/bin/icu-config --ldflags
-lpthread -lm -L/home/djdeath/src/buildroot/buildroot_rebase/output/staging/usr/lib -licui18n -licuuc -licudata -lpthread -lm


The script returns paths relative to the host. This is going to be a
problem when linking the webkit binaries because the linker will have to
deal with host binaries... That's the same kind of problem we are fixing
with the libtool patch.

I'm wondering whether adding the following to package/webkit/webkit.mk
should fix the problem :

WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config

This way the configure script should use the pre-filled variable and not
try to detect the icu-config's path. And we don't need to add the
staging bin directories to the TARGET_PATH. 

A long term fix could be to use pkg-config instead of self made tool.
Patching/contributing to libicu is required.

-- 
Lionel Landwerlin <llandwerlin@gmail.com>

[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Peter Korsgaard]

>>>>> "Lionel" == Lionel Landwerlin <llandwerlin@gmail.com> writes:

 Lionel> First let me argue that when buildroot is not using an external
 Lionel> toolchain, the $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin
 Lionel> paths are a part of the TARGET_PATH variable.

Hmm, you're right - what a mess. There's no good reason why these should
be different in the regard.

 Lionel> The script returns paths relative to the host. This is going to
 Lionel> be a problem when linking the webkit binaries because the
 Lionel> linker will have to deal with host binaries... That's the same
 Lionel> kind of problem we are fixing with the libtool patch.

Yes, but copying the script from staging to host shouldn't change that.

 Lionel> I'm wondering whether adding the following to package/webkit/webkit.mk
 Lionel> should fix the problem :

 Lionel> WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config

If the webkit configure script handles that (didn't check), then that
would certainly be the easiest/cleanest solution.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Lionel Landwerlin]

Le samedi 24 octobre 2009 ? 22:22 +0200, Peter Korsgaard a ?crit :
> >>>>> "Lionel" == Lionel Landwerlin <llandwerlin@gmail.com> writes:
> 
>  Lionel> First let me argue that when buildroot is not using an external
>  Lionel> toolchain, the $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin
>  Lionel> paths are a part of the TARGET_PATH variable.
> 
> Hmm, you're right - what a mess. There's no good reason why these should
> be different in the regard.
> 
>  Lionel> The script returns paths relative to the host. This is going to
>  Lionel> be a problem when linking the webkit binaries because the
>  Lionel> linker will have to deal with host binaries... That's the same
>  Lionel> kind of problem we are fixing with the libtool patch.
> 
> Yes, but copying the script from staging to host shouldn't change that.

Yeah, that an idea. I don't which one is the cleanest.

> 
>  Lionel> I'm wondering whether adding the following to package/webkit/webkit.mk
>  Lionel> should fix the problem :
> 
>  Lionel> WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config
> 
> If the webkit configure script handles that (didn't check), then that
> would certainly be the easiest/cleanest solution.
> 

Ok, I just tested it and it works.

So the next step would probably be to remove $(STAGING_DIR)/bin and
$(STAGING_DIR)/usr/bin from the TARGET_PATH with source built
toolchains, but I'm pretty sure it's going to break some other packages.

Please find a patch as attachment.

Regards,


-- 
Lionel Landwerlin <llandwerlin@gmail.com>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-package-Set-icu-config-path.patch
Type: text/x-patch
Size: 748 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20091025/02848f92/attachment.bin>

[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Thomas Petazzoni]

Le Sat, 24 Oct 2009 20:37:57 +0200,
Peter Korsgaard <jacmet@uclibc.org> a ?crit :

>  Lionel> Some time ago, an external toolchain improvement patch added
>  Lionel> $(HOST_DIR)/bin and $(HOST_DIR)/usr/bin to the TARGET_PATH
>  Lionel> variable. This patch also removed $(STAGING_DIR)/bin and
>  Lionel> $(STAGING_DIR)/usr/bin to TARGET_PATH which breaks the build
>  Lionel> of the WebKit package for example.
> 
>  Lionel> WebKit depends on the icu package, and when configuring the
>  Lionel> package, there is a need to run icu-config which is
>  Lionel> installed inside the staging directory.
> 
> I don't like this, as it mixes binaries/scripts for the host and the
> target - I think it's better for the icu package to install the host
> tool into host/usr/bin as well instead.

As part of my work on the package infrastructure, I converted icu over
to the new format.

At first, I'd agree with you that icu-config, being executed on the
host, should theorically be installed in $(HOST_DIR). However, this
raises an issue : icu is compiled twice, once for the host, once for
the target.

In the current package/icu/icu.mk, the host icu is not installed at
all, but my package infrastructure work converts it to a more normal
package where the host icu is installed in $(HOST_DIR). Therefore, the
host icu installs its own icu-config in $(HOST_DIR)/usr/bin.

Then, we build the target icu, which also installs its own icu-config,
in $(STAGING_DIR)/usr/bin. It is not possible to install this
icu-config into $(HOST_DIR)/usr/bin, otherwise it would overwrite the
host-icu one.

Therefore, I think we have no other choice than leaving the *-config
stuff into $(STAGING_DIR)/usr/bin and then finding some trick to get
them executed even if $(STAGING_DIR)/usr/bin is not in the PATH.

What do you think about this ?

Sincerly,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Lionel Landwerlin]

Le mardi 27 octobre 2009 ? 08:44 +0100, Thomas Petazzoni a ?crit :
> Le Sat, 24 Oct 2009 20:37:57 +0200,
> Peter Korsgaard <jacmet@uclibc.org> a ?crit :
> 
> >  Lionel> Some time ago, an external toolchain improvement patch added
> >  Lionel> $(HOST_DIR)/bin and $(HOST_DIR)/usr/bin to the TARGET_PATH
> >  Lionel> variable. This patch also removed $(STAGING_DIR)/bin and
> >  Lionel> $(STAGING_DIR)/usr/bin to TARGET_PATH which breaks the build
> >  Lionel> of the WebKit package for example.
> > 
> >  Lionel> WebKit depends on the icu package, and when configuring the
> >  Lionel> package, there is a need to run icu-config which is
> >  Lionel> installed inside the staging directory.
> > 
> > I don't like this, as it mixes binaries/scripts for the host and the
> > target - I think it's better for the icu package to install the host
> > tool into host/usr/bin as well instead.
> 
> As part of my work on the package infrastructure, I converted icu over
> to the new format.
> 
> At first, I'd agree with you that icu-config, being executed on the
> host, should theorically be installed in $(HOST_DIR). However, this
> raises an issue : icu is compiled twice, once for the host, once for
> the target.
> 
> In the current package/icu/icu.mk, the host icu is not installed at
> all, but my package infrastructure work converts it to a more normal
> package where the host icu is installed in $(HOST_DIR). Therefore, the
> host icu installs its own icu-config in $(HOST_DIR)/usr/bin.
> 
> Then, we build the target icu, which also installs its own icu-config,
> in $(STAGING_DIR)/usr/bin. It is not possible to install this
> icu-config into $(HOST_DIR)/usr/bin, otherwise it would overwrite the
> host-icu one.
> 
> Therefore, I think we have no other choice than leaving the *-config
> stuff into $(STAGING_DIR)/usr/bin and then finding some trick to get
> them executed even if $(STAGING_DIR)/usr/bin is not in the PATH.
> 
> What do you think about this ?
> 

Sounds right to me.

Let's add

 WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config

to the webkit makefile.

[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Thomas Petazzoni]

Le Tue, 27 Oct 2009 10:45:07 +0100,
Lionel Landwerlin <llandwerlin@gmail.com> a ?crit :

> Sounds right to me.
> 
> Let's add
> 
>  WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config
> 
> to the webkit makefile.

Sure. But we'll have to remember that this is the solution to apply on
all similar packages... for some of which a xxx_config configure
options might not be present.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Lionel Landwerlin]

Le mardi 27 octobre 2009 ? 10:53 +0100, Thomas Petazzoni a ?crit :
> Le Tue, 27 Oct 2009 10:45:07 +0100,
> Lionel Landwerlin <llandwerlin@gmail.com> a ?crit :
> 
> > Sounds right to me.
> > 
> > Let's add
> > 
> >  WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config
> > 
> > to the webkit makefile.
> 
> Sure. But we'll have to remember that this is the solution to apply on
> all similar packages... for some of which a xxx_config configure
> options might not be present.

Right.
In my config I built the following packages :

atk-1.22.0
atk-1.22.0-host
autoconf-2.63-host
automake-1.10-host
buildroot-config
busybox-1.15.1
busybox-1.15.2
cairo-1.6.4
cairo-1.6.4-host
dbus-1.2.16
directfb-1.4.2
directfb-examples-1.2.0
eina-0.0.2.060
enchant-1.5.0
ethtool-6
expat-2.0.1
expat-2.0.1-host
fakeroot-1.9.5
fakeroot-1.9.5-host
fontconfig-2.6.0
fontconfig-2.6.0-host
freetype-2.3.9
freetype-2.3.9-host
gettext-0.16.1
icu
icu-host
iostat-2.2
jpeg-6b
libcurl-7.19.2
libecore-0.9.9.062
libe_dbus-0.5.0.062
libedje-0.9.92.062
libeet-1.2.2
libeina-0.0.2.062
libembryo-0.9.9.062
liberation-fonts-1.04
libevas-0.9.9.062
libglib2-2.20.5
libglib2-2.20.5-host
libgtk2-2.12.12
libgtk2-2.12.12-host
libpng-1.2.38
libsoup-2.26.2
libtool-1.5.24-host
libungif-4.1.4
libxml2-2.7.3
libxslt-1.1.24
lzo-2.03
m4-1.4.9-host
makedevs-host
mtd_orig
openssh-5.1p1
openssl-0.9.8k
pango-1.20.5
pango-1.20.5-host
pixman-0.10.0
pixman-0.10.0-host
pkg-config-0.23-host
sawman-1.4.2
sqlite-3.6.18
strace-4.5.18
sysstat-9.0.5
tiff-3.8.2
udev-114
webkit-r44552
zlib-1.2.3


By the way, is there a set of configuration files that cover the list of
package that are known to work with buildroot ?

It could be interesting to setup some kind of non regression builds :)


-- 
Lionel Landwerlin <llandwerlin@gmail.com>

[Buildroot] [PATCH] Fix TARGET_PATH variable with external toolchain

[Peter Korsgaard]

>>>>> "Lionel" == Lionel Landwerlin <llandwerlin@gmail.com> writes:

Hi,

 Lionel> By the way, is there a set of configuration files that cover
 Lionel> the list of package that are known to work with buildroot ?

Not a complete list, no. In concept, we would like all packages to work,
but as the number of combinations with 600+ packages / 5+ archs is so
high, there's bound to be issues.

 Lionel> It could be interesting to setup some kind of non regression builds :)

Certainly. I have a buildbot instance running at where I work to test
the configs we're using, and I'm planning on setting up a public
buildbot soon.

I recently added make all{yes,no,random}packageconfig targets to make it
easier to test random package combinations.

-- 
Bye, Peter Korsgaard