[PATCH v2 0/2] allow linux-lib-headers with external-toolchain-csl

[PATCH v2 0/2] allow linux-lib-headers with external-toolchain-csl

[Ben Gardiner]

This patch series enables the use of linux-libc-headers with
external CSL toolchain. This is useful when building userspace
applications to use ABI features not present in the headers bundled
with the CSL toolchain.

Version 1 of this patch was tested in arago [1] and found to work there.

Testing was performed there also by Andrea Galbusera <gizero@gmail.com>.
Who found that the changes enabled a couple of packages to pickup kernel
headers that were not bundled with the CSL toolchain.

I tested this series on 3dc72c4ca0a9ce677f6c3a56517626984b5eaaa0 of
git://git.openembedded.org/openembedded#master

Using the following local.conf (comment- and blank-lines trimmed):

SCRATCH ?= "${OEBASE}"
DL_DIR = "${SCRATCH}/downloads"
TMPDIR = "${SCRATCH}/arago-tmp"
BBFILES = "/media/unixdata/src/openembedded/openembedded-master/recipes/*/*.bb"
MACHINE = "da850-omapl138-evm"
DISTRO = "angstrom-2008.1"
ENABLE_BINARY_LOCALE_GENERATION = "0"
IMAGE_FSTYPES = "jffs2 tar.gz ext2.gz"
PARALLEL_MAKE = "-j 9"
BB_NUMBER_THREADS = "9"
BBINCLUDELOGS = "yes"
SOURCE_MIRROR_URL = "http://mirror/oe/"
INHERIT += "own-mirrors"
TOOLCHAIN_VENDOR = "-none"
TOOLCHAIN_TYPE = "external"
TOOLCHAIN_BRAND = "csl"
TOOLCHAIN_PATH = "/opt/codesourcery-arm-none-eabi-2009q1"

'bitbake net-tools' fails with the familiar error [2] of 
'error: redefinition of 'struct iphdr'' which is caused by a missing
#ifdef in the 2.6.29 linux-headers ip.h bundled with the 2009q1
toolchain I am using. 

Whereas with the patches applied and the following added to local.conf:
PREFERRED_PROVIDER_linux-libc-headers = "linux-libc-headers"

'bitbake net-tools' is successful. Since specifying the
PREFERRED_PROVIDER in local.conf now results in installation of the
linux-lib-headers version 2.6.31 specified by angstrom conf.

Ben Gardiner (2):
  external-toolchain: allow override of linux-libc-headers provider
  external-toolchain-csl: allow linux-libc-headers

 conf/distro/include/toolchain-external.inc |    2 +-
 conf/local.conf.sample                     |    4 ++++
 recipes/meta/external-toolchain-csl.bb     |   15 ++++++++++-----
 3 files changed, 15 insertions(+), 6 deletions(-)

[1] https://gforge.ti.com/gf/project/arago/mailman/?_forum_action=ForumMessageBrowse&thread_id=4152&action=ListThreads&mailman_id=39
[2] http://www.gossamer-threads.com/lists/linux/kernel/1025676

[PATCH v2 1/2] external-toolchain: allow override of linux-libc-headers provider

[Ben Gardiner]

only set PREFERRED_PROVIDER_linux-libc-headers if it is not
already assigned a value.

Signed-off-by: Ben Gardiner <bengardiner@nanometrics.ca>
Tested-by: Andrea Galbusera <gizero@gmail.com>

---

Changes since v1:
 * rebased to 3dc72c4ca0a9ce677f6c3a56517626984b5eaaa0 of
   git://git.openembedded.org/openembedded#master
 * added Tested-by: Andrea Galbusera <gizero@gmail.com>

---
 conf/distro/include/toolchain-external.inc |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/conf/distro/include/toolchain-external.inc b/conf/distro/include/toolchain-external.inc
index ab50308..12dc454 100644
--- a/conf/distro/include/toolchain-external.inc
+++ b/conf/distro/include/toolchain-external.inc
@@ -4,7 +4,7 @@ PREFERRED_PROVIDER_virtual/${TARGET_PREFIX}gcc-initial = "external-toolchain-${T
 PREFERRED_PROVIDER_virtual/${TARGET_PREFIX}gcc-intermediate = "external-toolchain-${TOOLCHAIN_BRAND}"
 PREFERRED_PROVIDER_virtual/${TARGET_PREFIX}gcc = "external-toolchain-${TOOLCHAIN_BRAND}"
 PREFERRED_PROVIDER_virtual/${TARGET_PREFIX}g++ = "external-toolchain-${TOOLCHAIN_BRAND}"
-PREFERRED_PROVIDER_linux-libc-headers = "external-toolchain-${TOOLCHAIN_BRAND}"
+PREFERRED_PROVIDER_linux-libc-headers ?= "external-toolchain-${TOOLCHAIN_BRAND}"
 
 TOOLCHAIN_VENDOR ?= "${TARGET_VENDOR}"
 TARGET_VENDOR := "${TOOLCHAIN_VENDOR}"
-- 
1.7.1

[PATCH v2 2/2] external-toolchain-csl: allow linux-libc-headers

[Ben Gardiner]

If PREFERRED_PROVIDER_linux-libc-headers is defined as anything
other than external-toolchain-csl then don't PROVIDE, install or
stage the linux headers packaged with the CSL toolchain.

Signed-off-by: Ben Gardiner <bengardiner@nanometrics.ca>
Tested-by: Andrea Galbusera <gizero@gmail.com>

---

Changes since v1:
 * rebased to 3dc72c4ca0a9ce677f6c3a56517626984b5eaaa0 of
   git://git.openembedded.org/openembedded#master
 * added Tested-by: Andrea Galbusera <gizero@gmail.com>

---
 conf/local.conf.sample                 |    4 ++++
 recipes/meta/external-toolchain-csl.bb |   15 ++++++++++-----
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/conf/local.conf.sample b/conf/local.conf.sample
index ff3da96..51d0449 100644
--- a/conf/local.conf.sample
+++ b/conf/local.conf.sample
@@ -90,6 +90,10 @@ PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}g++:gcc-cross"
 # Stay away from unversioned distros unless you really know what you are doing
 # DISTRO = "angstrom-2008.1"
 
+#Uncomment this if you want to use linux-headers from the linux-lib-headers
+#recipe as opposed to those bundled with the CSL toolchain
+#PREFERRED_PROVIDER_linux-libc-headers = "linux-libc-headers"
+
 # So far, angstrom.conf sets ENABLE_BINARY_LOCALE_GENERATION
 # to generate binary locale packages at build time using qemu-native and
 # thereby guarantee i18n support on all devices. If your build breaks on 
diff --git a/recipes/meta/external-toolchain-csl.bb b/recipes/meta/external-toolchain-csl.bb
index 2e109c8..48c23eb 100644
--- a/recipes/meta/external-toolchain-csl.bb
+++ b/recipes/meta/external-toolchain-csl.bb
@@ -1,4 +1,4 @@
-PR = "r7"
+PR = "r8"
 
 INHIBIT_DEFAULT_DEPS = "1"
 
@@ -21,9 +21,10 @@ PROVIDES = "\
 	virtual/libintl \
 	virtual/libiconv \
 	glibc-thread-db \
-	linux-libc-headers \
+	${@base_conditional('PREFERRED_PROVIDER_linux-libc-headers', 'external-toolchain-csl', 'linux-libc-headers', '', d)} \
 "
 
+DEPENDS = "${@base_conditional('PREFERRED_PROVIDER_linux-libc-headers', 'external-toolchain-csl', '', 'linux-libc-headers', d)}"
 RPROVIDES_glibc-dev += "libc-dev libc6-dev virtual-libc-dev"
 PACKAGES_DYNAMIC += "glibc-gconv-*"
 PACKAGES_DYNAMIC += "glibc-locale-*"
@@ -35,7 +36,7 @@ PACKAGES = "\
 	libgcc-dev \
 	libstdc++ \
 	libstdc++-dev \
-	linux-libc-headers \
+	${@base_conditional('PREFERRED_PROVIDER_linux-libc-headers', 'external-toolchain-csl', 'linux-libc-headers', '', d)} \
 	glibc-dbg \
 	glibc \
 	catchsegv \
@@ -200,6 +201,8 @@ LICENSE_libstdc++ = "${CSL_LIC_RLE}"
 LICENSE_libstdc++-dev = "${CSL_LIC_RLE}"
 
 do_install() {
+	RSYNC_OPTS="${@base_conditional('PREFERRED_PROVIDER_linux-libc-headers', 'external-toolchain-csl', '', '--exclude=/include/linux', d)}"
+
 	install -d ${D}${sysconfdir}
 	install -d ${D}${bindir}
 	install -d ${D}${sbindir}
@@ -213,7 +216,7 @@ do_install() {
 	cp -a ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/etc/* ${D}${sysconfdir}
 	cp -a ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/sbin/* ${D}${base_sbindir} \
 		|| true
-	cp -a ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/usr/* ${D}/usr
+	rsync -a ${RSYNC_OPTS} ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/usr/* ${D}/usr
 	cp -a ${TOOLCHAIN_PATH}/${TARGET_SYS}/include/* ${D}/usr/include
 
 	rm -rf ${D}${bindir}/gdbserver
@@ -222,11 +225,13 @@ do_install() {
 }
 
 do_stage() {
+	RSYNC_OPTS="${@base_conditional('PREFERRED_PROVIDER_linux-libc-headers', 'external-toolchain-csl', '', '--exclude=/linux', d)}"
+
 	install -d ${STAGING_INCDIR}
 	install -d ${STAGING_LIBDIR}
 	install -d ${STAGING_DIR_TARGET}${base_libdir}
 
-	cp -a ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/usr/include/* ${STAGING_INCDIR}
+	rsync -a ${RSYNC_OPTS} ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/usr/include/* ${STAGING_INCDIR}
 	cp -a ${TOOLCHAIN_PATH}/${TARGET_SYS}/include/* ${STAGING_INCDIR}
 	cp -a ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/usr/lib/* ${STAGING_LIBDIR}
 	cp -a ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/lib/* ${STAGING_DIR_TARGET}${base_libdir}
-- 
1.7.1

Re: [PATCH v2 0/2] allow linux-lib-headers with external-toolchain-csl

[Tom Rini]

On 03/02/2011 08:05 AM, Ben Gardiner wrote:
> This patch series enables the use of linux-libc-headers with
> external CSL toolchain. This is useful when building userspace
> applications to use ABI features not present in the headers bundled
> with the CSL toolchain.
>
> Version 1 of this patch was tested in arago [1] and found to work there.
>
> Testing was performed there also by Andrea Galbusera<gizero@gmail.com>.
> Who found that the changes enabled a couple of packages to pickup kernel
> headers that were not bundled with the CSL toolchain.
>
> I tested this series on 3dc72c4ca0a9ce677f6c3a56517626984b5eaaa0 of
> git://git.openembedded.org/openembedded#master
>
> Using the following local.conf (comment- and blank-lines trimmed):

With some slight modifications to not use rsync (since the host may not 
have that) and to punt the local.conf.sample bit to another commit I'm 
doing shortly with a full -csl example, I've applied this locally and 
will push today.  Thanks!

-- 
Tom Rini
Mentor Graphics Corporation

Re: [PATCH v2 0/2] allow linux-lib-headers with external-toolchain-csl

[Ben Gardiner]

Hi Tom,

On Wed, Mar 2, 2011 at 11:52 AM, Tom Rini <tom_rini@mentor.com> wrote:
> On 03/02/2011 08:05 AM, Ben Gardiner wrote:
>>
>> This patch series enables the use of linux-libc-headers with
>> external CSL toolchain. This is useful when building userspace
>> applications to use ABI features not present in the headers bundled
>> with the CSL toolchain.
>>
>> Version 1 of this patch was tested in arago [1] and found to work there.
>>
>> Testing was performed there also by Andrea Galbusera<gizero@gmail.com>.
>> Who found that the changes enabled a couple of packages to pickup kernel
>> headers that were not bundled with the CSL toolchain.
>>
>> I tested this series on 3dc72c4ca0a9ce677f6c3a56517626984b5eaaa0 of
>> git://git.openembedded.org/openembedded#master
>>
>> Using the following local.conf (comment- and blank-lines trimmed):
>
> With some slight modifications to not use rsync (since the host may not have
> that) and to punt the local.conf.sample bit to another commit I'm doing
> shortly with a full -csl example, I've applied this locally and will push
> today.  Thanks!

Glad you like it!

I was not aware of the rsync restriction, thanks. What are the mod's
you're going to use? I'd be happy to re-spin the series with your
changes and the local.conf.sample extracted.

Best Regards,
Ben Gardiner

---
Nanometrics Inc.
http://www.nanometrics.ca

Re: [PATCH v2 0/2] allow linux-lib-headers with external-toolchain-csl

[Tom Rini]

On 03/02/2011 10:00 AM, Ben Gardiner wrote:
> Hi Tom,
>
> On Wed, Mar 2, 2011 at 11:52 AM, Tom Rini<tom_rini@mentor.com>  wrote:
>> On 03/02/2011 08:05 AM, Ben Gardiner wrote:
>>>
>>> This patch series enables the use of linux-libc-headers with
>>> external CSL toolchain. This is useful when building userspace
>>> applications to use ABI features not present in the headers bundled
>>> with the CSL toolchain.
>>>
>>> Version 1 of this patch was tested in arago [1] and found to work there.
>>>
>>> Testing was performed there also by Andrea Galbusera<gizero@gmail.com>.
>>> Who found that the changes enabled a couple of packages to pickup kernel
>>> headers that were not bundled with the CSL toolchain.
>>>
>>> I tested this series on 3dc72c4ca0a9ce677f6c3a56517626984b5eaaa0 of
>>> git://git.openembedded.org/openembedded#master
>>>
>>> Using the following local.conf (comment- and blank-lines trimmed):
>>
>> With some slight modifications to not use rsync (since the host may not have
>> that) and to punt the local.conf.sample bit to another commit I'm doing
>> shortly with a full -csl example, I've applied this locally and will push
>> today.  Thanks!
>
> Glad you like it!
>
> I was not aware of the rsync restriction, thanks. What are the mod's
> you're going to use? I'd be happy to re-spin the series with your
> changes and the local.conf.sample extracted.

It's all in now (and this spurred me to push something else I've been 
meaning to for a while, adding in gdbserver and dropping do_stage) but 
in short, we just cp -a like before but now will rm -rf as needed.

-- 
Tom Rini
Mentor Graphics Corporation

Re: [PATCH v2 0/2] allow linux-lib-headers with external-toolchain-csl

[Ben Gardiner]

Hi Tom and Denys,

On Wed, Mar 2, 2011 at 12:10 PM, Tom Rini <tom_rini@mentor.com> wrote:
>
> It's all in now (and this spurred me to push something else I've been
> meaning to for a while, adding in gdbserver and dropping do_stage)[...]

Ok, no worries then. I'm really glad that you found it useful, Tom and
it's making its way to upstream, thanks.

On Wed, Mar 2, 2011 at 12:10 PM, Tom Rini <tom_rini@mentor.com> wrote:
> [...]but in
> short, we just cp -a like before but now will rm -rf as needed.
On Wed, Mar 2, 2011 at 12:12 PM, Denys Dmytriyenko <denys@ti.com> wrote:
> Looks good to me, except:
>
> 1. Why use rsync instead of cp? I think it's less portable that way.

I had an inkling that my selection of rsync should have been defended
in the commit messages :)

In my own defense, I was concerned that the cp followed by an rm could
remove from staging other headers that were installed into the correct
locations -- rsync is my go-to for copying with exclusions.

I'll remember that it is not portable enough next time around.

On Wed, Mar 2, 2011 at 12:12 PM, Denys Dmytriyenko <denys@ti.com> wrote:
> 2. I saw several linux-lib-headers instead of linux-libc-headers (missing 'c')
On Wed, Mar 2, 2011 at 12:16 PM, Denys Dmytriyenko <denys@ti.com> wrote:
> Argh, you left one linux-lib-headers in the comment... :)

Oops. Sorry about that.

Thanks again for picking up this patch.

Best Regards,
Ben Gardiner

---
Nanometrics Inc.
http://www.nanometrics.ca

Re: [PATCH v2 2/2] external-toolchain-csl: allow linux-libc-headers

[Phil Blundell]

On Wed, 2011-03-02 at 10:05 -0500, Ben Gardiner wrote:
> -	cp -a ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/usr/include/* ${STAGING_INCDIR}
> +	rsync -a ${RSYNC_OPTS} ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/usr/include/* ${STAGING_INCDIR}

If you want to exclude some files then it's probably better to use
something like tar --exclude ... -cf - | tar -xf -.  (I think the
--exclude option is in POSIX so any reasonable version of tar ought to
have it.)  Or you could use some sort of "find | grep -v .. | xargs cp"
construct to make a list of the files that you do want to copy.  Either
of those is probably a better option than rsync.

p.

Re: [PATCH v2 2/2] external-toolchain-csl: allow linux-libc-headers

[Ben Gardiner]

Hi Phil,

On Wed, Mar 2, 2011 at 12:50 PM, Phil Blundell <philb@gnu.org> wrote:
> On Wed, 2011-03-02 at 10:05 -0500, Ben Gardiner wrote:
>> -     cp -a ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/usr/include/* ${STAGING_INCDIR}
>> +     rsync -a ${RSYNC_OPTS} ${TOOLCHAIN_PATH}/${TARGET_SYS}/libc/usr/include/* ${STAGING_INCDIR}
>
> If you want to exclude some files then it's probably better to use
> something like tar --exclude ... -cf - | tar -xf -.  (I think the
> --exclude option is in POSIX so any reasonable version of tar ought to
> have it.)  Or you could use some sort of "find | grep -v .. | xargs cp"
> construct to make a list of the files that you do want to copy.  Either
> of those is probably a better option than rsync.

Thanks for the alternatives.

I think the piped-tars option is pretty cool, it's alot like the
netcat-tar trick for when you don't have rsync handy, just no netcat.
:)

I will definitely keep that in mind for future tasks.

Best Regards,
Ben Gardiner

---
Nanometrics Inc.
http://www.nanometrics.ca