[PATCH 0/1] [1.2] poky-tiny: Separate the libc features required for meta-toolchain

[PATCH 0/1] [1.2] poky-tiny: Separate the libc features required for meta-toolchain

[Darren Hart]

This addresses [YOCTO #2295], in which the meta-toolchain target would fail for
the poky-tiny DISTRO definition due to its reduced libc feature set. This is not
an optimal solution as it significantly increases the size of the C libraries
(by 461KB). However, by making the features easily removed, those looking for
truly tiny libc configurations can easily revert to the smaller set. Out of the
box, we want poky-tiny to be able to build meta-toolchain.

In the future, we need to address WHY meta-toolchain cannot be built without
these features and look at how to reduce the impact of meta-toolchain on the
target eglibc build.

The following changes since commit c936fca6e163bc19bc08cbd0036b31ca6f6c3ed6:

  bitbake.conf: Add PARALLEL_MAKE to BB_HASHBASE_WHITELIST (2012-04-16 23:37:45 +0100)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib dvhart/tiny
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dvhart/tiny

Darren Hart (1):
  poky-tiny: Separate the libc features required for meta-toolchain

 meta-yocto/conf/distro/poky-tiny.conf |   18 ++++++++++++------
 1 files changed, 12 insertions(+), 6 deletions(-)

-- 
1.7.5.4

[PATCH 1/1] poky-tiny: Separate the libc features required for meta-toolchain

[Darren Hart]

This fixes bug [YOCTO #2295]

eglibc needs libc-posix-regexp-glibc & libc-libm-big enabled in its
configuration to avoid following eglibc build issue. Thanks to
Nitin for identifying the required features.

    ...
    | In file included from xregex.c:634:0:
    | xregex.c: In function 'byte_regex_compile':
    | xregex.c:3395:8: error: too few arguments to function 'findidx'
    | ../locale/weight.h:23:1: note: declared here
    ...

The libc features added to support building meta-toolchain add 461KB to the C
libraries. 320KB directly to libc (a 32% increase in size). If not building
meta-toolchain, the user should be able to easily configure these out.

Create a new variable to capture these dependencies,
DISTRO_FEATURES_LIBC_TOOLCHAIN, keeping them separate from the core tiny
requirements. Make it clear how to disable these if meta-toolchain is not
needed.

This patch has been tested by running the following for the qemux86 machine with
DISTRO=poky-tiny:

    $ bitbake -c cleansstate eglibc
    $ bitbake meta-toolchain
    $ bitbake core-image-minimal

The libc comparisons were made from core-image-minimal ext2 filesystems build
before and after the applicaiton of this patch and Nitin's previous poky-tiny
libc features patch in support of meta-toolchain:

    commit 8c48ab6183934fd32600997cc33e0cd3bf63139b
    Author: Nitin A Kamble <nitin.a.kamble@intel.com>
    Date:   Mon Apr 9 15:15:01 2012 -0700

        poky-tiny.conf: adjust eglibc options for poky-tiny

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
CC: Nitin A. Kamble <nitin.a.kamble@intel.com>
CC: Saul Wold <sgw@linux.intel.com>
CC: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta-yocto/conf/distro/poky-tiny.conf |   18 ++++++++++++------
 1 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/meta-yocto/conf/distro/poky-tiny.conf b/meta-yocto/conf/distro/poky-tiny.conf
index 58d64ec..327cc1e 100644
--- a/meta-yocto/conf/distro/poky-tiny.conf
+++ b/meta-yocto/conf/distro/poky-tiny.conf
@@ -63,12 +63,17 @@ ASSUME_PROVIDED += "pkgconfig$"
 # Reconfigure eglibc for a smaller installation
 # Comment out any of the lines below to disable them in the build
 DISTRO_FEATURES_LIBC_TINY = "libc-libm libc-crypt"
-# for gettext
-DISTRO_FEATURES_LIBC_TINY += "libc-posix-clang-wchar"
-# for m4
-DISTRO_FEATURES_LIBC_TINY += "libc-spawn libc-locale-code"
-# for elfutils
-DISTRO_FEATURES_LIBC_TINY += "libc-ftraverse"
+
+# Building meta-toolchain currently imposes some additional requirements:
+# If you do not plan to build meta-toolchain, you can save ~461KB by
+# commenting out the DISTRO_FEATURES_LIBC_TOOLCHAIN assignment
+# gettext needs wchar, m4 needs spawn and locale, elfutils needs ftraverse
+DISTRO_FEATURES_LIBC_TOOLCHAIN = "libc-posix-clang-wchar \
+                                  libc-spawn libc-locale-code \
+                                  libc-ftraverse libc-libm-big \
+                                  libc-posix-regexp-glibc \
+                                 "
+
 # Required for "who"
 DISTRO_FEATURES_LIBC_MINIMAL = "libc-utmp libc-getlogin"
 DISTRO_FEATURES_LIBC_REGEX = "libc-posix-regexp"
@@ -78,6 +83,7 @@ DISTRO_FEATURES_LIBC = "${DISTRO_FEATURES_LIBC_TINY} \
                         ${DISTRO_FEATURES_LIBC_MINIMAL} \
                         ${DISTRO_FEATURES_LIBC_REGEX} \
                         ${DISTRO_FEATURES_LIBC_NET} \
+                        ${DISTRO_FEATURES_LIBC_TOOLCHAIN} \
                        "
 
 # Comment out any of the lines below to disable them in the build
-- 
1.7.5.4

Re: [PATCH 0/1] [1.2] poky-tiny: Separate the libc features required for meta-toolchain

[Richard Purdie]

On Mon, 2012-04-16 at 18:06 -0700, Darren Hart wrote:
> This addresses [YOCTO #2295], in which the meta-toolchain target would fail for
> the poky-tiny DISTRO definition due to its reduced libc feature set. This is not
> an optimal solution as it significantly increases the size of the C libraries
> (by 461KB). However, by making the features easily removed, those looking for
> truly tiny libc configurations can easily revert to the smaller set. Out of the
> box, we want poky-tiny to be able to build meta-toolchain.
> 
> In the future, we need to address WHY meta-toolchain cannot be built without
> these features and look at how to reduce the impact of meta-toolchain on the
> target eglibc build.

I had a thought about this:

diff --git a/meta/conf/distro/include/default-distrovars.inc b/meta/conf/distro/include/default-distrovars.inc
index 16b3108..f770919 100644
--- a/meta/conf/distro/include/default-distrovars.inc
+++ b/meta/conf/distro/include/default-distrovars.inc
@@ -10,12 +10,14 @@ LIMIT_BUILT_LOCALES ?= "POSIX en_US en_GB"
 ENABLE_BINARY_LOCALE_GENERATION ?= "1"
 LOCALE_UTF8_ONLY ?= "0"
 
-DISTRO_FEATURES_LIBC ?= "ipv4 ipv6 libc-backtrace libc-big-macros libc-bsd libc-cxx-tests libc-catgets libc-charsets libc-crypt \
+DISTRO_FEATURES_LIBC_DEFAULT ?= "ipv4 ipv6 libc-backtrace libc-big-macros libc-bsd libc-cxx-tests libc-catgets libc-charsets libc-crypt \
                                        libc-crypt-ufc libc-db-aliases libc-envz libc-fcvt libc-fmtmsg libc-fstab libc-ftraverse \
                                        libc-getlogin libc-idn libc-inet-anl libc-libm libc-libm-big libc-locales libc-locale-code \
                                        libc-memusage libc-nis libc-nsswitch libc-rcmd libc-rtld-debug libc-spawn libc-streams libc-sunrpc \
                                        libc-utmp libc-utmpx libc-wordexp libc-posix-clang-wchar libc-posix-regexp libc-posix-regexp-glibc \
                                        libc-posix-wchar-io"
+DISTRO_FEATURES_LIBC ?= "${DISTRO_FEATURES_LIBC_DEFAULT}"
+DISTRO_FEATURES_LIBC_virtclass-nativesdk = "${DISTRO_FEATURES_LIBC_DEFAULT}"
 DISTRO_FEATURES ?= "alsa argp bluetooth ext2 irda largefile pcmcia usbgadget usbhost wifi xattr nfs zeroconf pci 3g x11 ${DISTRO_FEATURES_LIBC}"
 
 IMAGE_FEATURES ?= ""

Probably not something for release at this point but maybe after it...

Cheers,

Richard

Re: [PATCH 0/1] [1.2] poky-tiny: Separate the libc features required for meta-toolchain

[Darren Hart]

On 04/17/2012 01:09 AM, Richard Purdie wrote:
> On Mon, 2012-04-16 at 18:06 -0700, Darren Hart wrote:
>> This addresses [YOCTO #2295], in which the meta-toolchain target would fail for
>> the poky-tiny DISTRO definition due to its reduced libc feature set. This is not
>> an optimal solution as it significantly increases the size of the C libraries
>> (by 461KB). However, by making the features easily removed, those looking for
>> truly tiny libc configurations can easily revert to the smaller set. Out of the
>> box, we want poky-tiny to be able to build meta-toolchain.
>>
>> In the future, we need to address WHY meta-toolchain cannot be built without
>> these features and look at how to reduce the impact of meta-toolchain on the
>> target eglibc build.
> 
> I had a thought about this:
> 
> diff --git a/meta/conf/distro/include/default-distrovars.inc b/meta/conf/distro/include/default-distrovars.inc
> index 16b3108..f770919 100644
> --- a/meta/conf/distro/include/default-distrovars.inc
> +++ b/meta/conf/distro/include/default-distrovars.inc
> @@ -10,12 +10,14 @@ LIMIT_BUILT_LOCALES ?= "POSIX en_US en_GB"
>  ENABLE_BINARY_LOCALE_GENERATION ?= "1"
>  LOCALE_UTF8_ONLY ?= "0"
>  
> -DISTRO_FEATURES_LIBC ?= "ipv4 ipv6 libc-backtrace libc-big-macros libc-bsd libc-cxx-tests libc-catgets libc-charsets libc-crypt \
> +DISTRO_FEATURES_LIBC_DEFAULT ?= "ipv4 ipv6 libc-backtrace libc-big-macros libc-bsd libc-cxx-tests libc-catgets libc-charsets libc-crypt \
>                                         libc-crypt-ufc libc-db-aliases libc-envz libc-fcvt libc-fmtmsg libc-fstab libc-ftraverse \
>                                         libc-getlogin libc-idn libc-inet-anl libc-libm libc-libm-big libc-locales libc-locale-code \
>                                         libc-memusage libc-nis libc-nsswitch libc-rcmd libc-rtld-debug libc-spawn libc-streams libc-sunrpc \
>                                         libc-utmp libc-utmpx libc-wordexp libc-posix-clang-wchar libc-posix-regexp libc-posix-regexp-glibc \
>                                         libc-posix-wchar-io"
> +DISTRO_FEATURES_LIBC ?= "${DISTRO_FEATURES_LIBC_DEFAULT}"
> +DISTRO_FEATURES_LIBC_virtclass-nativesdk = "${DISTRO_FEATURES_LIBC_DEFAULT}"
>  DISTRO_FEATURES ?= "alsa argp bluetooth ext2 irda largefile pcmcia usbgadget usbhost wifi xattr nfs zeroconf pci 3g x11 ${DISTRO_FEATURES_LIBC}"
>  
>  IMAGE_FEATURES ?= ""
> 
> Probably not something for release at this point but maybe after it...

Interesting.... I need to investigate the meta-toolchain build a bit
more. I didn't think it was just the nativesdk recipes that were failing.

--
Darren

> 
> Cheers,
> 
> Richard
> 

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 0/1] [1.2] poky-tiny: Separate the libc features required for meta-toolchain

[Richard Purdie]

On Tue, 2012-04-17 at 07:04 -0700, Darren Hart wrote:
> On 04/17/2012 01:09 AM, Richard Purdie wrote:
> > I had a thought about this:
> > 
> > diff --git a/meta/conf/distro/include/default-distrovars.inc b/meta/conf/distro/include/default-distrovars.inc
> > index 16b3108..f770919 100644
> > --- a/meta/conf/distro/include/default-distrovars.inc
> > +++ b/meta/conf/distro/include/default-distrovars.inc
> > @@ -10,12 +10,14 @@ LIMIT_BUILT_LOCALES ?= "POSIX en_US en_GB"
> >  ENABLE_BINARY_LOCALE_GENERATION ?= "1"
> >  LOCALE_UTF8_ONLY ?= "0"
> >  
> > -DISTRO_FEATURES_LIBC ?= "ipv4 ipv6 libc-backtrace libc-big-macros libc-bsd libc-cxx-tests libc-catgets libc-charsets libc-crypt \
> > +DISTRO_FEATURES_LIBC_DEFAULT ?= "ipv4 ipv6 libc-backtrace libc-big-macros libc-bsd libc-cxx-tests libc-catgets libc-charsets libc-crypt \
> >                                         libc-crypt-ufc libc-db-aliases libc-envz libc-fcvt libc-fmtmsg libc-fstab libc-ftraverse \
> >                                         libc-getlogin libc-idn libc-inet-anl libc-libm libc-libm-big libc-locales libc-locale-code \
> >                                         libc-memusage libc-nis libc-nsswitch libc-rcmd libc-rtld-debug libc-spawn libc-streams libc-sunrpc \
> >                                         libc-utmp libc-utmpx libc-wordexp libc-posix-clang-wchar libc-posix-regexp libc-posix-regexp-glibc \
> >                                         libc-posix-wchar-io"
> > +DISTRO_FEATURES_LIBC ?= "${DISTRO_FEATURES_LIBC_DEFAULT}"
> > +DISTRO_FEATURES_LIBC_virtclass-nativesdk = "${DISTRO_FEATURES_LIBC_DEFAULT}"
> >  DISTRO_FEATURES ?= "alsa argp bluetooth ext2 irda largefile pcmcia usbgadget usbhost wifi xattr nfs zeroconf pci 3g x11 ${DISTRO_FEATURES_LIBC}"
> >  
> >  IMAGE_FEATURES ?= ""
> > 
> > Probably not something for release at this point but maybe after it...
> 
> Interesting.... I need to investigate the meta-toolchain build a bit
> more. I didn't think it was just the nativesdk recipes that were failing.

I screwed up and accidentally merged the above in another commit.
Thankfully the code actually works and I'm therefore not going to revert
it and it will go into the release. I tried a quick build with poky-tiny
and I think this diff is a good summary for a working meta-toolchain:

diff --git a/meta-yocto/conf/distro/poky-tiny.conf b/meta-yocto/conf/distro/poky-tiny.conf
index 327cc1e..503444f 100644
--- a/meta-yocto/conf/distro/poky-tiny.conf
+++ b/meta-yocto/conf/distro/poky-tiny.conf
@@ -68,11 +68,7 @@ DISTRO_FEATURES_LIBC_TINY = "libc-libm libc-crypt"
 # If you do not plan to build meta-toolchain, you can save ~461KB by
 # commenting out the DISTRO_FEATURES_LIBC_TOOLCHAIN assignment
 # gettext needs wchar, m4 needs spawn and locale, elfutils needs ftraverse
-DISTRO_FEATURES_LIBC_TOOLCHAIN = "libc-posix-clang-wchar \
-                                  libc-spawn libc-locale-code \
-                                  libc-ftraverse libc-libm-big \
-                                  libc-posix-regexp-glibc \
-                                 "
+DISTRO_FEATURES_LIBC_TOOLCHAIN = ""
 
 # Required for "who"
 DISTRO_FEATURES_LIBC_MINIMAL = "libc-utmp libc-getlogin"
diff --git a/meta/classes/gettext.bbclass b/meta/classes/gettext.bbclass
index 89faa30..95818c6 100644
--- a/meta/classes/gettext.bbclass
+++ b/meta/classes/gettext.bbclass
@@ -15,7 +15,7 @@ def gettext_oeconf(d):
         return '--disable-nls'
     return "--enable-nls"
 
-DEPENDS_GETTEXT = "virtual/gettext gettext-native"
+DEPENDS_GETTEXT ??= "virtual/gettext gettext-native"
 
 BASEDEPENDS =+ "${@gettext_dependencies(d)}"
 EXTRA_OECONF_append = " ${@gettext_oeconf(d)}"

so we should be able to drop DISTRO_FEATURES_LIBC_TOOLCHAIN early in the
1.3 cycle, we just need to tweak the gettext class (it was overwriting
the value of DEPENDS_GETTEXT set in canadian-cross.bbclass).

Cheers,

Richard