[PATCH 0/4] [PATCH 0/4] mklibs library optimization for all arches

[PATCH 0/4] [PATCH 0/4] mklibs library optimization for all arches

[Nitin A Kamble]

From: Nitin A Kamble <nitin.a.kamble@intel.com>

  I tested this with poky-image-minimal and I see some disk space savings by
using the mklibs library optimizations. (76MB rootfs became 75MB)

  The minimal poky image had 106 dynamically linked executables. And the
footprint savings will go up as number of dynamically linked executables in
an image decrease.

  I did not see any improvement on mips and ppc, but could see the footprint
improvement on x86, arm & x86-64.

  Also checked that the executables were not broken after the mklibs 
optimization.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: nitin/mklibs
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=nitin/mklibs

Thanks,
    Nitin A Kamble <nitin.a.kamble@intel.com>
---


Nitin A Kamble (4):
  distro_tracking: update latest upstream versions
  elfutils: add a missing depedancy
  mklibs-native: new recipe for optimizing size of library files
  image-mklibs.bbclass: add the library optimization functionality

 meta/classes/image-mklibs.bbclass                  |   57 ++++++++++++++++++++
 .../conf/distro/include/distro_tracking_fields.inc |   43 ++++++++-------
 meta/conf/local.conf.sample                        |    2 +
 meta/recipes-devtools/elfutils/elfutils_0.148.bb   |    3 +-
 .../mklibs/files/ac_init_fix.patch                 |   17 ++++++
 .../mklibs/mklibs-native_0.1.31.bb                 |   23 ++++++++
 6 files changed, 124 insertions(+), 21 deletions(-)
 create mode 100644 meta/classes/image-mklibs.bbclass
 create mode 100644 meta/recipes-devtools/mklibs/files/ac_init_fix.patch
 create mode 100644 meta/recipes-devtools/mklibs/mklibs-native_0.1.31.bb

-- 
1.7.2.2

[PATCH 1/4] distro_tracking: update latest upstream versions

[Nitin A Kamble]

From: Nitin A Kamble <nitin.a.kamble@intel.com>

Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
---
 .../conf/distro/include/distro_tracking_fields.inc |   43 ++++++++++---------
 1 files changed, 23 insertions(+), 20 deletions(-)

diff --git a/meta/conf/distro/include/distro_tracking_fields.inc b/meta/conf/distro/include/distro_tracking_fields.inc
index 9f99e50..1bdf521 100644
--- a/meta/conf/distro/include/distro_tracking_fields.inc
+++ b/meta/conf/distro/include/distro_tracking_fields.inc
@@ -102,9 +102,14 @@ RECIPE_LATEST_VERSION_pn-run-postinsts = "check"
 RECIPE_LAST_UPDATE_pn-run-postinsts = "Nov 22, 2010"
 RECIPE_MAINTAINER_pn-run-postinsts = "Dongxiao Xu <dongxiao.xu@intel.com"
 
-RECIPE_STATUS_pn-subversion = "green"
-RECIPE_LATEST_VERSION_pn-subversion = "1.6.13"
-RECIPE_LAST_UPDATE_pn-subversion = "Nov 18, 2010"
+RECIPE_STATUS_pn-screen = "red"
+RECIPE_LATEST_VERSION_pn-screen = "check"
+RECIPE_LAST_UPDATE_pn-screen = "Aug 31, 2010"
+RECIPE_MAINTAINER_pn-screen = "Qing He <qing.he@intel.com>"
+
+RECIPE_STATUS_pn-subversion = "red"
+RECIPE_LATEST_VERSION_pn-subversion = "1.6.15"
+RECIPE_MANUAL_CHECK_DATE_pn-subversion = "Jan 25, 2011" 
 RECIPE_MAINTAINER_pn-subversion = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
 RECIPE_STATUS_pn-trace-cmd = "red"
@@ -2807,17 +2812,16 @@ RECIPE_STATUS_pn-ldconfig-native = "red"
 RECIPE_LAST_UPDATE_pn-ldconfig-native = "Nov 23, 2007"
 DISTRO_PN_ALIAS_pn-ldconfig-native = "Ubuntu=libc-bin Fedora=glibc"
 
-RECIPE_STATUS_pn-gcc="green" 
-RECIPE_LATEST_VERSION_pn-gcc="4.5.1"
-RECIPE_NO_OF_PATCHES_pn-gcc="8"
-RECIPE_TIME_BETWEEN_LAST_TWO_RELEASES_pn-gcc="5 months"
+RECIPE_STATUS_pn-gcc="red" 
+RECIPE_LATEST_VERSION_pn-gcc="4.5.2"
 RECIPE_LATEST_RELEASE_DATE_pn-gcc="2010/03/03"
 RECIPE_LAST_UPDATE_pn-gcc = "Nov 8, 2010"
 RECIPE_MAINTAINER_pn-gcc = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
-RECIPE_STATUS_pn-autoconf="green" 
+RECIPE_STATUS_pn-autoconf="red" 
 RECIPE_LATEST_VERSION_pn-autoconf="2.68"
 RECIPE_LAST_UPDATE_pn-autoconf = "Jan 29, 2010"
+RECIPE_NO_UPDATE_REASON_autoconf="seeing rpm issue with the 2.68 version, a bug is open"
 RECIPE_MAINTAINER_pn-autoconf = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
 RECIPE_STATUS_pn-automake="green" 
@@ -2887,7 +2891,7 @@ RECIPE_MAINTAINER_pn-postinsts = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
 RECIPE_STATUS_pn-nasm="green" 
 RECIPE_LATEST_VERSION_pn-nasm="2.07"
-RECIPE_LAST_UPDATE_pn-nasm = "Jun 23, 2010"
+RECIPE_MANUAL_CHECK_DATE_pn-nasm = "Jan 25, 2011" 
 RECIPE_MAINTAINER_pn-nasm = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
 RECIPE_STATUS_pn-perl="red" # upgrade needed
@@ -2898,7 +2902,7 @@ RECIPE_MAINTAINER_pn-perl = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 RECIPE_STATUS_pn-prelink="red" 
 RECIPE_LATEST_VERSION_pn-prelink="1.0+git0+0x909470ee441237563d6236c505cb2d02ddc48704"
 RECIPE_LAST_UPDATE_pn-perl = "Jul 23, 2010"
-RECIPE_MAINTAINER_pn-prelink = "Nitin A Kamble <nitin.a.kamble@intel.com>"
+RECIPE_MAINTAINER_pn-prelink = "Mark Hatle <mark.hatle@windriver.com>"
 
 RECIPE_STATUS_pn-python-dbus="red" 
 RECIPE_LATEST_VERSION_pn-python-dbus="0.83.1"
@@ -2906,8 +2910,8 @@ RECIPE_LAST_UPDATE_pn-python-dbus = "Jul 7, 2010"
 RECIPE_MAINTAINER_pn-python-dbus = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 DISTRO_PN_ALIAS_pn-python-dbus = "Ubuntu=python-dbus Debian=python-dbus Mandriva=python-dbus"
 
-RECIPE_STATUS_pn-python-gst="green" 
-RECIPE_LATEST_VERSION_pn-python-gst="0.10.19"
+RECIPE_STATUS_pn-python-gst="red" 
+RECIPE_LATEST_VERSION_pn-python-gst="0.10.21"
 DISTRO_PN_ALIAS_pn-python-gst = "OpenSuSE=python-gstreamer Ubuntu=gst0.10-python Debian=gst0.10-python"
 RECIPE_LAST_UPDATE_pn-python-gst = "Nov 8, 2010"
 RECIPE_MAINTAINER_pn-python-gst = "Nitin A Kamble <nitin.a.kamble@intel.com>"
@@ -2948,9 +2952,6 @@ RECIPE_MAINTAINER_pn-python-scons = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
 RECIPE_STATUS_pn-python="red"
 RECIPE_LATEST_VERSION_pn-python="2.7.1"
-RECIPE_NO_OF_PATCHES_pn-python="8"
-RECIPE_LATEST_RELEASE_DATE_pn-python="2010/03/18"
-RECIPE_TIME_BETWEEN_LAST_TWO_RELEASES_pn-python="5 months"
 RECIPE_LAST_UPDATE_pn-python = "Nov 8, 2010"
 RECIPE_MAINTAINER_pn-python = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
@@ -2976,7 +2977,7 @@ RECIPE_MAINTAINER_pn-gnu-config = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
 RECIPE_STATUS_pn-mpfr="green"
 RECIPE_LATEST_VERSION_pn-mpfr="3.0.0"
-RECIPE_LAST_UPDATE_pn-mpfr = "Aug 27, 2010"
+RECIPE_MANUAL_CHECK_DATE_pn-mpfr = "Jan 25, 2011" 
 RECIPE_MAINTAINER_pn-mpfr = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
 RECIPE_STATUS_pn-gmp="green"
@@ -2986,16 +2987,18 @@ RECIPE_MAINTAINER_pn-gmp = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
 RECIPE_STATUS_pn-libmpc="green"
 RECIPE_LATEST_VERSION_pn-libmpc="0.8.2"
-RECIPE_LAST_UPDATE_pn-libmpc = "Jul 19, 2010"
+RECIPE_MANUAL_CHECK_DATE_pn-libmpc = "Jan 25, 2011" 
 RECIPE_MAINTAINER_pn-libmpc = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 DISTRO_PN_ALIAS_pn-libmpc = "Fedora=libmpc OpenSuse=libmpc2"
 
-RECIPE_STATUS_pn-byacc="green" 
-RECIPE_LATEST_VERSION_pn-byacc="20101127"
+RECIPE_STATUS_pn-byacc="red" 
+RECIPE_LATEST_VERSION_pn-byacc="20101229"
 RECIPE_MAINTAINER_pn-byacc = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
-RECIPE_STATUS_pn-libconvert-asn1-perl="green" # 0.22 version does not build, so created a recipe for version 0.21
+RECIPE_STATUS_pn-libconvert-asn1-perl="green" 
+# 0.22 version does not build, so created a recipe for version 0.21
 RECIPE_LATEST_VERSION_pn-libconvert-asn1-perl="0.22"
+RECIPE_NO_UPDATE_REASON_pn-libconvert-asn1-perl = "0.22 version does not build, so created a recipe for version 0.21"
 RECIPE_LAST_UPDATE_pn-libconvert-asn1-perl = "Aug 13, 2010"
 RECIPE_MAINTAINER_pn-libconvert-asn1-perl = "Nitin A Kamble <nitin.a.kamble@intel.com>"
 
-- 
1.7.2.2

[PATCH 3/4] mklibs-native: new recipe for optimizing size of library files

[Nitin A Kamble]

From: Nitin A Kamble <nitin.a.kamble@intel.com>

Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
---
 .../mklibs/files/ac_init_fix.patch                 |   17 ++++++++++++++
 .../mklibs/mklibs-native_0.1.31.bb                 |   23 ++++++++++++++++++++
 2 files changed, 40 insertions(+), 0 deletions(-)
 create mode 100644 meta/recipes-devtools/mklibs/files/ac_init_fix.patch
 create mode 100644 meta/recipes-devtools/mklibs/mklibs-native_0.1.31.bb

diff --git a/meta/recipes-devtools/mklibs/files/ac_init_fix.patch b/meta/recipes-devtools/mklibs/files/ac_init_fix.patch
new file mode 100644
index 0000000..7325931
--- /dev/null
+++ b/meta/recipes-devtools/mklibs/files/ac_init_fix.patch
@@ -0,0 +1,17 @@
+Get the version of mklibs by simpler means. The MKLIBS_VERSION string in the 
+configure.ac file is replaced with real version string by the 
+do_configure_prepend() function from the recipe .bb file.
+
+Nitin A Kamble <nitin.a.kamble@intel.com>
+Date: 2011/01/24
+
+Index: mklibs/configure.ac
+===================================================================
+--- mklibs.orig/configure.ac	2010-02-21 17:34:56.000000000 -0800
++++ mklibs/configure.ac	2011-01-24 18:52:19.943242079 -0800
+@@ -1,4 +1,4 @@
+-AC_INIT([mklibs],m4_esyscmd(dpkg-parsechangelog | perl -ne 'print $1 if m/^Version: (.*)$/;'))
++AC_INIT([mklibs], MKLIBS_VERSION)
+ AM_INIT_AUTOMAKE([foreign no-define])
+ AC_CONFIG_HEADERS([config.h])
+ AM_MAINTAINER_MODE
diff --git a/meta/recipes-devtools/mklibs/mklibs-native_0.1.31.bb b/meta/recipes-devtools/mklibs/mklibs-native_0.1.31.bb
new file mode 100644
index 0000000..2044d12
--- /dev/null
+++ b/meta/recipes-devtools/mklibs/mklibs-native_0.1.31.bb
@@ -0,0 +1,23 @@
+DESCRIPTION = "mklibs produces cut-down shared libraries that contain only the routines required by a particular set of executables."
+HOMEPAGE = "https://code.launchpad.net/mklibs"
+SECTION = "devel"
+LICENSE = "GPLv2+"
+LIC_FILES_CHKSUM = "file://debian/copyright;md5=98d31037b13d896e33890738ef01af64"
+DEPENDS = "python-native"
+
+PR = "r0"
+
+SRC_URI = "http://ftp.de.debian.org/debian/pool/main/m/mklibs/${BPN}_${PV}.tar.gz \
+	file://ac_init_fix.patch\
+"
+
+SRC_URI[md5sum] = "f4df0307ccbdf60070e42277513f27ed"
+SRC_URI[sha256sum] = "8f5595621eb09d52871c771861e81b032d10c31d15e5dd61fa7f5a9e5b7de405"
+
+S = "${WORKDIR}/${BPN}"
+
+inherit autotools gettext native
+
+do_configure_prepend() {
+	sed "s+MKLIBS_VERSION+${PV}+" ${S}/configure.ac
+}
-- 
1.7.2.2

[PATCH 4/4] image-mklibs.bbclass: add the library optimization functionality

[Nitin A Kamble]

From: Nitin A Kamble <nitin.a.kamble@intel.com>

Thanks to mark hatle for his help in implementation of this.

Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
---
 meta/classes/image-mklibs.bbclass |   57 +++++++++++++++++++++++++++++++++++++
 meta/conf/local.conf.sample       |    2 +
 2 files changed, 59 insertions(+), 0 deletions(-)
 create mode 100644 meta/classes/image-mklibs.bbclass

diff --git a/meta/classes/image-mklibs.bbclass b/meta/classes/image-mklibs.bbclass
new file mode 100644
index 0000000..9fd2bc0
--- /dev/null
+++ b/meta/classes/image-mklibs.bbclass
@@ -0,0 +1,57 @@
+do_rootfs[depends] += "mklibs-native:do_populate_sysroot"
+
+IMAGE_PREPROCESS_COMMAND += "mklibs_optimize_image; "
+
+mklibs_optimize_image () {
+	rm -rf ${WORKDIR}/mklibs
+	mkdir -p ${WORKDIR}/mklibs/dest
+	cd ${IMAGE_ROOTFS}
+	du -bs > ${WORKDIR}/mklibs/du.before.mklibs.txt
+	for i in `find .`; do file $i; done \
+		| grep ELF \
+		| grep "LSB executable" \
+		| grep "dynamically linked" \
+		| sed "s/:.*//" \
+		| sed "s+^\./++" \
+		> ${WORKDIR}/mklibs/executables.list
+
+	case ${TARGET_ARCH} in
+		powerpc | mips )
+			dynamic_loader="/lib/ld.so.1"
+			;;
+		x86_64)
+			dynamic_loader="/lib/ld-linux-x86-64.so.2"
+			;;
+		i586 )
+			dynamic_loader="/lib/ld-linux.so.2"
+			;;
+		arm )
+			dynamic_loader="/lib/ld-linux.so.3"
+			;;
+		* )
+			dynamic_loader="/unknown_dynamic_linker"
+			;;
+	esac
+
+	mklibs -v \
+		--ldlib ${dynamic_loader} \
+		--sysroot ${PKG_CONFIG_SYSROOT_DIR} \
+		--root ${IMAGE_ROOTFS} \
+		--target `echo ${TARGET_PREFIX} | sed 's/-$//' ` \
+		-d ${WORKDIR}/mklibs/dest \
+		`cat ${WORKDIR}/mklibs/executables.list`
+
+	cd ${WORKDIR}/mklibs/dest
+	for i in *
+	do
+		cp $i `find ${IMAGE_ROOTFS} -name $i`
+	done
+
+	cd ${IMAGE_ROOTFS}
+	du -bs > ${WORKDIR}/mklibs/du.after.mklibs.txt
+
+	echo rootfs size before mklibs optimization: `cat ${WORKDIR}/mklibs/du.before.mklibs.txt`
+	echo rootfs size after mklibs optimization: `cat ${WORKDIR}/mklibs/du.after.mklibs.txt`
+}
+
+EXPORT_FUNCTIONS mklibs_optimize_image
diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
index f9953d1..a3667e9 100644
--- a/meta/conf/local.conf.sample
+++ b/meta/conf/local.conf.sample
@@ -74,8 +74,10 @@ EXTRA_IMAGE_FEATURES_mx31ads = "tools-testapps debug-tweaks"
 PACKAGE_CLASSES ?= "package_rpm package_ipk"
 
 # A list of additional classes to use when building the system
+# include 'image-mklibs' to reduce shared library files size for an image
 # include 'image-prelink' in order to prelink the filesystem image
 # include 'image-swab' to perform host system intrusion detection
+# NOTE: if listing mklibs & prelink both, then make sure mklibs is before prelink
 USER_CLASSES ?= "image-prelink"
 
 # POKYMODE controls the characteristics of the generated packages/images by
-- 
1.7.2.2

[PATCH 2/4] elfutils: add a missing depedancy

[Nitin A Kamble]

From: Nitin A Kamble <nitin.a.kamble@intel.com>

while implementing the mklibs optimization, found out that an executable from
elfutils package, viz eu-findtextrel is dynamically linked to libbz2.so
library. But the poky-image-minimal did not have that library.
  adding the depedency to the bzip2 to get that library for eu-fundtextrel
executable.

Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
---
 meta/recipes-devtools/elfutils/elfutils_0.148.bb |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/meta/recipes-devtools/elfutils/elfutils_0.148.bb b/meta/recipes-devtools/elfutils/elfutils_0.148.bb
index 44b3c52..14060e6 100644
--- a/meta/recipes-devtools/elfutils/elfutils_0.148.bb
+++ b/meta/recipes-devtools/elfutils/elfutils_0.148.bb
@@ -5,8 +5,9 @@ LICENSE = "GPLv2 with exceptions"
 LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3\
                     file://EXCEPTION;md5=570adcb0c1218ab57f2249c67d0ce417"
 DEPENDS = "libtool zlib"
+RDEPENDS = "bzip2"
 
-PR = "r1"
+PR = "r2"
 
 SRC_URI = "https://fedorahosted.org/releases/e/l/elfutils/elfutils-${PV}.tar.bz2"
 
-- 
1.7.2.2

Re: [PATCH 2/4] elfutils: add a missing depedancy

[Richard Purdie]

On Tue, 2011-02-01 at 03:57 -0800, Nitin A Kamble wrote:
> From: Nitin A Kamble <nitin.a.kamble@intel.com>
> 
> while implementing the mklibs optimization, found out that an executable from
> elfutils package, viz eu-findtextrel is dynamically linked to libbz2.so
> library. But the poky-image-minimal did not have that library.
>   adding the depedency to the bzip2 to get that library for eu-fundtextrel
> executable.
> 
> Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
> ---
>  meta/recipes-devtools/elfutils/elfutils_0.148.bb |    3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/meta/recipes-devtools/elfutils/elfutils_0.148.bb b/meta/recipes-devtools/elfutils/elfutils_0.148.bb
> index 44b3c52..14060e6 100644
> --- a/meta/recipes-devtools/elfutils/elfutils_0.148.bb
> +++ b/meta/recipes-devtools/elfutils/elfutils_0.148.bb
> @@ -5,8 +5,9 @@ LICENSE = "GPLv2 with exceptions"
>  LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3\
>                      file://EXCEPTION;md5=570adcb0c1218ab57f2249c67d0ce417"
>  DEPENDS = "libtool zlib"
> +RDEPENDS = "bzip2"

There should be a package name here, e.g. RDEPENDS_${PN}.

Also, why is the shlibs code not picking up the dependency
automatically? Is it dlopen'd?

Cheers,

Richard

Re: [PATCH 2/4] elfutils: add a missing depedancy

[Kamble, Nitin A]

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: Tuesday, February 01, 2011 2:00 PM
> To: Kamble, Nitin A
> Cc: poky@yoctoproject.org
> Subject: Re: [poky] [PATCH 2/4] elfutils: add a missing depedancy
> 
> On Tue, 2011-02-01 at 03:57 -0800, Nitin A Kamble wrote:
> > From: Nitin A Kamble <nitin.a.kamble@intel.com>
> >
> > while implementing the mklibs optimization, found out that an
> executable from
> > elfutils package, viz eu-findtextrel is dynamically linked to
> libbz2.so
> > library. But the poky-image-minimal did not have that library.
> >   adding the depedency to the bzip2 to get that library for eu-
> fundtextrel
> > executable.
> >
> > Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
> > ---
> >  meta/recipes-devtools/elfutils/elfutils_0.148.bb |    3 ++-
> >  1 files changed, 2 insertions(+), 1 deletions(-)
> >
> > diff --git a/meta/recipes-devtools/elfutils/elfutils_0.148.bb
> b/meta/recipes-devtools/elfutils/elfutils_0.148.bb
> > index 44b3c52..14060e6 100644
> > --- a/meta/recipes-devtools/elfutils/elfutils_0.148.bb
> > +++ b/meta/recipes-devtools/elfutils/elfutils_0.148.bb
> > @@ -5,8 +5,9 @@ LICENSE = "GPLv2 with exceptions"
> >  LIC_FILES_CHKSUM =
> "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3\
> >
> file://EXCEPTION;md5=570adcb0c1218ab57f2249c67d0ce417"
> >  DEPENDS = "libtool zlib"
> > +RDEPENDS = "bzip2"
> 
Thanks Richard for the review,

> There should be a package name here, e.g. RDEPENDS_${PN}.
I will fix the RDEPENDS_${PN}, and also fix the gplv3 recipe

> 
> Also, why is the shlibs code not picking up the dependency
> automatically? Is it dlopen'd?
Don't know. Not sure what you are asking.

Thanks,
Nitin

> 
> Cheers,
> 
> Richard