[PATCH 0/8] Missing deps discovered in per-recipe builds

[PATCH 0/8] Missing deps discovered in per-recipe builds

[Martin Jansa]

I was using simple script to build each component alone:

#!/bin/bash

rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/*
rm -rf log.*

ls -d buildhistory/packages/*/* | xargs -n 1 basename | sort -u > component.list
ncomponents=`cat component.list | wc -l`
i=1
for component in `cat component.list`; do
  echo "Building component: ${component} ($i/$ncomponents)"
  bitbake -c cleansstate ${component} | tee log.${component};
  bitbake ${component} | tee -a log.${component};
  rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/*
  i=`expr $i + 1`
done
grep "NOTE: Tasks Summary: Attempted.*failed.$" log.*

That should cover missing mandatory dependencies.
Comparing depends.dot files in buildhistory between incremental build and this per-component build should detect at least some autodetected dependencies. In per-component builds it can find only libraries already included in dependency tree, so dependencies lost in buildhistory aren't deterministic.
Even better would be to run per-component build once after building world without removing tmp-eglibc, that will allow components to autodetect all possible libraries and then run another per-component build with removing tmp-eglibc which allows to detect libraries from dependency-tree.


The following changes since commit 3e1dbabbf33a2e461abc92ff10cd970fe604ee38:

  sysstat: backport a patch to fix a parallel building error (2013-07-02 07:58:10 -0700)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib jansa/deps
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=jansa/deps

Martin Jansa (8):
  (lib)telepathy*: add missing dependency on libxslt-native
  gst-plugins-bad: inherit gsettings
  gst-plugins-base: add missing dependency on glib-2.0-native
  taglib: add missing dependency on zlib
  quota: inherit pkgconfig
  libpam: inherit pkgconfig
  systemtap: inherit pkgconfig
  eglibc-locale: add missing dependency on
    virtual/${TARGET_PREFIX}binutils

 meta/recipes-connectivity/telepathy/libtelepathy_0.3.3.bb            | 2 +-
 meta/recipes-connectivity/telepathy/telepathy-glib_0.20.1.bb         | 3 +--
 meta/recipes-connectivity/telepathy/telepathy-idle_0.1.14.bb         | 3 +--
 .../telepathy/telepathy-mission-control_5.14.0.bb                    | 4 +---
 meta/recipes-connectivity/telepathy/telepathy-python_0.15.19.bb      | 1 +
 meta/recipes-core/eglibc/eglibc-locale.inc                           | 5 +++++
 meta/recipes-extended/pam/libpam_1.1.6.bb                            | 2 +-
 meta/recipes-extended/quota/quota_4.01.bb                            | 2 +-
 meta/recipes-kernel/systemtap/systemtap_git.bb                       | 2 +-
 meta/recipes-multimedia/gstreamer/gst-plugins-bad_0.10.23.bb         | 2 +-
 meta/recipes-multimedia/gstreamer/gst-plugins-base_0.10.36.bb        | 2 +-
 meta/recipes-support/taglib/taglib_1.8.bb                            | 2 ++
 12 files changed, 17 insertions(+), 13 deletions(-)

-- 
1.8.2.1

[PATCH 1/8] (lib)telepathy*: add missing dependency on libxslt-native

[Martin Jansa]

* do_configure fails without:
  | configure:13590: error: xsltproc (from the libxslt source package) is required

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-connectivity/telepathy/libtelepathy_0.3.3.bb             | 2 +-
 meta/recipes-connectivity/telepathy/telepathy-glib_0.20.1.bb          | 3 +--
 meta/recipes-connectivity/telepathy/telepathy-idle_0.1.14.bb          | 3 +--
 .../telepathy/telepathy-mission-control_5.14.0.bb                     | 4 +---
 meta/recipes-connectivity/telepathy/telepathy-python_0.15.19.bb       | 1 +
 5 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/meta/recipes-connectivity/telepathy/libtelepathy_0.3.3.bb b/meta/recipes-connectivity/telepathy/libtelepathy_0.3.3.bb
index 7d0ef79..f13e709 100644
--- a/meta/recipes-connectivity/telepathy/libtelepathy_0.3.3.bb
+++ b/meta/recipes-connectivity/telepathy/libtelepathy_0.3.3.bb
@@ -4,7 +4,7 @@ communication, including instant messaging, voice calls and video calls.  It \
 abstracts differences between protocols to provide a unified interface for \
 applications."
 HOMEPAGE = "http://telepathy.freedesktop.org/wiki/"
-DEPENDS = "glib-2.0 dbus dbus-glib telepathy-glib"
+DEPENDS = "glib-2.0 dbus dbus-glib telepathy-glib libxslt-native"
 LICENSE = "LGPLv2.1+"
 LIC_FILES_CHKSUM = "file://COPYING;md5=a6f89e2100d9b6cdffcea4f398e37343 \
                     file://src/tp-conn.c;beginline=1;endline=19;md5=4c58069f77d601cc59200bce5396c7cb"
diff --git a/meta/recipes-connectivity/telepathy/telepathy-glib_0.20.1.bb b/meta/recipes-connectivity/telepathy/telepathy-glib_0.20.1.bb
index a0c3103..85071c0 100644
--- a/meta/recipes-connectivity/telepathy/telepathy-glib_0.20.1.bb
+++ b/meta/recipes-connectivity/telepathy/telepathy-glib_0.20.1.bb
@@ -1,9 +1,8 @@
 SUMMARY = "Telepathy Framework glib-base helper library"
 DESCRIPTION = "Telepathy Framework: GLib-based helper library for connection managers"
 HOMEPAGE = "http://telepathy.freedesktop.org/wiki/"
-DEPENDS = "glib-2.0 dbus python-native-runtime dbus-native dbus-glib"
+DEPENDS = "glib-2.0 dbus python-native-runtime dbus-native dbus-glib libxslt-native"
 LICENSE = "LGPLv2.1+"
-PR = "r0"
 
 SRC_URI = "http://telepathy.freedesktop.org/releases/telepathy-glib/${BPN}-${PV}.tar.gz \
            file://pkgconfig.patch"
diff --git a/meta/recipes-connectivity/telepathy/telepathy-idle_0.1.14.bb b/meta/recipes-connectivity/telepathy/telepathy-idle_0.1.14.bb
index 02250ea..4edcb3b 100644
--- a/meta/recipes-connectivity/telepathy/telepathy-idle_0.1.14.bb
+++ b/meta/recipes-connectivity/telepathy/telepathy-idle_0.1.14.bb
@@ -1,11 +1,10 @@
 SUMMARY = "Telepathy IRC connection manager"
 DESCRIPTION = "Telepathy implementation of the Internet Relay Chat protocols."
 HOMEPAGE = "http://telepathy.freedesktop.org/wiki/"
-DEPENDS = "glib-2.0 dbus telepathy-glib openssl"
+DEPENDS = "glib-2.0 dbus telepathy-glib openssl libxslt-native"
 LICENSE = "LGPLv2.1"
 LIC_FILES_CHKSUM = "file://COPYING;md5=2d5025d4aa3495befef8f17206a5b0a1 \
                     file://src/idle.c;beginline=1;endline=19;md5=b06b1e2594423111a1a7910b0eefc7f9"
-PR = "r0"
 
 SRC_URI = "http://telepathy.freedesktop.org/releases/${BPN}/${BPN}-${PV}.tar.gz \
            file://fix-svc-gtk-doc.h-target.patch"
diff --git a/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.14.0.bb b/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.14.0.bb
index b758dd9..23e4c97 100644
--- a/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.14.0.bb
+++ b/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.14.0.bb
@@ -4,12 +4,10 @@ LICENSE = "LGPLv2.1+"
 LIC_FILES_CHKSUM = "file://COPYING;md5=d8045f3b8f929c1cb29a1e3fd737b499 \
                     file://src/request.h;beginline=1;endline=21;md5=f80534d9af1c33291b3b79609f196eb2"
 SECTION = "libs"
-DEPENDS = "libtelepathy dbus-glib gconf"
+DEPENDS = "libtelepathy dbus-glib gconf libxslt-native"
 
 SRC_URI = "http://telepathy.freedesktop.org/releases/telepathy-mission-control/telepathy-mission-control-${PV}.tar.gz"
 
-PR = "r0"
-
 inherit autotools pkgconfig pythonnative
 
 PACKAGES =+ " \
diff --git a/meta/recipes-connectivity/telepathy/telepathy-python_0.15.19.bb b/meta/recipes-connectivity/telepathy/telepathy-python_0.15.19.bb
index f83bb2c..5a02d81 100644
--- a/meta/recipes-connectivity/telepathy/telepathy-python_0.15.19.bb
+++ b/meta/recipes-connectivity/telepathy/telepathy-python_0.15.19.bb
@@ -4,6 +4,7 @@ LICENSE = "LGPLv2.1+"
 LIC_FILES_CHKSUM = "file://COPYING;md5=2d5025d4aa3495befef8f17206a5b0a1 \
                     file://src/utils.py;beginline=1;endline=17;md5=9a07d1a9791a7429a14e7b25c6c86822"
 
+DEPENDS = "libxslt-native"
 RDEPENDS_${PN} += "python-dbus"
 
 SRC_URI = "http://telepathy.freedesktop.org/releases/${BPN}/${BPN}-${PV}.tar.gz \
-- 
1.8.2.1

[PATCH 2/8] gst-plugins-bad: inherit gsettings

[Martin Jansa]

* do_configure fails without native glib-compile-schemas:
  | checking for glib-compile-schemas... no
  | configure: error: glib-compile-schemas not found.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-multimedia/gstreamer/gst-plugins-bad_0.10.23.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-multimedia/gstreamer/gst-plugins-bad_0.10.23.bb b/meta/recipes-multimedia/gstreamer/gst-plugins-bad_0.10.23.bb
index a3a1204..7e80ce4 100644
--- a/meta/recipes-multimedia/gstreamer/gst-plugins-bad_0.10.23.bb
+++ b/meta/recipes-multimedia/gstreamer/gst-plugins-bad_0.10.23.bb
@@ -10,7 +10,7 @@ DEPENDS += "gst-plugins-base"
 
 PR = "r4"
 
-inherit gettext
+inherit gettext gsettings
 
 EXTRA_OECONF += "--disable-experimental \
                  --disable-sdl --disable-cdaudio --disable-directfb \
-- 
1.8.2.1

[PATCH 3/8] gst-plugins-base: add missing dependency on glib-2.0-native

[Martin Jansa]

* glib-genmarshal is needed during do_compile:
  | /bin/bash: line 1: glib-genmarshal: command not found

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-multimedia/gstreamer/gst-plugins-base_0.10.36.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-multimedia/gstreamer/gst-plugins-base_0.10.36.bb b/meta/recipes-multimedia/gstreamer/gst-plugins-base_0.10.36.bb
index 26c562c..19473ba 100644
--- a/meta/recipes-multimedia/gstreamer/gst-plugins-base_0.10.36.bb
+++ b/meta/recipes-multimedia/gstreamer/gst-plugins-base_0.10.36.bb
@@ -6,7 +6,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3 \
                     file://COPYING.LIB;md5=55ca817ccb7d5b5b66355690e9abc605 \
                     file://gst/ffmpegcolorspace/utils.c;beginline=1;endline=20;md5=9c83a200b8e597b26ca29df20fc6ecd0"
 
-DEPENDS += "alsa-lib liboil libogg libvorbis libtheora util-linux tremor"
+DEPENDS += "alsa-lib liboil libogg libvorbis libtheora util-linux tremor glib-2.0-native"
 
 SRC_URI += "file://gst-plugins-base-tremor.patch \
             file://configure.ac-fix-subparse-plugin.patch"
-- 
1.8.2.1

[PATCH 4/8] taglib: add missing dependency on zlib

[Martin Jansa]

* without target zlib it tries to use native one:
 | /OE/sysroots/x86_64-linux/usr/lib/libz.so: could not read symbols: File in wrong format
 | collect2: error: ld returned 1 exit status
 | make[2]: *** [taglib/libtag.so.1.12.0] Error 1

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-support/taglib/taglib_1.8.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-support/taglib/taglib_1.8.bb b/meta/recipes-support/taglib/taglib_1.8.bb
index 76d2e51..4c151b7 100644
--- a/meta/recipes-support/taglib/taglib_1.8.bb
+++ b/meta/recipes-support/taglib/taglib_1.8.bb
@@ -6,6 +6,8 @@ LIC_FILES_CHKSUM = "file://COPYING.LGPL;md5=4fbd65380cdd255951079008b364516c \
                     file://COPYING.MPL;md5=bfe1f75d606912a4111c90743d6c7325 \
                     file://taglib/audioproperties.h;beginline=1;endline=24;md5=9df2c7399519b7310568a7c55042ecee"
 
+DEPENDS = "zlib"
+
 PR = "r1"
 # http://developer.kde.org/~wheeler/files/src/taglib-${PV}.tar.gz
 SRC_URI = "https://github.com/downloads/taglib/taglib/taglib-1.8.tar.gz \
-- 
1.8.2.1

[PATCH 5/8] quota: inherit pkgconfig

[Martin Jansa]

* missing dependency on pkgconfig-native was causing
  that PKG_CHECK_MODULES(DBUS, dbus-1) stayed unexpanded in
  configure script:
  checking for ext2fs_initialize in -lext2fs... yes
  quota/4.01-r1/quota-tools/configure: line 3746: syntax error near unexpected token `DBUS,'
  quota/4.01-r1/quota-tools/configure: line 3746: `        PKG_CHECK_MODULES(DBUS, dbus-1)'
  Configure failed. The contents of all config.log files follows to aid debugging

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-extended/quota/quota_4.01.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-extended/quota/quota_4.01.bb b/meta/recipes-extended/quota/quota_4.01.bb
index 4dc7e68..9835f06 100644
--- a/meta/recipes-extended/quota/quota_4.01.bb
+++ b/meta/recipes-extended/quota/quota_4.01.bb
@@ -17,7 +17,7 @@ S = "${WORKDIR}/quota-tools"
 
 DEPENDS = "gettext-native e2fsprogs"
 
-inherit autotools gettext
+inherit autotools gettext pkgconfig
 
 EXTRA_OEMAKE += 'STRIP=""'
 
-- 
1.8.2.1

[PATCH 6/8] libpam: inherit pkgconfig

[Martin Jansa]

* missing dependency on pkgconfig-native was causing
  that PKG_CHECK_MODULES(DBUS, dbus-1) stayed unexpanded in
  configure script:
  checking for dbm_store in -lndbm... no
  libpam/1.1.6-r2/Linux-PAM-1.1.6/configure:
  line 14217: syntax error near unexpected token `libtirpc,'
  libpam/1.1.6-r2/Linux-PAM-1.1.6/configure:
  line 14217: `      PKG_CHECK_MODULES(libtirpc, libtirpc,'
  Configure failed. The contents of all config.log files follows to aid
  debugging

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-extended/pam/libpam_1.1.6.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-extended/pam/libpam_1.1.6.bb b/meta/recipes-extended/pam/libpam_1.1.6.bb
index f811f4c..62ad7b1 100644
--- a/meta/recipes-extended/pam/libpam_1.1.6.bb
+++ b/meta/recipes-extended/pam/libpam_1.1.6.bb
@@ -43,7 +43,7 @@ PR = "r2"
 
 S = "${WORKDIR}/Linux-PAM-${PV}"
 
-inherit autotools gettext
+inherit autotools gettext pkgconfig
 
 PACKAGES += "${PN}-runtime ${PN}-xtests"
 FILES_${PN} = "${base_libdir}/lib*${SOLIBS}"
-- 
1.8.2.1

[PATCH 7/8] systemtap: inherit pkgconfig

[Martin Jansa]

* systemtap-native was failing with undefined AC_DEFINE
  configure.ac:56: error: possibly undefined macro: AC_DEFINE
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.bb b/meta/recipes-kernel/systemtap/systemtap_git.bb
index 22f8959..9bba5a9 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.bb
+++ b/meta/recipes-kernel/systemtap/systemtap_git.bb
@@ -20,7 +20,7 @@ STAP_DOCS ?= "--disable-docs --disable-publican --disable-refdocs"
 
 EXTRA_OECONF += "${STAP_DOCS} "
 
-inherit autotools gettext
+inherit autotools gettext pkgconfig
 
 BBCLASSEXTEND = "native nativesdk"
 
-- 
1.8.2.1

[PATCH 8/8] eglibc-locale: add missing dependency on virtual/${TARGET_PREFIX}binutils

[Martin Jansa]

* eglibc-collateral.inc inhibits all default deps, but do_package needs
  objcopy:
  ERROR: objcopy failed with exit code 127 (cmd was
  'i586-webos-linux-objcopy' --only-keep-debug
  'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so'
  'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
  ERROR: Function failed: split_and_strip_files

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-core/eglibc/eglibc-locale.inc | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/recipes-core/eglibc/eglibc-locale.inc b/meta/recipes-core/eglibc/eglibc-locale.inc
index f4cb814..fc51022 100644
--- a/meta/recipes-core/eglibc/eglibc-locale.inc
+++ b/meta/recipes-core/eglibc/eglibc-locale.inc
@@ -5,6 +5,11 @@ SUMMARY = "Locale data from eglibc"
 BPN = "eglibc"
 LOCALEBASEPN = "${MLPREFIX}eglibc"
 
+# eglibc-collateral.inc inhibits all default deps, but do_package needs objcopy
+# ERROR: objcopy failed with exit code 127 (cmd was 'i586-webos-linux-objcopy' --only-keep-debug 'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so' 'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
+# ERROR: Function failed: split_and_strip_files
+DEPENDS = "virtual/${TARGET_PREFIX}binutils"
+
 # Binary locales are generated at build time if ENABLE_BINARY_LOCALE_GENERATION
 # is set. The idea is to avoid running localedef on the target (at first boot)
 # to decrease initial boot time and avoid localedef being killed by the OOM
-- 
1.8.2.1

Re: [PATCH 8/8] eglibc-locale: add missing dependency on virtual/${TARGET_PREFIX}binutils

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2267 bytes --]

On Fri, Jul 05, 2013 at 03:29:00AM +0200, Martin Jansa wrote:
> * eglibc-collateral.inc inhibits all default deps, but do_package needs
>   objcopy:
>   ERROR: objcopy failed with exit code 127 (cmd was
>   'i586-webos-linux-objcopy' --only-keep-debug
>   'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so'
>   'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
>   ERROR: Function failed: split_and_strip_files
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/recipes-core/eglibc/eglibc-locale.inc | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/meta/recipes-core/eglibc/eglibc-locale.inc b/meta/recipes-core/eglibc/eglibc-locale.inc
> index f4cb814..fc51022 100644
> --- a/meta/recipes-core/eglibc/eglibc-locale.inc
> +++ b/meta/recipes-core/eglibc/eglibc-locale.inc
> @@ -5,6 +5,11 @@ SUMMARY = "Locale data from eglibc"
>  BPN = "eglibc"
>  LOCALEBASEPN = "${MLPREFIX}eglibc"
>  
> +# eglibc-collateral.inc inhibits all default deps, but do_package needs objcopy
> +# ERROR: objcopy failed with exit code 127 (cmd was 'i586-webos-linux-objcopy' --only-keep-debug 'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so' 'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
> +# ERROR: Function failed: split_and_strip_files
> +DEPENDS = "virtual/${TARGET_PREFIX}binutils"
> +

This one doesn't work with nativesdk:

ERROR: Nothing PROVIDES 'virtual/nativesdk-x86_64-oesdk-linux-binutils'
(but
virtual:nativesdk:/OE/shr-core/openembedded-core/meta/recipes-core/eglibc/eglibc-locale_2.18.bb
DEPENDS on or otherwise requires it)
ERROR: Nothing RPROVIDES 'nativesdk-eglibc-charmaps' (but
virtual:nativesdk:/OE/shr-core/openembedded-core/meta/recipes-core/eglibc/eglibc-locale_2.18.bb
RDEPENDS on or otherwise requires it)
ERROR: No eligible RPROVIDERs exist for 'nativesdk-eglibc-charmaps'

I'll send v2 when I have a fix.

>  # Binary locales are generated at build time if ENABLE_BINARY_LOCALE_GENERATION
>  # is set. The idea is to avoid running localedef on the target (at first boot)
>  # to decrease initial boot time and avoid localedef being killed by the OOM
> -- 
> 1.8.2.1
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[PATCHv2 1/2] eglibc-locale: add missing dependency on virtual/${TARGET_PREFIX}binutils

[Martin Jansa]

* eglibc-collateral.inc inhibits all default deps, but do_package needs
  objcopy:
  ERROR: objcopy failed with exit code 127 (cmd was
  'i586-webos-linux-objcopy' --only-keep-debug
  'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so'
  'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
  ERROR: Function failed: split_and_strip_files

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-core/eglibc/eglibc-locale.inc | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/recipes-core/eglibc/eglibc-locale.inc b/meta/recipes-core/eglibc/eglibc-locale.inc
index f4cb814..c3fcc6d 100644
--- a/meta/recipes-core/eglibc/eglibc-locale.inc
+++ b/meta/recipes-core/eglibc/eglibc-locale.inc
@@ -5,6 +5,11 @@ SUMMARY = "Locale data from eglibc"
 BPN = "eglibc"
 LOCALEBASEPN = "${MLPREFIX}eglibc"
 
+# eglibc-collateral.inc inhibits all default deps, but do_package needs objcopy
+# ERROR: objcopy failed with exit code 127 (cmd was 'i586-webos-linux-objcopy' --only-keep-debug 'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so' 'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
+# ERROR: Function failed: split_and_strip_files
+DEPENDS_class-target = "virtual/${TARGET_PREFIX}binutils"
+
 # Binary locales are generated at build time if ENABLE_BINARY_LOCALE_GENERATION
 # is set. The idea is to avoid running localedef on the target (at first boot)
 # to decrease initial boot time and avoid localedef being killed by the OOM
-- 
1.8.2.1

[PATCHv2 2/2] subversion: explicitly disable sasl

[Martin Jansa]

* cyrus-sasl is in meta-networking and isn't in subversion DEPENDS

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-devtools/subversion/subversion_1.6.15.bb | 1 +
 meta/recipes-devtools/subversion/subversion_1.7.10.bb | 1 +
 2 files changed, 2 insertions(+)

diff --git a/meta/recipes-devtools/subversion/subversion_1.6.15.bb b/meta/recipes-devtools/subversion/subversion_1.6.15.bb
index 244d7ea..2a072e8 100644
--- a/meta/recipes-devtools/subversion/subversion_1.6.15.bb
+++ b/meta/recipes-devtools/subversion/subversion_1.6.15.bb
@@ -24,6 +24,7 @@ EXTRA_OECONF = " \
                 --without-berkeley-db --without-apxs --without-apache \
                 --without-swig --with-apr=${STAGING_BINDIR_CROSS} \
                 --with-apr-util=${STAGING_BINDIR_CROSS} \
+                --without-sasl \
                 ac_cv_path_RUBY=none"
 
 inherit autotools
diff --git a/meta/recipes-devtools/subversion/subversion_1.7.10.bb b/meta/recipes-devtools/subversion/subversion_1.7.10.bb
index 9d189fe..ded1788 100644
--- a/meta/recipes-devtools/subversion/subversion_1.7.10.bb
+++ b/meta/recipes-devtools/subversion/subversion_1.7.10.bb
@@ -22,6 +22,7 @@ EXTRA_OECONF = " \
                 --without-berkeley-db --without-apxs \
                 --without-swig --with-apr=${STAGING_BINDIR_CROSS} \
                 --with-apr-util=${STAGING_BINDIR_CROSS} \
+                --without-sasl \
                 ac_cv_path_RUBY=none"
 
 inherit autotools
-- 
1.8.2.1

Re: [PATCHv2 2/2] subversion: explicitly disable sasl

[Jonathan Liu]

On 6/07/2013 6:39 PM, Martin Jansa wrote:
> * cyrus-sasl is in meta-networking and isn't in subversion DEPENDS
>
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>   meta/recipes-devtools/subversion/subversion_1.6.15.bb | 1 +
>   meta/recipes-devtools/subversion/subversion_1.7.10.bb | 1 +
>   2 files changed, 2 insertions(+)
>
> diff --git a/meta/recipes-devtools/subversion/subversion_1.6.15.bb b/meta/recipes-devtools/subversion/subversion_1.6.15.bb
> index 244d7ea..2a072e8 100644
> --- a/meta/recipes-devtools/subversion/subversion_1.6.15.bb
> +++ b/meta/recipes-devtools/subversion/subversion_1.6.15.bb
> @@ -24,6 +24,7 @@ EXTRA_OECONF = " \
>                   --without-berkeley-db --without-apxs --without-apache \
>                   --without-swig --with-apr=${STAGING_BINDIR_CROSS} \
>                   --with-apr-util=${STAGING_BINDIR_CROSS} \
> +                --without-sasl \
>                   ac_cv_path_RUBY=none"
>   
>   inherit autotools
> diff --git a/meta/recipes-devtools/subversion/subversion_1.7.10.bb b/meta/recipes-devtools/subversion/subversion_1.7.10.bb
> index 9d189fe..ded1788 100644
> --- a/meta/recipes-devtools/subversion/subversion_1.7.10.bb
> +++ b/meta/recipes-devtools/subversion/subversion_1.7.10.bb
> @@ -22,6 +22,7 @@ EXTRA_OECONF = " \
>                   --without-berkeley-db --without-apxs \
>                   --without-swig --with-apr=${STAGING_BINDIR_CROSS} \
>                   --with-apr-util=${STAGING_BINDIR_CROSS} \
> +                --without-sasl \
>                   ac_cv_path_RUBY=none"
>   
>   inherit autotools
I would prefer a PACKAGECONFIG for this as I depend on subversion with 
SASL support.

Regards,
Jonathan

Re: [PATCHv2 2/2] subversion: explicitly disable sasl

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1828 bytes --]

On Sat, Jul 06, 2013 at 10:39:22AM +0200, Martin Jansa wrote:
> * cyrus-sasl is in meta-networking and isn't in subversion DEPENDS
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/recipes-devtools/subversion/subversion_1.6.15.bb | 1 +
>  meta/recipes-devtools/subversion/subversion_1.7.10.bb | 1 +
>  2 files changed, 2 insertions(+)

I'll change this into PACKAGECONFIG, Jonathan Liu reported that he needs
subversion with sasl support.

> 
> diff --git a/meta/recipes-devtools/subversion/subversion_1.6.15.bb b/meta/recipes-devtools/subversion/subversion_1.6.15.bb
> index 244d7ea..2a072e8 100644
> --- a/meta/recipes-devtools/subversion/subversion_1.6.15.bb
> +++ b/meta/recipes-devtools/subversion/subversion_1.6.15.bb
> @@ -24,6 +24,7 @@ EXTRA_OECONF = " \
>                  --without-berkeley-db --without-apxs --without-apache \
>                  --without-swig --with-apr=${STAGING_BINDIR_CROSS} \
>                  --with-apr-util=${STAGING_BINDIR_CROSS} \
> +                --without-sasl \
>                  ac_cv_path_RUBY=none"
>  
>  inherit autotools
> diff --git a/meta/recipes-devtools/subversion/subversion_1.7.10.bb b/meta/recipes-devtools/subversion/subversion_1.7.10.bb
> index 9d189fe..ded1788 100644
> --- a/meta/recipes-devtools/subversion/subversion_1.7.10.bb
> +++ b/meta/recipes-devtools/subversion/subversion_1.7.10.bb
> @@ -22,6 +22,7 @@ EXTRA_OECONF = " \
>                  --without-berkeley-db --without-apxs \
>                  --without-swig --with-apr=${STAGING_BINDIR_CROSS} \
>                  --with-apr-util=${STAGING_BINDIR_CROSS} \
> +                --without-sasl \
>                  ac_cv_path_RUBY=none"
>  
>  inherit autotools
> -- 
> 1.8.2.1
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCHv2 1/2] eglibc-locale: add missing dependency on virtual/${TARGET_PREFIX}binutils

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]

On Sat, Jul 6, 2013 at 1:39 AM, Martin Jansa <martin.jansa@gmail.com> wrote:

> * eglibc-collateral.inc inhibits all default deps, but do_package needs
>   objcopy:
>   ERROR: objcopy failed with exit code 127 (cmd was
>   'i586-webos-linux-objcopy' --only-keep-debug
>   'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so'
>   'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
>   ERROR: Function failed: split_and_strip_files
>

Wouldn't it be better to add the necessary dep to the do_package[depends]
in package.bbclass?
-- 
Christopher Larson

[-- Attachment #2: Type: text/html, Size: 971 bytes --]

Re: [PATCHv2 1/2] eglibc-locale: add missing dependency on virtual/${TARGET_PREFIX}binutils

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

On Sat, Jul 06, 2013 at 03:56:59PM -0700, Chris Larson wrote:
> On Sat, Jul 6, 2013 at 1:39 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
> 
> > * eglibc-collateral.inc inhibits all default deps, but do_package needs
> >   objcopy:
> >   ERROR: objcopy failed with exit code 127 (cmd was
> >   'i586-webos-linux-objcopy' --only-keep-debug
> >   'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so'
> >   'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
> >   ERROR: Function failed: split_and_strip_files
> >
> 
> Wouldn't it be better to add the necessary dep to the do_package[depends]
> in package.bbclass?

In this case yes, but what about allarch recipes, creating
binutils-cross dependency in their do_package task would probably make
their stamps depending on architecture.

I have similar patch adding virtual/${TARGET_PREFIX}binutils in
BASEDEPENDS, that won't fix this eglibc-locale (because it has
INHIBIT_DEFAULT_DEPS), but it's needed in cases where
gcc-cross.do_configure -> binutils-cross.do_populate_sysroot is skipped
when gcc-cross is reused from sstate.

I'll send RFC for that one now (I was trying to debug why it also brings
dependency on do_package_setscene)

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[RFC][PATCH] SHLIBSSEARCHDIRS

[Martin Jansa]

---
 meta/classes/package.bbclass | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 3f2deca..55035b5 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1322,11 +1322,18 @@ python package_do_shlibs() {
     libdir_re = re.compile(".*/%s$" % d.getVar('baselib', True))
     
     shlibs_search_dirs = d.getVar('SHLIBSSEARCHDIRS', True)
-    for dir in shlibs_search_dirs:
+    shlibs_search_dirs_re_txt = ""
+    for dir in shlibs_search_dirs.split(' '):
+        # strip leading and trailing slash, it's added in regexp
+        if dir.endswith("/"):
+            dir = dir[:-1]
+        if dir.startswith("/"):
+            dir = dir[1:]
         if shlibs_search_dirs_re:
-            shlibs_search_dirs_re += "\|"
-        shlibs_search_dirs_re += "\(^.*\/%s\/.*$\)" % dir
-    bb.debug(2, "will use following RE to search for provides sonames %s" % shlibs_search_dirs)
+            shlibs_search_dirs_re_txt += "\|"
+        shlibs_search_dirs_re_txt += "\(^.*/%s/.*$\)" % dir
+    shlibs_search_dirs_re = re.compile(shlibs_search_dirs_re_txt)
+    bb.debug(2, "will use following RE to search for provides sonames %s" % shlibs_search_dirs_re_txt)
 
     packages = d.getVar('PACKAGES', True)
     targetos = d.getVar('TARGET_OS', True)
@@ -1389,7 +1396,7 @@ python package_do_shlibs() {
                         if not private_libs or -1 == private_libs.find(this_soname):
                             sonames.append(this_soname)
                     else:
-                        bb.debug(2, "ignoring soname %s from %s, because path doesn't match %s" % (this_soname, file, shlibs_search_dirs))
+                        bb.debug(2, "ignoring soname %s from %s, because path doesn't match %s" % (this_soname, file, shlibs_search_dirs_re_txt))
                 if libdir_re.match(os.path.dirname(file)):
                     needs_ldconfig = True
                 if snap_symlinks and (os.path.basename(file) != this_soname):
-- 
1.8.2.1

Re: [RFC][PATCH] SHLIBSSEARCHDIRS

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 304 bytes --]

On Sun, Jul 07, 2013 at 01:15:54AM +0200, Martin Jansa wrote:
> ---
>  meta/classes/package.bbclass | 17 ++++++++++++-----
>  1 file changed, 12 insertions(+), 5 deletions(-)

Ignore this one, wrong git revision passed to send-email..

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[RFC][PATCH] base.bbclass: add binutils to BASEDEPENDS

[Martin Jansa]

* when gcc-cross is reused from the sstate cache then
  gcc-cross's dependency on binutils-cross.do_populate_sysroot
  is ignored due to being in the safe dependency list in
  setscene_depvalid
* many packages still require binutils-cross.do_populate_sysroot
  to be executed before their do_configure (or do_compile)
* usually some other recipe runs do_package task (which also
  require binutils-cross.do_populate_sysroot, before do_configure
  is executed

* rebuilding one recipe with everything else reused from sstate-cache
  before:
  $ grep binutils-cross log
  NOTE: Running setscene task 251 of 412 (oe-core/meta/recipes-devtools/binutils/binutils-cross_2.23.1.bb, do_package_write_ipk_setscene)
  NOTE: recipe binutils-cross-2.23.1-r3: task do_package_write_ipk_setscene: Started
  NOTE: recipe binutils-cross-2.23.1-r3: task do_package_write_ipk_setscene: Succeeded
  NOTE: Running setscene task 254 of 412 (oe-core/meta/recipes-devtools/binutils/binutils-cross_2.23.1.bb, do_packagedata_setscene)
  NOTE: recipe binutils-cross-2.23.1-r3: task do_packagedata_setscene: Started
  NOTE: recipe binutils-cross-2.23.1-r3: task do_packagedata_setscene: Succeeded
  NOTE: Running noexec task 209 of 1157 (ID: 530, oe-core/meta/recipes-devtools/binutils/binutils-cross_2.23.1.bb, do_package_write)

  after:
  $ grep binutils-cross log2
  NOTE: Running setscene task 216 of 412 (oe-core/meta/recipes-devtools/binutils/binutils-cross_2.23.1.bb, do_package_write_ipk_setscene)
  NOTE: recipe binutils-cross-2.23.1-r3: task do_package_write_ipk_setscene: Started
  NOTE: recipe binutils-cross-2.23.1-r3: task do_package_write_ipk_setscene: Succeeded
  NOTE: Running setscene task 341 of 412 (oe-core/meta/recipes-devtools/binutils/binutils-cross_2.23.1.bb, do_packagedata_setscene)
  NOTE: recipe binutils-cross-2.23.1-r3: task do_packagedata_setscene: Started
  NOTE: recipe binutils-cross-2.23.1-r3: task do_packagedata_setscene: Succeeded
  NOTE: Running setscene task 344 of 412 (oe-core/meta/recipes-devtools/binutils/binutils-cross_2.23.1.bb, do_package_setscene)
  NOTE: recipe binutils-cross-2.23.1-r3: task do_package_setscene: Started
  NOTE: recipe binutils-cross-2.23.1-r3: task do_package_setscene: Succeeded
  NOTE: Running setscene task 389 of 412 (oe-core/meta/recipes-devtools/binutils/binutils-cross_2.23.1.bb, do_populate_sysroot_setscene)
  NOTE: recipe binutils-cross-2.23.1-r3: task do_populate_sysroot_setscene: Started
  NOTE: recipe binutils-cross-2.23.1-r3: task do_populate_sysroot_setscene: Succeeded
  NOTE: Running noexec task 187 of 1157 (ID: 108, oe-core/meta/recipes-devtools/binutils/binutils-cross_2.23.1.bb, do_package_write)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/base.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 27edea8..4383d2e 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -72,7 +72,7 @@ def base_dep_prepend(d):
     # the application.
     if not d.getVar('INHIBIT_DEFAULT_DEPS'):
         if (d.getVar('HOST_SYS', True) != d.getVar('BUILD_SYS', True)):
-            deps += " virtual/${TARGET_PREFIX}gcc virtual/${TARGET_PREFIX}compilerlibs virtual/libc "
+            deps += " virtual/${TARGET_PREFIX}gcc virtual/${TARGET_PREFIX}compilerlibs virtual/${TARGET_PREFIX}binutils virtual/libc "
     return deps
 
 BASEDEPENDS = "${@base_dep_prepend(d)}"
-- 
1.8.2.1

Re: [PATCHv2 1/2] eglibc-locale: add missing dependency on virtual/${TARGET_PREFIX}binutils

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

On Sat, Jul 6, 2013 at 4:08 PM, Martin Jansa <martin.jansa@gmail.com> wrote:

> In this case yes, but what about allarch recipes, creating
> binutils-cross dependency in their do_package task would probably make
> their stamps depending on architecture.
>

def get_binutils_dep(d):
    if 'target' in d.getVar('OVERRIDES', True).split():
        return d.expand('virtual/${TARGET_PREFIX}binutils')
    else:
        return ''

do_package[depends] += "${@get_binutils_dep(d)}"


-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1311 bytes --]

Re: [PATCHv2 1/2] eglibc-locale: add missing dependency on virtual/${TARGET_PREFIX}binutils

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

On Sat, Jul 06, 2013 at 04:20:00PM -0700, Chris Larson wrote:
> On Sat, Jul 6, 2013 at 4:08 PM, Martin Jansa <martin.jansa@gmail.com> wrote:
> 
> > In this case yes, but what about allarch recipes, creating
> > binutils-cross dependency in their do_package task would probably make
> > their stamps depending on architecture.
> >
> 
> def get_binutils_dep(d):
>     if 'target' in d.getVar('OVERRIDES', True).split():
>         return d.expand('virtual/${TARGET_PREFIX}binutils')
>     else:
>         return ''
> 
> do_package[depends] += "${@get_binutils_dep(d)}"

class-target will be in recipes which inherits allarch

that code should also respect
EXCLUDE_FROM_SHLIBS
INHIBIT_PACKAGE_DEBUG_SPLIT
INHIBIT_PACKAGE_STRIP
flags

I think objcopy is needed only with some combination of these.

With binutils-cross in BASEDEPENDS I don't know how many recipes will
need do_package -> binutils-cross.do_populate_sysroot, maybe just this
one eglibc-locale...

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCHv2 1/2] eglibc-locale: add missing dependency on virtual/${TARGET_PREFIX}binutils

[Khem Raj]

On Jul 6, 2013, at 1:39 AM, Martin Jansa <martin.jansa@gmail.com> wrote:

> * eglibc-collateral.inc inhibits all default deps, but do_package needs
>  objcopy:
>  ERROR: objcopy failed with exit code 127 (cmd was
>  'i586-webos-linux-objcopy' --only-keep-debug
>  'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so'
>  'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
>  ERROR: Function failed: split_and_strip_files
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
> meta/recipes-core/eglibc/eglibc-locale.inc | 5 +++++
> 1 file changed, 5 insertions(+)
> 
> diff --git a/meta/recipes-core/eglibc/eglibc-locale.inc b/meta/recipes-core/eglibc/eglibc-locale.inc
> index f4cb814..c3fcc6d 100644
> --- a/meta/recipes-core/eglibc/eglibc-locale.inc
> +++ b/meta/recipes-core/eglibc/eglibc-locale.inc
> @@ -5,6 +5,11 @@ SUMMARY = "Locale data from eglibc"
> BPN = "eglibc"
> LOCALEBASEPN = "${MLPREFIX}eglibc"
> 
> +# eglibc-collateral.inc inhibits all default deps, but do_package needs objcopy
> +# ERROR: objcopy failed with exit code 127 (cmd was 'i586-webos-linux-objcopy' --only-keep-debug 'eglibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so' 'eglibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
> +# ERROR: Function failed: split_and_strip_files
> +DEPENDS_class-target = "virtual/${TARGET_PREFIX}binutils"


Looks good.

Re: [PATCH 0/8] Missing deps discovered in per-recipe builds

[Burton, Ross]

On 5 July 2013 02:28, Martin Jansa <martin.jansa@gmail.com> wrote:
> I was using simple script to build each component alone:

I was literally going to write this script on Monday, great stuff.
Surprisingly few patches for oe-core! :)

Ross

Re: [PATCH 0/8] Missing deps discovered in per-recipe builds

[Paul Eggleton]

On Friday 05 July 2013 03:28:21 Martin Jansa wrote:
> I was using simple script to build each component alone:
> 
> #!/bin/bash
> 
> rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
> tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone
> tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps
> tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* rm -rf log.*
> 
> ls -d buildhistory/packages/*/* | xargs -n 1 basename | sort -u >
> component.list ncomponents=`cat component.list | wc -l`
> i=1
> for component in `cat component.list`; do
>   echo "Building component: ${component} ($i/$ncomponents)"
>   bitbake -c cleansstate ${component} | tee log.${component};
>   bitbake ${component} | tee -a log.${component};
>   rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
> tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone
> tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps
> tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* i=`expr $i +
> 1`
> done

A cleaned up form of this would be a great addition to the scripts/contrib 
directory if you have time...

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/8] Missing deps discovered in per-recipe builds

[Burton, Ross]

On 5 July 2013 16:28, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
>>   rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
>> tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone
>> tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps
>> tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/*

Do you really need all those rms?  wipe-sysroot will blast away the
sysroot which will then be re-populated from sstate, and the
per-package work directories will be cleaned out by the -ccleansstate
(that does a clean + remove from sstate, right?)

Ross

Re: [PATCH 0/8] Missing deps discovered in per-recipe builds

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1763 bytes --]

On Fri, Jul 05, 2013 at 04:28:24PM +0100, Paul Eggleton wrote:
> On Friday 05 July 2013 03:28:21 Martin Jansa wrote:
> > I was using simple script to build each component alone:
> > 
> > #!/bin/bash
> > 
> > rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
> > tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone
> > tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps
> > tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* rm -rf log.*
> > 
> > ls -d buildhistory/packages/*/* | xargs -n 1 basename | sort -u >
> > component.list ncomponents=`cat component.list | wc -l`
> > i=1
> > for component in `cat component.list`; do
> >   echo "Building component: ${component} ($i/$ncomponents)"
> >   bitbake -c cleansstate ${component} | tee log.${component};
> >   bitbake ${component} | tee -a log.${component};
> >   rm -rf tmp-eglibc/abi_version tmp-eglibc/buildstats tmp-eglibc/cache
> > tmp-eglibc/deploy tmp-eglibc/log tmp-eglibc/pkgdata pseudodone
> > tmp-eglibc/saved_tmpdir tmp-eglibc/sstate-control tmp-eglibc/stamps
> > tmp-eglibc/sysroots tmp-eglibc/work-shared tmp-eglibc/work/* i=`expr $i +
> > 1`
> > done
> 
> A cleaned up form of this would be a great addition to the scripts/contrib 
> directory if you have time...

I'm working on improved version of this script which will detect
automagically enabled dependencies. Once I get it working I'll clean it
and send patch with script.

Unfortunately running it takes very long time (even on fast server),
normal "bitbake world" is done in +- 24 hours, but with repopulating
tmp-eglibc so many times (even from sstate-cache) it's almost 48 hours.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[RFC][PATCH] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/test-dependencies.sh | 250 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 250 insertions(+)
 create mode 100755 scripts/test-dependencies.sh

diff --git a/scripts/test-dependencies.sh b/scripts/test-dependencies.sh
new file mode 100755
index 0000000..99e2ca0
--- /dev/null
+++ b/scripts/test-dependencies.sh
@@ -0,0 +1,250 @@
+#!/bin/sh
+
+# Used to detect missing dependencies or automagically
+# enabled dependencies which aren't explicitly enabled
+# or disabled.
+
+# It does 3 builds of <target>
+# 1st to populate sstate-cache directory and sysroot
+# 2nd to rebuild each recipe with every possible
+#     dependency found in sysroot (which stays populated
+#     from 1st build
+# 3rd to rebuild each recipe only with dependencies defined
+#     in DEPENDS
+# 4th (optional) repeat build like 3rd to make sure that
+#     minimal versions of dependencies defined in DEPENDS
+#     is also enough
+
+# Global vars
+tmpdir=
+targets=
+recipes=
+buildhistory=
+buildtype=
+default_targets="world"
+default_buildhistory="buildhistory"
+default_buildtype="1 2 3 c"
+
+usage () {
+  cat << EOF
+Welcome to utility to detect missing or autoenabled dependencies.
+WARNING: this utility will completely remove your tmpdir (make sure
+         you don't have important buildhistory or persistent dir there).
+$0 <OPTION>
+
+Options:
+  -h, --help
+        Display this help and exit.
+
+  --tmpdir=<tmpdir>
+        Specify tmpdir, will use the environment variable TMPDIR if it is not specified.
+        Something like /OE/oe-core/tmp-eglibc (no / at the end).
+
+  --targets=<targets>
+        List of targets separated by space, will use the environment variable TARGETS if it is not specified.
+        It will run "bitbake <targets>" to populate sysroots.
+        Default value is "world".
+
+  --recipes=<recipes>
+        File with list of recipes we want to rebuild with minimal and maximal sysroot.
+        Will use the environment variable RECIPES if it is not specified.
+        Default value will use all packages ever recorded in buildhistory directory.
+
+  --buildhistory=<buildhistory>
+        Path to buildhistory directory, it needs to be enabled in your config,
+        because it's used to detect different dependencies and to create list
+        of recipes to rebuild when it's not specified.
+        Will use the environment variable BUILDHISTORY if it is not specified.
+        Default value is "buildhistory"
+
+  --buildtype=<buildtype>
+        There are 4 types of build:
+          1: build to populate sstate-cache directory and sysroot
+          2: build to rebuild each recipe with every possible dep
+          3: build to rebuild each recipe with minimal dependencies
+          4: build to rebuild each recipe again with minimal dependencies
+          c: compare buildhistory directories from build 2 and 3
+        Will use the environment variable BUILDTYPE if it is not specified.
+        Default value is "1 2 3 c", order is important, type 4 is optional.
+EOF
+}
+
+# Print error information and exit.
+echo_error () {
+  echo "ERROR: $1" >&2
+  exit 1
+}
+
+while [ -n "$1" ]; do
+  case $1 in
+    --tmpdir=*)
+      tmpdir=`echo $1 | sed -e 's#^--tmpdir=##' | xargs readlink -e`
+      [ -d "$tmpdir" ] || echo_error "Invalid argument to --tmpdir"
+      shift
+        ;;
+    --targets=*)
+      targets=`echo $1 | sed -e 's#^--targets="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --recipes=*)
+      recipes=`echo $1 | sed -e 's#^--recipes="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --buildhistory=*)
+      buildhistory=`echo $1 | sed -e 's#^--buildhistory="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --buildtype=*)
+      buildtype=`echo $1 | sed -e 's#^--buildtype="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --help|-h)
+      usage
+      exit 0
+        ;;
+    *)
+      echo "Invalid arguments $*"
+      echo_error "Try '$0 -h' for more information."
+        ;;
+  esac
+done
+
+# tmpdir directory, use environment variable TMPDIR
+# if it was not specified, otherwise, error.
+[ -n "$tmpdir" ] || tmpdir=$TMPDIR
+[ -n "$tmpdir" ] || echo_error "No tmpdir found!"
+[ -d "$tmpdir" ] || echo_error "Invalid tmpdir \"$tmpdir\""
+[ -n "$targets" ] || targets=$TARGETS
+[ -n "$targets" ] || targets=$default_targets
+[ -n "$recipes" ] || recipes=$RECIPES
+[ -n "$recipes" -a ! -f "$recipes" ] && echo_error "Invalid file with list of recipes to rebuild"
+[ -n "$recipes" ] || echo "All packages ever recorded in buildhistory directory will be rebuilt"
+[ -n "$buildhistory" ] || buildhistory=$BUILDHISTORY
+[ -n "$buildhistory" ] || buildhistory=$default_buildhistory
+[ -d "$buildhistory" ] || echo_error "Invalid buildhistory directory \"$buildhistory\""
+[ -n "$buildtype" ] || buildtype=$BUILDTYPE
+[ -n "$buildtype" ] || buildtype=$default_buildtype
+echo "$buildtype" | grep -v '^[1234c ]*$' && echo_error "Invalid buildtype \"$buildtype\", only some combination of 1, 2, 3, 4, c separated by space is allowed"
+
+OUTPUT_BASE=test-dependencies/`date "+%s"`
+
+build_all() {
+  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
+  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
+  mkdir -p ${OUTPUT1}
+  echo "Logs will be stored in ${OUTPUT1} directory"
+  bitbake $targets | tee -a ${OUTPUT1}/complete.log
+}
+
+build_every_recipe_max() {
+  echo "===== 2nd build to rebuild each recipe with every possible dep ====="
+  OUTPUT2=${OUTPUT_BASE}/${TYPE}_max
+  mkdir -p ${OUTPUT2}
+  echo "Logs will be stored in ${OUTPUT2} directory"
+  if [ -z "$recipes" ]; then
+    ls -d $buildhistory/packages/*/* | xargs -n 1 basename | sort -u > ${OUTPUT2}/recipe.list
+    recipes=${OUTPUT2}/recipe.list
+  fi
+  i=1
+  count=`cat $recipes | wc -l`
+  for recipe in `cat $recipes`; do
+    echo "Building recipe: ${recipe} ($i/$count)"
+    bitbake -c cleansstate ${recipe} > ${OUTPUT2}/log.${recipe} 2>&1;
+    bitbake ${recipe} >> ${OUTPUT2}/log.${recipe} 2>&1;
+    grep "ERROR: Task.*failed" ${OUTPUT2}/log.${recipe}
+    # here we keep tmpdir between builds
+    i=`expr $i + 1`
+  done
+  echo "Copying buildhistory/packages to ${OUTPUT2}"
+  cp -ra $buildhistory/packages ${OUTPUT2}
+  grep "ERROR: Task.*failed" ${OUTPUT2}/log.* | tee ${OUTPUT2}/failed
+}
+
+build_every_recipe_min() {
+  echo "===== 3rd or 4th build to rebuild each recipe with minimal dependencies ====="
+  OUTPUT3=${OUTPUT_BASE}/${TYPE}_min
+  mkdir -p ${OUTPUT3}
+  echo "Logs will be stored in ${OUTPUT3} directory"
+
+  if [ -z "$recipes" ]; then
+    ls -d $buildhistory/packages/*/* | xargs -n 1 basename | sort -u > ${OUTPUT3}/recipe.list
+    recipes=${OUTPUT3}/recipe.list
+  fi
+  echo "!!!Removing tmpdir \"$tmpdir\"!!!"
+  rm -rf $tmpdir/* 2>/dev/null
+  i=1
+  count=`cat $recipes | wc -l`
+  for recipe in `cat $recipes`; do
+    echo "Building recipe: ${recipe} ($i/$count)"
+    bitbake -c cleansstate ${recipe} > ${OUTPUT3}/log.${recipe} 2>&1;
+    bitbake ${recipe} >> ${OUTPUT3}/log.${recipe} 2>&1;
+    grep "ERROR: Task.*failed" ${OUTPUT3}/log.${recipe}
+    rm -rf $tmpdir/* 2>/dev/null
+    i=`expr $i + 1`
+  done
+  echo "Copying buildhistory/packages to ${OUTPUT3}"
+  cp -ra $buildhistory/packages ${OUTPUT3}
+  grep "ERROR: Task.*failed" ${OUTPUT3}/log.* | tee ${OUTPUT3}/failed
+}
+
+compare_deps() {
+  # you can run just compare task with command like this
+  # OUTPUT2=test-dependencies/1373140172/2_max \
+  # OUTPUT3=test-dependencies/1373140172/3_min \
+  # OUTPUTC=test-dependencies/1373140172 \
+  # openembedded-core/scripts/test-dependencies.sh --tmpdir=tmp-eglibc --targets=glib-2.0 --recipes=recipe_list --buildtype=c
+  echo "===== Compare dependencies recorded in \"${OUTPUT2}\" and \"${OUTPUT3}\" ====="
+  [ -n "${OUTPUTC}" ] || OUTPUTC=${OUTPUT_BASE}
+  mkdir -p ${OUTPUTC}
+  OUTPUT_FILE=${OUTPUTC}/dependency-changes
+  echo "Differences will be stored in ${OUTPUT_FILE}, dot is shown for every 100 of checked packages"
+  echo > ${OUTPUT_FILE}
+
+  [ -d ${OUTPUT2} ] || echo_error "Directory with output from build 2 \"${OUTPUT2}\" does not exist"
+  [ -d ${OUTPUT3} ] || echo_error "Directory with output from build 3 \"${OUTPUT3}\" does not exist"
+  [ -d ${OUTPUT2}/packages/ ] || echo_error "Directory with packages from build 2 \"${OUTPUT2}/packages/\" does not exist"
+  [ -d ${OUTPUT3}/packages/ ] || echo_error "Directory with packages from build 3 \"${OUTPUT3}/packages/\" does not exist"
+  i=0
+  find ${OUTPUT2}/packages/ -name latest | sed "s#${OUTPUT2}/##g" | while read pkg; do
+    max_pkg=${OUTPUT2}/${pkg}
+    min_pkg=${OUTPUT3}/${pkg}
+    if [ ! -f "${min_pkg}" ] ; then
+      echo "ERROR: ${min_pkg} doesn't exist" | tee ${OUTPUT_FILE}
+    fi
+    # strip version information in parenthesis
+    max_deps=`grep "^RDEPENDS = " ${max_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
+    min_deps=`grep "^RDEPENDS = " ${min_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
+    if [ "$i" = 100 ] ; then
+      echo -n "." # cheap progressbar
+      i=0
+    fi
+    if [ "${max_deps}" = "${min_deps}" ] ; then
+      # it's annoying long, but at least it's showing some progress, warnings are grepped at the end
+      echo "NOTE: ${pkg} dependencies weren't changed" >> ${OUTPUT_FILE}
+    else
+      missing_deps=
+      for dep in ${max_deps}; do
+        echo "${min_deps}" | grep -q " ${dep} " || missing_deps="${missing_deps} ${dep}"
+      done
+      if [ -n "${missing_deps}" ] ; then
+        echo # to get rid of dots on last line
+        echo "WARN: ${pkg} lost dependency on ${missing_deps}" | tee -a ${OUTPUT_FILE}
+      fi
+    fi
+    i=`expr $i + 1`
+  done
+  echo # to get rid of dots on last line
+  echo "Found differences: "
+  grep "^WARN: " ${OUTPUT_FILE}
+}
+
+for TYPE in $buildtype; do
+  case ${TYPE} in
+    1) build_all;;
+    2) build_every_recipe_max;;
+    3) build_every_recipe_min;;
+    4) build_every_recipe_min;;
+    c) compare_deps;;
+    *) echo_error "Invalid buildtype \"$TYPE\""
+  esac
+done
-- 
1.8.2.1

Re: [RFC][PATCH] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

On Sat, Jul 06, 2013 at 11:24:46PM +0200, Martin Jansa wrote:
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  scripts/test-dependencies.sh | 250 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 250 insertions(+)
>  create mode 100755 scripts/test-dependencies.sh
> 
> +build_all() {
> +  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
> +  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
> +  mkdir -p ${OUTPUT1}
> +  echo "Logs will be stored in ${OUTPUT1} directory"
> +  bitbake $targets | tee -a ${OUTPUT1}/complete.log
> +}

I've just added "-k" here to bitbake call, more improvements are expected, 
that's why it's only RFC, but I wanted to share it soon in case someone
else also have access to some builder which is idle during weekends..

It would be nice to test and fix oe-core recipes first, but because my
target is to run this in dylan I'm running this with more layers
included already.

I'm sending patches for master + blacklisting broken recipes in dylan,
maybe we should backport them later as missing deps and autodetected
deps are relatively safe fixes.

Regards,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [RFC][PATCH] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Randy MacLeod]

On 13-07-06 05:39 PM, Martin Jansa wrote:
> On Sat, Jul 06, 2013 at 11:24:46PM +0200, Martin Jansa wrote:
>> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
>> ---
>>   scripts/test-dependencies.sh | 250 +++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 250 insertions(+)
>>   create mode 100755 scripts/test-dependencies.sh
>>
>> +build_all() {
>> +  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
>> +  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
>> +  mkdir -p ${OUTPUT1}
>> +  echo "Logs will be stored in ${OUTPUT1} directory"
>> +  bitbake $targets | tee -a ${OUTPUT1}/complete.log
>> +}
>
> I've just added "-k" here to bitbake call, more improvements are expected,
> that's why it's only RFC, but I wanted to share it soon in case someone
> else also have access to some builder which is idle during weekends..
>
> It would be nice to test and fix oe-core recipes first, but because my
> target is to run this in dylan I'm running this with more layers
> included already.

Martin,

Such a per-recipe dependency test has been on our to do list!
Once your script is in master, we can run it at least once a week.

// Randy

>
> I'm sending patches for master + blacklisting broken recipes in dylan,
> maybe we should backport them later as missing deps and autodetected
> deps are relatively safe fixes.
>
> Regards,
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350

Re: [RFC][PATCH] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2103 bytes --]

On Mon, Jul 08, 2013 at 02:00:20PM -0400, Randy MacLeod wrote:
> On 13-07-06 05:39 PM, Martin Jansa wrote:
> > On Sat, Jul 06, 2013 at 11:24:46PM +0200, Martin Jansa wrote:
> >> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> >> ---
> >>   scripts/test-dependencies.sh | 250 +++++++++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 250 insertions(+)
> >>   create mode 100755 scripts/test-dependencies.sh
> >>
> >> +build_all() {
> >> +  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
> >> +  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
> >> +  mkdir -p ${OUTPUT1}
> >> +  echo "Logs will be stored in ${OUTPUT1} directory"
> >> +  bitbake $targets | tee -a ${OUTPUT1}/complete.log
> >> +}
> >
> > I've just added "-k" here to bitbake call, more improvements are expected,
> > that's why it's only RFC, but I wanted to share it soon in case someone
> > else also have access to some builder which is idle during weekends..
> >
> > It would be nice to test and fix oe-core recipes first, but because my
> > target is to run this in dylan I'm running this with more layers
> > included already.
> 
> Martin,
> 
> Such a per-recipe dependency test has been on our to do list!
> Once your script is in master, we can run it at least once a week.
> 

Great, my tests revealed about 100 packages with autodetected extra
dependencies, I'll send improved version of this script soon (together
with results from my world builds (23 layers included).

Regards,

> > I'm sending patches for master + blacklisting broken recipes in dylan,
> > maybe we should backport them later as missing deps and autodetected
> > deps are relatively safe fixes.
> >
> > Regards,
> >
> >
> >
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >
> 
> 
> -- 
> # Randy MacLeod. SMTS, Linux, Wind River
> Direct: 613.963.1350

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[PATCHv2] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/test-dependencies.sh | 253 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 253 insertions(+)
 create mode 100755 scripts/test-dependencies.sh

diff --git a/scripts/test-dependencies.sh b/scripts/test-dependencies.sh
new file mode 100755
index 0000000..405c14e
--- /dev/null
+++ b/scripts/test-dependencies.sh
@@ -0,0 +1,253 @@
+#!/bin/sh
+
+# Author: Martin Jansa <martin.jansa@gmail.com>
+#
+# Copyright (c) 2013 Martin Jansa <Martin.Jansa@gmail.com>
+
+# Used to detect missing dependencies or automagically
+# enabled dependencies which aren't explicitly enabled
+# or disabled.
+
+# It does 3 builds of <target>
+# 1st to populate sstate-cache directory and sysroot
+# 2nd to rebuild each recipe with every possible
+#     dependency found in sysroot (which stays populated
+#     from 1st build
+# 3rd to rebuild each recipe only with dependencies defined
+#     in DEPENDS
+# 4th (optional) repeat build like 3rd to make sure that
+#     minimal versions of dependencies defined in DEPENDS
+#     is also enough
+
+# Global vars
+tmpdir=
+targets=
+recipes=
+buildhistory=
+buildtype=
+default_targets="world"
+default_buildhistory="buildhistory"
+default_buildtype="1 2 3 c"
+
+usage () {
+  cat << EOF
+Welcome to utility to detect missing or autoenabled dependencies.
+WARNING: this utility will completely remove your tmpdir (make sure
+         you don't have important buildhistory or persistent dir there).
+$0 <OPTION>
+
+Options:
+  -h, --help
+        Display this help and exit.
+
+  --tmpdir=<tmpdir>
+        Specify tmpdir, will use the environment variable TMPDIR if it is not specified.
+        Something like /OE/oe-core/tmp-eglibc (no / at the end).
+
+  --targets=<targets>
+        List of targets separated by space, will use the environment variable TARGETS if it is not specified.
+        It will run "bitbake <targets>" to populate sysroots.
+        Default value is "world".
+
+  --recipes=<recipes>
+        File with list of recipes we want to rebuild with minimal and maximal sysroot.
+        Will use the environment variable RECIPES if it is not specified.
+        Default value will use all packages ever recorded in buildhistory directory.
+
+  --buildhistory=<buildhistory>
+        Path to buildhistory directory, it needs to be enabled in your config,
+        because it's used to detect different dependencies and to create list
+        of recipes to rebuild when it's not specified.
+        Will use the environment variable BUILDHISTORY if it is not specified.
+        Default value is "buildhistory"
+
+  --buildtype=<buildtype>
+        There are 4 types of build:
+          1: build to populate sstate-cache directory and sysroot
+          2: build to rebuild each recipe with every possible dep
+          3: build to rebuild each recipe with minimal dependencies
+          4: build to rebuild each recipe again with minimal dependencies
+          c: compare buildhistory directories from build 2 and 3
+        Will use the environment variable BUILDTYPE if it is not specified.
+        Default value is "1 2 3 c", order is important, type 4 is optional.
+EOF
+}
+
+# Print error information and exit.
+echo_error () {
+  echo "ERROR: $1" >&2
+  exit 1
+}
+
+while [ -n "$1" ]; do
+  case $1 in
+    --tmpdir=*)
+      tmpdir=`echo $1 | sed -e 's#^--tmpdir=##' | xargs readlink -e`
+      [ -d "$tmpdir" ] || echo_error "Invalid argument to --tmpdir"
+      shift
+        ;;
+    --targets=*)
+      targets=`echo $1 | sed -e 's#^--targets="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --recipes=*)
+      recipes=`echo $1 | sed -e 's#^--recipes="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --buildhistory=*)
+      buildhistory=`echo $1 | sed -e 's#^--buildhistory="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --buildtype=*)
+      buildtype=`echo $1 | sed -e 's#^--buildtype="*\([^"]*\)"*#\1#'`
+      shift
+        ;;
+    --help|-h)
+      usage
+      exit 0
+        ;;
+    *)
+      echo "Invalid arguments $*"
+      echo_error "Try '$0 -h' for more information."
+        ;;
+  esac
+done
+
+# tmpdir directory, use environment variable TMPDIR
+# if it was not specified, otherwise, error.
+[ -n "$tmpdir" ] || tmpdir=$TMPDIR
+[ -n "$tmpdir" ] || echo_error "No tmpdir found!"
+[ -d "$tmpdir" ] || echo_error "Invalid tmpdir \"$tmpdir\""
+[ -n "$targets" ] || targets=$TARGETS
+[ -n "$targets" ] || targets=$default_targets
+[ -n "$recipes" ] || recipes=$RECIPES
+[ -n "$recipes" -a ! -f "$recipes" ] && echo_error "Invalid file with list of recipes to rebuild"
+[ -n "$recipes" ] || echo "All packages ever recorded in buildhistory directory will be rebuilt"
+[ -n "$buildhistory" ] || buildhistory=$BUILDHISTORY
+[ -n "$buildhistory" ] || buildhistory=$default_buildhistory
+[ -d "$buildhistory" ] || echo_error "Invalid buildhistory directory \"$buildhistory\""
+[ -n "$buildtype" ] || buildtype=$BUILDTYPE
+[ -n "$buildtype" ] || buildtype=$default_buildtype
+echo "$buildtype" | grep -v '^[1234c ]*$' && echo_error "Invalid buildtype \"$buildtype\", only some combination of 1, 2, 3, 4, c separated by space is allowed"
+
+OUTPUT_BASE=test-dependencies/`date "+%s"`
+
+build_all() {
+  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
+  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
+  mkdir -p ${OUTPUT1}
+  echo "Logs will be stored in ${OUTPUT1} directory"
+  bitbake -k $targets | tee -a ${OUTPUT1}/complete.log
+}
+
+build_every_recipe() {
+  if [ "${TYPE}" = "2" ] ; then
+    echo "===== 2nd build to rebuild each recipe with every possible dep ====="
+    OUTPUT_MAX=${OUTPUT_BASE}/${TYPE}_max
+    OUTPUTB=${OUTPUT_MAX}
+  else
+    echo "===== 3rd or 4th build to rebuild each recipe with minimal dependencies ====="
+    OUTPUT_MIN=${OUTPUT_BASE}/${TYPE}_min
+    OUTPUTB=${OUTPUT_MIN}
+  fi
+  
+  mkdir -p ${OUTPUTB} ${OUTPUTB}/failed ${OUTPUTB}/ok
+  echo "Logs will be stored in ${OUTPUTB} directory"
+  if [ -z "$recipes" ]; then
+    ls -d $buildhistory/packages/*/* | xargs -n 1 basename | sort -u > ${OUTPUTB}/recipe.list
+    recipes=${OUTPUTB}/recipe.list
+  fi
+  if [ "${TYPE}" != "2" ] ; then
+    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
+    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null
+  fi
+  i=1
+  count=`cat $recipes | wc -l`
+  for recipe in `cat $recipes`; do
+    echo "Building recipe: ${recipe} ($i/$count)"
+    bitbake -c cleansstate ${recipe} > ${OUTPUTB}/log.${recipe} 2>&1;
+    bitbake ${recipe} >> ${OUTPUTB}/log.${recipe} 2>&1;
+    grep "ERROR: Task.*failed" ${OUTPUTB}/log.${recipe} && mv ${OUTPUTB}/log.${recipe} ${OUTPUTB}/failed/${recipe} || mv ${OUTPUTB}/log.${recipe} ${OUTPUTB}/ok/${recipe}
+    if [ "${TYPE}" != "2" ] ; then
+      rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null
+    fi
+    i=`expr $i + 1`
+  done
+  echo "Copying buildhistory/packages to ${OUTPUTB}"
+  cp -ra $buildhistory/packages ${OUTPUTB}
+  # This will be usefull to see which library is pulling new dependency
+  echo "Copying do_package logs to ${OUTPUTB}/do_package/"
+  mkdir ${OUTPUTB}/do_package
+  find $tmpdir/work/ -name log.do_package | while read f; do
+    # pn is 3 levels back, but we don't know if there is just one log per pn (only one arch and version)
+    # dest=`echo $f | sed 's#^.*/\([^/]*\)/\([^/]*\)/\([^/]*\)/log.do_package#\1#g'`
+    dest=`echo $f | sed "s#$tmpdir/work/##g; s#/#_#g"`
+    cp $f ${OUTPUTB}/do_package/$dest
+  done
+  grep "ERROR: Task.*failed" ${OUTPUTB}/failed/*
+}
+
+compare_deps() {
+  # you can run just compare task with command like this
+  # OUTPUT_MAX=test-dependencies/1373140172/2_max \
+  # OUTPUT_MIN=test-dependencies/1373140172/3_min \
+  # OUTPUTC=test-dependencies/1373140172 \
+  # openembedded-core/scripts/test-dependencies.sh --tmpdir=tmp-eglibc --targets=glib-2.0 --recipes=recipe_list --buildtype=c
+  echo "===== Compare dependencies recorded in \"${OUTPUT_MAX}\" and \"${OUTPUT_MIN}\" ====="
+  [ -n "${OUTPUTC}" ] || OUTPUTC=${OUTPUT_BASE}
+  mkdir -p ${OUTPUTC}
+  OUTPUT_FILE=${OUTPUTC}/dependency-changes
+  echo "Differences will be stored in ${OUTPUT_FILE}, dot is shown for every 100 of checked packages"
+  echo > ${OUTPUT_FILE}
+
+  [ -d ${OUTPUT_MAX} ] || echo_error "Directory with output from build 2 \"${OUTPUT_MAX}\" does not exist"
+  [ -d ${OUTPUT_MIN} ] || echo_error "Directory with output from build 3 \"${OUTPUT_MIN}\" does not exist"
+  [ -d ${OUTPUT_MAX}/packages/ ] || echo_error "Directory with packages from build 2 \"${OUTPUT_MAX}/packages/\" does not exist"
+  [ -d ${OUTPUT_MIN}/packages/ ] || echo_error "Directory with packages from build 3 \"${OUTPUT_MIN}/packages/\" does not exist"
+  i=0
+  find ${OUTPUT_MAX}/packages/ -name latest | sed "s#${OUTPUT_MAX}/##g" | while read pkg; do
+    max_pkg=${OUTPUT_MAX}/${pkg}
+    min_pkg=${OUTPUT_MIN}/${pkg}
+    if [ ! -f "${min_pkg}" ] ; then
+      echo "ERROR: ${min_pkg} doesn't exist" | tee -a ${OUTPUT_FILE}
+      continue
+    fi
+    # strip version information in parenthesis
+    max_deps=`grep "^RDEPENDS = " ${max_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
+    min_deps=`grep "^RDEPENDS = " ${min_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
+    if [ "$i" = 100 ] ; then
+      echo -n "." # cheap progressbar
+      i=0
+    fi
+    if [ "${max_deps}" = "${min_deps}" ] ; then
+      # it's annoying long, but at least it's showing some progress, warnings are grepped at the end
+      echo "NOTE: ${pkg} dependencies weren't changed" >> ${OUTPUT_FILE}
+    else
+      missing_deps=
+      for dep in ${max_deps}; do
+        echo "${min_deps}" | grep -q " ${dep} " || missing_deps="${missing_deps} ${dep}"
+      done
+      if [ -n "${missing_deps}" ] ; then
+        echo # to get rid of dots on last line
+        echo "WARN: ${pkg} lost dependency on ${missing_deps}" | tee -a ${OUTPUT_FILE}
+      fi
+    fi
+    i=`expr $i + 1`
+  done
+  echo # to get rid of dots on last line
+  echo "Found differences: "
+  grep "^WARN: " ${OUTPUT_FILE}
+  echo "Found errors: "
+  grep "^ERROR: " ${OUTPUT_FILE}
+}
+
+for TYPE in $buildtype; do
+  case ${TYPE} in
+    1) build_all;;
+    2) build_every_recipe;;
+    3) build_every_recipe;;
+    4) build_every_recipe;;
+    c) compare_deps;;
+    *) echo_error "Invalid buildtype \"$TYPE\""
+  esac
+done
-- 
1.8.2.1

Re: [PATCHv2] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 11797 bytes --]

On Wed, Jul 10, 2013 at 02:38:13PM +0200, Martin Jansa wrote:
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  scripts/test-dependencies.sh | 253 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 253 insertions(+)
>  create mode 100755 scripts/test-dependencies.sh

Build with last version of this script is still running (after 37 hours)
Building recipe: gnome-desktop (485/1848)
but I've tested it in smaller builds and I'm happy with current
functionality, I'll send found issues in tomorrow or day after that when
the build is complete.

> diff --git a/scripts/test-dependencies.sh b/scripts/test-dependencies.sh
> new file mode 100755
> index 0000000..405c14e
> --- /dev/null
> +++ b/scripts/test-dependencies.sh
> @@ -0,0 +1,253 @@
> +#!/bin/sh
> +
> +# Author: Martin Jansa <martin.jansa@gmail.com>
> +#
> +# Copyright (c) 2013 Martin Jansa <Martin.Jansa@gmail.com>
> +
> +# Used to detect missing dependencies or automagically
> +# enabled dependencies which aren't explicitly enabled
> +# or disabled.
> +
> +# It does 3 builds of <target>
> +# 1st to populate sstate-cache directory and sysroot
> +# 2nd to rebuild each recipe with every possible
> +#     dependency found in sysroot (which stays populated
> +#     from 1st build
> +# 3rd to rebuild each recipe only with dependencies defined
> +#     in DEPENDS
> +# 4th (optional) repeat build like 3rd to make sure that
> +#     minimal versions of dependencies defined in DEPENDS
> +#     is also enough
> +
> +# Global vars
> +tmpdir=
> +targets=
> +recipes=
> +buildhistory=
> +buildtype=
> +default_targets="world"
> +default_buildhistory="buildhistory"
> +default_buildtype="1 2 3 c"
> +
> +usage () {
> +  cat << EOF
> +Welcome to utility to detect missing or autoenabled dependencies.
> +WARNING: this utility will completely remove your tmpdir (make sure
> +         you don't have important buildhistory or persistent dir there).
> +$0 <OPTION>
> +
> +Options:
> +  -h, --help
> +        Display this help and exit.
> +
> +  --tmpdir=<tmpdir>
> +        Specify tmpdir, will use the environment variable TMPDIR if it is not specified.
> +        Something like /OE/oe-core/tmp-eglibc (no / at the end).
> +
> +  --targets=<targets>
> +        List of targets separated by space, will use the environment variable TARGETS if it is not specified.
> +        It will run "bitbake <targets>" to populate sysroots.
> +        Default value is "world".
> +
> +  --recipes=<recipes>
> +        File with list of recipes we want to rebuild with minimal and maximal sysroot.
> +        Will use the environment variable RECIPES if it is not specified.
> +        Default value will use all packages ever recorded in buildhistory directory.
> +
> +  --buildhistory=<buildhistory>
> +        Path to buildhistory directory, it needs to be enabled in your config,
> +        because it's used to detect different dependencies and to create list
> +        of recipes to rebuild when it's not specified.
> +        Will use the environment variable BUILDHISTORY if it is not specified.
> +        Default value is "buildhistory"
> +
> +  --buildtype=<buildtype>
> +        There are 4 types of build:
> +          1: build to populate sstate-cache directory and sysroot
> +          2: build to rebuild each recipe with every possible dep
> +          3: build to rebuild each recipe with minimal dependencies
> +          4: build to rebuild each recipe again with minimal dependencies
> +          c: compare buildhistory directories from build 2 and 3
> +        Will use the environment variable BUILDTYPE if it is not specified.
> +        Default value is "1 2 3 c", order is important, type 4 is optional.
> +EOF
> +}
> +
> +# Print error information and exit.
> +echo_error () {
> +  echo "ERROR: $1" >&2
> +  exit 1
> +}
> +
> +while [ -n "$1" ]; do
> +  case $1 in
> +    --tmpdir=*)
> +      tmpdir=`echo $1 | sed -e 's#^--tmpdir=##' | xargs readlink -e`
> +      [ -d "$tmpdir" ] || echo_error "Invalid argument to --tmpdir"
> +      shift
> +        ;;
> +    --targets=*)
> +      targets=`echo $1 | sed -e 's#^--targets="*\([^"]*\)"*#\1#'`
> +      shift
> +        ;;
> +    --recipes=*)
> +      recipes=`echo $1 | sed -e 's#^--recipes="*\([^"]*\)"*#\1#'`
> +      shift
> +        ;;
> +    --buildhistory=*)
> +      buildhistory=`echo $1 | sed -e 's#^--buildhistory="*\([^"]*\)"*#\1#'`
> +      shift
> +        ;;
> +    --buildtype=*)
> +      buildtype=`echo $1 | sed -e 's#^--buildtype="*\([^"]*\)"*#\1#'`
> +      shift
> +        ;;
> +    --help|-h)
> +      usage
> +      exit 0
> +        ;;
> +    *)
> +      echo "Invalid arguments $*"
> +      echo_error "Try '$0 -h' for more information."
> +        ;;
> +  esac
> +done
> +
> +# tmpdir directory, use environment variable TMPDIR
> +# if it was not specified, otherwise, error.
> +[ -n "$tmpdir" ] || tmpdir=$TMPDIR
> +[ -n "$tmpdir" ] || echo_error "No tmpdir found!"
> +[ -d "$tmpdir" ] || echo_error "Invalid tmpdir \"$tmpdir\""
> +[ -n "$targets" ] || targets=$TARGETS
> +[ -n "$targets" ] || targets=$default_targets
> +[ -n "$recipes" ] || recipes=$RECIPES
> +[ -n "$recipes" -a ! -f "$recipes" ] && echo_error "Invalid file with list of recipes to rebuild"
> +[ -n "$recipes" ] || echo "All packages ever recorded in buildhistory directory will be rebuilt"
> +[ -n "$buildhistory" ] || buildhistory=$BUILDHISTORY
> +[ -n "$buildhistory" ] || buildhistory=$default_buildhistory
> +[ -d "$buildhistory" ] || echo_error "Invalid buildhistory directory \"$buildhistory\""
> +[ -n "$buildtype" ] || buildtype=$BUILDTYPE
> +[ -n "$buildtype" ] || buildtype=$default_buildtype
> +echo "$buildtype" | grep -v '^[1234c ]*$' && echo_error "Invalid buildtype \"$buildtype\", only some combination of 1, 2, 3, 4, c separated by space is allowed"
> +
> +OUTPUT_BASE=test-dependencies/`date "+%s"`
> +
> +build_all() {
> +  echo "===== 1st build to populate sstate-cache directory and sysroot ====="
> +  OUTPUT1=${OUTPUT_BASE}/${TYPE}_all
> +  mkdir -p ${OUTPUT1}
> +  echo "Logs will be stored in ${OUTPUT1} directory"
> +  bitbake -k $targets | tee -a ${OUTPUT1}/complete.log
> +}
> +
> +build_every_recipe() {
> +  if [ "${TYPE}" = "2" ] ; then
> +    echo "===== 2nd build to rebuild each recipe with every possible dep ====="
> +    OUTPUT_MAX=${OUTPUT_BASE}/${TYPE}_max
> +    OUTPUTB=${OUTPUT_MAX}
> +  else
> +    echo "===== 3rd or 4th build to rebuild each recipe with minimal dependencies ====="
> +    OUTPUT_MIN=${OUTPUT_BASE}/${TYPE}_min
> +    OUTPUTB=${OUTPUT_MIN}
> +  fi
> +  
> +  mkdir -p ${OUTPUTB} ${OUTPUTB}/failed ${OUTPUTB}/ok
> +  echo "Logs will be stored in ${OUTPUTB} directory"
> +  if [ -z "$recipes" ]; then
> +    ls -d $buildhistory/packages/*/* | xargs -n 1 basename | sort -u > ${OUTPUTB}/recipe.list
> +    recipes=${OUTPUTB}/recipe.list
> +  fi
> +  if [ "${TYPE}" != "2" ] ; then
> +    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
> +    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null
> +  fi
> +  i=1
> +  count=`cat $recipes | wc -l`
> +  for recipe in `cat $recipes`; do
> +    echo "Building recipe: ${recipe} ($i/$count)"
> +    bitbake -c cleansstate ${recipe} > ${OUTPUTB}/log.${recipe} 2>&1;
> +    bitbake ${recipe} >> ${OUTPUTB}/log.${recipe} 2>&1;
> +    grep "ERROR: Task.*failed" ${OUTPUTB}/log.${recipe} && mv ${OUTPUTB}/log.${recipe} ${OUTPUTB}/failed/${recipe} || mv ${OUTPUTB}/log.${recipe} ${OUTPUTB}/ok/${recipe}
> +    if [ "${TYPE}" != "2" ] ; then
> +      rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null
> +    fi
> +    i=`expr $i + 1`
> +  done
> +  echo "Copying buildhistory/packages to ${OUTPUTB}"
> +  cp -ra $buildhistory/packages ${OUTPUTB}
> +  # This will be usefull to see which library is pulling new dependency
> +  echo "Copying do_package logs to ${OUTPUTB}/do_package/"
> +  mkdir ${OUTPUTB}/do_package
> +  find $tmpdir/work/ -name log.do_package | while read f; do
> +    # pn is 3 levels back, but we don't know if there is just one log per pn (only one arch and version)
> +    # dest=`echo $f | sed 's#^.*/\([^/]*\)/\([^/]*\)/\([^/]*\)/log.do_package#\1#g'`
> +    dest=`echo $f | sed "s#$tmpdir/work/##g; s#/#_#g"`
> +    cp $f ${OUTPUTB}/do_package/$dest
> +  done
> +  grep "ERROR: Task.*failed" ${OUTPUTB}/failed/*
> +}
> +
> +compare_deps() {
> +  # you can run just compare task with command like this
> +  # OUTPUT_MAX=test-dependencies/1373140172/2_max \
> +  # OUTPUT_MIN=test-dependencies/1373140172/3_min \
> +  # OUTPUTC=test-dependencies/1373140172 \
> +  # openembedded-core/scripts/test-dependencies.sh --tmpdir=tmp-eglibc --targets=glib-2.0 --recipes=recipe_list --buildtype=c
> +  echo "===== Compare dependencies recorded in \"${OUTPUT_MAX}\" and \"${OUTPUT_MIN}\" ====="
> +  [ -n "${OUTPUTC}" ] || OUTPUTC=${OUTPUT_BASE}
> +  mkdir -p ${OUTPUTC}
> +  OUTPUT_FILE=${OUTPUTC}/dependency-changes
> +  echo "Differences will be stored in ${OUTPUT_FILE}, dot is shown for every 100 of checked packages"
> +  echo > ${OUTPUT_FILE}
> +
> +  [ -d ${OUTPUT_MAX} ] || echo_error "Directory with output from build 2 \"${OUTPUT_MAX}\" does not exist"
> +  [ -d ${OUTPUT_MIN} ] || echo_error "Directory with output from build 3 \"${OUTPUT_MIN}\" does not exist"
> +  [ -d ${OUTPUT_MAX}/packages/ ] || echo_error "Directory with packages from build 2 \"${OUTPUT_MAX}/packages/\" does not exist"
> +  [ -d ${OUTPUT_MIN}/packages/ ] || echo_error "Directory with packages from build 3 \"${OUTPUT_MIN}/packages/\" does not exist"
> +  i=0
> +  find ${OUTPUT_MAX}/packages/ -name latest | sed "s#${OUTPUT_MAX}/##g" | while read pkg; do
> +    max_pkg=${OUTPUT_MAX}/${pkg}
> +    min_pkg=${OUTPUT_MIN}/${pkg}
> +    if [ ! -f "${min_pkg}" ] ; then
> +      echo "ERROR: ${min_pkg} doesn't exist" | tee -a ${OUTPUT_FILE}
> +      continue
> +    fi
> +    # strip version information in parenthesis
> +    max_deps=`grep "^RDEPENDS = " ${max_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
> +    min_deps=`grep "^RDEPENDS = " ${min_pkg} | sed 's/^RDEPENDS = / /g; s/$/ /g; s/([^(]*)//g'`
> +    if [ "$i" = 100 ] ; then
> +      echo -n "." # cheap progressbar
> +      i=0
> +    fi
> +    if [ "${max_deps}" = "${min_deps}" ] ; then
> +      # it's annoying long, but at least it's showing some progress, warnings are grepped at the end
> +      echo "NOTE: ${pkg} dependencies weren't changed" >> ${OUTPUT_FILE}
> +    else
> +      missing_deps=
> +      for dep in ${max_deps}; do
> +        echo "${min_deps}" | grep -q " ${dep} " || missing_deps="${missing_deps} ${dep}"
> +      done
> +      if [ -n "${missing_deps}" ] ; then
> +        echo # to get rid of dots on last line
> +        echo "WARN: ${pkg} lost dependency on ${missing_deps}" | tee -a ${OUTPUT_FILE}
> +      fi
> +    fi
> +    i=`expr $i + 1`
> +  done
> +  echo # to get rid of dots on last line
> +  echo "Found differences: "
> +  grep "^WARN: " ${OUTPUT_FILE}
> +  echo "Found errors: "
> +  grep "^ERROR: " ${OUTPUT_FILE}
> +}
> +
> +for TYPE in $buildtype; do
> +  case ${TYPE} in
> +    1) build_all;;
> +    2) build_every_recipe;;
> +    3) build_every_recipe;;
> +    4) build_every_recipe;;
> +    c) compare_deps;;
> +    *) echo_error "Invalid buildtype \"$TYPE\""
> +  esac
> +done
> -- 
> 1.8.2.1
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCHv2] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Burton, Ross]

On 10 July 2013 13:41, Martin Jansa <martin.jansa@gmail.com> wrote:
> Build with last version of this script is still running (after 37 hours)
> Building recipe: gnome-desktop (485/1848)
> but I've tested it in smaller builds and I'm happy with current
> functionality, I'll send found issues in tomorrow or day after that when
> the build is complete.

As this has been on my todo list for a while now, come to ELC-E and
I'll get you a beer. :)

Anyone volunteering to run this on oe-core master this weekend?

Ross

Re: [PATCHv2] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 951 bytes --]

On Wed, Jul 10, 2013 at 03:34:58PM +0100, Burton, Ross wrote:
> On 10 July 2013 13:41, Martin Jansa <martin.jansa@gmail.com> wrote:
> > Build with last version of this script is still running (after 37 hours)
> > Building recipe: gnome-desktop (485/1848)
> > but I've tested it in smaller builds and I'm happy with current
> > functionality, I'll send found issues in tomorrow or day after that when
> > the build is complete.
> 
> As this has been on my todo list for a while now, come to ELC-E and
> I'll get you a beer. :)

OK :)

> Anyone volunteering to run this on oe-core master this weekend?

I have it running with more layers included so it will take longer, but
will report more "autodetected" deps.

Well for some reason the server is very slow now, it moved only couple
recipes since writting that previous e-mail :/

Building recipe: gnumeric (505/1848)
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

sstate-cache and making a package "host-dependent"

[Mike Looijmans]

I added a buildserver that also exports its "sstate-cache" directory, so 
that other build machines can grab their stuff from it. This works fine, 
but I have one problem. Some packages are meant to be dependent on the 
system that built it. I want to enforce that each build machine creates 
its own package, and does not grab it from the sstate-cache of the 
central server.

For example,

$ cat recipes-core/meta/distro-feed-configs.bbappend
PRINC="1"
DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
DISTRO_FEED_NAME ?= "feed"
DISTRO_FEED_PREFIX = "topic"
DISTRO_FEED_URI = 
"http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"


The purpose being that the host name of the machien that built the image 
ends up in the opkg config files. This works just fine.
Now that we have a central server, all images on all build hosts grab 
the package from the sstate-cache server, resulting in the feed pointing 
to the central server instead of the private one, which is not what I 
wanted to happen.

After reading the documentation, I added the following line to the recipe:

PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"

This did not have the desired effect. The package is still retrieved 
from the cache, and not rebuilt locally. Even if I clean and force a 
rebuild of the distro-feed-configs package, the package that ends up in 
the image is still the one from the central server.

What am I missing here?

Re: sstate-cache and making a package "host-dependent"

[Paul Eggleton]

Hi Mike,

On Wednesday 10 July 2013 15:05:23 Mike Looijmans wrote:
> I added a buildserver that also exports its "sstate-cache" directory, so
> that other build machines can grab their stuff from it. This works fine,
> but I have one problem. Some packages are meant to be dependent on the
> system that built it. I want to enforce that each build machine creates
> its own package, and does not grab it from the sstate-cache of the
> central server.
> 
> For example,
> 
> $ cat recipes-core/meta/distro-feed-configs.bbappend
> PRINC="1"
> DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
> DISTRO_FEED_NAME ?= "feed"
> DISTRO_FEED_PREFIX = "topic"
> DISTRO_FEED_URI =
> "http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"
> 
> 
> The purpose being that the host name of the machien that built the image
> ends up in the opkg config files. This works just fine.
> Now that we have a central server, all images on all build hosts grab
> the package from the sstate-cache server, resulting in the feed pointing
> to the central server instead of the private one, which is not what I
> wanted to happen.
> 
> After reading the documentation, I added the following line to the recipe:
> 
> PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"
> 
> This did not have the desired effect. The package is still retrieved
> from the cache, and not rebuilt locally. Even if I clean and force a
> rebuild of the distro-feed-configs package, the package that ends up in
> the image is still the one from the central server.
> 
> What am I missing here?

I think distro-feed-configs (from meta-oe, not OE-Core) is for package 
installation on the target, and has nothing to do with shared state.

I'm not sure if it's entirely appropriate, but you could take a similar 
approach to the one we use for preventing native sstate packages from mixing 
across different distributions (mostly to sidestep host glibc version 
dependency problems) - set SSTATE_EXTRAPATH to some value for the recipes you 
don't want to share and then the packages will go into a subdirectory named 
with that value; you could then delete these automatically from the server.

Cheers,
Paul

PS if you could please avoid replying to an existing thread for a new 
discussion, that would be great.

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: sstate-cache and making a package "host-dependent"

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2664 bytes --]

On Wed, Jul 10, 2013 at 06:39:20PM +0100, Paul Eggleton wrote:
> Hi Mike,
> 
> On Wednesday 10 July 2013 15:05:23 Mike Looijmans wrote:
> > I added a buildserver that also exports its "sstate-cache" directory, so
> > that other build machines can grab their stuff from it. This works fine,
> > but I have one problem. Some packages are meant to be dependent on the
> > system that built it. I want to enforce that each build machine creates
> > its own package, and does not grab it from the sstate-cache of the
> > central server.
> > 
> > For example,
> > 
> > $ cat recipes-core/meta/distro-feed-configs.bbappend
> > PRINC="1"
> > DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
> > DISTRO_FEED_NAME ?= "feed"
> > DISTRO_FEED_PREFIX = "topic"
> > DISTRO_FEED_URI =
> > "http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"
> > 
> > 
> > The purpose being that the host name of the machien that built the image
> > ends up in the opkg config files. This works just fine.
> > Now that we have a central server, all images on all build hosts grab
> > the package from the sstate-cache server, resulting in the feed pointing
> > to the central server instead of the private one, which is not what I
> > wanted to happen.
> > 
> > After reading the documentation, I added the following line to the recipe:
> > 
> > PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"
> > 
> > This did not have the desired effect. The package is still retrieved
> > from the cache, and not rebuilt locally. Even if I clean and force a
> > rebuild of the distro-feed-configs package, the package that ends up in
> > the image is still the one from the central server.
> > 
> > What am I missing here?
> 
> I think distro-feed-configs (from meta-oe, not OE-Core) is for package 
> installation on the target, and has nothing to do with shared state.
> 
> I'm not sure if it's entirely appropriate, but you could take a similar 
> approach to the one we use for preventing native sstate packages from mixing 
> across different distributions (mostly to sidestep host glibc version 
> dependency problems) - set SSTATE_EXTRAPATH to some value for the recipes you 
> don't want to share and then the packages will go into a subdirectory named 
> with that value; you could then delete these automatically from the server.

I would try to add
DISTRO_HOST_NAME[vardepvalue] = "${DISTRO_HOST_NAME}"

that way each builder should create own sstate archive.

use bitbake-diffsigs path-to-distro-feeds.sigdata-file to check that
value of os.uname is included in signature.

Regards,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: sstate-cache and making a package "host-dependent"

[Mike Looijmans]

On 07/10/2013 07:45 PM, Martin Jansa wrote:
> On Wed, Jul 10, 2013 at 06:39:20PM +0100, Paul Eggleton wrote:
>> Hi Mike,
>>
>> On Wednesday 10 July 2013 15:05:23 Mike Looijmans wrote:
>>> I added a buildserver that also exports its "sstate-cache" directory, so
>>> that other build machines can grab their stuff from it. This works fine,
>>> but I have one problem. Some packages are meant to be dependent on the
>>> system that built it. I want to enforce that each build machine creates
>>> its own package, and does not grab it from the sstate-cache of the
>>> central server.
>>>
>>> For example,
>>>
>>> $ cat recipes-core/meta/distro-feed-configs.bbappend
>>> PRINC="1"
>>> DISTRO_HOST_NAME ?= "${@os.uname()[1]}"
>>> DISTRO_FEED_NAME ?= "feed"
>>> DISTRO_FEED_PREFIX = "topic"
>>> DISTRO_FEED_URI =
>>> "http://${DISTRO_HOST_NAME}/${DISTRO_FEED_NAME}/${MACHINE}"
>>>
>>>
>>> The purpose being that the host name of the machien that built the image
>>> ends up in the opkg config files. This works just fine.
>>> Now that we have a central server, all images on all build hosts grab
>>> the package from the sstate-cache server, resulting in the feed pointing
>>> to the central server instead of the private one, which is not what I
>>> wanted to happen.
>>>
>>> After reading the documentation, I added the following line to the recipe:
>>>
>>> PACKAGE_ARCHS[vardeps] = "DISTRO_FEED_URI"
>>>
>>> This did not have the desired effect. The package is still retrieved
>>> from the cache, and not rebuilt locally. Even if I clean and force a
>>> rebuild of the distro-feed-configs package, the package that ends up in
>>> the image is still the one from the central server.
>>>
>>> What am I missing here?
>>
>> I think distro-feed-configs (from meta-oe, not OE-Core) is for package
>> installation on the target, and has nothing to do with shared state.
>>
>> I'm not sure if it's entirely appropriate, but you could take a similar
>> approach to the one we use for preventing native sstate packages from mixing
>> across different distributions (mostly to sidestep host glibc version
>> dependency problems) - set SSTATE_EXTRAPATH to some value for the recipes you
>> don't want to share and then the packages will go into a subdirectory named
>> with that value; you could then delete these automatically from the server.
>
> I would try to add
> DISTRO_HOST_NAME[vardepvalue] = "${DISTRO_HOST_NAME}"
>
> that way each builder should create own sstate archive.

I forgot to give my final feedback on this, it would be impolite to ask 
for advise and don't report back on whether it was useful, so here goes:

I've been using that setting for a while now and it does exactly what I 
wanted it to do. Every build machine creates its own feed configuration, 
while the big buildserver creates about everything else. The shared 
state cache is a big time saver for us.

Thanks!

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) – (0)499 - 33.69.79
Telefax: (+31) - (0)499 - 33.69.70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.

The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not  liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.

Re: [PATCHv2] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 956 bytes --]

On Wed, Jul 10, 2013 at 02:38:13PM +0200, Martin Jansa wrote:
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  scripts/test-dependencies.sh | 253 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 253 insertions(+)
>  create mode 100755 scripts/test-dependencies.sh
> 
> +    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
> +    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work $tmpdir/work-shared 2>/dev/null

I'll send v3 which doesn't remove $tmpdir/cache between the builds, it
takes too long already without reparsing all recipes every time.

Originally I was removing it as work around for
http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html
but that's not needed for master and people probably won't be changing
PNBLACKLIST while this script is running.
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCHv2] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

On Thu, Jul 11, 2013 at 5:38 AM, Martin Jansa <martin.jansa@gmail.com>wrote:

> On Wed, Jul 10, 2013 at 02:38:13PM +0200, Martin Jansa wrote:
> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> >  scripts/test-dependencies.sh | 253
> +++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 253 insertions(+)
> >  create mode 100755 scripts/test-dependencies.sh
> >
> > +    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
> > +    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata
> $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work
> $tmpdir/work-shared 2>/dev/null
>
> I'll send v3 which doesn't remove $tmpdir/cache between the builds, it
> takes too long already without reparsing all recipes every time.
>
> Originally I was removing it as work around for
> http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html
> but that's not needed for master and people probably won't be changing
> PNBLACKLIST while this script is running.
>

This is quite an interesting script. It looks quite similar to my
greedy-deps-tests[1] script, though you've taken it a step further in
adding builds #3 and #4.

1. https://github.com/kergoth/oe-test-scripts#oe-test-scripts
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2230 bytes --]

Re: [PATCHv2] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1801 bytes --]

On Thu, Jul 11, 2013 at 06:28:04AM -0700, Chris Larson wrote:
> On Thu, Jul 11, 2013 at 5:38 AM, Martin Jansa <martin.jansa@gmail.com>wrote:
> 
> > On Wed, Jul 10, 2013 at 02:38:13PM +0200, Martin Jansa wrote:
> > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > > ---
> > >  scripts/test-dependencies.sh | 253
> > +++++++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 253 insertions(+)
> > >  create mode 100755 scripts/test-dependencies.sh
> > >
> > > +    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
> > > +    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata
> > $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work
> > $tmpdir/work-shared 2>/dev/null
> >
> > I'll send v3 which doesn't remove $tmpdir/cache between the builds, it
> > takes too long already without reparsing all recipes every time.
> >
> > Originally I was removing it as work around for
> > http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html
> > but that's not needed for master and people probably won't be changing
> > PNBLACKLIST while this script is running.
> >
> 
> This is quite an interesting script. It looks quite similar to my
> greedy-deps-tests[1] script, though you've taken it a step further in
> adding builds #3 and #4.
> 
> 1. https://github.com/kergoth/oe-test-scripts#oe-test-scripts

pity I didn't know about your script before writing mine, it looks more
elegant in python

I've started with #3 the idea to detect greedy came later

Do you plan to integrate your script into oe-core? I'm fine with keeping
my shell script locally until you do. Your python script would better fit
in bitbake/oe environment and reloc-tests also looks useful.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCHv2] test-dependencies: add simple script to detect missing or autoenabled dependencies

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 2339 bytes --]

On Thu, Jul 11, 2013 at 6:49 AM, Martin Jansa <martin.jansa@gmail.com>wrote:

> On Thu, Jul 11, 2013 at 06:28:04AM -0700, Chris Larson wrote:
> > On Thu, Jul 11, 2013 at 5:38 AM, Martin Jansa <martin.jansa@gmail.com
> >wrote:
> >
> > > On Wed, Jul 10, 2013 at 02:38:13PM +0200, Martin Jansa wrote:
> > > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > > > ---
> > > >  scripts/test-dependencies.sh | 253
> > > +++++++++++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 253 insertions(+)
> > > >  create mode 100755 scripts/test-dependencies.sh
> > > >
> > > > +    echo "!!!Removing tmpdir \"$tmpdir\"!!!"
> > > > +    rm -rf $tmpdir/cache $tmpdir/deploy $tmpdir/pkgdata
> > > $tmpdir/sstate-control $tmpdir/stamps $tmpdir/sysroots $tmpdir/work
> > > $tmpdir/work-shared 2>/dev/null
> > >
> > > I'll send v3 which doesn't remove $tmpdir/cache between the builds, it
> > > takes too long already without reparsing all recipes every time.
> > >
> > > Originally I was removing it as work around for
> > >
> http://lists.openembedded.org/pipermail/bitbake-devel/2013-July/003624.html
> > > but that's not needed for master and people probably won't be changing
> > > PNBLACKLIST while this script is running.
> > >
> >
> > This is quite an interesting script. It looks quite similar to my
> > greedy-deps-tests[1] script, though you've taken it a step further in
> > adding builds #3 and #4.
> >
> > 1. https://github.com/kergoth/oe-test-scripts#oe-test-scripts
>
> pity I didn't know about your script before writing mine, it looks more
> elegant in python
>

Heh, it was shell at first, but I wanted to experiment. It ended up being a
good exercise in clean handling of SIGINT and SIGTERM.

I've started with #3 the idea to detect greedy came later
>
> Do you plan to integrate your script into oe-core? I'm fine with keeping
> my shell script locally until you do. Your python script would better fit
> in bitbake/oe environment and reloc-tests also looks useful.


I hadn't thought about it, but it's a good idea. Perhaps we can get the
best-of-both-worlds and integrate the features of both our scripts.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3442 bytes --]