[PATCH 0/8] Extensible SDK fixes

[PATCH 0/8] Extensible SDK fixes

[Chen Qi]

The following changes since commit 288c76a41aebaf54c7fca3782160830df462b9e4:

  builder: Fix multilib compile failure (2015-08-03 07:15:47 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib ChenQi/sdktool
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=ChenQi/sdktool

Chen Qi (8):
  populate_sdk_ext: install the latest buildtools-tarball
  copy_buildsystem: make sure bitbake directory is copied
  populate_sdk_ext: consider custom configuration in local.conf
  populate_sdk_ext: don't remove the native qemu dependencies
  Extensible SDK: allow for installation without preparing build system
  oe-publish-sdk: add script
  populate_sdk_ext: record SDK_TARGETS in devtool.conf
  devtool: add sdk.py plugin

 meta/classes/populate_sdk_ext.bbclass |  46 ++++++---
 meta/files/toolchain-shar-extract.sh  |   7 +-
 meta/lib/oe/copy_buildsystem.py       |   7 +-
 scripts/lib/devtool/sdk.py            | 182 ++++++++++++++++++++++++++++++++++
 scripts/oe-publish-sdk                | 124 +++++++++++++++++++++++
 5 files changed, 347 insertions(+), 19 deletions(-)
 create mode 100644 scripts/lib/devtool/sdk.py
 create mode 100755 scripts/oe-publish-sdk

-- 
1.9.1

[PATCH 1/8] populate_sdk_ext: install the latest buildtools-tarball

[Chen Qi]

If we do `bitbake buildtools-tarball' and then after one day do `bitbake
core-image-minimal -c populate_sdk_ext', we would meet errors like below.

| install: cannot stat '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/
poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone
-1.8+snapshot-20150429.sh': No such file or directory

The problem is that the output name for buildtools-tarball has ${DATE} in it.
So if populate_sdk_ext task is executed but buildtools-tarball is not rebuilt,
the above error appears.

Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the
install_tools() function, we should find the latest buildtools-tarball based
on the modification time and install it.

[YOCTO #7674]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index a36bf16..b6725e0 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -169,7 +169,9 @@ install_tools() {
 	lnr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/recipetool ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/recipetool
 	touch ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
 
-	install ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-${DISTRO_VERSION}.sh ${SDK_OUTPUT}/${SDKPATH}
+	# find latest buildtools-tarball and install it
+	buildtools_path=`ls -t1 ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-*.sh | head -n1`
+	install $buildtools_path ${SDK_OUTPUT}/${SDKPATH}
 
 	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2 ${SDK_OUTPUT}/${SDKPATH}
 }
-- 
1.9.1

Re: [PATCH 1/8] populate_sdk_ext: install the latest buildtools-tarball

[Paul Eggleton]

On Monday 10 August 2015 11:17:59 Chen Qi wrote:
> If we do `bitbake buildtools-tarball' and then after one day do `bitbake
> core-image-minimal -c populate_sdk_ext', we would meet errors like below.
> 
> | install: cannot stat
> | '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/
> 
> poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalon
> e -1.8+snapshot-20150429.sh': No such file or directory
> 
> The problem is that the output name for buildtools-tarball has ${DATE} in
> it. So if populate_sdk_ext task is executed but buildtools-tarball is not
> rebuilt, the above error appears.
> 
> Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the
> install_tools() function, we should find the latest buildtools-tarball based
> on the modification time and install it.
> 
> [YOCTO #7674]
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/classes/populate_sdk_ext.bbclass | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/populate_sdk_ext.bbclass
> b/meta/classes/populate_sdk_ext.bbclass index a36bf16..b6725e0 100644
> --- a/meta/classes/populate_sdk_ext.bbclass
> +++ b/meta/classes/populate_sdk_ext.bbclass
> @@ -169,7 +169,9 @@ install_tools() {
>  	lnr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/recipetool
> ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/recipetool touch
> ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
> 
> -	install
> ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKG
> ARCH}-buildtools-nativesdk-standalone-${DISTRO_VERSION}.sh
> ${SDK_OUTPUT}/${SDKPATH} +	# find latest buildtools-tarball and install it
> +	buildtools_path=`ls -t1
> ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKG
> ARCH}-buildtools-nativesdk-standalone-*.sh | head -n1` +	install
> $buildtools_path ${SDK_OUTPUT}/${SDKPATH}
> 
>  	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2
> ${SDK_OUTPUT}/${SDKPATH} }

Hmm, it turns out there's another problem here - this assumes SDK_NAME is set 
as it is in poky. Perhaps we correct that in a follow-up patch though, but 
it'll be a little bit tricky as it'll need to be done by getting the actual 
value of SDK_NAME as it would have been when IMAGE_BASENAME is "buildtools-
tarball".

The more I think about this though the more I wonder why we're bothering with 
buildtools instead of just bundling its contents inside the native part of the 
extensible SDK itself. Randy can you remind me why we did it that way?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 1/8] populate_sdk_ext: install the latest buildtools-tarball

[ChenQi]

On 08/18/2015 09:51 PM, Paul Eggleton wrote:
> On Monday 10 August 2015 11:17:59 Chen Qi wrote:
>> If we do `bitbake buildtools-tarball' and then after one day do `bitbake
>> core-image-minimal -c populate_sdk_ext', we would meet errors like below.
>>
>> | install: cannot stat
>> | '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/
>>
>> poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalon
>> e -1.8+snapshot-20150429.sh': No such file or directory
>>
>> The problem is that the output name for buildtools-tarball has ${DATE} in
>> it. So if populate_sdk_ext task is executed but buildtools-tarball is not
>> rebuilt, the above error appears.
>>
>> Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the
>> install_tools() function, we should find the latest buildtools-tarball based
>> on the modification time and install it.
>>
>> [YOCTO #7674]
>>
>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>> ---
>>   meta/classes/populate_sdk_ext.bbclass | 4 +++-
>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/populate_sdk_ext.bbclass
>> b/meta/classes/populate_sdk_ext.bbclass index a36bf16..b6725e0 100644
>> --- a/meta/classes/populate_sdk_ext.bbclass
>> +++ b/meta/classes/populate_sdk_ext.bbclass
>> @@ -169,7 +169,9 @@ install_tools() {
>>   	lnr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/recipetool
>> ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/recipetool touch
>> ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
>>
>> -	install
>> ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKG
>> ARCH}-buildtools-nativesdk-standalone-${DISTRO_VERSION}.sh
>> ${SDK_OUTPUT}/${SDKPATH} +	# find latest buildtools-tarball and install it
>> +	buildtools_path=`ls -t1
>> ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKG
>> ARCH}-buildtools-nativesdk-standalone-*.sh | head -n1` +	install
>> $buildtools_path ${SDK_OUTPUT}/${SDKPATH}
>>
>>   	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2
>> ${SDK_OUTPUT}/${SDKPATH} }
> Hmm, it turns out there's another problem here - this assumes SDK_NAME is set
> as it is in poky. Perhaps we correct that in a follow-up patch though, but
> it'll be a little bit tricky as it'll need to be done by getting the actual
> value of SDK_NAME as it would have been when IMAGE_BASENAME is "buildtools-
> tarball".
>
> The more I think about this though the more I wonder why we're bothering with
> buildtools instead of just bundling its contents inside the native part of the
> extensible SDK itself. Randy can you remind me why we did it that way?
>
> Cheers,
> Paul
>
Hi Paul,

I've dropped this patch from the patchset and sent out V2.
I think we can fix this buildtool problem in other patches.

Best Regards,
Chen Qi

[PATCH 2/8] copy_buildsystem: make sure bitbake directory is copied

[Chen Qi]

The previous code assumes that bitbake/ directory is under the core layer.
This is the case for Yocto project. But users might clone oe-core and bitbake
separately. So we use bb.__file__ to locate the bitbake directory to make sure
it's copied into the extensible SDK.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oe/copy_buildsystem.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index cf7fada..979578c 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -28,11 +28,10 @@ class BuildSystem(object):
         layers.append(corebase)
 
         corebase_files = self.d.getVar('COREBASE_FILES', True).split()
-
-        # bitbake belongs in corebase so make sure it goes there
-        if "bitbake" not in corebase_files:
-            corebase_files.append("bitbake")
         corebase_files = [corebase + '/' +x for x in corebase_files]
+        # Make sure bitbake goes in
+        bitbake_dir = bb.__file__.rsplit('/', 3)[0]
+        corebase_files.append(bitbake_dir)
 
         for layer in layers:
             layerconf = os.path.join(layer, 'conf', 'layer.conf')
-- 
1.9.1

[PATCH 3/8] populate_sdk_ext: consider custom configuration in local.conf

[Chen Qi]

Copy the contents of local.conf under TOPDIR into the final generated
local.conf. In this way, custom settings are also made into the final
local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure
when preparing the build system if the user has custom configuration for
DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf
also don't get built correctly.

This patch solves the above problem by making use of the bb.utils.edit_metadata.

In addition, we check to avoid any setting that might lead to host path
bleeding into SDK's configuration. Basically, variables with values starting
with '/' are removed. A whitelist mechanism is introduced so that users could
specify variables that should not be ignored. The name of the whitelist is
SDK_LOCAL_CONF_WHITELIST.

The SDK_META_CONF_WHITELIST is removed as it's of no use after this change.

Add SDK_INHERIT_BLACKLIST to forbit local.conf in SDK to herit certain classes
like 'buildhistory' or 'icecc'.

[YOCTO #7616]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index b6725e0..b111bdc 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -15,7 +15,8 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = " ${SDK_TARGETS}"
 
 SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
 
-SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
+SDK_LOCAL_CONF_WHITELIST ?= ""
+SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 
 SDK_TARGETS ?= "${PN}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
@@ -109,15 +110,33 @@ python copy_buildsystem () {
         f.write('    "\n')
 
     # Create local.conf
+    local_conf_whitelist = d.getVar('SDK_LOCAL_CONF_WHITELIST', True).split()
+    def handle_var(varname, origvalue, op, newlines):
+        if origvalue.strip().startswith('/') and not varname in local_conf_whitelist:
+            newlines.append('# Removed original setting of %s\n' % varname)
+            return None, op, 0, True
+        else:
+            return origvalue, op, 0, True
+    varlist = ['[^#=+ ]*']
+    builddir = d.getVar('TOPDIR', True)
+    with open(builddir + '/conf/local.conf', 'r') as f:
+        oldlines = f.readlines()
+    (updated, newlines) = bb.utils.edit_metadata(oldlines, varlist, handle_var)
+
     with open(baseoutpath + '/conf/local.conf', 'w') as f:
         f.write('# WARNING: this configuration has been automatically generated and in\n')
         f.write('# most cases should not be edited. If you need more flexibility than\n')
         f.write('# this configuration provides, it is strongly suggested that you set\n')
         f.write('# up a proper instance of the full build system and use that instead.\n\n')
+        for line in newlines:
+            f.write(line)
 
         f.write('INHERIT += "%s"\n\n' % 'uninative')
         f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION', False))
 
+        # Some classes are not suitable for SDK, remove them from INHERIT
+        f.write('INHERIT_remove = "%s"\n' % d.getVar('SDK_INHERIT_BLACKLIST'))
+    
         # This is a bit of a hack, but we really don't want these dependencies
         # (we're including them in the SDK as nativesdk- versions instead)
         f.write('POKYQEMUDEPS_forcevariable = ""\n\n')
@@ -134,8 +153,6 @@ python copy_buildsystem () {
         # Ensure locked sstate cache objects are re-used without error
         f.write('SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "warn"\n\n')
 
-        for varname in d.getVar('SDK_META_CONF_WHITELIST', True).split():
-            f.write('%s = "%s"\n' % (varname, d.getVar(varname, True)))
         f.write('require conf/locked-sigs.inc\n')
         f.write('require conf/work-config.inc\n')
 
-- 
1.9.1

[PATCH 4/8] populate_sdk_ext: don't remove the native qemu dependencies

[Chen Qi]

These dependencies were deliberately removed because it was assumed that
they were provided by nativesdk packages. On the one hand, nativesdk packages
in extensible SDK don't have these packages; on the other hand, even if we
add these nativesdk packages, they are still not useful because we we need
runqemu to run correctly.

So we don't remove these native qemu dependencies.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index b111bdc..781e70b 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -137,11 +137,6 @@ python copy_buildsystem () {
         # Some classes are not suitable for SDK, remove them from INHERIT
         f.write('INHERIT_remove = "%s"\n' % d.getVar('SDK_INHERIT_BLACKLIST'))
     
-        # This is a bit of a hack, but we really don't want these dependencies
-        # (we're including them in the SDK as nativesdk- versions instead)
-        f.write('POKYQEMUDEPS_forcevariable = ""\n\n')
-        f.write('EXTRA_IMAGEDEPENDS_remove = "qemu-native qemu-helper-native"\n\n')
-
         # Bypass the default connectivity check if any
         f.write('CONNECTIVITY_CHECK_URIS = ""\n\n')
 
-- 
1.9.1

[PATCH 5/8] Extensible SDK: allow for installation without preparing build system

[Chen Qi]

When publishing SDK, what we want is basically its metadata and sstate
cache objects. We don't want the SDK to be prepared with running bitbake
as it takes time which reproduces meaningless output for the published SDK.

So this patch adds an option to allow for SDK to be extracted without
preparing the build system.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 12 +++++++-----
 meta/files/toolchain-shar-extract.sh  |  7 ++++++-
 2 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 781e70b..3e36890 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -219,11 +219,13 @@ sdk_ext_postinst() {
 	# For now this is where uninative.bbclass expects the tarball
 	mv *-nativesdk-libc.tar.* $target_sdk_dir/`dirname ${oe_init_build_env_path}`
 
-	printf "Preparing build system...\n"
-	# dash which is /bin/sh on Ubuntu will not preserve the
-	# current working directory when first ran, nor will it set $1 when
-	# sourcing a script. That is why this has to look so ugly.
-	sh -c ". buildtools/environment-setup* > preparing_build_system.log && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> preparing_build_system.log && bitbake ${SDK_TARGETS} >> preparing_build_system.log" || { echo "SDK preparation failed: see `pwd`/preparing_build_system.log" ; exit 1 ; }
+	if [ "$prepare_buildsystem" != "no" ]; then
+	    printf "Preparing build system...\n"
+	    # dash which is /bin/sh on Ubuntu will not preserve the
+	    # current working directory when first ran, nor will it set $1 when
+	    # sourcing a script. That is why this has to look so ugly.
+	    sh -c ". buildtools/environment-setup* > preparing_build_system.log && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> preparing_build_system.log && bitbake ${SDK_TARGETS} >> preparing_build_system.log" || { echo "SDK preparation failed: see `pwd`/preparing_build_system.log" ; exit 1 ; }
+	fi
 	echo done
 }
 
diff --git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index 0a33ee8..b7f6350 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -32,7 +32,7 @@ answer=""
 relocate=1
 savescripts=0
 verbose=0
-while getopts ":yd:DRS" OPT; do
+while getopts ":yd:nDRS" OPT; do
 	case $OPT in
 	y)
 		answer="Y"
@@ -41,6 +41,9 @@ while getopts ":yd:DRS" OPT; do
 	d)
 		target_sdk_dir=$OPTARG
 		;;
+	n)
+		prepare_buildsystem="no"
+		;;
 	D)
 		verbose=1
 		;;
@@ -55,6 +58,8 @@ while getopts ":yd:DRS" OPT; do
 		echo "Usage: $(basename $0) [-y] [-d <dir>]"
 		echo "  -y         Automatic yes to all prompts"
 		echo "  -d <dir>   Install the SDK to <dir>"
+		echo "======== Extensible SDK only options ============"
+		echo "  -n         Do not prepare the build system"
 		echo "======== Advanced DEBUGGING ONLY OPTIONS ========"
 		echo "  -S         Save relocation scripts"
 		echo "  -R         Do not relocate executables"
-- 
1.9.1

Re: [PATCH 5/8] Extensible SDK: allow for installation without preparing build system

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 910 bytes --]

On Sun, Aug 9, 2015 at 8:18 PM, Chen Qi <Qi.Chen@windriver.com> wrote:

> When publishing SDK, what we want is basically its metadata and sstate
> cache objects. We don't want the SDK to be prepared with running bitbake
> as it takes time which reproduces meaningless output for the published SDK.
>
> So this patch adds an option to allow for SDK to be extracted without
> preparing the build system.
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>

I expect the intent behind the preparation was for it to be a build from
100% sstate, but that’s not actually the case on all systems. If that’s the
case, maybe it should be run with locked sigs and ignore failure, to avoid
doing a real from-scratch build at install time.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1387 bytes --]

Re: [PATCH 5/8] Extensible SDK: allow for installation without preparing build system

[Paul Eggleton]

Hi Chris / Qi,

On Monday 10 August 2015 08:39:01 Christopher Larson wrote:
> On Sun, Aug 9, 2015 at 8:18 PM, Chen Qi <Qi.Chen@windriver.com> wrote:
> > When publishing SDK, what we want is basically its metadata and sstate
> > cache objects. We don't want the SDK to be prepared with running bitbake
> > as it takes time which reproduces meaningless output for the published
> > SDK.
> > 
> > So this patch adds an option to allow for SDK to be extracted without
> > preparing the build system.
> > 
> > Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> 
> I expect the intent behind the preparation was for it to be a build from
> 100% sstate, but that’s not actually the case on all systems. If that’s the
> case, maybe it should be run with locked sigs and ignore failure, to avoid
> doing a real from-scratch build at install time.

I agree this is definitely a problem. In fact, I'm fairly sure the locked sigs 
functionality isn't working as it was originally intended to - last I checked 
it looked like it was warning and then going ahead and running the task; by 
default it was supposed to just extract the sstate package. I think we'll need 
to deal with that separately as all this patch does is make the extraction 
optional (in order to support the publishing script).

As an aside it would be nice if we could hide this -n option in the help when 
not building the extensible SDK, but on the other hand refactoring things so 
that would be possible would likely be more effort than it's worth.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH 6/8] oe-publish-sdk: add script

[Chen Qi]

Add a script to publish extensible SDK to a specified destination.

oe-publish-sdk <ext-sdk> <destination>

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 scripts/oe-publish-sdk | 124 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)
 create mode 100755 scripts/oe-publish-sdk

diff --git a/scripts/oe-publish-sdk b/scripts/oe-publish-sdk
new file mode 100755
index 0000000..90bedac
--- /dev/null
+++ b/scripts/oe-publish-sdk
@@ -0,0 +1,124 @@
+#!/usr/bin/env python
+
+# OpenEmbedded SDK Tool
+
+# oe-publish-sdk publish <ext-sdk> <destination>
+# <ext-sdk>: extensible SDK to publish
+# <destination>: local or remote location which servers as an SDK update server
+# e.g.
+# oe-publish-sdk publish sdk-ext.sh /mnt/poky/sdk-ext
+# oe-publish-sdk publish sdk-ext.sh user@host:/opt/poky/sdk-ext
+#
+
+import sys
+import os
+import argparse
+import glob
+import re
+import subprocess
+import logging
+import shutil
+import errno
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + '/lib'
+sys.path = sys.path + [lib_path]
+import scriptutils
+logger = scriptutils.logger_create('sdktool')
+
+def mkdir(d):
+    try:
+        os.makedirs(d)
+    except OSError as e:
+        if e.errno != errno.EEXIST:
+            raise e
+
+def publish(args):
+    logger.debug("In publish function")
+    target_sdk = args.sdk
+    destination = args.dest
+    logger.debug("target_sdk = %s, update_server = %s" % (target_sdk, destination))
+    sdk_basename = os.path.basename(target_sdk)
+
+    # Ensure the SDK exists
+    if not os.path.exists(target_sdk):
+        logger.error("%s doesn't exist" % target_sdk)
+        return -1
+    
+    is_remote = False
+    if ':' in destination:
+        is_remote = True
+
+    if not is_remote:
+        dest_sdk = os.path.join(destination, sdk_basename)
+        logger.debug("dest_sdk = %s" % dest_sdk)
+        mkdir(destination)
+        if os.path.exists(dest_sdk):
+            os.remove(dest_sdk)
+        if (os.stat(target_sdk).st_dev == os.stat(destination).st_dev):
+            os.link(target_sdk, dest_sdk)
+        else:
+            shutil.copy(target_sdk, dest_sdk)
+        # Unpack the SDK to get the sstate objects and manifests
+        # so that client side could do updates
+        logger.debug("Unpacking %s to %s ..." % (dest_sdk, destination))
+        ret = subprocess.call("sh %s -n -y -d %s" % (dest_sdk, destination), shell=True)
+        if ret == 0:
+            logger.info('Successfully unpacked %s to %s' % (dest_sdk, destination))
+        else:
+            logger.error('Failed to unpack %s to %s' % (dest_sdk, destination))
+
+        return ret
+    else:
+        host, destdir = destination.split(':')
+        logger.debug('Making sure %s on %s exists' % (destdir, host))
+        ret = subprocess.call("ssh %s 'rm -rf %s && mkdir -p %s'" % (host, destdir, destdir), shell=True)
+        if ret != 0:
+            logger.error("Making directory %s on %s failed" % (destdir, host))
+            return ret
+        logger.debug('Copying %s to %s' % (target_sdk, destination))
+        ret = subprocess.call("scp %s %s" % (target_sdk, destination), shell=True)
+        if ret != 0:
+            logger.error("scp failed")
+            return ret
+        dest_sdk = os.path.join(destdir, sdk_basename)
+        cmd = 'sh %s -n -y -d %s && cd %s/layers && git init . && mv .git/hooks/post-update.sample .git/hooks/post-update && git add . && git commit -m "init repo"' % (dest_sdk, destdir, destdir)
+        logger.debug("cmd is %s" % cmd)
+        logger.debug("Unpacking %s in %s" % (dest_sdk, destination))
+        ret = subprocess.call("ssh %s '%s'" % (host, cmd), shell=True)
+        if ret != 0:
+            logger.error("Failed to unpack SDK on %s" % destination)
+            return ret
+        else:
+            logger.info("Successfully published SDK to %s" % destination)
+            return 0
+        
+def main():
+    parser = argparse.ArgumentParser(description="OpenEmbedded development tool",
+                                     epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
+    parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
+    parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
+
+    parser.add_argument('sdk', help='Extensible SDK to publish')
+    parser.add_argument('dest', help='Destination to publish SDK to')
+    
+    parser.set_defaults(func=publish)
+
+    args = parser.parse_args()
+
+    if args.debug:
+        logger.setLevel(logging.DEBUG)
+    elif args.quiet:
+        logger.setLevel(logging.ERROR)
+
+    ret = args.func(args)
+    return ret
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc(5)
+    sys.exit(ret)
-- 
1.9.1

Re: [PATCH 6/8] oe-publish-sdk: add script

[Paul Eggleton]

Hi Qi,

A couple of issues noted below.

On Monday 10 August 2015 11:18:04 Chen Qi wrote:
> Add a script to publish extensible SDK to a specified destination.
> 
> oe-publish-sdk <ext-sdk> <destination>
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  scripts/oe-publish-sdk | 124
> +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124
> insertions(+)
>  create mode 100755 scripts/oe-publish-sdk
> 
> diff --git a/scripts/oe-publish-sdk b/scripts/oe-publish-sdk
> new file mode 100755
> index 0000000..90bedac
> --- /dev/null
> +++ b/scripts/oe-publish-sdk
> @@ -0,0 +1,124 @@
> +#!/usr/bin/env python
> +
> +# OpenEmbedded SDK Tool
> +
> +# oe-publish-sdk publish <ext-sdk> <destination>
> +# <ext-sdk>: extensible SDK to publish
> +# <destination>: local or remote location which servers as an SDK update
> server 
> +# e.g.
> +# oe-publish-sdk publish sdk-ext.sh /mnt/poky/sdk-ext
> +# oe-publish-sdk publish sdk-ext.sh user@host:/opt/poky/sdk-ext
> +#
> +
> +import sys
> +import os
> +import argparse
> +import glob
> +import re
> +import subprocess
> +import logging
> +import shutil
> +import errno
> +
> +scripts_path = os.path.dirname(os.path.realpath(__file__))
> +lib_path = scripts_path + '/lib'
> +sys.path = sys.path + [lib_path]
> +import scriptutils
> +logger = scriptutils.logger_create('sdktool')
> +
> +def mkdir(d):
> +    try:
> +        os.makedirs(d)
> +    except OSError as e:
> +        if e.errno != errno.EEXIST:
> +            raise e
>
> +def publish(args):
> +    logger.debug("In publish function")
> +    target_sdk = args.sdk
> +    destination = args.dest
> +    logger.debug("target_sdk = %s, update_server = %s" % (target_sdk,
> destination)) 
> +    sdk_basename = os.path.basename(target_sdk)
> +
> +    # Ensure the SDK exists
> +    if not os.path.exists(target_sdk):
> +        logger.error("%s doesn't exist" % target_sdk)
> +        return -1
> +
> +    is_remote = False
> +    if ':' in destination:
> +        is_remote = True
> +
> +    if not is_remote:
> +        dest_sdk = os.path.join(destination, sdk_basename)
> +        logger.debug("dest_sdk = %s" % dest_sdk)
> +        mkdir(destination)
> +        if os.path.exists(dest_sdk):
> +            os.remove(dest_sdk)
> +        if (os.stat(target_sdk).st_dev == os.stat(destination).st_dev):
> +            os.link(target_sdk, dest_sdk)
> +        else:
> +            shutil.copy(target_sdk, dest_sdk)
> +        # Unpack the SDK to get the sstate objects and manifests
> +        # so that client side could do updates
> +        logger.debug("Unpacking %s to %s ..." % (dest_sdk, destination))
> +        ret = subprocess.call("sh %s -n -y -d %s" % (dest_sdk,
> destination), shell=True) +        if ret == 0:
> +            logger.info('Successfully unpacked %s to %s' % (dest_sdk,
> destination)) +        else:
> +            logger.error('Failed to unpack %s to %s' % (dest_sdk,
> destination)) +
> +        return ret
> +    else:
> +        host, destdir = destination.split(':')
> +        logger.debug('Making sure %s on %s exists' % (destdir, host))
> +        ret = subprocess.call("ssh %s 'rm -rf %s && mkdir -p %s'" % (host,
> destdir, destdir), shell=True) 

This makes me feel very uneasy. I really think we should not be doing rm -rf 
on whatever path the user specifies in case it's wrong.

> +        if ret != 0:
> +            logger.error("Making directory %s on %s failed" % (destdir,
> host)) 
> +            return ret
> +        logger.debug('Copying %s to %s' % (target_sdk, destination))
> +        ret = subprocess.call("scp %s %s" % (target_sdk, destination),
> shell=True) 
> +        if ret != 0:
> +            logger.error("scp failed")
> +            return ret
> +        dest_sdk = os.path.join(destdir, sdk_basename)
> +        cmd = 'sh %s -n -y -d %s && cd %s/layers && git init . && mv
> .git/hooks/post-update.sample .git/hooks/post-update && git add . && git
> commit -m "init repo"' % (dest_sdk, destdir, destdir) +       
> logger.debug("cmd is %s" % cmd)
> +        logger.debug("Unpacking %s in %s" % (dest_sdk, destination))
> +        ret = subprocess.call("ssh %s '%s'" % (host, cmd), shell=True)
> +        if ret != 0:
> +            logger.error("Failed to unpack SDK on %s" % destination)
> +            return ret
> +        else:
> +            logger.info("Successfully published SDK to %s" % destination)
> +            return 0

It looks like you are only doing the setup of the embedded git repository when 
the publishing destination is remote - that's not right. The destination might 
still be local and shared for remote access. Could this be refactored so 
rather than having a big if statement at the start and having completely 
separate implementations for local and remote, you check it when running each 
step and choose to wrap the command in ssh, or run scp instead of cp etc.?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH 7/8] populate_sdk_ext: record SDK_TARGETS in devtool.conf

[Chen Qi]

Record the value of SDK_TARGETS for later use of devtool's sdk plugin
for updating SDK.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 3e36890..5e60dbf 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -84,6 +84,8 @@ python copy_buildsystem () {
     config.set('General', 'bitbake_subdir', conf_bbpath)
     config.set('General', 'init_path', conf_initpath)
     config.set('General', 'core_meta_subdir', core_meta_subdir)
+    config.add_section('SDK')
+    config.set('SDK', 'sdk_targets', d.getVar('SDK_TARGETS', True))
     bb.utils.mkdirhier(os.path.join(baseoutpath, 'conf'))
     with open(os.path.join(baseoutpath, 'conf', 'devtool.conf'), 'w') as f:
         config.write(f)
-- 
1.9.1

[PATCH 8/8] devtool: add sdk.py plugin

[Chen Qi]

E.g.
devtool sdk-update /mnt/sdk-repo/
devtool sdk-update http://mysdkhost/sdk

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 scripts/lib/devtool/sdk.py | 182 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100644 scripts/lib/devtool/sdk.py

diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
new file mode 100644
index 0000000..231755c
--- /dev/null
+++ b/scripts/lib/devtool/sdk.py
@@ -0,0 +1,182 @@
+# Development tool - sdk-update command plugin
+
+import os
+import subprocess
+import logging
+import glob
+import shutil
+import errno
+import sys
+from devtool import exec_build_env_command, setup_tinfoil
+
+logger = logging.getLogger('devtool')
+
+def plugin_init(pluginlist):
+    """Plugin initialization"""
+    pass
+
+def parse_locked_sigs(sigfile_path):
+    """Return <pn:task>:<hash> dictionary"""
+    sig_dict = {}
+    with open(sigfile_path) as f:
+        lines = f.readlines()
+        for line in lines:
+            if ':' in line:
+                taskkey, _, hashval = line.rpartition(':')
+                sig_dict[taskkey.strip()] = hashval.split()[0]
+    return sig_dict
+
+def generate_update_dict(sigfile_new, sigfile_old):
+    """Return a dict containing <pn:task>:<hash> which indicates what need to be updated"""
+    update_dict = {}
+    sigdict_new = parse_locked_sigs(sigfile_new)
+    sigdict_old = parse_locked_sigs(sigfile_old)
+    for k in sigdict_new:
+        if k not in sigdict_old:
+            update_dict[k] = sigdict_new[k]
+            continue
+        if sigdict_new[k] != sigdict_old[k]:
+            update_dict[k] = sigdict_new[k]
+            continue
+    return update_dict
+
+def get_sstate_objects(update_dict, newsdk_path):
+    """Return a list containing sstate objects which are to be installed"""
+    sstate_objects = []
+    # Ensure newsdk_path points to an extensible SDK
+    sstate_dir = os.path.join(newsdk_path, 'sstate-cache')
+    if not os.path.exists(sstate_dir):
+        logger.error("sstate-cache directory not found under %s" % newsdk_path)
+        raise
+    for k in update_dict:
+        files = set()
+        hashval = update_dict[k]
+        p = sstate_dir + '/' + hashval[:2] + '/*' + hashval + '*.tgz'
+        files |= set(glob.glob(p))
+        p = sstate_dir + '/*/' + hashval[:2] + '/*' + hashval + '*.tgz'
+        files |= set(glob.glob(p))
+        files = list(files)
+        if len(files) == 1:
+            sstate_objects.extend(files)
+        elif len(files) > 1:
+            logger.error("More than one matching sstate object found for %s" % hashval)
+        
+    return sstate_objects
+
+def mkdir(d):
+    try:
+        os.makedirs(d)
+    except OSError as e:
+        if e.errno != errno.EEXIST:
+            raise e
+
+def install_sstate_objects(sstate_objects, src_sdk, dest_sdk):
+    """Install sstate objects into destination SDK"""
+    sstate_dir = os.path.join(dest_sdk, 'sstate-cache')
+    if not os.path.exists(sstate_dir):
+        logger.error("Missing sstate-cache directory in %s, it might not be an extensible SDK." % dest_sdk)
+        raise
+    for sb in sstate_objects:
+        dst = sb.replace(src_sdk, dest_sdk)
+        destdir = os.path.dirname(dst)
+        mkdir(destdir)
+        logger.debug("Copying %s to %s" % (sb, dst))
+        shutil.copy(sb, dst)
+
+def sdk_update(args, config, basepath, workspace):
+    logger.debug("In sdk_update")
+    # Fetch locked-sigs.inc file from remote/local destination
+    # For now, we only support local destination or a destination hosted on a webserver
+    logger.debug("updateserver: %s" % args.updateserver)
+
+    # Make sure we are using sdk-update from within SDK
+    logger.debug("basepath = %s" % basepath)
+    old_locked_sig_file_path = os.path.join(basepath, 'conf/locked-sigs.inc')
+    if not os.path.exists(old_locked_sig_file_path):
+        logger.error("Not using devtool's sdk-update command from within an extensible SDK. Please specify correct basepath via --basepath option")
+        return -1
+    else:
+        logger.debug("Found conf/locked-sigs.inc in %s" % basepath)
+        
+    is_remote = False
+    if ':' in args.updateserver:
+        is_remote = True
+        
+    if not is_remote:
+        # devtool sdk-update /local/path/to/latest/sdk
+        new_locked_sig_file_path = os.path.join(args.updateserver, 'conf/locked-sigs.inc')
+        if not os.path.exists(new_locked_sig_file_path):
+            logger.error("%s doesn't exist or is not an extensible SDK" % args.updateserver)
+            return -1
+        else:
+            logger.debug("Found conf/locked-sigs.inc in %s" % args.updateserver)
+        update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
+        logger.debug("update_dict = %s" % update_dict)
+        sstate_objects = get_sstate_objects(update_dict, args.updateserver)
+        logger.debug("sstate_objects = %s" % sstate_objects)
+        if len(sstate_objects) == 0:
+            logger.info("No need to update.")
+            return 0
+        logger.info("Installing sstate objects into %s", basepath)
+        install_sstate_objects(sstate_objects, args.updateserver.rstrip('/'), basepath)
+        logger.info("Updating configuration files")
+        new_conf_dir = os.path.join(args.updateserver, 'conf')
+        old_conf_dir = os.path.join(basepath, 'conf')
+        shutil.rmtree(old_conf_dir)
+        shutil.copytree(new_conf_dir, old_conf_dir)
+        logger.info("Updating layers")
+        new_layers_dir = os.path.join(args.updateserver, 'layers')
+        old_layers_dir = os.path.join(basepath, 'layers')
+        shutil.rmtree(old_layers_dir)
+        shutil.copytree(new_layers_dir, old_layers_dir)
+    else:
+        # devtool sdk-update http://myhost/sdk
+        tmpsdk_dir = '/tmp/sdk-ext'
+        if os.path.exists(tmpsdk_dir):
+            shutil.rmtree(tmpsdk_dir)
+        os.makedirs(tmpsdk_dir)
+        os.makedirs(os.path.join(tmpsdk_dir, 'conf'))
+        # Fetch locked-sigs.inc from update server
+        ret = subprocess.call("wget -q -O - %s/conf/locked-sigs.inc > %s/locked-sigs.inc" % (args.updateserver, os.path.join(tmpsdk_dir, 'conf')), shell=True)
+        if ret != 0:
+            logger.error("Fetching conf/locked-sigs.inc from %s to %s/locked-sigs.inc failed" % (args.updateserver, os.path.join(tmpsdk_dir, 'conf')))
+            return ret
+        else:
+            logger.info("Fetching conf/locked-sigs.inc from %s to %s/locked-sigs.inc succeeded" % (args.updateserver, os.path.join(tmpsdk_dir, 'conf')))
+        new_locked_sig_file_path = os.path.join(tmpsdk_dir, 'conf/locked-sigs.inc')
+        update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
+        logger.debug("update_dict = %s" % update_dict)
+        if len(update_dict) == 0:
+            logger.info("No need to update.")
+            return 0
+        # Update metadata
+        logger.debug("Updating meta data via git ...")
+        ret = subprocess.call("rm -rf layers && git clone %s/layers" % args.updateserver, shell=True)
+        if ret != 0:
+            logger.error("Updating meta data via git clone failed")
+            return ret
+        logger.debug("Updating conf files ...")
+        conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'work-config.inc', 'locked-sigs.inc']
+        for conf in conf_files:
+            ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (args.updateserver, conf, conf), shell=True)
+            if ret != 0:
+                logger.error("Update %s failed" % conf)
+                return ret
+        with open(os.path.join(basepath, 'conf/local.conf'), 'a') as f:
+            f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % args.updateserver)
+
+    # Run bitbake command for the whole SDK
+    sdk_targets = config.get('SDK', 'sdk_targets')
+    logger.info("Executing 'bitbake %s' ... (This may take some time.)" % sdk_targets)
+    try:
+        exec_build_env_command(config.init_path, basepath, 'bitbake %s' % sdk_targets)
+    except:
+        logger.error('bitbake %s failed' % sdk_targets)
+        return -1
+    return 0
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from the sdk plugin"""
+    parser_sdk = subparsers.add_parser('sdk-update', help='Update SDK components from a nominated destination')
+    parser_sdk.add_argument('updateserver', help='The update server to fetch latest SDK components')
+    parser_sdk.set_defaults(func=sdk_update)
-- 
1.9.1

Re: [PATCH 8/8] devtool: add sdk.py plugin

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 632 bytes --]

On Sun, Aug 9, 2015 at 8:18 PM, Chen Qi <Qi.Chen@windriver.com> wrote:

> E.g.
> devtool sdk-update /mnt/sdk-repo/
> devtool sdk-update http://mysdkhost/sdk
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>

Shouldn’t there be a description of exactly what the purpose of this is, in
the commit message, for those of us not intimately familiar with the sdk
construction process? It’s best to include the why in commit messages, not
just the what/how.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1145 bytes --]

Re: [PATCH 8/8] devtool: add sdk.py plugin

[Paul Eggleton]

On Monday 10 August 2015 11:18:06 Chen Qi wrote:
> E.g.
> devtool sdk-update /mnt/sdk-repo/
> devtool sdk-update http://mysdkhost/sdk

As noted by Chris we definitely need a better commit message here.

> +        logger.debug("Updating meta data via git ...")
> +        ret = subprocess.call("rm -rf layers && git clone %s/layers" %
> args.updateserver, shell=True) 
> +        if ret != 0:
> +            logger.error("Updating meta data via git clone failed")
> +            return ret

If we're using git we should at least try to pull (assuming we update the 
remote URL first in case the update server is different); if the pull fails we 
can fall back to rm -rf and clone. (Actually, would git pull -f work if the 
remote repo is completely different?)

> +
> +def register_commands(subparsers, context):
> +    """Register devtool subcommands from the sdk plugin"""
> +    parser_sdk = subparsers.add_parser('sdk-update', help='Update SDK
> components from a nominated destination')
> +    parser_sdk.add_argument('updateserver', help='The update server to
> fetch latest SDK components')
> +    parser_sdk.set_defaults(func=sdk_update)

Could you please add a check here for context.fixed_setup before adding the 
sdk-update command - since this command is only useful within the extensible 
SDK and should not be accessible when devtool is running next to a full 
install of the build system.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/8] Extensible SDK fixes

[Paul Eggleton]

On Monday 10 August 2015 11:17:58 Chen Qi wrote:
> The following changes since commit 288c76a41aebaf54c7fca3782160830df462b9e4:
> 
>   builder: Fix multilib compile failure (2015-08-03 07:15:47 +0100)
> 
> are available in the git repository at:
> 
>   git://git.openembedded.org/openembedded-core-contrib ChenQi/sdktool
>  
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=Chen
> Qi/sdktool
> 
> Chen Qi (8):
>   populate_sdk_ext: install the latest buildtools-tarball
>   copy_buildsystem: make sure bitbake directory is copied
>   populate_sdk_ext: consider custom configuration in local.conf
>   populate_sdk_ext: don't remove the native qemu dependencies
>   Extensible SDK: allow for installation without preparing build system
>   oe-publish-sdk: add script
>   populate_sdk_ext: record SDK_TARGETS in devtool.conf
>   devtool: add sdk.py plugin

FWIW I'm happy with patches 1-5; the rest need minor work, however 1-5 can 
probably go in ahead without them.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre