[PATCH 0/5] Extensible SDK fixes

[PATCH 0/5] Extensible SDK fixes

[Paul Eggleton]

The following changes since commit e3fc8ef152494e9b7cff8de110a784150295f17e:

  recipes-support/rng-tools: Change runlevel start from S to 2, 3, 4, 5. (2016-03-30 21:32:11 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/extsdkfixes10-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/extsdkfixes10-oe

Paul Eggleton (5):
  classes/populate_sdk_ext: support setting vars from environment at build time
  oe-publish-sdk: prevent specifying a directory for the SDK argument
  oe-publish-sdk: exclude sstate-cache if publishing minimal SDK
  toolchain-shar-extract.sh: ensure all_proxy is allowed through
  devtool: modify: call shutdown on tinfoil when done

 meta/classes/populate_sdk_ext.bbclass | 26 ++++++++++++++++++++++++++
 meta/files/toolchain-shar-extract.sh  | 13 ++++++++++---
 scripts/lib/devtool/standard.py       |  2 ++
 scripts/oe-publish-sdk                | 13 +++++++------
 4 files changed, 45 insertions(+), 9 deletions(-)

-- 
2.5.5

[PATCH 1/5] classes/populate_sdk_ext: support setting vars from environment at build time

[Paul Eggleton]

When running bitbake you may pass in values of variables from the
external environment (making use of BB_ENV_EXTRAWHITE), and you may
choose to do this when building the extensible SDK, for example:

  MACHINE=qemuarm bitbake -c populate_sdk_ext core-image-minimal

You would naturally expect those settings to be reflected in the
extensible SDK itself; however they were not, since we were only
considering local.conf and auto.conf. Check the variables mentioned in
BB_ENV_EXTRAWHITE to see if any are different than the values set in
local.conf/auto.conf and add lines setting them in the SDK's local.conf
if so.

Fixes [YOCTO #9339].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 0ea974d..5e2ebd7 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -164,6 +164,9 @@ python copy_buildsystem () {
         f.write('    $' + '{SDKBASEMETAPATH}/workspace \\\n')
         f.write('    "\n')
 
+    env_whitelist = (d.getVar('BB_ENV_EXTRAWHITE', True) or '').split()
+    env_whitelist_values = {}
+
     # Create local.conf
     builddir = d.getVar('TOPDIR', True)
     if derivative:
@@ -176,6 +179,8 @@ python copy_buildsystem () {
                 newlines.append('# Removed original setting of %s\n' % varname)
                 return None, op, 0, True
             else:
+                if varname in env_whitelist:
+                    env_whitelist_values[varname] = origvalue
                 return origvalue, op, 0, True
         varlist = ['[^#=+ ]*']
         with open(builddir + '/conf/local.conf', 'r') as f:
@@ -241,6 +246,21 @@ python copy_buildsystem () {
                     if line.strip() and not line.startswith('#'):
                         f.write(line)
 
+    # Ensure any variables set from the external environment (by way of
+    # BB_ENV_EXTRAWHITE) are set in the SDK's configuration
+    extralines = []
+    for name, value in env_whitelist_values.iteritems():
+        actualvalue = d.getVar(name, True) or ''
+        if value != actualvalue:
+            extralines.append('%s = "%s"\n' % (name, actualvalue))
+    if extralines:
+        with open(baseoutpath + '/conf/local.conf', 'a') as f:
+            f.write('\n')
+            f.write('# Extra settings from environment:\n')
+            for line in extralines:
+                f.write(line)
+            f.write('\n')
+
     # Filter the locked signatures file to just the sstate tasks we are interested in
     excluded_targets = d.getVar('SDK_TARGETS', True)
     sigfile = d.getVar('WORKDIR', True) + '/locked-sigs.inc'
-- 
2.5.5

[PATCH 2/5] oe-publish-sdk: prevent specifying a directory for the SDK argument

[Paul Eggleton]

The SDK argument is expected to be an installer .sh file; if a directory
is specified we can get an ugly failure later on; best to check up
front.

Fixes [YOCTO #9065].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/oe-publish-sdk | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/oe-publish-sdk b/scripts/oe-publish-sdk
index 1729a0d..cd912df 100755
--- a/scripts/oe-publish-sdk
+++ b/scripts/oe-publish-sdk
@@ -50,7 +50,10 @@ def publish(args):
 
     # Ensure the SDK exists
     if not os.path.exists(target_sdk):
-        logger.error("%s doesn't exist" % target_sdk)
+        logger.error("Specified SDK %s doesn't exist" % target_sdk)
+        return -1
+    if os.path.isdir(target_sdk):
+        logger.error("%s is a directory - expected path to SDK installer file" % target_sdk)
         return -1
 
     if ':' in destination:
-- 
2.5.5

[PATCH 3/5] oe-publish-sdk: exclude sstate-cache if publishing minimal SDK

[Paul Eggleton]

If SDK_EXT_TYPE is set to "minimal" then the SDK won't contain many
sstate artifacts, and you're required to set up an sstate mirror in this
case anyway so there's no point publishing the "stub" sstate-cache
directory from within the SDK since it won't be useful for update
purposes and may be confused with the real sstate-cache.

There is however a possibility that people might publish the real
sstate-cache directory under the same output directory provided to
oe-publish-sdk, thus deleting it after extracting (as we were doing with
other files we wanted to clean up at the end) would be problematic,
besides which extracting it and then deleting it is wasteful. Thus,
introduce a "-p" command line option to the SDK installer that we can
use to tell tar not to extract the items we don't want when publishing.
This has the added benefit of mostly keeping references to these in the
place they belong i.e. in populate_sdk_ext.bbclass.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass |  6 ++++++
 meta/files/toolchain-shar-extract.sh  | 11 +++++++++--
 scripts/oe-publish-sdk                |  8 +++-----
 3 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 5e2ebd7..2bbd181 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -365,6 +365,12 @@ sdk_ext_preinst() {
 		exit 1
 	fi
 	SDK_EXTENSIBLE="1"
+	if [ "$publish" = "1" ] ; then
+		EXTRA_TAR_OPTIONS="$EXTRA_TAR_OPTIONS --exclude=ext-sdk-prepare.py"
+		if [ "${SDK_EXT_TYPE}" = "minimal" ] ; then
+			EXTRA_TAR_OPTIONS="$EXTRA_TAR_OPTIONS --exclude=sstate-cache"
+		fi
+	fi
 }
 SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}"
 
diff --git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index 0295bde..23a86dd 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -38,12 +38,14 @@ fi
 
 DEFAULT_INSTALL_DIR="@SDKPATH@"
 SUDO_EXEC=""
+EXTRA_TAR_OPTIONS=""
 target_sdk_dir=""
 answer=""
 relocate=1
 savescripts=0
 verbose=0
-while getopts ":yd:nDRS" OPT; do
+publish=0
+while getopts ":yd:npDRS" OPT; do
 	case $OPT in
 	y)
 		answer="Y"
@@ -54,6 +56,10 @@ while getopts ":yd:nDRS" OPT; do
 	n)
 		prepare_buildsystem="no"
 		;;
+	p)
+		prepare_buildsystem="no"
+		publish=1
+		;;
 	D)
 		verbose=1
 		;;
@@ -70,6 +76,7 @@ while getopts ":yd:nDRS" OPT; do
 		echo "  -d <dir>   Install the SDK to <dir>"
 		echo "======== Extensible SDK only options ============"
 		echo "  -n         Do not prepare the build system"
+		echo "  -p         Publish mode (implies -n)"
 		echo "======== Advanced DEBUGGING ONLY OPTIONS ========"
 		echo "  -S         Save relocation scripts"
 		echo "  -R         Do not relocate executables"
@@ -181,7 +188,7 @@ fi
 payload_offset=$(($(grep -na -m1 "^MARKER:$" $0|cut -d':' -f1) + 1))
 
 printf "Extracting SDK..."
-tail -n +$payload_offset $0| $SUDO_EXEC tar xJ -C $target_sdk_dir --checkpoint=.2500 || exit 1
+tail -n +$payload_offset $0| $SUDO_EXEC tar xJ -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1
 echo "done"
 
 printf "Setting it up..."
diff --git a/scripts/oe-publish-sdk b/scripts/oe-publish-sdk
index cd912df..71bc972 100755
--- a/scripts/oe-publish-sdk
+++ b/scripts/oe-publish-sdk
@@ -94,19 +94,17 @@ def publish(args):
 
     # Unpack the SDK
     logger.info("Unpacking SDK")
-    cleanupfiles = [dest_sdk, os.path.join(destdir, 'ext-sdk-prepare.py')]
     if not is_remote:
-        cmd = "sh %s -n -y -d %s" % (dest_sdk, destination)
+        cmd = "sh %s -p -y -d %s" % (dest_sdk, destination)
         ret = subprocess.call(cmd, shell=True)
         if ret == 0:
             logger.info('Successfully unpacked %s to %s' % (dest_sdk, destination))
-            for cleanupfile in cleanupfiles:
-                os.remove(cleanupfile)
+            os.remove(dest_sdk)
         else:
             logger.error('Failed to unpack %s to %s' % (dest_sdk, destination))
             return ret
     else:
-        cmd = "ssh %s 'sh %s -n -y -d %s && rm -f %s'" % (host, dest_sdk, destdir, ' '.join(cleanupfiles))
+        cmd = "ssh %s 'sh %s -p -y -d %s && rm -f %s'" % (host, dest_sdk, destdir, dest_sdk)
         ret = subprocess.call(cmd, shell=True)
         if ret == 0:
             logger.info('Successfully unpacked %s to %s' % (dest_sdk, destdir))
-- 
2.5.5

[PATCH 4/5] toolchain-shar-extract.sh: ensure all_proxy is allowed through

[Paul Eggleton]

all_proxy is referred to by oe-git-proxy so ensure it is allowed through
into the installer environment in case the extensible SDK install
process needs to query a remote git repository.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/files/toolchain-shar-extract.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index 23a86dd..397be11 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -2,7 +2,7 @@
 
 [ -z "$ENVCLEANED" ] && exec /usr/bin/env -i ENVCLEANED=1 HOME="$HOME" \
 	http_proxy="$http_proxy" https_proxy="$https_proxy" ftp_proxy="$ftp_proxy" \
-	no_proxy="$no_proxy" GIT_PROXY_COMMAND="$GIT_PROXY_COMMAND" "$0" "$@"
+	no_proxy="$no_proxy" all_proxy="$all_proxy" GIT_PROXY_COMMAND="$GIT_PROXY_COMMAND" "$0" "$@"
 [ -f /etc/environment ] && . /etc/environment
 export PATH=`echo "$PATH" | sed -e 's/:\.//' -e 's/::/:/'`
 
-- 
2.5.5

[PATCH 5/5] devtool: modify: call shutdown on tinfoil when done

[Paul Eggleton]

Strictly speaking we ought to explicitly shut down a tinfoil instance
when we're done with it. This doesn't affect modify's operation but is
important if you want to be able to call into modify() from another
plugin (though anyone doing so should be advised that the function is
by no means a stable API and is subject to change in future releases).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index f3f3e98..77a82d5 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -805,6 +805,8 @@ def modify(args, config, basepath, workspace):
 
     logger.info('Recipe %s now set up to build from %s' % (pn, srctree))
 
+    tinfoil.shutdown()
+
     return 0
 
 def _get_patchset_revs(args, srctree, recipe_path):
-- 
2.5.5