[PATCH 0/7] Extensible SDK fixes

[PATCH 0/7] Extensible SDK fixes

[Paul Eggleton]

Fixes for the extensible SDK and related items.


The following changes since commit 6f98c39418c60b7c0b25b30983d2e5257158a6a4:

  gcc: Drop 4.8 (2015-11-16 14:59:07 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/extsdkfixes2
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/extsdkfixes2

Paul Eggleton (7):
  scripts/gen-lockedsig-cache: improve output
  nativesdk-buildtools-perl-dummy: fix rebuilding when SDKMACHINE changes
  toolchain-shar-extract.sh: do not allow $ in paths for ext SDK
  classes/populate_sdk_ext: tidy up preparation log file writing
  classes/populate_sdk_ext: make it clear when SDK installation has failed
  classes/populate_sdk_ext: tweak reporting of workspace exclusion
  classes/populate_sdk_ext: fail if SDK_ARCH != BUILD_ARCH

 meta/classes/populate_sdk_ext.bbclass                     | 10 ++++++++--
 meta/files/toolchain-shar-extract.sh                      |  4 ++--
 meta/lib/oe/copy_buildsystem.py                           |  5 +++--
 meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb | 11 +++++++++--
 scripts/gen-lockedsig-cache                               | 13 ++++++++++---
 5 files changed, 32 insertions(+), 11 deletions(-)

-- 
2.1.0

[PATCH 1/7] scripts/gen-lockedsig-cache: improve output

[Paul Eggleton]

* Print some status when running
* When incorrect number of arguments specified, print usage text

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/gen-lockedsig-cache | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/scripts/gen-lockedsig-cache b/scripts/gen-lockedsig-cache
index 806c1e4..9c16506 100755
--- a/scripts/gen-lockedsig-cache
+++ b/scripts/gen-lockedsig-cache
@@ -1,7 +1,4 @@
 #!/usr/bin/env python
-#
-# gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir>
-#
 
 import os
 import sys
@@ -18,14 +15,17 @@ def mkdir(d):
 
 if len(sys.argv) < 3:
     print("Incorrect number of arguments specified")
+    print("syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir>")
     sys.exit(1)
 
+print('Reading %s' % sys.argv[1])
 sigs = []
 with open(sys.argv[1]) as f:
     for l in f.readlines():
         if ":" in l:
             sigs.append(l.split(":")[2].split()[0])
 
+print('Gathering file list')
 files = set()
 for s in sigs:
     p = sys.argv[2] + "/" + s[:2] + "/*" + s + "*"
@@ -33,10 +33,13 @@ for s in sigs:
     p = sys.argv[2] + "/*/" + s[:2] + "/*" + s + "*"
     files |= set(glob.glob(p))
 
+print('Processing files')
 for f in files:
+    sys.stdout.write('Processing %s... ' % f)
     _, ext = os.path.splitext(f)
     if not ext in ['.tgz', '.siginfo', '.sig']:
         # Most likely a temp file, skip it
+        print('skipping')
         continue
     dst = f.replace(sys.argv[2], sys.argv[3])
     destdir = os.path.dirname(dst)
@@ -45,6 +48,10 @@ for f in files:
     if os.path.exists(dst):
         os.remove(dst)
     if (os.stat(f).st_dev == os.stat(destdir).st_dev):
+        print('linking')
         os.link(f, dst)
     else:
+        print('copying')
         shutil.copyfile(f, dst)
+
+print('Done!')
-- 
2.1.0

[PATCH 2/7] nativesdk-buildtools-perl-dummy: fix rebuilding when SDKMACHINE changes

[Paul Eggleton]

This recipe produces an empty dummy package (in order to satisfy
dependencies on perl so we don't have perl within buildtools-tarball).
Because we were inheriting nativesdk here the recipe was being rebuilt,
but having forced PACKAGE_ARCH to a particular value the packages for
each architecture were stepping on eachother. Since the packages are
empty they can in fact be allarch (even though they won't actually go
into the "all" package feed). It turns out that nheriting nativesdk
wasn't actually necessary either, so drop that.

Fixes [YOCTO #8509].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb b/meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb
index d971c3c..6fb2b64 100644
--- a/meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb
+++ b/meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb
@@ -2,9 +2,16 @@ SUMMARY = "Dummy package which ensures perl is excluded from buildtools"
 LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
 
-inherit nativesdk
+inherit allarch
+
+python() {
+    # Put the package somewhere separate to ensure it's never used except
+    # when we want it
+    # (note that we have to do this in anonymous python here to avoid
+    # allarch.bbclass disabling itself)
+    d.setVar('PACKAGE_ARCH', 'buildtools-dummy-${SDKPKGSUFFIX}')
+}
 
-# Put it somewhere separate to ensure it's never used except when we want it
 PACKAGE_ARCH = "buildtools-dummy-${SDKPKGSUFFIX}"
 
 PERLPACKAGES = "nativesdk-perl \
-- 
2.1.0

[PATCH 3/7] toolchain-shar-extract.sh: do not allow $ in paths for ext SDK

[Paul Eggleton]

If you put an $ character in the path, SDK installation fails during the
preparation stage, so add this to the disallowed characters.

Fixes [YOCTO #8625].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/files/toolchain-shar-extract.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index 98b9f1c..954b6b3 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -101,9 +101,9 @@ fi
 
 if [ "$SDK_EXTENSIBLE" = "1" ]; then
 	# We're going to be running the build system, additional restrictions apply
-	if echo "$target_sdk_dir" | grep -q '[+\ @]'; then
+	if echo "$target_sdk_dir" | grep -q '[+\ @$]'; then
 		echo "The target directory path ($target_sdk_dir) contains illegal" \
-		     "characters such as spaces, @ or +. Abort!"
+		     "characters such as spaces, @, \$ or +. Abort!"
 		exit 1
 	fi
 else
-- 
2.1.0

[PATCH 4/7] classes/populate_sdk_ext: tidy up preparation log file writing

[Paul Eggleton]

Use a variable for the log file which includes the full path; this is
not only neater but avoids us writing the first part (the output of
oe-init-build-env) to a file in another directory since we are
changing directory as part of this subshell.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 249ec36..d368d4f 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -249,7 +249,8 @@ sdk_ext_postinst() {
 		# dash which is /bin/sh on Ubuntu will not preserve the
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
-		sh -c ". buildtools/environment-setup* > preparing_build_system.log && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> preparing_build_system.log && $target_sdk_dir/ext-sdk-prepare.sh $target_sdk_dir '${SDK_TARGETS}' >> preparing_build_system.log 2>&1" || { echo "SDK preparation failed: see `pwd`/preparing_build_system.log" ; exit 1 ; }
+		LOGFILE="$target_sdk_dir/preparing_build_system.log"
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && $target_sdk_dir/ext-sdk-prepare.sh $target_sdk_dir '${SDK_TARGETS}' >> $LOGFILE 2>&1" || { echo "SDK preparation failed: see $LOGFILE" ; exit 1 ; }
 	fi
 	rm -f $target_sdk_dir/ext-sdk-prepare.sh
 	echo done
-- 
2.1.0

[PATCH 5/7] classes/populate_sdk_ext: make it clear when SDK installation has failed

[Paul Eggleton]

When SDK preparation fails:

* Insert an ERROR: in front of the error message
* Add an error message to the environment setup script

Hopefully this should make it more obvious when this happens.

Fixes [YOCTO #8658].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index d368d4f..e806d32 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -250,7 +250,7 @@ sdk_ext_postinst() {
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && $target_sdk_dir/ext-sdk-prepare.sh $target_sdk_dir '${SDK_TARGETS}' >> $LOGFILE 2>&1" || { echo "SDK preparation failed: see $LOGFILE" ; exit 1 ; }
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && $target_sdk_dir/ext-sdk-prepare.sh $target_sdk_dir '${SDK_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK wasn't fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
 	fi
 	rm -f $target_sdk_dir/ext-sdk-prepare.sh
 	echo done
-- 
2.1.0

[PATCH 6/7] classes/populate_sdk_ext: tweak reporting of workspace exclusion

[Paul Eggleton]

If you have a local workspace layer enabled when building the
extensible SDK, we explicitly exclude that from the SDK (mostly because
the SDK has its own for the user to use). Adjust the message we print
notifying the user of this so it's clear that we're excluding it from
the SDK, and scale it back from a warning to a note printed with
bb.plain().

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 meta/lib/oe/copy_buildsystem.py       | 5 +++--
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index e806d32..736f17f 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -51,7 +51,7 @@ python copy_buildsystem () {
     core_meta_subdir = ''
 
     # Copy in all metadata layers + bitbake (as repositories)
-    buildsystem = oe.copy_buildsystem.BuildSystem(d)
+    buildsystem = oe.copy_buildsystem.BuildSystem('extensible SDK', d)
     baseoutpath = d.getVar('SDK_OUTPUT', True) + '/' + d.getVar('SDKPATH', True)
     layers_copied = buildsystem.copy_bitbake_and_layers(baseoutpath + '/layers')
 
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index 979578c..c0e7541 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -14,8 +14,9 @@ def _smart_copy(src, dest):
         shutil.copymode(src, dest)
 
 class BuildSystem(object):
-    def __init__(self, d):
+    def __init__(self, context, d):
         self.d = d
+        self.context = context
         self.layerdirs = d.getVar('BBLAYERS', True).split()
 
     def copy_bitbake_and_layers(self, destdir):
@@ -38,7 +39,7 @@ class BuildSystem(object):
             if os.path.exists(layerconf):
                 with open(layerconf, 'r') as f:
                     if f.readline().startswith("# ### workspace layer auto-generated by devtool ###"):
-                        bb.warn("Skipping local workspace layer %s" % layer)
+                        bb.plain("NOTE: Excluding local workspace layer %s from %s" % (layer, self.context))
                         continue
 
             # If the layer was already under corebase, leave it there
-- 
2.1.0

[PATCH 7/7] classes/populate_sdk_ext: fail if SDK_ARCH != BUILD_ARCH

[Paul Eggleton]

The extensible SDK relies upon uninative, and with the way that
uninative works, the build system architecture must be the same as the
SDK architecture or the extensible SDK won't be usable. At some point in
future hopefully we can remove this limitation, but until then it's
disingenuous to allow this to build, so add a check to ensure
SDK_ARCH == BUILD_ARCH and fail if it isn't.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 736f17f..dc05261 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -261,6 +261,11 @@ SDK_POST_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_postinst}"
 SDK_POSTPROCESS_COMMAND_prepend_task-populate-sdk-ext = "copy_buildsystem; install_tools; "
 
 fakeroot python do_populate_sdk_ext() {
+    # FIXME hopefully we can remove this restriction at some point, but uninative
+    # currently forces this upon us
+    if d.getVar('SDK_ARCH', True) != d.getVar('BUILD_ARCH', True):
+        bb.fatal('The extensible SDK can currently only be built for the same architecture as the machine being built on - SDK_ARCH is set to %s (likely via setting SDKMACHINE) which is different from the architecture of the build machine (%s). Unable to continue.' % (d.getVar('SDK_ARCH', True), d.getVar('BUILD_ARCH', True)))
+
     bb.build.exec_func("do_populate_sdk", d)
 }
 
-- 
2.1.0