[PATCH 0/9] Show progress during build and eSDK install

[PATCH 0/9] Show progress during build and eSDK install

[Paul Eggleton]

Show progress during build and extensible SDK installation.

I've often thought it would be nice if we could print progress information
for individual tasks so that you could get some idea of whether the task
is doing anything and how much longer it might take, and so I've been working
on a set of patches to implement just that in the background for a while now
and this is the result.  Unfortunately this isn't necessarily practical to do
for all tasks, but things like do_fetch are running processes that do output
progress information so it's not too hard to tap into that and present it in
the form of a progress bar next to the task, which is what this patchset
does. I was also able to enable progress for do_compile in recipes that use
cmake (since that outputs % complete as it runs) as well as do_rootfs where
it has a number of internal steps that can be counted and weighted.

I have contemplated and even prototyped going a step further and using the
count of lines in the output to track progress for *all* long-running tasks,
which does work reasonably well - the caveat is you actually need the
expected line count ahead of time for each task, so you have to be able to
get that from somewhere. I  haven't worked out a practical means of
delivering that, suggestions would be very much welcome - my prototype stuff
is on poky-contrib paule/progress-wip and relies on someone profiling a
build containing all the tasks beforehand and that information somehow being
made available to everyone else.

Also included in this patchset is support for showing progress in a couple
of other stages during the build - when checking for the availability of
sstate artifacts, when initialising the tasks (formerly "Preparing RunQueue")
and overall progress of running setscene and normal tasks.

There are also a few semi-related changes to improve progress reporting and
speed of extensible SDK installation.

NOTE: this patchset requires the corresponding set of changes just sent
to the bitbake-devel list in order to work.


The following changes since commit 2fdd2c1434e7c741b8048145bd460c68bea7ee53:

  systemtap: add build dependency on ncurses (2016-06-12 23:47:00 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/startup-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/startup-oe

Paul Eggleton (9):
  bitbake.conf: whitelist progress varflag
  classes/cmake: enable progress for do_compile
  classes/image: implement progress support for do_rootfs
  classes/sstate: show progress during sstate object availability check
  classes/sstate: add a mode to error if sstate package unavailable
  classes/populate_sdk_ext: eliminate double execution on install
  lib/oe/sstatesig: print locked sigs file message only when explicitly called
  classes/populate_sdk_ext: show progress when preparing build system
  toolchain-shar-extract.sh: allow TERM through into SDK installer env

 meta/classes/cmake.bbclass                        |   1 +
 meta/classes/image.bbclass                        |  10 +-
 meta/classes/populate_sdk_ext.bbclass             |   7 +-
 meta/classes/sstate.bbclass                       |  29 +++++-
 meta/conf/bitbake.conf                            |   6 +-
 meta/files/ext-sdk-prepare.py                     | 117 ++++++++--------------
 meta/files/toolchain-shar-extract.sh              |   1 +
 meta/lib/oe/rootfs.py                             | 101 ++++++++++++++++---
 meta/lib/oe/sstatesig.py                          |  10 +-
 meta/recipes-devtools/cmake/cmake-native_3.5.2.bb |   2 +
 scripts/oe-buildenv-internal                      |   2 +-
 11 files changed, 184 insertions(+), 102 deletions(-)

-- 
2.5.5

[PATCH 1/9] bitbake.conf: whitelist progress varflag

[Paul Eggleton]

We don't want the value of this varflag in any signatures since it's
only there for the purpose of aiding display of task execution.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/conf/bitbake.conf | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 942b8b1..2e95819 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -822,7 +822,8 @@ BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends \
     lockfiles type vardepsexclude vardeps vardepvalue vardepvalueexclude \
     file-checksums python func task export unexport noexec nostamp dirs cleandirs \
     sstate-lockfile-shared prefuncs postfuncs export_func deptask rdeptask \
-    recrdeptask nodeprrecs stamp-extra-info sstate-outputdirs filename lineno"
+    recrdeptask nodeprrecs stamp-extra-info sstate-outputdirs filename lineno \
+    progress"
 
 MLPREFIX ??= ""
 MULTILIB_VARIANTS ??= ""
-- 
2.5.5

[PATCH 2/9] classes/cmake: enable progress for do_compile

[Paul Eggleton]

cmake outputs percentage complete as part of its compilation process, so
we can enable BitBake's new progress scanning for do_compile here.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/cmake.bbclass                        | 1 +
 meta/recipes-devtools/cmake/cmake-native_3.5.2.bb | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index e788310..b18152a 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -124,6 +124,7 @@ cmake_do_configure() {
 	  -Wno-dev
 }
 
+do_compile[progress] = "percent"
 cmake_do_compile()  {
 	cd ${B}
 	base_do_compile VERBOSE=1
diff --git a/meta/recipes-devtools/cmake/cmake-native_3.5.2.bb b/meta/recipes-devtools/cmake/cmake-native_3.5.2.bb
index cb4e749..33930fb 100644
--- a/meta/recipes-devtools/cmake/cmake-native_3.5.2.bb
+++ b/meta/recipes-devtools/cmake/cmake-native_3.5.2.bb
@@ -14,3 +14,5 @@ CMAKE_EXTRACONF = "\
     -DENABLE_ACL=0 -DHAVE_ACL_LIBACL_H=0 \
     -DHAVE_SYS_ACL_H=0 \
 "
+
+do_compile[progress] = "percent"
-- 
2.5.5

[PATCH 3/9] classes/image: implement progress support for do_rootfs

[Paul Eggleton]

Use the new task progress functionality to report progress during
do_rootfs. I did some timings of the various parts of the do_rootfs code
and set some weightings accordingly. This is a little coarse and ideally
we would have some progress within the installation section, but it's
better than nothing.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/image.bbclass |  10 ++++-
 meta/lib/oe/rootfs.py      | 101 +++++++++++++++++++++++++++++++++++++++------
 2 files changed, 96 insertions(+), 15 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 65ce6bb..cf15cf1 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -198,6 +198,10 @@ fakeroot python do_rootfs () {
     from oe.rootfs import create_rootfs
     from oe.manifest import create_manifest
 
+    stage_weights = [0.01, 0.1, 0.18, 0.1, 0.03, 0.5, 0, 0.01, 0.02, 0.02, 0.02, 0.01, 0]
+    progress_reporter = bb.progress.MultiStageProgressReporter(d, stage_weights)
+    progress_reporter.next_stage()
+
     # Handle package exclusions
     excl_pkgs = d.getVar("PACKAGE_EXCLUDE", True).split()
     inst_pkgs = d.getVar("PACKAGE_INSTALL", True).split()
@@ -230,8 +234,10 @@ fakeroot python do_rootfs () {
     # Generate the initial manifest
     create_manifest(d)
 
-    # Generate rootfs
-    create_rootfs(d)
+    progress_reporter.next_stage()
+
+    # generate rootfs
+    create_rootfs(d, progress_reporter=progress_reporter)
 }
 do_rootfs[dirs] = "${TOPDIR}"
 do_rootfs[cleandirs] += "${S}"
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 1fc35bd..ed0bab1 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -15,11 +15,12 @@ class Rootfs(object, metaclass=ABCMeta):
     This is an abstract class. Do not instantiate this directly.
     """
 
-    def __init__(self, d):
+    def __init__(self, d, progress_reporter=None):
         self.d = d
         self.pm = None
         self.image_rootfs = self.d.getVar('IMAGE_ROOTFS', True)
         self.deploy_dir_image = self.d.getVar('DEPLOY_DIR_IMAGE', True)
+        self.progress_reporter = progress_reporter
 
         self.install_order = Manifest.INSTALL_ORDER
 
@@ -191,6 +192,9 @@ class Rootfs(object, metaclass=ABCMeta):
 
         execute_pre_post_process(self.d, pre_process_cmds)
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         # call the package manager dependent create method
         self._create()
 
@@ -205,6 +209,9 @@ class Rootfs(object, metaclass=ABCMeta):
 
         execute_pre_post_process(self.d, post_process_cmds)
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         if bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs",
                          True, False, self.d):
             delayed_postinsts = self._get_delayed_postinsts()
@@ -218,6 +225,9 @@ class Rootfs(object, metaclass=ABCMeta):
 
         self._uninstall_unneeded()
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         self._insert_feed_uris()
 
         self._run_ldconfig()
@@ -228,6 +238,10 @@ class Rootfs(object, metaclass=ABCMeta):
         self._cleanup()
         self._log_check()
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
+
     def _uninstall_unneeded(self):
         # Remove unneeded init script symlinks
         delayed_postinsts = self._get_delayed_postinsts()
@@ -359,8 +373,8 @@ class Rootfs(object, metaclass=ABCMeta):
 
 
 class RpmRootfs(Rootfs):
-    def __init__(self, d, manifest_dir):
-        super(RpmRootfs, self).__init__(d)
+    def __init__(self, d, manifest_dir, progress_reporter=None):
+        super(RpmRootfs, self).__init__(d, progress_reporter)
         self.log_check_regex = '(unpacking of archive failed|Cannot find package'\
                                '|exit 1|ERROR: |Error: |Error |ERROR '\
                                '|Failed |Failed: |Failed$|Failed\(\d+\):)'
@@ -418,11 +432,17 @@ class RpmRootfs(Rootfs):
 
         execute_pre_post_process(self.d, rpm_pre_process_cmds)
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         self.pm.dump_all_available_pkgs()
 
         if self.inc_rpm_image_gen == "1":
             self._create_incremental(pkgs_to_install)
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         self.pm.update()
 
         pkgs = []
@@ -433,12 +453,24 @@ class RpmRootfs(Rootfs):
             else:
                 pkgs += pkgs_to_install[pkg_type]
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         self.pm.install(pkgs)
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         self.pm.install(pkgs_attempt, True)
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         self.pm.install_complementary()
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         self._setup_dbg_rootfs(['/etc/rpm', '/var/lib/rpm', '/var/lib/smart'])
 
         execute_pre_post_process(self.d, rpm_post_process_cmds)
@@ -450,6 +482,10 @@ class RpmRootfs(Rootfs):
 
         self.pm.rpm_setup_smart_target_config()
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
+
     @staticmethod
     def _depends_list():
         return ['DEPLOY_DIR_RPM', 'INC_RPM_IMAGE_GEN', 'RPM_PREPROCESS_COMMANDS',
@@ -494,8 +530,8 @@ class RpmRootfs(Rootfs):
            bb.utils.remove(self.pm.install_dir_path, True)
 
 class DpkgOpkgRootfs(Rootfs):
-    def __init__(self, d):
-        super(DpkgOpkgRootfs, self).__init__(d)
+    def __init__(self, d, progress_reporter=None):
+        super(DpkgOpkgRootfs, self).__init__(d, progress_reporter)
 
     def _get_pkgs_postinsts(self, status_file):
         def _get_pkg_depends_list(pkg_depends):
@@ -589,8 +625,8 @@ class DpkgOpkgRootfs(Rootfs):
             num += 1
 
 class DpkgRootfs(DpkgOpkgRootfs):
-    def __init__(self, d, manifest_dir):
-        super(DpkgRootfs, self).__init__(d)
+    def __init__(self, d, manifest_dir, progress_reporter=None):
+        super(DpkgRootfs, self).__init__(d, progress_reporter)
         self.log_check_regex = '^E:'
         self.log_check_expected_regexes = \
         [
@@ -618,15 +654,31 @@ class DpkgRootfs(DpkgOpkgRootfs):
 
         execute_pre_post_process(self.d, deb_pre_process_cmds)
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+            # Don't support incremental, so skip that
+            self.progress_reporter.next_stage()
+
         self.pm.update()
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         for pkg_type in self.install_order:
             if pkg_type in pkgs_to_install:
                 self.pm.install(pkgs_to_install[pkg_type],
                                 [False, True][pkg_type == Manifest.PKG_TYPE_ATTEMPT_ONLY])
 
+        if self.progress_reporter:
+            # Don't support attemptonly, so skip that
+            self.progress_reporter.next_stage()
+            self.progress_reporter.next_stage()
+
         self.pm.install_complementary()
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         self._setup_dbg_rootfs(['/var/lib/dpkg'])
 
         self.pm.fix_broken_dependencies()
@@ -637,6 +689,9 @@ class DpkgRootfs(DpkgOpkgRootfs):
 
         execute_pre_post_process(self.d, deb_post_process_cmds)
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
     @staticmethod
     def _depends_list():
         return ['DEPLOY_DIR_DEB', 'DEB_SDK_ARCH', 'APTCONF_TARGET', 'APT_ARGS', 'DPKG_ARCH', 'DEB_PREPROCESS_COMMANDS', 'DEB_POSTPROCESS_COMMANDS']
@@ -662,8 +717,8 @@ class DpkgRootfs(DpkgOpkgRootfs):
 
 
 class OpkgRootfs(DpkgOpkgRootfs):
-    def __init__(self, d, manifest_dir):
-        super(OpkgRootfs, self).__init__(d)
+    def __init__(self, d, manifest_dir, progress_reporter=None):
+        super(OpkgRootfs, self).__init__(d, progress_reporter)
         self.log_check_regex = '(exit 1|Collected errors)'
 
         self.manifest = OpkgManifest(d, manifest_dir)
@@ -857,13 +912,24 @@ class OpkgRootfs(DpkgOpkgRootfs):
 
         execute_pre_post_process(self.d, opkg_pre_process_cmds)
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+            # Steps are a bit different in order, skip next
+            self.progress_reporter.next_stage()
+
         self.pm.update()
 
         self.pm.handle_bad_recommendations()
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         if self.inc_opkg_image_gen == "1":
             self._remove_extra_packages(pkgs_to_install)
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         for pkg_type in self.install_order:
             if pkg_type in pkgs_to_install:
                 # For multilib, we perform a sanity test before final install
@@ -875,8 +941,14 @@ class OpkgRootfs(DpkgOpkgRootfs):
                 self.pm.install(pkgs_to_install[pkg_type],
                                 [False, True][pkg_type == Manifest.PKG_TYPE_ATTEMPT_ONLY])
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         self.pm.install_complementary()
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
         self._setup_dbg_rootfs(['/etc', '/var/lib/opkg', '/usr/lib/ssl'])
 
         execute_pre_post_process(self.d, opkg_post_process_cmds)
@@ -884,6 +956,9 @@ class OpkgRootfs(DpkgOpkgRootfs):
         if self.inc_opkg_image_gen == "1":
             self.pm.backup_packaging_data()
 
+        if self.progress_reporter:
+            self.progress_reporter.next_stage()
+
     @staticmethod
     def _depends_list():
         return ['IPKGCONF_SDK', 'IPK_FEED_URIS', 'DEPLOY_DIR_IPK', 'IPKGCONF_TARGET', 'INC_IPK_IMAGE_GEN', 'OPKG_ARGS', 'OPKGLIBDIR', 'OPKG_PREPROCESS_COMMANDS', 'OPKG_POSTPROCESS_COMMANDS', 'OPKGLIBDIR']
@@ -919,16 +994,16 @@ def variable_depends(d, manifest_dir=None):
     cls = get_class_for_type(img_type)
     return cls._depends_list()
 
-def create_rootfs(d, manifest_dir=None):
+def create_rootfs(d, manifest_dir=None, progress_reporter=None):
     env_bkp = os.environ.copy()
 
     img_type = d.getVar('IMAGE_PKGTYPE', True)
     if img_type == "rpm":
-        RpmRootfs(d, manifest_dir).create()
+        RpmRootfs(d, manifest_dir, progress_reporter).create()
     elif img_type == "ipk":
-        OpkgRootfs(d, manifest_dir).create()
+        OpkgRootfs(d, manifest_dir, progress_reporter).create()
     elif img_type == "deb":
-        DpkgRootfs(d, manifest_dir).create()
+        DpkgRootfs(d, manifest_dir, progress_reporter).create()
 
     os.environ.clear()
     os.environ.update(env_bkp)
-- 
2.5.5

[PATCH 4/9] classes/sstate: show progress during sstate object availability check

[Paul Eggleton]

Show progress through bitbake's standard terminal UI when checking for
shared state object availability, since this can take some time if there
are a large number of tasks to be executed and/or the network connection
is slow.

Part of the implementation for [YOCTO #5853].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/sstate.bbclass | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 994eae6..4e81fc9 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -800,7 +800,8 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
             except:
                 missed.append(task)
                 bb.debug(2, "SState: Unsuccessful fetch test for %s" % srcuri)
-                pass     
+                pass
+            bb.event.fire(bb.event.ProcessProgress("Checking sstate mirror object availability", len(tasklist) - thread_worker.tasks.qsize()), d)
 
         tasklist = []
         for task in range(len(sq_fn)):
@@ -811,7 +812,7 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
             tasklist.append((task, sstatefile))
 
         if tasklist:
-            bb.note("Checking sstate mirror object availability (for %s objects)" % len(tasklist))
+            bb.event.fire(bb.event.ProcessStarted("Checking sstate mirror object availability", len(tasklist)), d)
             import multiprocessing
             nproc = min(multiprocessing.cpu_count(), len(tasklist))
 
@@ -821,6 +822,7 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
                 pool.add_task(checkstatus, t)
             pool.start()
             pool.wait_completion()
+            bb.event.fire(bb.event.ProcessFinished("Checking sstate mirror object availability"), d)
 
     inheritlist = d.getVar("INHERIT", True)
     if "toaster" in inheritlist:
-- 
2.5.5

[PATCH 5/9] classes/sstate: add a mode to error if sstate package unavailable

[Paul Eggleton]

If BB_SETSCENE_ENFORCE is set to "1" and an sstate package fails to
download outside of the whitelist specified by
BB_SETSCENE_ENFORCE_WHITELIST, then fail immediately so you can tell
that the problem was caused by failing to restore the task from sstate.

Part of the implementation of [YOCTO #9367].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/sstate.bbclass  | 23 +++++++++++++++++++++++
 meta/conf/bitbake.conf       |  3 ++-
 scripts/oe-buildenv-internal |  2 +-
 3 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 4e81fc9..621dc37 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -719,6 +719,7 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
 
     ret = []
     missed = []
+    missing = []
     extension = ".tgz"
     if siginfo:
         extension = extension + ".siginfo"
@@ -740,6 +741,18 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
 
         return spec, extrapath, tname
 
+    def sstate_pkg_to_pn(pkg, d):
+        """
+        Translate an sstate filename to a PN value by way of SSTATE_PKGSPEC. This is slightly hacky but
+        we don't have access to everything in this context.
+        """
+        pkgspec = d.getVar('SSTATE_PKGSPEC', False)
+        try:
+            idx = pkgspec.split(':').index('${PN}')
+        except ValueError:
+            bb.fatal('Unable to find ${PN} in SSTATE_PKGSPEC')
+        return pkg.split(':')[idx]
+
 
     for task in range(len(sq_fn)):
 
@@ -774,6 +787,8 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
         if localdata.getVar('BB_NO_NETWORK', True) == "1" and localdata.getVar('SSTATE_MIRROR_ALLOW_NETWORK', True) == "1":
             localdata.delVar('BB_NO_NETWORK')
 
+        whitelist = bb.runqueue.get_setscene_enforce_whitelist(d)
+
         from bb.fetch2 import FetchConnectionCache
         def checkstatus_init(thread_worker):
             thread_worker.connection_cache = FetchConnectionCache()
@@ -800,6 +815,12 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
             except:
                 missed.append(task)
                 bb.debug(2, "SState: Unsuccessful fetch test for %s" % srcuri)
+                if whitelist:
+                    pn = sstate_pkg_to_pn(sstatefile, d)
+                    taskname = sq_task[task]
+                    if not bb.runqueue.check_setscene_enforce_whitelist(pn, taskname, whitelist):
+                        missing.append(task)
+                        bb.error('Sstate artifact unavailable for %s.%s' % (pn, taskname))
                 pass
             bb.event.fire(bb.event.ProcessProgress("Checking sstate mirror object availability", len(tasklist) - thread_worker.tasks.qsize()), d)
 
@@ -823,6 +844,8 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
             pool.start()
             pool.wait_completion()
             bb.event.fire(bb.event.ProcessFinished("Checking sstate mirror object availability"), d)
+            if whitelist and missing:
+                bb.fatal('Required artifacts were unavailable - exiting')
 
     inheritlist = d.getVar("INHERIT", True)
     if "toaster" in inheritlist:
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 2e95819..45d9862 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -817,7 +817,8 @@ BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
     SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \
     PARALLEL_MAKE BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \
     GIT_PROXY_COMMAND ALL_PROXY all_proxy NO_PROXY no_proxy FTP_PROXY ftp_proxy \
-    HTTP_PROXY http_proxy HTTPS_PROXY https_proxy SOCKS5_USER SOCKS5_PASSWD"
+    HTTP_PROXY http_proxy HTTPS_PROXY https_proxy SOCKS5_USER SOCKS5_PASSWD \
+    BB_SETSCENE_ENFORCE"
 BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends \
     lockfiles type vardepsexclude vardeps vardepvalue vardepvalueexclude \
     file-checksums python func task export unexport noexec nostamp dirs cleandirs \
diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv-internal
index 56d3419..03dc50f 100755
--- a/scripts/oe-buildenv-internal
+++ b/scripts/oe-buildenv-internal
@@ -118,7 +118,7 @@ BB_ENV_EXTRAWHITE_OE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy \
 HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY \
 all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY \
 SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND \
-SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA"
+SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA BB_SETSCENE_ENFORCE"
 
 BB_ENV_EXTRAWHITE="$(echo $BB_ENV_EXTRAWHITE $BB_ENV_EXTRAWHITE_OE | tr ' ' '\n' | LC_ALL=C sort --unique | tr '\n' ' ')"
 
-- 
2.5.5

[PATCH 6/9] classes/populate_sdk_ext: eliminate double execution on install

[Paul Eggleton]

Use the new BB_SETSCENE_ENFORCE functionality to avoid having to run
bitbake twice on installing the extensible SDK - we can now do it all in
one invocation and we should get more meaningful errors.

Part of the implementation of [YOCTO #9367].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass |  5 +++-
 meta/files/ext-sdk-prepare.py         | 52 +----------------------------------
 2 files changed, 5 insertions(+), 52 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index b9d9543..67297e5 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -223,10 +223,13 @@ python copy_buildsystem () {
             # warning.
             f.write('SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"\n\n')
 
-            # Error if the sigs in the locked-signature file don't match
+            # Warn if the sigs in the locked-signature file don't match
             # the sig computed from the metadata.
             f.write('SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n\n')
 
+            # Set up whitelist for run on install
+            f.write('BB_SETSCENE_ENFORCE_WHITELIST = "%:* *:do_shared_workdir *:do_rm_work"\n\n')
+
             # Hide the config information from bitbake output (since it's fixed within the SDK)
             f.write('BUILDCFG_HEADER = ""\n')
 
diff --git a/meta/files/ext-sdk-prepare.py b/meta/files/ext-sdk-prepare.py
index bf0d033..3b33c0f 100644
--- a/meta/files/ext-sdk-prepare.py
+++ b/meta/files/ext-sdk-prepare.py
@@ -27,22 +27,6 @@ def exec_watch(cmd, **options):
 
     return process.returncode, buf
 
-def check_unexpected(lines, recipes):
-    """Check for unexpected output lines from dry run"""
-    unexpected = []
-    for line in lines.splitlines():
-        if 'Running task' in line:
-            for recipe in recipes:
-                if recipe in line:
-                    break
-            else:
-                line = line.split('Running', 1)[-1]
-                if 'do_rm_work' not in line:
-                    unexpected.append(line.rstrip())
-        elif 'Running setscene' in line:
-            unexpected.append(line.rstrip())
-    return unexpected
-
 def main():
     if len(sys.argv) < 2:
         sdk_targets = []
@@ -55,44 +39,10 @@ def main():
 
     print('Preparing SDK for %s...' % ', '.join(sdk_targets))
 
-    ret, out = exec_watch('bitbake %s --setscene-only' % ' '.join(sdk_targets))
+    ret, out = exec_watch('BB_SETSCENE_ENFORCE=1 bitbake %s' % ' '.join(sdk_targets))
     if ret:
         return ret
 
-    targetlist = []
-    for target in sdk_targets:
-        if ':' in target:
-            target = target.split(':')[0]
-        if not target in targetlist:
-            targetlist.append(target)
-
-    recipes = []
-    for target in targetlist:
-        try:
-            out = subprocess.check_output(('bitbake -e %s' % target).split(), stderr=subprocess.STDOUT)
-            for line in out.splitlines():
-                if line.startswith('FILE='):
-                    splitval = line.rstrip().split('=')
-                    if len(splitval) > 1:
-                        recipes.append(splitval[1].strip('"'))
-                    break
-        except subprocess.CalledProcessError as e:
-            print('ERROR: Failed to get recipe for target %s:\n%s' % (target, e.output))
-            return 1
-
-    try:
-        out = subprocess.check_output('bitbake %s -n' % ' '.join(sdk_targets), stderr=subprocess.STDOUT, shell=True)
-        unexpected = check_unexpected(out, recipes)
-    except subprocess.CalledProcessError as e:
-        print('ERROR: Failed to execute dry-run:\n%s' % e.output)
-        return 1
-
-    if unexpected:
-        print('ERROR: Unexpected tasks or setscene left over to be executed:')
-        for line in unexpected:
-            print('  ' + line)
-        return 1
-
 if __name__ == "__main__":
     try:
         ret = main()
-- 
2.5.5

[PATCH 7/9] lib/oe/sstatesig: print locked sigs file message only when explicitly called

[Paul Eggleton]

If we're building the extensible SDK we don't need to see the "Writing
locked sigs" message; it's only necessary when the user explicitly runs
bitbake -S none <target>.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/sstatesig.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index a58f03a..8224e3a 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -130,7 +130,9 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
         super(bb.siggen.SignatureGeneratorBasicHash, self).set_taskdata(coredata)
 
     def dump_sigs(self, dataCache, options):
-        self.dump_lockedsigs()
+        sigfile = os.getcwd() + "/locked-sigs.inc"
+        bb.plain("Writing locked sigs to %s" % sigfile)
+        self.dump_lockedsigs(sigfile)
         return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigs(dataCache, options)
 
     def get_taskhash(self, fn, task, deps, dataCache):
@@ -181,11 +183,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
             return
         super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime)
 
-    def dump_lockedsigs(self, sigfile=None, taskfilter=None):
-        if not sigfile:
-            sigfile = os.getcwd() + "/locked-sigs.inc"
-
-        bb.plain("Writing locked sigs to %s" % sigfile)
+    def dump_lockedsigs(self, sigfile, taskfilter=None):
         types = {}
         for k in self.runtaskdeps:
             if taskfilter:
-- 
2.5.5

[PATCH 8/9] classes/populate_sdk_ext: show progress when preparing build system

[Paul Eggleton]

During the extensible SDK installation process the final step is to
prepare the internal copy of the build system. This can take some time,
especially if you have SDK_EXT_TYPE set to "minimal" (downloading
sstate artifacts) and SDK_INCLUDE_PKGDATA set to "1" (restoring
pkgdata for world). To make this a bit less painful, use BitBake's new
quiet mode to display status during this operation so you have some idea
of how it's progressing; instead of redirecting the output to
preparing_build_system.log we grab the last console log and append it
instead.

Part of the implementation for [YOCTO #9613].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass |  2 +-
 meta/files/ext-sdk-prepare.py         | 67 ++++++++++++++++++++++-------------
 2 files changed, 44 insertions(+), 25 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 67297e5..d049fb5 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -427,7 +427,7 @@ sdk_ext_postinst() {
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py '${SDK_INSTALL_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; cat "$LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { cat "$LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
 		rm $target_sdk_dir/ext-sdk-prepare.py
 	fi
 	echo done
diff --git a/meta/files/ext-sdk-prepare.py b/meta/files/ext-sdk-prepare.py
index 3b33c0f..8b15982 100644
--- a/meta/files/ext-sdk-prepare.py
+++ b/meta/files/ext-sdk-prepare.py
@@ -5,43 +5,62 @@
 import sys
 import os
 import subprocess
+import signal
 
-def exec_watch(cmd, **options):
-    """Run program with stdout shown on sys.stdout"""
-    if isinstance(cmd, str) and not "shell" in options:
-        options["shell"] = True
+def reenable_sigint():
+    signal.signal(signal.SIGINT, signal.SIG_DFL)
 
-    process = subprocess.Popen(
-        cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **options
-    )
-
-    buf = ''
-    while True:
-        out = process.stdout.read(1)
-        if out:
-            sys.stdout.write(out)
-            sys.stdout.flush()
-            buf += out
-        elif out == '' and process.poll() != None:
-            break
+def run_command_interruptible(cmd):
+    """
+    Run a command with output displayed on the console, but ensure any Ctrl+C is
+    processed only by the child process.
+    """
+    signal.signal(signal.SIGINT, signal.SIG_IGN)
+    try:
+        ret = subprocess.call(cmd, shell=True, preexec_fn=reenable_sigint)
+    finally:
+        signal.signal(signal.SIGINT, signal.SIG_DFL)
+    return ret
 
-    return process.returncode, buf
+def get_last_consolelog():
+    '''Return the most recent console log file'''
+    logdir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'tmp', 'log', 'cooker')
+    if os.path.exists(logdir):
+        mcdir = os.listdir(logdir)
+        if mcdir:
+            logdir = os.path.join(logdir, mcdir[0])
+            logfiles = [os.path.join(logdir, fn) for fn in os.listdir(logdir)]
+            logfiles.sort(key=os.path.getmtime)
+            if logfiles:
+                return os.path.join(logdir, logfiles[-1])
+    return None
 
 def main():
     if len(sys.argv) < 2:
+        print('Please specify output log file')
+        return 1
+    logfile = sys.argv[1]
+    if len(sys.argv) < 3:
         sdk_targets = []
     else:
-        sdk_targets = ' '.join(sys.argv[1:]).split()
+        sdk_targets = ' '.join(sys.argv[2:]).split()
     if not sdk_targets:
         # Just do a parse so the cache is primed
-        ret, _ = exec_watch('bitbake -p')
+        ret = run_command_interruptible('bitbake -p --quiet')
         return ret
 
-    print('Preparing SDK for %s...' % ', '.join(sdk_targets))
+    with open(logfile, 'a') as logf:
+        logf.write('Preparing SDK for %s...\n' % ', '.join(sdk_targets))
 
-    ret, out = exec_watch('BB_SETSCENE_ENFORCE=1 bitbake %s' % ' '.join(sdk_targets))
-    if ret:
-        return ret
+        ret = run_command_interruptible('BB_SETSCENE_ENFORCE=1 bitbake --quiet %s' % ' '.join(sdk_targets))
+        lastlog = get_last_consolelog()
+        if lastlog:
+            with open(lastlog, 'r') as f:
+                for line in f:
+                    logf.write(line)
+        if ret:
+            print('ERROR: SDK preparation failed: see %s' % logfile)
+            return ret
 
 if __name__ == "__main__":
     try:
-- 
2.5.5

[PATCH 9/9] toolchain-shar-extract.sh: allow TERM through into SDK installer env

[Paul Eggleton]

In order to show the task progress during the extensible SDK
installation, knotty needs to know it is using a terminal it can
support the "interactive" mode on (i.e. ncurses can be used), which
means it needs access to the TERM variable value, so allow that
through into the cleaned environment within the SDK installation
script.

Part of the implementation for [YOCTO #9613].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/files/toolchain-shar-extract.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index d1ce86d..9279310 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -2,6 +2,7 @@
 
 [ -z "$ENVCLEANED" ] && exec /usr/bin/env -i ENVCLEANED=1 HOME="$HOME" \
 	LC_ALL=en_US.UTF-8 \
+	TERM=$TERM \
 	http_proxy="$http_proxy" https_proxy="$https_proxy" ftp_proxy="$ftp_proxy" \
 	no_proxy="$no_proxy" all_proxy="$all_proxy" GIT_PROXY_COMMAND="$GIT_PROXY_COMMAND" "$0" "$@"
 [ -f /etc/environment ] && . /etc/environment
-- 
2.5.5

Re: [PATCH 0/9] Show progress during build and eSDK install

[Randle, William C]

On Mon, 2016-06-13 at 15:13 +1200, Paul Eggleton wrote:
> 
> I have contemplated and even prototyped going a step further and using the
> count of lines in the output to track progress for *all* long-running tasks,
> which does work reasonably well - the caveat is you actually need the
> expected line count ahead of time for each task, so you have to be able to
> get that from somewhere. I  haven't worked out a practical means of
> delivering that, suggestions would be very much welcome - my prototype stuff
> is on poky-contrib paule/progress-wip and relies on someone profiling a
> build containing all the tasks beforehand and that information somehow being
> made available to everyone else.

Could this be done somewhat analagous to the way the sstate stuff works, in the
sense you could specify a local or remote line count mirror, or use a local
directory. The first time a unique task is run, the line count from the output
is saved in the line count db. On subsequent runs, it would detect there's
already an entry in the db and would be able to provide the progress report.

Similar to the way an autobuilder can create a shared sstate mirror, when a task
is run on an autobuilder, it could save the line count and make it available to
other consumers, so on first run you would get a progress report, rather than
having to bootstrap the line counts first.

    -Bill

Re: [PATCH 0/9] Show progress during build and eSDK install

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2005 bytes --]

On Mon, Jun 13, 2016 at 8:47 AM, Randle, William C <
william.c.randle@intel.com> wrote:

> On Mon, 2016-06-13 at 15:13 +1200, Paul Eggleton wrote:
> >
> > I have contemplated and even prototyped going a step further and using
> the
> > count of lines in the output to track progress for *all* long-running
> tasks,
> > which does work reasonably well - the caveat is you actually need the
> > expected line count ahead of time for each task, so you have to be able
> to
> > get that from somewhere. I  haven't worked out a practical means of
> > delivering that, suggestions would be very much welcome - my prototype
> stuff
> > is on poky-contrib paule/progress-wip and relies on someone profiling a
> > build containing all the tasks beforehand and that information somehow
> being
> > made available to everyone else.
>
> Could this be done somewhat analagous to the way the sstate stuff works,
> in the
> sense you could specify a local or remote line count mirror, or use a local
> directory. The first time a unique task is run, the line count from the
> output
> is saved in the line count db. On subsequent runs, it would detect there's
> already an entry in the db and would be able to provide the progress
> report.
>
> Similar to the way an autobuilder can create a shared sstate mirror, when
> a task
> is run on an autobuilder, it could save the line count and make it
> available to
> other consumers, so on first run you would get a progress report, rather
> than
> having to bootstrap the line counts first.


I think if you want to do this, then we should do it in a generic way, to
store info of that sort from a build and use it as input to another build,
i.e. another sqlite3 db or something, as this isn't the only case where
build output could be used as hints for another build.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2450 bytes --]

Re: [PATCH 0/9] Show progress during build and eSDK install

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 3121 bytes --]

On Sun, Jun 12, 2016 at 8:13 PM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> Show progress during build and extensible SDK installation.
>
> I've often thought it would be nice if we could print progress information
> for individual tasks so that you could get some idea of whether the task
> is doing anything and how much longer it might take, and so I've been
> working
> on a set of patches to implement just that in the background for a while
> now
> and this is the result.  Unfortunately this isn't necessarily practical to
> do
> for all tasks, but things like do_fetch are running processes that do
> output
> progress information so it's not too hard to tap into that and present it
> in
> the form of a progress bar next to the task, which is what this patchset
> does. I was also able to enable progress for do_compile in recipes that use
> cmake (since that outputs % complete as it runs) as well as do_rootfs where
> it has a number of internal steps that can be counted and weighted.
>
> I have contemplated and even prototyped going a step further and using the
> count of lines in the output to track progress for *all* long-running
> tasks,
> which does work reasonably well - the caveat is you actually need the
> expected line count ahead of time for each task, so you have to be able to
> get that from somewhere. I  haven't worked out a practical means of
> delivering that, suggestions would be very much welcome - my prototype
> stuff
> is on poky-contrib paule/progress-wip and relies on someone profiling a
> build containing all the tasks beforehand and that information somehow
> being
> made available to everyone else.
>
> Also included in this patchset is support for showing progress in a couple
> of other stages during the build - when checking for the availability of
> sstate artifacts, when initialising the tasks (formerly "Preparing
> RunQueue")
> and overall progress of running setscene and normal tasks.
>
> There are also a few semi-related changes to improve progress reporting and
> speed of extensible SDK installation.
>
> NOTE: this patchset requires the corresponding set of changes just sent
> to the bitbake-devel list in order to work.
>

This is pretty sweet, nice work on this, Paul. A couple random thoughts:

1. do_fetch progress is *awesome*, especially for the kernel :)
2. there are do_setscene progress bars, but they seem to just pop up and
vanish in the common case, since they run so quickly. we might want to add
a minimum time threshold to the UI, so if the task has been running less
than 5 seconds, don't even bother showing it
3. it may be worth adding a spinner for cases where we don't know the
expected line count, but want to be able to indicate that something is
happening, i.e. just spin the spinner if output is still coming from the
task's stdout, so the user doesn't get paranoid and think the compile hung
:)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3645 bytes --]