[PATCH 0/2] Python 3 runqemu fixes

[PATCH 0/2] Python 3 runqemu fixes

[Joshua Lock]

Note: This patch is against master-next, as the Python 3 runqemu hasn't made it
to master yet.
This supercedes my earlier one patch series "Python3 runqemu: remove use of subprocess.run()" (both versions)

Two fixes for the Python3 runqemu:
* Use of subprocess.run() in recent runqemu changes (by me) elevates the Python
  3 requirement from 3.4+ to 3.5+. Replace subprocess.run() with
  subprocess.check_output(), as that is available in Python 3.1+
* testimage calls runqemu from within a bitbake run, which prevents us from 
  using `bitbake -e` to determine bitbake environment variables. The test code
  exports an OE_TMPDIR variable which was used by the shell runqemu to 
  infer/guess paths into the sysroot. This patch does the equivalent.

Changes since "Python3 runqemu: remove use of subprocess.run()" v2:
* Fix typo in exception handler log message.
* Add extra fix to support testimage

Changes since "Python3 runqemu: remove use of subprocess.run()" v1:
* Fix printing of error message in exception handler

The following changes since commit 3df3462d81216d8b26a87d9915abd9de1b2c2faa:

  bitbake: cooker: record events on cooker exit (2016-09-06 23:18:06 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib joshuagl/runqemu
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=joshuagl/runqemu

Joshua Lock (2):
  runqemu: remove use of subprocess.run()
  runqemu: fixes for when invoked during a bitbake run

 scripts/runqemu | 37 +++++++++++++++++++++++++++----------
 1 file changed, 27 insertions(+), 10 deletions(-)

-- 
2.7.4

[PATCH 1/2] runqemu: remove use of subprocess.run()

[Joshua Lock]

We aim to support Python 3.4+ whereas subprocess.run() was added
in Python 3.5.
Replace subprocess.run() with subprocess.check_output().

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 0a56c60..cbc5cc6 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -858,10 +858,11 @@ class BaseConfig(object):
             cmd = 'bitbake -e'
 
         logger.info('Running %s...' % cmd)
-        proc = subprocess.run(cmd, shell=True, stdout=subprocess.PIPE)
-        if proc.returncode != 0:
-            logger.warn("Couldn't run 'bitbake -e' to gather environment information")
-        self.bitbake_e = proc.stdout.decode('utf-8')
+        try:
+            self.bitbake_e = subprocess.check_output(cmd, shell=True).decode('utf-8')
+        except subprocess.CalledProcessError as err:
+            self.bitbake_e = ''
+            logger.warn("Couldn't run 'bitbake -e' to gather environment information:\n%s" % err.output.decode('utf-8'))
 
 def main():
     if len(sys.argv) == 1 or "help" in sys.argv:
-- 
2.7.4

[PATCH 2/2] runqemu: fixes for when invoked during a bitbake run

[Joshua Lock]

When runqemu is invoked from a running bitbake instance it will be
unable to call `bitbake -e` due to the lock held by the calling
bitbake instance.

Our test code sets an OE_TMPDIR environment variable from which we
can infer/guess paths. Add code to do so when self.bitbake_e can't
be set, much as the sh version of runqemu did.

[YOCTO #10240]

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 28 ++++++++++++++++++++++------
 1 file changed, 22 insertions(+), 6 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index cbc5cc6..7919564 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -570,12 +570,28 @@ class BaseConfig(object):
         if not havenative:
             if not self.bitbake_e:
                 self.load_bitbake_env()
-            native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE']
-            for nv in native_vars:
-                s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
-                if s and s.group(1) != self.get(nv):
-                    logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
-                    self.set(nv, s.group(1))
+
+            if self.bitbake_e:
+                native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE']
+                for nv in native_vars:
+                    s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
+                    if s and s.group(1) != self.get(nv):
+                        logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
+                        self.set(nv, s.group(1))
+            else:
+                # when we're invoked from a running bitbake instance we won't
+                # be able to call `bitbake -e` but should have OE_TMPDIR set in
+                # the environment and can guess paths based on it
+                tmpdir = os.environ.get('OE_TMPDIR', None)
+                if tmpdir:
+                    logger.info('Setting STAGING_DIR_NATIVE and STAGING_BINDIR_NATIVE relative to OE_TMPDIR (%s)' % tmpdir)
+                    hostos, _, _, _, machine = os.uname()
+                    buildsys = '%s-%s' % (machine, hostos.lower())
+                    staging_dir_native = '%s/sysroots/%s' % (tmpdir, buildsys)
+                    self.set('STAGING_DIR_NATIVE', staging_dir_native)
+                    # we have to assume that STAGING_BINDIR_NATIVE is at usr/bin
+                    staging_bindir_native = '%s/usr/bin' % staging_dir_native
+                    self.set('STAGING_BINDIR_NATIVE', staging_bindir_native)
 
     def print_config(self):
         logger.info('Continuing with the following parameters:\n')
-- 
2.7.4