[PATCH v2 0/3] Python3 runqemu

[PATCH v2 0/3] Python3 runqemu

[Joshua Lock]

This small series against master-next (as the python3 runqemu hasn't made it to
master yet) fixes the new runqemu for a workflow of mine where I build on a
headless machine and copy images locally to test them with runqemu.

Changes since v1:
* drop a patch to replace an ls -t call with Python's glob()
* Fix and enhance reading STAGING_*_NATIVE from the bitbake environment, tested
  with artefacts in a completely different path to the build path and invoking 
  with both the qemuboot.conf file as the arg to runqemu and the parent 
  directory.

The following changes since commit 5b3ac3cd92d1f021c06b22eaa0e74ab8dfb22764:

  kernel-module-split.bbclass: no need for running depmod (2016-09-05 17:45:53 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib joshuagl/runqemu
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=joshuagl/runqemu

Joshua Lock (3):
  runqemu: fix typos
  runqemu: assume artefacts are relative to *.qemuboot.conf
  runqemu: better handle running on a host with different paths

 scripts/runqemu | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 54 insertions(+), 3 deletions(-)

-- 
2.7.4

[PATCH v2 1/3] runqemu: fix typos

[Joshua Lock]

Remove a stray whitespace when accessing a member variable and fix a
spelling mistake in an Exception message.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index a94cc65..72c6352 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -160,7 +160,7 @@ class BaseConfig(object):
         self.custombiosdir = ''
         self.lock = ''
         self.lock_descriptor = ''
-        self. bitbake_e = ''
+        self.bitbake_e = ''
         self.fstypes = ('ext2', 'ext3', 'ext4', 'jffs2', 'nfs', 'btrfs', 'cpio.gz', 'cpio', 'ramfs')
         self.vmtypes = ('hddimg', 'hdddirect', 'wic', 'vmdk', 'qcow2', 'vdi', 'iso')
 
@@ -289,7 +289,7 @@ class BaseConfig(object):
             self.set_machine_deploy_dir(arg, deploy_dir_image)
         else:
             logger.error("%s not a directory valid DEPLOY_DIR_IMAGE" % deploy_dir_image)
-            raise Exception("Falied to set MACHINE to %s. Unknown arg: %s" % (arg, arg))
+            raise Exception("Failed to set MACHINE to %s. Unknown arg: %s" % (arg, arg))
 
     def check_args(self):
         unknown_arg = ""
-- 
2.7.4

[PATCH v2 2/3] runqemu: assume artefacts are relative to *.qemuboot.conf

[Joshua Lock]

When runqemu is started with a *.qemuboot.conf arg assume that image
artefacts are relative to that file, rather than in whatever
directory the DEPLOY_DIR_IMAGE variable in the conf file points to.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 72c6352..6ad0ec0 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -143,6 +143,7 @@ class BaseConfig(object):
         self.nfs_server = ''
         self.rootfs = ''
         self.qemuboot = ''
+        self.qbconfload = False
         self.kernel = ''
         self.kernel_cmdline = ''
         self.kernel_cmdline_script = ''
@@ -239,6 +240,7 @@ class BaseConfig(object):
         """
         if p.endswith('.qemuboot.conf'):
             self.qemuboot = p
+            self.qbconfload = True
         elif re.search('\.bin$', p) or re.search('bzImage', p) or \
              re.search('zImage', p) or re.search('vmlinux', p) or \
              re.search('fitImage', p) or re.search('uImage', p):
@@ -250,7 +252,8 @@ class BaseConfig(object):
             if m:
                 qb = '%s%s' % (re.sub('\.rootfs$', '', m.group(1)), '.qemuboot.conf')
                 if os.path.exists(qb):
-                    self.qemuboot = qb 
+                    self.qemuboot = qb
+                    self.qbconfload = True
                 else:
                     logger.warn("%s doesn't exist" % qb)
                 fst = m.group(2)
@@ -533,6 +536,7 @@ class BaseConfig(object):
                 qbs = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
                 if qbs:
                     self.qemuboot = qbs.split()[0]
+                    self.qbconfload = True
 
         if not os.path.exists(self.qemuboot):
             raise Exception("Failed to find <image>.qemuboot.conf!")
@@ -545,6 +549,15 @@ class BaseConfig(object):
             k_upper = k.upper()
             self.set(k_upper, v)
 
+        # When we're started with a *.qemuboot.conf arg assume that image
+        # artefacts are relative to that file, rather than in whatever
+        # directory DEPLOY_DIR_IMAGE in the conf file points to.
+        if self.qbconfload:
+            imgdir = os.path.dirname(self.qemuboot)
+            if imgdir != self.get('DEPLOY_DIR_IMAGE'):
+                logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qemuboot, imgdir))
+                self.set('DEPLOY_DIR_IMAGE', imgdir)
+
     def print_config(self):
         logger.info('Continuing with the following parameters:\n')
         if not self.fstype in self.vmtypes:
-- 
2.7.4

[PATCH v2 3/3] runqemu: better handle running on a host with different paths

[Joshua Lock]

If the STAGING_*_NATIVE directories from the config file don't exist
and we're in a sourced OE build directory try to extract the paths
from `bitbake -e`

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/scripts/runqemu b/scripts/runqemu
index 6ad0ec0..558cbc2 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -558,6 +558,22 @@ class BaseConfig(object):
                 logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qemuboot, imgdir))
                 self.set('DEPLOY_DIR_IMAGE', imgdir)
 
+        # If the STAGING_*_NATIVE directories from the config file don't exist
+        # and we're in a sourced OE build directory try to extract the paths
+        # from `bitbake -e`
+        havenative = os.path.exists(self.get('STAGING_DIR_NATIVE')) and \
+            os.path.exists(self.get('STAGING_BINDIR_NATIVE'))
+
+        if not havenative:
+            if not self.bitbake_e:
+                self.load_bitbake_env()
+            native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE']
+            for nv in native_vars:
+                s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
+                if s and s.group(1) != self.get(nv):
+                    logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
+                    self.set(nv, s.group(1))
+
     def print_config(self):
         logger.info('Continuing with the following parameters:\n')
         if not self.fstype in self.vmtypes:
@@ -818,6 +834,28 @@ class BaseConfig(object):
             shutil.rmtree(self.nfs_dir)
             shutil.rmtree('%s.pseudo_state' % self.nfs_dir)
 
+    def load_bitbake_env(self, mach=None):
+        if self.bitbake_e:
+            return
+
+        bitbake = shutil.which('bitbake')
+        if not bitbake:
+            return
+
+        if not mach:
+            mach = self.get('MACHINE')
+
+        if mach:
+            cmd = 'MACHINE=%s bitbake -e' % mach
+        else:
+            cmd = 'bitbake -e'
+
+        logger.info('Running %s...' % cmd)
+        proc = subprocess.run(cmd, shell=True, stdout=subprocess.PIPE)
+        if proc.returncode != 0:
+            logger.warn("Couldn't run 'bitbake -e' to gather environment information")
+        self.bitbake_e = proc.stdout.decode('utf-8')
+
 def main():
     if len(sys.argv) == 1 or "help" in sys.argv:
         print_usage()
-- 
2.7.4

Re: [PATCH v2 0/3] Python3 runqemu

[Robert Yang]

Thanks, I did some rough testing, they worked well.

// Robert

On 09/06/2016 04:32 AM, Joshua Lock wrote:
> This small series against master-next (as the python3 runqemu hasn't made it to
> master yet) fixes the new runqemu for a workflow of mine where I build on a
> headless machine and copy images locally to test them with runqemu.
>
> Changes since v1:
> * drop a patch to replace an ls -t call with Python's glob()
> * Fix and enhance reading STAGING_*_NATIVE from the bitbake environment, tested
>   with artefacts in a completely different path to the build path and invoking
>   with both the qemuboot.conf file as the arg to runqemu and the parent
>   directory.
>
> The following changes since commit 5b3ac3cd92d1f021c06b22eaa0e74ab8dfb22764:
>
>   kernel-module-split.bbclass: no need for running depmod (2016-09-05 17:45:53 +0100)
>
> are available in the git repository at:
>
>   git://git.openembedded.org/openembedded-core-contrib joshuagl/runqemu
>   http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=joshuagl/runqemu
>
> Joshua Lock (3):
>   runqemu: fix typos
>   runqemu: assume artefacts are relative to *.qemuboot.conf
>   runqemu: better handle running on a host with different paths
>
>  scripts/runqemu | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 54 insertions(+), 3 deletions(-)
>