[PATCH 0/6] runqemu fixes

[PATCH 0/6] runqemu fixes

[Joshua Lock]

This series includes several fixes and enhancements for runqemu to better align
with the previous shell based implementation.

The following changes since commit 059d475b6bce1e5414170a4fe2e7989f6b0eacd6:

  oeqa/utils/decorators: LogResults fix race condition in linkfile (2016-09-15 12:14:40 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib joshuagl/runqemu
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=joshuagl/runqemu

Joshua Lock (6):
  runqemu: add guidance to resolve issues with missing files
  qemuboot: write the full kernel filename, not the link name
  runqemu: clarify an INFO message
  qemuboot: also write the kernel link name to the conf file
  runqemu: try symlinks when kernel or rootfs can't be found
  runqemu: work even if a *.qemuboot.conf isn't found

 meta/classes/qemuboot.bbclass | 13 +++++++++++--
 scripts/runqemu               | 43 ++++++++++++++++++++++++++++++++++++-------
 2 files changed, 47 insertions(+), 9 deletions(-)

-- 
2.7.4

[PATCH 1/6] runqemu: add guidance to resolve issues with missing files

[Joshua Lock]

When a required binary cannot be found print some guidance pointing
to using a sourced OE build environment or a qemuboot.conf file,
based on a similar message from the previous shell-based runqemu.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index b6bc0ba..c71a47c 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -28,6 +28,16 @@ import shutil
 import glob
 import configparser
 
+class OEPathError(Exception):
+    """Custom Exception to give better guidance on missing binaries"""
+    def __init__(self, message):
+        self.message = "In order for this script to dynamically infer paths\n \
+kernels or filesystem images, you either need bitbake in your PATH\n \
+or to source oe-init-build-env before running this script.\n\n \
+Dynamic path inference can be avoided by passing a *.qemuboot.conf to\n \
+runqemu, i.e. `runqemu /path/to/my-image-name.qemuboot.conf`\n\n %s" % message
+
+
 def create_logger():
     logger = logging.getLogger('runqemu')
     logger.setLevel(logging.INFO)
@@ -537,7 +547,7 @@ class BaseConfig(object):
             elif os.getenv('DEPLOY_DIR_IMAGE'):
                 deploy_dir_image = os.getenv('DEPLOY_DIR_IMAGE')
             else:
-                raise Exception("DEPLOY_DIR_IMAGE is NULL!")
+                raise OEPathError("DEPLOY_DIR_IMAGE is NULL!")
 
             if self.rootfs and not os.path.exists(self.rootfs):
                 # Lazy rootfs
@@ -691,7 +701,7 @@ class BaseConfig(object):
         lockdir = "/tmp/qemu-tap-locks"
 
         if not (self.qemuifup and self.qemuifdown and ip):
-            raise Exception("runqemu-ifup, runqemu-ifdown or ip not found")
+            raise OEPathError("runqemu-ifup, runqemu-ifdown or ip not found")
 
         if not os.path.exists(lockdir):
             # There might be a race issue when multi runqemu processess are
@@ -808,7 +818,7 @@ class BaseConfig(object):
 
         qemu_bin = '%s/%s' % (self.get('STAGING_BINDIR_NATIVE'), qemu_system)
         if not os.access(qemu_bin, os.X_OK):
-            raise Exception("No QEMU binary '%s' could be found" % qemu_bin)
+            raise OEPathError("No QEMU binary '%s' could be found" % qemu_bin)
 
         check_libgl(qemu_bin)
 
@@ -923,6 +933,9 @@ def main():
 if __name__ == "__main__":
     try:
         ret = main()
+    except OEPathError as err:
+        ret = 1
+        logger.error(err.message)
     except Exception as esc:
         ret = 1
         import traceback
-- 
2.7.4

[PATCH 2/6] qemuboot: write the full kernel filename, not the link name

[Joshua Lock]

KERNEL_IMAGETYPE gives the filename of a symlink to the kernel,
which may not be available i.e. if the user downloads some build
artefacts to run on a local machine. It's also possible that the
link will point to a newer kernel than was intended for use with
the rootfs in the qemuboot.conf.

It's much more reliable to read the name of the file
KERNEL_IMAGETYPE is linking to and assign the full filename to
QB_DEFAULT_KERNEL.

[YOCTO #10285]

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/classes/qemuboot.bbclass | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/classes/qemuboot.bbclass b/meta/classes/qemuboot.bbclass
index 8500c73..0892db3 100644
--- a/meta/classes/qemuboot.bbclass
+++ b/meta/classes/qemuboot.bbclass
@@ -63,6 +63,14 @@ python write_qemuboot_conf() {
     cf.add_section('config_bsp')
     for k in build_vars + qb_vars:
         cf.set('config_bsp', k, '%s' % d.getVar(k, True))
+
+    # QB_DEFAULT_KERNEL's value of KERNEL_IMAGETYPE is the name of a symlink
+    # to the kernel file, which hinders relocatability of the qb conf.
+    # Read the link and replace it with the full filename of the target.
+    kernel_link = os.path.join(d.getVar('DEPLOY_DIR_IMAGE', True), d.getVar('QB_DEFAULT_KERNEL', True))
+    kernel = os.readlink(kernel_link)
+    cf.set('config_bsp', 'QB_DEFAULT_KERNEL', kernel)
+
     with open(qemuboot, 'w') as f:
         cf.write(f)
 
-- 
2.7.4

[PATCH 3/6] runqemu: clarify an INFO message

[Joshua Lock]

Make it clearer that we are looking for a file which ends with
qemuboot.conf

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index c71a47c..6aaae44 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -205,7 +205,7 @@ class BaseConfig(object):
     def is_deploy_dir_image(self, p):
         if os.path.isdir(p):
             if not re.search('.qemuboot.conf$', '\n'.join(os.listdir(p)), re.M):
-                logger.info("Can't find required qemuboot.conf in %s" % p)
+                logger.info("Can't find required *.qemuboot.conf in %s" % p)
                 return False
             if not re.search('-image-', '\n'.join(os.listdir(p))):
                 logger.info("Can't find *-image-* in %s" % p)
-- 
2.7.4

[PATCH 4/6] qemuboot: also write the kernel link name to the conf file

[Joshua Lock]

This will allow runqemu to fall back to trying the link name when
a file matching the full name can't be found.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/classes/qemuboot.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/classes/qemuboot.bbclass b/meta/classes/qemuboot.bbclass
index 0892db3..802eb59 100644
--- a/meta/classes/qemuboot.bbclass
+++ b/meta/classes/qemuboot.bbclass
@@ -48,8 +48,9 @@ python write_qemuboot_conf() {
     import configparser
 
     build_vars = ['MACHINE', 'TUNE_ARCH', 'DEPLOY_DIR_IMAGE', \
-                'IMAGE_NAME', 'IMAGE_LINK_NAME', 'STAGING_DIR_NATIVE', \
-                'STAGING_BINDIR_NATIVE', 'STAGING_DIR_HOST']
+                'KERNEL_IMAGETYPE', 'IMAGE_NAME', 'IMAGE_LINK_NAME', \
+                'STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE', \
+                'STAGING_DIR_HOST']
 
     # Vars from bsp
     qb_vars = []
-- 
2.7.4

[PATCH 5/6] runqemu: try symlinks when kernel or rootfs can't be found

[Joshua Lock]

If the kernel or rootfs names written to the qemuboot.conf can't
be found, try and find the symlinked variant of the filename.

This will help usability of runqemu, for example where a user
downloads an image and associated files as the symlinked names
yet the qemuboot.conf variables point to the full, non-linked,
file names.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 6aaae44..38f9b30 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -451,7 +451,12 @@ class BaseConfig(object):
             if all_files:
                 self.rootfs = all_files[0]
             else:
-                raise Exception("Failed to find rootfs: %s" % cmd)
+                cmd = '%s/%s*.%s' % (self.get('DEPLOY_DIR_IMAGE'), self.get('IMAGE_LINK_NAME'), self.fstype)
+                all_files = glob.glob(cmd)
+                if all_files:
+                    self.rootfs = all_files[0]
+                else:
+                    raise Exception("Failed to find rootfs: %s" % cmd)
 
         if not os.path.exists(self.rootfs):
             raise Exception("Can't find rootfs: %s" % self.rootfs)
@@ -462,13 +467,18 @@ class BaseConfig(object):
         if self.fstype in self.vmtypes:
             return
         kernel = self.kernel
+        deploy_dir_image = self.get('DEPLOY_DIR_IMAGE')
         if not kernel:
-            kernel = "%s/%s" % (self.get('DEPLOY_DIR_IMAGE'), self.get('QB_DEFAULT_KERNEL'))
+            kernel = "%s/%s" % (deploy_dir_image, self.get('QB_DEFAULT_KERNEL'))
 
         if os.path.exists(kernel):
             self.kernel = kernel
         else:
-            raise Exception("KERNEL %s not found" % kernel)
+            kernel = "%s/%s" % (deploy_dir_image, self.get('KERNEL_IMAGETYPE'))
+            if kernel != deploy_dir_image and os.path.exists(kernel):
+                self.kernel = kernel
+            else:
+                raise Exception("KERNEL %s not found" % kernel)
 
         dtb = self.get('QB_DTB')
         if dtb:
-- 
2.7.4

[PATCH 6/6] runqemu: work even if a *.qemuboot.conf isn't found

[Joshua Lock]

A qemuboot conf file is a convenience but it should still be
possible to invoke runqemu without them, especially for examples
such as using the SDK with an extracted rootfs via NFS.

As read_qemuboot() is always called we need to be sure that function
can return cleanly, without throwing Exceptions, even if a qemuboot
conf file isn't found.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 scripts/runqemu | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 38f9b30..10122af 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -557,7 +557,8 @@ class BaseConfig(object):
             elif os.getenv('DEPLOY_DIR_IMAGE'):
                 deploy_dir_image = os.getenv('DEPLOY_DIR_IMAGE')
             else:
-                raise OEPathError("DEPLOY_DIR_IMAGE is NULL!")
+                logger.info("Can't find qemuboot conf file, DEPLOY_DIR_IMAGE is NULL!")
+                return
 
             if self.rootfs and not os.path.exists(self.rootfs):
                 # Lazy rootfs
@@ -574,6 +575,11 @@ class BaseConfig(object):
                     self.qemuboot = qbs.split()[0]
                     self.qbconfload = True
 
+        if not self.qemuboot:
+            # If we haven't found a .qemuboot.conf at this point it probably
+            # doesn't exist, continue without
+            return
+
         if not os.path.exists(self.qemuboot):
             raise Exception("Failed to find <image>.qemuboot.conf!")
 
-- 
2.7.4

Re: [PATCH 5/6] runqemu: try symlinks when kernel or rootfs can't be found

[Robert Yang]

On 09/16/2016 08:52 PM, Joshua Lock wrote:
> If the kernel or rootfs names written to the qemuboot.conf can't
> be found, try and find the symlinked variant of the filename.
>
> This will help usability of runqemu, for example where a user
> downloads an image and associated files as the symlinked names
> yet the qemuboot.conf variables point to the full, non-linked,
> file names.
>
> Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
> ---
>  scripts/runqemu | 16 +++++++++++++---
>  1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/runqemu b/scripts/runqemu
> index 6aaae44..38f9b30 100755
> --- a/scripts/runqemu
> +++ b/scripts/runqemu
> @@ -451,7 +451,12 @@ class BaseConfig(object):
>              if all_files:
>                  self.rootfs = all_files[0]
>              else:
> -                raise Exception("Failed to find rootfs: %s" % cmd)
> +                cmd = '%s/%s*.%s' % (self.get('DEPLOY_DIR_IMAGE'), self.get('IMAGE_LINK_NAME'), self.fstype)
> +                all_files = glob.glob(cmd)
> +                if all_files:
> +                    self.rootfs = all_files[0]
> +                else:
> +                    raise Exception("Failed to find rootfs: %s" % cmd)
>
>          if not os.path.exists(self.rootfs):
>              raise Exception("Can't find rootfs: %s" % self.rootfs)
> @@ -462,13 +467,18 @@ class BaseConfig(object):
>          if self.fstype in self.vmtypes:
>              return
>          kernel = self.kernel
> +        deploy_dir_image = self.get('DEPLOY_DIR_IMAGE')
>          if not kernel:
> -            kernel = "%s/%s" % (self.get('DEPLOY_DIR_IMAGE'), self.get('QB_DEFAULT_KERNEL'))
> +            kernel = "%s/%s" % (deploy_dir_image, self.get('QB_DEFAULT_KERNEL'))
>
>          if os.path.exists(kernel):
>              self.kernel = kernel
>          else:
> -            raise Exception("KERNEL %s not found" % kernel)
> +            kernel = "%s/%s" % (deploy_dir_image, self.get('KERNEL_IMAGETYPE'))
> +            if kernel != deploy_dir_image and os.path.exists(kernel):


Thank you very much for fixing this, can we use KERNEL_IMAGETYPE* here, please ?
Please take a look at here:
http://autobuilder.yoctoproject.org/pub/releases/yocto-2.2_M3.rc1/machines/qemu/qemuppc/

Its kernel name might be vmlinux-qemuppc.bin, while KERNEL_IMAGETYPE is vmlinux.

I have local patches conflicted with this, they also fix dtb,
OECORE_NATIVE_SYSROOT and nfs. I will send a V2 including your patches,
and please feel free to comment.

// Robert

> +                self.kernel = kernel
> +            else:
> +                raise Exception("KERNEL %s not found" % kernel)
>
>          dtb = self.get('QB_DTB')
>          if dtb:
>