[PATCH 0/5] runqemu: 5 fixes

[PATCH 0/5] runqemu: 5 fixes

[Robert Yang]

The following changes since commit ef68005a8c527e9b1d05b7769f0ec8ebe9ec3f91:

  webkitgtk: Upgrade to 2.16.5 (2017-07-17 13:49:04 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/runqemu
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/runqemu

Robert Yang (5):
  runqemu: add --debug and --quiet
  runqemu: check qbconfload before running bitbake
  runqemu: check tar.bz2 and .tar.gz
  runqemu: validate combos
  runqemu: chmod 0o777 for lockdir

 scripts/runqemu | 64 ++++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 43 insertions(+), 21 deletions(-)

-- 
2.11.0

[PATCH 1/5] runqemu: add --debug and --quiet

[Robert Yang]

And move some debug info into logger.debug(), this can make it easy to
read key messages like errors or warnings.

I checked meta/lib/oeqa/ they don't depend on these messages. And I have
run "oe-selftest -a", it doesn't break anything.

[YOCTO #10474]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 scripts/runqemu | 50 +++++++++++++++++++++++++++++++-------------------
 1 file changed, 31 insertions(+), 19 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index d44afc7e7a1..7188bd78830 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -48,7 +48,7 @@ def create_logger():
 
     # create console handler and set level to debug
     ch = logging.StreamHandler()
-    ch.setLevel(logging.INFO)
+    ch.setLevel(logging.DEBUG)
 
     # create formatter
     formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
@@ -85,6 +85,8 @@ of the following environment variables (in any order):
   qemuparams=<xyz> - specify custom parameters to QEMU
   bootparams=<xyz> - specify custom kernel parameters during boot
   help, -h, --help: print this text
+  -d, --debug: Enable debug output
+  -q, --quite: Hide most output except error messages
 
 Examples:
   runqemu
@@ -112,7 +114,7 @@ def check_tun():
 
 def check_libgl(qemu_bin):
     cmd = 'ldd %s' % qemu_bin
-    logger.info('Running %s...' % cmd)
+    logger.debug('Running %s...' % cmd)
     need_gl = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
     if re.search('libGLU', need_gl):
         # We can't run without a libGL.so
@@ -229,12 +231,12 @@ class BaseConfig(object):
         self.mac_slirp = "52:54:00:12:35:"
 
     def acquire_lock(self):
-        logger.info("Acquiring lockfile %s..." % self.lock)
+        logger.debug("Acquiring lockfile %s..." % self.lock)
         try:
             self.lock_descriptor = open(self.lock, 'w')
             fcntl.flock(self.lock_descriptor, fcntl.LOCK_EX|fcntl.LOCK_NB)
         except Exception as e:
-            logger.info("Acquiring lockfile %s failed: %s" % (self.lock, e))
+            logger.error("Acquiring lockfile %s failed: %s" % (self.lock, e))
             if self.lock_descriptor:
                 self.lock_descriptor.close()
             return False
@@ -259,10 +261,10 @@ class BaseConfig(object):
     def is_deploy_dir_image(self, p):
         if os.path.isdir(p):
             if not re.search('.qemuboot.conf$', '\n'.join(os.listdir(p)), re.M):
-                logger.info("Can't find required *.qemuboot.conf in %s" % p)
+                logger.debug("Can't find required *.qemuboot.conf in %s" % p)
                 return False
             if not any(map(lambda name: '-image-' in name, os.listdir(p))):
-                logger.info("Can't find *-image-* in %s" % p)
+                logger.debug("Can't find *-image-* in %s" % p)
                 return False
             return True
         else:
@@ -281,9 +283,9 @@ class BaseConfig(object):
 
     def set_machine_deploy_dir(self, machine, deploy_dir_image):
         """Set MACHINE and DEPLOY_DIR_IMAGE"""
-        logger.info('MACHINE: %s' % machine)
+        logger.debug('MACHINE: %s' % machine)
         self.set("MACHINE", machine)
-        logger.info('DEPLOY_DIR_IMAGE: %s' % deploy_dir_image)
+        logger.debug('DEPLOY_DIR_IMAGE: %s' % deploy_dir_image)
         self.set("DEPLOY_DIR_IMAGE", deploy_dir_image)
 
     def check_arg_nfs(self, p):
@@ -337,10 +339,10 @@ class BaseConfig(object):
 
         elif os.path.isdir(p) or re.search(':', p) and re.search('/', p):
             if self.is_deploy_dir_image(p):
-                logger.info('DEPLOY_DIR_IMAGE: %s' % p)
+                logger.debug('DEPLOY_DIR_IMAGE: %s' % p)
                 self.set("DEPLOY_DIR_IMAGE", p)
             else:
-                logger.info("Assuming %s is an nfs rootfs" % p)
+                logger.debug("Assuming %s is an nfs rootfs" % p)
                 self.check_arg_nfs(p)
         elif os.path.basename(p).startswith('ovmf'):
             self.ovmf_bios.append(p)
@@ -356,7 +358,7 @@ class BaseConfig(object):
         elif re.search('/', arg):
             raise RunQemuError("Unknown arg: %s" % arg)
 
-        logger.info('Assuming MACHINE = %s' % arg)
+        logger.debug('Assuming MACHINE = %s' % arg)
 
         # if we're running under testimage, or similarly as a child
         # of an existing bitbake invocation, we can't invoke bitbake
@@ -393,6 +395,16 @@ class BaseConfig(object):
             self.set("MACHINE", arg)
 
     def check_args(self):
+        for debug in ("-d", "--debug"):
+            if debug in sys.argv:
+                logger.setLevel(logging.DEBUG)
+                sys.argv.remove(debug)
+
+        for quiet in ("-q", "--quiet"):
+            if quiet in sys.argv:
+                logger.setLevel(logging.ERROR)
+                sys.argv.remove(quiet)
+
         unknown_arg = ""
         for arg in sys.argv[1:]:
             if arg in self.fstypes + self.vmtypes:
@@ -608,7 +620,7 @@ class BaseConfig(object):
                 break
 
         if biosdir:
-            logger.info("Assuming biosdir is: %s" % biosdir)
+            logger.debug("Assuming biosdir is: %s" % biosdir)
             self.qemu_opt_script += ' -L %s' % biosdir
         else:
             logger.error("Custom BIOS directory not found. Tried: %s, %s, and %s" % (self.custombiosdir, biosdir_native, biosdir_host))
@@ -663,7 +675,7 @@ class BaseConfig(object):
             if self.get('DEPLOY_DIR_IMAGE'):
                 deploy_dir_image = self.get('DEPLOY_DIR_IMAGE')
             else:
-                logger.info("Can't find qemuboot conf file, DEPLOY_DIR_IMAGE is NULL!")
+                logger.warn("Can't find qemuboot conf file, DEPLOY_DIR_IMAGE is NULL!")
                 return
 
             if self.rootfs and not os.path.exists(self.rootfs):
@@ -675,7 +687,7 @@ class BaseConfig(object):
                         self.rootfs, machine)
             else:
                 cmd = 'ls -t %s/*.qemuboot.conf' %  deploy_dir_image
-                logger.info('Running %s...' % cmd)
+                logger.debug('Running %s...' % cmd)
                 try:
                     qbs = subprocess.check_output(cmd, shell=True).decode('utf-8')
                 except subprocess.CalledProcessError as err:
@@ -700,7 +712,7 @@ class BaseConfig(object):
         if not os.path.exists(self.qemuboot):
             raise RunQemuError("Failed to find %s (wrong image name or BSP does not support running under qemu?)." % self.qemuboot)
 
-        logger.info('CONFFILE: %s' % self.qemuboot)
+        logger.debug('CONFFILE: %s' % self.qemuboot)
 
         cf = configparser.ConfigParser()
         cf.read(self.qemuboot)
@@ -899,7 +911,7 @@ class BaseConfig(object):
                 pass
 
         cmd = '%s link' % ip
-        logger.info('Running %s...' % cmd)
+        logger.debug('Running %s...' % cmd)
         ip_link = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
         # Matches line like: 6: tap0: <foo>
         possibles = re.findall('^[1-9]+: +(tap[0-9]+): <.*', ip_link, re.M)
@@ -931,7 +943,7 @@ class BaseConfig(object):
             self.lock = lockfile + '.lock'
             self.acquire_lock()
             self.cleantap = True
-            logger.info('Created tap: %s' % tap)
+            logger.debug('Created tap: %s' % tap)
 
         if not tap:
             logger.error("Failed to setup tap device. Run runqemu-gen-tapdevs to manually create.")
@@ -1150,7 +1162,7 @@ class BaseConfig(object):
     def cleanup(self):
         if self.cleantap:
             cmd = 'sudo %s %s %s' % (self.qemuifdown, self.tap, self.bindir_native)
-            logger.info('Running %s' % cmd)
+            logger.debug('Running %s' % cmd)
             subprocess.check_call(cmd, shell=True)
         if self.lock_descriptor:
             logger.info("Releasing lockfile for tap device '%s'" % self.tap)
@@ -1159,7 +1171,7 @@ class BaseConfig(object):
         if self.nfs_running:
             logger.info("Shutting down the userspace NFS server...")
             cmd = "runqemu-export-rootfs stop %s" % self.rootfs
-            logger.info('Running %s' % cmd)
+            logger.debug('Running %s' % cmd)
             subprocess.check_call(cmd, shell=True)
 
         if self.saved_stty:
-- 
2.11.0

[PATCH 2/5] runqemu: check qbconfload before running bitbake

[Robert Yang]

If qbconfload (.qemuboot.conf is found) is present, we can get
DEPLOY_DIR_IMAGE from it rather than "bitbake -e".

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 scripts/runqemu | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 7188bd78830..781814ae651 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -466,7 +466,7 @@ class BaseConfig(object):
 
             self.check_arg_machine(unknown_arg)
 
-        if not self.get('DEPLOY_DIR_IMAGE'):
+        if not (self.get('DEPLOY_DIR_IMAGE') or self.qbconfload):
             self.load_bitbake_env()
             s = re.search('^DEPLOY_DIR_IMAGE="(.*)"', self.bitbake_e, re.M)
             if s:
-- 
2.11.0

[PATCH 3/5] runqemu: check tar.bz2 and .tar.gz

[Robert Yang]

Handle them as nfs, so that cmd like the following can be boot:
$ runqemu tmp/deploy/images/qemux86/core-image-minimal-qemux86.tar.bz2

[YOCTO #11286]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 scripts/runqemu | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 781814ae651..bdb559f82ff 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -218,7 +218,8 @@ class BaseConfig(object):
         self.lock_descriptor = ''
         self.bitbake_e = ''
         self.snapshot = False
-        self.fstypes = ('ext2', 'ext3', 'ext4', 'jffs2', 'nfs', 'btrfs', 'cpio.gz', 'cpio', 'ramfs')
+        self.fstypes = ('ext2', 'ext3', 'ext4', 'jffs2', 'nfs', 'btrfs',
+                        'cpio.gz', 'cpio', 'ramfs', 'tar.bz2', 'tar.gz')
         self.vmtypes = ('hddimg', 'hdddirect', 'wic', 'vmdk', 'qcow2', 'vdi', 'iso')
         self.network_device = "-device e1000,netdev=net0,mac=@MAC@"
         # Use different mac section for tap and slirp to avoid
@@ -277,6 +278,8 @@ class BaseConfig(object):
         if not self.fstype or self.fstype == fst:
             if fst == 'ramfs':
                 fst = 'cpio.gz'
+            if fst in ('tar.bz2', 'tar.gz'):
+                fst = 'nfs'
             self.fstype = fst
         else:
             raise RunQemuError("Conflicting: FSTYPE %s and %s" % (self.fstype, fst))
-- 
2.11.0

[PATCH 4/5] runqemu: validate combos

[Robert Yang]

Error out ealier if the combos is invalid, e.g.:
$ runqemu tmp/deploy/images/qemux86/bzImage-qemux86.bin tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic

This will fail at kernel panic, no we check and error out early. We can
add other checkings in the future.

[YOCTO #11286]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 scripts/runqemu | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/runqemu b/scripts/runqemu
index bdb559f82ff..e9ed890bb27 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -1209,6 +1209,10 @@ class BaseConfig(object):
             self.bitbake_e = ''
             logger.warn("Couldn't run 'bitbake -e' to gather environment information:\n%s" % err.output.decode('utf-8'))
 
+    def validate_combos(self):
+        if (self.fstype in self.vmtypes) and self.kernel:
+            raise Exception("%s doesn't need kernel %s!" % (self.fstype, self.kernel))
+
     @property
     def bindir_native(self):
         result = self.get('STAGING_BINDIR_NATIVE')
@@ -1240,6 +1244,8 @@ def main():
         config.check_args()
         config.read_qemuboot()
         config.check_and_set()
+        # Check whether the combos is valid or not
+        config.validate_combos()
         config.print_config()
         config.setup_network()
         config.setup_rootfs()
-- 
2.11.0

Re: [PATCH 4/5] runqemu: validate combos

[Robert Yang]

On 07/18/2017 11:21 AM, Robert Yang wrote:
> Error out ealier if the combos is invalid, e.g.:
> $ runqemu tmp/deploy/images/qemux86/bzImage-qemux86.bin tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
>
> This will fail at kernel panic, no we check and error out early. We can
> add other checkings in the future.
>
> [YOCTO #11286]
>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  scripts/runqemu | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/scripts/runqemu b/scripts/runqemu
> index bdb559f82ff..e9ed890bb27 100755
> --- a/scripts/runqemu
> +++ b/scripts/runqemu
> @@ -1209,6 +1209,10 @@ class BaseConfig(object):
>              self.bitbake_e = ''
>              logger.warn("Couldn't run 'bitbake -e' to gather environment information:\n%s" % err.output.decode('utf-8'))
>
> +    def validate_combos(self):
> +        if (self.fstype in self.vmtypes) and self.kernel:
> +            raise Exception("%s doesn't need kernel %s!" % (self.fstype, self.kernel))

I just found that here should be raise RunQemuError(), updated it in the repo.

// Robert

> +
>      @property
>      def bindir_native(self):
>          result = self.get('STAGING_BINDIR_NATIVE')
> @@ -1240,6 +1244,8 @@ def main():
>          config.check_args()
>          config.read_qemuboot()
>          config.check_and_set()
> +        # Check whether the combos is valid or not
> +        config.validate_combos()
>          config.print_config()
>          config.setup_network()
>          config.setup_rootfs()
>

[PATCH 5/5] runqemu: chmod 0o777 for lockdir

[Robert Yang]

Multi-users may run qemu on the same host, all of them should be able to
create or remove lock in lockdir, so set lockdir's mode to 0o777.

Note, os.mkdir()'s mode is default to 0o777, but the current umask value is
first masked out, so use os.chmod() to set it.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 scripts/runqemu | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/runqemu b/scripts/runqemu
index e9ed890bb27..2be27804841 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -910,6 +910,7 @@ class BaseConfig(object):
             # running at the same time.
             try:
                 os.mkdir(lockdir)
+                os.chmod(lockdir, 0o777)
             except FileExistsError:
                 pass
 
-- 
2.11.0

Re: [PATCH 0/5] runqemu: 5 fixes

[Robert Yang]

Ping.

// Robert

On 07/18/2017 11:21 AM, Robert Yang wrote:
> The following changes since commit ef68005a8c527e9b1d05b7769f0ec8ebe9ec3f91:
>
>   webkitgtk: Upgrade to 2.16.5 (2017-07-17 13:49:04 +0100)
>
> are available in the git repository at:
>
>   git://git.openembedded.org/openembedded-core-contrib rbt/runqemu
>   http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/runqemu
>
> Robert Yang (5):
>   runqemu: add --debug and --quiet
>   runqemu: check qbconfload before running bitbake
>   runqemu: check tar.bz2 and .tar.gz
>   runqemu: validate combos
>   runqemu: chmod 0o777 for lockdir
>
>  scripts/runqemu | 64 ++++++++++++++++++++++++++++++++++++++-------------------
>  1 file changed, 43 insertions(+), 21 deletions(-)
>