[PATCH 0/5] Make runqemu to work when rm_work is enabled

[PATCH 0/5] Make runqemu to work when rm_work is enabled

[Ed Bartosh]

Hi,

runqemu fails when rm_work is enabled as it can't get qemu-system and tunctl
binaries from the native sysroot as it's removed by rm_work.

This is fixed by disabling removal of the qemu-helper-native sysroot directory
and changing runqemu code to run qemu-system and tunctl from it.

[YOCTO #11266]
[YOCTO #11193]

The following changes since commit 94467d5087ca3b13e7baac670af107550701b333:

  dev-manual, ref-manual: Edits to the "Incrementing Binary Package" section. (2017-04-12 19:35:00 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/oe-core/rm_work-runqemu-ed
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/rm_work-runqemu-ed

Ed Bartosh (4):
  runqemu: get qemu from qemu-helper-native sysroot
  runqemu: add bindir_native property
  runqemu: use bindir_native property to run ifup/down scripts
  qemuboot: write native sysroot of qemu-helper into qemuboot.conf

brian avery (1):
  qemu-helper-native: prepare native sysroot for runqemu

 meta/classes/qemuboot.bbclass                      |  9 ++++++-
 .../qemu/qemu-helper-native_1.0.bb                 |  4 +++
 scripts/runqemu                                    | 30 +++++++++++++++++++---
 scripts/runqemu-ifdown                             |  6 ++---
 scripts/runqemu-ifup                               |  6 ++---
 5 files changed, 44 insertions(+), 11 deletions(-)

--
Regards,
Ed

[PATCH 1/5] qemu-helper-native: prepare native sysroot for runqemu

[Ed Bartosh]

From: brian avery <brian.avery@intel.com>

Make sure that native sysroot contains qemu and tunctl binaries for
runqemu usage:
  - excluded native sysroot from rm_work
  - added qemu-native to DEPENDS to put qemu binaries into native sysroot
  - forced addto_recipe_sysroot task

[YOCTO #11266]
[YOCTO #11193]

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb b/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb
index 8d27c4d..27d5315 100644
--- a/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb
+++ b/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb
@@ -19,3 +19,7 @@ do_install() {
 	install -d ${D}${bindir}
 	install tunctl ${D}${bindir}/
 }
+
+RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
+DEPENDS += "qemu-native"
+addtask addto_recipe_sysroot after do_populate_sysroot before do_build
-- 
2.1.4

[PATCH 2/5] runqemu: get qemu from qemu-helper-native sysroot

[Ed Bartosh]

If rm_work is enabled image native sysroot can be removed.
This makes runqemu to fail trying to find qemu binary.

Used native sysroot of qemu-helper-native to find system qemu
binary.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/runqemu | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 5b5d56b..a8bb9eb 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -1056,7 +1056,18 @@ class BaseConfig(object):
         if not qemu_system:
             raise Exception("Failed to boot, QB_SYSTEM_NAME is NULL!")
 
-        qemu_bin = '%s/%s' % (self.get('STAGING_BINDIR_NATIVE'), qemu_system)
+        cmd = 'bitbake qemu-helper-native -e'
+        logger.info('Running %s...' % cmd)
+        out = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
+        out = out.stdout.read().decode('utf-8')
+
+        match = re.search('^STAGING_BINDIR_NATIVE="(.*)"', out, re.M)
+        if match:
+            bindir_native  = match.group(1)
+        else:
+            raise Exception("Can't find STAGING_BINDIR_NATIVE in '%s' output" % cmd)
+
+        qemu_bin = '%s/%s' % (bindir_native, qemu_system)
 
         # It is possible to have qemu-native in ASSUME_PROVIDED, and it won't
         # find QEMU in sysroot, it needs to use host's qemu.
-- 
2.1.4

[PATCH 3/5] runqemu: add bindir_native property

[Ed Bartosh]

Isolated logic of getting path to native bin directory in
new bindir_native property method.

This property is going to be used to obtain location of
qemu-sytem and tunctl.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/runqemu | 37 ++++++++++++++++++++++++-------------
 1 file changed, 24 insertions(+), 13 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index a8bb9eb..6cdedd8 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -721,7 +721,7 @@ class BaseConfig(object):
                 self.load_bitbake_env()
 
             if self.bitbake_e:
-                native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE']
+                native_vars = ['STAGING_DIR_NATIVE']
                 for nv in native_vars:
                     s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
                     if s and s.group(1) != self.get(nv):
@@ -1056,18 +1056,7 @@ class BaseConfig(object):
         if not qemu_system:
             raise Exception("Failed to boot, QB_SYSTEM_NAME is NULL!")
 
-        cmd = 'bitbake qemu-helper-native -e'
-        logger.info('Running %s...' % cmd)
-        out = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
-        out = out.stdout.read().decode('utf-8')
-
-        match = re.search('^STAGING_BINDIR_NATIVE="(.*)"', out, re.M)
-        if match:
-            bindir_native  = match.group(1)
-        else:
-            raise Exception("Can't find STAGING_BINDIR_NATIVE in '%s' output" % cmd)
-
-        qemu_bin = '%s/%s' % (bindir_native, qemu_system)
+        qemu_bin = '%s/%s' % (self.bindir_native, qemu_system)
 
         # It is possible to have qemu-native in ASSUME_PROVIDED, and it won't
         # find QEMU in sysroot, it needs to use host's qemu.
@@ -1196,6 +1185,28 @@ class BaseConfig(object):
             self.bitbake_e = ''
             logger.warn("Couldn't run 'bitbake -e' to gather environment information:\n%s" % err.output.decode('utf-8'))
 
+    @property
+    def bindir_native(self):
+        result = self.get('STAGING_BINDIR_NATIVE')
+        if result and os.path.exists(result):
+            return result
+
+        cmd = 'bitbake qemu-helper-native -e'
+        logger.info('Running %s...' % cmd)
+        out = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
+        out = out.stdout.read().decode('utf-8')
+
+        match = re.search('^STAGING_BINDIR_NATIVE="(.*)"', out, re.M)
+        if match:
+            result = match.group(1)
+            if os.path.exists(result):
+                self.set('STAGING_BINDIR_NATIVE', result)
+                return result
+            raise Exception("Native sysroot directory %s doesn't exist" % result)
+        else:
+            raise Exception("Can't find STAGING_BINDIR_NATIVE in '%s' output" % cmd)
+
+
 def main():
     if "help" in sys.argv or '-h' in sys.argv or '--help' in sys.argv:
         print_usage()
-- 
2.1.4

[PATCH 4/5] runqemu: use bindir_native property to run ifup/down scripts

[Ed Bartosh]

Used self.bindir_native to point out to the native sysroot
when running runqemu-ifup and runqemu-ifdown scripts.

[YOCTO #11266]
[YOCTO #11193]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/runqemu        | 4 ++--
 scripts/runqemu-ifdown | 6 +++---
 scripts/runqemu-ifup   | 6 +++---
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 6cdedd8..3744c67 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -920,7 +920,7 @@ class BaseConfig(object):
             gid = os.getgid()
             uid = os.getuid()
             logger.info("Setting up tap interface under sudo")
-            cmd = 'sudo %s %s %s %s' % (self.qemuifup, uid, gid, self.get('STAGING_DIR_NATIVE'))
+            cmd = 'sudo %s %s %s %s' % (self.qemuifup, uid, gid, self.bindir_native)
             tap = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8').rstrip('\n')
             lockfile = os.path.join(lockdir, tap)
             self.lock = lockfile + '.lock'
@@ -1140,7 +1140,7 @@ class BaseConfig(object):
 
     def cleanup(self):
         if self.cleantap:
-            cmd = 'sudo %s %s %s' % (self.qemuifdown, self.tap, self.get('STAGING_DIR_NATIVE'))
+            cmd = 'sudo %s %s %s' % (self.qemuifdown, self.tap, self.bindir_native)
             logger.info('Running %s' % cmd)
             subprocess.call(cmd, shell=True)
         if self.lock_descriptor:
diff --git a/scripts/runqemu-ifdown b/scripts/runqemu-ifdown
index 8f66cfa2..ffbc9de 100755
--- a/scripts/runqemu-ifdown
+++ b/scripts/runqemu-ifdown
@@ -41,11 +41,11 @@ if [ $# -ne 2 ]; then
 fi
 
 TAP=$1
-NATIVE_SYSROOT_DIR=$2
+STAGING_BINDIR_NATIVE=$2
 
-TUNCTL=$NATIVE_SYSROOT_DIR/usr/bin/tunctl
+TUNCTL=$STAGING_BINDIR_NATIVE/tunctl
 if [ ! -e "$TUNCTL" ]; then
-	echo "Error: Unable to find tunctl binary in '$NATIVE_SYSROOT_DIR/usr/bin', please bitbake qemu-helper-native"
+	echo "Error: Unable to find tunctl binary in '$STAGING_BINDIR_NATIVE', please bitbake qemu-helper-native"
 	exit 1
 fi
 
diff --git a/scripts/runqemu-ifup b/scripts/runqemu-ifup
index d9bd894..59a15ea 100755
--- a/scripts/runqemu-ifup
+++ b/scripts/runqemu-ifup
@@ -49,11 +49,11 @@ fi
 
 USERID="-u $1"
 GROUP="-g $2"
-NATIVE_SYSROOT_DIR=$3
+STAGING_BINDIR_NATIVE=$3
 
-TUNCTL=$NATIVE_SYSROOT_DIR/usr/bin/tunctl
+TUNCTL=$STAGING_BINDIR_NATIVE/tunctl
 if [ ! -x "$TUNCTL" ]; then
-       echo "Error: Unable to find tunctl binary in '$NATIVE_SYSROOT_DIR/usr/bin', please bitbake qemu-helper-native"
+       echo "Error: Unable to find tunctl binary in '$STAGING_BINDIR_NATIVE', please bitbake qemu-helper-native"
 	exit 1
 fi
 
-- 
2.1.4

[PATCH 5/5] qemuboot: write native sysroot of qemu-helper into qemuboot.conf

[Ed Bartosh]

Native sysroot of qemu-helper contains all required tools
(qemu-system and tunctl atm) for runqemu to work. It's not
removed by rm_dir and should always exist. It makes sense
to write it into qemuboot.conf to make runqemu to use it
as a default directory for native tools.

This should also speed up runqemu as it doesn't need to run
to run 'bitbake qemu-helper -e' to get its native sysroot.

[YOCTO #11266]
[YOCTO #11193]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/qemuboot.bbclass | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/meta/classes/qemuboot.bbclass b/meta/classes/qemuboot.bbclass
index 2870388..cc5314e 100644
--- a/meta/classes/qemuboot.bbclass
+++ b/meta/classes/qemuboot.bbclass
@@ -87,7 +87,14 @@ python do_write_qemuboot_conf() {
     cf = configparser.ConfigParser()
     cf.add_section('config_bsp')
     for k in qemuboot_vars(d):
-        cf.set('config_bsp', k, '%s' % d.getVar(k))
+        # qemu-helper-native sysroot is not removed by rm_work and
+        # contains all tools required by runqemu
+        if k == 'STAGING_BINDIR_NATIVE':
+            val = os.path.join(d.getVar('BASE_WORKDIR'), d.getVar('BUILD_SYS'),
+                               'qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin/')
+        else:
+            val = d.getVar(k)
+        cf.set('config_bsp', k, '%s' % val)
 
     # QB_DEFAULT_KERNEL's value of KERNEL_IMAGETYPE is the name of a symlink
     # to the kernel file, which hinders relocatability of the qb conf.
-- 
2.1.4