[PATCH 0/4] #10833: fix testing of wic images by testimage

[PATCH 0/4] #10833: fix testing of wic images by testimage

[Ed Bartosh]

Hi,

It was not possible to test wic images with testimage as this framework configured
guest networking by passing "ip=<host ip>::<guest ip>:<netmask>" to the kernel.
This approach doesn't work for wic images as kernel is run by bootloader and the
kernel command line is defined when image is built.

This patchset implements networking configuration by running commands on guests's
serial console. runqemu script was modified to output IP addresses and netmask to
be able to get these parameters from its output in testimage code.

The following changes since commit 8e9769773fd6d04402581c005bb423530a726457:

  meta-yocto-bsp: bump to the latest linux stable kernel for the non-x86 BSPs (2017-03-16 22:12:07 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/oe-core/tap-networking-wic-10833
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/tap-networking-wic-10833

Ed Bartosh (4):
  runqemu: output network configuration
  qemurunner: get network params from runqemu output
  qemurunner: configure guest networking
  bitbake.conf: add sudo to HOSTTOOLS_NONFATAL

 meta/conf/bitbake.conf            |  2 +-
 meta/lib/oeqa/utils/qemurunner.py | 49 ++++++++++++++++++++++++++++-----------
 scripts/runqemu                   |  4 +++-
 3 files changed, 39 insertions(+), 16 deletions(-)

--
Regards,
Ed

[PATCH 1/4] runqemu: output network configuration

[Ed Bartosh]

runqemu adds network configuration parameters to the kernel
command line to configure guest networking. This works only
for the images that run with external kernel using qemu -kernel
parameter. It doesn't work for the images that use bootloader
to boot kernel as -kernel parameter is not used and network
configuration is not possible to get.

Added host and guest ip addresses and netmask of tap link
to the runqemu output. This should allow external programs
that execute runqemu to get network configuration.

[YOCTO #10833]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/runqemu | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 1721956..453f9d8 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -917,7 +917,9 @@ class BaseConfig(object):
         client = gateway + 1
         if self.fstype == 'nfs':
             self.setup_nfs()
-        self.kernel_cmdline_script += " ip=192.168.7.%s::192.168.7.%s:255.255.255.0" % (client, gateway)
+        netconf = "192.168.7.%s::192.168.7.%s:255.255.255.0" % (client, gateway)
+        logger.info("Network configuration: %s", netconf)
+        self.kernel_cmdline_script += " ip=%s" % netconf
         mac = "%s%02x" % (self.mac_tap, client)
         qb_tap_opt = self.get('QB_TAP_OPT')
         if qb_tap_opt:
-- 
2.1.4

[PATCH 2/4] qemurunner: get network params from runqemu output

[Ed Bartosh]

Parsed runqemu output to get guest network configuration
if it's not present in runqemu command line.

[YOCTO #10833]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/utils/qemurunner.py | 36 ++++++++++++++++++++++--------------
 1 file changed, 22 insertions(+), 14 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 59dc11d..7e5f588 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -37,10 +37,12 @@ class QemuRunner:
         self.runqemu = None
         # pid of the qemu process that runqemu will start
         self.qemupid = None
-        # target ip - from the command line
+        # target ip - from the command line or runqemu output
         self.ip = None
         # host ip - where qemu is running
         self.server_ip = None
+        # target ip netmask
+        self.netmask = None
 
         self.machine = machine
         self.rootfs = rootfs
@@ -192,6 +194,7 @@ class QemuRunner:
                     return False
             time.sleep(1)
 
+        out = self.getOutput(output)
         if self.is_alive():
             logger.info("qemu started - qemu procces pid is %s" % self.qemupid)
             if get_ip:
@@ -203,17 +206,23 @@ class QemuRunner:
                     cmdline = re_control_char.sub('', cmdline)
                 try:
                     ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
-                    if not ips or len(ips) != 3:
-                        raise ValueError
-                    else:
-                        self.ip = ips[0]
-                        self.server_ip = ips[1]
+                    self.ip = ips[0]
+                    self.server_ip = ips[1]
+                    logger.info("qemu cmdline used:\n{}".format(cmdline))
                 except (IndexError, ValueError):
-                    logger.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used:\n%s\nand output from runqemu:\n%s" % (cmdline, self.getOutput(output)))
-                    self._dump_host()
-                    self.stop()
-                    return False
-                logger.info("qemu cmdline used:\n{}".format(cmdline))
+                    # Try to get network configuration from runqemu output
+                    match = re.match('.*Network configuration: ([0-9.]+)::([0-9.]+):([0-9.]+)$.*',
+                                     out, re.MULTILINE|re.DOTALL)
+                    if match:
+                        self.ip, self.server_ip, self.netmask = match.groups()
+                    else:
+                        logger.error("Couldn't get ip from qemu command line and runqemu output! "
+                                     "Here is the qemu command line used:\n%s\n"
+                                     "and output from runqemu:\n%s" % (cmdline, out))
+                        self._dump_host()
+                        self.stop()
+                        return False
+
                 logger.info("Target IP: %s" % self.ip)
                 logger.info("Server IP: %s" % self.server_ip)
 
@@ -222,12 +231,11 @@ class QemuRunner:
             if not self.thread.connection_established.wait(self.boottime):
                 logger.error("Didn't receive a console connection from qemu. "
                              "Here is the qemu command line used:\n%s\nand "
-                             "output from runqemu:\n%s" % (cmdline,
-                                                           self.getOutput(output)))
+                             "output from runqemu:\n%s" % (cmdline, out))
                 self.stop_thread()
                 return False
 
-            logger.info("Output from runqemu:\n%s", self.getOutput(output))
+            logger.info("Output from runqemu:\n%s", out)
             logger.info("Waiting at most %d seconds for login banner" % self.boottime)
             endtime = time.time() + self.boottime
             socklist = [self.server_socket]
-- 
2.1.4

[PATCH 3/4] qemurunner: configure guest networking

[Ed Bartosh]

Configured guest network interface through serial connection
when kernel is not run by qemu.

This should make it possible to test wic images with testimage.

[YOCTO #10833]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/utils/qemurunner.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 7e5f588..9ef7629 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -195,6 +195,7 @@ class QemuRunner:
             time.sleep(1)
 
         out = self.getOutput(output)
+        netconf = False # network configuration is not required by default
         if self.is_alive():
             logger.info("qemu started - qemu procces pid is %s" % self.qemupid)
             if get_ip:
@@ -215,6 +216,10 @@ class QemuRunner:
                                      out, re.MULTILINE|re.DOTALL)
                     if match:
                         self.ip, self.server_ip, self.netmask = match.groups()
+                        # network configuration is required as we couldn't get it
+                        # from the runqemu command line, so qemu doesn't run kernel
+                        # and guest networking is not configured
+                        netconf = True
                     else:
                         logger.error("Couldn't get ip from qemu command line and runqemu output! "
                                      "Here is the qemu command line used:\n%s\n"
@@ -287,6 +292,14 @@ class QemuRunner:
                 if re.search("root@[a-zA-Z0-9\-]+:~#", output):
                     self.logged = True
                     logger.info("Logged as root in serial console")
+                    if netconf:
+                        # configure guest networking
+                        cmd = "ifconfig eth0 %s netmask %s up\n" % (self.ip, self.netmask)
+                        output = self.run_serial(cmd, raw=True)[1]
+                        if re.search("root@[a-zA-Z0-9\-]+:~#", output):
+                            logger.info("configured ip address %s", self.ip)
+                        else:
+                            logger.info("Couldn't configure guest networking")
                 else:
                     logger.info("Couldn't login into serial console"
                             " as root using blank password")
-- 
2.1.4

[PATCH 4/4] bitbake.conf: add sudo to HOSTTOOLS_NONFATAL

[Ed Bartosh]

runqemu is using sudo to configure tap networking. Without sudo
in HOSTTOOLS_NONFATAL it may cause bitbake -c testimage to fail
with this error:
    runqemu - INFO - Setting up tap interface under sudo
    /bin/sh: sudo: command not found

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9fdbdfd..649dcef 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -469,7 +469,7 @@ HOSTTOOLS += " \
 HOSTTOOLS += "ps stty ip ssh scp ping vi"
 
 # Link to these if present
-HOSTTOOLS_NONFATAL += "ccache ld.bfd ld.gold gcc-ar gpg sftp nc socat"
+HOSTTOOLS_NONFATAL += "ccache ld.bfd ld.gold gcc-ar gpg sftp nc socat sudo"
 
 CCACHE ??= ""
 # Disable ccache explicitly if CCACHE is null since gcc may be a symlink
-- 
2.1.4