[PATCH 00/11] Properly fix oe-selftest QEMU usage + related fixes

[PATCH 00/11] Properly fix oe-selftest QEMU usage + related fixes

[Paul Eggleton]

The tests in oeqa/selftest/imagefeatures have had a long list of issues
that we've worked around; this patchset implements the ability to run
QEMU from oe-selftest tests properly and attempts to fix the remaining
known issues with the tests. At the same time we can make the devtool
test that uses QEMU use the new runqemu() logic, so do that as well.

When we did do the proper runqemu() implementation however, I uncovered
a number of other problems to do with error handling, process
termination and logging. This patchset should address those as well.

Thanks to Richard for the initial set of fixes which I built upon here.


The following changes since commit 03d01393d14b7b20dcb40ff89b1628883fd3b545:

  toaster.bbclass: Fix ValueError (2015-07-27 12:26:57 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/oe-selftest-qemu-fixes
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/oe-selftest-qemu-fixes

Paul Eggleton (9):
  oe-selftest: add scripts/lib and bitbake/lib to path
  oeqa/targetcontrol: create test directory before copying rootfs image
  oeqa/utils/qemurunner: fix logging
  oeqa/utils/qemurunner: avoid blocking on stty when running under
    oe-selftest
  oeqa/utils/qemurunner: fix error handling if runqemu exits with an
    error
  oeqa/targetcontrol: write QemuRunner log output to a file
  oeqa/selftest/imagefeatures: fix RPM4 test
  oeqa/selftest/imagefeatures: remove gummiboot tests
  oeqa/selftest/devtool: use new runqemu function

Richard Purdie (2):
  oeqa/selftest/imagefeatures: Use QemuTarget code
  oeqa/selftest/imagefeatures: fix SSH without password tests

 meta/lib/oeqa/selftest/devtool.py       |  72 +++++-----
 meta/lib/oeqa/selftest/imagefeatures.py | 247 +++++++++-----------------------
 meta/lib/oeqa/targetcontrol.py          |  12 ++
 meta/lib/oeqa/utils/commands.py         |  49 +++++++
 meta/lib/oeqa/utils/qemurunner.py       |  70 +++++----
 scripts/oe-selftest                     |   5 +-
 6 files changed, 207 insertions(+), 248 deletions(-)

-- 
2.1.0

[PATCH 01/11] oe-selftest: add scripts/lib and bitbake/lib to path

[Paul Eggleton]

In particular, this allows us to use code from bitbake's bb module
(such as tinfoil).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/oe-selftest | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index fa6245a..6db6e23 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -31,7 +31,10 @@ import unittest
 import logging
 import argparse
 
-sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'meta/lib')))
+sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + '/lib')
+import scriptpath
+scriptpath.add_bitbake_lib_path()
+scriptpath.add_oe_lib_path()
 
 import oeqa.selftest
 import oeqa.utils.ftools as ftools
-- 
2.1.0

[PATCH 02/11] oeqa/targetcontrol: create test directory before copying rootfs image

[Paul Eggleton]

The test directory might not exist at this point so just go ahead and
create it.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/targetcontrol.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index 60b09b2..beacdaf 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -143,6 +143,7 @@ class QemuTarget(BaseTarget):
 
     def deploy(self):
         try:
+            bb.utils.mkdirhier(self.testdir)
             shutil.copyfile(self.origrootfs, self.rootfs)
         except Exception as e:
             bb.fatal("Error copying rootfs: %s" % e)
-- 
2.1.0

[PATCH 03/11] oeqa/utils/qemurunner: fix logging

[Paul Eggleton]

OE-Core commit 519e381278d40bdac79add340e4c0460a9f97e17 unfortunately
broke logging in two different ways:

1) it prevented logging to the task log from working within bitbake
   -c testimage. This is due to the logger object being set up too early
   which interferes with BitBake's own logging. If we prefix the name
   with "BitBake." everything works (and we don't need to set the
   logging level).

2) Additionally because it called the log functions on the logging
   module and not the logger object it set up, this caused the
   oe-selftest logging to start printing everything from that point
   forward.

Fix these two issues and return us to the desired behaviour for
do_testimage.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/utils/qemurunner.py | 49 +++++++++++++++++++--------------------
 1 file changed, 24 insertions(+), 25 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 59c9934..4de3c64 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -14,8 +14,7 @@ import socket
 import select
 
 import logging
-logger = logging.getLogger("QemuRunner")
-logger.setLevel(logging.DEBUG - 2)
+logger = logging.getLogger("BitBake.QemuRunner")
 
 class QemuRunner:
 
@@ -51,11 +50,11 @@ class QemuRunner:
             self.server_socket.bind(("127.0.0.1",0))
             self.server_socket.listen(2)
             self.serverport = self.server_socket.getsockname()[1]
-            logging.info("Created listening socket for qemu serial console on: 127.0.0.1:%s" % self.serverport)
+            logger.info("Created listening socket for qemu serial console on: 127.0.0.1:%s" % self.serverport)
             return True
         except socket.error, msg:
             self.server_socket.close()
-            logging.error("Failed to create listening socket: %s" % msg[1])
+            logger.error("Failed to create listening socket: %s" % msg[1])
             return False
 
 
@@ -68,18 +67,18 @@ class QemuRunner:
         if self.display:
             os.environ["DISPLAY"] = self.display
         else:
-            logging.error("To start qemu I need a X desktop, please set DISPLAY correctly (e.g. DISPLAY=:1)")
+            logger.error("To start qemu I need a X desktop, please set DISPLAY correctly (e.g. DISPLAY=:1)")
             return False
         if not os.path.exists(self.rootfs):
-            logging.error("Invalid rootfs %s" % self.rootfs)
+            logger.error("Invalid rootfs %s" % self.rootfs)
             return False
         if not os.path.exists(self.tmpdir):
-            logging.error("Invalid TMPDIR path %s" % self.tmpdir)
+            logger.error("Invalid TMPDIR path %s" % self.tmpdir)
             return False
         else:
             os.environ["OE_TMPDIR"] = self.tmpdir
         if not os.path.exists(self.deploy_dir_image):
-            logging.error("Invalid DEPLOY_DIR_IMAGE path %s" % self.deploy_dir_image)
+            logger.error("Invalid DEPLOY_DIR_IMAGE path %s" % self.deploy_dir_image)
             return False
         else:
             os.environ["DEPLOY_DIR_IMAGE"] = self.deploy_dir_image
@@ -97,28 +96,28 @@ class QemuRunner:
         launch_cmd = 'runqemu %s %s %s' % (self.machine, self.rootfs, self.qemuparams)
         self.runqemu = subprocess.Popen(launch_cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,preexec_fn=os.setpgrp)
 
-        logging.info("runqemu started, pid is %s" % self.runqemu.pid)
-        logging.info("waiting at most %s seconds for qemu pid" % self.runqemutime)
+        logger.info("runqemu started, pid is %s" % self.runqemu.pid)
+        logger.info("waiting at most %s seconds for qemu pid" % self.runqemutime)
         endtime = time.time() + self.runqemutime
         while not self.is_alive() and time.time() < endtime:
             time.sleep(1)
 
         if self.is_alive():
-            logging.info("qemu started - qemu procces pid is %s" % self.qemupid)
+            logger.info("qemu started - qemu procces pid is %s" % self.qemupid)
             cmdline = ''
             with open('/proc/%s/cmdline' % self.qemupid) as p:
                 cmdline = p.read()
             ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
             if not ips or len(ips) != 3:
-                logging.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used: %s" % cmdline)
+                logger.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used: %s" % cmdline)
                 self.stop()
                 return False
             else:
                 self.ip = ips[0]
                 self.server_ip = ips[1]
-            logging.info("Target IP: %s" % self.ip)
-            logging.info("Server IP: %s" % self.server_ip)
-            logging.info("Waiting at most %d seconds for login banner" % self.boottime)
+            logger.info("Target IP: %s" % self.ip)
+            logger.info("Server IP: %s" % self.server_ip)
+            logger.info("Waiting at most %d seconds for login banner" % self.boottime)
             endtime = time.time() + self.boottime
             socklist = [self.server_socket]
             reachedlogin = False
@@ -131,7 +130,7 @@ class QemuRunner:
                         self.qemusock.setblocking(0)
                         socklist.append(self.qemusock)
                         socklist.remove(self.server_socket)
-                        logging.info("Connection from %s:%s" % addr)
+                        logger.info("Connection from %s:%s" % addr)
                     else:
                         data = sock.recv(1024)
                         if data:
@@ -140,24 +139,24 @@ class QemuRunner:
                             if re.search(".* login:", self.bootlog):
                                 stopread = True
                                 reachedlogin = True
-                                logging.info("Reached login banner")
+                                logger.info("Reached login banner")
                         else:
                             socklist.remove(sock)
                             sock.close()
                             stopread = True
 
             if not reachedlogin:
-                logging.info("Target didn't reached login boot in %d seconds" % self.boottime)
+                logger.info("Target didn't reached login boot in %d seconds" % self.boottime)
                 lines = "\n".join(self.bootlog.splitlines()[-5:])
-                logging.info("Last 5 lines of text:\n%s" % lines)
-                logging.info("Check full boot log: %s" % self.logfile)
+                logger.info("Last 5 lines of text:\n%s" % lines)
+                logger.info("Check full boot log: %s" % self.logfile)
                 self.stop()
                 return False
         else:
-            logging.info("Qemu pid didn't appeared in %s seconds" % self.runqemutime)
+            logger.info("Qemu pid didn't appeared in %s seconds" % self.runqemutime)
             output = self.runqemu.stdout
             self.stop()
-            logging.info("Output from runqemu:\n%s" % output.read())
+            logger.info("Output from runqemu:\n%s" % output.read())
             return False
 
         return self.is_alive()
@@ -165,13 +164,13 @@ class QemuRunner:
     def stop(self):
 
         if self.runqemu:
-            logging.info("Sending SIGTERM to runqemu")
+            logger.info("Sending SIGTERM to runqemu")
             os.killpg(self.runqemu.pid, signal.SIGTERM)
             endtime = time.time() + self.runqemutime
             while self.runqemu.poll() is None and time.time() < endtime:
                 time.sleep(1)
             if self.runqemu.poll() is None:
-                logging.info("Sending SIGKILL to runqemu")
+                logger.info("Sending SIGKILL to runqemu")
                 os.killpg(self.runqemu.pid, signal.SIGKILL)
             self.runqemu = None
         if self.server_socket:
@@ -181,7 +180,7 @@ class QemuRunner:
         self.ip = None
 
     def restart(self, qemuparams = None):
-        logging.info("Restarting qemu process")
+        logger.info("Restarting qemu process")
         if self.runqemu.poll() is None:
             self.stop()
         if self.start(qemuparams):
-- 
2.1.0

[PATCH 04/11] oeqa/utils/qemurunner: avoid blocking on stty when running under oe-selftest

[Paul Eggleton]

runqemu-internal runs stty to return the terminal to its previous state
in case QEMU hasn't done that properly (which it at least used to do
when it crashed). For some reason I have yet to determine, stty blocks
(on tcsetattr() according to gdb) when run within QemuRunner() under
oe-selftest, with the result that we always wait until the timeout and
then we kill the script, which adds an extra delay after QEMU is
stopped. Naturally you would assume that this is something to do with
the nature of the terminal under which it is being run; however no
amount of playing around with stdin/stdout/stderr seemed to fix the
issue, apart from passing in subprocess.PIPE as stdin which makes stty
error out with "stty: standard input: Inappropriate ioctl for device". I
was also unable to come up with a reliable test for the terminal which
we could use inside runqemu-internal to avoid calling stty. For now, go
with the stdin=subprocess.PIPE workaround to at least avoid the delay
with minimal ill effect.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/utils/qemurunner.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 4de3c64..11186d6 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -94,7 +94,11 @@ class QemuRunner:
             self.qemuparams = self.qemuparams[:-1] + " " + qemuparams + " " + '\"'
 
         launch_cmd = 'runqemu %s %s %s' % (self.machine, self.rootfs, self.qemuparams)
-        self.runqemu = subprocess.Popen(launch_cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,preexec_fn=os.setpgrp)
+        # FIXME: We pass in stdin=subprocess.PIPE here to work around stty
+        # blocking at the end of the runqemu script when using this within
+        # oe-selftest (this makes stty error out immediately). There ought
+        # to be a proper fix but this will suffice for now.
+        self.runqemu = subprocess.Popen(launch_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE, preexec_fn=os.setpgrp)
 
         logger.info("runqemu started, pid is %s" % self.runqemu.pid)
         logger.info("waiting at most %s seconds for qemu pid" % self.runqemutime)
-- 
2.1.0

[PATCH 05/11] oeqa/utils/qemurunner: fix error handling if runqemu exits with an error

[Paul Eggleton]

* Don't wait for QEMU to start if it's never going to (because runqemu
  exited with an error)
* Don't error out if killing the process fails with "no such process"
  (we don't care if it's already dead)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/utils/qemurunner.py | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 11186d6..71da21c 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -12,6 +12,7 @@ import signal
 import re
 import socket
 import select
+import errno
 
 import logging
 logger = logging.getLogger("BitBake.QemuRunner")
@@ -104,6 +105,14 @@ class QemuRunner:
         logger.info("waiting at most %s seconds for qemu pid" % self.runqemutime)
         endtime = time.time() + self.runqemutime
         while not self.is_alive() and time.time() < endtime:
+            if self.runqemu.poll():
+                if self.runqemu.returncode:
+                    # No point waiting any longer
+                    logger.info('runqemu exited with code %d' % self.runqemu.returncode)
+                    output = self.runqemu.stdout
+                    self.stop()
+                    logger.info("Output from runqemu:\n%s" % output.read())
+                    return False
             time.sleep(1)
 
         if self.is_alive():
@@ -169,7 +178,11 @@ class QemuRunner:
 
         if self.runqemu:
             logger.info("Sending SIGTERM to runqemu")
-            os.killpg(self.runqemu.pid, signal.SIGTERM)
+            try:
+                os.killpg(self.runqemu.pid, signal.SIGTERM)
+            except OSError as e:
+                if e.errno != errno.ESRCH:
+                    raise
             endtime = time.time() + self.runqemutime
             while self.runqemu.poll() is None and time.time() < endtime:
                 time.sleep(1)
-- 
2.1.0

[PATCH 06/11] oeqa/targetcontrol: write QemuRunner log output to a file

[Paul Eggleton]

If we use this outside of testimage we don't have a task log; so let's
just explicitly write the log output to a file all the time so it's
always there to look at (particularly useful when runqemu exits
immediately with an error.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/targetcontrol.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index beacdaf..138e6178 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -10,6 +10,7 @@ import subprocess
 import bb
 import traceback
 import sys
+import logging
 from oeqa.utils.sshcontrol import SSHControl
 from oeqa.utils.qemurunner import QemuRunner
 from oeqa.utils.qemutinyrunner import QemuTinyRunner
@@ -123,6 +124,16 @@ class QemuTarget(BaseTarget):
         self.rootfs = os.path.join(self.testdir, d.getVar("IMAGE_LINK_NAME", True) + '-testimage.' + self.image_fstype)
         self.kernel = os.path.join(d.getVar("DEPLOY_DIR_IMAGE", True), d.getVar("KERNEL_IMAGETYPE", False) + '-' + d.getVar('MACHINE', False) + '.bin')
 
+        # Log QemuRunner log output to a file
+        import oe.path
+        bb.utils.mkdirhier(self.testdir)
+        self.qemurunnerlog = os.path.join(self.testdir, 'qemurunner_log.%s' % self.datetime)
+        logger = logging.getLogger('BitBake.QemuRunner')
+        loggerhandler = logging.FileHandler(self.qemurunnerlog)
+        loggerhandler.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
+        logger.addHandler(loggerhandler)
+        oe.path.symlink(os.path.basename(self.qemurunnerlog), os.path.join(self.testdir, 'qemurunner_log'), force=True)
+
         if d.getVar("DISTRO", True) == "poky-tiny":
             self.runner = QemuTinyRunner(machine=d.getVar("MACHINE", True),
                             rootfs=self.rootfs,
-- 
2.1.0

[PATCH 07/11] oeqa/selftest/imagefeatures: Use QemuTarget code

[Paul Eggleton]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Create a runqemu function which uses the QemuTarget() code from
oeqa.targetcontrol to setup the QEMU instance, with all of the added
robustness that that gives us. To do this, a datastore is needed for the
recipe in question (core-image-minimal) so we do the work needed to set
this up. We then use this runqemu function within the imagefeatures
tests instead of a hand-rolled implementation.

We can then use SSHControl to run the SSH tests rather than rolling our
own code to do that as an added bonus.

Fixed and extended by Paul Eggleton <paul.eggleton@linux.intel.com>.

Part of the fix for [YOCTO #7994].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/selftest/imagefeatures.py | 96 +++++++--------------------------
 meta/lib/oeqa/utils/commands.py         | 49 +++++++++++++++++
 2 files changed, 67 insertions(+), 78 deletions(-)

diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index 1795b7b..d48435f 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -1,20 +1,18 @@
 from oeqa.selftest.base import oeSelfTest
-from oeqa.utils.commands import runCmd, bitbake, get_bb_var
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, runqemu
 from oeqa.utils.decorators import testcase
-import pexpect
+from oeqa.utils.sshcontrol import SSHControl
 from os.path import isfile
-from os import system, killpg
+from os import system
 import glob
-import signal
-
+import os
+import sys
+import logging
 
 class ImageFeatures(oeSelfTest):
 
     test_user = 'tester'
     root_user = 'root'
-    prompt = r'qemux86:\S+[$#]\s+'
-    ssh_cmd = "ssh {} -l {} -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
-    get_ip_patt = r'\s+ip=(?P<qemu_ip>(\d+.){3}\d+)::'
 
     @testcase(1107)
     def test_non_root_user_can_connect_via_ssh_without_password(self):
@@ -37,41 +35,12 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-minimal
         bitbake('core-image-minimal')
 
-        # Boot qemu image
-        proc_qemu = pexpect.spawn('runqemu qemux86 nographic')
-        try:
-            proc_qemu.expect(self.get_ip_patt, timeout=100)
-            qemu_ip = proc_qemu.match.group('qemu_ip')
-            proc_qemu.expect('qemux86 login:', timeout=100)
-        except Exception as e:
-            try:
-                killpg(proc_qemu.pid, signal.SIGTERM)
-            except:
-                pass
-            self.fail('Failed to start qemu: %s' % e)
-
-        # Attempt to ssh with each user into qemu with empty password
-        for user in [self.root_user, self.test_user]:
-            proc_ssh = pexpect.spawn(self.ssh_cmd.format(qemu_ip, user))
-            index = proc_ssh.expect([self.prompt, pexpect.TIMEOUT, pexpect.EOF])
-            if index == 0:
-                # user successfully logged in with empty password
-                pass
-            elif index == 1:
-                killpg(proc_qemu.pid, signal.SIGTERM)
-                proc_ssh.terminate()
-                self.fail('Failed to ssh with {} user into qemu (timeout).'.format(user))
-            else:
-                killpg(proc_qemu.pid, signal.SIGTERM)
-                proc_ssh.terminate()
-                self.fail('Failed to ssh with {} user into qemu (eof).'.format(user))
-            proc_ssh.terminate()
-
-        # Cleanup
-        try:
-            killpg(proc_qemu.pid, signal.SIGTERM)
-        except:
-            pass
+        with runqemu("core-image-minimal", self) as qemu:
+            # Attempt to ssh with each user into qemu with empty password
+            for user in [self.root_user, self.test_user]:
+                ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user=user)
+                status, output = ssh.run("true")
+                self.assertEqual(status, 0, 'ssh to user %s failed with %s' % (user, output))
 
     @testcase(1115)
     def test_all_users_can_connect_via_ssh_without_password(self):
@@ -82,7 +51,6 @@ class ImageFeatures(oeSelfTest):
         Author:      Ionut Chisanovici <ionutx.chisanovici@intel.com>
         AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
         """
-
         features = 'EXTRA_IMAGE_FEATURES += "ssh-server-openssh allow-empty-password"\n'
         features += 'INHERIT += "extrausers"\n'
         features += 'EXTRA_USERS_PARAMS = "useradd -p \'\' {}; usermod -s /bin/sh {};"'.format(self.test_user, self.test_user)
@@ -93,41 +61,13 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-minimal
         bitbake('core-image-minimal')
 
-        # Boot qemu image
-        proc_qemu = pexpect.spawn('runqemu qemux86 nographic')
-        try:
-            proc_qemu.expect(self.get_ip_patt, timeout=100)
-            qemu_ip = proc_qemu.match.group('qemu_ip')
-            proc_qemu.expect('qemux86 login:', timeout=100)
-        except Exception as e:
-            try:
-                killpg(proc_qemu.pid, signal.SIGTERM)
-            except:
-                pass
-            self.fail('Failed to start qemu: %s' % e)
-
-        # Attempt to ssh with each user into qemu with empty password
-        for user in [self.root_user, self.test_user]:
-            proc_ssh = pexpect.spawn(self.ssh_cmd.format(qemu_ip, user))
-            index = proc_ssh.expect([self.prompt, pexpect.TIMEOUT, pexpect.EOF])
-            if index == 0:
-                # user successfully logged in with empty password
-                pass
-            elif index == 1:
-                killpg(proc_qemu.pid, signal.SIGTERM)
-                proc_ssh.terminate()
-                self.fail('Failed to ssh with {} user into qemu (timeout).'.format(user))
-            else:
-                killpg(proc_qemu.pid, signal.SIGTERM)
-                proc_ssh.terminate()
-                self.fail('Failed to ssh with {} user into qemu (eof).'.format(user))
-            proc_ssh.terminate()
+        with runqemu("core-image-minimal", self) as qemu:
+            # Attempt to ssh with each user into qemu with empty password
+            for user in [self.root_user, self.test_user]:
+                ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user=user)
+                status, output = ssh.run("true")
+                self.assertEqual(status, 0, 'ssh to user tester failed with %s' % output)
 
-        # Cleanup
-        try:
-            killpg(proc_qemu.pid, signal.SIGTERM)
-        except:
-            pass
 
     @testcase(1114)
     def test_rpm_version_4_support_on_image(self):
diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
index 1be7bed..50a08dc 100644
--- a/meta/lib/oeqa/utils/commands.py
+++ b/meta/lib/oeqa/utils/commands.py
@@ -17,6 +17,7 @@ import logging
 from oeqa.utils import CommandError
 from oeqa.utils import ftools
 import re
+import contextlib
 
 class Command(object):
     def __init__(self, command, bg=False, timeout=None, data=None, **options):
@@ -173,3 +174,51 @@ def create_temp_layer(templayerdir, templayername, priority=999, recipepathspec=
         f.write('BBFILE_PATTERN_%s = "^${LAYERDIR}/"\n' % templayername)
         f.write('BBFILE_PRIORITY_%s = "%d"\n' % (templayername, priority))
         f.write('BBFILE_PATTERN_IGNORE_EMPTY_%s = "1"\n' % templayername)
+
+
+@contextlib.contextmanager
+def runqemu(pn, test):
+
+    import bb.tinfoil
+    import bb.build
+
+    tinfoil = bb.tinfoil.Tinfoil()
+    tinfoil.prepare(False)
+    try:
+        tinfoil.logger.setLevel(logging.WARNING)
+        import oeqa.targetcontrol
+        tinfoil.config_data.setVar("TEST_LOG_DIR", "${WORKDIR}/testimage")
+        tinfoil.config_data.setVar("TEST_QEMUBOOT_TIMEOUT", "90")
+        import oe.recipeutils
+        recipefile = oe.recipeutils.pn_to_recipe(tinfoil.cooker, pn)
+        recipedata = oe.recipeutils.parse_recipe(recipefile, [], tinfoil.config_data)
+
+        # The QemuRunner log is saved out, but we need to ensure it is at the right
+        # log level (and then ensure that since it's a child of the BitBake logger,
+        # we disable propagation so we don't then see the log events on the console)
+        logger = logging.getLogger('BitBake.QemuRunner')
+        logger.setLevel(logging.DEBUG)
+        logger.propagate = False
+        logdir = recipedata.getVar("TEST_LOG_DIR", True)
+
+        qemu = oeqa.targetcontrol.QemuTarget(recipedata)
+    finally:
+        # We need to shut down tinfoil early here in case we actually want
+        # to run tinfoil-using utilities with the running QEMU instance.
+        # Luckily QemuTarget doesn't need it after the constructor.
+        tinfoil.shutdown()
+
+    try:
+        qemu.deploy()
+        try:
+            qemu.start()
+        except bb.build.FuncFailed:
+            raise Exception('Failed to start QEMU - see the logs in %s' % logdir)
+
+        yield qemu
+
+    finally:
+        try:
+            qemu.stop()
+        except:
+            pass
-- 
2.1.0

[PATCH 08/11] oeqa/selftest/imagefeatures: fix SSH without password tests

[Paul Eggleton]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

* We need to set EXTRA_IMAGE_FEATURES outright or existing values will
  affect the test
* For test case 1107 we need "empty-root-password" to match the behaviour
  described in the test case
* For test case 1115 we shouldn't be able to connect as root with the
  features we are setting

Test cases 1107 and 1115 have been updated in Testopia to match.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/selftest/imagefeatures.py | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index d48435f..20cc58d 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -25,7 +25,7 @@ class ImageFeatures(oeSelfTest):
         AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
         """
 
-        features = 'EXTRA_IMAGE_FEATURES += "ssh-server-openssh empty-root-password"\n'
+        features = 'EXTRA_IMAGE_FEATURES = "ssh-server-openssh empty-root-password allow-empty-password"\n'
         features += 'INHERIT += "extrausers"\n'
         features += 'EXTRA_USERS_PARAMS = "useradd -p \'\' {}; usermod -s /bin/sh {};"'.format(self.test_user, self.test_user)
 
@@ -46,12 +46,14 @@ class ImageFeatures(oeSelfTest):
     def test_all_users_can_connect_via_ssh_without_password(self):
         """
         Summary:     Check if all users can connect via ssh without password
-        Expected:    1. Connection to the image via ssh using root or tester user without providing a password should be allowed.
+        Expected: 1. Connection to the image via ssh using root user without providing a password should NOT be allowed.
+                  2. Connection to the image via ssh using tester user without providing a password should be allowed.
         Product:     oe-core
         Author:      Ionut Chisanovici <ionutx.chisanovici@intel.com>
         AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
         """
-        features = 'EXTRA_IMAGE_FEATURES += "ssh-server-openssh allow-empty-password"\n'
+
+        features = 'EXTRA_IMAGE_FEATURES = "ssh-server-openssh allow-empty-password"\n'
         features += 'INHERIT += "extrausers"\n'
         features += 'EXTRA_USERS_PARAMS = "useradd -p \'\' {}; usermod -s /bin/sh {};"'.format(self.test_user, self.test_user)
 
@@ -66,7 +68,10 @@ class ImageFeatures(oeSelfTest):
             for user in [self.root_user, self.test_user]:
                 ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user=user)
                 status, output = ssh.run("true")
-                self.assertEqual(status, 0, 'ssh to user tester failed with %s' % output)
+                if user == 'root':
+                    self.assertNotEqual(status, 0, 'ssh to user root was allowed when it should not have been')
+                else:
+                    self.assertEqual(status, 0, 'ssh to user tester failed with %s' % output)
 
 
     @testcase(1114)
-- 
2.1.0

[PATCH 09/11] oeqa/selftest/imagefeatures: fix RPM4 test

[Paul Eggleton]

* Use our new runqemu function
* Don't hard-code the RPM4 version
* Double-check the native version is RPM4
* Check that an rpm 4.x package is in the image manifest (this isn't
  strictly necessary, but "belt-and-braces" and it serves as an example
  of how to do that)
* Check that the database is working on the target
* Ensure the image actually has openssh in it so we can connect to it

Initial runqemu adaptation by Richard Purdie <richard.purdie@linuxfoundation.org>

Part of the fix for [YOCTO #7994].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/imagefeatures.py | 75 +++++++++++++++++++--------------
 1 file changed, 44 insertions(+), 31 deletions(-)

diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index 20cc58d..a370d1e 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -78,16 +78,17 @@ class ImageFeatures(oeSelfTest):
     def test_rpm_version_4_support_on_image(self):
         """
         Summary:     Check rpm version 4 support on image
-        Expected:    Rpm version must be 4.11.2
+        Expected:    Rpm version must be 4.x
         Product:     oe-core
         Author:      Ionut Chisanovici <ionutx.chisanovici@intel.com>
         AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
         """
 
-        rpm_version = '4.11.2'
-        features = 'IMAGE_INSTALL_append = " rpm"\n'
-        features += 'PREFERRED_VERSION_rpm = "{}"\n'.format(rpm_version)
-        features += 'PREFERRED_VERSION_rpm-native = "{}"\n'.format(rpm_version)
+        features = 'PREFERRED_VERSION_rpm = "4.%"\n'
+        features += 'PREFERRED_VERSION_rpm-native = "4.%"\n'
+        # Use openssh in IMAGE_INSTALL instead of ssh-server-openssh in EXTRA_IMAGE_FEATURES as a workaround for bug 8047
+        features += 'IMAGE_INSTALL_append = " openssh"\n'
+        features += 'EXTRA_IMAGE_FEATURES = "empty-root-password allow-empty-password package-management"\n'
         features += 'RPMROOTFSDEPENDS_remove = "rpmresolve-native:do_populate_sysroot"'
 
         # Append 'features' to local.conf
@@ -96,32 +97,44 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-minimal
         bitbake('core-image-minimal')
 
-        # Boot qemu image & get rpm version
-        proc_qemu = pexpect.spawn('runqemu qemux86 nographic')
-        try:
-            proc_qemu.expect('qemux86 login:', timeout=100)
-            proc_qemu.sendline(self.root_user)
-            proc_qemu.expect(self.prompt)
-            proc_qemu.sendline('rpm --version')
-            proc_qemu.expect(self.prompt)
-        except Exception as e:
-            try:
-                killpg(proc_qemu.pid, signal.SIGTERM)
-            except:
-                pass
-            self.fail('Failed to start qemu: %s' % e)
-
-        found_rpm_version = proc_qemu.before
-
-        # Make sure the retrieved rpm version is the expected one
-        self.assertIn(rpm_version, found_rpm_version,
-                      'RPM version is not {}, found instead {}.'.format(rpm_version, found_rpm_version))
-
-        # Cleanup (close qemu)
-        try:
-            killpg(proc_qemu.pid, signal.SIGTERM)
-        except:
-            pass
+        # Check the native version of rpm is correct
+        native_bindir = get_bb_var('STAGING_BINDIR_NATIVE')
+        result = runCmd(os.path.join(native_bindir, 'rpm') + ' --version')
+        self.assertIn('version 4.', result.output)
+
+        # Check manifest for the rpm package
+        deploydir = get_bb_var('DEPLOY_DIR_IMAGE')
+        imgname = get_bb_var('IMAGE_LINK_NAME', 'core-image-minimal')
+        with open(os.path.join(deploydir, imgname) + '.manifest', 'r') as f:
+            for line in f:
+                splitline = line.split()
+                if len(splitline) > 2:
+                    rpm_version = splitline[2]
+                    if splitline[0] == 'rpm':
+                        if not rpm_version.startswith('4.'):
+                            self.fail('rpm version %s found in image, expected 4.x' % rpm_version)
+                        break
+            else:
+                self.fail('No rpm package found in image')
+
+        # Now do a couple of runtime tests
+        with runqemu("core-image-minimal", self) as qemu:
+            command = "rpm --version"
+            status, output = qemu.run(command)
+            self.assertEqual(0, status, 'Failed to run command "%s": %s' % (command, output))
+            found_rpm_version = output.strip()
+
+            # Make sure the retrieved rpm version is the expected one
+            if rpm_version not in found_rpm_version:
+                self.fail('RPM version is not {}, found instead {}.'.format(rpm_version, found_rpm_version))
+
+            # Test that the rpm database is there and working
+            command = "rpm -qa"
+            status, output = qemu.run(command)
+            self.assertEqual(0, status, 'Failed to run command "%s": %s' % (command, output))
+            self.assertIn('packagegroup-core-boot', output)
+            self.assertIn('busybox', output)
+
 
     @testcase(1116)
     def test_clutter_image_can_be_built(self):
-- 
2.1.0

[PATCH 10/11] oeqa/selftest/imagefeatures: remove gummiboot tests

[Paul Eggleton]

These tests were _deleting_ meta-intel if it happened to appear under
COREBASE, which could have been catastrophic if there was any work in
progress in that directory. It turns out we don't even need meta-intel,
but we do need a machine that's set up appropriately (e.g.
genericx86-64). Tests that involve layers outside of OE-Core don't
really belong in OE-Core, and genericx86-64 is in meta-yocto-bsp;
however we will soon have the capability to have selftest tests in other
layers, so remove this here so we can add a fixed version in
meta-yocto-bsp after that happens.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/imagefeatures.py | 75 ---------------------------------
 1 file changed, 75 deletions(-)

diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index a370d1e..fcffc42 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -2,9 +2,6 @@ from oeqa.selftest.base import oeSelfTest
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var, runqemu
 from oeqa.utils.decorators import testcase
 from oeqa.utils.sshcontrol import SSHControl
-from os.path import isfile
-from os import system
-import glob
 import os
 import sys
 import logging
@@ -169,75 +166,3 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-weston
         bitbake('core-image-weston')
 
-
-class Gummiboot(oeSelfTest):
-
-    meta_intel_dir = ''
-
-    def setUpLocal(self):
-        """
-        Common setup for test cases: 1101, 1103
-        """
-
-        self.meta_intel_dir = get_bb_var('COREBASE') + '/meta-intel'
-        meta_nuc_dir = self.meta_intel_dir + '/meta-nuc'
-        meta_intel_repo = 'http://git.yoctoproject.org/git/meta-intel'
-
-        # Delete meta_intel_dir
-        system('rm -rf ' + self.meta_intel_dir)
-
-        # Delete meta-intel dir even if the setUp fails
-        self.add_command_to_tearDown('rm -rf ' + self.meta_intel_dir)
-
-        # Clone meta-intel
-        runCmd('git clone ' + meta_intel_repo + ' ' + self.meta_intel_dir)
-
-        # Add meta-intel and meta-nuc layers in conf/bblayers.conf
-        features = 'BBLAYERS += "' + self.meta_intel_dir + ' ' + meta_nuc_dir + '"'
-        self.append_bblayers_config(features)
-
-        # Set EFI_PROVIDER = "gummiboot" and MACHINE = "nuc" in conf/local.conf
-        features = 'EFI_PROVIDER = "gummiboot"\n'
-        features += 'MACHINE = "nuc"'
-        self.append_config(features)
-
-        # Run "bitbake syslinux syslinux-native parted-native dosfstools-native mtools-native core-image-minimal "
-        # to build a nuc/efi gummiboot image
-
-        bitbake('syslinux syslinux-native parted-native dosfstools-native mtools-native core-image-minimal')
-
-    @testcase(1101)
-    def test_efi_gummiboot_images_can_be_build(self):
-        """
-        Summary:     Check if efi/gummiboot images can be buit
-        Expected:    1. File gummibootx64.efi should be available in build/tmp/deploy/images/nuc
-                     2. Efi/gummiboot images can be built
-        Product:     oe-core
-        Author:      Ionut Chisanovici <ionutx.chisanovici@intel.com>
-        AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
-        """
-
-        look_for_file = 'gummibootx64.efi'
-        file_location = get_bb_var('COREBASE') + '/build/tmp/deploy/images/nuc/'
-
-        found = isfile(file_location + look_for_file)
-        self.assertTrue(found, 'File {} not found under {}.'.format(look_for_file, file_location))
-
-    @testcase(1103)
-    def test_wic_command_can_create_efi_gummiboot_installation_images(self):
-        """
-        Summary:     Check that wic command can create efi/gummiboot installation images
-        Expected:    A .direct file in folder /var/tmp/wic/ must be created.
-        Product:     oe-core
-        Author:      Ionut Chisanovici <ionutx.chisanovici@intel.com>
-        AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
-        """
-
-        # Create efi/gummiboot installation images
-        wic_create_cmd = 'wic create mkgummidisk -e core-image-minimal'
-        runCmd(wic_create_cmd)
-
-        # Verify that a .direct file was created
-        direct_file = '/var/tmp/wic/build/*.direct'
-        ret = glob.glob(direct_file)
-        self.assertEqual(1, len(ret), 'Failed to find the .direct file')
-- 
2.1.0

[PATCH 11/11] oeqa/selftest/devtool: use new runqemu function

[Paul Eggleton]

Use the common code we now have to run QEMU instead of running it
ourselves, avoiding reliance on the machine showing up at 192.168.7.2.
This also makes a copy of the image rather than using -snapshot so if we
need to inspect the image after a failure, we can.

Fixes [YOCTO #7928].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 72 ++++++++++++++++++---------------------
 1 file changed, 34 insertions(+), 38 deletions(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 21cd7f5..c833e54 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -8,7 +8,7 @@ import glob
 
 import oeqa.utils.ftools as ftools
 from oeqa.selftest.base import oeSelfTest
-from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer, runqemu
 from oeqa.utils.decorators import testcase
 
 class DevtoolBase(oeSelfTest):
@@ -799,12 +799,10 @@ class DevtoolTests(DevtoolBase):
             self.skipTest('No tap devices found - you must set up tap devices with scripts/runqemu-gen-tapdevs before running this test')
         workspacedir = os.path.join(self.builddir, 'workspace')
         self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
-        import pexpect
         # Definitions
         testrecipe = 'mdadm'
         testfile = '/sbin/mdadm'
         testimage = 'oe-selftest-image'
-        testhost = '192.168.7.2'
         testcommand = '/sbin/mdadm --help'
         # Build an image to run
         bitbake("%s qemu-native qemu-helper-native" % testimage)
@@ -821,44 +819,42 @@ class DevtoolTests(DevtoolBase):
         self.add_command_to_tearDown('bitbake -c clean %s' % testrecipe)
         result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
         # Test that deploy-target at this point fails (properly)
-        result = runCmd('devtool deploy-target -n %s root@%s' % (testrecipe, testhost), ignore_status=True)
+        result = runCmd('devtool deploy-target -n %s root@localhost' % testrecipe, ignore_status=True)
         self.assertNotEqual(result.output, 0, 'devtool deploy-target should have failed, output: %s' % result.output)
         self.assertNotIn(result.output, 'Traceback', 'devtool deploy-target should have failed with a proper error not a traceback, output: %s' % result.output)
         result = runCmd('devtool build %s' % testrecipe)
         # First try a dry-run of deploy-target
-        result = runCmd('devtool deploy-target -n %s root@%s' % (testrecipe, testhost))
+        result = runCmd('devtool deploy-target -n %s root@localhost' % testrecipe)
         self.assertIn('  %s' % testfile, result.output)
         # Boot the image
-        console = pexpect.spawn('runqemu %s %s qemuparams="-snapshot" nographic' % (machine, testimage))
-        console.expect("login:", timeout=120)
-        # Now really test deploy-target
-        result = runCmd('devtool deploy-target -c %s root@%s' % (testrecipe, testhost))
-        # Run a test command to see if it was installed properly
-        sshargs = '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
-        result = runCmd('ssh %s root@%s %s' % (sshargs, testhost, testcommand))
-        # Check if it deployed all of the files with the right ownership/perms
-        # First look on the host - need to do this under pseudo to get the correct ownership/perms
-        installdir = get_bb_var('D', testrecipe)
-        fakerootenv = get_bb_var('FAKEROOTENV', testrecipe)
-        fakerootcmd = get_bb_var('FAKEROOTCMD', testrecipe)
-        result = runCmd('%s %s find . -type f -exec ls -l {} \;' % (fakerootenv, fakerootcmd), cwd=installdir)
-        filelist1 = self._process_ls_output(result.output)
-
-        # Now look on the target
-        tempdir2 = tempfile.mkdtemp(prefix='devtoolqa')
-        self.track_for_cleanup(tempdir2)
-        tmpfilelist = os.path.join(tempdir2, 'files.txt')
-        with open(tmpfilelist, 'w') as f:
-            for line in filelist1:
-                splitline = line.split()
-                f.write(splitline[-1] + '\n')
-        result = runCmd('cat %s | ssh -q %s root@%s \'xargs ls -l\'' % (tmpfilelist, sshargs, testhost))
-        filelist2 = self._process_ls_output(result.output)
-        filelist1.sort(key=lambda item: item.split()[-1])
-        filelist2.sort(key=lambda item: item.split()[-1])
-        self.assertEqual(filelist1, filelist2)
-        # Test undeploy-target
-        result = runCmd('devtool undeploy-target -c %s root@%s' % (testrecipe, testhost))
-        result = runCmd('ssh %s root@%s %s' % (sshargs, testhost, testcommand), ignore_status=True)
-        self.assertNotEqual(result, 0, 'undeploy-target did not remove command as it should have')
-        console.close()
+        with runqemu(testimage, self) as qemu:
+            # Now really test deploy-target
+            result = runCmd('devtool deploy-target -c %s root@%s' % (testrecipe, qemu.ip))
+            # Run a test command to see if it was installed properly
+            sshargs = '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
+            result = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, testcommand))
+            # Check if it deployed all of the files with the right ownership/perms
+            # First look on the host - need to do this under pseudo to get the correct ownership/perms
+            installdir = get_bb_var('D', testrecipe)
+            fakerootenv = get_bb_var('FAKEROOTENV', testrecipe)
+            fakerootcmd = get_bb_var('FAKEROOTCMD', testrecipe)
+            result = runCmd('%s %s find . -type f -exec ls -l {} \;' % (fakerootenv, fakerootcmd), cwd=installdir)
+            filelist1 = self._process_ls_output(result.output)
+
+            # Now look on the target
+            tempdir2 = tempfile.mkdtemp(prefix='devtoolqa')
+            self.track_for_cleanup(tempdir2)
+            tmpfilelist = os.path.join(tempdir2, 'files.txt')
+            with open(tmpfilelist, 'w') as f:
+                for line in filelist1:
+                    splitline = line.split()
+                    f.write(splitline[-1] + '\n')
+            result = runCmd('cat %s | ssh -q %s root@%s \'xargs ls -l\'' % (tmpfilelist, sshargs, qemu.ip))
+            filelist2 = self._process_ls_output(result.output)
+            filelist1.sort(key=lambda item: item.split()[-1])
+            filelist2.sort(key=lambda item: item.split()[-1])
+            self.assertEqual(filelist1, filelist2)
+            # Test undeploy-target
+            result = runCmd('devtool undeploy-target -c %s root@%s' % (testrecipe, qemu.ip))
+            result = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, testcommand), ignore_status=True)
+            self.assertNotEqual(result, 0, 'undeploy-target did not remove command as it should have')
-- 
2.1.0