[Qemu-devel] [PATCH 0/1] integrate qemu-test into kvm-userspace

[Qemu-devel] [PATCH 0/1] integrate qemu-test into kvm-userspace

[Ryan Harper]

This patch provides a relatively simple method for exercising various
features of qemu/kvm by interacting with the guest via serial and qemu
via the monitor that individual developers can use to validate that
their changes having broken fundamental feature/function.  Such tests
need to be very accessable and easy-to-use to encourge there use by
users/developers.  As a bonus, in-tree make test is trivially integrated
into higher-level test frameworks such as autotest.

Using such interfaces the guest requirements are very low (serial login)
and thus we can execute tests across a large number of differing
guests.  Writing new test cases should also be fairly simple,  for
example, the host_shutdown.py test logic is as follows:

def run_test(vm):
        vm.wait_for_boot()
        vm.login()
        vm.guest_command("echo pressing power button")
        output = vm.monitor_command('system_powerdown')
        if vm.wait_for_shutdown(): return 1

There is a configuration file (test/CONFIG) which can be used to provide
per-distro overrides for various functions.  The current defaults have
been tested against: RHEL5, openSUSE 11, Ubuntu 8.04, and Fedora 9 w.r.t
the regular expressions used for login, and prompt.

Getting started is relatively simple for someone familiar with
configuring a guest for serial login and boot output via serial.  Once
completed, add a file to $(TOPDIR)/test/images like the following:

% cat images/rhel5.2
DISK="/home/rharper/work/images/rhel5.2_x86_64_10G.qcow2"
COMMAND="qemu-system-x86_64 -m 512 -smp 2 -net tap -net nic,model=e1000
         -net nic,model=rtl8139 -drive file=${DISK},if=ide,boot=on
         -vnc none"

You can omit the COMMAND variable and a default qemu command will be run
instead (see run.sh).  For additional guest image or config just add a
new file in the images dir.

To run the tests, at the toplevel run:

% make && sudo make test
<snip massive amount of output>
************************************************************
Results: 5 passed, 4 FAILED
passed: e820_memory.py:fedora-9-x86_64
passed: host_shutdown.py:fedora-9-x86_64
passed: networking.py:fedora-9-x86_64
passed: reboot.py:fedora-9-x86_64
passed: smp_hotplug.py:fedora-9-x86_64
FAILED: Test:host_reset.py,Image:fedora-9-x86_64
FAILED: Test:migrate.py,Image:fedora-9-x86_64
FAILED: Test:timedrift.py,Image:fedora-9-x86_64
FAILED: Test:writeverify.py,Image:fedora-9-x86_64
************************************************************


Ultimately, I'd like to push this into qemu since nothing we're doing
here is arch or kvm specific.  To do so, I'd move the test bits into qemu/tests
and invoke run.sh from there, iterating over the various QEMU_PROG names
available for each configured TARGET_DIRS so that we support running
when qemu has been configured with multiple targets.

The current kvm-userspace integration handles removing and loading the
compiled modules in the kernel dir and some additional symlink-fu to
ensure we run qemu from the build tree and don't need to have run make
install.  This integration would stay in the kvm-userspace tree and
change to recursing into qemu and invoking make test.

There is still plenty to do on the framework as well as the test, but I
wanted to get feedback on the current setup before investing significant
efforts.

[Qemu-devel] [PATCH 1/1] integrate qemu-test into kvm-userspace

[Ryan Harper]

diff --git a/Makefile b/Makefile
index 2c54e95..1c1223e 100644
--- a/Makefile
+++ b/Makefile
@@ -7,7 +7,7 @@ rpmrelease = devel
 
 sane-arch = $(subst i386,x86,$(subst x86_64,x86,$(subst s390x,s390,$(ARCH))))
 
-.PHONY: kernel user libkvm qemu bios vgabios extboot clean libfdt
+.PHONY: kernel user libkvm qemu bios vgabios extboot clean libfdt test
 
 all: libkvm qemu
 ifneq '$(filter $(ARCH), x86_64 i386 ia64)' ''
@@ -104,9 +104,12 @@ srpm:
 	$(RM) $(tmpspec)
 
 clean:
-	for i in $(if $(WANT_MODULE), kernel) user libkvm qemu libfdt; do \
+	for i in $(if $(WANT_MODULE), kernel) user libkvm qemu libfdt test; do \
 		make -C $$i clean; \
 	done
 
 distclean: clean
 	rm -f config.mak user/config.mak
+
+test: all
+	$(MAKE) -C test $@
diff --git a/test/CONFIG b/test/CONFIG
new file mode 100644
index 0000000..8509f22
--- /dev/null
+++ b/test/CONFIG
@@ -0,0 +1,56 @@
+###############################################################################
+# QEMU-TEST options and defaults
+#
+###############################################################################
+[options]
+debug=1
+logfile=qemu-test.log
+serial=/dev/stdout
+
+###############################################################################
+# !!!!WARNING!!!!: don't change the defaults, instead add an override for your
+# distro, see below
+###############################################################################
+[defaults]
+username=root
+password=
+login_prompt=^(\S+) login:
+shell_prompt=(%(username)s@)*%(hostname)s.*#\ $
+
+###############################################################################
+# OVERRIDES
+#
+# Distro overrides section, to override the default values as specified in the
+# [defaults] section, add the same option name and a new value, e.g, overriding
+# the default username:
+#
+# [rhel]
+# username=foobar
+#
+#
+# Fields:
+# username:
+# password:
+# login_prompt:
+# shell_prompt:
+# banner:
+# add a regular expression to match the output from your distro's welcome banner
+# (/etc/issue) the option name will be used for overriding the default section.
+# For example, if you add the following option:
+#
+###############################################################################
+
+[rhel]
+banner=^Red Hat Enterprise Linux
+shell_prompt=^\[%(username)s@%(hostname)s.*\]#\ $
+
+[opensuse]
+banner=^Welcome to openSUSE
+shell_prompt=%(hostname)s:.*#\ $
+
+[ubuntu]
+banner=^Ubuntu
+shell_prompt=^%(username)s@%(hostname)s:.*#\ $
+
+[fedora]
+banner=^Fedora
diff --git a/test/HOWTO b/test/HOWTO
new file mode 100644
index 0000000..d7d1543
--- /dev/null
+++ b/test/HOWTO
@@ -0,0 +1,33 @@
+Configure your Guest - FIXME: more details
+-------------------
+1. Configure guest to support serial login
+
+
+2.  Create an image file in qemu-test/images:
+
+For example:
+
+% cat images/rhel5.2
+DISK="/home/rharper/work/images/rhel5.2_x86_64_10G.qcow2"
+COMMAND="qemu-system-x86_64 -m 512 -smp 2 -net tap -net nic,model=e1000 -net nic,model=rtl8139 -drive file=${DISK},if=ide,boot=on -vnc none"
+
+You can omit the COMMAND variable and a default qemu command will be run instead
+(see run.sh).  For each guest image or config add additional files in
+qemu-test/images.
+
+3. Run the test and observe the results:
+
+% make && sudo make test
+<snip massive amount of output>
+************************************************************
+Results: 5 passed, 4 FAILED
+passed: e820_memory.py:fedora-9-x86_64
+passed: host_shutdown.py:fedora-9-x86_64
+passed: networking.py:fedora-9-x86_64
+passed: reboot.py:fedora-9-x86_64
+passed: smp_hotplug.py:fedora-9-x86_64
+FAILED: Test:host_reset.py,Image:fedora-9-x86_64
+FAILED: Test:migrate.py,Image:fedora-9-x86_64
+FAILED: Test:timedrift.py,Image:fedora-9-x86_64
+FAILED: Test:writeverify.py,Image:fedora-9-x86_64
+************************************************************
diff --git a/test/Makefile b/test/Makefile
new file mode 100644
index 0000000..7694695
--- /dev/null
+++ b/test/Makefile
@@ -0,0 +1,21 @@
+include ../qemu/config-host.mak
+
+modules: ../kernel/kvm.ko ../kernel/kvm-intel.ko ../kernel/kvm-amd.ko
+	echo "Reloading kvm modules..."
+	if grep -q kvm_intel /proc/modules; then rmmod kvm-intel; else rmmod kvm-amd; fi
+	if grep -q kvm /proc/modules; then rmmod kvm; fi
+	insmod ../kernel/kvm.ko && sleep 1
+	if grep -q Intel /proc/cpuinfo; then insmod ../kernel/kvm-intel.ko; else insmod ../kernel/kvm-amd.ko; fi
+	
+setup: ../qemu/pc-bios ../qemu/keymaps
+	cd ../qemu/pc-bios && ln -sf ../keymaps
+
+clean:
+	rm -f ../qemu/pc-bios/keymaps
+
+test:	all	
+
+all: ../qemu/config-host.mak setup modules
+	./run.sh QEMU="../qemu/$(TARGET_DIRS)/qemu-system-$(ARCH) -L ../qemu/pc-bios" \
+		 TESTS="$(TESTS)" ARGS="$(ARGS)"
+
diff --git a/test/THEORY b/test/THEORY
new file mode 100644
index 0000000..16d5c48
--- /dev/null
+++ b/test/THEORY
@@ -0,0 +1,4 @@
+Each test should work with as many guests as possible with the least number
+of parameters required.
+
+Minimum guest requirements (only configuration)
diff --git a/test/TODO b/test/TODO
new file mode 100644
index 0000000..69353d3
--- /dev/null
+++ b/test/TODO
@@ -0,0 +1,31 @@
+* improve failure detection
+   o oops/soft lockup detection
+   o freeze detection
+      - hard lockup detection (100% cpu usage, min timeout expired)
+      - soft lockup detection (0% cpu usage, min timeout expired)
+      - qemu freeze detection (timeout in monitor command)
+* detect guest type
+   o automagically select appropriate defaults for prompt
+* lots more tests
+  o device validation
+  o disk emu/paravirt verification
+  o pause/resume
+  o savevm/loadvm
+* randomize sockets
+* redirect serial output to file, print to screen as option
+* write up on how to configure various distros to be ready to run any test
+* rewrite networking.py to parse info out of sysfs
+* pass test name to launch for logging
+
+
+#### Completed #####
+2008-07-23
+* make check|test integration into kvm-userspace tree
+* fedora guest support
+
+2008-07-16
+* integrate config file
+  o support per-distro overrides
+* ubuntu guest support
+* rhel5 guest support
+* opensuse geust support
diff --git a/test/e820_memory.py b/test/e820_memory.py
new file mode 100644
index 0000000..f02d85d
--- /dev/null
+++ b/test/e820_memory.py
@@ -0,0 +1,114 @@
+# Copyright IBM Corp. 2008
+# Authors: Anthony Liguori <aliguori@us.ibm.com>
+#        : Ryan Harper <ryanh@us.ibm.com
+#
+
+from qemu.test import launch, TimeoutException
+import sys
+import ConfigParser, os
+
+
+def check_table_entry(e820, entry, field, value):
+    v1 = e820[entry][field]
+    v2 = value
+    # third field is a string, so strip to trim newlines and such
+    if field == 2:
+        v1 = v1.strip()
+        v2 = v2.strip()
+    if v1 != v2:
+        print 'e820 entry %d field %d should be "%s", found [%s]' %(
+               entry, field, str(value), e820[entry][field])
+        return 1
+    print 'e820[%d][%d] OK' %(entry, field)
+    return 0
+
+
+def compare_table_entry(e820_a, e820_b, entry, field):
+    v1 = e820_a[entry][field]
+    v2 = e820_b[entry][field]
+    # third field is a string, so strip to trim newlines and such
+    if field == 2:
+        v1 = v1.strip()
+        v2 = v2.strip()
+    if v1 != v2:
+        print 'e820 table entry %d field %d do not match' %(entry, field)
+        print '[%s] != [%s]' %(v1, v2)
+        return 1
+
+    print 'e820[%d][%d] match' %(entry, field)
+    return 0
+
+
+def run_test(vm, memory_size):
+    vm.wait_for_boot()
+
+    vm.login()
+
+    e820 = []
+    output = vm.guest_command('dmesg | grep BIOS-e820')
+    for line in output.split('\n'):
+        # handle timestamps in dmesg
+        if line.startswith('['):
+            _, line = line.split('] ', 1)
+
+        words = line.split(' ', 5)
+        start = long('0x%s' % words[2], 0)
+        end = long('0x%s' % words[4], 0)
+        e820.append((start, end, words[5]))
+
+    if len(e820) != 6:
+        print 'Unusual number of e820 entries: %d' % len(e820)
+        return 1
+
+    e820_static = [(0x00000, 0x09fc00, '(usable)'),
+                   (0x9fc00, 0x0a0000, '(reserved)'),
+                   (0xe8000, 0x100000, '(reserved)')]
+
+    for i in range(3):
+        if compare_table_entry(e820_static, e820, i, 0): return 1
+        if compare_table_entry(e820_static, e820, i, 1): return 1
+        if compare_table_entry(e820_static, e820, i, 2): return 1
+
+    if check_table_entry(e820, 3, 0, 0x100000): return 1
+    if check_table_entry(e820, 3, 2, '(usable)'): return 1
+    if check_table_entry(e820, 4, 2, '(ACPI data)'): return 1
+       
+    if (e820[4][1] - e820[4][0]) != (64 << 10):
+        print 'e820 entry 4 has invalid size'
+        return 1
+
+    if e820[4][1] != memory_size:
+        print 'e820 entry 4 has invalid start address'
+        return 1
+
+    if check_table_entry(e820, 5, 0, 0xfffbd000): return 1
+    if check_table_entry(e820, 5, 1, 0x100000000): return 1
+    if check_table_entry(e820, 5, 2, '(reserved)'): return 1
+
+    return 0
+
+
+def main(args):
+    memory_size = 128 << 20
+    for i in range(len(args)):
+        if args[i] == '-m':
+            memory_size = long(args[i + 1]) << 20
+            break
+
+    vm = launch(*args)
+    err = 1
+    try:
+        err = run_test(vm, memory_size)
+    finally:
+        vm.shutdown()
+        vm.wait_for_shutdown()
+        vm.wait_for_quit()
+
+    return err
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except TimeoutException, e:
+        print 'Timeout occurred waiting for %s' % e.message
+        sys.exit(1)
diff --git a/test/host_reset.py b/test/host_reset.py
new file mode 100644
index 0000000..f1bf610
--- /dev/null
+++ b/test/host_reset.py
@@ -0,0 +1,38 @@
+# Copyright IBM Corp. 2008
+# Authors: Ryan Harper <ryanh@us.ibm.com
+#
+
+from qemu.test import launch, TimeoutException
+import sys
+
+def run_test(vm):
+    vm.wait_for_boot()
+    vm.login()
+    vm.guest_command("echo pressing reset button")
+    output = vm.monitor_command('system_reset')
+    vm.wait_for_restart()
+    vm.wait_for_boot()
+
+    return 0
+
+
+def main(args):
+    err = 1
+    vm = launch(*args)
+
+    try:
+        err = run_test(vm)
+    finally:
+        vm.shutdown()
+        vm.wait_for_shutdown()
+        vm.wait_for_quit()
+
+    return err
+
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except TimeoutException, e:
+        print 'Timeout occurred waiting for %s' % e.message
+        sys.exit(1)
diff --git a/test/host_shutdown.py b/test/host_shutdown.py
new file mode 100644
index 0000000..eefe232
--- /dev/null
+++ b/test/host_shutdown.py
@@ -0,0 +1,35 @@
+# Copyright IBM Corp. 2008
+# Authors: Ryan Harper <ryanh@us.ibm.com
+
+from qemu.test import launch, TimeoutException
+import sys
+
+def run_test(vm):
+    vm.wait_for_boot()
+    vm.login()
+    vm.guest_command("echo pressing power button")
+    output = vm.monitor_command('system_powerdown')
+    if vm.wait_for_shutdown(): return 1
+
+    return 0
+
+
+def main(args):
+    err = 1
+    vm = launch(*args)
+
+    try:
+        err = run_test(vm)
+    finally:
+        vm.monitor_command('quit')
+        vm.wait_for_quit()
+
+    return err
+
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except TimeoutException, e:
+        print 'Timeout occurred waiting for %s' % e.message
+        sys.exit(1)
diff --git a/test/migrate.py b/test/migrate.py
new file mode 100644
index 0000000..b13714d
--- /dev/null
+++ b/test/migrate.py
@@ -0,0 +1,55 @@
+# Copyright IBM Corp. 2008
+# Authors: Anthony Liguori <aliguori@us.ibm.com>
+#
+
+from qemu.test import launch
+import sys, time
+
+def test_ping(vm):
+    #gw_info = vm.get_gateway_info()
+    #if vm.ping(gw_info['address']): return False
+    output = vm.guest_command('ping -c 1 10.0.1.1')
+    _, stats, __ = output.rsplit('\n', 2)
+    words = stats.split(' ')
+    if words[0] != words[3]:
+        return False
+    return True
+
+def main(args):
+    A = launch(*args)
+    A.wait_for_boot()
+    A.login('root', '')
+
+    for i in range(10):
+        B = launch(*(args + ['-incoming', 'tcp://localhost:1025']))
+
+        B.booted = A.booted
+        B.logged_in = A.logged_in
+        B.hostname = A.hostname
+        B.username = A.username
+        B.password = A.password
+        B.prompt = A.prompt
+
+        if not test_ping(A):
+            print 'Ping test failed before migration'
+            A.monitor_command('quit')
+            B.monitor_command('quit')
+            return 1
+
+        A.monitor_command('migrate tcp://localhost:1025')
+        A.monitor_command('quit')
+
+        if not test_ping(B):
+            print 'Ping test failed after migration'
+            B.monitor_command('quit')
+            return 1
+
+        A = B
+        B = None
+
+    B.monitor_command('quit')
+
+    return 0
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv[1:]))
diff --git a/test/networking.py b/test/networking.py
new file mode 100644
index 0000000..18ca242
--- /dev/null
+++ b/test/networking.py
@@ -0,0 +1,80 @@
+# Copyright IBM Corp. 2008
+# Authors: Ryan Harper <ryanh@us.ibm.com
+#
+
+from qemu.test import launch, TimeoutException
+import sys
+
+def run_test(vm, args):
+    configured_nics = []
+    detected_nics = []
+    nic_static = { 'Unknown device 1af4:1000': 'virtio',
+                   'Realtek Semiconductor Co., Ltd.  RTL-8139/8139C/8139 C+ (rev 20)': 'rtl8319',
+                   'Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)': 'rtl8319',
+                   'Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)': 'e1000' }
+    drivers_static = { 'e1000': 'e1000', 'virtio_net': 'virtio', '8139cp': 'rtl8139' }
+    devices = []
+                      
+    vm.wait_for_boot()
+    vm.login()
+
+    for i in range(len(args)):
+        if args[i].startswith('nic') and 'model=' in args[i]:
+            configured_nics.append(args[i].split('model=')[1].split(',')[0])
+
+    if len(configured_nics) < 1:
+        print "this test requires a configured nic"
+        print "try again with: '-net nic,model=<ethernet device>'"
+        return 1
+
+    output = vm.guest_command('lspci -v | grep Ethernet')
+    for line in output.split('\n'):
+        device = " ".join(line.split()[3:]).replace('\r','')
+        if device in nic_static.keys():
+            detected_nics.append(nic_static[device])
+
+    if len(detected_nics) != len(configured_nics):
+        print 'Failed to detect all configured nics'
+        print 'Configured nics: %s' %(configured_nics)
+        print 'Detected nics: %s' %(detected_nics)
+        return 1
+
+    # build interface to driver mapping array
+    output = vm.guest_command('for d in `ls -1 /sys/class/net/ | grep eth`; do driver=$(basename `readlink /sys/class/net/$d/device/driver/module`); echo ${driver}:$d; done');
+    for line in output.split('\n'):
+        (driver,device) = line.split(':')
+        devices.append((device,driver))
+    
+    gw_info = vm.get_gateway_info() 
+
+    # if device is configured and connected to a gateway, do a ping check
+    # FIXME multi-gateway setups not tested
+    for (device,driver) in devices:
+        dev_info = vm.get_netinfo(device)
+        if dev_info['state'] == 'up':
+            if vm.ping(gw_info['address'], iface=device): return 1
+        else:
+            print "device %s, driver %s not enabled in guest, skipping..." %(device, driver)
+
+    return 0
+
+
+def main(args):
+    err = 1
+    vm = launch(*args)
+
+    try:
+        err = run_test(vm, args)
+    finally:
+        vm.shutdown()
+        vm.wait_for_shutdown()
+        vm.wait_for_quit()
+
+    return err
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except TimeoutException, e:
+        print 'Timeout occurred waiting for %s' % e.message
+        sys.exit(1)
diff --git a/test/qemu/__init__.py b/test/qemu/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/test/qemu/test.py b/test/qemu/test.py
new file mode 100644
index 0000000..a603c34
--- /dev/null
+++ b/test/qemu/test.py
@@ -0,0 +1,418 @@
+# Copyright IBM Corp. 2008
+# Authors: Anthony Liguori <aliguori@us.ibm.com>
+#        : Ryan Harper <ryanh@us.ibm.com
+#
+
+import os, sys, socket, re, time, select, ConfigParser
+
+class TimeoutException(Exception):
+    def __init__(self, message):
+        Exception.__init__(self)
+        self.message = message
+
+
+class QEMUInstance(object):
+    def __init__(self, pid, serial_fd, monitor_fd, configfile=None):
+        self.configfile = None
+        self.DEBUG = None
+        self.logfile = None
+        self.logfd = None
+        self.pid = pid
+        self.serial = serial_fd
+        self.serialfile = None
+        self.serialoutfd = None
+        self.monitor = monitor_fd
+        self.booted = False
+        self.logged_in = False
+        self.distro = None
+        self.login_prompt = None
+        self.shell_prompt = None
+        self.username = None
+        self.hostname = None
+        self.prompt = None
+ 
+        # parse config file, check if custom path has been provided
+        if configfile:
+            self.configfile = configfile
+        else:
+            # look for CONFIG file in current dir
+            self.configfile = os.path.join(os.getcwd(),'CONFIG')
+
+        # fire up the parser
+        try:
+            self.config = ConfigParser.ConfigParser()
+            self.config.readfp(open(self.configfile))
+        except Exception, e:
+            print 'Failed to parse config: %s'%(self.configfile)
+            print e.message
+            self.quit()
+ 
+        # fetch common options from config
+        if self.config.has_option("options", "debug"):
+            self.DEBUG = self.config.get("options", "debug")
+        if self.config.has_option("options", "logfile"):
+            self.logfile = self.config.get("options", "logfile")
+        if self.config.has_option("options", "serialfile"):
+            self.serialfile = self.config.get("options", "serialfile")
+
+        # dump header if we're debuggin
+        if self.DEBUG:
+            self.logfd = open(self.logfile, "a")
+            self.log("****************************************************************")
+            self.log("qemu-test v0.1 init")
+            self.log("****************************************************************")
+
+        #if self.serialfile:
+        #    self.serialoutfd = open(self.serialfile, "a")
+        #    self.log("Writing serial output to %s", self.serialfile)
+                
+
+        # wait for qemu monitor to come up -- ensure it's working
+        self.wait_for_monitor_prompt()
+
+
+    def log(self, message):
+        if self.DEBUG:
+                self.logfd.write('[%s] %s\n' %(time.time(), message))
+                self.logfd.flush()
+
+    ###########################################################  
+    # waiting methods 
+    ###########################################################  
+
+    # base method which reads from the serial line checking to see
+    # the current input matches a designated regular expression
+    # also provide two timers, total (timeout) and activity timeout
+    # (min_activity).  The latter is specified when the caller 
+    # wants to ensure some output is being generated with some 
+    # frequency.  Return 'message' in the exception if either
+    # timer fails
+    def wait_for_rexpr(self, rexpr, timeout=None, min_activity=None,
+                       message=None):
+        output = ''
+        m = None
+        self.log("wait_for_rexpr(%s) ->"%(rexpr))
+
+        if message == None:
+            message = "rexpr `%s'\n" % rexpr
+
+        if timeout != None:
+            end_time = time.time() + timeout
+        else:
+            end_time = None
+
+        if min_activity == None:
+            min_activity = timeout
+
+        while m == None:
+            now = time.time()
+            if end_time != None and end_time < now:
+                raise TimeoutException(message)
+
+            if end_time != None or min_activity != None:
+                if end_time == None:
+                    wait = min_activity
+                else:
+                    wait = min(min_activity, end_time - now)
+
+                rdfds, _, __ = select.select([self.serial], [], [], wait)
+                if self.serial not in rdfds:
+                    raise TimeoutException(message)
+
+            ch = self.serial.recv(1)
+            if ch == '':
+                break
+            # TODO: support configuring output to file 
+            sys.stdout.write(ch)
+            sys.stdout.flush()
+            output += ch
+            if output.find('\n') != -1:
+                _, line = output.rsplit('\n', 1)
+                #self.log("searching line: [%s] -> [%s]"%(rexpr, line))
+                m = re.search(rexpr, line)
+        self.log("wait_for_rexpr() <-")
+        return output, m
+
+
+    def wait_for_monitor_prompt(self):
+        info = ''
+        self.log("wait_for_monitor_prompt() ->")
+        while not info.endswith('\n(qemu) '):
+            ch = self.monitor.recv(1)
+            if ch == '':
+                break
+            #self.log("got: %s"%(ch))
+            info += ch
+
+        self.log("wait_for_monitor_prompt() <-")
+        return info
+
+
+    def wait_for_linux_password(self, rexpr='^Password: ', timeout=None):
+        self.wait_for_rexpr(rexpr, timeout=timeout)
+
+
+    def wait_for_boot(self, timeout=None):
+        try:
+            self.log("wait_for_boot() ->")
+            self.detect_distro()
+            self.log("after distro detect")
+            if not self.login_prompt:
+                self.login_prompt = self.getconfig("login_prompt")
+            output, m = self.wait_for_rexpr(self.login_prompt, timeout, min_activity=60,
+                                            message="Linux login prompt")
+            self.hostname = m.group(1)
+            self.booted = True
+            self.log("wait_for_boot() <-")
+        except Exception, e:
+            print "wait_for_boot error: %s" % e.message
+
+
+    def detect_distro(self, timeout=None):
+        self.log("detect_distro() ->")
+        if self.distro == None:
+            match_map = []
+            self.log("distro not set yet")
+            # list potential distros to check for based on config overrides
+            distros = filter(lambda x: self.config.has_option(x, 'banner'),
+                             self.config.sections())
+            self.log("distros in config: %s"%(distros))
+
+            # keep a position mapping
+            for d in distros:
+                match_map.append(d)
+
+            banner = "|".join(map(lambda y: "(%s)" % self.config.get(y, 'banner',
+                            raw=True), distros))
+            self.log("distro rexpr: [%s]"%(banner))
+            try:
+                output, m = self.wait_for_rexpr(banner, timeout, min_activity=20,
+                                                message="Detect distro banner")
+
+                # groups returns a tuple of which parts of the regular expression
+                # were involved in the match, extracting which group was
+                # not None will give us an index into our regrexpress for each
+                # distro and we can determine which distro we detected
+                matchlist = list(m.groups())
+                for x in range(len(matchlist)):
+                    if matchlist[x] != None:
+                        self.distro = match_map[x]
+
+                self.log("detected distro: %s"%(self.distro))
+            except Exception, e:
+                self.log("waiting for distro rexpr FAIL: %s"%(e.message))
+                self.distro = 'defaults'
+                # kick the console to spit out a new login prompt
+                # as we probably ate it looking for a banner
+                self.send_to_guest("\n")
+                
+
+        self.log("detect_distro() <-")
+
+    def wait_for_shutdown(self, rexpr='^Power down.', timeout=180):
+        output, m = self.wait_for_rexpr(rexpr, timeout, min_activity=60,
+                                        message="Linux shutdown message")
+        self.booted = False
+
+
+    def wait_for_restart(self, rexpr='^Restarting system.', timeout=180):
+        output, m = self.wait_for_rexpr(rexpr, timeout, min_activity=60,
+                                        message="Linux shutdown message")
+        self.booted = False
+
+
+    def wait_for_quit(self):
+        self.log("waidpid() on PID=%d"%(self.pid))
+        os.waitpid(self.pid, 0)
+
+
+    def getconfig(self, field):
+        def __getconfig(self, section, field):
+            self.log("getconfig: fetching section:%s field:%s "%(section,
+                        field))
+            try:
+                return self.config.get(section, field, raw=True)
+            except Exception, e:
+                self.log("getconfig: %s"%(e.message))
+                pass
+            return None
+
+        self.log("getconfig() ->")
+        # try looking in distro override section, otherwise defaults
+        value = __getconfig(self, self.distro, field)
+        if value == None:
+            value = __getconfig(self, "defaults", field)
+
+        self.log("getconfig() <-")
+        return value
+
+
+    def login(self):
+        self.log("login() ->")
+        if not self.prompt:
+            username = self.getconfig("username")
+            password = self.getconfig("password")
+            prompt   = self.getconfig("shell_prompt")
+
+            if username != 'root':
+                    raise Exception("login requires username=root");
+                   
+            if not self.booted:
+                self.wait_for_boot()
+
+            self.username = username
+            self.password = password
+            self.prompt = prompt % {'username': username, 'hostname': self.hostname}
+        
+
+        # XXX: maybe I should check for self.booted here as well
+        self.send_to_guest('%s\n' % self.username)
+        self.wait_for_rexpr('^Password: ')
+        # There seems to be a race with the password entry
+        time.sleep(1)
+        self.send_to_guest('%s\n' % self.password)
+        self.wait_for_rexpr(self.prompt, timeout=10)
+        self.logged_in = True
+
+
+    def send_to_guest(self, data):
+        self.serial.sendall(data)
+
+
+    def __sanitize(self, string):
+        return string.replace('\r','')
+
+
+    # common networking helpers
+    def get_gateway_info(self):
+        cmd = "route -n | grep ^0 | awk '{print $8\":\"$2}'"
+        output = self.__sanitize(self.guest_command(cmd))
+        output = output.split(":")
+        return { 'device':output[0], 'address': output[1] }
+
+
+    def get_netinfo(self, devicename):
+        d = self.__sanitize(devicename)
+        netinfo = {}
+        c = "ifconfig %s | grep [A-Za-z]" %(d)
+        ifconfig_raw = self.guest_command(c).split('\n')
+        if 'error' in ifconfig_raw:
+            return netinfo
+
+        netinfo['device'] = d
+        netinfo['macaddr'] = ifconfig_raw[0].split()[-1:]
+        #FIXME: counting # of lines of ifconfig output isn't a good way
+        # to determine if the device is configured
+        if len(ifconfig_raw) >= 8:
+            netinfo['address'] = ifconfig_raw[1].split()[1].split(':')[1]
+            netinfo['netmask'] = ifconfig_raw[1].split()[-1:][0].split(':')[1]
+            if 'UP' in ifconfig_raw[3]:
+                netinfo['state'] = 'up'
+            else:
+                netinfo['state'] = 'down'
+            netinfo['mtu'] = ifconfig_raw[3].split()[4].split(':')[1]
+        else:
+            # device not configured
+            netinfo['state'] = 'down'
+                
+        return netinfo
+
+
+    def ping(self, address, iface=None):
+        cmd = "ping -c 5 %s -w 10" %(address)
+        if iface:
+            iface = self.__sanitize(iface)
+            cmd = "%s -I %s" %(cmd, iface)
+        cmd = "%s; echo $?" %(cmd)
+        raw = self.guest_command(cmd).split('\n')
+        # extract the return code, dropping any empty entries from the list
+        rc = filter(lambda x: len(x) > 0, raw)[-1]
+        return int(rc)
+
+
+    def shutdown(self):
+        self.guest_command("halt; echo shutdown")
+
+
+    def quit(self):
+        self.monitor_command("quit")
+        self.wait_for_quit()
+
+    def guest_command(self, command, timeout=300):
+        self.log("guest_command(command=%s) ->"%(command))
+        self.send_to_guest('%s\n' % command)
+        output, _ = self.wait_for_rexpr('%s' % self.prompt, timeout,
+                                        message="Guest command: %s"%(command))
+        try:
+            output, _ = output.rsplit('\n', 1)
+            _, output = output.split('\n', 1)
+        except Exception, e:
+            pass
+        self.log("guest_command() <-")
+        return output.rsplit('\r', 1)[0]
+
+
+    def monitor_command(self, command):
+        self.log("monitor_command(command=%s) ->"%(command))
+        self.monitor.sendall('%s\n' % command)
+        info = self.wait_for_monitor_prompt()
+        self.log("info = %s"%(info))
+        index = info.find('\n')
+        if index != -1:
+            info = info[index+1:]
+        index = info.rfind('\n')
+        if index != -1:
+            info = info[0:index]
+        self.log("monitor_command() <-")
+        return info
+
+
+def launch(executable, *args):
+    def __extract_config(args):
+        if 'config=' in args:
+            configfile = args.split("config=")[1].split()[0]
+            newargs = " ".join(args.split("config=")[1].split()[1:])
+            return (configfile, newargs)
+        
+        return (None, args)
+            
+
+    (config, args) = __extract_config(args)
+
+    serial_path = os.tmpnam()
+    monitor_path = os.tmpnam()
+
+    try:
+        os.unlink(serial_path)
+    except Exception, e:
+        pass
+    try:
+        os.unlink(monitor_path)
+    except Exception, e:
+        pass
+
+    serial = socket.socket(socket.AF_UNIX)
+    serial.bind(serial_path)
+    serial.listen(1)
+
+    monitor = socket.socket(socket.AF_UNIX)
+    monitor.bind(monitor_path)
+    monitor.listen(1)
+
+    pid = os.fork()
+    if pid == 0:
+        os.execvp(executable, (executable, '-serial', 'unix:%s' % serial_path,
+                               '-monitor', 'unix:%s' % monitor_path) + args)
+        sys.exit(1)
+
+    serial_fd, _ = serial.accept()
+    monitor_fd, _ = monitor.accept()
+
+    os.unlink(serial_path)
+    os.unlink(monitor_path)
+
+    serial.close()
+    monitor.close()
+
+    return QEMUInstance(pid, serial_fd, monitor_fd, config)
+
diff --git a/test/reboot.py b/test/reboot.py
new file mode 100644
index 0000000..ded7980
--- /dev/null
+++ b/test/reboot.py
@@ -0,0 +1,42 @@
+# Copyright IBM Corp. 2008
+# Authors: Ryan Harper <ryanh@us.ibm.com
+#
+
+from qemu.test import launch, TimeoutException
+import sys
+
+def run_test(vm):
+    reboots=10
+    vm.wait_for_boot()
+    vm.login()
+
+    for x in range(1,reboots+1):
+        # background the reboot command so we match the prompt string
+        vm.guest_command("echo Reboot %s; reboot&"%(x))
+        vm.wait_for_restart()
+        vm.wait_for_boot()
+        vm.login()
+
+    return 0
+
+
+def main(args):
+    err = 1
+    vm = launch(*args)
+
+    try:
+        err = run_test(vm)
+    finally:
+        vm.shutdown()
+        vm.wait_for_shutdown()
+        vm.wait_for_quit()
+
+    return err
+
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except TimeoutException, e:
+        print 'Timeout occurred waiting for %s' % e.message
+        sys.exit(1)
diff --git a/test/run.sh b/test/run.sh
new file mode 100755
index 0000000..bc2f1b7
--- /dev/null
+++ b/test/run.sh
@@ -0,0 +1,62 @@
+#!/bin/bash
+#
+# Copyright IBM Corp. 2008
+# Authors: Ryan Harper <ryanh@us.ibm.com
+#
+
+
+while [ $# -gt 0 ]; do
+        var=${1%=*}
+        value=${1#*=}
+        export ${var}="${value}"
+        shift;
+done
+
+[ -z "${QEMU}" ] && { QEMU="qemu-system-x86_64"; }
+[ -z "${TESTS}" ] && { TESTS="`ls *.py`"; }
+[ -z "${ARGS}" ] && { ARGS="-m 512 -smp 2 -net tap -net nic,model=e1000 -net nic,model=rtl8139 -snapshot -vnc none"; }
+
+mkdir -p $PWD/images
+DISKS=( `ls $PWD/images` )
+# error out if we don't have any images defined
+[ ${#DISKS[@]} -le 0 ] && {
+    echo "No test images defined in $PWD/images"
+    exit 0;
+}
+
+declare -a PASS
+declare -a FAIL
+for t in ${TESTS[@]}; do
+        for d in ${DISKS[@]}; do
+                . $PWD/images/${d}
+                echo "Running Test:${t} on Image:${d}"
+                if [ -z "${COMMAND}" ]; then
+                        COMMAND="${QEMU} ${ARGS} -drive file=${DISK},if=ide,boot=on"
+                        echo "Using default command: ${COMMAND}"
+                else
+                        # filter out QEMU from COMMAND, image files should use
+                        # QEMU if they need to specify
+                        COMMAND="${QEMU} $(echo $COMMAND | fmt -w 1 | grep -v qemu-system | fmt -w 255)"
+                        echo "Using override command: ${COMMAND}"
+                fi
+                sleep 2
+                sudo python ${t} ${COMMAND}
+                rv="$?"
+                if [ "${rv}" == "0" ]; then
+                        PASS[${#PASS[@]}]="${t}:${d}"
+                else
+                        FAIL[${#FAIL[@]}]="Test:${t},Image:${d}"
+                fi                                
+        done
+done
+
+echo ""
+echo "************************************************************"
+echo "Results: ${#PASS[@]} passed, ${#FAIL[@]} FAILED"
+for f in ${PASS[@]}; do
+        echo "passed: ${f}"
+done
+for f in ${FAIL[@]}; do
+        echo "FAILED: ${f}"
+done
+echo "************************************************************"
diff --git a/test/smp_hotplug.py b/test/smp_hotplug.py
new file mode 100644
index 0000000..671e8cf
--- /dev/null
+++ b/test/smp_hotplug.py
@@ -0,0 +1,69 @@
+# Copyright IBM Corp. 2008
+# Authors: Anthony Liguori <aliguori@us.ibm.com>
+#
+
+from qemu.test import launch, TimeoutException
+import sys, time
+
+def count_guest_cpus(vm):
+    output = vm.guest_command('grep "^processor" /proc/cpuinfo')
+    return len(output.split('\n'))
+
+def hotplug_cpu(vm, cpu, value):
+    vm.guest_command('echo %d | sudo dd of=/sys/devices/system/cpu/cpu%d/online' %
+                     (value, cpu))
+
+def run_test(vm, n_cpus):
+    vm.wait_for_boot()
+    output = vm.monitor_command('info cpus')
+
+    if n_cpus != len(output.split('\n')):
+        print 'info cpus shows the wrong number of cpus'
+        return 1
+    
+    vm.login()
+
+    if count_guest_cpus(vm) != n_cpus:
+        print 'guest cannot see all of the cpus'
+        return 1
+
+    for i in range(1, n_cpus):
+        hotplug_cpu(vm, i, 0)
+        if count_guest_cpus(vm) != n_cpus - i:
+            print 'failed to hot unplug cpu %d' % i
+            return 1
+        # give the scheduler a chance to react
+        time.sleep(1)
+
+    for i in range(1, n_cpus):
+        hotplug_cpu(vm, i, 1)
+        if count_guest_cpus(vm) != i + 1:
+            print 'failed to hot plug cpu %d' % i
+            return 1
+        # give the scheduler a chance to react
+        time.sleep(1)
+
+def main(args):
+    for i in range(len(args)):
+        if args[i] == '-smp':
+            n_cpus = int(args[i + 1])
+            break
+
+    vm = launch(*args)
+
+    err = 1
+    try:
+        err = run_test(vm, n_cpus)
+    finally:
+        vm.shutdown()
+        vm.wait_for_shutdown()
+        vm.wait_for_quit()
+
+    return err
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except TimeoutException, e:
+        print 'Timeout occurred waiting for %s' % e.message
+        sys.exit(1)
diff --git a/test/timedrift.py b/test/timedrift.py
new file mode 100644
index 0000000..4e9f541
--- /dev/null
+++ b/test/timedrift.py
@@ -0,0 +1,67 @@
+# Copyright IBM Corp. 2008
+# Authors: Ryan Harper <ryanh@us.ibm.com
+#
+
+from qemu.test import launch, TimeoutException
+import sys, time
+
+def run_test(vm):
+    badclocks = []
+    vm.wait_for_boot()
+    vm.login()
+
+    # calculate how much time it takes to run a very simple guest command
+    length = 600
+    now = time.time()
+    vm.guest_command("echo true")
+    end = time.time()
+    overhead = int(end - now)
+    # obviously it takes *some* amount of time to issue the command
+    if overhead == 0:
+        overhead = 1
+    vm.guest_command("echo calculated overhead: %s seconds"%(overhead))
+
+    available = vm.guest_command("cat /sys/devices/system/clocksource/clocksource0/available_clocksource")
+    clocksource = vm.guest_command("cat /sys/devices/system/clocksource/clocksource0/current_clocksource")
+    for clock in available.split():
+        if clock == "jiffies":
+                vm.guest_command("echo skipping crappy clock=jiffies")
+                continue
+        vm.guest_command("echo switching clocksource to %s"%(clock))
+        vm.guest_command("echo %s | dd of=/sys/devices/system/clocksource/clocksource0/current_clocksource" %(clock))
+        now = time.time()
+        vm.guest_command("date -u +%s; sleep 600; date -u +%s", timeout=1200)
+        end = time.time()
+        delta = int(end - now)
+        vm.guest_command("echo %s took %s seconds to complete"%(clock, delta))
+        
+        if (delta-length) > overhead:
+                badclocks.append(clock)
+                
+    if len(badclocks) > 0:
+        print "Bad clocks: %s"%(badclocks)
+        return 1
+
+    return 0
+
+
+def main(args):
+    err = 1
+    vm = launch(*args)
+
+    try:
+        err = run_test(vm)
+    finally:
+        vm.shutdown()
+        vm.wait_for_shutdown()
+        vm.wait_for_quit()
+
+    return err
+
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except TimeoutException, e:
+        print 'Timeout occurred waiting for %s' % e.message
+        sys.exit(1)
diff --git a/test/writeverify.py b/test/writeverify.py
new file mode 100644
index 0000000..ba072dd
--- /dev/null
+++ b/test/writeverify.py
@@ -0,0 +1,57 @@
+# Copyright IBM Corp. 2008
+# Authors: Ryan Harper <ryanh@us.ibm.com
+#
+
+from qemu.test import launch, TimeoutException
+import sys
+
+def run_test(vm):
+    # definitions
+    fio_cmd = 'fio --thread --bs=64k --direct=1 --ioengine=sync --verify=md5 --verify_pattern=0xaa555aa5 --verify_interval=1000 --size=%(size)s --filename=%(filename)s --name=write-phase --rw=write --fill_device=1 --do_verify=0 --name=verify-phase --stonewall --create_serialize=0 --rw=read --do_verify=1'
+
+    write_size = '4g'
+    write_file = '/dev/vda'
+    vm.wait_for_boot()
+    vm.login()
+
+    # make sure we have fio installed
+    output = vm.guest_command('which fio')
+
+    # find mounted path on virtio device or unmounted device
+    cmd = fio_cmd  % { 'size': write_size, 'filename': write_file }
+    cmd = cmd + "; echo $?"
+    raw = vm.guest_command(cmd, timeout=None).split('\n')
+    rc = filter(lambda x: len(x) > 0, raw)[-1]
+    return int(rc)
+
+
+def main(args):
+    err = 1
+    virtio_device = 0
+
+    #for a in args:
+    #    if 'file=' in a: and 'if=virtio' in a:
+    #        virtio_device=1
+    #        break;
+
+
+    # scan args for virtio blk device
+    # create one and attach if not present
+    vm = launch(*args)
+
+    try:
+        err = run_test(vm)
+    finally:
+        vm.shutdown()
+        vm.wait_for_shutdown()
+        vm.wait_for_quit()
+
+    return err
+
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except TimeoutException, e:
+        print 'Timeout occurred waiting for %s' % e.message
+        sys.exit(1)

Re: [Qemu-devel] [PATCH 0/1] integrate qemu-test into kvm-userspace

[Paul Brook]

> Ultimately, I'd like to push this into qemu since nothing we're doing
> here is arch or kvm specific.

What advantage does inclusion with qemu give? 
AFAICS You're going to have to download gigabytes of images anyway, so 
wouldn't it make more sense to include the test script with those?

Paul

Re: [Qemu-devel] [PATCH 0/1] integrate qemu-test into kvm-userspace

[Anthony Liguori]

Paul Brook wrote:
>> Ultimately, I'd like to push this into qemu since nothing we're doing
>> here is arch or kvm specific.
>>     
>
> What advantage does inclusion with qemu give? 
> AFAICS You're going to have to download gigabytes of images anyway, so 
> wouldn't it make more sense to include the test script with those?
>   

The tests themselves are meant to run against pretty much any Linux 
image that's configured to use the serial port as a console.  The idea 
is to be able to run the individual tests with arbitrary QEMU command 
lines (think -smp 2 for instance) against arbitrary Linux images.  Yeah, 
it can't guarantee to be perfect but in practice, it seems to work 
pretty well.

Ryan's also included a "make check" framework that assumes a set of 
existing images.  That may not be useful within QEMU but I think the 
tests themselves are.

Regards,

Anthony Liguori



> Paul
>   

Re: [Qemu-devel] [PATCH 0/1] integrate qemu-test into kvm-userspace

[Ryan Harper]

* Paul Brook <paul@codesourcery.com> [2008-07-25 12:13]:
> > Ultimately, I'd like to push this into qemu since nothing we're doing
> > here is arch or kvm specific.
> 
> What advantage does inclusion with qemu give? 

We're working toward getting kvm merged with qemu so at some point there
won't be a separate kvm userspace tree with qemu.  That aside, I know
Anthony was interested in having this available to test out incoming
patches against different qemu build targets, kvm and x86 are just one
of those.

> AFAICS You're going to have to download gigabytes of images anyway, so 
> wouldn't it make more sense to include the test script with those?

We're punting on the images; developers already have images they use or
can easily create[1] such an images.  We should be writing tests that are
general enough to work with differing images in any case.

1. http://kvm.qumranet.com/kvmwiki/KVMTest

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

[Qemu-devel] Re: [PATCH 0/1] integrate qemu-test into kvm-userspace

[Avi Kivity]

Ryan Harper wrote:
> This patch provides a relatively simple method for exercising various
> features of qemu/kvm by interacting with the guest via serial and qemu
> via the monitor that individual developers can use to validate that
> their changes having broken fundamental feature/function.  Such tests
> need to be very accessable and easy-to-use to encourge there use by
> users/developers.  As a bonus, in-tree make test is trivially integrated
> into higher-level test frameworks such as autotest.
>
>   

The downside of this is that it effectively makes me the qemu-test 
maintainer, which will degrade my response times further.  Also, I have 
no real understanding of this code base.

I would really prefer an independent testsuite maintainer to pick this up.

-- 
error compiling committee.c: too many arguments to function