[Qemu-devel] [PATCH 0/2] Add qemu-test for automated testing

[Ryan Harper <ryanh@us.ibm.com>]

This patch provides a relatively simple method for exercising various
features of qemu by interacting with the guest via serial and qemu
via the monitor that individual developers can use to validate that
their changes haven't broken feature/function.  Such tests
need to be very accessable and easy-to-use to encourge there use by
users/developers.  As a bonus, in-tree make test is trivially integrated
into higher-level test frameworks such as autotest.

Using such interfaces the guest requirements are very low (serial login)
and thus we can execute tests across a large number of differing
guests.  Writing new test cases should also be fairly simple,  for
example, the host_shutdown.py test logic is as follows:

def run_test(vm):
        vm.wait_for_boot()
        vm.login()
        vm.guest_command("echo pressing power button")
        output = vm.monitor_command('system_powerdown')
        if vm.wait_for_shutdown(): return 1

There is a configuration file (tests/qemu-test/CONFIG) which can be used to
provide per-distro overrides for various functions.  The current defaults have
been tested against: RHEL5, openSUSE 11, Ubuntu 8.04, and Fedora 9 w.r.t the
regular expressions used for login, and prompt.

Getting started is relatively simple for someone familiar with
configuring a guest for serial login and boot output via serial.  Once
completed, add a file to tests/qemu-test/images like the following:

% cat images/rhel5.2
DISK="/home/rharper/work/images/rhel5.2_x86_64_10G.qcow2"
COMMAND="qemu-system-x86_64 -m 512 -smp 2 -net tap -net nic,model=e1000
         -net nic,model=rtl8139 -drive file=${DISK},if=ide,boot=on
         -vnc none"

You can omit the COMMAND variable and a default qemu command will be run
instead (see run.sh).  For additional guest image or config just add a
new file in the images dir.

To run the tests, at the toplevel run:

% make qemu-test


Some additional parameters can be passed through to run.sh to control how we
execute qemu-test: TESTS, ARGS, and IMAGEDIR.  By default, run.sh will execute
each test (*.py) in tests/qemu-test/.  If you want to restrict that list to
something more specific, you can pass a space separated list to the make
command:

% make qemu-test TESTS="reboot.py networking.py"

ARGS allows you to specific what parameters are passed to the qemu-system
command.  You can either modify the default args in tests/qemu-test/run.sh or
override them via the command line

IMAGEDIR is a relative path where the image input files are located.  It
defaults to 'images' in tests/qemu-test, but if you have a specific directory
you want to specify, for example tests/qemu-test/images.i386, this can be done
with:

% make qemu-test IMAGEDIR="images.i386"

Output of the run will be logged to qemu/${TARGET_ARCH}_test_results.out.
This includes all data generated during the run and the results, each test run
will include whether it passed or failed, the testname, and the name of the
image in which the test was run.  Some sample output:

% head -n 10 i386-softmmu_test_results.out
Running Test:migrate.py on Image:ubuntu-6.06
Using default command: ../../i386-softmmu/qemu -L ../../pc-bios -m 128 -net tap -net nic,model=e1000 -net nic,model=rtl8139 -snapshot -vnc none -drive file=/home/rharper/work/images/ubuntu_6.06_i386_10G.qcow2,if=ide -boot c
[42949372.960000] Linux version 2.6.15-51-server (buildd@rothera) (gcc version 4.0.3 (Ubuntu 4.0.3-1ubuntu5)) #1 SMP Thu Dec 6 21:37:18 UTC 2007
[42949372.960000] BIOS-provided physical RAM map:
[42949372.960000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
[42949372.960000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
[42949372.960000]  BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
[42949372.960000]  BIOS-e820: 0000000000100000 - 0000000007ff0000 (usable)
[42949372.960000]  BIOS-e820: 0000000007ff0000 - 0000000008000000 (ACPI data)
[42949372.960000]  BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)

% tail -n 7 i386-softmmu_test_results.out
0
root@vm1:~# 
************************************************************
Results: 2 passed, 0 FAILED
passed: migrate.py:ubuntu-6.06
passed: migrate_networking.py:ubuntu-6.06
************************************************************