Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi <stefanha@gmail.com>]

On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues <lmr@redhat.com> wrote:
> One of our main goals is to provide useful tools for the qemu community,
> since we have a good number of tests and libraries written to perform
> integration/QA testing for that tool, being successfuly used by a number of
> QA teams that work on qemu. Also, we recently provided a subset of that
> infrastructure to test libvirt, one of our virtualization projects of
> interest.

Thanks for sharing.

My feeling is that although this proposal opens some interesting
possibilities we're missing the key ingredient to attract people to
autotest as a virtualization test platform.

As a developer who writes tests and would like to test more, my number
one requirement is productivity.  It is frustrating when there is a
test case in my mind but I lack the vocabulary to express it as an
automated test.  In other words, we need an API to that makes it easy
to express virtualization tests.  This is the most important thing for
me and I won't make any other points in this post but will try to
expand more on it.

It shouldn't be a heavyweight framework that imposes constraints on
tests or have a high learning curve, it should be a library that
provides a vocabulary for configuring VMs, interacting with the serial
port, etc.

I think autotest can already do a lot of that today, but I don't know
how.  Actually I kind of do because I've looked at test source but it
doesn't look easy for me to consume.  It hasn't been made accessible
to the level where I know autotest is the right choice.  I worry that
digging into autotest will require significant time investment and
impose constraints on simple test cases I want to write.

I've preferred to write plain Python unittest tests that invoke
qemu-img, qemu-io, and qemu-kvm directly.  It's a pain to write them
from scratch but still quicker than battling with autotest to write a
test every now and then.

What would make me use autotest:
 * A clear productivity win over writing tests from scratch
 * Python API docs for launching a VM, creating disk image,
interacting with the serial port, sending QMP commands, etc

I think these unmet needs are why QEMU has been growing its own test
infrastructure.

Stefan