Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori <anthony@codemonkey.ws>]

Hi Dor,

On 12/25/2011 09:19 AM, Dor Laor wrote:
> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>
> Well, I'm still not convinced that a new standalone package should handle these
> cases instead of kvm autotest. I'll be happy to integrate the tests to kvm
> autotest anyway and the more the merrier but imho it's a duplicate.

I'm sure kvm autotest could be taught to do exactly what qemu-test is doing. 
But why does kvm autotest have to do everything?  I doubt there would be much 
code reuse.

I think it's not a bad thing to have multiple test suites when there isn't 
considerable overlap.

>> It has the following characteristics:
>>
>> 1) It builds a custom kernel and initramfs based on busybox. This is
>> fairly important to ensure that we can run tests with no device
>> pre-requisites.
>
> This can be done easily w/ autotest too.

Okay, please demonstrate :-)  The procedure to do this with qemu test is:

$ git clone git://git.qemu.org/qemu-test.git
$ cd qemu-test
$ git submodule update --init
$ make

How would one do this with kvm autotest?

>> 2) Tests are scripts that are launched in the initramfs
>>
>> 3) The test controls exactly how QEMU is launched which allows easy
>> testing of various QEMU options
>
> afaik kvm autotest do the same. It's true that kvm autotest might look less
> friendly but its may only be this way since it has lots of options.

Actually, kvm-autotest has various layers of abstraction in how QEMU ends up 
being launched.  As you mention below, those layers are there to allow for 
things like using libvirt.

That's desirable when you're doing "virt testing", but not so desirably when 
you're trying to write specific unit tests against QEMU.

>> 5) The tests execute very quickly, can be run stand alone, and do not
>> require root privileges
>
> ditto for kvm auotest. It's possible to configure it w/o root too which is not a
> huge issue.

When I say, "run quickly", I mean, they execute very quickly.

$ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 
tests/virtio-serial.sh

real	0m4.385s
user	0m1.460s
sys	0m1.860s

I've used kvm-autotest a lot, there is no test in kvm-autotest that is even 
close to completing in 4 seconds start to finish.  I don't think kvm-autotest 
can even run without installing a guest first which puts puts a simple test more 
at the 30 minute mark.  If we're talking about TCG testing, then we're in hours 
territory.

If there's a way to run kvm-autotest as a non-privileged user, I've not figured 
it out yet.

Of course, kvm-autotest can test a lot more things than qemu-test can since the 
tests are guest agnostic.  That is the primary architectural difference.  By 
writing the tests to one specific guest (busybox initrd), they can be extremely 
fast in execution, but the tests are less general purpose.

That makes qemu-test very useful for development testing (such as long automated 
bisect runs) but less useful for acceptance testing.

> Please compare your own virtio-serial test w/ the autotest version of it:
> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
>
>
> This single file tests functionality, limits, console, live migration and
> performance. Of course one can add a very basic 'hello world' sanity test too
> that will run quickly and will identify basic breakage fast.

Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.

There is a lot of value in being able to write simple tests in 50 lines of code. 
  In fact, the largest qemu-test at the moment is only ~70 lines of code.  And 
besides finding regressions in my own code (which is the primary use atm), I've 
found and fixed a few real bugs in upstream QEMU.

For a couple hundred lines of bash script, how can it be anything but a good 
thing :-)

> Noways we abstract kvm autotest so libvirt will be optionally tested too w/ the
> same tests.
>
> Again, I'm not advocating that kvm autotest is a solution for anything but for
> plain guest-host communication, monitor commands, etc it's a really good tool.
>
> I agree that kvm autotest may be less friendly for developer users since it
> carries allot of options for testing a huge matrix. Lucas and Cleber are working
> these days to add a make kvmautotest target to qemu so one would be able to
> quickly execute autotest and we can think of additional parameters like
> sanity-set, migration-set, etc.

We (the QEMU project) need to get more serious about testing.  We need to 
(quickly) get to a point where we can mandate that features come with test 
cases.  It is extremely hard to implement touch everything features with no good 
way to test all of the different things we support.

kvm autotest cannot fill that role because it's too complicated and too many 
layers removed from QEMU.

virtio-console/virtio-serial-bus in about 1400 LOC in QEMU.  I'm not about to 
mandate that someone writes 2.2k LOC in a python test framework in order to get 
1.4k of code merged in QEMU.

But 50 lines of bash seems like a more than reasonable requirement.

Regards,

Anthony Liguori

>>
>> 6) They are random by nature with the ability to fix the seed in order
>> to be used in git-bisect.
>>
>> I think Gerd had been looking at doing something similar with a custom
>> initrd.
>>
>> I've tried to consider other architectures and had hoped that we could
>> commit the vmlinuz and initramfs into git so that it was easy to test
>> other architectures without having a full build environment.
>> Unfortunately, busybox doesn't link statically with glibc and I can't
>> see an obvious way to commit binaries while respecting the GPL since we
>> need to pull glibc into the initramfs.
>>
>> I know buildroot exists specifically to deal with this but in my
>> experience, buildroot is very unreliable and extremely heavy weight
>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>> environment.
>>
>> Anyway, the code is available at:
>>
>> http://git.qemu.org/qemu-test.git
>>
>> See the README for instructions on how to use it.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>