Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori <anthony@codemonkey.ws>]

On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:
> On 03/08/2012 06:24 PM, Anthony Liguori wrote:
>>>
>>> Cons:
>>> - Lot of code will be duplicated to cover the main code paths:
>>> writting tests will require writting/supporting considerable
>>> ammount of code (that already exists in autotest).
>>
>> Again, existence proof that this isn't true.
>
> Case in point, the virtio test (that uses an auxiliary script to send data to
> the host). Can you tell me if both tests cover even remotely the same amount of
> functionality?
>
> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
>
>
> https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py
>
>
> Here is the qemu-test version
>
> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master

I seem to recall us having this same discussion in the past....  But 
nevertheless, I started to respond in detail but decided to wait until tomorrow. 
  It's been a long day and I think this thread is a bit too heated already.  I 
think I'll be a bit less grumpy in the morning :-)

But there's one thing I want to point out...

> What the qemu-test version covers:
> * host starts qemu with one virtio console device backed by a file
> * guest verifies if the name of the device is correct
> * guest writes to the console device
> * host verifies if guest wrote to the virtio console
>
> What the virtio-console covers:
>
> * Sends data between host and guest back and forth, validates the data being
> sent, for both small and large amounts of data, both random or sequential.
> * Tests write/send in blocking, polling, selecting mode, with port mode sync/async

This bullet (and many of the bullets) below are not tests of QEMU.  They are 
tests of the Linux kernel.  This is an integration test, not a functional test 
of QEMU.

We absolutely need integration tests and hands down, that's where autotest is 
going to shine.  This is clearly not in the scope of qemu-test and there's been 
a few discussions on the ML on this including me rejecting a patch that crossed 
this boundary.

Right now autotest effectively does everything because we (QEMU) have not 
fulfilled our responsibility here.

Regards,

Anthony Liguori