Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori <anthony@codemonkey.ws>]

On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:
> On 03/08/2012 06:24 PM, Anthony Liguori wrote:
>>>
>>> Cons:
>>> - Lot of code will be duplicated to cover the main code paths:
>>> writting tests will require writting/supporting considerable
>>> ammount of code (that already exists in autotest).
>>
>> Again, existence proof that this isn't true.
>
> Case in point, the virtio test (that uses an auxiliary script to send data to
> the host). Can you tell me if both tests cover even remotely the same amount of
> functionality?
>
> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
>
>
> https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py
>
>
> Here is the qemu-test version
>
> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master

So virtio-serial is an exception in autotest:

2174 virtio_console.py
1875 cgroup.py
615 ksm_overcommit.py
439 qemu_img.py
407 qmp_basic.py
389 qmp_basic_rhel6.py
356 stepmaker.py
247 steps.py

// The next largest actual test of QEMU
203 pci_hotplug.py
190 cdrom.py
182 physical_resources_check.py
181 timedrift.py
170 enospc.py
150 balloon_check.py
138 multi_disk.py
121 unittest.py
117 migration.py
111 cpu_hotplug.py
107 migration_multi_host.py
104 nic_hotplug.py
103 timedrift_with_stop.py
96 timedrift_with_migration.py
...

So picking virtio-serial as your comparison point is not really representative 
of kvm-autotest but at any rate...

>
>
> What the qemu-test version covers:
> * host starts qemu with one virtio console device backed by a file
> * guest verifies if the name of the device is correct
> * guest writes to the console device
> * host verifies if guest wrote to the virtio console
>
> What the virtio-console covers:
>
> * Sends data between host and guest back and forth, validates the data being
> sent, for both small and large amounts of data, both random or sequential.

This is an improvement.  qemu-test has shared random seed so adding a mode like 
this wouldn't be hard.

> * Tests write/send in blocking, polling, selecting mode, with port mode sync/async

This is a Linux regression test, not a QEMU test.

> * Verifies if the maximum amount of ports was created and it's available in the
> guest

This is an improvement and would be trivial to add.

> * Tries lseek on the device

This is a Linux regression test.

> * Verifies if concomitant access to a single port passes or fails

I think this is also a Linux test.

> * Verifies data throughput and resource utilization

This is outside the scope of qemu-test.

> * Repeats the data transfer with live migration to see if the port connections
> survive

This is neat and an improvement.  I haven't attempted to do migration testing 
with qemu-test.

> * Keeps an eye on the guest OS to see if we have kernel panics, and if it does,
> it'll clean up and put the guest in a working state so other functionality can
> be checked

This is a Linux test.

> * Probably something else I'm forgetting right now.
>
> Good luck implementing that in a shell script. I'd love to see how you implement
> that amount of coverage in less lines in shell.
>
> So, more coverage, more code. It's as simple as that. We don't write code just
> for the sake of it.

I hope it's clear though that most of what you're doing here is an integration 
test with Linux.  This is exactly what autotest should be doing IMHO and exactly 
not what we should be doing in qemu.git.

>> I really see this all as over architecting to be honest.
>>
>> Can someone explain in clear terms (without appealing to maturity,
>> flexibility, or any other qualitative aspect) why it takes anything more
>> than a few dozen shell functions to write all of the tests that are in
>> kvm-autotest test today?
>
> Clearly as your requirements are different than the ones we had when the project
> was written, qemu-test doesn't need any of the stuff present there and you can
> do it all with a handful of shell script functions.
>
> But to do cover the same things covered in autotest today with the same
> requirements, I really doubt you can do the same with the same handful of shell
> script functions. See the example about the virtio test above, not to mention
> other functionality we have with the tests, such as make possible to do tests
> run with a migration thread going on the background, while it's
> plugging/unplugging devices, running a stress test or ltp, or a benchmark, or
> measuring the time drift experienced by the guest.

That's fine.  Your requirements are different and that's why you need autotest. 
  But we don't have the same requirements in qemu.git.

Regards,

Anthony Liguori