Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues <lmr@redhat.com>]

On 03/08/2012 06:24 PM, Anthony Liguori wrote:
>>
>> Cons:
>> - Lot of code will be duplicated to cover the main code paths:
>> writting tests will require writting/supporting considerable
>> ammount of code (that already exists in autotest).
>
> Again, existence proof that this isn't true.

Case in point, the virtio test (that uses an auxiliary script to send 
data to the host). Can you tell me if both tests cover even remotely the 
same amount of functionality?

https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py

https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py

Here is the qemu-test version

http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master

What the qemu-test version covers:
  * host starts qemu with one virtio console device backed by a file
  * guest verifies if the name of the device is correct
  * guest writes to the console device
  * host verifies if guest wrote to the virtio console

What the virtio-console covers:

  * Sends data between host and guest back and forth, validates the data 
being sent, for both small and large amounts of data, both random or 
sequential.
  * Tests write/send in blocking, polling, selecting mode, with port 
mode sync/async
  * Verifies if the maximum amount of ports was created and it's 
available in the guest
  * Tries lseek on the device
  * Verifies if concomitant access to a single port passes or fails
  * Verifies data throughput and resource utilization
  * Repeats the data transfer with live migration to see if the port 
connections survive
  * Keeps an eye on the guest OS to see if we have kernel panics, and if 
it does, it'll clean up and put the guest in a working state so other 
functionality can be checked
  * Probably something else I'm forgetting right now.

Good luck implementing that in a shell script. I'd love to see how you 
implement that amount of coverage in less lines in shell.

So, more coverage, more code. It's as simple as that. We don't write 
code just for the sake of it.

>> - QE will be alienated from the qemu test effort. There will be
>> no integration between the QE efforts and the maintenance of
>> the qemu developer-level tests.
>
> I think we're a pretty friendly and open community :-) There is no
> reason that QE should be "alienated" unless folks are choosing not to
> participate upstream.
>
>> - You don't get the goodies from autotest (standardized system
>> info collection, video recording, grid support, etc).
>> - The new tests will work only on the master branch.
>
> This last one is a legitimate point that I have considered myself. But I
> think the value of having tests in HEAD outweigh the cost here.
>
>> 3. A mix of (1) and (2): we "move" everything under qemu.git, but
>> keep the compatibility layer and provide a "libqemutest" for
>> third-parties.
>>
>> Anybody writting a test that interacts with qemu will use this
>> library: qemu, libvirt, spice, ovirt, QE tests, etc.
>
> I really see this all as over architecting to be honest.
>
> Can someone explain in clear terms (without appealing to maturity,
> flexibility, or any other qualitative aspect) why it takes anything more
> than a few dozen shell functions to write all of the tests that are in
> kvm-autotest test today?

Clearly as your requirements are different than the ones we had when the 
project was written, qemu-test doesn't need any of the stuff present 
there and you can do it all with a handful of shell script functions.

But to do cover the same things covered in autotest today with the same 
requirements, I really doubt you can do the same with the same handful 
of shell script functions. See the example about the virtio test above, 
not to mention other functionality we have with the tests, such as make 
possible to do tests run with a migration thread going on the 
background, while it's plugging/unplugging devices, running a stress 
test or ltp, or a benchmark, or measuring the time drift experienced by 
the guest.