Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori <anthony@codemonkey.ws>]

On 03/09/2012 09:34 AM, Paolo Bonzini wrote:
> Il 09/03/2012 16:24, Anthony Liguori ha scritto:
>>> At the very least dump the inquiry pages, mode pages, etc. and see that
>>> they make sense and correspond to the device properties.
>>
>> Is this not something that's reasonably easy to do in qtest?
>
> Yes (at least with virtio-scsi the libos bits are relatively small; just
> think of what it would have been like when the only HBA was LSI), but
> with one gotcha...
>
>> Is it possible to write a C program that does the ioctl and dump the
>> inquiry page in a text format conducive to shell parsing?
>
> ... sg_utils also parses the pages and dumps them in human-readable
> format.  This is useful because it provides a completely separate
> implementation and avoids problems with misinterpretation of the
> standard.  Of course it would work just as well if someone wrote tests
> instead of me.

I don't recommend it in the general case, but it should be trivial to add 
additional packages to qemu-jeos and reuse the toolchain to build them.

I don't think it's all that valuable here.  I think you really want to test this 
via qtest.  You could easy copy/paste code from sg_utils to do the parsing if 
you were so inclined...

>>>> Are these the sort of tests that would be interesting to also run on
>>>> Fedora, Windows, and Ubuntu?
>>>
>>> They should give exactly the same output on any guest.
>>
>> Is it valuable to have a per-platform test or since this is mostly
>> passthrough to the device (I assume), do you just need a single test?
>
> Ah, understood.  Yeah, a single test is enough for the purpose of
> testing QEMU.  If you want to test the driver too, running under Windows
> would be useful.

We should separate integration test (testing multiple components where we want 
to be able to use different versions/implementations of one component) from 
functional/unit tests that are strictly testing a single component.  There isn't 
always a clean separation and there may be overlap, but I don't think we should 
stress out about the overlap.

For instance, doing general purpose I/O testing is something where we want to 
test I/O with a Linux guest, a Windows guest, etc.  This is an integration test 
and we should focus on it.

But we probably want some sort of I/O test in qemu-test too since it provides a 
simple functional test.  Yes, there's overlap, but the functional form of the 
test is so simple that it really isn't that important.

But testing how QEMU handles malformatted virtio-blk requests is something 
that's clearly QEMU specific.  There's no value doing that with a Linux and 
Windows guest (even if it's somehow possible).  It's definitely a unit test 
that's strictly specific to QEMU.

I think autotest should be able to execute QEMU's unit and functional tests.  By 
using a test framework like gtest that exposes a test protocol, it should be 
trivial to add that support.

Regards,

Anthony Liguori

> Paolo