Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Cleber Rosa <crosa@redhat.com>]

On 03/09/2012 12:48 PM, Anthony Liguori wrote:
> On 03/09/2012 09:34 AM, Paolo Bonzini wrote:
>> Il 09/03/2012 16:24, Anthony Liguori ha scritto:
>>>> At the very least dump the inquiry pages, mode pages, etc. and see 
>>>> that
>>>> they make sense and correspond to the device properties.
>>>
>>> Is this not something that's reasonably easy to do in qtest?
>>
>> Yes (at least with virtio-scsi the libos bits are relatively small; just
>> think of what it would have been like when the only HBA was LSI), but
>> with one gotcha...
>>
>>> Is it possible to write a C program that does the ioctl and dump the
>>> inquiry page in a text format conducive to shell parsing?
>>
>> ... sg_utils also parses the pages and dumps them in human-readable
>> format.  This is useful because it provides a completely separate
>> implementation and avoids problems with misinterpretation of the
>> standard.  Of course it would work just as well if someone wrote tests
>> instead of me.
>
> I don't recommend it in the general case, but it should be trivial to 
> add additional packages to qemu-jeos and reuse the toolchain to build 
> them.
>

And then the two code bases (now buildroot and qemu-jeos) would have and 
increasing number of similar features.

> I don't think it's all that valuable here.  I think you really want to 
> test this via qtest.  You could easy copy/paste code from sg_utils to 
> do the parsing if you were so inclined...

And yet more code duplication. Even if a few KLOCs, but still... 
Hopefully I'm not the only one that fears those suggestions becoming real.

>
>>>>> Are these the sort of tests that would be interesting to also run on
>>>>> Fedora, Windows, and Ubuntu?
>>>>
>>>> They should give exactly the same output on any guest.
>>>
>>> Is it valuable to have a per-platform test or since this is mostly
>>> passthrough to the device (I assume), do you just need a single test?
>>
>> Ah, understood.  Yeah, a single test is enough for the purpose of
>> testing QEMU.  If you want to test the driver too, running under Windows
>> would be useful.
>
> We should separate integration test (testing multiple components where 
> we want to be able to use different versions/implementations of one 
> component) from functional/unit tests that are strictly testing a 
> single component.  There isn't always a clean separation and there may 
> be overlap, but I don't think we should stress out about the overlap.
>
> For instance, doing general purpose I/O testing is something where we 
> want to test I/O with a Linux guest, a Windows guest, etc.  This is an 
> integration test and we should focus on it.
>
> But we probably want some sort of I/O test in qemu-test too since it 
> provides a simple functional test.  Yes, there's overlap, but the 
> functional form of the test is so simple that it really isn't that 
> important.
>
> But testing how QEMU handles malformatted virtio-blk requests is 
> something that's clearly QEMU specific.  There's no value doing that 
> with a Linux and Windows guest (even if it's somehow possible).  It's 
> definitely a unit test that's strictly specific to QEMU.
>
> I think autotest should be able to execute QEMU's unit and functional 
> tests.  By using a test framework like gtest that exposes a test 
> protocol, it should be trivial to add that support.
>
> Regards,
>
> Anthony Liguori
>
>> Paolo
>