Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori <anthony@codemonkey.ws>]

On 03/09/2012 08:43 AM, Paolo Bonzini wrote:
> Il 09/03/2012 14:56, Anthony Liguori ha scritto:
>> On 03/09/2012 07:07 AM, Paolo Bonzini wrote:
>>> Il 08/03/2012 22:24, Anthony Liguori ha scritto:
>>>> The qemu-test tests are smaller than the corresponding autotest
>>>> tests.
>>>
>>> They also do much less.
>>>
>>> It's true that a combination of qemu-test + qtests could do 99% of
>>> the job more simply than autotest.  But the last 1% (including
>>> migration) would require a large effort, while it would be just
>>> there in autotest.
>>
>> Can you clarify here?  When you say "migration", what I think you
>> mean is that kvm-autotest has the ability to run autotest client
>> tests within a guest while migration.
>
> 1) Migration is quite different from other tests in terms of VM
> lifecycle, even if you use migrate-to-file + restore.  It requires
> special infrastructure in the drivers.
>
> 2) Migration bugs are subtle and often nondeterministic, so they require
> long running tests that do the same thing over and over again.  These
> requirements are more similar to integration tests than unit tests.  So
> integration tests cannot really be dismissed.

I think migration really requires special test support in QEMU.  I've long 
wanted to introduce a migration torture mode that would constantly tear down the 
device model and then restore it during normal operation.

I would then want to make -migration-torture an option to any test.

> BTW, I don't see the problem with requiring a large "make check" time.
> I probably already said this: when I was working on GCC, I ran tests
> overnight every time I had piled up enough changes, and it was simply
> not possible to skip that step because compiler changes often had
> surprising ramifications.  Yes, in some cases I had a bad surprise the
> following day, but it's unavoidable.
>
> Even today, bootstrapping and testing GCC on embedded systems can be
> done but it takes 4-5 *days*.  Of course you do it first in a
> cross-compiler, but surprises still can happen.

I think it's fair to have multiple levels of check.  There should be a short 
sanity check one at least but I have no problem with make check-full taking a 
long period of time.

Regards,

Anthony Liguori

> But we're lucky: compilers and QEMU are pretty much the opposite, and a
> bad surprise should be much more rare here.  QEMU tests and
> infrastructure are harder, but the components are delimited much better
> than in a compiler with 200 cascading passes, which is what makes qtest
> possible at all.
>
> Paolo