Re: autotest (was Re: [Qemu-devel] Re: [Qemu-commits] [COMMIT f80f9ec] Convert machine registration to use module initfunctions)

[Dor Laor <dlaor@redhat.com>]

Anthony Liguori wrote:
> Dor Laor wrote:
>> Anthony Liguori wrote:
>>> Avi Kivity wrote:
>>>> Anthony Liguori wrote:
>>>>> Autotest doesn't currently test the regressions that I want to test.  
>>>>
>>>> Which regressions do you want to test?
>>>
>>> For instance, we often have networking regressions.  In particular, 
>>> with the refactoring going on in slirp and tun/tap, I'd really like 
>>> to have an automated way to test slirp and tap with TCP 
>>> transmit/receive, and slirp redir.  I have something locally to do 
>>> this that I intend on posting in the next few days.
>>>
>> Current kvm autotest do test slirp. The step-files for guest 
>> installation open ssh/telnet access in order
>> to let the host reach them.
>> We trapped exactly this regression using autotest.
>
> What I want is a single test that completes quickly.  This means a 
> preconfigured guest that I can run a quick test case against.
>
> Installation tests a lot of things.  I'm looking for more targeted tests.
No doubt unit tests and some tools run manually are the best.
Nevertheless, the projects needs fully automated test framework that 
test it all.
Even today you do not have to go through the guest install each time.
Guest images are not erased after the install and autotest itself spawn 
additional
tests over this image. Even a VM can be kept alive at the end of a test, 
waiting for
additional tests to be executed over it.

In the future, I envision that you can go to auto-serv web interface, 
manually pick
a host and a test case and run it. The test will have built-in 
dependencies, and
if the VM is installed and alive it would just run the test as quick as 
possible.
Otherwise it will automatically do the dependency chain (or prompt for it).
>
>>> I wouldn't require kvm-autotest, but I'd like to have an interest 
>>> set of unit tests that are then usable within kvm-autotest.  Once we 
>>> have that, asking politely for tests to be written I think is quite 
>>> reasonable.
>> Some features are easily tested using a large framework, for example 
>> migration, slirp, time drift, etc.
>> In addition kvm's tests unit tests, qemu-io and similar should be 
>> written too in order to test various cases that
>> are only reasonable to be found using low level operations.
>> No doubt all of the unit tests should be executed from autotest.
>
> Yup, I agree 100%.
>
> Regards,
>
> Anthony Liguori