Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori <anthony@codemonkey.ws>]

On 12/29/2011 05:17 PM, Lucas Meneghel Rodrigues wrote:
> On 12/29/2011 03:02 PM, Anthony Liguori wrote:
>> On 12/29/2011 10:53 AM, Avi Kivity wrote:
>>> On 12/29/2011 06:39 PM, Anthony Liguori wrote:
>>>
>>> It might have made sense to split the kvm-testing functionality of
>>> autotest, and have autotest drive that. We could even have called it
>>> qemu-test.
>>
>> I specifically advocated this during Lucas' KVM Forum talk and he was
>> strongly opposed to it.
>
> Ok, this is a point where I have failed in communicating things.
>
> I've watched the presentation on video just to see if my memory was not
> betraying me, and I wouldn't say I was 'strongly opposed', it's just that there
> are some practical problems, not impossible to overcome of course, but worth
> thinking.

So I decided to do some snooping.  Here are some stats:

anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py
    150 balloon_check.py
     68 boot_savevm.py
    190 cdrom.py
   1875 cgroup.py
    111 cpu_hotplug.py
    170 enospc.py
     71 floppy.py
     72 getfd.py
     89 hdparm.py
      0 __init__.py
    615 ksm_overcommit.py
    107 migration_multi_host.py
    117 migration.py
     85 migration_with_file_transfer.py
     43 migration_with_reboot.py
    138 multi_disk.py
     62 nic_bonding.py
    104 nic_hotplug.py
     60 nmi_watchdog.py
    203 pci_hotplug.py
    182 physical_resources_check.py
    439 qemu_img.py
     48 qemu_iotests.py
    407 qmp_basic.py
    389 qmp_basic_rhel6.py
     54 set_link.py
     67 smbios_table.py
    356 stepmaker.py
    247 steps.py
     43 stop_continue.py
     31 system_reset_bootable.py
    181 timedrift.py
     96 timedrift_with_migration.py
     91 timedrift_with_reboot.py
    103 timedrift_with_stop.py
     28 unittest_kvmctl.py
    121 unittest.py
     90 usb.py
   2174 virtio_console.py
     85 vmstop.py
   9562 total

anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>" kvm | sort -u

Amos Kong <akong@redhat.com>
Chris Evich <cevich@redhat.com>
Cleber Rosa <crosa@redhat.com>
Jerry Tang <jtang@suse.com>
lmr <lmr@592f7852-d20e-0410-864c-8624ca9c26a4>
Lucas Meneghel Rodrigues <lmr@redhat.com>
Lucas Meneghel Rodrigues <lookkas@gmail.com>
Lukas Doktor <ldoktor@redhat.com>
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4>
Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com>
pradeep <psuriset@linux.vnet.ibm.com>
Qingtang Zhou <qzhou@redhat.com>
Thomas Jarosch <thomas.jarosch@intra2net.com>
Yiqiao Pu <ypu@redhat.com>

Which leads me to the following conclusions:

1) No one outside of autotest developers is actually contributing tests to 
kvm-autotest.

2) Most of the tests are relatively small and simple enough that they could be 
trivially ported to a stand alone utility.  For instance, looking at the 
pci_hotplug.py, it just executes monitor commands and does basic pci enumeration 
in the guest.

I don't see a lot of pitfalls here.  At the end of the day, we're not talking 
about all that much code.

>
> * We rely on a fair bit of autotest libraries/API, so by extracting the APIs we
> rely on to start a new project we'd be creating some code duplication, and
> updates/bugfixes on the autotest API wouldn't be easily carried on to the new
> project.
> * I'm not sure that this per se would fix the perceived problems with the tests.
> It'd be nearly the same code, just outside the autotest tree, I fail to see how
> that would be much better, and therefore, justify the work on doing this. Of
> course I might be wrong.

It's not about where they live, it's about the ability to execute them in a 
simple and direct fashion.

>
>> I think kvm autotest would get a lot more interest if the test cases
>> were pulled out of autotest, made more stand alone. They also should be
>> more Unix like being divided into individual executables with
>> independent command line options over a single do-everything
>> configuration file.
>
> You have a point with regards to having the test cases more stand alone, but
> it's not as easy as one would think mainly because of the large amount of user
> configurable options that we have to pass to the tests.

Configuration options are really the devil when it comes to testing.  As a 
developer, I want my test suite to just run and tell me if I have bugs.  I have 
no interest in configuring it to run a certain way.  I don't want to think that 
much about what I'm testing, I just want the tests to run and tell me if I 
screwed something up.

Regards,

Anthony Liguori

> This is a place where
> the config file format shines, since we just have to override some parameters on
> different variants to have a test working among Windows and Linux guests, for
> example, all expressed in a concise way. The config files might look enormous,
> but they are actually pretty small compared with the amount of test combinations
> we can get out of it. This is nothing I'd call insurmountable as well.
>
> If I get more feedback of people saying "Ok, this would make things better for
> me and make me more interested", then we'll go that route. I'm sorry if I gave
> the impression I was strongly opposed to your idea.
>