From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:37836) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgQPj-0001sJ-Bn for qemu-devel@nongnu.org; Thu, 29 Dec 2011 19:33:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RgQPh-0006QN-IH for qemu-devel@nongnu.org; Thu, 29 Dec 2011 19:33:39 -0500 Received: from mail-gy0-f173.google.com ([209.85.160.173]:38956) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgQPh-0006QG-Db for qemu-devel@nongnu.org; Thu, 29 Dec 2011 19:33:37 -0500 Received: by ghbg16 with SMTP id g16so5032757ghb.4 for ; Thu, 29 Dec 2011 16:33:36 -0800 (PST) Message-ID: <4EFD06DC.4050409@codemonkey.ws> Date: Thu, 29 Dec 2011 18:33:32 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> <4EF8FC88.70809@redhat.com> <4EFA4829.4000207@redhat.com> <4EFA80EA.3050405@codemonkey.ws> <4EFAA2A2.4000107@redhat.com> <4EFB2764.7040006@codemonkey.ws> <4EFB2F36.2090408@redhat.com> <4EFB46DD.4000905@codemonkey.ws> <4EFB5014.9030609@redhat.com> <4EFC7B72.9050802@redhat.com> <4EFC97C3.4080603@codemonkey.ws> <4EFC9AFF.5070707@redhat.com> <4EFC9D2B.1020606@codemonkey.ws> <4EFCF505.9030004@redhat.com> In-Reply-To: <4EFCF505.9030004@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Lucas Meneghel Rodrigues Cc: Stefan Hajnoczi , cleber@redhat.com, dlaor@redhat.com, qemu-devel , Gerd Hoffmann , Cleber Rosa , Avi Kivity On 12/29/2011 05:17 PM, Lucas Meneghel Rodrigues wrote: > On 12/29/2011 03:02 PM, Anthony Liguori wrote: >> On 12/29/2011 10:53 AM, Avi Kivity wrote: >>> On 12/29/2011 06:39 PM, Anthony Liguori wrote: >>> >>> It might have made sense to split the kvm-testing functionality of >>> autotest, and have autotest drive that. We could even have called it >>> qemu-test. >> >> I specifically advocated this during Lucas' KVM Forum talk and he was >> strongly opposed to it. > > Ok, this is a point where I have failed in communicating things. > > I've watched the presentation on video just to see if my memory was not > betraying me, and I wouldn't say I was 'strongly opposed', it's just that there > are some practical problems, not impossible to overcome of course, but worth > thinking. So I decided to do some snooping. Here are some stats: anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py 150 balloon_check.py 68 boot_savevm.py 190 cdrom.py 1875 cgroup.py 111 cpu_hotplug.py 170 enospc.py 71 floppy.py 72 getfd.py 89 hdparm.py 0 __init__.py 615 ksm_overcommit.py 107 migration_multi_host.py 117 migration.py 85 migration_with_file_transfer.py 43 migration_with_reboot.py 138 multi_disk.py 62 nic_bonding.py 104 nic_hotplug.py 60 nmi_watchdog.py 203 pci_hotplug.py 182 physical_resources_check.py 439 qemu_img.py 48 qemu_iotests.py 407 qmp_basic.py 389 qmp_basic_rhel6.py 54 set_link.py 67 smbios_table.py 356 stepmaker.py 247 steps.py 43 stop_continue.py 31 system_reset_bootable.py 181 timedrift.py 96 timedrift_with_migration.py 91 timedrift_with_reboot.py 103 timedrift_with_stop.py 28 unittest_kvmctl.py 121 unittest.py 90 usb.py 2174 virtio_console.py 85 vmstop.py 9562 total anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>" kvm | sort -u Amos Kong Chris Evich Cleber Rosa Jerry Tang lmr Lucas Meneghel Rodrigues Lucas Meneghel Rodrigues Lukas Doktor mbligh Onkar N Mahajan pradeep Qingtang Zhou Thomas Jarosch Yiqiao Pu Which leads me to the following conclusions: 1) No one outside of autotest developers is actually contributing tests to kvm-autotest. 2) Most of the tests are relatively small and simple enough that they could be trivially ported to a stand alone utility. For instance, looking at the pci_hotplug.py, it just executes monitor commands and does basic pci enumeration in the guest. I don't see a lot of pitfalls here. At the end of the day, we're not talking about all that much code. > > * We rely on a fair bit of autotest libraries/API, so by extracting the APIs we > rely on to start a new project we'd be creating some code duplication, and > updates/bugfixes on the autotest API wouldn't be easily carried on to the new > project. > * I'm not sure that this per se would fix the perceived problems with the tests. > It'd be nearly the same code, just outside the autotest tree, I fail to see how > that would be much better, and therefore, justify the work on doing this. Of > course I might be wrong. It's not about where they live, it's about the ability to execute them in a simple and direct fashion. > >> I think kvm autotest would get a lot more interest if the test cases >> were pulled out of autotest, made more stand alone. They also should be >> more Unix like being divided into individual executables with >> independent command line options over a single do-everything >> configuration file. > > You have a point with regards to having the test cases more stand alone, but > it's not as easy as one would think mainly because of the large amount of user > configurable options that we have to pass to the tests. Configuration options are really the devil when it comes to testing. As a developer, I want my test suite to just run and tell me if I have bugs. I have no interest in configuring it to run a certain way. I don't want to think that much about what I'm testing, I just want the tests to run and tell me if I screwed something up. Regards, Anthony Liguori > This is a place where > the config file format shines, since we just have to override some parameters on > different variants to have a test working among Windows and Linux guests, for > example, all expressed in a concise way. The config files might look enormous, > but they are actually pretty small compared with the amount of test combinations > we can get out of it. This is nothing I'd call insurmountable as well. > > If I get more feedback of people saying "Ok, this would make things better for > me and make me more interested", then we'll go that route. I'm sorry if I gave > the impression I was strongly opposed to your idea. >