From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:47128) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgR8t-0006pw-WC for qemu-devel@nongnu.org; Thu, 29 Dec 2011 20:20:21 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RgR8r-0005fT-OR for qemu-devel@nongnu.org; Thu, 29 Dec 2011 20:20:19 -0500 Received: from mx1.redhat.com ([209.132.183.28]:61612) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgR8r-0005fH-93 for qemu-devel@nongnu.org; Thu, 29 Dec 2011 20:20:17 -0500 Message-ID: <4EFD11D0.4000107@redhat.com> Date: Thu, 29 Dec 2011 23:20:16 -0200 From: Lucas Meneghel Rodrigues MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> <4EF8FC88.70809@redhat.com> <4EFA4829.4000207@redhat.com> <4EFA80EA.3050405@codemonkey.ws> <4EFAA2A2.4000107@redhat.com> <4EFB2764.7040006@codemonkey.ws> <4EFB2F36.2090408@redhat.com> <4EFB46DD.4000905@codemonkey.ws> <4EFB5014.9030609@redhat.com> <4EFC7B72.9050802@redhat.com> <4EFC97C3.4080603@codemonkey.ws> <4EFC9AFF.5070707@redhat.com> <4EFC9D2B.1020606@codemonkey.ws> <4EFCF505.9030004@redhat.com> <4EFD06DC.4050409@codemonkey.ws> In-Reply-To: <4EFD06DC.4050409@codemonkey.ws> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: Stefan Hajnoczi , cleber@redhat.com, dlaor@redhat.com, qemu-devel , Gerd Hoffmann , Cleber Rosa , Avi Kivity On 12/29/2011 10:33 PM, Anthony Liguori wrote: > So I decided to do some snooping. Here are some stats: > > anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py > 150 balloon_check.py > 68 boot_savevm.py > 190 cdrom.py > 1875 cgroup.py > 111 cpu_hotplug.py > 170 enospc.py > 71 floppy.py > 72 getfd.py > 89 hdparm.py > 0 __init__.py > 615 ksm_overcommit.py > 107 migration_multi_host.py > 117 migration.py > 85 migration_with_file_transfer.py > 43 migration_with_reboot.py > 138 multi_disk.py > 62 nic_bonding.py > 104 nic_hotplug.py > 60 nmi_watchdog.py > 203 pci_hotplug.py > 182 physical_resources_check.py > 439 qemu_img.py > 48 qemu_iotests.py > 407 qmp_basic.py > 389 qmp_basic_rhel6.py > 54 set_link.py > 67 smbios_table.py > 356 stepmaker.py > 247 steps.py > 43 stop_continue.py > 31 system_reset_bootable.py > 181 timedrift.py > 96 timedrift_with_migration.py > 91 timedrift_with_reboot.py > 103 timedrift_with_stop.py > 28 unittest_kvmctl.py > 121 unittest.py > 90 usb.py > 2174 virtio_console.py > 85 vmstop.py > 9562 total There are more tests under client/virt/tests: [lmr@freedom tests]$ wc -l *.py 25 autotest.py 100 boot.py 42 build.py 32 clock_getres.py 235 ethtool.py 84 file_transfer.py 53 fillup_disk.py 76 guest_s4.py 80 guest_test.py 45 image_copy.py 0 __init__.py 136 iofuzz.py 31 ioquit.py 40 iozone_windows.py 127 jumbo.py 85 kdump.py 41 linux_s3.py 84 lvm.py 60 mac_change.py 46 module_probe.py 90 multicast.py 106 netperf.py 149 netstress_kill_guest.py 255 nfs_corrupt.py 63 nicdriver_unload.py 39 nic_promisc.py 73 ping.py 29 pxe.py 42 shutdown.py 147 softlockup.py 53 stress_boot.py 164 trans_hugepage_defrag.py 127 trans_hugepage.py 113 trans_hugepage_swapping.py 852 unattended_install.py 175 vlan.py 46 watchdog.py 136 whql_client_install.py 275 whql_submission.py 49 yum_update.py 4405 total Not to mention the infrastructure libraries: [lmr@freedom virt]$ wc -l *.py 1351 aexpect.py 705 base_installer.py 9 common.py 0 __init__.py 195 installer.py 64 installer_unittest.py 240 kvm_installer.py 921 kvm_monitor.py 1635 kvm_vm.py 324 libvirt_monitor.py 1271 libvirt_vm.py 110 passfd.py 237 ppm_utils.py 519 rss_client.py 527 virt_env_process.py 124 virt_http_server.py 51 virt_passfd_setup.py 229 virt_scheduler.py 1401 virt_step_editor.py 131 virt_test.py 315 virt_test_setup.py 813 virt_test_utils.py 3556 virt_utils.py 196 virt_utils_unittest.py 224 virt_video_maker.py 1032 virt_vm.py 16180 total There's a bit more outside this dir, not to mention the auxiliary data files. The support libraries account for roughly for the same number of lines of test code written, so we can say in very rough terms we have around 33 k of python code for kvm/libvirt tests. I agree it is more than an order of magnitude smaller than the qemu code base, but still it demands effort that has to be justified. > anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>" > kvm | sort -u > > Amos Kong > Chris Evich > Cleber Rosa > Jerry Tang > lmr > Lucas Meneghel Rodrigues > Lucas Meneghel Rodrigues > Lukas Doktor > mbligh > Onkar N Mahajan > pradeep > Qingtang Zhou > Thomas Jarosch > Yiqiao Pu > > Which leads me to the following conclusions: > > 1) No one outside of autotest developers is actually contributing tests > to kvm-autotest. Autotest was maintained using subversion until recently, all commits from people outside autotest were commited by autotest developers, the git tree was just a mirror until recently. Grepping for Signed-off-by: lines will give a much better idea of the contributors: [lmr@freedom autotest.git]$ git log client/virt | grep "Signed-off-by:" | sort -u Signed-off-by: Xu He Jie Signed-off-by: Alon Levy Signed-off-by: Amos Kong Signed-off-by: Chen Cao Signed-off-by: Chris Evich Signed-off-by: Cleber Rosa Signed-off-by: Feng Yang Signed-off-by: Gerd Hoffmann Signed-off-by: Jason D. Gaston Signed-off-by: Jason Wang Signed-off-by: Jiri Zupka Signed-off-by: Lucas Meneghel Rodrigues Signed-off-by: Lukas Doktor Signed-off-by: Martin Jenner Signed-off-by: Pradeep K Surisetty Signed-off-by: Pradeep Kumar Surisetty Signed-off-by: Qingtang Zhou Signed-off-by: Quentin Deldycke Signed-off-by: Ray Chauvin Signed-off-by: Satheesh Rajendran Signed-off-by: Suqin Huang Signed-off-by: Wei Yang Signed-off-by: Xu He Jie Signed-off-by: Yiqiao Pu Signed-off-by: Yogananth Subramanian Signed-off-by: Yolkfull Chow [lmr@freedom autotest.git]$ git log client/tests/kvm | grep "Signed-off-by:" | sort -u Signed-off-by: Alon Levy Signed-off-by: Amos Kong Signed-off-by: Avi Kivity Signed-off-by: Bear Yang Signed-off-by: Cao, Chen Signed-off-by: Chen Cao Signed-off-by: Chris Evich Signed-off-by: Cleber Rosa Signed-off-by: David Huff Signed-off-by: Eduardo Habkost Signed-off-by: Feng Yang Signed-off-by: Gerd Hoffmann Signed-off-by: Jason Wang Signed-off-by: Jerry Tang Signed-off-by: Jes Sorensen Signed-off-by: Jin Bing Guo Signed-off-by: Jiri Zupka Signed-off-by: Ken Cao Signed-off-by: Lucas Meneghel Rodrigues Signed-off-by: Lukas Doktor Signed-off-by: Marcelo Tosatti Signed-off-by: Martin Jenner Signed-off-by: Michael Goldish Signed-off-by: Michael S. Tsirkin Signed-off-by: Onkar N Mahajan Signed-off-by: Pradeep K Surisetty Signed-off-by: Qingtang Zhou Signed-off-by: Ryan Harper Signed-off-by: Shuxi Shang Signed-off-by: Sudhir Kumar Signed-off-by: Supriya Kannery Signed-off-by: Suqin Huang Signed-off-by: Tang Chen Signed-off-by: Thomas Jarosch Signed-off-by: Uri Lublin Signed-off-by: Yiqiao Pu Signed-off-by: Yogananth Subramanian Signed-off-by: Yolkfull Chow So, not quite what you wanted to demonstrate. > 2) Most of the tests are relatively small and simple enough that they > could be trivially ported to a stand alone utility. For instance, > looking at the pci_hotplug.py, it just executes monitor commands and > does basic pci enumeration in the guest. They do depend on infrastructure code to be simple and small. The infrastructure code was written using autotest libraries, so although it is possible to make it stand alone, still there's the problem of having these library/classes out of sync with autotest evolving. > I don't see a lot of pitfalls here. At the end of the day, we're not > talking about all that much code. Even so (you see it's not so small), it demands effort that has to be well justified, as I pointed out. I'm starting to believe it might be worth the effort. > It's not about where they live, it's about the ability to execute them > in a simple and direct fashion. Ok, fair enough, we'll see what we can do, although I wish it were a simpler problem, see below. >> You have a point with regards to having the test cases more stand >> alone, but >> it's not as easy as one would think mainly because of the large amount >> of user >> configurable options that we have to pass to the tests. > > Configuration options are really the devil when it comes to testing. As > a developer, I want my test suite to just run and tell me if I have > bugs. I have no interest in configuring it to run a certain way. I don't > want to think that much about what I'm testing, I just want the tests to > run and tell me if I screwed something up. Still, you need to tell your tests, say, the format of the terminal prompt for Fedora might be slightly different of the [insert-distro-here], that the src and destination paths on a file transfer test are different for Windows and Linux guests, that the output of a given command when you are running a test with a virtio card differs from the output when you are using rtl8139, that you want to build qemu from git repo X, or from brew build Y, using a custom version of seabios, and so on and so forth.