From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:56145) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ri4nP-0002KF-Nr for qemu-devel@nongnu.org; Tue, 03 Jan 2012 08:53:00 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Ri4nK-00071n-Bf for qemu-devel@nongnu.org; Tue, 03 Jan 2012 08:52:55 -0500 Received: from mail-tul01m020-f173.google.com ([209.85.214.173]:44723) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ri4nK-00071f-26 for qemu-devel@nongnu.org; Tue, 03 Jan 2012 08:52:50 -0500 Received: by obcwp18 with SMTP id wp18so13785403obc.4 for ; Tue, 03 Jan 2012 05:52:49 -0800 (PST) Message-ID: <4F03082B.2080801@codemonkey.ws> Date: Tue, 03 Jan 2012 07:52:43 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> <4EF8FC88.70809@redhat.com> <4EFA4829.4000207@redhat.com> <4EFA80EA.3050405@codemonkey.ws> <4EFAA2A2.4000107@redhat.com> <4EFB2764.7040006@codemonkey.ws> <4EFB2F36.2090408@redhat.com> <4EFB46DD.4000905@codemonkey.ws> <4EFB5014.9030609@redhat.com> <4EFC7B72.9050802@redhat.com> <4EFC97C3.4080603@codemonkey.ws> <4EFC9AFF.5070707@redhat.com> <4EFC9D2B.1020606@codemonkey.ws> <4EFCF505.9030004@redhat.com> <4EFD06DC.4050409@codemonkey.ws> <4EFD11D0.4000107@redhat.com> In-Reply-To: <4EFD11D0.4000107@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Lucas Meneghel Rodrigues Cc: Stefan Hajnoczi , cleber@redhat.com, dlaor@redhat.com, qemu-devel , Gerd Hoffmann , Cleber Rosa , Avi Kivity On 12/29/2011 07:20 PM, Lucas Meneghel Rodrigues wrote: > On 12/29/2011 10:33 PM, Anthony Liguori wrote: >> So I decided to do some snooping. Here are some stats: >> >> anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py >> 150 balloon_check.py >> 68 boot_savevm.py >> 190 cdrom.py >> 1875 cgroup.py >> 111 cpu_hotplug.py >> 170 enospc.py >> 71 floppy.py >> 72 getfd.py >> 89 hdparm.py >> 0 __init__.py >> 615 ksm_overcommit.py >> 107 migration_multi_host.py >> 117 migration.py >> 85 migration_with_file_transfer.py >> 43 migration_with_reboot.py >> 138 multi_disk.py >> 62 nic_bonding.py >> 104 nic_hotplug.py >> 60 nmi_watchdog.py >> 203 pci_hotplug.py >> 182 physical_resources_check.py >> 439 qemu_img.py >> 48 qemu_iotests.py >> 407 qmp_basic.py >> 389 qmp_basic_rhel6.py >> 54 set_link.py >> 67 smbios_table.py >> 356 stepmaker.py >> 247 steps.py >> 43 stop_continue.py >> 31 system_reset_bootable.py >> 181 timedrift.py >> 96 timedrift_with_migration.py >> 91 timedrift_with_reboot.py >> 103 timedrift_with_stop.py >> 28 unittest_kvmctl.py >> 121 unittest.py >> 90 usb.py >> 2174 virtio_console.py >> 85 vmstop.py >> 9562 total > > There are more tests under client/virt/tests: > > [lmr@freedom tests]$ wc -l *.py > 25 autotest.py > 100 boot.py > 42 build.py > 32 clock_getres.py > 235 ethtool.py > 84 file_transfer.py > 53 fillup_disk.py > 76 guest_s4.py > 80 guest_test.py > 45 image_copy.py > 0 __init__.py > 136 iofuzz.py > 31 ioquit.py > 40 iozone_windows.py > 127 jumbo.py > 85 kdump.py > 41 linux_s3.py > 84 lvm.py > 60 mac_change.py > 46 module_probe.py > 90 multicast.py > 106 netperf.py > 149 netstress_kill_guest.py > 255 nfs_corrupt.py > 63 nicdriver_unload.py > 39 nic_promisc.py > 73 ping.py > 29 pxe.py > 42 shutdown.py > 147 softlockup.py > 53 stress_boot.py > 164 trans_hugepage_defrag.py > 127 trans_hugepage.py > 113 trans_hugepage_swapping.py > 852 unattended_install.py > 175 vlan.py > 46 watchdog.py > 136 whql_client_install.py > 275 whql_submission.py > 49 yum_update.py > 4405 total Hrm, what's the significance of the two separate directories? At any rate, my point still stands. 13k LOC is still small enough that making large changes to the infrastructure shouldn't be that big of a deal. I think the important observation is that each test is pretty small which means they can be reasonably moved 1-by-1 to a new stand alone infrastructure. > Not to mention the infrastructure libraries: > > [lmr@freedom virt]$ wc -l *.py > 1351 aexpect.py > 705 base_installer.py > 9 common.py > 0 __init__.py > 195 installer.py > 64 installer_unittest.py > 240 kvm_installer.py > 921 kvm_monitor.py > 1635 kvm_vm.py > 324 libvirt_monitor.py > 1271 libvirt_vm.py > 110 passfd.py > 237 ppm_utils.py > 519 rss_client.py > 527 virt_env_process.py > 124 virt_http_server.py > 51 virt_passfd_setup.py > 229 virt_scheduler.py > 1401 virt_step_editor.py > 131 virt_test.py > 315 virt_test_setup.py > 813 virt_test_utils.py > 3556 virt_utils.py > 196 virt_utils_unittest.py > 224 virt_video_maker.py > 1032 virt_vm.py > 16180 total > > There's a bit more outside this dir, not to mention the auxiliary data files. > The support libraries account for roughly for the same number of lines of test > code written, so we can say in very rough terms we have around 33 k of python > code for kvm/libvirt tests. I agree it is more than an order of magnitude > smaller than the qemu code base, but still it demands effort that has to be > justified. > >> anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>" >> kvm | sort -u >> >> Amos Kong >> Chris Evich >> Cleber Rosa >> Jerry Tang >> lmr >> Lucas Meneghel Rodrigues >> Lucas Meneghel Rodrigues >> Lukas Doktor >> mbligh >> Onkar N Mahajan >> pradeep >> Qingtang Zhou >> Thomas Jarosch >> Yiqiao Pu >> >> Which leads me to the following conclusions: >> >> 1) No one outside of autotest developers is actually contributing tests >> to kvm-autotest. > > Autotest was maintained using subversion until recently, all commits from people > outside autotest were commited by autotest developers, the git tree was just a > mirror until recently. Grepping for Signed-off-by: lines will give a much better > idea of the contributors: > > [lmr@freedom autotest.git]$ git log client/virt | grep "Signed-off-by:" | sort -u > Signed-off-by: Xu He Jie > Signed-off-by: Alon Levy > Signed-off-by: Amos Kong > Signed-off-by: Chen Cao > Signed-off-by: Chris Evich > Signed-off-by: Cleber Rosa > Signed-off-by: Feng Yang > Signed-off-by: Gerd Hoffmann > Signed-off-by: Jason D. Gaston > Signed-off-by: Jason Wang > Signed-off-by: Jiri Zupka > Signed-off-by: Lucas Meneghel Rodrigues > Signed-off-by: Lukas Doktor > Signed-off-by: Martin Jenner > Signed-off-by: Pradeep K Surisetty > Signed-off-by: Pradeep Kumar Surisetty > Signed-off-by: Qingtang Zhou > Signed-off-by: Quentin Deldycke > Signed-off-by: Ray Chauvin > Signed-off-by: Satheesh Rajendran > Signed-off-by: Suqin Huang > Signed-off-by: Wei Yang > Signed-off-by: Xu He Jie > Signed-off-by: Yiqiao Pu > Signed-off-by: Yogananth Subramanian > Signed-off-by: Yolkfull Chow > > [lmr@freedom autotest.git]$ git log client/tests/kvm | grep "Signed-off-by:" | > sort -u > Signed-off-by: Alon Levy > Signed-off-by: Amos Kong > Signed-off-by: Avi Kivity > Signed-off-by: Bear Yang > Signed-off-by: Cao, Chen > Signed-off-by: Chen Cao > Signed-off-by: Chris Evich > Signed-off-by: Cleber Rosa > Signed-off-by: David Huff > Signed-off-by: Eduardo Habkost > Signed-off-by: Feng Yang > Signed-off-by: Gerd Hoffmann > Signed-off-by: Jason Wang > Signed-off-by: Jerry Tang > Signed-off-by: Jes Sorensen > Signed-off-by: Jin Bing Guo > Signed-off-by: Jiri Zupka > Signed-off-by: Ken Cao > Signed-off-by: Lucas Meneghel Rodrigues Signed-off-by: Luiz Capitulino > Signed-off-by: Lukas Doktor > Signed-off-by: Marcelo Tosatti > Signed-off-by: Martin Jenner > Signed-off-by: Michael Goldish > Signed-off-by: Michael S. Tsirkin > Signed-off-by: Onkar N Mahajan > Signed-off-by: Pradeep K Surisetty > Signed-off-by: Qingtang Zhou > Signed-off-by: Ryan Harper > Signed-off-by: Shuxi Shang > Signed-off-by: Sudhir Kumar > Signed-off-by: Supriya Kannery > Signed-off-by: Suqin Huang > Signed-off-by: Tang Chen > Signed-off-by: Thomas Jarosch > Signed-off-by: Uri Lublin > Signed-off-by: Yiqiao Pu > Signed-off-by: Yogananth Subramanian > Signed-off-by: Yolkfull Chow > > So, not quite what you wanted to demonstrate. Well it actually still is. There isn't a tremendous overlap between that list and the list of QEMU contributors. >> 2) Most of the tests are relatively small and simple enough that they >> could be trivially ported to a stand alone utility. For instance, >> looking at the pci_hotplug.py, it just executes monitor commands and >> does basic pci enumeration in the guest. > > They do depend on infrastructure code to be simple and small. The infrastructure > code was written using autotest libraries, so although it is possible to make it > stand alone, still there's the problem of having these library/classes out of > sync with autotest evolving. > >> I don't see a lot of pitfalls here. At the end of the day, we're not >> talking about all that much code. > > Even so (you see it's not so small), it demands effort that has to be well > justified, as I pointed out. I'm starting to believe it might be worth the effort. There are two discussions tied into one here so just to summarize: 1) I think we can get more people working on kvm autotest and by extracting and making them stand alone. We can potentially even start merging tests into QEMU getting more developers working on kvm autotest tests and mandating new ones for certain types of features. 2) kvm autotest focuses on producing guest-OS agnostic tests which require infrastructure (Python, ssh, etc.). qemu-test essentially uses Linux as a libOS for writing unit tests. (1) does not change the need for qemu-test IMHO. >> It's not about where they live, it's about the ability to execute them >> in a simple and direct fashion. > > Ok, fair enough, we'll see what we can do, although I wish it were a simpler > problem, see below. > >>> You have a point with regards to having the test cases more stand >>> alone, but >>> it's not as easy as one would think mainly because of the large amount >>> of user >>> configurable options that we have to pass to the tests. >> >> Configuration options are really the devil when it comes to testing. As >> a developer, I want my test suite to just run and tell me if I have >> bugs. I have no interest in configuring it to run a certain way. I don't >> want to think that much about what I'm testing, I just want the tests to >> run and tell me if I screwed something up. > > Still, you need to tell your tests, say, the format of the terminal prompt for > Fedora might be slightly different of the [insert-distro-here], that the src and > destination paths on a file transfer test are different for Windows and Linux > guests, There are various clever things we can do. Essentially, the problem is that we want to deliver a test payload to a guest, right? Why not create a simple PCI device and expose the payload through the bar or something? There's no need to screen scrap a terminal session when we can do something more direct. > that the output of a given command when you are running a test with a > virtio card differs from the output when you are using rtl8139, that you want to > build qemu from git repo X, or from brew build Y, using a custom version of > seabios, and so on and so forth. Building qemu should not be tied into the tests themselves. That's useful for regression testing but that just complicates things for development testing. The building part is somethign that probably should continue to live in autotest. Regards, Anthony Liguori >