From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:53113) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RfJX7-0001Qc-Pe for qemu-devel@nongnu.org; Mon, 26 Dec 2011 18:00:43 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RfJX4-0007BG-M5 for qemu-devel@nongnu.org; Mon, 26 Dec 2011 18:00:41 -0500 Received: from mx1.redhat.com ([209.132.183.28]:46439) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RfJX4-0007As-FF for qemu-devel@nongnu.org; Mon, 26 Dec 2011 18:00:38 -0500 Message-ID: <4EF8FC88.70809@redhat.com> Date: Tue, 27 Dec 2011 01:00:24 +0200 From: Dor Laor MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> In-Reply-To: <4EF88EC0.8020301@codemonkey.ws> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU Reply-To: dlaor@redhat.com List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: "lmr@redhat.com" , Gerd Hoffmann , Stefan Hajnoczi , qemu-devel On 12/26/2011 05:12 PM, Anthony Liguori wrote: > Hi Dor, > Merry Christmas Anthony, > On 12/25/2011 09:19 AM, Dor Laor wrote: >> On 12/19/2011 07:13 PM, Anthony Liguori wrote: >> >> Well, I'm still not convinced that a new standalone package should >> handle these >> cases instead of kvm autotest. I'll be happy to integrate the tests to >> kvm >> autotest anyway and the more the merrier but imho it's a duplicate. > > I'm sure kvm autotest could be taught to do exactly what qemu-test is > doing. But why does kvm autotest have to do everything? I doubt there > would be much code reuse. > > I think it's not a bad thing to have multiple test suites when there > isn't considerable overlap. I agree but in this case, it loos to me that qemu-test is likely to do a subset of what kvm autotest do in a potentially less elegant way but due to the influence of a key maintainer may gain traction and dilute kvm autotest.. > >>> It has the following characteristics: >>> >>> 1) It builds a custom kernel and initramfs based on busybox. This is >>> fairly important to ensure that we can run tests with no device >>> pre-requisites. >> >> This can be done easily w/ autotest too. > > Okay, please demonstrate :-) The procedure to do this with qemu test is: > > $ git clone git://git.qemu.org/qemu-test.git > $ cd qemu-test > $ git submodule update --init > $ make > > How would one do this with kvm autotest? git clone git://github.com/autotest/autotest.git cd autotest/client/tests/kvm sudo ./get_started.py One command shorter and the above downloads F15 automatically and runs it. (from https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript) >>> 2) Tests are scripts that are launched in the initramfs >>> >>> 3) The test controls exactly how QEMU is launched which allows easy >>> testing of various QEMU options >> >> afaik kvm autotest do the same. It's true that kvm autotest might look >> less >> friendly but its may only be this way since it has lots of options. > > Actually, kvm-autotest has various layers of abstraction in how QEMU > ends up being launched. As you mention below, those layers are there to > allow for things like using libvirt. It goes beyond that, since it also related to the monitor interface as well. > That's desirable when you're doing "virt testing", but not so desirably > when you're trying to write specific unit tests against QEMU. True, one may not need it at all but it's nice that a test for migration/stress/hotplug will be tested directly w/ qemu and libvirt w/ the same effort. > >>> 5) The tests execute very quickly, can be run stand alone, and do not >>> require root privileges >> >> ditto for kvm auotest. It's possible to configure it w/o root too >> which is not a >> huge issue. > > When I say, "run quickly", I mean, they execute very quickly. /me too > > $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 > tests/virtio-serial.sh > > real 0m4.385s > user 0m1.460s > sys 0m1.860s That's impressive but it's more of a function of the guest being used - if instead of running a full Fedora install, you'll chose your busybox image w/ -kernel/initrd you'll get a similar result. > > I've used kvm-autotest a lot, there is no test in kvm-autotest that is > even close to completing in 4 seconds start to finish. I don't think > kvm-autotest can even run without installing a guest first which puts > puts a simple test more at the 30 minute mark. If we're talking about > TCG testing, then we're in hours territory. Autotest today may be too verbose by default and will call echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way and Lucas can easily create a slim mode that does not even need to be root. > > If there's a way to run kvm-autotest as a non-privileged user, I've not > figured it out yet. Again, there is no magic in a test suite so if required we can wave it (maybe we need to do that regardless) > > Of course, kvm-autotest can test a lot more things than qemu-test can > since the tests are guest agnostic. That is the primary architectural > difference. By writing the tests to one specific guest (busybox initrd), > they can be extremely fast in execution, but the tests are less general > purpose. > > That makes qemu-test very useful for development testing (such as long > automated bisect runs) but less useful for acceptance testing. Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and the tests will be slimmer or each test will have such a mode. > >> Please compare your own virtio-serial test w/ the autotest version of it: >> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py >> >> >> >> This single file tests functionality, limits, console, live migration and >> performance. Of course one can add a very basic 'hello world' sanity >> test too >> that will run quickly and will identify basic breakage fast. > > Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52. I bet that virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will be less than 52 lines. It depends what you like to test. as noted, the above test fully covers all aspects of virtio-serial. It does not have to be this way and I do agree it is nicer to split it to multiple sub files. > > There is a lot of value in being able to write simple tests in 50 lines > of code. In fact, the largest qemu-test at the moment is only ~70 lines > of code. And besides finding regressions in my own code (which is the > primary use atm), I've found and fixed a few real bugs in upstream QEMU. Now imagine what virtio_console.py buys you :) > > For a couple hundred lines of bash script, how can it be anything but a > good thing :-) > >> Noways we abstract kvm autotest so libvirt will be optionally tested >> too w/ the >> same tests. >> >> Again, I'm not advocating that kvm autotest is a solution for anything >> but for >> plain guest-host communication, monitor commands, etc it's a really >> good tool. >> >> I agree that kvm autotest may be less friendly for developer users >> since it >> carries allot of options for testing a huge matrix. Lucas and Cleber >> are working >> these days to add a make kvmautotest target to qemu so one would be >> able to >> quickly execute autotest and we can think of additional parameters like >> sanity-set, migration-set, etc. > > We (the QEMU project) need to get more serious about testing. We need to Like > (quickly) get to a point where we can mandate that features come with > test cases. It is extremely hard to implement touch everything features > with no good way to test all of the different things we support. > > kvm autotest cannot fill that role because it's too complicated and too > many layers removed from QEMU. > > virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not > about to mandate that someone writes 2.2k LOC in a python test framework > in order to get 1.4k of code merged in QEMU. > > But 50 lines of bash seems like a more than reasonable requirement. Hmm, what's the coverage of these 50 LOC? Maybe when your maintainer hat is on, it is indeed a huge task to keep all of qemu platforms/architectures to compile and run so that's enough but when I wear my Red Hat manager role, I wish that it will not only compile and boot but will test as much functionality as possible and will be easily re-used by R&D, QA, libvirt, while all supported guests are _potentially_ exercised, not just busybox. Since kvm autotest will be there anyway, I wish that developers will contribute 50 LOC to autotest (and not 2.2kloc) and hope the ROI will justify it. I agree autotest is not perfect but it likes to be such. If you wish, you can challenge Lucas and Cleber w/ these type of requirements and we'll all improve as a result. Cheers, Dor > > Regards, > > Anthony Liguori > >>> >>> 6) They are random by nature with the ability to fix the seed in order >>> to be used in git-bisect. >>> >>> I think Gerd had been looking at doing something similar with a custom >>> initrd. >>> >>> I've tried to consider other architectures and had hoped that we could >>> commit the vmlinuz and initramfs into git so that it was easy to test >>> other architectures without having a full build environment. >>> Unfortunately, busybox doesn't link statically with glibc and I can't >>> see an obvious way to commit binaries while respecting the GPL since we >>> need to pull glibc into the initramfs. >>> >>> I know buildroot exists specifically to deal with this but in my >>> experience, buildroot is very unreliable and extremely heavy weight >>> since it rebuilds gcc multiple times in order to bootstrap a ulibc >>> environment. >>> >>> Anyway, the code is available at: >>> >>> http://git.qemu.org/qemu-test.git >>> >>> See the README for instructions on how to use it. >>> >>> Regards, >>> >>> Anthony Liguori >>> >> >