From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:41810) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Rflcw-0004e1-MO for qemu-devel@nongnu.org; Wed, 28 Dec 2011 00:00:36 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Rflcu-0000Nj-5Q for qemu-devel@nongnu.org; Wed, 28 Dec 2011 00:00:34 -0500 Received: from mx1.redhat.com ([209.132.183.28]:54151) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Rflct-0000Nf-N1 for qemu-devel@nongnu.org; Wed, 28 Dec 2011 00:00:32 -0500 Message-ID: <4EFAA2A2.4000107@redhat.com> Date: Wed, 28 Dec 2011 02:01:22 -0300 From: Cleber Rosa MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> <4EF8FC88.70809@redhat.com> <4EFA4829.4000207@redhat.com> <4EFA80EA.3050405@codemonkey.ws> In-Reply-To: <4EFA80EA.3050405@codemonkey.ws> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU Reply-To: cleber@redhat.com List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: "lmr@redhat.com" , Stefan Hajnoczi , cleber@redhat.com, dlaor@redhat.com, qemu-devel , Gerd Hoffmann On 12/27/2011 11:37 PM, Anthony Liguori wrote: > On 12/27/2011 04:35 PM, Cleber Rosa wrote: >> On 12/26/2011 08:00 PM, Dor Laor wrote: >>> On 12/26/2011 05:12 PM, Anthony Liguori wrote: >>>> Hi Dor, >>>> >>> >>> Merry Christmas Anthony, >>> >>>> On 12/25/2011 09:19 AM, Dor Laor wrote: >>>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote: >>>>> >>>>> Well, I'm still not convinced that a new standalone package should >>>>> handle these >>>>> cases instead of kvm autotest. I'll be happy to integrate the >>>>> tests to >>>>> kvm >>>>> autotest anyway and the more the merrier but imho it's a duplicate. >>>> >>>> I'm sure kvm autotest could be taught to do exactly what qemu-test is >>>> doing. But why does kvm autotest have to do everything? I doubt there >>>> would be much code reuse. >>>> >>>> I think it's not a bad thing to have multiple test suites when there >>>> isn't considerable overlap. >> >> I think the main goal of qemu-tests (may be implicit) is to be quick >> and simple. > > qemu-test doesn't have a main goal. My goal is to improve QEMU's > quality. qemu-test is just a tool to help achieve that goal. Maybe I've used the wrong wording. I got the feeling that, besides testing qemu the way you need it, keeping qemu-test simple was really important. If not, I'd imagine you'd try to re-use something that already exists (yes, kvm autotest). > >> That is indeed great, but if one thinks that's all we'll ever going >> to need, >> that thought is pretty naive. > > I don't know who "we" is, but I can tell you that qemu-test is exactly > what *I* need. Consider that I spent a good portion of every single > day testing QEMU with either my own or other people's patches, making > that job easier and more automated is fairly important to me. "We" is everyone that somehow contributes to QEMU, that is, the QEMU community. If you're only concerned about what *you* need, then you're on the right track. If, besides that, you feel it'd be nice to *try to* concentrate our efforts, then we're all on the same track. > > I'm sharing it because I suspect that a lot of other developers have a > similar need. > >> And it may be true that there's room for both test >> suites... or that, as busy developers, we're refusing to deal with >> the added >> complexity (qemu alone accounts for a lot) and delaying to fix the >> fixable. I >> believe on the later. >> >> One example: kvm-autotest has a complex configuration file format >> with a steep >> learning curve, while a test such as qemu-tests/tests/simple-ping.sh >> would have >> to be tweaked if somehow the kernel detects the first ethernet >> interface as em1 >> (such as recent Fedora systems do). Simple, but not scalable. > > I can tell by this comment that you don't actually understand how > qemu-test works. Please take a look at it before jumping to > conclusions about whether it should or shouldn't be part of kvm-autotest. > > Hint: qemu-test always uses the same kernel because it builds it as > part of the test suite. The behavior of how a nic test will never > change unless someone explicitly changes the kernel. I can tell you did not get my point: I'm giving some reasons of why current kvm autotest is somehow complex, and how qemu-tests gets away and keeps it simple. BTW, I did not jumped at any conclusion. I'm just trying to enrich the discussion, which may end up proving that there's no other way to have what qemu-tests does. > >>>>>> 1) It builds a custom kernel and initramfs based on busybox. This is >>>>>> fairly important to ensure that we can run tests with no device >>>>>> pre-requisites. >>>>> >>>>> This can be done easily w/ autotest too. >> >> The Python requirement inside the guest is true *if* we want to run >> regular >> autotest tests inside the guest (see >> autotest/client/virt/tests/autotest.py) and >> this accounts for very very little of kvm autotest usage. All other >> tests >> interact with the monitor directly and with the guest via >> ssh/telnet/serial. > > qemu-test does not require any specific hardware to be used in the > guest which lets it test a wider variety of scenarios in QEMU. So you > cannot assume there is ssh/telnet/serial available. I really thought we could rely on, at least, a serial connection. If not, then indeed the current kvm autotest approach is not compatible with that test environment. That is not to say that kvm autotest couldn't incorporate the qemu-tests approach/functionality. BTW, I just don't like the idea of having lots of functionalities/tests implemented on two test suites for a single piece of software, unless proven that there's no way around it. To me, this is the whole point of this discussion. > >> >> So, I see no reason for not using a more expressive language, > > I seriously doubt you can build a useful initramfs that contains > python without doing something crazy like livecd-tools does.... You're right. Again, I was thinking we could rely at least on a serial connection. Can we not? > >>>> Actually, kvm-autotest has various layers of abstraction in how QEMU >>>> ends up being launched. As you mention below, those layers are >>>> there to >>>> allow for things like using libvirt. >> >> Indeed the qemu command line parameters gets generated depending on many >> configuration parameters. It'd be *really* simple to add a configuration >> parameters that overwrites the qemu command with an static one. > > But if you're a QEMU developer, you want to have as much control of > the command line as possible. For instance, one of the tests in > qemu-test makes sure to test invocations without -device as this > triggers a different code path (there was a recent regression in this > too). You can't just add arguments to reproduce this behavior. > >>> >>> It goes beyond that, since it also related to the monitor interface >>> as well. >>> >>>> That's desirable when you're doing "virt testing", but not so >>>> desirably >>>> when you're trying to write specific unit tests against QEMU. >>> >>> True, one may not need it at all but it's nice that a test for >>> migration/stress/hotplug will be tested directly w/ qemu and libvirt >>> w/ the >>> same effort. >>> >>>> >>>>>> 5) The tests execute very quickly, can be run stand alone, and do >>>>>> not >>>>>> require root privileges >>>>> >>>>> ditto for kvm auotest. It's possible to configure it w/o root too >>>>> which is not a >>>>> huge issue. >>>> >>>> When I say, "run quickly", I mean, they execute very quickly. >>> >>> /me too >>> >>>> >>>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 >>>> tests/virtio-serial.sh >>>> >>>> real 0m4.385s >>>> user 0m1.460s >>>> sys 0m1.860s >>> >>> That's impressive but it's more of a function of the guest being >>> used - if >>> instead of running a full Fedora install, you'll chose your busybox >>> image w/ >>> -kernel/initrd you'll get a similar result. >> >> I also think so. Maybe kvm-autotest would take a little more time >> because of the >> different approach we take when communicating with the guest, but I >> bet it'd be >> irrelevant. > > I don't see any reason why everything needs to live in kvm-autotest... > but if you really feel that way, please provide patches that > demonstrate how this would work. If it's technically viable, I think that having it as part of kvm autotest, shows that the project is more cohesive, attracts more contributions, and makes better use of our efforts. > We could argue indefinitely about how things could work, it's much > better to compare how things actually do work :-) Sure, that's also what I suggested when I mentioned it'd be a "nice exercise for all of us". > >>> I agree autotest is not perfect but it likes to be such. >>> If you wish, you can challenge Lucas and Cleber w/ these type of >>> requirements >>> and we'll all improve as a result. >> >> Yes, I believe it'd be a nice exercise for all of us. >> >> The only thing I ask is that we bear at least with some of the >> complexity that >> kvm-autotest inherently holds... > > I think there's something of a knee jerk reaction here. The existence > of qemu-test does not take anything away from kvm-autotest. It's just > another tool in our arsenal to achieve our joint goal of making QEMU > (and KVM) higher quality. You're right, It does not take anything away from kvm autotest today. But suppose we can prove that kvm autotest can indeed absorve all of qemu-tests functionalities, it'd be itself a reason for doing so. It'd avoid finding ourselves with two evolved test tools that do some of the same things, but are separate implementations. > > autotest is made to invoke third party tests so the two tools can > co-exist in a complimentary way. That's a no brainier and everyone so far agrees on that. The delicate issue is whether qemu-tests functionality could live in kvm autotest in a nice way. > > Regards, > > Anthony Liguori > >> the last requests we had was to get rid of all >> the complexity, while retaining all the other nice characteristics. >> Pretty hard, >> so I think we failed, or maybe only half-succeeded at it. >> >>> >>> Cheers, >>> Dor >>> >>>> >>>> Regards, >>>> >>>> Anthony Liguori >>>> >>>>>> >>>>>> 6) They are random by nature with the ability to fix the seed in >>>>>> order >>>>>> to be used in git-bisect. >>>>>> >>>>>> I think Gerd had been looking at doing something similar with a >>>>>> custom >>>>>> initrd. >>>>>> >>>>>> I've tried to consider other architectures and had hoped that we >>>>>> could >>>>>> commit the vmlinuz and initramfs into git so that it was easy to >>>>>> test >>>>>> other architectures without having a full build environment. >>>>>> Unfortunately, busybox doesn't link statically with glibc and I >>>>>> can't >>>>>> see an obvious way to commit binaries while respecting the GPL >>>>>> since we >>>>>> need to pull glibc into the initramfs. >>>>>> >>>>>> I know buildroot exists specifically to deal with this but in my >>>>>> experience, buildroot is very unreliable and extremely heavy weight >>>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc >>>>>> environment. >>>>>> >>>>>> Anyway, the code is available at: >>>>>> >>>>>> http://git.qemu.org/qemu-test.git >>>>>> >>>>>> See the README for instructions on how to use it. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Anthony Liguori >>>>>> >>>>> >>>> >>> >>> >> >> > >