From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:41134) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RfuTy-0007uk-TL for qemu-devel@nongnu.org; Wed, 28 Dec 2011 09:27:56 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RfuTx-0001kz-2g for qemu-devel@nongnu.org; Wed, 28 Dec 2011 09:27:54 -0500 Received: from mail-gy0-f173.google.com ([209.85.160.173]:39270) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RfuTw-0001kk-Qx for qemu-devel@nongnu.org; Wed, 28 Dec 2011 09:27:52 -0500 Received: by ghbg16 with SMTP id g16so4397302ghb.4 for ; Wed, 28 Dec 2011 06:27:52 -0800 (PST) Message-ID: <4EFB2764.7040006@codemonkey.ws> Date: Wed, 28 Dec 2011 08:27:48 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> <4EF8FC88.70809@redhat.com> <4EFA4829.4000207@redhat.com> <4EFA80EA.3050405@codemonkey.ws> <4EFAA2A2.4000107@redhat.com> In-Reply-To: <4EFAA2A2.4000107@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: cleber@redhat.com Cc: "lmr@redhat.com" , Stefan Hajnoczi , dlaor@redhat.com, qemu-devel , Gerd Hoffmann , Cleber Rosa On 12/27/2011 11:01 PM, Cleber Rosa wrote: > On 12/27/2011 11:37 PM, Anthony Liguori wrote: >>> I think the main goal of qemu-tests (may be implicit) is to be quick and simple. >> >> qemu-test doesn't have a main goal. My goal is to improve QEMU's quality. >> qemu-test is just a tool to help achieve that goal. > > Maybe I've used the wrong wording. I got the feeling that, besides testing qemu > the way you need it, keeping qemu-test simple was really important. Simple is always important. In the case of qemu-test, there are some important trade offs made to keep things simple and fast. It doesn't try to work with arbitrary guests which means the tests need to handle only one environment. The guest is pre-made to have exactly what is needed for the tests so there is no setup required. The guest is as small as possible such that the test can run as quickly as possible. >>> That is indeed great, but if one thinks that's all we'll ever going to need, >>> that thought is pretty naive. >> >> I don't know who "we" is, but I can tell you that qemu-test is exactly what >> *I* need. Consider that I spent a good portion of every single day testing >> QEMU with either my own or other people's patches, making that job easier and >> more automated is fairly important to me. > > "We" is everyone that somehow contributes to QEMU, that is, the QEMU community. > If you're only concerned about what *you* need, then you're on the right track. > If, besides that, you feel it'd be nice to *try to* concentrate our efforts, > then we're all on the same track. There is no need to have a single tool that meets every possible need. In fact, the Unix tradition is to have separate single purposed tools. Having two test tools it not a bad thing provided that the overlap isn't significant. We shouldn't be discussing whether it's possible to merge the two tools, but rather what the technical benefits of doing so would be. Since at this point, there is almost no overlap between the two, I don't see any actual technical benefit to merging them. I see benefit to autotest executing qemu-test, of course. >> I'm sharing it because I suspect that a lot of other developers have a similar >> need. >> >>> And it may be true that there's room for both test >>> suites... or that, as busy developers, we're refusing to deal with the added >>> complexity (qemu alone accounts for a lot) and delaying to fix the fixable. I >>> believe on the later. >>> >>> One example: kvm-autotest has a complex configuration file format with a steep >>> learning curve, while a test such as qemu-tests/tests/simple-ping.sh would have >>> to be tweaked if somehow the kernel detects the first ethernet interface as em1 >>> (such as recent Fedora systems do). Simple, but not scalable. >> >> I can tell by this comment that you don't actually understand how qemu-test >> works. Please take a look at it before jumping to conclusions about whether it >> should or shouldn't be part of kvm-autotest. >> >> Hint: qemu-test always uses the same kernel because it builds it as part of >> the test suite. The behavior of how a nic test will never change unless >> someone explicitly changes the kernel. > > I can tell you did not get my point: I'm giving some reasons of why current kvm > autotest is somehow complex, and how qemu-tests gets away and keeps it simple. You're claiming that "we're refusing to deal with the added complexity (qemu alone accounts for a lot) and delaying to fix the fixable". There is no way that qemu-test would ever need to deal with how Fedora configures udev to name ethernet devices without becoming something totally different than it is. So there's no "delaying to fix the fixable" here. qemu-test makes a simplifying assumption. By restricting the guest to a fixed environment (initramfs w/busybox), things inherently become much, much simpler. Of course, this is not an adequate assumption to make if it were our only test tool but fortunately, we have an existing test suite that does a very good job at testing a wide variety of guests :-) >>> >>> The Python requirement inside the guest is true *if* we want to run regular >>> autotest tests inside the guest (see autotest/client/virt/tests/autotest.py) and >>> this accounts for very very little of kvm autotest usage. All other tests >>> interact with the monitor directly and with the guest via ssh/telnet/serial. >> >> qemu-test does not require any specific hardware to be used in the guest which >> lets it test a wider variety of scenarios in QEMU. So you cannot assume there >> is ssh/telnet/serial available. > > I really thought we could rely on, at least, a serial connection. If not, then > indeed the current kvm autotest approach is not compatible with that test > environment. That is not to say that kvm autotest couldn't incorporate the > qemu-tests approach/functionality. With the scope that qemu-test has, it cannot assume any hardware is present because it wants to test every piece of hardware. > BTW, I just don't like the idea of having lots of functionalities/tests > implemented on two test suites for a single piece of software, unless proven > that there's no way around it. To me, this is the whole point of this discussion. I actually disagree on principle. Two tools shouldn't be combined unless there's a proven reason that they benefit from being together. You get more flexibility at the end of the day having strong, independent components that can be combined together. >>> So, I see no reason for not using a more expressive language, >> >> I seriously doubt you can build a useful initramfs that contains python >> without doing something crazy like livecd-tools does.... > > You're right. Again, I was thinking we could rely at least on a serial > connection. Can we not? No. How would you test creating a guest with no serial devices? The nice thing about delivering a test via the initramfs is that you're only injecting software into the guest (usually via some non guest visible mechanism). >> I don't see any reason why everything needs to live in kvm-autotest... but if >> you really feel that way, please provide patches that demonstrate how this >> would work. > > If it's technically viable, I think that having it as part of kvm autotest, > shows that the project is more cohesive, attracts more contributions, and makes > better use of our efforts. Just putting the code in kvm-autotest.git in a directory doesn't make sense to me. Beyond the lack of a technical reason to do so, logistically, it makes it harder for me to ask people to submit test cases with a patch series if I can't actually apply the test case when I'm applying the patches to qemu.git. If qemu-test didn't use large submodules (like linux.git), I would have made qemu-test part of qemu.git. As far as I'm concerned, qemu-test.git is just an extension to qemu.git. >> I think there's something of a knee jerk reaction here. The existence of >> qemu-test does not take anything away from kvm-autotest. It's just another >> tool in our arsenal to achieve our joint goal of making QEMU (and KVM) higher >> quality. > > You're right, It does not take anything away from kvm autotest today. But > suppose we can prove that kvm autotest can indeed absorve all of qemu-tests > functionalities, it'd be itself a reason for doing so. That's not obvious to me. > It'd avoid finding > ourselves with two evolved test tools that do some of the same things, but are > separate implementations. It's just too easy to argue about abstract things like this. This discussion about testing has come up many times and we've talked to death how much we need to do better testing and get to test driven development. Yet, not much has materialized. I'm not interested in talking about things in abstract anymore. That's why I announced qemu-test. If folks think it should behave differently, then patches are welcome. If you think that kvm-autotest can do everything that qemu-test does, just make the changes to kvm-autotest to make it behave the same way and show me. If it makes sense, I'll happily embrace it. Regards, Anthony Liguori