From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:53578) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5xql-0007Hz-CT for qemu-devel@nongnu.org; Fri, 09 Mar 2012 06:19:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1S5xqc-0000gR-9I for qemu-devel@nongnu.org; Fri, 09 Mar 2012 06:19:06 -0500 Received: from mx1.redhat.com ([209.132.183.28]:35281) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5xqc-0000gB-0W for qemu-devel@nongnu.org; Fri, 09 Mar 2012 06:18:58 -0500 Message-ID: <4F59E77B.7080806@redhat.com> Date: Fri, 09 Mar 2012 08:20:27 -0300 From: Cleber Rosa MIME-Version: 1.0 References: <20120308144958.GA25750@t420s.optimusnet> <4F58C897.5020405@codemonkey.ws> <20120308150722.GA30576@t420s.optimusnet> <4F58CCBA.9000702@codemonkey.ws> <20120308160505.GA32360@t420s.optimusnet> <4F58E67A.3050708@codemonkey.ws> <20120308175907.GA4900@t420s.optimusnet> <4F5905AA.3060304@codemonkey.ws> <20120308210209.GA11998@t420s.optimusnet> <4F59237F.6010406@codemonkey.ws> <20120308222433.GB11998@t420s.optimusnet> <4F593F08.8050606@codemonkey.ws> In-Reply-To: <4F593F08.8050606@codemonkey.ws> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests Reply-To: cleber@redhat.com List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: Lucas Meneghel Rodrigues , Scott Zawalski , QEMU devel , Ademar Reis On 03/08/2012 08:21 PM, Anthony Liguori wrote: > On 03/08/2012 04:24 PM, Ademar Reis wrote: >> On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote: >>> On 03/08/2012 03:02 PM, Ademar Reis wrote: >>>> On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote: >>>>> On 03/08/2012 11:59 AM, Ademar Reis wrote: >>>> >>>> >>>> >>>>> >>>>>>> I expect QEMU to grow tests for anything that involves launching >>>>>>> QEMU directly. Where I would not see QEMU growing tests for is >>>>>>> things like launching QEMU through libvirt. >>>>>> >>>>>> Agree. For QEMU developers, libvirt should not be on the way, the >>>>>> interaction should be minimal or non-existent. >>>>>> >>>>>> That's an area which will require some work in libautotest, >>>>>> because due to previous QE requirements, it now invokes libvirt >>>>>> methods instead of QEMU directly for a lot of stuff. But it can >>>>>> be done and is part of the plan. >>>>> >>>>> If you're talking about libautotest as an abstraction layer for >>>>> launching QEMU, I don't think that's something we should use in >>>>> upstream QEMU. Abstraction layers are okay when the things you are >>>>> abstracting are well defined and relatively static. We're talking >>>>> about testing the tip of a development tree though and coordinating >>>>> changes with an abstraction layer would be counter productive. >>>>> >>>> >>>> Valid point. If I'm a qemu developer and I want to make huge >>>> changes, ideally I shouldn't have to submit patches to >>>> libautotest because the tests in qemu.org will break. >>>> >>>> I don't know how likely such scenario is (Lucas is maitaining >>>> kvm-autotest for years now and the integration hardly breaks) >>>> But we do have a conflict of interests, so lets evaluate a >>>> few options: >>> >>> Because kvm-autotest doesn't have thousands of tests and isn't part >>> of 200+ developers fast paths :-) >>> >>>> >>>> My main interests are: >>>> - remove barriers for developers to write tests >>> >>> This is my number one priority. >>> >>>> - share the testing effort between multiple projects (qemu, >>>> libvirt, ovirt, spice, QE, etc...) >>> >>> This is my number 99 priority. >> >> You have the right to consider the interests of qemu only, but >> qemu is used by several other teams who depend on it to implement >> their own tests. Some collaboration would be extremely beneficial >> for everybody. >> >> Collaboration goes both ways: you get benefits from the >> interaction and code reuse with different projects and teams, but >> it may require a few tradeoffs. > > I think the problem here effectively boils down to "code sharing at > any cost". > > Let's look at it in purely practical terms. How many lines of code > would be saved here and what does that code do? > > If the only sharing is some infrastructure bits in libautotest, then I > suspect we're talking in the low 10ks of lines at the absolute maximum > (and I don't think it's even close to that in reality). Launching > QEMU and connecting to QMP isn't that hard. It's in the hundreds of > lines of code. What else is there that's need to write a test? > > I don't think this is sufficiently compelling to add an unconditional > autotest dependency to QEMU. > >>> I think you could achieve practical code sharing by decomposing >>> kvm-autotest into a set of useful, independent tools. For >>> instance.... >>> >>>> Now the options: >>>> >>>> 1. We keep the invocation stuff in libautotest and when writting >>>> a complex test in qemu.git, the test writter can choose to use it >>>> because of the goodies there. >>>> >>>> Pros: >>>> - Lots of codepaths implemented both in python and as cmd-line >>>> utilities: less lines of code to write a test, smaller >>>> barrier for the developer. >>> >>> I've got an existence proof that this is not true. The qemu-test >>> tests are smaller than the corresponding autotest tests. >> >> You're comparing developer-level tests with the existent QA-level >> tests (much more complex). > > Let's be specific then. Look at device-add.sh in qemu-test. It's > 71LOC. pci_hotplug.py in autotest is 204LOC. pci_hotplug.py does much more than device-add.sh: * tests both pci_add and device_add commands * checks the monitor syntax for adding a new drive, that is, it works on HEAD and on other versions (such as the ones in some RHEL releases); * tests both nic and block hotplug * for block, tests with both virtio and scsi * also does device removal, both for pci_add and device_add syntaxes > > What's the "much more complex" part of pci_hotplug.py that is enabled > by having the autotest framework? > >> We're talking about an approach where you'll need just a few >> lines of code to instantiate a VM and interact with it. > > This isn't that hard of a problem. It just takes a few lines to > launch QEMU in qemu-test too. > >>>> - Mature code in the test library. >>>> - Goodies such as video-recording and system info collection >>>> in a standard way for different projects. >>> >>> Why does video-recording have to be part of libautotest? Why can't >>> you just have a simply utility that connects to a VNC session and >>> records it? Presumably that's all it's doing here. >>> >>> Likewise, there are dozens of "system info collection" such as >>> sosreport.. why fold this all into libautotest? >> >> For standardization: think outside of the qemu community. A lot >> of our developers contribute code to multiple projects. It gets >> easier if there's a standard way to achieve the same results when >> writting tests for different projects. >> >> Anyway, it's a minor issue. > > Yeah, I think this is outside of the scope of the current discussion > as it's sort of like trying to boil the ocean :-) > >>>> - Code shared with other teams. >>>> - The same test code can run on old platforms because >>>> libautotest has a compatibility layer, thus enabling a >>>> larger grid of tests. >>> >>> This is not a requirement from a QEMU perspective nor do I think >>> it's useful. >> >> Well, it's not a requirement, but you can't expect QE teams to >> spend their time on tests that work only on HEAD. It makes no >> sense from their point of view. > > They're certainly testing some sort of HEAD, even if it's not > qemu.git. I don't see any conflicting goals here. Write a test > against upstream, backport to a downstream if necessarily. > >> >>> >>>> - QE teams will be closer to the development, sharing >>>> some of their testing code and maintenance burden. >>> >>> The QE team could simply contribute tests to qemu.git to achieve >>> this same goal. >> >> They won't help you if you don't help them. Again, it doesn't >> make sense for QE to work on HEAD. At least not the QE teams I'm >> talking about. >> >> And I say from my own experience: maintaining a set of thousands >> of tests is far from trivial. You'll waste a lot of potential >> contributions by alienating the fully-dedicated teams currently >> working on QE. > > And trying to maintain a thousand tests against any possible version > is probably also unrealistic. > > Is the expectation that all of the tests in WebKit work against any > possible version of WebKit? How do they enforce this since it's all > in trunk? > >>>> 2. We "move" (or implement) all the libautotest stuff related to >>>> qemu invocation and interaction into qemu.git, forbidding the >>>> dependency of autotest in qemu.git. >>> >>> Let me just be really clear here. In order to do TDD, tests have to >>> be a mandatory part of the QEMU build process. We cannot introduce >>> new dependencies in order to build tests. >>> >>> Making autotest an unconditional dependency of qemu.git is a >>> non-starter for me. >> >> You have dependencies on gtest and other libraries. it's not that >> different. > > We have a dependency on glib, which gtest happens to be a part of. It > was a big deal introducing that dependency and gtest was not the only > (or primary) reason to introduce that dependency. > >> Plus it's not unconditional: the test runner will report tests >> SKIPPED if a dependency is not present. > > But then the tests aren't run so if most developers didn't have it > installed, and most tests were written with it, most developers > wouldn't be running most tests which defeats the purpose, no? > >> It's actually the oposite: you'll have to forbid developers from >> submiting tests that use libautotest if you don't want the >> dependency. > > I think part of the problem here is the abstract nature of this > discussion. What is libautotest actually supposed to do? > > This would be an easier discussion if we were talking about patches > instead of proposals. > >>>> - QE will be alienated from the qemu test effort. There will be >>>> no integration between the QE efforts and the maintenance of >>>> the qemu developer-level tests. >>> >>> I think we're a pretty friendly and open community :-) There is no >>> reason that QE should be "alienated" unless folks are choosing not >>> to participate upstream. >> >> For the exact same reasons you as a developer don't want to >> implement tests inside autotest, QE won't want to implement tests >> for qemu.git. It's out of their comfort zone, just put yourself >> on their shoes. > > This is a really, really poor argument and I hope I don't need to go > into details of why. If the primary reason for libautotest is so the > people writing tests for QEMU can avoid actually working with the > developers of QEMU... we've got a problem. > >>> >>>> - You don't get the goodies from autotest (standardized system >>>> info collection, video recording, grid support, etc). >>>> - The new tests will work only on the master branch. >>> >>> This last one is a legitimate point that I have considered myself. >>> But I think the value of having tests in HEAD outweigh the cost >>> here. >> >> Thinking about qemu only, agree. >> >>> >>>> 3. A mix of (1) and (2): we "move" everything under qemu.git, but >>>> keep the compatibility layer and provide a "libqemutest" for >>>> third-parties. >>>> >>>> Anybody writting a test that interacts with qemu will use this >>>> library: qemu, libvirt, spice, ovirt, QE tests, etc. >>> >>> I really see this all as over architecting to be honest. >>> >>> Can someone explain in clear terms (without appealing to maturity, >>> flexibility, or any other qualitative aspect) why it takes anything >>> more than a few dozen shell functions to write all of the tests that >>> are in kvm-autotest test today? >> >> This is close to being offensive IMO, so I won't answer. :-) > > I honestly don't mean it to be so I apologize if it comes off that > way. But I dislike arguing in such abstract terms. What are the > features that you think libautotest is needed for? > >>> You launch qemu (by just using a command), and you interact with the >>> monitor via QMP and HMP. Why do you ever need anything more than >>> that? >> >> Well, again, I'll agree to disagree. >> >>> >>> Why does libautotest even need to exist? >> >> You're looking at things from the sole point of view of a qemu >> developer. If you don't want to see things from a different >> perspective and accept any kind of tradeoffs for better >> collaboration with other teams, then unfortunately there's not >> much we can do. It's a pity, because collaboration goes both >> ways. > > I need something more concrete then this. What is it about > libautotest that would allow collaboration? > >> We're trying very hard to satisfy your requirements and >> interests. Don't you see any way you can help us satisfy >> ours? > > I'm having a hard time understanding what your requirements are other > than that we should write our tests using libautotest (which has not > really been explained other than being levied as a requirement). > > Regards, > > Anthony Liguori