From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:59118) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5ipq-0007gq-0S for qemu-devel@nongnu.org; Thu, 08 Mar 2012 14:17:12 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1S5ipm-0003Ug-Fe for qemu-devel@nongnu.org; Thu, 08 Mar 2012 14:17:09 -0500 Received: from mail-pw0-f45.google.com ([209.85.160.45]:42529) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5ipm-0003Ua-53 for qemu-devel@nongnu.org; Thu, 08 Mar 2012 14:17:06 -0500 Received: by pbcuo5 with SMTP id uo5so2127955pbc.4 for ; Thu, 08 Mar 2012 11:17:04 -0800 (PST) Message-ID: <4F5905AA.3060304@codemonkey.ws> Date: Thu, 08 Mar 2012 13:16:58 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <4F582EDB.1040608@redhat.com> <4F58B5CB.8040503@codemonkey.ws> <20120308144958.GA25750@t420s.optimusnet> <4F58C897.5020405@codemonkey.ws> <20120308150722.GA30576@t420s.optimusnet> <4F58CCBA.9000702@codemonkey.ws> <20120308160505.GA32360@t420s.optimusnet> <4F58E67A.3050708@codemonkey.ws> <20120308175907.GA4900@t420s.optimusnet> In-Reply-To: <20120308175907.GA4900@t420s.optimusnet> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Ademar Reis Cc: Lucas Meneghel Rodrigues , Scott Zawalski , QEMU devel , "kvm-autotest@redhat.com" , Cleber Rosa On 03/08/2012 11:59 AM, Ademar Reis wrote: > On Thu, Mar 08, 2012 at 11:03:54AM -0600, Anthony Liguori wrote: >> On 03/08/2012 10:05 AM, Ademar Reis wrote: >>> On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote: >>>> On 03/08/2012 09:07 AM, Ademar Reis wrote: >>>>> On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote: >>>>>> On 03/08/2012 08:49 AM, Ademar Reis wrote: >>>>>>> On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote: >>>>>>>> On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote: >>>>>>>>> Virt/qemu tests: Minimal guest images >>>>>>>>> ------------------------------------- >>>>>>>>> >>>>>>>>> In order to make development level test possible, we need the tests to run fast. >>>>>>>>> In order to do that, a set of minimal guest images is being developed and we >>>>>>>>> have a version for x86_64 ready and functional: >>>>>>>>> >>>>>>>>> https://github.com/autotest/buildroot-autotest >>>>>>>> >>>>>>>> I'm really not a fan of buildroot. Note that in order to ship >>>>>>>> binaries, full source needs to be provided in order to comply with >>>>>>>> the GPL. The FSF at least states that referring to another website >>>>>>>> for source that's not under your control doesn't satisfy the >>>>>>>> requirements of the GPL. >>>>>>>> >>>>>>>> Just out of curiosity, did you try to use qemu-test? Is there a >>>>>>>> reason you created something different? >>>>>>>> >>>>>>>> I think it's good that you're thinking about how to make writing >>>>>>>> tests easier, but we have a growing test infrastructure in QEMU and >>>>>>>> that's what I'd prefer people focused on. >>>>>>>> >>>>>>> >>>>>>> You probably remember the long thread we had back in December on >>>>>>> qemu-devel on this topic. Back then our message was "we have a >>>>>>> growing test infrastructure in s/QEMU/autotest/ and that's what >>>>>>> we'd prefer people focused on". :-) >>>>>>> >>>>>>> From Dor: >>>>>>> >>>>>>> (http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html) >>>>>>> >>>>>>> """ >>>>>>> If you wish, you can challenge Lucas and Cleber w/ these type of >>>>>>> requirements and we'll all improve as a result. >>>>>>> """ >>>>>>> >>>>>>> Your response was: >>>>>>> >>>>>>> """ >>>>>>> Well consider qemu-test the challenge. It's an existence proof >>>>>>> that we can have a very easy to use framework for testing that >>>>>>> runs extremely fast and is very easy to write tests for. >>>>>>> """ >>>>>>> >>>>>>> http://knowyourmeme.com/memes/challenge-accepted ;-) >>>>>>> >>>>>>> I particularly agreed with basically everything you said on that >>>>>>> discussion regarding test simplification (I had just joined the >>>>>>> team back then). To me, autotest has been focusing on QE-level, >>>>>>> leaving the developer-level test requirements out. Now we're >>>>>>> attacking this new front, and a lot of the requirements are >>>>>>> indeed from that discussion. >>>>>> >>>>>> If you want to talk about this in terms of "requirements", my >>>>>> requirement is for "developer-level" tests to live in qemu.git and >>>>>> be integrated into make check. >>>>>> >>>>>> Just as we've been discussing and working on since the previous set of discussions. >>>>>> >>>>>>> By simplifying the design and bringing barriers down, we hope to >>>>>>> reach a broader audience and help developers write and maintain >>>>>>> tests, benefiting from all the instrumentation that autotest >>>>>>> brings. It's not going to be just about qemu (check the new test >>>>>>> examples). >>>>>>> >>>>>>> We have a team fully dedicated to autotest and it's used not only >>>>>>> by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc >>>>>>> (these all have code contributions in autotest) >>>>>>> >>>>>>> That said, the current qemu-tests will probably be easily >>>>>>> integrated into (the new) autotest and we hope that, given enough >>>>>>> time, autotest will be good enough to relieve qemu from the >>>>>>> framework maintenance and code duplication with other projects. >>>>>> >>>>>> autotest should not be the focal point for integration. qemu.git should be. >>>>>> >>>>>> I'd be perfectly happy to review patches submitting the test >>>>>> infrastructure from kvm-autotest into qemu.git (provided it didn't >>>>>> have unreasonable external dependencies and fit into QEMU). >>>>>> >>>>>> Developer-level tests need to live where the developers live. The >>>>>> developers live in qemu.git. See my other response on this thread >>>>>> for the explanation of why this is so important. >>>>>> >>>>> >>>>> Excelent, we're in the same page then. This was my number 1 >>>>> requirement when I was discussing the changes with Lucas and >>>>> Cleber. For convenience, I'll repeat here what I wrote in a >>>>> previous e-mail (no qemu-devel archive available yet to use as a >>>>> reference). >>>>> >>>>> In summary, autotest is (or is going to be) a framework that >>>>> provides: >>>>> >>>>> - A test runner, with grid/cluster support and advanced >>>>> instrumentation >>>>> - A devel library and set of utilities for test writers >>>>> - A set of pre-built images (JeOS – Just Enough OS) for >>>>> test writers >>>> >>>> I don't think autotest is the right place for this to live. We need >>>> this directly in qemu.git otherwise we're severely limited in what >>>> tests we can write. >>>> >>>> I guess that doesn't preclude autotest having its own JeOS >>>> mechanism, but we clearly need one in qemu.git. >>>> >>>>> >>>>> (attached is a picture showing what we want to achieve) >>>>> >>>>> If a project has an internal library or set of utilities that can >>>>> be of general use, they can be submitted to autotest.git for >>>>> inclusion, thus reaching a broader audience. >>>>> >>>>> A short summary of the plans: >>>>> >>>>> - Tests can live anywhere and each devel team implements and >>>>> maintains their own set of tests >>>> >>>> Let me change this to: >>>> >>>> - Autotest will learn how to harness the tests that each development >>>> team creates in their respective git repository. >>> >>> Yep, given this simple requirement: "tests return 0 or an error >>> code". The output from the test runner will be a simple PASS/FAIL >>> and stdout/stderr will be properly collected by the test runner >>> and made available in a standard location. If using TAP, even >>> better. >> >> We use gtest, so this will not be the case. I think it would be >> best for autotest to learn how to interact with the gtest protocol. > > I don't see a problem with this. It's very minor IMO, we can > probably speak gtest as well, but in the worst case, all logs > will be collected and are human-readable anyway. > >>> >>> This will allow trivial tests. As the test complexity grows, then >>> developers may start using parts of the autotest library or >>> utilities, thus requiring it installed (a trivial bootstrap >>> procedure is also a requirement, see the e-mail from Lucas). >> >> What is the value of the autotest library to a test writer other >> than being part of the "autotest framework"? > > Well, that's the whole idea: there are tons of "goodies" in the > autotest library and in the utilities. You can either replicate > the code and maintain everything inside qemu.git or use what > autotest provides not just to QEMU but other projects as well. > > For example, the autotest library provides ways to talk to > guests, check if they're alive, interact via ssh, open serial > channels, start/stop/install/kill/migrate VMs, record video, etc. > Why would you want to duplicate and maintain all this > functionality inside QEMU when this could be provided for free by > autotest, and shared with other projects? > > Maybe we can split things a bit more and consider the creation of > a "libvirttest" (or whatever name makes sense) that would > implement all the virt testing infrastructure to be shared among > the interested parties: > > - QEMU > - libvirt > - spice > - ... > - QE teams (the ones who write complex and comprehensive tests > in their own repo and will help a lot with the maintenance > of such library) > > But such code already exists and there are active contributors to > it in autotest.git. Our plan is to considerably improve it, but > if something is wrong with our plans, that's the moment to > improve things up by raising requirements. > >> >>>> This is where you start to lose me. If all you're saying is, "QEMU >>>> can continue to use gtest to build out it's test infrastructure and >>>> autotest will learn how to use it", then we're in violent agreement. >>> >>> Good :-) >>> >>> Inside qemu.git, you can add as many instrumentation and helper >>> libraries as you want. As a developer it'll be up to you to setup >>> your environment to run the tests, and autotest will be just yet >>> another dependency. >> >> I think there's an implicit assumption here that the tests in >> qemu.git will have a dependency on autotest. I'd like to understand >> why this dependency is necessary. >> >> Normally, autotest executes third party tests, what makes QEMU special here? > > We're just offering what we already have, thanks to all the > resources invested in the "kvm-" part of "kvm-autotest" through > years. Again, we could split that part out of autotest, but in > the end it's just a naming convention. > > A different question would be: what's the point of QEMU writting > code to handle interaction with VMs if such code can be shared > among different projects? Why rewrite everything if there's a > team allocated to this kind of task already, with a substancial > codebase and expertise? Why not raise requirements instead? > >> >>> QE will probably be interested in setting up a bot with several >>> tests from differente repositories, but that's their problem, not >>> yours. >> >> Right, and this is what autotest is very good at :-) > > Yep, but a lot of code can be shared and we can have a common > API, there's no need to duplicate efforts. I think the right way to address this is sending patches to qemu-devel. If you think there are bits of autotest that could be integrated, wonderful. But you are the autotest experts, not me. You can tell me that there's wonderful things that should be reused but I have no idea what they are and am not in a position to figure them out. Hopefully you'll agree, that autotest is not currently conducive to a buffet style consumption model. As an outsider, it looks pretty much all or nothing to me. > I clearly see difference motivations for tests written by QE and > developers, but I don't see why they can't share a common library > and why they can't exchange tests, expertise and contribute with > each other. I have no moral objections to sharing code. But I feel strongly that we need to increase our testing in qemu.git. If you think that there is code in autotest that could contribute to that, patches are more than welcome. > By the book, this would be the difference: > > unit-test: > void main() > { > my_function(); > } > > integration test (or validation test): > void main() > { > exec("my-application"); > } > > But that's all semantics, not important for this discussion IMO. Heh, the lines are not quite that clear. We can debate the Unified Process here but I agree, it's not important for this discussion :-) >> I expect QEMU to grow tests for anything that involves launching >> QEMU directly. Where I would not see QEMU growing tests for is >> things like launching QEMU through libvirt. > > Agree. For QEMU developers, libvirt should not be on the way, the > interaction should be minimal or non-existent. > > That's an area which will require some work in libautotest, > because due to previous QE requirements, it now invokes libvirt > methods instead of QEMU directly for a lot of stuff. But it can > be done and is part of the plan. If you're talking about libautotest as an abstraction layer for launching QEMU, I don't think that's something we should use in upstream QEMU. Abstraction layers are okay when the things you are abstracting are well defined and relatively static. We're talking about testing the tip of a development tree though and coordinating changes with an abstraction layer would be counter productive. Regards, Anthony Liguori