From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:49383) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5gl6-0005q5-I0 for qemu-devel@nongnu.org; Thu, 08 Mar 2012 12:04:31 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1S5gkz-0002HD-8I for qemu-devel@nongnu.org; Thu, 08 Mar 2012 12:04:08 -0500 Received: from mail-pw0-f45.google.com ([209.85.160.45]:51436) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5gky-0002El-V3 for qemu-devel@nongnu.org; Thu, 08 Mar 2012 12:04:01 -0500 Received: by pbcuo5 with SMTP id uo5so2007773pbc.4 for ; Thu, 08 Mar 2012 09:03:58 -0800 (PST) Message-ID: <4F58E67A.3050708@codemonkey.ws> Date: Thu, 08 Mar 2012 11:03:54 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <4F582EDB.1040608@redhat.com> <4F58B5CB.8040503@codemonkey.ws> <20120308144958.GA25750@t420s.optimusnet> <4F58C897.5020405@codemonkey.ws> <20120308150722.GA30576@t420s.optimusnet> <4F58CCBA.9000702@codemonkey.ws> <20120308160505.GA32360@t420s.optimusnet> In-Reply-To: <20120308160505.GA32360@t420s.optimusnet> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Ademar Reis Cc: Lucas Meneghel Rodrigues , Scott Zawalski , QEMU devel , "kvm-autotest@redhat.com" , Cleber Rosa On 03/08/2012 10:05 AM, Ademar Reis wrote: > On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote: >> On 03/08/2012 09:07 AM, Ademar Reis wrote: >>> On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote: >>>> On 03/08/2012 08:49 AM, Ademar Reis wrote: >>>>> On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote: >>>>>> On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote: >>>>>>> Virt/qemu tests: Minimal guest images >>>>>>> ------------------------------------- >>>>>>> >>>>>>> In order to make development level test possible, we need the tests to run fast. >>>>>>> In order to do that, a set of minimal guest images is being developed and we >>>>>>> have a version for x86_64 ready and functional: >>>>>>> >>>>>>> https://github.com/autotest/buildroot-autotest >>>>>> >>>>>> I'm really not a fan of buildroot. Note that in order to ship >>>>>> binaries, full source needs to be provided in order to comply with >>>>>> the GPL. The FSF at least states that referring to another website >>>>>> for source that's not under your control doesn't satisfy the >>>>>> requirements of the GPL. >>>>>> >>>>>> Just out of curiosity, did you try to use qemu-test? Is there a >>>>>> reason you created something different? >>>>>> >>>>>> I think it's good that you're thinking about how to make writing >>>>>> tests easier, but we have a growing test infrastructure in QEMU and >>>>>> that's what I'd prefer people focused on. >>>>>> >>>>> >>>>> You probably remember the long thread we had back in December on >>>>> qemu-devel on this topic. Back then our message was "we have a >>>>> growing test infrastructure in s/QEMU/autotest/ and that's what >>>>> we'd prefer people focused on". :-) >>>>> >>>>> From Dor: >>>>> >>>>> (http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html) >>>>> >>>>> """ >>>>> If you wish, you can challenge Lucas and Cleber w/ these type of >>>>> requirements and we'll all improve as a result. >>>>> """ >>>>> >>>>> Your response was: >>>>> >>>>> """ >>>>> Well consider qemu-test the challenge. It's an existence proof >>>>> that we can have a very easy to use framework for testing that >>>>> runs extremely fast and is very easy to write tests for. >>>>> """ >>>>> >>>>> http://knowyourmeme.com/memes/challenge-accepted ;-) >>>>> >>>>> I particularly agreed with basically everything you said on that >>>>> discussion regarding test simplification (I had just joined the >>>>> team back then). To me, autotest has been focusing on QE-level, >>>>> leaving the developer-level test requirements out. Now we're >>>>> attacking this new front, and a lot of the requirements are >>>>> indeed from that discussion. >>>> >>>> If you want to talk about this in terms of "requirements", my >>>> requirement is for "developer-level" tests to live in qemu.git and >>>> be integrated into make check. >>>> >>>> Just as we've been discussing and working on since the previous set of discussions. >>>> >>>>> By simplifying the design and bringing barriers down, we hope to >>>>> reach a broader audience and help developers write and maintain >>>>> tests, benefiting from all the instrumentation that autotest >>>>> brings. It's not going to be just about qemu (check the new test >>>>> examples). >>>>> >>>>> We have a team fully dedicated to autotest and it's used not only >>>>> by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc >>>>> (these all have code contributions in autotest) >>>>> >>>>> That said, the current qemu-tests will probably be easily >>>>> integrated into (the new) autotest and we hope that, given enough >>>>> time, autotest will be good enough to relieve qemu from the >>>>> framework maintenance and code duplication with other projects. >>>> >>>> autotest should not be the focal point for integration. qemu.git should be. >>>> >>>> I'd be perfectly happy to review patches submitting the test >>>> infrastructure from kvm-autotest into qemu.git (provided it didn't >>>> have unreasonable external dependencies and fit into QEMU). >>>> >>>> Developer-level tests need to live where the developers live. The >>>> developers live in qemu.git. See my other response on this thread >>>> for the explanation of why this is so important. >>>> >>> >>> Excelent, we're in the same page then. This was my number 1 >>> requirement when I was discussing the changes with Lucas and >>> Cleber. For convenience, I'll repeat here what I wrote in a >>> previous e-mail (no qemu-devel archive available yet to use as a >>> reference). >>> >>> In summary, autotest is (or is going to be) a framework that >>> provides: >>> >>> - A test runner, with grid/cluster support and advanced >>> instrumentation >>> - A devel library and set of utilities for test writers >>> - A set of pre-built images (JeOS – Just Enough OS) for >>> test writers >> >> I don't think autotest is the right place for this to live. We need >> this directly in qemu.git otherwise we're severely limited in what >> tests we can write. >> >> I guess that doesn't preclude autotest having its own JeOS >> mechanism, but we clearly need one in qemu.git. >> >>> >>> (attached is a picture showing what we want to achieve) >>> >>> If a project has an internal library or set of utilities that can >>> be of general use, they can be submitted to autotest.git for >>> inclusion, thus reaching a broader audience. >>> >>> A short summary of the plans: >>> >>> - Tests can live anywhere and each devel team implements and >>> maintains their own set of tests >> >> Let me change this to: >> >> - Autotest will learn how to harness the tests that each development >> team creates in their respective git repository. > > Yep, given this simple requirement: "tests return 0 or an error > code". The output from the test runner will be a simple PASS/FAIL > and stdout/stderr will be properly collected by the test runner > and made available in a standard location. If using TAP, even > better. We use gtest, so this will not be the case. I think it would be best for autotest to learn how to interact with the gtest protocol. > > This will allow trivial tests. As the test complexity grows, then > developers may start using parts of the autotest library or > utilities, thus requiring it installed (a trivial bootstrap > procedure is also a requirement, see the e-mail from Lucas). What is the value of the autotest library to a test writer other than being part of the "autotest framework"? >> This is where you start to lose me. If all you're saying is, "QEMU >> can continue to use gtest to build out it's test infrastructure and >> autotest will learn how to use it", then we're in violent agreement. > > Good :-) > > Inside qemu.git, you can add as many instrumentation and helper > libraries as you want. As a developer it'll be up to you to setup > your environment to run the tests, and autotest will be just yet > another dependency. I think there's an implicit assumption here that the tests in qemu.git will have a dependency on autotest. I'd like to understand why this dependency is necessary. Normally, autotest executes third party tests, what makes QEMU special here? > QE will probably be interested in setting up a bot with several > tests from differente repositories, but that's their problem, not > yours. Right, and this is what autotest is very good at :-) > BTW, please note that we're not trying to cover unit-tests in > autotest. At least not by the definition of unit-tests that I > know, which are code-level tests without running the application > itself (unit-tests test functions/methods only, not the > application as a whole). There is never a clear line between unit-tests and integration tests as long as you're talking about a single project. I expect QEMU to grow tests for anything that involves launching QEMU directly. Where I would not see QEMU growing tests for is things like launching QEMU through libvirt. > Autotest is more about integration tests, which are complementary > to unit-tests in a "test driven development" environment: you > instantiate your program by using scripts and check if things > happen as you expect, as in this trivial example: > > qemu-img-convert.sh > #!/bin/bash > qemu-img convert -O qcow2 $DATA/qemu-imgs/reference.vdi $TEMPDIR/output.qcow2 > exec diff -q $TEMPDIR/output.qcow2 $DATA/qemu-imgs/reference.qcow2 > > (of course, you could encapsulate your unit-test runner as one > test inside autotest) > > The way I see test automation working at the developer level is > similar to what we have in WebKit (my previous project): dozens > of thousands of small test scripts (28k last time I counted), run > all the time on a grid of buildbots and on developer machines. > Tests are maintained by the devel team and constantly growing > because bugfixes and new features are acompanied by new tests. > > And a broken test is considered as bad as a broken build. > > BUT we'll need a lot of effort and time to get there. We hope > autotest will help somehow, but it's just a small baby step. Yes, I have a similar view of the long term goal. And just like those thousands of tests live in the main webkit tree, I think we need thousands of tests in qemu.git. Regards, Anthony Liguori > Cheers, > - Ademar >