Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori <anthony@codemonkey.ws>]

On 03/08/2012 10:05 AM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote:
>> On 03/08/2012 09:07 AM, Ademar Reis wrote:
>>> On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote:
>>>> On 03/08/2012 08:49 AM, Ademar Reis wrote:
>>>>> On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
>>>>>> On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
>>>>>>> Virt/qemu tests: Minimal guest images
>>>>>>> -------------------------------------
>>>>>>>
>>>>>>> In order to make development level test possible, we need the tests to run fast.
>>>>>>> In order to do that, a set of minimal guest images is being developed and we
>>>>>>> have a version for x86_64 ready and functional:
>>>>>>>
>>>>>>> https://github.com/autotest/buildroot-autotest
>>>>>>
>>>>>> I'm really not a fan of buildroot.  Note that in order to ship
>>>>>> binaries, full source needs to be provided in order to comply with
>>>>>> the GPL.  The FSF at least states that referring to another website
>>>>>> for source that's not under your control doesn't satisfy the
>>>>>> requirements of the GPL.
>>>>>>
>>>>>> Just out of curiosity, did you try to use qemu-test?  Is there a
>>>>>> reason you created something different?
>>>>>>
>>>>>> I think it's good that you're thinking about how to make writing
>>>>>> tests easier, but we have a growing test infrastructure in QEMU and
>>>>>> that's what I'd prefer people focused on.
>>>>>>
>>>>>
>>>>> You probably remember the long thread we had back in December on
>>>>> qemu-devel on this topic. Back then our message was "we have a
>>>>> growing test infrastructure in s/QEMU/autotest/ and that's what
>>>>> we'd prefer people focused on". :-)
>>>>>
>>>>>  From Dor:
>>>>>
>>>>> (http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
>>>>>
>>>>> """
>>>>> If you wish, you can challenge Lucas and Cleber w/ these type of
>>>>> requirements and we'll all improve as a result.
>>>>> """
>>>>>
>>>>> Your response was:
>>>>>
>>>>> """
>>>>> Well consider qemu-test the challenge. It's an existence proof
>>>>> that we can have a very easy to use framework for testing that
>>>>> runs extremely fast and is very easy to write tests for.
>>>>> """
>>>>>
>>>>> http://knowyourmeme.com/memes/challenge-accepted ;-)
>>>>>
>>>>> I particularly agreed with basically everything you said on that
>>>>> discussion regarding test simplification (I had just joined the
>>>>> team back then). To me, autotest has been focusing on QE-level,
>>>>> leaving the developer-level test requirements out. Now we're
>>>>> attacking this new front, and a lot of the requirements are
>>>>> indeed from that discussion.
>>>>
>>>> If you want to talk about this in terms of "requirements", my
>>>> requirement is for "developer-level" tests to live in qemu.git and
>>>> be integrated into make check.
>>>>
>>>> Just as we've been discussing and working on since the previous set of discussions.
>>>>
>>>>> By simplifying the design and bringing barriers down, we hope to
>>>>> reach a broader audience and help developers write and maintain
>>>>> tests, benefiting from all the instrumentation that autotest
>>>>> brings. It's not going to be just about qemu (check the new test
>>>>> examples).
>>>>>
>>>>> We have a team fully dedicated to autotest and it's used not only
>>>>> by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
>>>>> (these all have code contributions in autotest)
>>>>>
>>>>> That said, the current qemu-tests will probably be easily
>>>>> integrated into (the new) autotest and we hope that, given enough
>>>>> time, autotest will be good enough to relieve qemu from the
>>>>> framework maintenance and code duplication with other projects.
>>>>
>>>> autotest should not be the focal point for integration.  qemu.git should be.
>>>>
>>>> I'd be perfectly happy to review patches submitting the test
>>>> infrastructure from kvm-autotest into qemu.git (provided it didn't
>>>> have unreasonable external dependencies and fit into QEMU).
>>>>
>>>> Developer-level tests need to live where the developers live.  The
>>>> developers live in qemu.git.  See my other response on this thread
>>>> for the explanation of why this is so important.
>>>>
>>>
>>> Excelent, we're in the same page then. This was my number 1
>>> requirement when I was discussing the changes with Lucas and
>>> Cleber. For convenience, I'll repeat here what I wrote in a
>>> previous e-mail (no qemu-devel archive available yet to use as a
>>> reference).
>>>
>>> In summary, autotest is (or is going to be) a framework that
>>> provides:
>>>
>>>   - A test runner, with grid/cluster support and advanced
>>>     instrumentation
>>>   - A devel library and set of utilities for test writers
>>>   - A set of pre-built images (JeOS – Just Enough OS) for
>>>     test writers
>>
>> I don't think autotest is the right place for this to live.  We need
>> this directly in qemu.git otherwise we're severely limited in what
>> tests we can write.
>>
>> I guess that doesn't preclude autotest having its own JeOS
>> mechanism, but we clearly need one in qemu.git.
>>
>>>
>>> (attached is a picture showing what we want to achieve)
>>>
>>> If a project has an internal library or set of utilities that can
>>> be of general use, they can be submitted to autotest.git for
>>> inclusion, thus reaching a broader audience.
>>>
>>> A short summary of the plans:
>>>
>>>   - Tests can live anywhere and each devel team implements and
>>>     maintains their own set of tests
>>
>> Let me change this to:
>>
>> - Autotest will learn how to harness the tests that each development
>> team creates in their respective git repository.
>
> Yep, given this simple requirement: "tests return 0 or an error
> code". The output from the test runner will be a simple PASS/FAIL
> and stdout/stderr will be properly collected by the test runner
> and made available in a standard location. If using TAP, even
> better.

We use gtest, so this will not be the case.  I think it would be best for 
autotest to learn how to interact with the gtest protocol.

>
> This will allow trivial tests. As the test complexity grows, then
> developers may start using parts of the autotest library or
> utilities, thus requiring it installed (a trivial bootstrap
> procedure is also a requirement, see the e-mail from Lucas).

What is the value of the autotest library to a test writer other than being part 
of the "autotest framework"?

>> This is where you start to lose me.  If all you're saying is, "QEMU
>> can continue to use gtest to build out it's test infrastructure and
>> autotest will learn how to use it", then we're in violent agreement.
>
> Good :-)
>
> Inside qemu.git, you can add as many instrumentation and helper
> libraries as you want. As a developer it'll be up to you to setup
> your environment to run the tests, and autotest will be just yet
> another dependency.

I think there's an implicit assumption here that the tests in qemu.git will have 
a dependency on autotest.  I'd like to understand why this dependency is necessary.

Normally, autotest executes third party tests, what makes QEMU special here?

> QE will probably be interested in setting up a bot with several
> tests from differente repositories, but that's their problem, not
> yours.

Right, and this is what autotest is very good at :-)

> BTW, please note that we're not trying to cover unit-tests in
> autotest. At least not by the definition of unit-tests that I
> know, which are code-level tests without running the application
> itself (unit-tests test functions/methods only, not the
> application as a whole).

There is never a clear line between unit-tests and integration tests as long as 
you're talking about a single project.

I expect QEMU to grow tests for anything that involves launching QEMU directly. 
  Where I would not see QEMU growing tests for is things like launching QEMU 
through libvirt.

> Autotest is more about integration tests, which are complementary
> to unit-tests in a "test driven development" environment: you
> instantiate your program by using scripts and check if things
> happen as you expect, as in this trivial example:
>
> qemu-img-convert.sh
>      #!/bin/bash
>      qemu-img convert -O qcow2 $DATA/qemu-imgs/reference.vdi $TEMPDIR/output.qcow2
>      exec diff -q $TEMPDIR/output.qcow2 $DATA/qemu-imgs/reference.qcow2
>
> (of course, you could encapsulate your unit-test runner as one
> test inside autotest)
>
> The way I see test automation working at the developer level is
> similar to what we have in WebKit (my previous project): dozens
> of thousands of small test scripts (28k last time I counted), run
> all the time on a grid of buildbots and on developer machines.
> Tests are maintained by the devel team and constantly growing
> because bugfixes and new features are acompanied by new tests.
>
> And a broken test is considered as bad as a broken build.
>
> BUT we'll need a lot of effort and time to get there. We hope
> autotest will help somehow, but it's just a small baby step.

Yes, I have a similar view of the long term goal.  And just like those thousands 
of tests live in the main webkit tree, I think we need thousands of tests in 
qemu.git.

Regards,

Anthony Liguori

> Cheers,
>     - Ademar
>