Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori <anthony@codemonkey.ws>]

On 03/08/2012 03:02 PM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>> On 03/08/2012 11:59 AM, Ademar Reis wrote:
>
> <snip>
>
>>
>>>> I expect QEMU to grow tests for anything that involves launching
>>>> QEMU directly.  Where I would not see QEMU growing tests for is
>>>> things like launching QEMU through libvirt.
>>>
>>> Agree. For QEMU developers, libvirt should not be on the way, the
>>> interaction should be minimal or non-existent.
>>>
>>> That's an area which will require some work in libautotest,
>>> because due to previous QE requirements, it now invokes libvirt
>>> methods instead of QEMU directly for a lot of stuff. But it can
>>> be done and is part of the plan.
>>
>> If you're talking about libautotest as an abstraction layer for
>> launching QEMU, I don't think that's something we should use in
>> upstream QEMU.  Abstraction layers are okay when the things you are
>> abstracting are well defined and relatively static.  We're talking
>> about testing the tip of a development tree though and coordinating
>> changes with an abstraction layer would be counter productive.
>>
>
> Valid point. If I'm a qemu developer and I want to make huge
> changes, ideally I shouldn't have to submit patches to
> libautotest because the tests in qemu.org will break.
>
> I don't know how likely such scenario is (Lucas is maitaining
> kvm-autotest for years now and the integration hardly breaks)
> But we do have a conflict of interests, so lets evaluate a
> few options:

Because kvm-autotest doesn't have thousands of tests and isn't part of 200+ 
developers fast paths :-)

>
> My main interests are:
>      - remove barriers for developers to write tests

This is my number one priority.

>      - share the testing effort between multiple projects (qemu,
>        libvirt, ovirt, spice, QE, etc...)

This is my number 99 priority.

I think you could achieve practical code sharing by decomposing kvm-autotest 
into a set of useful, independent tools.  For instance....

> Now the options:
>
> 1. We keep the invocation stuff in libautotest and when writting
> a complex test in qemu.git, the test writter can choose to use it
> because of the goodies there.
>
> Pros:
>    - Lots of codepaths implemented both in python and as cmd-line
>      utilities: less lines of code to write a test, smaller
>      barrier for the developer.

I've got an existence proof that this is not true.  The qemu-test tests are 
smaller than the corresponding autotest tests.

>    - Mature code in the test library.
>    - Goodies such as video-recording and system info collection
>      in a standard way for different projects.

Why does video-recording have to be part of libautotest?  Why can't you just 
have a simply utility that connects to a VNC session and records it?  Presumably 
that's all it's doing here.

Likewise, there are dozens of "system info collection" such as sosreport..  why 
fold this all into libautotest?

>    - Code shared with other teams.
>    - The same test code can run on old platforms because
>      libautotest has a compatibility layer, thus enabling a
>      larger grid of tests.

This is not a requirement from a QEMU perspective nor do I think it's useful.

>    - QE teams will be closer to the development, sharing
>      some of their testing code and maintenance burden.

The QE team could simply contribute tests to qemu.git to achieve this same goal.

> 2. We "move" (or implement) all the libautotest stuff related to
> qemu invocation and interaction into qemu.git, forbidding the
> dependency of autotest in qemu.git.

Let me just be really clear here.  In order to do TDD, tests have to be a 
mandatory part of the QEMU build process.  We cannot introduce new dependencies 
in order to build tests.

Making autotest an unconditional dependency of qemu.git is a non-starter for me.

> In a way, that's what tends to happen with qemu-test if things
> keep going on AS IS in qemu.org (I'm absolutely sure that you'll
> replicate a lot of what autotest does as you increase the test
> coverage and complexity).
>
> Pros:
>    - Everything under qemu.git control: you break it, you fix it.
>    - Organic growth of the test infra-structure in QEMU
>
> Cons:
>    - Lot of code will be duplicated to cover the main code paths:
>      writting tests will require writting/supporting considerable
>      ammount of code (that already exists in autotest).

Again, existence proof that this isn't true.

>    - QE will be alienated from the qemu test effort. There will be
>      no integration between the QE efforts and the maintenance of
>      the qemu developer-level tests.

I think we're a pretty friendly and open community :-)  There is no reason that 
QE should be "alienated" unless folks are choosing not to participate upstream.

>    - You don't get the goodies from autotest (standardized system
>      info collection, video recording, grid support, etc).
>    - The new tests will work only on the master branch.

This last one is a legitimate point that I have considered myself.  But I think 
the value of having tests in HEAD outweigh the cost here.

> 3. A mix of (1) and (2): we "move" everything under qemu.git, but
> keep the compatibility layer and provide a "libqemutest" for
> third-parties.
>
> Anybody writting a test that interacts with qemu will use this
> library: qemu, libvirt, spice, ovirt, QE tests, etc.

I really see this all as over architecting to be honest.

Can someone explain in clear terms (without appealing to maturity, flexibility, 
or any other qualitative aspect) why it takes anything more than a few dozen 
shell functions to write all of the tests that are in kvm-autotest test today?

You launch qemu (by just using a command), and you interact with the monitor via 
QMP and HMP.  Why do you ever need anything more than that?

Why does libautotest even need to exist?

Regards,

Anthony Liguori