Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Cleber Rosa <crosa@redhat.com>]

On 03/08/2012 08:21 PM, Anthony Liguori wrote:
> On 03/08/2012 04:24 PM, Ademar Reis wrote:
>> On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
>>> On 03/08/2012 03:02 PM, Ademar Reis wrote:
>>>> On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>>>>> On 03/08/2012 11:59 AM, Ademar Reis wrote:
>>>>
>>>> <snip>
>>>>
>>>>>
>>>>>>> I expect QEMU to grow tests for anything that involves launching
>>>>>>> QEMU directly.  Where I would not see QEMU growing tests for is
>>>>>>> things like launching QEMU through libvirt.
>>>>>>
>>>>>> Agree. For QEMU developers, libvirt should not be on the way, the
>>>>>> interaction should be minimal or non-existent.
>>>>>>
>>>>>> That's an area which will require some work in libautotest,
>>>>>> because due to previous QE requirements, it now invokes libvirt
>>>>>> methods instead of QEMU directly for a lot of stuff. But it can
>>>>>> be done and is part of the plan.
>>>>>
>>>>> If you're talking about libautotest as an abstraction layer for
>>>>> launching QEMU, I don't think that's something we should use in
>>>>> upstream QEMU.  Abstraction layers are okay when the things you are
>>>>> abstracting are well defined and relatively static.  We're talking
>>>>> about testing the tip of a development tree though and coordinating
>>>>> changes with an abstraction layer would be counter productive.
>>>>>
>>>>
>>>> Valid point. If I'm a qemu developer and I want to make huge
>>>> changes, ideally I shouldn't have to submit patches to
>>>> libautotest because the tests in qemu.org will break.
>>>>
>>>> I don't know how likely such scenario is (Lucas is maitaining
>>>> kvm-autotest for years now and the integration hardly breaks)
>>>> But we do have a conflict of interests, so lets evaluate a
>>>> few options:
>>>
>>> Because kvm-autotest doesn't have thousands of tests and isn't part
>>> of 200+ developers fast paths :-)
>>>
>>>>
>>>> My main interests are:
>>>>      - remove barriers for developers to write tests
>>>
>>> This is my number one priority.
>>>
>>>>      - share the testing effort between multiple projects (qemu,
>>>>        libvirt, ovirt, spice, QE, etc...)
>>>
>>> This is my number 99 priority.
>>
>> You have the right to consider the interests of qemu only, but
>> qemu is used by several other teams who depend on it to implement
>> their own tests. Some collaboration would be extremely beneficial
>> for everybody.
>>
>> Collaboration goes both ways: you get benefits from the
>> interaction and code reuse with different projects and teams, but
>> it may require a few tradeoffs.
>
> I think the problem here effectively boils down to "code sharing at 
> any cost".
>
> Let's look at it in purely practical terms.  How many lines of code 
> would be saved here and what does that code do?
>
> If the only sharing is some infrastructure bits in libautotest, then I 
> suspect we're talking in the low 10ks of lines at the absolute maximum 
> (and I don't think it's even close to that in reality).  Launching 
> QEMU and connecting to QMP isn't that hard.  It's in the hundreds of 
> lines of code.  What else is there that's need to write a test?
>
> I don't think this is sufficiently compelling to add an unconditional 
> autotest dependency to QEMU.
>
>>> I think you could achieve practical code sharing by decomposing
>>> kvm-autotest into a set of useful, independent tools.  For
>>> instance....
>>>
>>>> Now the options:
>>>>
>>>> 1. We keep the invocation stuff in libautotest and when writting
>>>> a complex test in qemu.git, the test writter can choose to use it
>>>> because of the goodies there.
>>>>
>>>> Pros:
>>>>    - Lots of codepaths implemented both in python and as cmd-line
>>>>      utilities: less lines of code to write a test, smaller
>>>>      barrier for the developer.
>>>
>>> I've got an existence proof that this is not true.  The qemu-test
>>> tests are smaller than the corresponding autotest tests.
>>
>> You're comparing developer-level tests with the existent QA-level
>> tests (much more complex).
>
> Let's be specific then.  Look at device-add.sh in qemu-test.  It's 
> 71LOC. pci_hotplug.py in autotest is 204LOC.

pci_hotplug.py does much more than device-add.sh:

    * tests both pci_add and device_add commands
    * checks the monitor syntax for adding a new drive, that is, it 
works on HEAD and on other versions (such as the ones in some RHEL 
releases);
    * tests both nic and block hotplug
    * for block, tests with both virtio and scsi
    * also does device removal, both for pci_add and device_add syntaxes

>
> What's the "much more complex" part of pci_hotplug.py that is enabled 
> by having the autotest framework?
>
>> We're talking about an approach where you'll need just a few
>> lines of code to instantiate a VM and interact with it.
>
> This isn't that hard of a problem.  It just takes a few lines to 
> launch QEMU in qemu-test too.
>
>>>>    - Mature code in the test library.
>>>>    - Goodies such as video-recording and system info collection
>>>>      in a standard way for different projects.
>>>
>>> Why does video-recording have to be part of libautotest?  Why can't
>>> you just have a simply utility that connects to a VNC session and
>>> records it?  Presumably that's all it's doing here.
>>>
>>> Likewise, there are dozens of "system info collection" such as
>>> sosreport..  why fold this all into libautotest?
>>
>> For standardization: think outside of the qemu community. A lot
>> of our developers contribute code to multiple projects. It gets
>> easier if there's a standard way to achieve the same results when
>> writting tests for different projects.
>>
>> Anyway, it's a minor issue.
>
> Yeah, I think this is outside of the scope of the current discussion 
> as it's sort of like trying to boil the ocean :-)
>
>>>>    - Code shared with other teams.
>>>>    - The same test code can run on old platforms because
>>>>      libautotest has a compatibility layer, thus enabling a
>>>>      larger grid of tests.
>>>
>>> This is not a requirement from a QEMU perspective nor do I think 
>>> it's useful.
>>
>> Well, it's not a requirement, but you can't expect QE teams to
>> spend their time on tests that work only on HEAD. It makes no
>> sense from their point of view.
>
> They're certainly testing some sort of HEAD, even if it's not 
> qemu.git.  I don't see any conflicting goals here.  Write a test 
> against upstream, backport to a downstream if necessarily.
>
>>
>>>
>>>>    - QE teams will be closer to the development, sharing
>>>>      some of their testing code and maintenance burden.
>>>
>>> The QE team could simply contribute tests to qemu.git to achieve 
>>> this same goal.
>>
>> They won't help you if you don't help them. Again, it doesn't
>> make sense for QE to work on HEAD. At least not the QE teams I'm
>> talking about.
>>
>> And I say from my own experience: maintaining a set of thousands
>> of tests is far from trivial. You'll waste a lot of potential
>> contributions by alienating the fully-dedicated teams currently
>> working on QE.
>
> And trying to maintain a thousand tests against any possible version 
> is probably also unrealistic.
>
> Is the expectation that all of the tests in WebKit work against any 
> possible version of WebKit?  How do they enforce this since it's all 
> in trunk?
>
>>>> 2. We "move" (or implement) all the libautotest stuff related to
>>>> qemu invocation and interaction into qemu.git, forbidding the
>>>> dependency of autotest in qemu.git.
>>>
>>> Let me just be really clear here.  In order to do TDD, tests have to
>>> be a mandatory part of the QEMU build process.  We cannot introduce
>>> new dependencies in order to build tests.
>>>
>>> Making autotest an unconditional dependency of qemu.git is a 
>>> non-starter for me.
>>
>> You have dependencies on gtest and other libraries. it's not that
>> different.
>
> We have a dependency on glib, which gtest happens to be a part of.  It 
> was a big deal introducing that dependency and gtest was not the only 
> (or primary) reason to introduce that dependency.
>
>> Plus it's not unconditional: the test runner will report tests
>> SKIPPED if a dependency is not present.
>
> But then the tests aren't run so if most developers didn't have it 
> installed, and most tests were written with it, most developers 
> wouldn't be running most tests which defeats the purpose, no?
>
>> It's actually the oposite: you'll have to forbid developers from
>> submiting tests that use libautotest if you don't want the
>> dependency.
>
> I think part of the problem here is the abstract nature of this 
> discussion. What is libautotest actually supposed to do?
>
> This would be an easier discussion if we were talking about patches 
> instead of proposals.
>
>>>>    - QE will be alienated from the qemu test effort. There will be
>>>>      no integration between the QE efforts and the maintenance of
>>>>      the qemu developer-level tests.
>>>
>>> I think we're a pretty friendly and open community :-)  There is no
>>> reason that QE should be "alienated" unless folks are choosing not
>>> to participate upstream.
>>
>> For the exact same reasons you as a developer don't want to
>> implement tests inside autotest, QE won't want to implement tests
>> for qemu.git. It's out of their comfort zone, just put yourself
>> on their shoes.
>
> This is a really, really poor argument and I hope I don't need to go 
> into details of why.  If the primary reason for libautotest is so the 
> people writing tests for QEMU can avoid actually working with the 
> developers of QEMU...  we've got a problem.
>
>>>
>>>>    - You don't get the goodies from autotest (standardized system
>>>>      info collection, video recording, grid support, etc).
>>>>    - The new tests will work only on the master branch.
>>>
>>> This last one is a legitimate point that I have considered myself.
>>> But I think the value of having tests in HEAD outweigh the cost
>>> here.
>>
>> Thinking about qemu only, agree.
>>
>>>
>>>> 3. A mix of (1) and (2): we "move" everything under qemu.git, but
>>>> keep the compatibility layer and provide a "libqemutest" for
>>>> third-parties.
>>>>
>>>> Anybody writting a test that interacts with qemu will use this
>>>> library: qemu, libvirt, spice, ovirt, QE tests, etc.
>>>
>>> I really see this all as over architecting to be honest.
>>>
>>> Can someone explain in clear terms (without appealing to maturity,
>>> flexibility, or any other qualitative aspect) why it takes anything
>>> more than a few dozen shell functions to write all of the tests that
>>> are in kvm-autotest test today?
>>
>> This is close to being offensive IMO, so I won't answer. :-)
>
> I honestly don't mean it to be so I apologize if it comes off that 
> way.  But I dislike arguing in such abstract terms.  What are the 
> features that you think libautotest is needed for?
>
>>> You launch qemu (by just using a command), and you interact with the
>>> monitor via QMP and HMP.  Why do you ever need anything more than
>>> that?
>>
>> Well, again, I'll agree to disagree.
>>
>>>
>>> Why does libautotest even need to exist?
>>
>> You're looking at things from the sole point of view of a qemu
>> developer. If you don't want to see things from a different
>> perspective and accept any kind of tradeoffs for better
>> collaboration with other teams, then unfortunately there's not
>> much we can do. It's a pity, because collaboration goes both
>> ways.
>
> I need something more concrete then this.  What is it about 
> libautotest that would allow collaboration?
>
>> We're trying very hard to satisfy your requirements and
>> interests. Don't you see any way you can help us satisfy
>> ours?
>
> I'm having a hard time understanding what your requirements are other 
> than that we should write our tests using libautotest (which has not 
> really been explained other than being levied as a requirement).
>
> Regards,
>
> Anthony Liguori