Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori <anthony@codemonkey.ws>]

On 12/27/2011 11:01 PM, Cleber Rosa wrote:
> On 12/27/2011 11:37 PM, Anthony Liguori wrote:
>>> I think the main goal of qemu-tests (may be implicit) is to be quick and simple.
>>
>> qemu-test doesn't have a main goal. My goal is to improve QEMU's quality.
>> qemu-test is just a tool to help achieve that goal.
>
> Maybe I've used the wrong wording. I got the feeling that, besides testing qemu
> the way you need it, keeping qemu-test simple was really important.

Simple is always important.  In the case of qemu-test, there are some important 
trade offs made to keep things simple and fast.  It doesn't try to work with 
arbitrary guests which means the tests need to handle only one environment.  The 
guest is pre-made to have exactly what is needed for the tests so there is no 
setup required.  The guest is as small as possible such that the test can run as 
quickly as possible.

>>> That is indeed great, but if one thinks that's all we'll ever going to need,
>>> that thought is pretty naive.
>>
>> I don't know who "we" is, but I can tell you that qemu-test is exactly what
>> *I* need. Consider that I spent a good portion of every single day testing
>> QEMU with either my own or other people's patches, making that job easier and
>> more automated is fairly important to me.
>
> "We" is everyone that somehow contributes to QEMU, that is, the QEMU community.
> If you're only concerned about what *you* need, then you're on the right track.
> If, besides that, you feel it'd be nice to *try to* concentrate our efforts,
> then we're all on the same track.

There is no need to have a single tool that meets every possible need.  In fact, 
the Unix tradition is to have separate single purposed tools.

Having two test tools it not a bad thing provided that the overlap isn't 
significant.  We shouldn't be discussing whether it's possible to merge the two 
tools, but rather what the technical benefits of doing so would be.

Since at this point, there is almost no overlap between the two, I don't see any 
actual technical benefit to merging them.  I see benefit to autotest executing 
qemu-test, of course.

>> I'm sharing it because I suspect that a lot of other developers have a similar
>> need.
>>
>>> And it may be true that there's room for both test
>>> suites... or that, as busy developers, we're refusing to deal with the added
>>> complexity (qemu alone accounts for a lot) and delaying to fix the fixable. I
>>> believe on the later.
>>>
>>> One example: kvm-autotest has a complex configuration file format with a steep
>>> learning curve, while a test such as qemu-tests/tests/simple-ping.sh would have
>>> to be tweaked if somehow the kernel detects the first ethernet interface as em1
>>> (such as recent Fedora systems do). Simple, but not scalable.
>>
>> I can tell by this comment that you don't actually understand how qemu-test
>> works. Please take a look at it before jumping to conclusions about whether it
>> should or shouldn't be part of kvm-autotest.
>>
>> Hint: qemu-test always uses the same kernel because it builds it as part of
>> the test suite. The behavior of how a nic test will never change unless
>> someone explicitly changes the kernel.
>
> I can tell you did not get my point: I'm giving some reasons of why current kvm
> autotest is somehow complex, and how qemu-tests gets away and keeps it simple.

You're claiming that "we're refusing to deal with the added complexity (qemu 
alone accounts for a lot) and delaying to fix the fixable".

There is no way that qemu-test would ever need to deal with how Fedora 
configures udev to name ethernet devices without becoming something totally 
different than it is.  So there's no "delaying to fix the fixable" here.

qemu-test makes a simplifying assumption.  By restricting the guest to a fixed 
environment (initramfs w/busybox), things inherently become much, much simpler.

Of course, this is not an adequate assumption to make if it were our only test 
tool but fortunately, we have an existing test suite that does a very good job 
at testing a wide variety of guests :-)

>>>
>>> The Python requirement inside the guest is true *if* we want to run regular
>>> autotest tests inside the guest (see autotest/client/virt/tests/autotest.py) and
>>> this accounts for very very little of kvm autotest usage. All other tests
>>> interact with the monitor directly and with the guest via ssh/telnet/serial.
>>
>> qemu-test does not require any specific hardware to be used in the guest which
>> lets it test a wider variety of scenarios in QEMU. So you cannot assume there
>> is ssh/telnet/serial available.
>
> I really thought we could rely on, at least, a serial connection. If not, then
> indeed the current kvm autotest approach is not compatible with that test
> environment. That is not to say that kvm autotest couldn't incorporate the
> qemu-tests approach/functionality.

With the scope that qemu-test has, it cannot assume any hardware is present 
because it wants to test every piece of hardware.

> BTW, I just don't like the idea of having lots of functionalities/tests
> implemented on two test suites for a single piece of software, unless proven
> that there's no way around it. To me, this is the whole point of this discussion.

I actually disagree on principle.  Two tools shouldn't be combined unless 
there's a proven reason that they benefit from being together.  You get more 
flexibility at the end of the day having strong, independent components that can 
be combined together.

>>> So, I see no reason for not using a more expressive language,
>>
>> I seriously doubt you can build a useful initramfs that contains python
>> without doing something crazy like livecd-tools does....
>
> You're right. Again, I was thinking we could rely at least on a serial
> connection. Can we not?

No.  How would you test creating a guest with no serial devices?

The nice thing about delivering a test via the initramfs is that you're only 
injecting software into the guest (usually via some non guest visible mechanism).

>> I don't see any reason why everything needs to live in kvm-autotest... but if
>> you really feel that way, please provide patches that demonstrate how this
>> would work.
>
> If it's technically viable, I think that having it as part of kvm autotest,
> shows that the project is more cohesive, attracts more contributions, and makes
> better use of our efforts.

Just putting the code in kvm-autotest.git in a directory doesn't make sense to 
me.  Beyond the lack of a technical reason to do so, logistically, it makes it 
harder for me to ask people to submit test cases with a patch series if I can't 
actually apply the test case when I'm applying the patches to qemu.git.

If qemu-test didn't use large submodules (like linux.git), I would have made 
qemu-test part of qemu.git.  As far as I'm concerned, qemu-test.git is just an 
extension to qemu.git.

>> I think there's something of a knee jerk reaction here. The existence of
>> qemu-test does not take anything away from kvm-autotest. It's just another
>> tool in our arsenal to achieve our joint goal of making QEMU (and KVM) higher
>> quality.
>
> You're right, It does not take anything away from kvm autotest today. But
> suppose we can prove that kvm autotest can indeed absorve all of qemu-tests
> functionalities, it'd be itself a reason for doing so.

That's not obvious to me.

> It'd avoid finding
> ourselves with two evolved test tools that do some of the same things, but are
> separate implementations.

It's just too easy to argue about abstract things like this.  This discussion 
about testing has come up many times and we've talked to death how much we need 
to do better testing and get to test driven development.

Yet, not much has materialized.  I'm not interested in talking about things in 
abstract anymore.  That's why I announced qemu-test.  If folks think it should 
behave differently, then patches are welcome.

If you think that kvm-autotest can do everything that qemu-test does, just make 
the changes to kvm-autotest to make it behave the same way and show me.  If it 
makes sense, I'll happily embrace it.

Regards,

Anthony Liguori