Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Cleber Rosa <crosa@redhat.com>]

On 12/27/2011 11:37 PM, Anthony Liguori wrote:
> On 12/27/2011 04:35 PM, Cleber Rosa wrote:
>> On 12/26/2011 08:00 PM, Dor Laor wrote:
>>> On 12/26/2011 05:12 PM, Anthony Liguori wrote:
>>>> Hi Dor,
>>>>
>>>
>>> Merry Christmas Anthony,
>>>
>>>> On 12/25/2011 09:19 AM, Dor Laor wrote:
>>>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>>>>>
>>>>> Well, I'm still not convinced that a new standalone package should
>>>>> handle these
>>>>> cases instead of kvm autotest. I'll be happy to integrate the 
>>>>> tests to
>>>>> kvm
>>>>> autotest anyway and the more the merrier but imho it's a duplicate.
>>>>
>>>> I'm sure kvm autotest could be taught to do exactly what qemu-test is
>>>> doing. But why does kvm autotest have to do everything? I doubt there
>>>> would be much code reuse.
>>>>
>>>> I think it's not a bad thing to have multiple test suites when there
>>>> isn't considerable overlap.
>>
>> I think the main goal of qemu-tests (may be implicit) is to be quick 
>> and simple.
>
> qemu-test doesn't have a main goal.  My goal is to improve QEMU's 
> quality. qemu-test is just a tool to help achieve that goal.

Maybe I've used the wrong wording. Besides the different approach 
(simpler requirements, initramfs with busybox, etc), it looks to me that 
keeping it simple played an important role in your decision to write 
qemu-tests, which is indeed a sign of good design. But read on...

>
>> That is indeed great, but if one thinks that's all we'll ever going 
>> to need,
>> that thought is pretty naive.
>
> I don't know who "we" is, but I can tell you that qemu-test is exactly 
> what *I* need.  Consider that I spent a good portion of every single 
> day testing QEMU with either my own or other people's patches, making 
> that job easier and more automated is fairly important to me.

"We" is everyone that contributes one way or another to qemu. So if you 
only concerned with your needs, you're definitely on the right track.

If not, I think (trying to think as a community) it'd be beneficial to 
concentrate all efforts, unless it's not possible to do so. It's as 
simple as that. All my reasoning here revolves around this.

>
> I'm sharing it because I suspect that a lot of other developers have a 
> similar need.
>
>> And it may be true that there's room for both test
>> suites... or that, as busy developers, we're refusing to deal with 
>> the added
>> complexity (qemu alone accounts for a lot) and delaying to fix the 
>> fixable. I
>> believe on the later.
>>
>> One example: kvm-autotest has a complex configuration file format 
>> with a steep
>> learning curve, while a test such as qemu-tests/tests/simple-ping.sh 
>> would have
>> to be tweaked if somehow the kernel detects the first ethernet 
>> interface as em1
>> (such as recent Fedora systems do). Simple, but not scalable.
>
> I can tell by this comment that you don't actually understand how 
> qemu-test works.  Please take a look at it before jumping to 
> conclusions about whether it should or shouldn't be part of kvm-autotest.

I can tell you did not grasp my point: kvm autotest is more complex, but 
more capable and flexible. And I did *not* come to a conclusion, I'm 
giving examples in an attempt to enrich the discussion.

>
> Hint: qemu-test always uses the same kernel because it builds it as 
> part of the test suite.  The behavior of how a nic test will never 
> change unless someone explicitly changes the kernel.

Hopefully you understand by now that I'm giving some reasons of why 
kvm-autotest does some things the way it does (usually more complex), 
and how qemu-tests, because of its approach, does not have to deal with 
that.

>
>>>>>> 1) It builds a custom kernel and initramfs based on busybox. This is
>>>>>> fairly important to ensure that we can run tests with no device
>>>>>> pre-requisites.
>>>>>
>>>>> This can be done easily w/ autotest too.
>>
>> The Python requirement inside the guest is true *if* we want to run 
>> regular
>> autotest tests inside the guest (see 
>> autotest/client/virt/tests/autotest.py) and
>> this accounts for very very little of kvm autotest usage. All other 
>> tests
>> interact with the monitor directly and with the guest via 
>> ssh/telnet/serial.
>
> qemu-test does not require any specific hardware to be used in the 
> guest which lets it test a wider variety of scenarios in QEMU.  So you 
> cannot assume there is ssh/telnet/serial available.

I was assuming that we could count on at least a serial port in the 
guest. If not, then current kvm-autotest can not absorb the same 
functionality of qemu-tests without re-writing it. A

>
>>
>> So, I see no reason for not using a more expressive language,
>
> I seriously doubt you can build a useful initramfs that contains 
> python without doing something crazy like livecd-tools does....
>
>>>> Actually, kvm-autotest has various layers of abstraction in how QEMU
>>>> ends up being launched. As you mention below, those layers are 
>>>> there to
>>>> allow for things like using libvirt.
>>
>> Indeed the qemu command line parameters gets generated depending on many
>> configuration parameters. It'd be *really* simple to add a configuration
>> parameters that overwrites the qemu command with an static one.
>
> But if you're a QEMU developer, you want to have as much control of 
> the command line as possible.  For instance, one of the tests in 
> qemu-test makes sure to test invocations without -device as this 
> triggers a different code path (there was a recent regression in this 
> too).  You can't just add arguments to reproduce this behavior.
>
>>>
>>> It goes beyond that, since it also related to the monitor interface 
>>> as well.
>>>
>>>> That's desirable when you're doing "virt testing", but not so 
>>>> desirably
>>>> when you're trying to write specific unit tests against QEMU.
>>>
>>> True, one may not need it at all but it's nice that a test for
>>> migration/stress/hotplug will be tested directly w/ qemu and libvirt 
>>> w/ the
>>> same effort.
>>>
>>>>
>>>>>> 5) The tests execute very quickly, can be run stand alone, and do 
>>>>>> not
>>>>>> require root privileges
>>>>>
>>>>> ditto for kvm auotest. It's possible to configure it w/o root too
>>>>> which is not a
>>>>> huge issue.
>>>>
>>>> When I say, "run quickly", I mean, they execute very quickly.
>>>
>>> /me too
>>>
>>>>
>>>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
>>>> tests/virtio-serial.sh
>>>>
>>>> real 0m4.385s
>>>> user 0m1.460s
>>>> sys 0m1.860s
>>>
>>> That's impressive but it's more of a function of the guest being 
>>> used - if
>>> instead of running a full Fedora install, you'll chose your busybox 
>>> image w/
>>> -kernel/initrd you'll get a similar result.
>>
>> I also think so. Maybe kvm-autotest would take a little more time 
>> because of the
>> different approach we take when communicating with the guest, but I 
>> bet it'd be
>> irrelevant.
>
> I don't see any reason why everything needs to live in kvm-autotest... 
> but if you really feel that way, please provide patches that 
> demonstrate how this would work.  We could argue indefinitely about 
> how things could work, it's much better to compare how things actually 
> do work :-)
>
>>> I agree autotest is not perfect but it likes to be such.
>>> If you wish, you can challenge Lucas and Cleber w/ these type of 
>>> requirements
>>> and we'll all improve as a result.
>>
>> Yes, I believe it'd be a nice exercise for all of us.
>>
>> The only thing I ask is that we bear at least with some of the 
>> complexity that
>> kvm-autotest inherently holds...
>
> I think there's something of a knee jerk reaction here.  The existence 
> of qemu-test does not take anything away from kvm-autotest.  It's just 
> another tool in our arsenal to achieve our joint goal of making QEMU 
> (and KVM) higher quality.
>
> autotest is made to invoke third party tests so the two tools can 
> co-exist in a complimentary way.
>
> Regards,
>
> Anthony Liguori
>
>> the last requests we had was to get rid of all
>> the complexity, while retaining all the other nice characteristics. 
>> Pretty hard,
>> so I think we failed, or maybe only half-succeeded at it.
>>
>>>
>>> Cheers,
>>> Dor
>>>
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>>>>
>>>>>> 6) They are random by nature with the ability to fix the seed in 
>>>>>> order
>>>>>> to be used in git-bisect.
>>>>>>
>>>>>> I think Gerd had been looking at doing something similar with a 
>>>>>> custom
>>>>>> initrd.
>>>>>>
>>>>>> I've tried to consider other architectures and had hoped that we 
>>>>>> could
>>>>>> commit the vmlinuz and initramfs into git so that it was easy to 
>>>>>> test
>>>>>> other architectures without having a full build environment.
>>>>>> Unfortunately, busybox doesn't link statically with glibc and I 
>>>>>> can't
>>>>>> see an obvious way to commit binaries while respecting the GPL 
>>>>>> since we
>>>>>> need to pull glibc into the initramfs.
>>>>>>
>>>>>> I know buildroot exists specifically to deal with this but in my
>>>>>> experience, buildroot is very unreliable and extremely heavy weight
>>>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>>>>>> environment.
>>>>>>
>>>>>> Anyway, the code is available at:
>>>>>>
>>>>>> http://git.qemu.org/qemu-test.git
>>>>>>
>>>>>> See the README for instructions on how to use it.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Anthony Liguori
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>