Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Cleber Rosa <crosa@redhat.com>]

On 12/27/2011 11:37 PM, Anthony Liguori wrote:
> On 12/27/2011 04:35 PM, Cleber Rosa wrote:
>> On 12/26/2011 08:00 PM, Dor Laor wrote:
>>> On 12/26/2011 05:12 PM, Anthony Liguori wrote:
>>>> Hi Dor,
>>>>
>>>
>>> Merry Christmas Anthony,
>>>
>>>> On 12/25/2011 09:19 AM, Dor Laor wrote:
>>>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>>>>>
>>>>> Well, I'm still not convinced that a new standalone package should
>>>>> handle these
>>>>> cases instead of kvm autotest. I'll be happy to integrate the 
>>>>> tests to
>>>>> kvm
>>>>> autotest anyway and the more the merrier but imho it's a duplicate.
>>>>
>>>> I'm sure kvm autotest could be taught to do exactly what qemu-test is
>>>> doing. But why does kvm autotest have to do everything? I doubt there
>>>> would be much code reuse.
>>>>
>>>> I think it's not a bad thing to have multiple test suites when there
>>>> isn't considerable overlap.
>>
>> I think the main goal of qemu-tests (may be implicit) is to be quick 
>> and simple.
>
> qemu-test doesn't have a main goal.  My goal is to improve QEMU's 
> quality. qemu-test is just a tool to help achieve that goal.

Maybe I've used the wrong wording. I got the feeling that, besides 
testing qemu the way you need it, keeping qemu-test simple was really 
important. If not, I'd imagine you'd try to re-use something that 
already exists (yes, kvm autotest).

>
>> That is indeed great, but if one thinks that's all we'll ever going 
>> to need,
>> that thought is pretty naive.
>
> I don't know who "we" is, but I can tell you that qemu-test is exactly 
> what *I* need.  Consider that I spent a good portion of every single 
> day testing QEMU with either my own or other people's patches, making 
> that job easier and more automated is fairly important to me.

"We" is everyone that somehow contributes to QEMU, that is, the QEMU 
community. If you're only concerned about what *you* need, then you're 
on the right track. If, besides that, you feel it'd be nice to *try to* 
concentrate our efforts, then we're all on the same track.

>
> I'm sharing it because I suspect that a lot of other developers have a 
> similar need.
>
>> And it may be true that there's room for both test
>> suites... or that, as busy developers, we're refusing to deal with 
>> the added
>> complexity (qemu alone accounts for a lot) and delaying to fix the 
>> fixable. I
>> believe on the later.
>>
>> One example: kvm-autotest has a complex configuration file format 
>> with a steep
>> learning curve, while a test such as qemu-tests/tests/simple-ping.sh 
>> would have
>> to be tweaked if somehow the kernel detects the first ethernet 
>> interface as em1
>> (such as recent Fedora systems do). Simple, but not scalable.
>
> I can tell by this comment that you don't actually understand how 
> qemu-test works.  Please take a look at it before jumping to 
> conclusions about whether it should or shouldn't be part of kvm-autotest.
>
> Hint: qemu-test always uses the same kernel because it builds it as 
> part of the test suite.  The behavior of how a nic test will never 
> change unless someone explicitly changes the kernel.

I can tell you did not get my point: I'm giving some reasons of why 
current kvm autotest is somehow complex, and how qemu-tests gets away 
and keeps it simple.

BTW, I did not jumped at any conclusion. I'm just trying to enrich the 
discussion, which may end up proving that there's no other way to have 
what qemu-tests does.

>
>>>>>> 1) It builds a custom kernel and initramfs based on busybox. This is
>>>>>> fairly important to ensure that we can run tests with no device
>>>>>> pre-requisites.
>>>>>
>>>>> This can be done easily w/ autotest too.
>>
>> The Python requirement inside the guest is true *if* we want to run 
>> regular
>> autotest tests inside the guest (see 
>> autotest/client/virt/tests/autotest.py) and
>> this accounts for very very little of kvm autotest usage. All other 
>> tests
>> interact with the monitor directly and with the guest via 
>> ssh/telnet/serial.
>
> qemu-test does not require any specific hardware to be used in the 
> guest which lets it test a wider variety of scenarios in QEMU.  So you 
> cannot assume there is ssh/telnet/serial available.

I really thought we could rely on, at least, a serial connection. If 
not, then indeed the current kvm autotest approach is not compatible 
with that test environment. That is not to say that kvm autotest 
couldn't incorporate the qemu-tests approach/functionality.

BTW, I just don't like the idea of having lots of functionalities/tests 
implemented on two test suites for a single piece of software, unless 
proven that there's no way around it. To me, this is the whole point of 
this discussion.

>
>>
>> So, I see no reason for not using a more expressive language,
>
> I seriously doubt you can build a useful initramfs that contains 
> python without doing something crazy like livecd-tools does....

You're right. Again, I was thinking we could rely at least on a serial 
connection. Can we not?

>
>>>> Actually, kvm-autotest has various layers of abstraction in how QEMU
>>>> ends up being launched. As you mention below, those layers are 
>>>> there to
>>>> allow for things like using libvirt.
>>
>> Indeed the qemu command line parameters gets generated depending on many
>> configuration parameters. It'd be *really* simple to add a configuration
>> parameters that overwrites the qemu command with an static one.
>
> But if you're a QEMU developer, you want to have as much control of 
> the command line as possible.  For instance, one of the tests in 
> qemu-test makes sure to test invocations without -device as this 
> triggers a different code path (there was a recent regression in this 
> too).  You can't just add arguments to reproduce this behavior.
>
>>>
>>> It goes beyond that, since it also related to the monitor interface 
>>> as well.
>>>
>>>> That's desirable when you're doing "virt testing", but not so 
>>>> desirably
>>>> when you're trying to write specific unit tests against QEMU.
>>>
>>> True, one may not need it at all but it's nice that a test for
>>> migration/stress/hotplug will be tested directly w/ qemu and libvirt 
>>> w/ the
>>> same effort.
>>>
>>>>
>>>>>> 5) The tests execute very quickly, can be run stand alone, and do 
>>>>>> not
>>>>>> require root privileges
>>>>>
>>>>> ditto for kvm auotest. It's possible to configure it w/o root too
>>>>> which is not a
>>>>> huge issue.
>>>>
>>>> When I say, "run quickly", I mean, they execute very quickly.
>>>
>>> /me too
>>>
>>>>
>>>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
>>>> tests/virtio-serial.sh
>>>>
>>>> real 0m4.385s
>>>> user 0m1.460s
>>>> sys 0m1.860s
>>>
>>> That's impressive but it's more of a function of the guest being 
>>> used - if
>>> instead of running a full Fedora install, you'll chose your busybox 
>>> image w/
>>> -kernel/initrd you'll get a similar result.
>>
>> I also think so. Maybe kvm-autotest would take a little more time 
>> because of the
>> different approach we take when communicating with the guest, but I 
>> bet it'd be
>> irrelevant.
>
> I don't see any reason why everything needs to live in kvm-autotest... 
> but if you really feel that way, please provide patches that 
> demonstrate how this would work. 

If it's technically viable, I think that having it as part of kvm 
autotest, shows that the project is more cohesive, attracts more 
contributions, and makes better use of our efforts.

> We could argue indefinitely about how things could work, it's much 
> better to compare how things actually do work :-)

Sure, that's also what I suggested when I mentioned it'd be a "nice 
exercise for all of us".

>
>>> I agree autotest is not perfect but it likes to be such.
>>> If you wish, you can challenge Lucas and Cleber w/ these type of 
>>> requirements
>>> and we'll all improve as a result.
>>
>> Yes, I believe it'd be a nice exercise for all of us.
>>
>> The only thing I ask is that we bear at least with some of the 
>> complexity that
>> kvm-autotest inherently holds...
>
> I think there's something of a knee jerk reaction here.  The existence 
> of qemu-test does not take anything away from kvm-autotest.  It's just 
> another tool in our arsenal to achieve our joint goal of making QEMU 
> (and KVM) higher quality.

You're right, It does not take anything away from kvm autotest today. 
But suppose we can prove that kvm autotest can indeed absorve all of 
qemu-tests functionalities, it'd be itself a reason for doing so. It'd 
avoid finding ourselves with two evolved test tools that do some of the 
same things, but are separate implementations.

>
> autotest is made to invoke third party tests so the two tools can 
> co-exist in a complimentary way.

That's a no brainier and everyone so far agrees on that. The delicate 
issue is whether qemu-tests functionality could live in kvm autotest in 
a nice way.

>
> Regards,
>
> Anthony Liguori
>
>> the last requests we had was to get rid of all
>> the complexity, while retaining all the other nice characteristics. 
>> Pretty hard,
>> so I think we failed, or maybe only half-succeeded at it.
>>
>>>
>>> Cheers,
>>> Dor
>>>
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>>>>
>>>>>> 6) They are random by nature with the ability to fix the seed in 
>>>>>> order
>>>>>> to be used in git-bisect.
>>>>>>
>>>>>> I think Gerd had been looking at doing something similar with a 
>>>>>> custom
>>>>>> initrd.
>>>>>>
>>>>>> I've tried to consider other architectures and had hoped that we 
>>>>>> could
>>>>>> commit the vmlinuz and initramfs into git so that it was easy to 
>>>>>> test
>>>>>> other architectures without having a full build environment.
>>>>>> Unfortunately, busybox doesn't link statically with glibc and I 
>>>>>> can't
>>>>>> see an obvious way to commit binaries while respecting the GPL 
>>>>>> since we
>>>>>> need to pull glibc into the initramfs.
>>>>>>
>>>>>> I know buildroot exists specifically to deal with this but in my
>>>>>> experience, buildroot is very unreliable and extremely heavy weight
>>>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>>>>>> environment.
>>>>>>
>>>>>> Anyway, the code is available at:
>>>>>>
>>>>>> http://git.qemu.org/qemu-test.git
>>>>>>
>>>>>> See the README for instructions on how to use it.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Anthony Liguori
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>
>