Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori <anthony@codemonkey.ws>]

On 12/27/2011 04:35 PM, Cleber Rosa wrote:
> On 12/26/2011 08:00 PM, Dor Laor wrote:
>> On 12/26/2011 05:12 PM, Anthony Liguori wrote:
>>> Hi Dor,
>>>
>>
>> Merry Christmas Anthony,
>>
>>> On 12/25/2011 09:19 AM, Dor Laor wrote:
>>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>>>>
>>>> Well, I'm still not convinced that a new standalone package should
>>>> handle these
>>>> cases instead of kvm autotest. I'll be happy to integrate the tests to
>>>> kvm
>>>> autotest anyway and the more the merrier but imho it's a duplicate.
>>>
>>> I'm sure kvm autotest could be taught to do exactly what qemu-test is
>>> doing. But why does kvm autotest have to do everything? I doubt there
>>> would be much code reuse.
>>>
>>> I think it's not a bad thing to have multiple test suites when there
>>> isn't considerable overlap.
>
> I think the main goal of qemu-tests (may be implicit) is to be quick and simple.

qemu-test doesn't have a main goal.  My goal is to improve QEMU's quality. 
qemu-test is just a tool to help achieve that goal.

> That is indeed great, but if one thinks that's all we'll ever going to need,
> that thought is pretty naive.

I don't know who "we" is, but I can tell you that qemu-test is exactly what *I* 
need.  Consider that I spent a good portion of every single day testing QEMU 
with either my own or other people's patches, making that job easier and more 
automated is fairly important to me.

I'm sharing it because I suspect that a lot of other developers have a similar need.

> And it may be true that there's room for both test
> suites... or that, as busy developers, we're refusing to deal with the added
> complexity (qemu alone accounts for a lot) and delaying to fix the fixable. I
> believe on the later.
>
> One example: kvm-autotest has a complex configuration file format with a steep
> learning curve, while a test such as qemu-tests/tests/simple-ping.sh would have
> to be tweaked if somehow the kernel detects the first ethernet interface as em1
> (such as recent Fedora systems do). Simple, but not scalable.

I can tell by this comment that you don't actually understand how qemu-test 
works.  Please take a look at it before jumping to conclusions about whether it 
should or shouldn't be part of kvm-autotest.

Hint: qemu-test always uses the same kernel because it builds it as part of the 
test suite.  The behavior of how a nic test will never change unless someone 
explicitly changes the kernel.

>>>>> 1) It builds a custom kernel and initramfs based on busybox. This is
>>>>> fairly important to ensure that we can run tests with no device
>>>>> pre-requisites.
>>>>
>>>> This can be done easily w/ autotest too.
>
> The Python requirement inside the guest is true *if* we want to run regular
> autotest tests inside the guest (see autotest/client/virt/tests/autotest.py) and
> this accounts for very very little of kvm autotest usage. All other tests
> interact with the monitor directly and with the guest via ssh/telnet/serial.

qemu-test does not require any specific hardware to be used in the guest which 
lets it test a wider variety of scenarios in QEMU.  So you cannot assume there 
is ssh/telnet/serial available.

>
> So, I see no reason for not using a more expressive language,

I seriously doubt you can build a useful initramfs that contains python without 
doing something crazy like livecd-tools does....

>>> Actually, kvm-autotest has various layers of abstraction in how QEMU
>>> ends up being launched. As you mention below, those layers are there to
>>> allow for things like using libvirt.
>
> Indeed the qemu command line parameters gets generated depending on many
> configuration parameters. It'd be *really* simple to add a configuration
> parameters that overwrites the qemu command with an static one.

But if you're a QEMU developer, you want to have as much control of the command 
line as possible.  For instance, one of the tests in qemu-test makes sure to 
test invocations without -device as this triggers a different code path (there 
was a recent regression in this too).  You can't just add arguments to reproduce 
this behavior.

>>
>> It goes beyond that, since it also related to the monitor interface as well.
>>
>>> That's desirable when you're doing "virt testing", but not so desirably
>>> when you're trying to write specific unit tests against QEMU.
>>
>> True, one may not need it at all but it's nice that a test for
>> migration/stress/hotplug will be tested directly w/ qemu and libvirt w/ the
>> same effort.
>>
>>>
>>>>> 5) The tests execute very quickly, can be run stand alone, and do not
>>>>> require root privileges
>>>>
>>>> ditto for kvm auotest. It's possible to configure it w/o root too
>>>> which is not a
>>>> huge issue.
>>>
>>> When I say, "run quickly", I mean, they execute very quickly.
>>
>> /me too
>>
>>>
>>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
>>> tests/virtio-serial.sh
>>>
>>> real 0m4.385s
>>> user 0m1.460s
>>> sys 0m1.860s
>>
>> That's impressive but it's more of a function of the guest being used - if
>> instead of running a full Fedora install, you'll chose your busybox image w/
>> -kernel/initrd you'll get a similar result.
>
> I also think so. Maybe kvm-autotest would take a little more time because of the
> different approach we take when communicating with the guest, but I bet it'd be
> irrelevant.

I don't see any reason why everything needs to live in kvm-autotest... but if 
you really feel that way, please provide patches that demonstrate how this would 
work.  We could argue indefinitely about how things could work, it's much better 
to compare how things actually do work :-)

>> I agree autotest is not perfect but it likes to be such.
>> If you wish, you can challenge Lucas and Cleber w/ these type of requirements
>> and we'll all improve as a result.
>
> Yes, I believe it'd be a nice exercise for all of us.
>
> The only thing I ask is that we bear at least with some of the complexity that
> kvm-autotest inherently holds...

I think there's something of a knee jerk reaction here.  The existence of 
qemu-test does not take anything away from kvm-autotest.  It's just another tool 
in our arsenal to achieve our joint goal of making QEMU (and KVM) higher quality.

autotest is made to invoke third party tests so the two tools can co-exist in a 
complimentary way.

Regards,

Anthony Liguori

> the last requests we had was to get rid of all
> the complexity, while retaining all the other nice characteristics. Pretty hard,
> so I think we failed, or maybe only half-succeeded at it.
>
>>
>> Cheers,
>> Dor
>>
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>>
>>>>>
>>>>> 6) They are random by nature with the ability to fix the seed in order
>>>>> to be used in git-bisect.
>>>>>
>>>>> I think Gerd had been looking at doing something similar with a custom
>>>>> initrd.
>>>>>
>>>>> I've tried to consider other architectures and had hoped that we could
>>>>> commit the vmlinuz and initramfs into git so that it was easy to test
>>>>> other architectures without having a full build environment.
>>>>> Unfortunately, busybox doesn't link statically with glibc and I can't
>>>>> see an obvious way to commit binaries while respecting the GPL since we
>>>>> need to pull glibc into the initramfs.
>>>>>
>>>>> I know buildroot exists specifically to deal with this but in my
>>>>> experience, buildroot is very unreliable and extremely heavy weight
>>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>>>>> environment.
>>>>>
>>>>> Anyway, the code is available at:
>>>>>
>>>>> http://git.qemu.org/qemu-test.git
>>>>>
>>>>> See the README for instructions on how to use it.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Anthony Liguori
>>>>>
>>>>
>>>
>>
>>
>
>