Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Dor Laor <dlaor@redhat.com>]

On 12/26/2011 05:12 PM, Anthony Liguori wrote:
> Hi Dor,
>

Merry Christmas Anthony,

> On 12/25/2011 09:19 AM, Dor Laor wrote:
>> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>>
>> Well, I'm still not convinced that a new standalone package should
>> handle these
>> cases instead of kvm autotest. I'll be happy to integrate the tests to
>> kvm
>> autotest anyway and the more the merrier but imho it's a duplicate.
>
> I'm sure kvm autotest could be taught to do exactly what qemu-test is
> doing. But why does kvm autotest have to do everything? I doubt there
> would be much code reuse.
>
> I think it's not a bad thing to have multiple test suites when there
> isn't considerable overlap.

I agree but in this case, it loos to me that qemu-test is likely to do a 
subset of what kvm autotest do in a potentially less elegant way but due 
to the influence of a key maintainer may gain traction and dilute kvm 
autotest..

>
>>> It has the following characteristics:
>>>
>>> 1) It builds a custom kernel and initramfs based on busybox. This is
>>> fairly important to ensure that we can run tests with no device
>>> pre-requisites.
>>
>> This can be done easily w/ autotest too.
>
> Okay, please demonstrate :-) The procedure to do this with qemu test is:
>
> $ git clone git://git.qemu.org/qemu-test.git
> $ cd qemu-test
> $ git submodule update --init
> $ make
>
> How would one do this with kvm autotest?

git clone git://github.com/autotest/autotest.git
cd autotest/client/tests/kvm
sudo ./get_started.py

One command shorter and the above downloads F15 automatically and runs it.

(from 
https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript)

>>> 2) Tests are scripts that are launched in the initramfs
>>>
>>> 3) The test controls exactly how QEMU is launched which allows easy
>>> testing of various QEMU options
>>
>> afaik kvm autotest do the same. It's true that kvm autotest might look
>> less
>> friendly but its may only be this way since it has lots of options.
>
> Actually, kvm-autotest has various layers of abstraction in how QEMU
> ends up being launched. As you mention below, those layers are there to
> allow for things like using libvirt.

It goes beyond that, since it also related to the monitor interface as well.

> That's desirable when you're doing "virt testing", but not so desirably
> when you're trying to write specific unit tests against QEMU.

True, one may not need it at all but it's nice that a test for 
migration/stress/hotplug will be tested directly w/ qemu and libvirt w/ 
the same effort.

>
>>> 5) The tests execute very quickly, can be run stand alone, and do not
>>> require root privileges
>>
>> ditto for kvm auotest. It's possible to configure it w/o root too
>> which is not a
>> huge issue.
>
> When I say, "run quickly", I mean, they execute very quickly.

/me too

>
> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
> tests/virtio-serial.sh
>
> real 0m4.385s
> user 0m1.460s
> sys 0m1.860s

That's impressive but it's more of a function of the guest being used - 
if instead of running a full Fedora install, you'll chose your busybox 
image w/ -kernel/initrd you'll get a similar result.

>
> I've used kvm-autotest a lot, there is no test in kvm-autotest that is
> even close to completing in 4 seconds start to finish. I don't think
> kvm-autotest can even run without installing a guest first which puts
> puts a simple test more at the 30 minute mark. If we're talking about
> TCG testing, then we're in hours territory.

Autotest today may be too verbose by default and will call
echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way 
and Lucas can easily create a slim mode that does not even need to be root.

>
> If there's a way to run kvm-autotest as a non-privileged user, I've not
> figured it out yet.

Again, there is no magic in a test suite so if required we can wave it 
(maybe we need to do that regardless)

>
> Of course, kvm-autotest can test a lot more things than qemu-test can
> since the tests are guest agnostic. That is the primary architectural
> difference. By writing the tests to one specific guest (busybox initrd),
> they can be extremely fast in execution, but the tests are less general
> purpose.
>
> That makes qemu-test very useful for development testing (such as long
> automated bisect runs) but less useful for acceptance testing.

Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and 
the tests will be slimmer or each test will have such a mode.

>
>> Please compare your own virtio-serial test w/ the autotest version of it:
>> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
>>
>>
>>
>> This single file tests functionality, limits, console, live migration and
>> performance. Of course one can add a very basic 'hello world' sanity
>> test too
>> that will run quickly and will identify basic breakage fast.
>
> Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.

I bet that 
virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will be 
less than 52 lines. It depends what you like to test. as noted, the 
above test fully covers all aspects of virtio-serial. It does not have 
to be this way and I do agree it is nicer to split it to multiple sub files.

>
> There is a lot of value in being able to write simple tests in 50 lines
> of code. In fact, the largest qemu-test at the moment is only ~70 lines
> of code. And besides finding regressions in my own code (which is the
> primary use atm), I've found and fixed a few real bugs in upstream QEMU.

Now imagine what virtio_console.py buys you :)

>
> For a couple hundred lines of bash script, how can it be anything but a
> good thing :-)
>
>> Noways we abstract kvm autotest so libvirt will be optionally tested
>> too w/ the
>> same tests.
>>
>> Again, I'm not advocating that kvm autotest is a solution for anything
>> but for
>> plain guest-host communication, monitor commands, etc it's a really
>> good tool.
>>
>> I agree that kvm autotest may be less friendly for developer users
>> since it
>> carries allot of options for testing a huge matrix. Lucas and Cleber
>> are working
>> these days to add a make kvmautotest target to qemu so one would be
>> able to
>> quickly execute autotest and we can think of additional parameters like
>> sanity-set, migration-set, etc.
>
> We (the QEMU project) need to get more serious about testing. We need to

Like

> (quickly) get to a point where we can mandate that features come with
> test cases. It is extremely hard to implement touch everything features
> with no good way to test all of the different things we support.
>
> kvm autotest cannot fill that role because it's too complicated and too
> many layers removed from QEMU.
>
> virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not
> about to mandate that someone writes 2.2k LOC in a python test framework
> in order to get 1.4k of code merged in QEMU.
>
> But 50 lines of bash seems like a more than reasonable requirement.

Hmm, what's the coverage of these 50 LOC?
Maybe when your maintainer hat is on, it is indeed a huge task to keep 
all of qemu platforms/architectures to compile and run so that's enough 
but when I wear my Red Hat manager role, I wish that it will not only 
compile and boot but will test as much functionality as possible and 
will be easily re-used by R&D, QA, libvirt, while all supported guests 
are _potentially_ exercised, not just busybox.

Since kvm autotest will be there anyway, I wish that developers will 
contribute 50 LOC to autotest (and not 2.2kloc) and hope the ROI will 
justify it.

I agree autotest is not perfect but it likes to be such.
If you wish, you can challenge Lucas and Cleber w/ these type of 
requirements and we'll all improve as a result.

Cheers,
Dor

>
> Regards,
>
> Anthony Liguori
>
>>>
>>> 6) They are random by nature with the ability to fix the seed in order
>>> to be used in git-bisect.
>>>
>>> I think Gerd had been looking at doing something similar with a custom
>>> initrd.
>>>
>>> I've tried to consider other architectures and had hoped that we could
>>> commit the vmlinuz and initramfs into git so that it was easy to test
>>> other architectures without having a full build environment.
>>> Unfortunately, busybox doesn't link statically with glibc and I can't
>>> see an obvious way to commit binaries while respecting the GPL since we
>>> need to pull glibc into the initramfs.
>>>
>>> I know buildroot exists specifically to deal with this but in my
>>> experience, buildroot is very unreliable and extremely heavy weight
>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>>> environment.
>>>
>>> Anyway, the code is available at:
>>>
>>> http://git.qemu.org/qemu-test.git
>>>
>>> See the README for instructions on how to use it.
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>>
>>
>