Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori <anthony@codemonkey.ws>]

On 03/09/2012 06:13 AM, Kevin Wolf wrote:
> Am 09.03.2012 12:59, schrieb Anthony Liguori:
>> On 03/09/2012 05:14 AM, Kevin Wolf wrote:
>>> Am 09.03.2012 00:51, schrieb Ademar Reis:
>>>> On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>>>>>> Plus it's not unconditional: the test runner will report tests
>>>>>> SKIPPED if a dependency is not present.
>>>>>
>>>>> But then the tests aren't run so if most developers didn't have it
>>>>> installed, and most tests were written with it, most developers
>>>>> wouldn't be running most tests which defeats the purpose, no?
>>>>
>>>> Part of a TDD approach is to have build and test bots frequently
>>>> running tests on multiple platforms with different
>>>> configurations.
>>>>
>>>> You can't expect developers to run all tests all the time.
>>>
>>> I think this is one of the most important points: Not all developers
>>> must run all the tests all the time.
>>>
>>> Actually, Anthony agreed with me when I said that developers should run
>>> some sanity tests for all of qemu and maybe a few more tests for the
>>> subsystems they're touching.
>>
>> And a small number of randomly chosen test cases.
>>
>> We don't want to have test cases that never run under normal circumstances or
>> else they're prone to break.  That's why I've talked a lot about keeping 'make
>> check' time bound.
>
> This sounds horrible. make check results must be reproducible,

It is reproducible by fixing the random seed.  When you run qemu-test, you get 
output like:

$ ./qemu-test qemu-system-x86_64 tests/device-add.sh
Using RANDOM seed 56782
Formatting '.tmp-2398/disk.img', fmt=qcow2 size=10737418240 encryption=off 
cluster_size=65536
ls: cannot access .tmp-2398/logfile-2398.log: No such file or directory
/home/anthony/build/qemu/x86_64-softmmu/qemu-system-x86_64 -kernel 
bin/vmlinuz-3.0 -initrd .tmp-2398/initramfs-2398.img.gz -append console=ttyS0 
seed=56782 -nographic -enable-kvm -device virtio-balloon-pci,id=balloon0 
-pidfile .tmp-2398/pidfile-2398.pid -qmp 
unix:.tmp-2398/qmpsock-2398.sock,server,nowait
test: 70: =: unexpected operator
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.0.0 (anthony@titi) (gcc version 4.5.2 
(Ubuntu/Linaro 4.5.2-8ubuntu4) ) #7 SMP Mon Dec 19 15:54:15 CST 2011

To run the exact same test again, you would do:

$ QEMU_TEST_SEED=56782 ./qemu-test qemu-system-x86_64 tests/devices-add.sh

This lets you design high coverage tests without having to mess around with 
changing configuration files.  In the latest qemu-test, you can even fix your 
random choices if you want to configure a very specific test.

> not
> depend on a randomly chosen set. If you do it like this, a passed make
> check means exactly _nothing_.

You can have it both ways.  You can fix choices in order to have a deterministic 
test of a specific thing or you can allow choices to be made randomly (with 
weighting).

So you can make make check validate very precise things but chances are, there's 
a class of things that you would like people to test but maybe not every time 
make check runs.  Having a contingent of occasionally executed tests means 
you'll get better coverage for less commonly used features.

>>> A test bot would be great, but even if people just
>>> run them occasionally by hand, that would already detect bugs that are
>>> currently left in the code for months. If maintainers do it before
>>> pushing code into master, you'll even catch everything before it goes
>>> into master. This is as good as it gets.
>>
>> We'll integrate make check into buildbot which currently does look at
>> submaintainer trees.
>
> But make check will never be the full thing. And if you want to
> integrate make check into automated testing, then choosing a random
> subset of tests for each is an even worse idea than it is anyway.

make check-full.

> I believe the only sane thing to do is to distinguish between quick
> sanity tests that are run by make check, and a full test suite that is
> not run by every developer, but ideally by some test bots.

Yeah, I don't think we're disagreeing at all.

Regards,

Anthony Liguori

>
> Kevin