From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:40387) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5ysZ-0006DY-9U for qemu-devel@nongnu.org; Fri, 09 Mar 2012 07:25:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1S5ysW-0006lN-VR for qemu-devel@nongnu.org; Fri, 09 Mar 2012 07:25:02 -0500 Received: from mail-gx0-f173.google.com ([209.85.161.173]:59045) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5ysW-0006lA-Pw for qemu-devel@nongnu.org; Fri, 09 Mar 2012 07:25:00 -0500 Received: by ggnj2 with SMTP id j2so939858ggn.4 for ; Fri, 09 Mar 2012 04:24:59 -0800 (PST) Message-ID: <4F59F698.5090203@codemonkey.ws> Date: Fri, 09 Mar 2012 06:24:56 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <20120308150722.GA30576@t420s.optimusnet> <4F58CCBA.9000702@codemonkey.ws> <20120308160505.GA32360@t420s.optimusnet> <4F58E67A.3050708@codemonkey.ws> <20120308175907.GA4900@t420s.optimusnet> <4F5905AA.3060304@codemonkey.ws> <20120308210209.GA11998@t420s.optimusnet> <4F59237F.6010406@codemonkey.ws> <20120308222433.GB11998@t420s.optimusnet> <4F593F08.8050606@codemonkey.ws> <20120308235101.GA24883@t420s.optimusnet> <4F59E5FE.60906@redhat.com> <4F59F0BF.9000507@codemonkey.ws> <4F59F3F3.9050200@redhat.com> In-Reply-To: <4F59F3F3.9050200@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Kevin Wolf Cc: Lucas Meneghel Rodrigues , Scott Zawalski , Ademar Reis , QEMU devel , Cleber Rosa On 03/09/2012 06:13 AM, Kevin Wolf wrote: > Am 09.03.2012 12:59, schrieb Anthony Liguori: >> On 03/09/2012 05:14 AM, Kevin Wolf wrote: >>> Am 09.03.2012 00:51, schrieb Ademar Reis: >>>> On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote: >>>>>> Plus it's not unconditional: the test runner will report tests >>>>>> SKIPPED if a dependency is not present. >>>>> >>>>> But then the tests aren't run so if most developers didn't have it >>>>> installed, and most tests were written with it, most developers >>>>> wouldn't be running most tests which defeats the purpose, no? >>>> >>>> Part of a TDD approach is to have build and test bots frequently >>>> running tests on multiple platforms with different >>>> configurations. >>>> >>>> You can't expect developers to run all tests all the time. >>> >>> I think this is one of the most important points: Not all developers >>> must run all the tests all the time. >>> >>> Actually, Anthony agreed with me when I said that developers should run >>> some sanity tests for all of qemu and maybe a few more tests for the >>> subsystems they're touching. >> >> And a small number of randomly chosen test cases. >> >> We don't want to have test cases that never run under normal circumstances or >> else they're prone to break. That's why I've talked a lot about keeping 'make >> check' time bound. > > This sounds horrible. make check results must be reproducible, It is reproducible by fixing the random seed. When you run qemu-test, you get output like: $ ./qemu-test qemu-system-x86_64 tests/device-add.sh Using RANDOM seed 56782 Formatting '.tmp-2398/disk.img', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536 ls: cannot access .tmp-2398/logfile-2398.log: No such file or directory /home/anthony/build/qemu/x86_64-softmmu/qemu-system-x86_64 -kernel bin/vmlinuz-3.0 -initrd .tmp-2398/initramfs-2398.img.gz -append console=ttyS0 seed=56782 -nographic -enable-kvm -device virtio-balloon-pci,id=balloon0 -pidfile .tmp-2398/pidfile-2398.pid -qmp unix:.tmp-2398/qmpsock-2398.sock,server,nowait test: 70: =: unexpected operator [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.0 (anthony@titi) (gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4) ) #7 SMP Mon Dec 19 15:54:15 CST 2011 To run the exact same test again, you would do: $ QEMU_TEST_SEED=56782 ./qemu-test qemu-system-x86_64 tests/devices-add.sh This lets you design high coverage tests without having to mess around with changing configuration files. In the latest qemu-test, you can even fix your random choices if you want to configure a very specific test. > not > depend on a randomly chosen set. If you do it like this, a passed make > check means exactly _nothing_. You can have it both ways. You can fix choices in order to have a deterministic test of a specific thing or you can allow choices to be made randomly (with weighting). So you can make make check validate very precise things but chances are, there's a class of things that you would like people to test but maybe not every time make check runs. Having a contingent of occasionally executed tests means you'll get better coverage for less commonly used features. >>> A test bot would be great, but even if people just >>> run them occasionally by hand, that would already detect bugs that are >>> currently left in the code for months. If maintainers do it before >>> pushing code into master, you'll even catch everything before it goes >>> into master. This is as good as it gets. >> >> We'll integrate make check into buildbot which currently does look at >> submaintainer trees. > > But make check will never be the full thing. And if you want to > integrate make check into automated testing, then choosing a random > subset of tests for each is an even worse idea than it is anyway. make check-full. > I believe the only sane thing to do is to distinguish between quick > sanity tests that are run by make check, and a full test suite that is > not run by every developer, but ideally by some test bots. Yeah, I don't think we're disagreeing at all. Regards, Anthony Liguori > > Kevin