From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:49986) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RkMz2-0002tj-EM for qemu-devel@nongnu.org; Mon, 09 Jan 2012 16:42:26 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RkMyx-0000cX-60 for qemu-devel@nongnu.org; Mon, 09 Jan 2012 16:42:24 -0500 Received: from e9.ny.us.ibm.com ([32.97.182.139]:58167) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RkMyw-0000bn-Hr for qemu-devel@nongnu.org; Mon, 09 Jan 2012 16:42:18 -0500 Received: from /spool/local by e9.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 9 Jan 2012 16:42:16 -0500 Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216]) by d01relay03.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q09LgDhx308512 for ; Mon, 9 Jan 2012 16:42:13 -0500 Received: from d01av02.pok.ibm.com (loopback [127.0.0.1]) by d01av02.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q09LgB5g023724 for ; Mon, 9 Jan 2012 19:42:13 -0200 Message-ID: <4F0B5F2D.1080904@us.ibm.com> Date: Mon, 09 Jan 2012 15:42:05 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <1326124572-8312-1-git-send-email-aliguori@us.ibm.com> <4F0B1016.2010600@us.ibm.com> <201201091747.52115.paul@codesourcery.com> <4F0B3FE2.8080909@us.ibm.com> <4F0B54A7.9060504@web.de> In-Reply-To: <4F0B54A7.9060504@web.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Qemu-devel] Please read: make check framework List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: =?ISO-8859-1?Q?Andreas_F=E4rber?= Cc: Mark McLoughlin , "Michael S. Tsirkin" , qemu-devel@nongnu.org, Alexander Graf , Blue Swirl , Max Filippov , Gerd Hoffmann , "Edgar E. Iglesias" , Peter Maydell , Markus Armbruster , =?ISO-8859-1?Q?Herv=E9_Poussine?= =?ISO-8859-1?Q?au?= , Avi Kivity , Stefan Hajnoczi , Stefano Stabellini , Stefan Weil , Riku Voipio , Jan Kiszka , Paul Brook , Paolo Bonzini , Daniel Gollub , Luiz Capitulino , "Venkateswararao Jujjuri (JV)" , Richard Henderson , Kevin Wolf , Marcelo Tosatti , Michael Walle , Amit Shah , "Vassili Karpov (malc)" , Aurelien Jarno On 01/09/2012 02:57 PM, Andreas Färber wrote: > Am 09.01.2012 20:28, schrieb Anthony Liguori: >> On 01/09/2012 11:47 AM, Paul Brook wrote: >>> - What level of testing is appropriate? As a maintainer when can/should I >>> bounce a patch due to lack of tests? >> >> Yes. > > Note the "when". :) I read "should". I think I answered this below... > >>> e.g should new device emulation come with >>> unit tests? New infrastructure? What about fixes to both of the above, >>> should >>> these include regression tests? >> >> There's a careful balance that we're going to need to work out over >> time. We need to balance test coverage with setting reasonable criteria >> for patch inclusion. > > Can we interpret that as new API infrastructure for now? As opposed to > devices (needing qtest), I'm working on the qtest integration patches right now actually.. Right now, there's no excuse to not write test cases against anything that is infrastructure and reasonably stand alone. I hope to make it easy to write device specific test cases very soon too. > TCG (not unified), KVM (autotest), build > infrastructure (manually). > > Do you envision also testing scripts? (e.g., simpletrace.py, > checkpatch.pl, tracetool[.py]) I don't see any of that stuff being in the critical path. >> We're still short on infrastructure right now so I expect that a lot of >> stuff will get merged without tests at first. As we get more test >> infrastructure like qtest and qemu-test, I expect that to change. >> >> In terms of total coverage for any given feature, I think ultimately the >> goal is to move the responsibility for regression testing from the >> contributors of new features to the subsystems themselves. >> >> IOW, after we get going with our test infrastructure, when we do big >> sweeping changes like the memory API or QOM, I will have much less >> sympathy for breakages that are caused by subsystems with an incomplete >> test suite. >> >> So I think the most important thing for maintainers to start thinking >> about is how they can add the necessary infrastructure for testing their >> subsystems. > > Well, I already tried pointing out that qemu-test in the way previously > proposed does not help catching the recent regressions for PReP. Actually, qemu-test did catch it. The only reason I didn't see it is that I didn't add qemu-system-powerpc to my regression script that I'm using to drive it. But simply booting up a Linux guest would have caught it. > Neither am I a kernel guru to propose my own JeOS build and testing > framework, nor do I think an abundance of test frameworks merely > orchestrated by a common driver will do us much good... It's really not that complicated... it's just building four packages in sequence. >> From what I understand, qemu-test with building binutils, gcc, linux > would not fit the stated 10 minutes goal. The idea would be that qemu-jeos would generate the test images, make check could probe for the existence of those images, and run tests if they are there. The same principle could apply to anything else that has an external dependency. > The intentions here are certainly good, just the process needs some more > thought - or less absoluteness in announcing it. There's no magic here. If you care about your code not breaking, write a unit test so that other people can check to see if there changes broke your code. We can bike shed all day long about how long things should run, or whether the wording of the "rules" require us to write tests for checkpatch, but it's counter productive. If there's some piece of QEMU you care about, start writing tests and tie it into make check. It's that simple :-) > If the infrastructure > we provide in a first step is good and people benefit from using it, we > will gain coverage and supporting infrastructure over time. It's > unlikely to work as a requirement from day one though. Maintainers are going to have exercise discretion, yes. But the mandate here (if there even is one), is while you can continue to complain when people break your subsystem, if you aren't working to add test infrastructure so people can reasonably test your subsystem, then your complaints aren't going to receive much sympathy. Regards, Anthony Liguori > > Regards, > Andreas >