From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:54421) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RkP4t-0000UD-AR for qemu-devel@nongnu.org; Mon, 09 Jan 2012 18:56:36 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RkP4r-0006TF-9P for qemu-devel@nongnu.org; Mon, 09 Jan 2012 18:56:35 -0500 Received: from e5.ny.us.ibm.com ([32.97.182.145]:46599) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RkP4r-0006Sx-5Y for qemu-devel@nongnu.org; Mon, 09 Jan 2012 18:56:33 -0500 Received: from /spool/local by e5.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 9 Jan 2012 18:56:27 -0500 Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216]) by d01relay01.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q09NuPPb248838 for ; Mon, 9 Jan 2012 18:56:25 -0500 Received: from d01av02.pok.ibm.com (loopback [127.0.0.1]) by d01av02.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q09NuNmU010964 for ; Mon, 9 Jan 2012 21:56:25 -0200 Message-ID: <4F0B7EA2.50909@us.ibm.com> Date: Mon, 09 Jan 2012 17:56:18 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <1326124572-8312-1-git-send-email-aliguori@us.ibm.com> <4F0B1016.2010600@us.ibm.com> <201201091747.52115.paul@codesourcery.com> <4F0B3FE2.8080909@us.ibm.com> <4F0B54A7.9060504@web.de> <4F0B5F2D.1080904@us.ibm.com> <4F0B76A2.1090408@web.de> In-Reply-To: <4F0B76A2.1090408@web.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Qemu-devel] Please read: make check framework List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: =?UTF-8?B?QW5kcmVhcyBGw6RyYmVy?= Cc: Mark McLoughlin , "Michael S. Tsirkin" , qemu-devel@nongnu.org, Alexander Graf , Blue Swirl , Max Filippov , Gerd Hoffmann , "Edgar E. Iglesias" , Peter Maydell , Markus Armbruster , =?UTF-8?B?SGVydsOpIFBvdXNzaW5lYXU=?= , Avi Kivity , Stefan Hajnoczi , Stefano Stabellini , Stefan Weil , Riku Voipio , Jan Kiszka , Paul Brook , Paolo Bonzini , Daniel Gollub , Luiz Capitulino , "Venkateswararao Jujjuri (JV)" , Richard Henderson , Kevin Wolf , Marcelo Tosatti , Michael Walle , Amit Shah , "Vassili Karpov (malc)" , Aurelien Jarno On 01/09/2012 05:22 PM, Andreas Färber wrote: > Am 09.01.2012 22:42, schrieb Anthony Liguori: >> On 01/09/2012 02:57 PM, Andreas Färber wrote: >>> Am 09.01.2012 20:28, schrieb Anthony Liguori: >>>> We're still short on infrastructure right now so I expect that a lot of >>>> stuff will get merged without tests at first. As we get more test >>>> infrastructure like qtest and qemu-test, I expect that to change. >>>> >>>> In terms of total coverage for any given feature, I think ultimately the >>>> goal is to move the responsibility for regression testing from the >>>> contributors of new features to the subsystems themselves. >>>> >>>> IOW, after we get going with our test infrastructure, when we do big >>>> sweeping changes like the memory API or QOM, I will have much less >>>> sympathy for breakages that are caused by subsystems with an incomplete >>>> test suite. >>>> >>>> So I think the most important thing for maintainers to start thinking >>>> about is how they can add the necessary infrastructure for testing their >>>> subsystems. >>> >>> Well, I already tried pointing out that qemu-test in the way previously >>> proposed does not help catching the recent regressions for PReP. >> >> Actually, qemu-test did catch it. The only reason I didn't see it is >> that I didn't add qemu-system-powerpc to my regression script that I'm >> using to drive it. But simply booting up a Linux guest would have >> caught it. > > Booting up a Linux kernel yes, but not the latest-and-greatest like you > proposed. Meaning, we need to be able to reference different branches or > trees instead of a single submodule (cf. qemu-test thread). I don't understand what you mean. Given the set of submodules currently used, I'm not aware of any target that has issues. You asserted that it may not be the case that a single version of Linux, busybox, uclibc would all work on all targets. But so far, I don't see any evidence to suggest that this isn't, in fact, possible. >> If there's some piece of QEMU you care about, start writing tests and >> tie it into make check. It's that simple :-) > > Not quite. It would be easier if we just set up some storage space with > images like the handful of *-test images on qemu.org. It's not that simple. We cannot just host an image on qemu.org without getting into GPL considerations. Even beyond the GPL, I don't think having opaque images that people don't understand how to recreate if they need to is a good strategy. I've considered trying to do something based on automated installation (using kickstart/preseed). This has limitations though. It takes an extremely long time to generate these having the necessary hooks with preseeds appears to be very difficult. > That way we can > supply working images for all targets, you can supply build scripts to > rebuild them and we can all be done quickly and have fun. That's what qemu-jeos is. > I was just saying, stop bike-shedding about how we're supposed to deny > patches without test cases from tomorrow on, I never said that. > and instead let's get the > test infrastructure in place and see how well we get along with it. > That's even simpler and much more convenient. That's what I'm proposing. Specifically: "I'm going to apply this series quickly and will start running 'make check-quick' as part a sniff test before pushing patches. I'd like to request that all maintainers/submaintainers do the same and that everyone contributes unit tests to this target." > You're building up quite a lot of time pressure on other people lately, > claiming two days for review were long already, wanting to merge QOM and The latest QOM series has been sitting on the list for a few weeks and I'm still waiting to give people time to review. I think it's hard to claim I'm rushing it. > now make check ASAP - despite all inchoate. Yes, I am going to rush through make check but there's nothing new or exciting here. > Me, I've reviewed most of this series in what I believe is a > constructive way; Yes, and I appreciate that :-) > I'd love to build test cases for my code based on > qtest - you've made this series a prerequisite, so getting libqtest for > writing test cases based on it is going to take some more time until we > can expect contributors to contribute test cases. The reason I sent this out before qtest (which is my real goal) is based on past experience. We have had two unit test frameworks in use, libcheck and gtester. I'd wager that most developers haven't used either. The libcheck tests spent a while with a small break too that no one noticed. I don't want to introduce yet another test framework without having a simple-to-consume mechanism for our existing test suites while also unifying what we already have into a single mechanism. > If however you're trying to push the work of writing tons of test cases > for legacy code - be it qtest or qemu-test or a qemu-test alternative - > upfront to submaintainers, like you seemed to with this thread, then > you're on the wrong track IMO. I think you mean 'existing code', not legacy code. I don't consider all of the code we have today as legacy. > KVM on x86_64 has the biggest user base > in terms of people testing and potentially contributing test cases; this > imbalance of manpower is not going to be remedied by any unit test > frameworks we introduce. Actually, I think it will. Why do non-x86 architectures break so often? Because they aren't tested. Why aren't they tested? Because it's extremely difficult. AFAIK, the majority of folks that do test non-x86 architectures do so by running images manually. Most of them use Aurelien's debian images. What I'm trying to get at with make check/qemu-test is a simple way that we can add test cases for less commonly used features that makes it very easy to test these things. That's the only way we'll stop breaking other targets. Don't view this as, "here is extra work you have to do to get your feature in", but rather as, "here is an opportunity to make sure that no one else does something silly and breaks your feature, forcing you to spend your time tracking it down." Regards, Anthony Liguori > > Regards, > Andreas >