From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:50295) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RkJKO-0007XI-Nr for qemu-devel@nongnu.org; Mon, 09 Jan 2012 12:48:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RkJKM-0003jD-BX for qemu-devel@nongnu.org; Mon, 09 Jan 2012 12:48:12 -0500 Received: from relay1.mentorg.com ([192.94.38.131]:33001) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RkJKM-0003iy-66 for qemu-devel@nongnu.org; Mon, 09 Jan 2012 12:48:10 -0500 From: Paul Brook Date: Mon, 9 Jan 2012 17:47:51 +0000 References: <1326124572-8312-1-git-send-email-aliguori@us.ibm.com> <4F0B1016.2010600@us.ibm.com> In-Reply-To: <4F0B1016.2010600@us.ibm.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201201091747.52115.paul@codesourcery.com> Subject: Re: [Qemu-devel] Please read: make check framework List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: Mark McLoughlin , "Michael S. Tsirkin" , qemu-devel@nongnu.org, Alexander Graf , Blue Swirl , Max Filippov , Gerd Hoffmann , "Edgar E. Iglesias" , Peter Maydell , Markus Armbruster , Andreas =?iso-8859-1?q?F=E4rber?= , =?iso-8859-1?q?Herv=E9_Poussineau?= , Avi Kivity , Stefan Hajnoczi , Stefano Stabellini , Stefan Weil , Riku Voipio , Jan Kiszka , Daniel Gollub , Luiz Capitulino , "Venkateswararao Jujjuri (JV)" , Richard Henderson , Kevin Wolf , "Vassili Karpov (malc)" , Marcelo Tosatti , Michael Walle , Amit Shah , Paolo Bonzini , Aurelien Jarno > I'm going to apply this series quickly and will start running 'make > check-quick' as part a sniff test before pushing patches. > > I'd like to request that all maintainers/submaintainers do the same and > that everyone contributes unit tests to this target. > > The general rules for 'make check-quick': > > 1) It must complete in less than 10 minutes start to finish (the entire > rule). We can re-examine this over time but for now, it seems like a > reasonable limit. No objection in principle, though I'm a bit unclear on the guidelines. In particular: - 10 minutes on what hardware? 10 minutes on one of my fat build machines is an hour on an average year-old laptop/desktop, and I guess 6+ hours on the netbook I use when travelling. Maybe relating this to the time taken to do a clean build would make more sense? - What level of testing is appropriate? As a maintainer when can/should I bounce a patch due to lack of tests? e.g should new device emulation come with unit tests? New infrastructure? What about fixes to both of the above, should these include regression tests? Given the size of the test surface for many components (in particular emulated CPUs), I'm guessing we're looking at extremely basic smoke-tests. Consistent regression tests or any sort of architecture conformance tests are going to completely blow your time budget. Obviously level of testing is always a bit of a judgement call - anyone who claims of have complete test coverage is either lying or writing trivially uninteresting code. However given qemu has historically had zero active test coverage I'd appreciate some guidance (as both maintainers and contrubutor). Paul