From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K4I19-0004OX-HV for qemu-devel@nongnu.org; Thu, 05 Jun 2008 12:08:47 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K4I16-0004LG-31 for qemu-devel@nongnu.org; Thu, 05 Jun 2008 12:08:47 -0400 Received: from [199.232.76.173] (port=36156 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K4I15-0004L2-Ua for qemu-devel@nongnu.org; Thu, 05 Jun 2008 12:08:43 -0400 Received: from mx20.gnu.org ([199.232.41.8]:4330) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1K4HKm-00006Z-5u for qemu-devel@nongnu.org; Thu, 05 Jun 2008 11:25:00 -0400 Received: from il.qumranet.com ([212.179.150.194]) by mx20.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1K4Eo0-0001tm-TJ for qemu-devel@nongnu.org; Thu, 05 Jun 2008 08:43:01 -0400 Message-ID: <4847DF51.7050606@qumranet.com> Date: Thu, 05 Jun 2008 15:42:57 +0300 From: Avi Kivity MIME-Version: 1.0 Subject: Re: [Qemu-devel] Suggestion for testing framework References: <767386.58386.qm@web57006.mail.re3.yahoo.com> <200806032302.09778.paul@codesourcery.com> <5d6222a80806031505t15f18fe7u256514ccbdc9960@mail.gmail.com> <200806032325.02120.paul@codesourcery.com> <18503.48210.408423.146974@mariner.uk.xensource.com> In-Reply-To: <18503.48210.408423.146974@mariner.uk.xensource.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Reply-To: qemu-devel@nongnu.org List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Ian Jackson wrote: > Paul Brook writes ("Re: [Qemu-devel] Suggestion for testing framework"): > >> We were talking about a tester that does periodic long running tests off svn >> trunk, and reports the results. Individual developers are not directly >> involved. >> > > Just reporting the results is all very well, but as others have said > testing is only useful with a very firm commitment to deal with the > results of reports. Few Free Software projects can manage to do this > without some kind of formal and automatic linkage of QA passes into > code distribution or propagation. Without that, continuous discipline > is needed - and if it ever slips, the test failures become an > ever-deepening swamp. > > One common approach to this problem used with success in very > different ways by both Debian and Xen and probably many others, is to > maintain parallel branches: one (call it `unstable') is changed > freely, and the other (call it `testing') is updated from unstable > only when the QA criteria (whatever those are) are met. > IMO parallel branches are overkill, especially given the current lack of reviewing and committing. I'd suggest having just two rules: - patch authors make a reasonable effort to ensure no regression in compilation or execution; this does not include running the entire test suite - if a regression is discovered by the test suite, the offending patch is either fixed in a short time or reverted This way progress can still be made, but quality is maintained -- error compiling committee.c: too many arguments to function