From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K3qMc-00078o-F3 for qemu-devel@nongnu.org; Wed, 04 Jun 2008 06:37:06 -0400 Received: from [199.232.76.173] (port=46910 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K3qMb-000762-58 for qemu-devel@nongnu.org; Wed, 04 Jun 2008 06:37:05 -0400 Received: from mx1.polytechnique.org ([129.104.30.34]:39199) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1K3qMa-0004in-R5 for qemu-devel@nongnu.org; Wed, 04 Jun 2008 06:37:05 -0400 Received: from fbe1.dev.netgem.com (gw.netgem.com [195.68.2.34]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ssl.polytechnique.org (Postfix) with ESMTP id 38A7E33179 for ; Wed, 4 Jun 2008 12:36:37 +0200 (CEST) Message-ID: <48467029.5040707@bellard.org> Date: Wed, 04 Jun 2008 12:36:25 +0200 From: Fabrice Bellard MIME-Version: 1.0 Subject: Re: [Qemu-devel] Suggestion for testing framework References: <767386.58386.qm@web57006.mail.re3.yahoo.com> In-Reply-To: <767386.58386.qm@web57006.mail.re3.yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Reply-To: qemu-devel@nongnu.org List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Balazs Attila-Mihaly (Cd-MaN) wrote: > Hello all > > It seems that there is agreement that some sort of automated testing > is "a good thing" ;-). I'll have some free time in the next couple of > days and plan on throwing something like this together on a spare > box. I was thinking along the lines: > > - several qemu images (one with Debian, one with Windows XP - I can > get a free student license for it, etc) - a script does a checkout of > the trunk, checks if the version number is different from the last > checkout (to avoid spamming the list :-)) - the script introduces the > source in each VM, starts the VM and lets the different compilers > available in the VM (like gcc 3.3, 3.4, mingw) compile the source - > if the compile fails, it collects the error logs - if the compile > succeeds, performance and functionality tests are run with the > resulting binary - the is the most nebulous part for the moment for > me - if I recall Fabrice said that compiling something inside a VM is > a good performance test... - results are sumitted to the list - if > you are ok with that, I wouldn't want to spam the list > > Please comment if you find the testing methodology good and what > performance and functionality test should the process include... I want to do such automated testing and performance measurements since a long time, but never found the time to do them. My idea was close to what you suggest, i.e: Every day: - do a checkout - compile all the versions on i386 and x86_64 - publish the binary packages - run the linux-user tests with the user emulators - run some of the disk images with the system emulators. - run benchmarks (gcc compilation, nbench) - install and test some commercial OSes (this part cannot be redistributed of course) The results should be published on a web site, not on the mailing list. Fabrice.