From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1K7xLY-0001Sy-TR for qemu-devel@nongnu.org; Sun, 15 Jun 2008 14:53:00 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1K7xLY-0001Sm-9k for qemu-devel@nongnu.org; Sun, 15 Jun 2008 14:53:00 -0400 Received: from [199.232.76.173] (port=45975 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1K7xLY-0001Sj-6B for qemu-devel@nongnu.org; Sun, 15 Jun 2008 14:53:00 -0400 Received: from moutng.kundenserver.de ([212.227.126.186]:52004) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1K7xLX-0007Hq-Jf for qemu-devel@nongnu.org; Sun, 15 Jun 2008 14:53:00 -0400 Message-ID: <48556507.9070004@mail.berlios.de> Date: Sun, 15 Jun 2008 20:52:55 +0200 From: Stefan Weil MIME-Version: 1.0 Subject: Re: [Qemu-devel][Patch] Suggestion for testing framework References: <767386.58386.qm@web57006.mail.re3.yahoo.com> In-Reply-To: <767386.58386.qm@web57006.mail.re3.yahoo.com> Content-Type: multipart/mixed; boundary="------------030809060500050209000000" Reply-To: qemu-devel@nongnu.org List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: QEMU Developers This is a multi-part message in MIME format. --------------030809060500050209000000 Content-Type: multipart/alternative; boundary="------------080804040003060909070709" --------------080804040003060909070709 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Balazs Attila-Mihaly (Cd-MaN) schrieb: > Hello all > > It seems that there is agreement that some sort of automated testing is "a good thing" ;-). I'll have some free time in the next couple of days and plan on throwing something like this together on a spare box. I was thinking along the lines: > > - several qemu images (one with Debian, one with Windows XP - I can get a free student license for it, etc) > - a script does a checkout of the trunk, checks if the version number is different from the last checkout (to avoid spamming the list :-)) > - the script introduces the source in each VM, starts the VM and lets the different compilers available in the VM (like gcc 3.3, 3.4, mingw) compile the source > - if the compile fails, it collects the error logs > - if the compile succeeds, performance and functionality tests are run with the resulting binary - the is the most nebulous part for the moment for me - if I recall Fabrice said that compiling something inside a VM is a good performance test... > - results are sumitted to the list - if you are ok with that, I wouldn't want to spam the list > > Please comment if you find the testing methodology good and what performance and functionality test should the process include... > > Best regards. Hi, using an existing framework for continuous integration tests will provide many of the features needed. Here is an example for an automated mail (which might be sent to the Qemu mailing list). I used CruiseControl, a Java based framework. The configuration and a helper script are appended to this mail and could be added to Qemu trunk, so everyone can run his/her own continuous integration. The current configuration is just a simple compile test for all targets. It can be extended to add more information to the mail, to inform those who changed the code about success or failure, to run a web interface, to cross compile, to run tests... Of course, running the continuous integration on a server with access from the Internet would be better than running it on my private machine, but even my private machine could send mails to the list when something goes wrong. I tried to run CI on a virtual server but it did not have enough resources to run Java... Regards Stefan Example of mail sent by CruiseControl: View results here -> http://localhost/cruisecontrol/buildresults/qemu trunk x86?log=log20080615200821Lbuild.7 BUILD COMPLETE - build.7 Date of build: 06/15/2008 20:08:21 Time to build: 3 minute(s) 11 second(s) Last changed: 06/15/2008 20:06:39 Last log entry: Avoid temporary variable use across basic blocks for udivx Unit Tests: (0) No Tests Run This project doesn't have any tests Modifications since last successful build: (2) modified blueswir1 /trunk/target-sparc/translate.c 06/15/2008 20:06:39 Avoid temporary variable use across basic blocks for udivx modified blueswir1 /trunk/linux-user/main.c 06/15/2008 20:02:48 Fix Sparc32plus & Sparc64 debug output --------------080804040003060909070709 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit Balazs Attila-Mihaly (Cd-MaN) schrieb: 
Hello all

It seems that there is agreement that some sort of automated testing is "a good thing" ;-). I'll have some free time in the next couple of days and plan on throwing something like this together on a spare box. I was thinking along the lines:

- several qemu images (one with Debian, one with Windows XP - I can get a free student license for it, etc)
- a script does a checkout of the trunk, checks if the version number is different from the last checkout (to avoid spamming the list :-))
- the script introduces the source in each VM, starts the VM and lets the different compilers available in the VM (like gcc 3.3, 3.4, mingw) compile the source
- if the compile fails, it collects the error logs
- if the compile succeeds, performance and functionality tests are run with the resulting binary - the is the most nebulous part for the moment for me - if I recall Fabrice said that compiling something inside a VM is a good performance test...
- results are sumitted to the list - if you are ok with that, I wouldn't want to spam the list

Please comment if you find the testing methodology good and what performance and functionality test should the process include...

Best regards.

Hi,

using an existing framework for continuous integration tests will provide many of the
features needed. Here is an example for an automated mail (which might be sent to
the Qemu mailing list). I used CruiseControl, a Java based framework. The
configuration and a helper script are appended to this mail and could be added to
Qemu trunk, so everyone can run his/her own continuous integration.
The current configuration is just a simple compile test for all targets.
It can be extended to add more information to the mail, to inform those who
changed the code about success or failure, to run a web interface,
to cross compile, to run tests...

Of course, running the continuous integration on a server with access from the
Internet would be better than running it on my private machine, but even my
private machine could send mails to the list when something goes wrong.
I tried to run CI on a virtual server but it did not have enough resources to run Java...

Regards
Stefan

Example of mail sent by CruiseControl:

View results here -> http://localhost/cruisecontrol/buildresults/qemu trunk x86?log=log20080615200821Lbuild.7

BUILD COMPLETE -  build.7
Date of build: 06/15/2008 20:08:21
Time to build: 3 minute(s) 11 second(s)
Last changed: 06/15/2008 20:06:39
Last log entry: Avoid temporary variable use across basic blocks for udivx

 Unit Tests: (0)
No Tests Run
This project doesn't have any tests
 

 Modifications since last successful build:  (2)
modified blueswir1 /trunk/target-sparc/translate.c 06/15/2008 20:06:39 Avoid temporary variable use across basic blocks for udivx
modified blueswir1 /trunk/linux-user/main.c 06/15/2008 20:02:48 Fix Sparc32plus & Sparc64 debug output


--------------080804040003060909070709-- --------------030809060500050209000000 Content-Type: text/x-diff; name="cruisecontrol.patch" Content-Transfer-Encoding: 8bit Content-Disposition: inline; filename="cruisecontrol.patch" Index: tests/cruisecontrol/config.xml =================================================================== --- tests/cruisecontrol/config.xml (Revision 0) +++ tests/cruisecontrol/config.xml (Revision 0) @@ -0,0 +1,71 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Index: tests/cruisecontrol/cc-build.sh =================================================================== --- tests/cruisecontrol/cc-build.sh (Revision 0) +++ tests/cruisecontrol/cc-build.sh (Revision 0) @@ -0,0 +1,19 @@ +#!/bin/sh + +# $Id$ + +# Build script for Continuous Integration with Cruise Control. +# Used by CI for Qemu. + +export LANG=C + +bindir=bin/native + +mkdir -p $bindir +cd $bindir + +../../configure + +(make || echo build failed) 2>&1 | tee make.log + +# eof Eigenschaftsänderungen: tests/cruisecontrol/cc-build.sh ___________________________________________________________________ Name: svn:executable + * --------------030809060500050209000000--