Re: [Qemu-devel][Patch] Suggestion for testing framework

[Stefan Weil <weil@mail.berlios.de>]

[-- Attachment #1.1: Type: text/plain, Size: 3078 bytes --]

Balazs Attila-Mihaly (Cd-MaN) schrieb:
> Hello all
>
> It seems that there is agreement that some sort of automated testing is "a good thing" ;-). I'll have some free time in the next couple of days and plan on throwing something like this together on a spare box. I was thinking along the lines:
>
> - several qemu images (one with Debian, one with Windows XP - I can get a free student license for it, etc)
> - a script does a checkout of the trunk, checks if the version number is different from the last checkout (to avoid spamming the list :-))
> - the script introduces the source in each VM, starts the VM and lets the different compilers available in the VM (like gcc 3.3, 3.4, mingw) compile the source
> - if the compile fails, it collects the error logs
> - if the compile succeeds, performance and functionality tests are run with the resulting binary - the is the most nebulous part for the moment for me - if I recall Fabrice said that compiling something inside a VM is a good performance test...
> - results are sumitted to the list - if you are ok with that, I wouldn't want to spam the list
>
> Please comment if you find the testing methodology good and what performance and functionality test should the process include...
>
> Best regards.

Hi,

using an existing framework for continuous integration tests will 
provide many of the
features needed. Here is an example for an automated mail (which might 
be sent to
the Qemu mailing list). I used CruiseControl, a Java based framework. The
configuration and a helper script are appended to this mail and could be 
added to
Qemu trunk, so everyone can run his/her own continuous integration.
The current configuration is just a simple compile test for all targets.
It can be extended to add more information to the mail, to inform those who
changed the code about success or failure, to run a web interface,
to cross compile, to run tests...

Of course, running the continuous integration on a server with access 
from the
Internet would be better than running it on my private machine, but even my
private machine could send mails to the list when something goes wrong.
I tried to run CI on a virtual server but it did not have enough 
resources to run Java...

Regards
Stefan

Example of mail sent by CruiseControl:

View results here -> http://localhost/cruisecontrol/buildresults/qemu 
trunk x86?log=log20080615200821Lbuild.7 
<http://localhost/cruisecontrol/buildresults/qemu%20trunk%20x86?log=log20080615200821Lbuild.7> 


BUILD COMPLETE -  build.7
Date of build: 	06/15/2008 20:08:21
Time to build: 	3 minute(s) 11 second(s)
Last changed: 	06/15/2008 20:06:39
Last log entry: 	Avoid temporary variable use across basic blocks for udivx

 Unit Tests: (0)
No Tests Run
This project doesn't have any tests

 

 Modifications since last successful build:  (2)
modified 	blueswir1 	/trunk/target-sparc/translate.c 	06/15/2008 
20:06:39 	Avoid temporary variable use across basic blocks for udivx
modified 	blueswir1 	/trunk/linux-user/main.c 	06/15/2008 20:02:48 	Fix 
Sparc32plus & Sparc64 debug output



[-- Attachment #1.2: Type: text/html, Size: 8913 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: cruisecontrol.patch --]
[-- Type: text/x-diff; name="cruisecontrol.patch", Size: 3157 bytes --]

Index: tests/cruisecontrol/config.xml
===================================================================
--- tests/cruisecontrol/config.xml	(Revision 0)
+++ tests/cruisecontrol/config.xml	(Revision 0)
@@ -0,0 +1,71 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+CruiseControl configuration file for continuous integration tests of Qemu.
+-->
+
+<cruisecontrol>
+  <!-- fix next two lines for your installation -->
+  <property name="srcdir" value="/home/stefan/src/qemu-trunk/trunk" />
+  <property name="mailto" value="name@domain.de" />
+
+  <!-- Native compilation with default configuration. -->
+  <project name="qemu trunk x86" buildafterfailed="false">
+    <listeners>
+      <currentbuildstatuslistener file="logs/${project.name}/status.txt"/>
+    </listeners>
+    <bootstrappers>
+      <svnbootstrapper localworkingcopy="${srcdir}" />
+    </bootstrappers>
+    <modificationset quietperiod="0">
+      <svn localworkingcopy="${srcdir}" uselocalrevision="true" />
+    </modificationset>
+    <schedule interval="600">
+      <exec
+        command="tests/cruisecontrol/cc-build.sh"
+        workingdir="${srcdir}"
+        errorstr="build failed"
+      />
+    </schedule>
+    <publishers>
+      <artifactspublisher file="${srcdir}/bin/native/make.log" dest="artifacts/${project.name}" />
+      <htmlemail
+        mailhost="localhost"
+        reportsuccess="always"
+        returnaddress="qemu-devel@nongnu.org"
+        returnname="QEMU Developers CruiseControl"
+        subjectprefix="[Qemu-devel][CruiseControl]"
+        skipusers="true"
+        buildresultsurl="http://localhost/cruisecontrol/buildresults/${project.name}"
+        logdir="logs/${project.name}"
+      >
+        <always address="${mailto}" />
+      </htmlemail>
+
+    </publishers>
+  </project>
+
+  <!-- Native compilation (64 bit) with default configuration. -->
+<!--
+  <project name="qemu trunk amd64" buildafterfailed="false">
+  </project>
+-->
+
+  <!-- Some tests. Might also be added to one of the compilation projects. -->
+<!--
+  <project name="qemu trunk tests" buildafterfailed="false">
+  </project>
+-->
+
+  <!-- Cross compilation (Win32) with default configuration. -->
+<!--
+  <project name="qemu trunk win32" buildafterfailed="false">
+  </project>
+-->
+
+  <!-- Cross compilation (Mips) with default configuration. -->
+  <!-- Not working because of missing tcg support for mips host. -->
+<!--
+  <project name="qemu trunk mips" buildafterfailed="false">
+  </project>
+-->
+</cruisecontrol>
Index: tests/cruisecontrol/cc-build.sh
===================================================================
--- tests/cruisecontrol/cc-build.sh	(Revision 0)
+++ tests/cruisecontrol/cc-build.sh	(Revision 0)
@@ -0,0 +1,19 @@
+#!/bin/sh
+
+# $Id$
+
+# Build script for Continuous Integration with Cruise Control.
+# Used by CI for Qemu.
+
+export LANG=C
+
+bindir=bin/native
+
+mkdir -p $bindir
+cd $bindir
+
+../../configure
+
+(make || echo build failed) 2>&1 | tee make.log
+
+# eof

Eigenschaftsänderungen: tests/cruisecontrol/cc-build.sh
___________________________________________________________________
Name: svn:executable
   + *