Re: [Qemu-devel] Suggestion for testing framework

[Fabrice Bellard <fabrice@bellard.org>]

Balazs Attila-Mihaly (Cd-MaN) wrote:
> Hello all
> 
> It seems that there is agreement that some sort of automated testing
> is "a good thing" ;-). I'll have some free time in the next couple of
> days and plan on throwing something like this together on a spare
> box. I was thinking along the lines:
> 
> - several qemu images (one with Debian, one with Windows XP - I can
> get a free student license for it, etc) - a script does a checkout of
> the trunk, checks if the version number is different from the last
> checkout (to avoid spamming the list :-)) - the script introduces the
> source in each VM, starts the VM and lets the different compilers
> available in the VM (like gcc 3.3, 3.4, mingw) compile the source -
> if the compile fails, it collects the error logs - if the compile
> succeeds, performance and functionality tests are run with the
> resulting binary - the is the most nebulous part for the moment for
> me - if I recall Fabrice said that compiling something inside a VM is
> a good performance test... - results are sumitted to the list - if
> you are ok with that, I wouldn't want to spam the list
> 
> Please comment if you find the testing methodology good and what
> performance and functionality test should the process include...

I want to do such automated testing and performance measurements since a 
long time, but never found the time to do them. My idea was close to 
what you suggest, i.e:

Every day:
- do a checkout
- compile all the versions on i386 and x86_64
- publish the binary packages
- run the linux-user tests with the user emulators
- run some of the disk images with the system emulators.
- run benchmarks (gcc compilation, nbench)
- install and test some commercial OSes (this part cannot be 
redistributed of course)

The results should be published on a web site, not on the mailing list.

Fabrice.