Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Anthony Liguori <anthony@codemonkey.ws>]

FYI, I've started building a VNC based automated tester.  You provide it 
a series of screenshots with masks of data that's likely to be 
semi-random and it'll wait for each screen shot.  It also has the smarts 
to find where the mouse is so that you can move it to an exact location.

I'll have something functional (along with a paper) within a month.

Regards,

Anthony Liguori

Dan Shearer wrote:
> On Sun, Apr 08, 2007 at 09:43:20PM +0100, Natalia Portillo wrote:
>
> Whoops, slightly late reply :-)
>
>   
>> I have a huge list of operating systems (both closed and open source, that
>> works and that doesn't work under QEMU) that can be used to check that QEMU
>> doesn't broke (or even, that it corrects a non-working state).
>>     
>
> Where the problem with QEMU is not something basic (eg doesn't boot) it
> can be useful to use nested virtualisation. Although QEMU in QEMU is of
> course slow because that case hasn't been optimised, if you have a
> single image within which are many other images savevm'd just before
> the error condition occurs, you can run a repeatable testsuite quite
> quickly. And since you can snapshot the outer image, you know that you
> are running these tests in exactly the same way every time. (There are a
> lot of other uses for recursive virtualisation, I'm a fan of it!)
>
>   
>> And I think it is better to check an installed image that to test it only
>> boots or installs (faster at least).
>>     
>
> Agreed. As to installs I had to do something similar for different VM
> systems a while ago and I found it useful to have a frozen image just
> before the typical "detecting hardware" phase of an installation. The
> point of this was not to compare the hardware that was detected, just to
> detect major malfunctions. Hardware probing like this is uniquely
> stressful. It isn't hard to notice if an installer crashes, or the VM
> crashes, or there's lot's of errors. 
>
>   
>> Fabrice and me discussed about taking screenshot per some seconds to compare
>> with known took screenshots and if something fails shutdown the VM and send
>> an email.
>>
>> But that required some macro interface "click at x,y, wait some seconds,
>> press 'k' key", that is not currently under QEMU.
>>     
>
> Why can't you redirect the QEMU monitor to something easily accessible,
> eg a virtual serial port, and then use mouse_move and sendkey monitor
> commands to drive testing, and screendump to save images for comparison?
> Those commands have been in the monitor for years, so is there something
> I'm not understanding here?
>
>   
>> The best solution I think is to get a way to send QEMU the screenshot to
>> file command some times and stop the VM when both are equal, then send the
>> last took screenshot to a mail address so breakability can be checked. (If
>> it is the login window, great, it BOOTS!, if not, it doesn't)
>>
>> What do you think?
>>     
>
> The problem is synchronisation given that QEMU has no guaranteed time
> domain, so you don't know if your target is failing or just being slow
> so you keep on taking screenshots for a ridiculously long time.  You can
> take fewer screenshots if you have some out-of-band signal. For example
> knowing to expect a particular screenshot around the time that a
> particular file is touched or a particular hardware device is
> initialised, or a magic QEMU-specific CPU instruction is executed.
>
>   
>> I have enough spare space to hold the boot images (600Gibibytes free), and a
>> machine that should be able to automatically checkout and compile QEMU
>> (Athlon XP 2000+, 768Mb RAM, 600GiB free, Gentoo Linux).
>>     
>
> Compiling is really important. Even without testing targets, this would
> be a very good community service if you want to do even just this much.
> Compiling a simulator under itself in all supported configurations is an
> important set of tests (compile/run MIPS under IA32, IA32 under PPC32,
> etc especially 32/64 mixes, then the different supported compiler
> versions, and then a couple of different Linux distributions with their
> different choices of library versions and configurations.) This is
> probably 24 hours of hard work. I have seen simulators choke on these
> sorts of tests.
>
>   
>> Just, I have not the knowledge to make the script that boots qemu, says qemu
>> to take the screenshot, compares the took one with the last one, stops qemu,
>> sends the last screenshot by email, compresses all took screenshots, goes to
>> next VM, so on. (Preferibly without X11 running, as the machine is a mostly
>> headless P2P and File server)
>>     
>
> I think this can be simplified. Care to send a copy offlist of what
> you've done and i'll have a look at it?
>
> --
> Dan Shearer
> dan@shearer.org
>
>
>
>