Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori <anthony@codemonkey.ws>]

On 12/29/2011 11:22 AM, Peter Maydell wrote:
> On 29 December 2011 16:36, Avi Kivity<avi@redhat.com>  wrote:
>> Yes; but using Linux limits you to what it exposes (of course Linux
>> exposes quite a lot, so that's mostly a non issue; but we'll have
>> counterexamples).
>
> Actually IME Linux is pretty conservative about how it uses devices.
> A lot of the ARM device models have gaping holes in their emulation
> which we've never needed to fix because Linux simply doesn't use
> those features (eg the PL080 DMA controller which doesn't actually
> implement DMA!)
>
> I think for devices what would be particularly useful would be
> if you can write a (simple) test for something at the register
> level, which generates an image which you can run on the real
> hardware as well as on QEMU. Then you can confirm that your test
> case is correct. Otherwise the tests are just going to bake in the
> same assumptions/misconceptions about what the hardware does as
> the QEMU model.

I think that could be useful, but it orthogonal to what we need for Test Driven 
Development.

The problem today is that it is very hard to make broad changes in QEMU simply 
because it's so difficult to know whether you're breaking another target.  Look 
at the challenges around merging the memory API.

We need a simple way that a developer can sanity check that they've not broken 
other targets and all of the various devices that we support.

>
> My guess is that a serious attempt at tests covering all the
> functionality of a device is probably approximately doubling
> the effort required for a device model, incidentally. A
> half-hearted attempt probably doesn't buy you much over
> automating "boot the guest OS and prod its driver".

Right, but "boot the guest OS and prod its driver" is very useful particularly 
if it can cover all of the various targets we support.

I don't think we should focus much on qtest for non-x86 targets.  I mean, if you 
are interested in it for ARM, fantastic, but I don't think we would mandate it.

For x86, it's reasonable to have stricter requirements on testing, particularly 
for new devices.

FWIW, I expect virtio-scsi to be the first guinea pig here...  I believe Stefan 
has already started looking at writing some qtest based tests for virtio-scsi.

Regards,

Anthony Liguori

> -- PMM
>