Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori <anthony@codemonkey.ws>]

On 12/29/2011 10:36 AM, Avi Kivity wrote:
> On 12/29/2011 06:12 PM, Anthony Liguori wrote:
>>>> That's why I've also proposed qtest.  But having written quite a few
>>>> qtest unit tests by now, you hit the limits of this type of testing
>>>> pretty quickly.
>>>
>>> Can you describe those limits?
>>
>>
>> I started writing a finger print test.  While it's easy to do PCI
>> enumeration, it gets pretty nasty if you want to actually access BARs
>> (to read virtio registers)
>
> Why is that? it should be pretty easy to poke values into the BAR registers.

You have to map them which means you need to figure out what the memory layout 
looks like.  Since there's no BIOS, you need to poke fw_cfg to figure all of 
this out and then program the BARs appropriately.

It's definitely non-trivial.

> Something else we can do is access the BARs directly.  We have APIs to
> poke values into mmio, add an api to poke a value into a device's BAR.
> Now that each BAR is represented by exactly one memory region, known to
> the pci core even if it is not mapped, it should be easy.
>
>> and forget about trying to get SMBIOS information as that involves
>> mucking around with ACPI.
>
> SMBIOS is built by seabios, yes?

I think QEMU actually builds the blob of information and passes the full blob to 
SeaBIOS.  You could read and parse the blob out of fw_cfg I guess but that's 
still pretty sucky.  It's a lot nicer to just poke sysfs.

> So fingerprinting it should be done
> from seabios-test.git.  Let's divide it into two problems:

That seems awkward especially since fingerprinting Just Works with qemu-test 
already (and it was only 50 lines of shell code!).

> - seabios pokes qemu to get information.  We should fingerprint this
> information to make sure different qemu version can interoperate with
> different seabios versions (needed since we migrate the bios along with
> the rest of the guest).  I'm guessing most of this info is from fwcfg?
> We can easily fingerprint it like any other device.
> - we need to make sure seabios generates the same tables when using -M.
> Here, we're not testing a device, rather we're testing guest code, so it
> makes sense to use a guest for this.
>
> However, I don't think it's even necessary.  From a quick read of the
> manual, SMBIOS is just a set of static tables in memory which are picked
> up using a signature.  So all we need to do is boot an empty guest, read
> its memory, and look for the tables.

Doesn't it sound a whole nicer use linux.git to parse this information for us in 
a friendly, easy to consume fashion?

>> OTOH, it was very simple to write with qemu-test:
>>
>> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/finger-print.sh;h=fc715378a3bbae0b458cc419981c2493d98f5c3d;hb=HEAD
>>
>
> Yes; but using Linux limits you to what it exposes (of course Linux
> exposes quite a lot, so that's mostly a non issue; but we'll have
> counterexamples).

Maybe.  And for those counter examples, we can drill down to qtest.  But just 
because we may need to do fancy things, it doesn't mean we shouldn't take the 
easy approach 95% of the time.

>> And I ended up finding some long standing bugs in the process:
>>
>> http://mid.gmane.org/1324305949-20960-2-git-send-email-aliguori@us.ibm.com
>>
>>
>> It's fairly nasty as it would only show up if doing migration from a
>> new QEMU with a new guest to a old QEMU.
>
> Yes, good catch.
>
>>
>> I think it's a good example of the type of test that I'm targeting at
>> qemu-test.  It's really not interesting to run it across multiple
>> distro types.  But doing it with qtest would be prohibitively hard.
>
> I don't see why.  A python library to read pci config should be a couple
> of dozens of lines long.  Then you iterate over all functions and print
> selected registers.

PCI enumeration is easy.  It's mapping the bars and then poking other sources of 
information that's more challenging.

Everything is doable by writing a libOS but I'd rather just use Linux than 
invent a libOS just for testing.

>>>> Please review the qtest series.  I think it offers a pretty good
>>>> approach to writing this style of test.  But as I mentioned, you hit
>>>> the limits pretty quickly.
>>>
>>> I think it's great, it looks like exactly what I wanted, except it's
>>> been delivered on time.
>>
>>
>> Can you take a look at how interrupts are handled?  From what I
>> gather, the only real limitation of this approach is that we won't be
>> able to simulate MSI vectors but I don't think that's a huge problem.
>>
>> I looked at integrating interrupt handling at CPUs itself and that
>> turned out to be fairly invasive.
>
> You mention in the changelog replacing the APIC.  Why can't you do that?

I currently replace the I/O APIC which seems to be limited to 16 IRQs.  I think 
MSI takes a side channel directly to the local APIC, no?

>>
>>> I'd really like to see it integrated quickly
>>> with some flesh around it, then replying -ENOTEST to all patches.  This
>>> will improve qemu's quality a lot more than guest boot/ping tests, which
>>> we do regularly with kvm-autotest anyway.
>>>
>>> Think of how new instruction emulations are always accompanied by new
>>> kvm-unit-tests tests, I often don't even have to ask for them.
>>
>> Yes, this is exactly where I'm heading with all of this.
>
> It looks great.  One thing I'd change is to use the qmp protocol
> (perhaps not the monitor, just the schema/codegen).  Does something
> prohibit this?

Yeah, I thought about using QMP.  But events are posted in QMP which means that 
you never get an explicit ACK.  That may not be a problem but it's something I 
had in mind.

It also seemed to be reasonably complicated without a clear advantage.  qtest 
isn't going to be a supported protocol so we can certainly change it down the 
road if we want to.

Regards,

Anthony Liguori

>