Re: [Qemu-devel] Please read: make check framework

[Anthony Liguori <aliguori@us.ibm.com>]

On 01/09/2012 02:57 PM, Andreas Färber wrote:
> Am 09.01.2012 20:28, schrieb Anthony Liguori:
>> On 01/09/2012 11:47 AM, Paul Brook wrote:
>>> - What level of testing is appropriate? As a maintainer when can/should I
>>> bounce a patch due to lack of tests?
>>
>> Yes.
>
> Note the "when". :)

I read "should".

I think I answered this below...

>
>>> e.g should new device emulation come with
>>> unit tests? New infrastructure? What about fixes to both of the above,
>>> should
>>> these include regression tests?
>>
>> There's a careful balance that we're going to need to work out over
>> time.  We need to balance test coverage with setting reasonable criteria
>> for patch inclusion.
>
> Can we interpret that as new API infrastructure for now? As opposed to
> devices (needing qtest),

I'm working on the qtest integration patches right now actually..  Right now, 
there's no excuse to not write test cases against anything that is 
infrastructure and reasonably stand alone.

I hope to make it easy to write device specific test cases very soon too.

> TCG (not unified), KVM (autotest), build
> infrastructure (manually).
>
> Do you envision also testing scripts? (e.g., simpletrace.py,
> checkpatch.pl, tracetool[.py])

I don't see any of that stuff being in the critical path.


>> We're still short on infrastructure right now so I expect that a lot of
>> stuff will get merged without tests at first.  As we get more test
>> infrastructure like qtest and qemu-test, I expect that to change.
>>
>> In terms of total coverage for any given feature, I think ultimately the
>> goal is to move the responsibility for regression testing from the
>> contributors of new features to the subsystems themselves.
>>
>> IOW, after we get going with our test infrastructure, when we do big
>> sweeping changes like the memory API or QOM, I will have much less
>> sympathy for breakages that are caused by subsystems with an incomplete
>> test suite.
>>
>> So I think the most important thing for maintainers to start thinking
>> about is how they can add the necessary infrastructure for testing their
>> subsystems.
>
> Well, I already tried pointing out that qemu-test in the way previously
> proposed does not help catching the recent regressions for PReP.

Actually, qemu-test did catch it.  The only reason I didn't see it is that I 
didn't add qemu-system-powerpc to my regression script that I'm using to drive 
it.  But simply booting up a Linux guest would have caught it.

> Neither am I a kernel guru to propose my own JeOS build and testing
> framework, nor do I think an abundance of test frameworks merely
> orchestrated by a common driver will do us much good...

It's really not that complicated... it's just building four packages in sequence.

>> From what I understand, qemu-test with building binutils, gcc, linux
> would not fit the stated 10 minutes goal.

The idea would be that qemu-jeos would generate the test images, make check 
could probe for the existence of those images, and run tests if they are there. 
  The same principle could apply to anything else that has an external dependency.

> The intentions here are certainly good, just the process needs some more
> thought - or less absoluteness in announcing it.

There's no magic here.  If you care about your code not breaking, write a unit 
test so that other people can check to see if there changes broke your code.

We can bike shed all day long about how long things should run, or whether the 
wording of the "rules" require us to write tests for checkpatch, but it's 
counter productive.

If there's some piece of QEMU you care about, start writing tests and tie it 
into make check.  It's that simple :-)

> If the infrastructure
> we provide in a first step is good and people benefit from using it, we
> will gain coverage and supporting infrastructure over time. It's
> unlikely to work as a requirement from day one though.

Maintainers are going to have exercise discretion, yes.

But the mandate here (if there even is one), is while you can continue to 
complain when people break your subsystem, if you aren't working to add test 
infrastructure so people can reasonably test your subsystem, then your 
complaints aren't going to receive much sympathy.

Regards,

Anthony Liguori

>
> Regards,
> Andreas
>