Re: [Qemu-devel] Please read: make check framework

[Anthony Liguori <aliguori@us.ibm.com>]

On 01/09/2012 11:47 AM, Paul Brook wrote:
>> I'm going to apply this series quickly and will start running 'make
>> check-quick' as part a sniff test before pushing patches.
>>
>> I'd like to request that all maintainers/submaintainers do the same and
>> that everyone contributes unit tests to this target.
>>
>> The general rules for 'make check-quick':
>>
>>    1) It must complete in less than 10 minutes start to finish (the entire
>> rule). We can re-examine this over time but for now, it seems like a
>> reasonable limit.
>
> No objection in principle, though I'm a bit unclear on the guidelines.
> In particular:
>
> - 10 minutes on what hardware?  10 minutes on one of my fat build machines is
> an hour on an average year-old laptop/desktop, and I guess 6+ hours on the
> netbook I use when travelling.  Maybe relating this to the time taken to do a
> clean build would make more sense?

10 minutes on my laptop.  It's a soft rule and is meant to give people an idea 
of what to target.

gtest has a set of functions to use to select run profiles (quick, slow, 
thorough, etc.).  It's just a matter of making sure that you don't add anything 
ridiculous to the quick profile.

We can tweak the profiles over time

>
> - What level of testing is appropriate? As a maintainer when can/should I
> bounce a patch due to lack of tests?

Yes.

> e.g should new device emulation come with
> unit tests? New infrastructure? What about fixes to both of the above, should
> these include regression tests?

There's a careful balance that we're going to need to work out over time.  We 
need to balance test coverage with setting reasonable criteria for patch inclusion.

We're still short on infrastructure right now so I expect that a lot of stuff 
will get merged without tests at first.  As we get more test infrastructure like 
qtest and qemu-test, I expect that to change.

In terms of total coverage for any given feature, I think ultimately the goal is 
to move the responsibility for regression testing from the contributors of new 
features to the subsystems themselves.

IOW, after we get going with our test infrastructure, when we do big sweeping 
changes like the memory API or QOM, I will have much less sympathy for breakages 
that are caused by subsystems with an incomplete test suite.

So I think the most important thing for maintainers to start thinking about is 
how they can add the necessary infrastructure for testing their subsystems.

> Given the size of the test surface for many
> components (in particular emulated CPUs), I'm guessing we're looking at
> extremely basic smoke-tests.  Consistent regression tests or any sort of
> architecture conformance tests are going to completely blow your time budget.

Right, testing is a cost that should be minimized.  Ultimately, the right 
testing investment is something like the amount of hours that you would have 
spent fixing introduced regressions multiplied by the impact of the regression 
appear in the first place.  That's not an easy formula to use for prediction :-)

That probably means that for subsystems that are well isolated and not in 
anyone's critical path, not a lot of test infrastructure is practical.

> Obviously level of testing is always a bit of a judgement call - anyone who
> claims of have complete test coverage is either lying or writing trivially
> uninteresting code.  However given qemu has historically had zero active test
> coverage I'd appreciate some guidance (as both maintainers and contrubutor).

We're going to have to strike a balance.  To start with, I'd recommend enforcing 
at least sniff tests.  We'll get a huge return on investment with a reasonable 
set of sniff tests.  Longer term, we're going to have to just wait and see at 
what point the ROI starts to diminish significantly.

Regards,

Anthony Liguori

> Paul
>