Re: [Qemu-devel] Improving QMP test coverage

[Markus Armbruster <armbru@redhat.com>]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Thu, Jul 13, 2017 at 05:28:52PM +0200, Markus Armbruster wrote:
>> What can we do to improve QMP testing?  Sadly, I don't have the master
>> plan ready.  I can tell people their new code needs to come with tests,
>> but that won't help much unless subsystem maintainers pick up the habit,
>> too.  There are a few obvious tests to write for old code, such as a
>> generic test of query-like commands without arguments and without side
>> effects, similar to what test-hmp.c does for HMP command info (I hope to
>> get around to that one).  But for much of the old code, we don't even
>> know where the test coverage holes are.
>> 
>> Ideas anyone?
>
> It makes sense for maintainers to ask that new QMP commands come with
> comprehensive tests.
>
> For me the main question is how to begin?  What is the easiest and
> preferred way to write QMP command test cases?
>
> Today the most common approach is a qtest test case that sends commands
> and verifies that the expected response is returned.  This approach
> works but we could benefit from a discussion about the alternatives
> (e.g.  qemu-iotests style shell scripts with output diffing).

Output testing style delegates checking ouput to diff.  I rather like it
when text output is readily available.  It is when testing QMP.  A
non-trivial example using this style could be useful, as discussing
ideas tends to be more productive when they come with patches.

> If it's easy to write tests then developers will contribute more tests.

Yes.  Maintainers can more easily demand them, too.