Re: [Qemu-devel] Improving QMP test coverage

[Stefan Hajnoczi <stefanha@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]

On Tue, Jul 18, 2017 at 06:24:19PM +0200, Markus Armbruster wrote:
> Stefan Hajnoczi <stefanha@gmail.com> writes:
> 
> > On Thu, Jul 13, 2017 at 05:28:52PM +0200, Markus Armbruster wrote:
> >> What can we do to improve QMP testing?  Sadly, I don't have the master
> >> plan ready.  I can tell people their new code needs to come with tests,
> >> but that won't help much unless subsystem maintainers pick up the habit,
> >> too.  There are a few obvious tests to write for old code, such as a
> >> generic test of query-like commands without arguments and without side
> >> effects, similar to what test-hmp.c does for HMP command info (I hope to
> >> get around to that one).  But for much of the old code, we don't even
> >> know where the test coverage holes are.
> >> 
> >> Ideas anyone?
> >
> > It makes sense for maintainers to ask that new QMP commands come with
> > comprehensive tests.
> >
> > For me the main question is how to begin?  What is the easiest and
> > preferred way to write QMP command test cases?
> >
> > Today the most common approach is a qtest test case that sends commands
> > and verifies that the expected response is returned.  This approach
> > works but we could benefit from a discussion about the alternatives
> > (e.g.  qemu-iotests style shell scripts with output diffing).
> 
> Output testing style delegates checking ouput to diff.  I rather like it
> when text output is readily available.  It is when testing QMP.  A
> non-trivial example using this style could be useful, as discussing
> ideas tends to be more productive when they come with patches.

Yes, I was considering how many of the Python iotests could be rewritten
comfortably in shell.  It is nice when the test simply executes commands
and the output file shows the entire history of what happened.  Great
for debugging.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]