[Qemu-devel] Improving QMP test coverage

[Qemu-devel] Improving QMP test coverage

[Markus Armbruster]

I've seen enough QMP-related bugs that should've been caught in basic
testing to know we need to become better at testing this stuff.  That
means investing into tests/.

QMP is now almost eight years old, but its test coverage has always been
spotty.  We don't even really know where the holes are.

The tried-and-true approach to get such a problem under control is to
require new code to come with tests, and buckle down to write the
missing tests for old code.  This is more difficult for QMP than for
other things, because QMP is in so many places.  The QMP core is a
proper subsystem[*], but many of the 150+ commands are parts of other
subsystems.

Let's review tests we already have.

Two unit tests have been around since "forever": test-qmp-commands.c
exercises QMP command dispatch, and test-qmp-event.c exercises QMP event
sending.  The former also has a few tests of qapi_free_FOO(), but they
should really be somewhere else.  Perhaps they can even be deleted.

We additionally have quite a few tests that play with a real QEMU's QMP
monitor, using libqtest or qtest.py.

Many of them use QMP to test something else; they are not necessarily
comprehensive tests for the QMP commands they use.  For instance,
numa-test.c uses QMP command query-cpus to check command line options
have the expected effect.  It's not a comprehensive test of query-cpus.
qemu-iotests/030 uses QMP to exercise image streaming.  Uses QMP command
block-stream a lot, but whether it is a comprehensive test for it I
can't say without careful review of both the command and the test.

What do I mean by "comprehensive"?  A comprehensive test exercises the
command systematically, including its failure modes (testing how things
fail is important, because that's where we often screw up).

There's just one test dedicated to QMP: qmp-test.c.  It's quite new, and
doesn't do much, yet.  It mostly checks QMP protocol implementation,
i.e. aspects of QMP that are independent of specific commands.  For
instance, it tests QEMU cleanly rejects a command object with an
"arguments" member that isn't an object.  It also tests query-version.
It can serve as a relatively uncluttered example of how to test QMP
commands.

What can we do to improve QMP testing?  Sadly, I don't have the master
plan ready.  I can tell people their new code needs to come with tests,
but that won't help much unless subsystem maintainers pick up the habit,
too.  There are a few obvious tests to write for old code, such as a
generic test of query-like commands without arguments and without side
effects, similar to what test-hmp.c does for HMP command info (I hope to
get around to that one).  But for much of the old code, we don't even
know where the test coverage holes are.

Ideas anyone?


[*] Or could be, if we repaid technical debt in the monitor.

Re: [Qemu-devel] Improving QMP test coverage

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

On Thu, Jul 13, 2017 at 05:28:52PM +0200, Markus Armbruster wrote:
> What can we do to improve QMP testing?  Sadly, I don't have the master
> plan ready.  I can tell people their new code needs to come with tests,
> but that won't help much unless subsystem maintainers pick up the habit,
> too.  There are a few obvious tests to write for old code, such as a
> generic test of query-like commands without arguments and without side
> effects, similar to what test-hmp.c does for HMP command info (I hope to
> get around to that one).  But for much of the old code, we don't even
> know where the test coverage holes are.
> 
> Ideas anyone?

It makes sense for maintainers to ask that new QMP commands come with
comprehensive tests.

For me the main question is how to begin?  What is the easiest and
preferred way to write QMP command test cases?

Today the most common approach is a qtest test case that sends commands
and verifies that the expected response is returned.  This approach
works but we could benefit from a discussion about the alternatives
(e.g.  qemu-iotests style shell scripts with output diffing).

If it's easy to write tests then developers will contribute more tests.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Improving QMP test coverage

[Markus Armbruster]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Thu, Jul 13, 2017 at 05:28:52PM +0200, Markus Armbruster wrote:
>> What can we do to improve QMP testing?  Sadly, I don't have the master
>> plan ready.  I can tell people their new code needs to come with tests,
>> but that won't help much unless subsystem maintainers pick up the habit,
>> too.  There are a few obvious tests to write for old code, such as a
>> generic test of query-like commands without arguments and without side
>> effects, similar to what test-hmp.c does for HMP command info (I hope to
>> get around to that one).  But for much of the old code, we don't even
>> know where the test coverage holes are.
>> 
>> Ideas anyone?
>
> It makes sense for maintainers to ask that new QMP commands come with
> comprehensive tests.
>
> For me the main question is how to begin?  What is the easiest and
> preferred way to write QMP command test cases?
>
> Today the most common approach is a qtest test case that sends commands
> and verifies that the expected response is returned.  This approach
> works but we could benefit from a discussion about the alternatives
> (e.g.  qemu-iotests style shell scripts with output diffing).

Output testing style delegates checking ouput to diff.  I rather like it
when text output is readily available.  It is when testing QMP.  A
non-trivial example using this style could be useful, as discussing
ideas tends to be more productive when they come with patches.

> If it's easy to write tests then developers will contribute more tests.

Yes.  Maintainers can more easily demand them, too.

Re: [Qemu-devel] Improving QMP test coverage

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]

On Tue, Jul 18, 2017 at 06:24:19PM +0200, Markus Armbruster wrote:
> Stefan Hajnoczi <stefanha@gmail.com> writes:
> 
> > On Thu, Jul 13, 2017 at 05:28:52PM +0200, Markus Armbruster wrote:
> >> What can we do to improve QMP testing?  Sadly, I don't have the master
> >> plan ready.  I can tell people their new code needs to come with tests,
> >> but that won't help much unless subsystem maintainers pick up the habit,
> >> too.  There are a few obvious tests to write for old code, such as a
> >> generic test of query-like commands without arguments and without side
> >> effects, similar to what test-hmp.c does for HMP command info (I hope to
> >> get around to that one).  But for much of the old code, we don't even
> >> know where the test coverage holes are.
> >> 
> >> Ideas anyone?
> >
> > It makes sense for maintainers to ask that new QMP commands come with
> > comprehensive tests.
> >
> > For me the main question is how to begin?  What is the easiest and
> > preferred way to write QMP command test cases?
> >
> > Today the most common approach is a qtest test case that sends commands
> > and verifies that the expected response is returned.  This approach
> > works but we could benefit from a discussion about the alternatives
> > (e.g.  qemu-iotests style shell scripts with output diffing).
> 
> Output testing style delegates checking ouput to diff.  I rather like it
> when text output is readily available.  It is when testing QMP.  A
> non-trivial example using this style could be useful, as discussing
> ideas tends to be more productive when they come with patches.

Yes, I was considering how many of the Python iotests could be rewritten
comfortably in shell.  It is nice when the test simply executes commands
and the output file shows the entire history of what happened.  Great
for debugging.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Improving QMP test coverage

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 2606 bytes --]


On 07/21/2017 11:33 AM, Stefan Hajnoczi wrote:
>> Output testing style delegates checking ouput to diff.  I rather like it
>> when text output is readily available.  It is when testing QMP.  A
>> non-trivial example using this style could be useful, as discussing
>> ideas tends to be more productive when they come with patches.
> 
> Yes, I was considering how many of the Python iotests could be rewritten
> comfortably in shell.  It is nice when the test simply executes commands
> and the output file shows the entire history of what happened.  Great
> for debugging.
> 
> Stefan
> 
I'd like to have a better understanding of the major pain points here.

Although this can be seen as a matter of taste, style preferences and
even religion, I guess it's safe to say that Python can scale better
than shell.  The upside of shell based tests is the "automatic" and
complete logging, right?  Running "bash -x /path/to/test.sh" will give
much more *useful* information than "python -v /path/to/test.py" will, fact.

I believe this has to do with how *generic* Python code is written, and
how builtin functions and most of the standard Python libraries work as
they do.  Now, when writing code aimed at testing, making use of testing
oriented libraries and tools, one would expect much more useful and
readily available debug information.

I'm biased, for sure, but that's what you get when you write basic tests
using the Avocado libraries.  For instance, when using process.run()[1]
within a test, you can choose to see its command output quite easily
with a command such as "avocado --show=avocado.test.stdout run test.py".

Using other custom logging channels is also trivial (for instance for
specific QMP communication)[2][3].

I wonder if such logging capabilities fill in the gap of what you
describe as "[when the] output file shows the entire history of what
happened".

BTW, I'll defer the discussion of using an external tool to check the
output and determine test success/failure, because it is IMO a
complementary topic, and I believe I understand its use cases.

Regards,
- Cleber.

[1] -
http://avocado-framework.readthedocs.io/en/52.0/api/utils/avocado.utils.html#avocado.utils.process.run
[2] -
http://avocado-framework.readthedocs.io/en/52.0/WritingTests.html#advanced-logging-capabilities
[3] - https://www.youtube.com/watch?v=htUbOsh8MZI

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] Improving QMP test coverage

[Markus Armbruster]

Cleber Rosa <crosa@redhat.com> writes:

> On 07/21/2017 11:33 AM, Stefan Hajnoczi wrote:
>>> Output testing style delegates checking ouput to diff.  I rather like it
>>> when text output is readily available.  It is when testing QMP.  A
>>> non-trivial example using this style could be useful, as discussing
>>> ideas tends to be more productive when they come with patches.
>> 
>> Yes, I was considering how many of the Python iotests could be rewritten
>> comfortably in shell.  It is nice when the test simply executes commands
>> and the output file shows the entire history of what happened.  Great
>> for debugging.
>> 
>> Stefan
>> 
> I'd like to have a better understanding of the major pain points here.
>
> Although this can be seen as a matter of taste, style preferences and
> even religion, I guess it's safe to say that Python can scale better
> than shell.  The upside of shell based tests is the "automatic" and
> complete logging, right?  Running "bash -x /path/to/test.sh" will give
> much more *useful* information than "python -v /path/to/test.py" will, fact.
>
> I believe this has to do with how *generic* Python code is written, and
> how builtin functions and most of the standard Python libraries work as
> they do.  Now, when writing code aimed at testing, making use of testing
> oriented libraries and tools, one would expect much more useful and
> readily available debug information.
>
> I'm biased, for sure, but that's what you get when you write basic tests
> using the Avocado libraries.  For instance, when using process.run()[1]
> within a test, you can choose to see its command output quite easily
> with a command such as "avocado --show=avocado.test.stdout run test.py".
>
> Using other custom logging channels is also trivial (for instance for
> specific QMP communication)[2][3].
>
> I wonder if such logging capabilities fill in the gap of what you
> describe as "[when the] output file shows the entire history of what
> happened".

Test code language is orthogonal to verification method (with code
vs. with diff).  Except verifying with shell code would be obviously
nuts[*].

The existing iotests written in Python verify with code, and the ones
written in shell verify with diff.  Doesn't mean that we have to port
from Python to shell to gain "verify with diff".

I don't doubt that featureful test frameworks like Avocado provide
adequate tools to debug tests.  The lure of the shell is its perceived
simplicity: everybody knows (well, should know) how to write and debug
simple shell scripts.  Of course, the simplicity evaporates when the
scripts grow beyond "simple".  Scare quotes, because what's simple for
Alice may not be so simple for Bob.

> BTW, I'll defer the discussion of using an external tool to check the
> output and determine test success/failure, because it is IMO a
> complementary topic, and I believe I understand its use cases.

Yes.  Regardless, I want to tell you *now* how tired of writing code to
verify test output I am.  Even of reading it.  Output text is easier to
read than code that tries to verify output text.  Diffs are easier to
read than stack backtraces.

> [1] -
> http://avocado-framework.readthedocs.io/en/52.0/api/utils/avocado.utils.html#avocado.utils.process.run
> [2] -
> http://avocado-framework.readthedocs.io/en/52.0/WritingTests.html#advanced-logging-capabilities
> [3] - https://www.youtube.com/watch?v=htUbOsh8MZI

[*] Very many nutty things are more obviously nuts in shell.  It's an
advantage of sorts ;)

Re: [Qemu-devel] Improving QMP test coverage

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 5436 bytes --]


On 07/24/2017 02:56 AM, Markus Armbruster wrote:
> Test code language is orthogonal to verification method (with code
> vs. with diff).  Except verifying with shell code would be obviously
> nuts[*].
> 
> The existing iotests written in Python verify with code, and the ones
> written in shell verify with diff.  Doesn't mean that we have to port
> from Python to shell to gain "verify with diff".
> 

The fact that they are still subject to "verify with diff" is
interesting.  IMO, exit status makes a lot more sense.

I only raised this point, and gave my opinion, because:

1) Shell based tests *with* output check has a lot of use cases and
proven value
2) Python based tests *without* output check also have use cases and value

To me, this shows that language flexibility is a must, and output check
can be an unnecessary burden.  If at all possible, it should be optional.

Implementation-wise, this seems pretty simple.  Tests based on any
language can communicate the basic success/failure by exit status code
and/or by recorded output.  If the ".out" file exists, let's use it, if
not, fallback to exit status (or the other way around, as long as it's
predictable).  For instance, the current implementation of
iotests.main() discards the exit status that could be one of the
PASS/FAIL criteria.

> I don't doubt that featureful test frameworks like Avocado provide
> adequate tools to debug tests.  The lure of the shell is its perceived
> simplicity: everybody knows (well, should know) how to write and debug
> simple shell scripts.  Of course, the simplicity evaporates when the
> scripts grow beyond "simple".  Scare quotes, because what's simple for
> Alice may not be so simple for Bob.
> 

Agreed.  That's why I avoided "simple" and "complex" in my previous
messages.  A statement such as "tool XYZ usually scales better" has
bigger chances of being agreed upon by both Alice and Bob.

>> BTW, I'll defer the discussion of using an external tool to check the
>> output and determine test success/failure, because it is IMO a
>> complementary topic, and I believe I understand its use cases.
> 
> Yes.  Regardless, I want to tell you *now* how tired of writing code to
> verify test output I am.  Even of reading it.  Output text is easier to
> read than code that tries to verify output text.  Diffs are easier to
> read than stack backtraces.
> 

Point taken.  And truth be told: the current shell based qemu-iotests
excel at this, with its simple and effective "run_qemu <<<QMP_INPUT"
pattern.  The cons are:

1) Some use cases do not fall so nicely in this pattern, and end up
requiring workarounds (such as filters)

2) The expected output is not readily accessible

Considering that for the specific case of tests targeting QMP, most of
the action comes down to the message exchanges (commands sent, response
received), how would you feel about a descriptive approach to the
communication?  Something along these lines:

"""
-> { "execute": "change",
     "arguments": { "device": "ide1-cd0",
                    "target": "/srv/images/Fedora-12-x86_64-DVD.iso" } }
<- { "return": {} }
"""

If this looks familiar, that's because it is a snippet from the
qapi-schema.json file.  It's documentation, so it should mean that's
easy to read, *but*, there's no reason this can not be treated as code too.

This approach is similar to Python's doctest[1], in which a docstring
contains snippets of Python code to be evaluated, and the expected
outcome.  Example:

def factorial(n):
    """Return the factorial of n, an exact integer >= 0.

    If the result is small enough to fit in an int, return an int.
    Else return a long.

    >>> [factorial(n) for n in range(6)]
    [1, 1, 2, 6, 24, 120]
    """

The lines that match the interpreter input will be evaluated, and the
subsequent lines are the expected output.

I don't think all QMP tests can be modeled like this, inside literal
strings, but having blocks that can be used when it makes sense seems
logical to me.  Pseudo code:

test_change() {

  init_qemu()
  failUnless(fileExists('/srv/images/Fedora-12-x86_64-DVD.iso'))

  qmp("""
-> { "execute": "change",
     "arguments": { "device": "ide1-cd0",
                    "target": "/srv/images/Fedora-12-x86_64-DVD.iso" } }
<- { "return": {} }
""")

  check_qemu_has_fd_open_for_iso()
}

IMO, the qmp() function can provide similar quality of readability to diff.

How does it sound?  As with previous interactions, if people see value
in this, I can certainly follow up with a PoC.

>> [1] -
>> http://avocado-framework.readthedocs.io/en/52.0/api/utils/avocado.utils.html#avocado.utils.process.run
>> [2] -
>> http://avocado-framework.readthedocs.io/en/52.0/WritingTests.html#advanced-logging-capabilities
>> [3] - https://www.youtube.com/watch?v=htUbOsh8MZI
> 
> [*] Very many nutty things are more obviously nuts in shell.  It's an
> advantage of sorts ;)
> 

We all know a number of amazing software that gets written using the
most improbable tools and languages.  It makes the world a much more
interesting place!

[1] - https://docs.python.org/2/library/doctest.html

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] Improving QMP test coverage

[Markus Armbruster]

Cleber Rosa <crosa@redhat.com> writes:

> On 07/24/2017 02:56 AM, Markus Armbruster wrote:
>> Test code language is orthogonal to verification method (with code
>> vs. with diff).  Except verifying with shell code would be obviously
>> nuts[*].
>> 
>> The existing iotests written in Python verify with code, and the ones
>> written in shell verify with diff.  Doesn't mean that we have to port
>> from Python to shell to gain "verify with diff".
>> 
>
> The fact that they are still subject to "verify with diff" is
> interesting.  IMO, exit status makes a lot more sense.
>
> I only raised this point, and gave my opinion, because:
>
> 1) Shell based tests *with* output check has a lot of use cases and
> proven value
> 2) Python based tests *without* output check also have use cases and value
>
> To me, this shows that language flexibility is a must, and output check
> can be an unnecessary burden.  If at all possible, it should be optional.
>
> Implementation-wise, this seems pretty simple.  Tests based on any
> language can communicate the basic success/failure by exit status code
> and/or by recorded output.  If the ".out" file exists, let's use it, if
> not, fallback to exit status (or the other way around, as long as it's
> predictable).  For instance, the current implementation of
> iotests.main() discards the exit status that could be one of the
> PASS/FAIL criteria.

I think we basically agree.

"Verify output with diff" is great when it fits the test, but it doesn't
fit all tests.

A test program terminating unsuccessfully should be treated as test
failure, regardless of any output checking the test harness may do.

>> I don't doubt that featureful test frameworks like Avocado provide
>> adequate tools to debug tests.  The lure of the shell is its perceived
>> simplicity: everybody knows (well, should know) how to write and debug
>> simple shell scripts.  Of course, the simplicity evaporates when the
>> scripts grow beyond "simple".  Scare quotes, because what's simple for
>> Alice may not be so simple for Bob.
>> 
>
> Agreed.  That's why I avoided "simple" and "complex" in my previous
> messages.  A statement such as "tool XYZ usually scales better" has
> bigger chances of being agreed upon by both Alice and Bob.

Of course, "scales better" is actually better only if you need it to
scale :)

>>> BTW, I'll defer the discussion of using an external tool to check the
>>> output and determine test success/failure, because it is IMO a
>>> complementary topic, and I believe I understand its use cases.
>> 
>> Yes.  Regardless, I want to tell you *now* how tired of writing code to
>> verify test output I am.  Even of reading it.  Output text is easier to
>> read than code that tries to verify output text.  Diffs are easier to
>> read than stack backtraces.
>> 
>
> Point taken.  And truth be told: the current shell based qemu-iotests
> excel at this, with its simple and effective "run_qemu <<<QMP_INPUT"
> pattern.  The cons are:
>
> 1) Some use cases do not fall so nicely in this pattern, and end up
> requiring workarounds (such as filters)

Output filtering is just fine as long as it's reasonably simple, and you
have to mess with it only rarely.  You can almost ignore it in actual
testing work then.

> 2) The expected output is not readily accessible

Expected output would be in git, readily accessible, so that can't be
what you mean.  What do you mean?

> Considering that for the specific case of tests targeting QMP, most of
> the action comes down to the message exchanges (commands sent, response
> received), how would you feel about a descriptive approach to the
> communication?  Something along these lines:
>
> """
> -> { "execute": "change",
>      "arguments": { "device": "ide1-cd0",
>                     "target": "/srv/images/Fedora-12-x86_64-DVD.iso" } }
> <- { "return": {} }
> """
>
> If this looks familiar, that's because it is a snippet from the
> qapi-schema.json file.  It's documentation, so it should mean that's
> easy to read, *but*, there's no reason this can not be treated as code too.
>
> This approach is similar to Python's doctest[1], in which a docstring
> contains snippets of Python code to be evaluated, and the expected
> outcome.  Example:
>
> def factorial(n):
>     """Return the factorial of n, an exact integer >= 0.
>
>     If the result is small enough to fit in an int, return an int.
>     Else return a long.
>
>     >>> [factorial(n) for n in range(6)]
>     [1, 1, 2, 6, 24, 120]
>     """
>
> The lines that match the interpreter input will be evaluated, and the
> subsequent lines are the expected output.

Executable examples in documentation are a really, really good idea,
because they greatly improve the chances the examples reflect reality.

But as you remark below, they can't be the one source of tests, because
(1) it would clutter documentation too much, and (2) it would limit what
tests can do.

An executable example provides QMP input and output, no more.  What if
we need to run QEMU a certain way to make the example work?  We'd have
to add that information to the example.  Might even be an improvement.

More seriously, executable examples make the test case *data*.  Good,
because data is better than code.  Until it isn't.  Sometimes you just
need a loop, a timeout, or want to print a bit of extra information that
isn't QMP output.

> I don't think all QMP tests can be modeled like this, inside literal
> strings, but having blocks that can be used when it makes sense seems
> logical to me.  Pseudo code:
>
> test_change() {
>
>   init_qemu()
>   failUnless(fileExists('/srv/images/Fedora-12-x86_64-DVD.iso'))
>
>   qmp("""
> -> { "execute": "change",
>      "arguments": { "device": "ide1-cd0",
>                     "target": "/srv/images/Fedora-12-x86_64-DVD.iso" } }
> <- { "return": {} }
> """)
>
>   check_qemu_has_fd_open_for_iso()
> }
>
> IMO, the qmp() function can provide similar quality of readability to diff.
>
> How does it sound?  As with previous interactions, if people see value
> in this, I can certainly follow up with a PoC.

This brings some advantages of "verify output with diff" to tests that
verify with code.  Improvement if it simplifies the verification code.

I'd still prefer *no* verification code (by delegating the job to diff)
for tests where I can get away wit it.

>>> [1] -
>>> http://avocado-framework.readthedocs.io/en/52.0/api/utils/avocado.utils.html#avocado.utils.process.run
>>> [2] -
>>> http://avocado-framework.readthedocs.io/en/52.0/WritingTests.html#advanced-logging-capabilities
>>> [3] - https://www.youtube.com/watch?v=htUbOsh8MZI
>> 
>> [*] Very many nutty things are more obviously nuts in shell.  It's an
>> advantage of sorts ;)
>> 
>
> We all know a number of amazing software that gets written using the
> most improbable tools and languages.  It makes the world a much more
> interesting place!
>
> [1] - https://docs.python.org/2/library/doctest.html

Re: [Qemu-devel] Improving QMP test coverage

[Fam Zheng]

On Thu, 07/27 10:14, Markus Armbruster wrote:
> This brings some advantages of "verify output with diff" to tests that
> verify with code.  Improvement if it simplifies the verification code.
> 
> I'd still prefer *no* verification code (by delegating the job to diff)
> for tests where I can get away wit it.

Python based iotests can be (re)done in such a way that they print actual logs
(interactions with qtest/monitor, stdout/stderr of QEMU, etc) instead of the
current dot dot dot summary, then we automatically have diff based verification,
no?

One thing I feel painful with bash iotests is how harder it is to write
complicated test scenarios such as migration, incremental backup, etc.

On the other hand the iotests are more difficult to debug when things go wrong
because it eats the output which, if done with shell, should be very easy to
get.

Fam

Re: [Qemu-devel] Improving QMP test coverage

[Fam Zheng]

On Thu, 07/27 17:19, Fam Zheng wrote:
> On the other hand the iotests are more difficult to debug when things go wrong

s/iotests/python tests/

Re: [Qemu-devel] Improving QMP test coverage

[Daniel P. Berrange]

On Thu, Jul 27, 2017 at 05:19:57PM +0800, Fam Zheng wrote:
> On Thu, 07/27 10:14, Markus Armbruster wrote:
> > This brings some advantages of "verify output with diff" to tests that
> > verify with code.  Improvement if it simplifies the verification code.
> > 
> > I'd still prefer *no* verification code (by delegating the job to diff)
> > for tests where I can get away wit it.
> 
> Python based iotests can be (re)done in such a way that they print actual logs
> (interactions with qtest/monitor, stdout/stderr of QEMU, etc) instead of the
> current dot dot dot summary, then we automatically have diff based verification,
> no?

The python test 149 that I wrote does exactly that. There's no reason why
the others couldn't do the same.

> One thing I feel painful with bash iotests is how harder it is to write
> complicated test scenarios such as migration, incremental backup, etc.

Yes, shell is an awful language if you need non-trivial control
logic or data structures

> On the other hand the iotests are more difficult to debug when things go wrong
> because it eats the output which, if done with shell, should be very easy to
> get.

Even if the python tests are not doing verify-by-diff, it should be fairly
easy to wire up an env variable IOTESTS_DEBUG=1 which would force them
to propagate stdout/err of all commands run (or at least save it to a
log file somewhere).

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Improving QMP test coverage

[Fam Zheng]

On Thu, 07/27 11:09, Daniel P. Berrange wrote:
> On Thu, Jul 27, 2017 at 05:19:57PM +0800, Fam Zheng wrote:
> > On Thu, 07/27 10:14, Markus Armbruster wrote:
> > > This brings some advantages of "verify output with diff" to tests that
> > > verify with code.  Improvement if it simplifies the verification code.
> > > 
> > > I'd still prefer *no* verification code (by delegating the job to diff)
> > > for tests where I can get away wit it.
> > 
> > Python based iotests can be (re)done in such a way that they print actual logs
> > (interactions with qtest/monitor, stdout/stderr of QEMU, etc) instead of the
> > current dot dot dot summary, then we automatically have diff based verification,
> > no?
> 
> The python test 149 that I wrote does exactly that. There's no reason why
> the others couldn't do the same.

Yes, and IMO it should be the default and recommended way.

> 
> > One thing I feel painful with bash iotests is how harder it is to write
> > complicated test scenarios such as migration, incremental backup, etc.
> 
> Yes, shell is an awful language if you need non-trivial control
> logic or data structures
> 
> > On the other hand the iotests are more difficult to debug when things go wrong
> > because it eats the output which, if done with shell, should be very easy to
> > get.
> 
> Even if the python tests are not doing verify-by-diff, it should be fairly
> easy to wire up an env variable IOTESTS_DEBUG=1 which would force them
> to propagate stdout/err of all commands run (or at least save it to a
> log file somewhere).

Good point. Maybe we should extend the -d option of ./check for more log, if the
above "full output" still isn't enough.

Fam

Re: [Qemu-devel] Improving QMP test coverage

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1161 bytes --]

On Thu, Jul 27, 2017 at 07:16:52PM +0800, Fam Zheng wrote:
> On Thu, 07/27 11:09, Daniel P. Berrange wrote:
> > On Thu, Jul 27, 2017 at 05:19:57PM +0800, Fam Zheng wrote:
> > > On Thu, 07/27 10:14, Markus Armbruster wrote:
> > > > This brings some advantages of "verify output with diff" to tests that
> > > > verify with code.  Improvement if it simplifies the verification code.
> > > > 
> > > > I'd still prefer *no* verification code (by delegating the job to diff)
> > > > for tests where I can get away wit it.
> > > 
> > > Python based iotests can be (re)done in such a way that they print actual logs
> > > (interactions with qtest/monitor, stdout/stderr of QEMU, etc) instead of the
> > > current dot dot dot summary, then we automatically have diff based verification,
> > > no?
> > 
> > The python test 149 that I wrote does exactly that. There's no reason why
> > the others couldn't do the same.
> 
> Yes, and IMO it should be the default and recommended way.

I agree.  It's something I embarked on but never finished a few years
ago.  I wanted to modify iotests.py to use logging and drop the unittest
framework.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Improving QMP test coverage

[Daniel P. Berrange]

On Mon, Jul 24, 2017 at 08:56:50AM +0200, Markus Armbruster wrote:
> Cleber Rosa <crosa@redhat.com> writes:
> 
> > On 07/21/2017 11:33 AM, Stefan Hajnoczi wrote:
> >>> Output testing style delegates checking ouput to diff.  I rather like it
> >>> when text output is readily available.  It is when testing QMP.  A
> >>> non-trivial example using this style could be useful, as discussing
> >>> ideas tends to be more productive when they come with patches.
> >> 
> >> Yes, I was considering how many of the Python iotests could be rewritten
> >> comfortably in shell.  It is nice when the test simply executes commands
> >> and the output file shows the entire history of what happened.  Great
> >> for debugging.
> >> 
> >> Stefan
> >> 
> > I'd like to have a better understanding of the major pain points here.
> >
> > Although this can be seen as a matter of taste, style preferences and
> > even religion, I guess it's safe to say that Python can scale better
> > than shell.  The upside of shell based tests is the "automatic" and
> > complete logging, right?  Running "bash -x /path/to/test.sh" will give
> > much more *useful* information than "python -v /path/to/test.py" will, fact.
> >
> > I believe this has to do with how *generic* Python code is written, and
> > how builtin functions and most of the standard Python libraries work as
> > they do.  Now, when writing code aimed at testing, making use of testing
> > oriented libraries and tools, one would expect much more useful and
> > readily available debug information.
> >
> > I'm biased, for sure, but that's what you get when you write basic tests
> > using the Avocado libraries.  For instance, when using process.run()[1]
> > within a test, you can choose to see its command output quite easily
> > with a command such as "avocado --show=avocado.test.stdout run test.py".
> >
> > Using other custom logging channels is also trivial (for instance for
> > specific QMP communication)[2][3].
> >
> > I wonder if such logging capabilities fill in the gap of what you
> > describe as "[when the] output file shows the entire history of what
> > happened".
> 
> Test code language is orthogonal to verification method (with code
> vs. with diff).  Except verifying with shell code would be obviously
> nuts[*].
> 
> The existing iotests written in Python verify with code, and the ones
> written in shell verify with diff.  Doesn't mean that we have to port
> from Python to shell to gain "verify with diff".

Nb, not all the python tests verify with code. The LUKS test 149 that
I wrote in python verifies  with diff. I chose python because shell is
an awful programming language if the code needs conditionals, non-scalar
data structures, or is more than 10 lines long in total :-)

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|