Re: [Qemu-devel] Improving QMP test coverage

[Fam Zheng <famz@redhat.com>]

On Thu, 07/27 11:09, Daniel P. Berrange wrote:
> On Thu, Jul 27, 2017 at 05:19:57PM +0800, Fam Zheng wrote:
> > On Thu, 07/27 10:14, Markus Armbruster wrote:
> > > This brings some advantages of "verify output with diff" to tests that
> > > verify with code.  Improvement if it simplifies the verification code.
> > > 
> > > I'd still prefer *no* verification code (by delegating the job to diff)
> > > for tests where I can get away wit it.
> > 
> > Python based iotests can be (re)done in such a way that they print actual logs
> > (interactions with qtest/monitor, stdout/stderr of QEMU, etc) instead of the
> > current dot dot dot summary, then we automatically have diff based verification,
> > no?
> 
> The python test 149 that I wrote does exactly that. There's no reason why
> the others couldn't do the same.

Yes, and IMO it should be the default and recommended way.

> 
> > One thing I feel painful with bash iotests is how harder it is to write
> > complicated test scenarios such as migration, incremental backup, etc.
> 
> Yes, shell is an awful language if you need non-trivial control
> logic or data structures
> 
> > On the other hand the iotests are more difficult to debug when things go wrong
> > because it eats the output which, if done with shell, should be very easy to
> > get.
> 
> Even if the python tests are not doing verify-by-diff, it should be fairly
> easy to wire up an env variable IOTESTS_DEBUG=1 which would force them
> to propagate stdout/err of all commands run (or at least save it to a
> log file somewhere).

Good point. Maybe we should extend the -d option of ./check for more log, if the
above "full output" still isn't enough.

Fam