[Qemu-devel] What tests should "make check-block" run?

[Qemu-devel] What tests should "make check-block" run?

[Markus Armbruster]

I posted a patch with a stupid mistake last night.  Running the right
tests would've saved me the embarrassment.  Trouble is to run them you
first have to know they exist, and then you have to know the right test
script incantation.  "make check-block" can't do that for you.

I believe we can reasonably expect people patching block stuff to know
about "make check-block" in addition to plain "make check", but asking
them to know everything tests/qemu-iotests/check can do is a bit much.

Here's what it can test, as far as I can tell, judging from
tests/qemu-iotests/{common,group}:

* Image formats raw, bochs, cow, cloop, parallels, qcow, qcow2, qed,
  vdi, vmdk, vpc, vhdx

* Protocols file, rbd, sheepdog, nbd, ssh, nfs

* Test groups aio, auto, backing, blkdbg, img, io, quick, rw, snapshot

* Arbitrary cache modes (usual caveats re cache=none on funny file
  systems apply)

* A few miscellaneous test options

Current "make check-block" runs tests/qemu-iotests-quick.sh, which tests
format qcow2, protocol file, group quick with cache=none.  Runs 55 tests
and takes about 2 minutes for me on a not particularly fast test box
(spinning rust).

Should "make check-block" run more tests?  Which ones?

Should we have a variant of "make check-block" for testing other
(format, protocol) combinations?

Should we have a variant of "make check-block" that tests everything
that's testable without special setup?

How can we make advanced iotest features more discoverable for
developers not already intimately familiar with it?

Re: [Qemu-devel] What tests should "make check-block" run?

[Paolo Bonzini]

Il 21/08/2014 14:27, Markus Armbruster ha scritto:
> Current "make check-block" runs tests/qemu-iotests-quick.sh, which tests
> format qcow2, protocol file, group quick with cache=none.  Runs 55 tests
> and takes about 2 minutes for me on a not particularly fast test box
> (spinning rust).

Perhaps "make check" should run these quick tests, while "make
check-block" should run a bigger testsuite.

The caveat about cache=none applies; perhaps tests/qemu-iotests-quick.sh
could check for that.

Paolo

Re: [Qemu-devel] What tests should "make check-block" run?

[Peter Crosthwaite]

On Thu, Aug 21, 2014 at 10:52 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 21/08/2014 14:27, Markus Armbruster ha scritto:
>> Current "make check-block" runs tests/qemu-iotests-quick.sh, which tests
>> format qcow2, protocol file, group quick with cache=none.  Runs 55 tests
>> and takes about 2 minutes for me on a not particularly fast test box
>> (spinning rust).
>
> Perhaps "make check" should run these quick tests, while "make
> check-block" should run a bigger testsuite.
>

Is there a make check-foo target to test absolutely everything
possible? Sometimes when you have far-reaching changes it might be
good to run an everything test case as an overnighter rather than
having to think hard about which test suites are relevant to you.

Regards,
Peter

> The caveat about cache=none applies; perhaps tests/qemu-iotests-quick.sh
> could check for that.
>
> Paolo
>

Re: [Qemu-devel] What tests should "make check-block" run?

[Fam Zheng]

On Thu, 08/21 23:00, Peter Crosthwaite wrote:
> On Thu, Aug 21, 2014 at 10:52 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > Il 21/08/2014 14:27, Markus Armbruster ha scritto:
> >> Current "make check-block" runs tests/qemu-iotests-quick.sh, which tests
> >> format qcow2, protocol file, group quick with cache=none.  Runs 55 tests
> >> and takes about 2 minutes for me on a not particularly fast test box
> >> (spinning rust).
> >
> > Perhaps "make check" should run these quick tests, while "make
> > check-block" should run a bigger testsuite.
> >
> 
> Is there a make check-foo target to test absolutely everything
> possible? Sometimes when you have far-reaching changes it might be
> good to run an everything test case as an overnighter rather than
> having to think hard about which test suites are relevant to you.
> 

I second that. It will be nice to have a make check-full.

Fam

Re: [Qemu-devel] What tests should "make check-block" run?

[Kevin Wolf]

Am 21.08.2014 um 14:52 hat Paolo Bonzini geschrieben:
> Il 21/08/2014 14:27, Markus Armbruster ha scritto:
> > Current "make check-block" runs tests/qemu-iotests-quick.sh, which tests
> > format qcow2, protocol file, group quick with cache=none.  Runs 55 tests
> > and takes about 2 minutes for me on a not particularly fast test box
> > (spinning rust).
> 
> Perhaps "make check" should run these quick tests, while "make
> check-block" should run a bigger testsuite.

Yes, I think that make sense.

> The caveat about cache=none applies; perhaps tests/qemu-iotests-quick.sh
> could check for that.

We could just use -c writeback now that qemu-iotests has this option.
The reason for -nocache was simply that writethrough is too slow.

Kevin

Re: [Qemu-devel] What tests should "make check-block" run?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

On Thu, Aug 21, 2014 at 02:27:00PM +0200, Markus Armbruster wrote:
> Should "make check-block" run more tests?  Which ones?

Let's add image format specific test cases to "make check-block" (e.g.
VHDX and VMDK).  For these formats we would not run the standard tests,
only the format-specific test cases.

That way "make check-block" stays fast but gets a lot more coverage.

> Should we have a variant of "make check-block" for testing other
> (format, protocol) combinations?

I don't think variants are useful.  If you need control, use ./check.

> Should we have a variant of "make check-block" that tests everything
> that's testable without special setup?
> 
> How can we make advanced iotest features more discoverable for
> developers not already intimately familiar with it?

The limiting factor is test execution speed.  If tests are very fast we
can run more of them without annoying the user.

If anyone wants to optimize test run times, that would be very welcome.

Here is a list of the longest test cases:
026 116s
041 56s
023 43s
015 29s
014 29s

There are some more test cases that take more than 10s which I have not
listed.  Ideally all test cases would take less than 5s.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] What tests should "make check-block" run?

[Peter Maydell]

On 21 August 2014 14:12, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> On Thu, Aug 21, 2014 at 02:27:00PM +0200, Markus Armbruster wrote:
>> Should we have a variant of "make check-block" for testing other
>> (format, protocol) combinations?
>
> I don't think variants are useful.  If you need control, use ./check.

That seems pretty undiscoverable to me. I know about 'make check',
and 'make check-help' tells me about 'make check-block', but how
do I find out about 'check' ? I just had to bounce through the makefile
and a wrapper script to even figure out which directory it lives in,
and there's no help text or usage comments in it...

thanks
-- PMM

Re: [Qemu-devel] What tests should "make check-block" run?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 858 bytes --]

On Thu, Aug 21, 2014 at 02:16:36PM +0100, Peter Maydell wrote:
> On 21 August 2014 14:12, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > On Thu, Aug 21, 2014 at 02:27:00PM +0200, Markus Armbruster wrote:
> >> Should we have a variant of "make check-block" for testing other
> >> (format, protocol) combinations?
> >
> > I don't think variants are useful.  If you need control, use ./check.
> 
> That seems pretty undiscoverable to me. I know about 'make check',
> and 'make check-help' tells me about 'make check-block', but how
> do I find out about 'check' ? I just had to bounce through the makefile
> and a wrapper script to even figure out which directory it lives in,
> and there's no help text or usage comments in it...

http://qemu-project.org/Documentation/QemuIoTests

Any suggestions for making it more discoverable?

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] What tests should "make check-block" run?

[Jeff Cody]

On Thu, Aug 28, 2014 at 12:05:56PM +0100, Stefan Hajnoczi wrote:
> On Thu, Aug 21, 2014 at 02:16:36PM +0100, Peter Maydell wrote:
> > On 21 August 2014 14:12, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > On Thu, Aug 21, 2014 at 02:27:00PM +0200, Markus Armbruster wrote:
> > >> Should we have a variant of "make check-block" for testing other
> > >> (format, protocol) combinations?
> > >
> > > I don't think variants are useful.  If you need control, use ./check.
> > 
> > That seems pretty undiscoverable to me. I know about 'make check',
> > and 'make check-help' tells me about 'make check-block', but how
> > do I find out about 'check' ? I just had to bounce through the makefile
> > and a wrapper script to even figure out which directory it lives in,
> > and there's no help text or usage comments in it...
> 
> http://qemu-project.org/Documentation/QemuIoTests
> 
> Any suggestions for making it more discoverable?
> 

A couple of suggestions:

1.) Add a README file to the 'tests' directory

2.) Make the 'tests' directory more hierarchical.  It is not obvious
that 'qemu-iotests' (and the other tests) are something different than
the giant ball of .c files that is also in the tests directory. I
think it would be ideal to have the 'tests' directory contain 2 files
(Makefile, README), and then sub-directories.

Re: [Qemu-devel] What tests should "make check-block" run?

[Markus Armbruster]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Thu, Aug 21, 2014 at 02:16:36PM +0100, Peter Maydell wrote:
>> On 21 August 2014 14:12, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>> > On Thu, Aug 21, 2014 at 02:27:00PM +0200, Markus Armbruster wrote:
>> >> Should we have a variant of "make check-block" for testing other
>> >> (format, protocol) combinations?
>> >
>> > I don't think variants are useful.  If you need control, use ./check.
>> 
>> That seems pretty undiscoverable to me. I know about 'make check',
>> and 'make check-help' tells me about 'make check-block', but how
>> do I find out about 'check' ? I just had to bounce through the makefile
>> and a wrapper script to even figure out which directory it lives in,
>> and there's no help text or usage comments in it...
>
> http://qemu-project.org/Documentation/QemuIoTests

I'm afraid this needs updating for Max's recent "iotests: Allow
out-of-tree run" series.  Max?

> Any suggestions for making it more discoverable?

How do you discover tests?

Developers familiar with GNU-ish make target conventions will probably
try "make check".  If it works (and it does in QEMU), then that's that.

People who actually read build instructions exist.  For those people,
the instructions should describe how to run tests.  Try 'git-grep "make
check" and weep.

"make check" won't cut it for the block layer, you need "make
check-block" for a quick test, and different, still more involved
incantations for a thorough test.

"make help" could lead people to idiosyncratic targets like check-block.
No go, it's "make check-help".  Nice reminder if you know it already,
but it's not discoverable.

I think we could do a few things:

* Have make check print "More tests are available, run 'make check-help'
  to list them" at the very end.

* Either have a make target for thorough block tests, or have check-help
  point to instructions.

* "make help" would be nice to have.

* Have a file INSTALL describing how to build and install this beast.
  Autoconf comes with one fit for autoconfiscated projects.  QEMU isn't,
  but it's vaguely similar, so autoconf's INSTALL could serve as a
  starting point.

Re: [Qemu-devel] What tests should "make check-block" run?

[Peter Maydell]

On 28 August 2014 16:24, Markus Armbruster <armbru@redhat.com> wrote:
> * Have a file INSTALL describing how to build and install this beast.
>   Autoconf comes with one fit for autoconfiscated projects.  QEMU isn't,
>   but it's vaguely similar, so autoconf's INSTALL could serve as a
>   starting point.

Yes please, and remove the (now pretty elderly and incorrect)
build instructions from qemu-doc.texi (which is an unhelpful
place to put them since you don't have prebuilt docs if you're
trying to figure out how to build...)

-- PMM

Re: [Qemu-devel] What tests should "make check-block" run?

[Max Reitz]

On 28.08.2014 17:24, Markus Armbruster wrote:
> Stefan Hajnoczi <stefanha@gmail.com> writes:
>
>> On Thu, Aug 21, 2014 at 02:16:36PM +0100, Peter Maydell wrote:
>>> On 21 August 2014 14:12, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>> On Thu, Aug 21, 2014 at 02:27:00PM +0200, Markus Armbruster wrote:
>>>>> Should we have a variant of "make check-block" for testing other
>>>>> (format, protocol) combinations?
>>>> I don't think variants are useful.  If you need control, use ./check.
>>> That seems pretty undiscoverable to me. I know about 'make check',
>>> and 'make check-help' tells me about 'make check-block', but how
>>> do I find out about 'check' ? I just had to bounce through the makefile
>>> and a wrapper script to even figure out which directory it lives in,
>>> and there's no help text or usage comments in it...
>> http://qemu-project.org/Documentation/QemuIoTests
> I'm afraid this needs updating for Max's recent "iotests: Allow
> out-of-tree run" series.  Max?

Done. Thanks for notifying me.

Max

Re: [Qemu-devel] What tests should "make check-block" run?

[Markus Armbruster]

Max Reitz <mreitz@redhat.com> writes:

> On 28.08.2014 17:24, Markus Armbruster wrote:
>> Stefan Hajnoczi <stefanha@gmail.com> writes:
>>
>>> On Thu, Aug 21, 2014 at 02:16:36PM +0100, Peter Maydell wrote:
>>>> On 21 August 2014 14:12, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>>> On Thu, Aug 21, 2014 at 02:27:00PM +0200, Markus Armbruster wrote:
>>>>>> Should we have a variant of "make check-block" for testing other
>>>>>> (format, protocol) combinations?
>>>>> I don't think variants are useful.  If you need control, use ./check.
>>>> That seems pretty undiscoverable to me. I know about 'make check',
>>>> and 'make check-help' tells me about 'make check-block', but how
>>>> do I find out about 'check' ? I just had to bounce through the makefile
>>>> and a wrapper script to even figure out which directory it lives in,
>>>> and there's no help text or usage comments in it...
>>> http://qemu-project.org/Documentation/QemuIoTests
>> I'm afraid this needs updating for Max's recent "iotests: Allow
>> out-of-tree run" series.  Max?
>
> Done. Thanks for notifying me.

Looks good, thank you!

Re: [Qemu-devel] What tests should "make check-block" run?

[Fam Zheng]

On Thu, 08/28 12:05, Stefan Hajnoczi wrote:
> On Thu, Aug 21, 2014 at 02:16:36PM +0100, Peter Maydell wrote:
> > On 21 August 2014 14:12, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > On Thu, Aug 21, 2014 at 02:27:00PM +0200, Markus Armbruster wrote:
> > >> Should we have a variant of "make check-block" for testing other
> > >> (format, protocol) combinations?
> > >
> > > I don't think variants are useful.  If you need control, use ./check.
> > 
> > That seems pretty undiscoverable to me. I know about 'make check',
> > and 'make check-help' tells me about 'make check-block', but how
> > do I find out about 'check' ? I just had to bounce through the makefile
> > and a wrapper script to even figure out which directory it lives in,
> > and there's no help text or usage comments in it...
> 
> http://qemu-project.org/Documentation/QemuIoTests
> 
> Any suggestions for making it more discoverable?
> 

Maybe mention it in
http://qemu-project.org/Testing
or
http://qemu-project.org/Contribute/SubmitAPatch
?

Fam