[Qemu-devel] [PATCH v2 0/3] make check: Add qemu-iotests subset

[Qemu-devel] [PATCH v2 0/3] make check: Add qemu-iotests subset

[Kevin Wolf]

Kevin Wolf (3):
  qemu-iotests: Mark some tests as quick
  make check: Add qemu-iotests subset
  Add 'make check-block'

 tests/Makefile              |   11 ++++++++++-
 tests/check-block.sh        |   21 +++++++++++++++++++++
 tests/qemu-iotests-quick.sh |   17 +++++++++++++++++
 tests/qemu-iotests/group    |   24 ++++++++++++------------
 4 files changed, 60 insertions(+), 13 deletions(-)
 create mode 100755 tests/check-block.sh
 create mode 100755 tests/qemu-iotests-quick.sh

-- 
1.7.6.5

[Qemu-devel] [PATCH v2 1/3] qemu-iotests: Mark some tests as quick

[Kevin Wolf]

This creates a new test group 'quick' for some test case that take at
most a couple of seconds each, so that the group can be run during a
quick 'make check'

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/qemu-iotests/group |   24 ++++++++++++------------
 1 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index fcf869d..b549f10 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -8,32 +8,32 @@
 # test-group association ... one line per test
 #
 001 rw auto
-002 rw auto
+002 rw auto quick
 003 rw auto
-004 rw auto
+004 rw auto quick
 005 img auto
 006 img auto
 007 snapshot auto
 008 rw auto
 009 rw auto
 010 rw auto
-011 rw auto
-012 auto
+011 rw auto quick
+012 auto quick
 013 rw auto
 014 rw auto
 015 rw snapshot auto
-016 rw auto
-017 rw backing auto
+016 rw auto quick
+017 rw backing auto quick
 018 rw backing auto
-019 rw backing auto
-020 rw backing auto
+019 rw backing auto quick
+020 rw backing auto quick
 021 io auto
 022 rw snapshot auto
 023 rw auto
-024 rw backing auto
-025 rw auto
+024 rw backing auto quick
+025 rw auto quick
 026 rw blkdbg auto
-027 rw auto
+027 rw auto quick
 028 rw backing auto
-029 rw auto
+029 rw auto quick
 030 rw auto
-- 
1.7.6.5

[Qemu-devel] [PATCH v2 2/3] make check: Add qemu-iotests subset

[Kevin Wolf]

Run the 'quick' group from qemu-iotests during 'make check'.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/Makefile              |    5 +++++
 tests/qemu-iotests-quick.sh |   17 +++++++++++++++++
 2 files changed, 22 insertions(+), 0 deletions(-)
 create mode 100755 tests/qemu-iotests-quick.sh

diff --git a/tests/Makefile b/tests/Makefile
index 74b29dc..571ad42 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -1,6 +1,9 @@
+export SRC_PATH
+
 CHECKS = check-qdict check-qfloat check-qint check-qstring check-qlist
 CHECKS += check-qjson test-qmp-output-visitor test-qmp-input-visitor
 CHECKS += test-string-input-visitor test-string-output-visitor test-coroutine
+CHECKS += $(SRC_PATH)/tests/qemu-iotests-quick.sh
 
 check-qint.o check-qstring.o check-qdict.o check-qlist.o check-qfloat.o check-qjson.o test-coroutine.o: $(GENERATED_HEADERS)
 
@@ -42,6 +45,8 @@ test-qmp-input-visitor: test-qmp-input-visitor.o $(qobject-obj-y) $(qapi-obj-y)
 test-qmp-commands.o: $(addprefix $(qapi-dir)/, test-qapi-types.c test-qapi-types.h test-qapi-visit.c test-qapi-visit.h test-qmp-marshal.c test-qmp-commands.h) $(qapi-obj-y)
 test-qmp-commands: test-qmp-commands.o $(qobject-obj-y) $(qapi-obj-y) $(tools-obj-y) $(qapi-dir)/test-qapi-visit.o $(qapi-dir)/test-qapi-types.o $(qapi-dir)/test-qmp-marshal.o module.o
 
+$(SRC_PATH)/tests/qemu-iotests-quick.sh: qemu-img qemu-io
+
 .PHONY: check
 check: $(CHECKS)
 	$(call quiet-command, gtester $(CHECKS), "  CHECK")
diff --git a/tests/qemu-iotests-quick.sh b/tests/qemu-iotests-quick.sh
new file mode 100755
index 0000000..cf90de0
--- /dev/null
+++ b/tests/qemu-iotests-quick.sh
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+# We don't know which of the system emulator binaries there is (or if there is
+# any at all), so the 'quick' group doesn't contain any tests that require
+# running qemu proper. Assign a fake binary name so that qemu-iotests doesn't
+# complain about the missing binary.
+export QEMU_PROG="this_should_be_unused"
+
+export QEMU_IMG_PROG="$(pwd)/qemu-img"
+export QEMU_IO_PROG="$(pwd)/qemu-io"
+
+cd $SRC_PATH/tests/qemu-iotests
+
+ret=0
+./check -T -nocache -qcow2 -g quick || ret=1
+
+exit $ret
-- 
1.7.6.5

[Qemu-devel] [PATCH v2 3/3] Add 'make check-block'

[Kevin Wolf]

Runs the full qemu-iotests suite for various image formats.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/Makefile       |    6 +++++-
 tests/check-block.sh |   21 +++++++++++++++++++++
 2 files changed, 26 insertions(+), 1 deletions(-)
 create mode 100755 tests/check-block.sh

diff --git a/tests/Makefile b/tests/Makefile
index 571ad42..6b4f4aa 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -47,6 +47,10 @@ test-qmp-commands: test-qmp-commands.o $(qobject-obj-y) $(qapi-obj-y) $(tools-ob
 
 $(SRC_PATH)/tests/qemu-iotests-quick.sh: qemu-img qemu-io
 
-.PHONY: check
+
+.PHONY: check check-block
 check: $(CHECKS)
 	$(call quiet-command, gtester $(CHECKS), "  CHECK")
+
+check-block:
+	$(call quiet-command, $(SRC_PATH)/tests/check-block.sh , "  CHECK")
diff --git a/tests/check-block.sh b/tests/check-block.sh
new file mode 100755
index 0000000..b9d9c6a
--- /dev/null
+++ b/tests/check-block.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+export QEMU_PROG="$(pwd)/x86_64-softmmu/qemu-system-x86_64"
+export QEMU_IMG_PROG="$(pwd)/qemu-img"
+export QEMU_IO_PROG="$(pwd)/qemu-io"
+
+if [ ! -x $QEMU_PROG ]; then
+    echo "'make check-block' requires qemu-system-x86_64"
+    exit 1
+fi
+
+cd $SRC_PATH/tests/qemu-iotests
+
+ret=0
+./check -T -nocache -raw || ret=1
+./check -T -nocache -qcow2 || ret=1
+./check -T -nocache -qed|| ret=1
+./check -T -nocache -vmdk|| ret=1
+./check -T -nocache -vpc || ret=1
+
+exit $ret
-- 
1.7.6.5

Re: [Qemu-devel] [PATCH v2 2/3] make check: Add qemu-iotests subset

[Stefan Hajnoczi]

On Fri, Mar 9, 2012 at 12:46 PM, Kevin Wolf <kwolf@redhat.com> wrote:
> Run the 'quick' group from qemu-iotests during 'make check'.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  tests/Makefile              |    5 +++++
>  tests/qemu-iotests-quick.sh |   17 +++++++++++++++++
>  2 files changed, 22 insertions(+), 0 deletions(-)
>  create mode 100755 tests/qemu-iotests-quick.sh
>
> diff --git a/tests/Makefile b/tests/Makefile
> index 74b29dc..571ad42 100644
> --- a/tests/Makefile
> +++ b/tests/Makefile
> @@ -1,6 +1,9 @@
> +export SRC_PATH

I'm not sure I like this.  Could we avoid this with:

cd $(dirname "$0")/tests/qemu-iotests

instead of using SRC_PATH in qemu-iotests-quick.sh?

Re: [Qemu-devel] [PATCH v2 3/3] Add 'make check-block'

[Andreas Färber]

Am 09.03.2012 13:46, schrieb Kevin Wolf:
> Runs the full qemu-iotests suite for various image formats.
> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  tests/Makefile       |    6 +++++-
>  tests/check-block.sh |   21 +++++++++++++++++++++
>  2 files changed, 26 insertions(+), 1 deletions(-)
>  create mode 100755 tests/check-block.sh
> 
> diff --git a/tests/Makefile b/tests/Makefile
> index 571ad42..6b4f4aa 100644
> --- a/tests/Makefile
> +++ b/tests/Makefile
> @@ -47,6 +47,10 @@ test-qmp-commands: test-qmp-commands.o $(qobject-obj-y) $(qapi-obj-y) $(tools-ob
>  
>  $(SRC_PATH)/tests/qemu-iotests-quick.sh: qemu-img qemu-io
>  
> -.PHONY: check
> +
> +.PHONY: check check-block

Minor nit: Since you already add a second empty line before .PHONY, care
to also add one before check: analog to check-block:?

>  check: $(CHECKS)
>  	$(call quiet-command, gtester $(CHECKS), "  CHECK")
> +
> +check-block:
> +	$(call quiet-command, $(SRC_PATH)/tests/check-block.sh , "  CHECK")

Please use $(SHELL) here.

Andreas

> diff --git a/tests/check-block.sh b/tests/check-block.sh
> new file mode 100755
> index 0000000..b9d9c6a
> --- /dev/null
> +++ b/tests/check-block.sh
> @@ -0,0 +1,21 @@
> +#!/bin/sh
[...]

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH v2 3/3] Add 'make check-block'

[Kevin Wolf]

Am 09.03.2012 14:26, schrieb Andreas Färber:
> Am 09.03.2012 13:46, schrieb Kevin Wolf:
>> Runs the full qemu-iotests suite for various image formats.
>>
>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>> ---
>>  tests/Makefile       |    6 +++++-
>>  tests/check-block.sh |   21 +++++++++++++++++++++
>>  2 files changed, 26 insertions(+), 1 deletions(-)
>>  create mode 100755 tests/check-block.sh
>>
>> diff --git a/tests/Makefile b/tests/Makefile
>> index 571ad42..6b4f4aa 100644
>> --- a/tests/Makefile
>> +++ b/tests/Makefile
>> @@ -47,6 +47,10 @@ test-qmp-commands: test-qmp-commands.o $(qobject-obj-y) $(qapi-obj-y) $(tools-ob
>>  
>>  $(SRC_PATH)/tests/qemu-iotests-quick.sh: qemu-img qemu-io
>>  
>> -.PHONY: check
>> +
>> +.PHONY: check check-block
> 
> Minor nit: Since you already add a second empty line before .PHONY, care
> to also add one before check: analog to check-block:?

Sure.

>>  check: $(CHECKS)
>>  	$(call quiet-command, gtester $(CHECKS), "  CHECK")
>> +
>> +check-block:
>> +	$(call quiet-command, $(SRC_PATH)/tests/check-block.sh , "  CHECK")
> 
> Please use $(SHELL) here.

Ok, I can do that. But what's the difference?

Kevin

Re: [Qemu-devel] [PATCH v2 2/3] make check: Add qemu-iotests subset

[Kevin Wolf]

Am 09.03.2012 14:20, schrieb Stefan Hajnoczi:
> On Fri, Mar 9, 2012 at 12:46 PM, Kevin Wolf <kwolf@redhat.com> wrote:
>> Run the 'quick' group from qemu-iotests during 'make check'.
>>
>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>> ---
>>  tests/Makefile              |    5 +++++
>>  tests/qemu-iotests-quick.sh |   17 +++++++++++++++++
>>  2 files changed, 22 insertions(+), 0 deletions(-)
>>  create mode 100755 tests/qemu-iotests-quick.sh
>>
>> diff --git a/tests/Makefile b/tests/Makefile
>> index 74b29dc..571ad42 100644
>> --- a/tests/Makefile
>> +++ b/tests/Makefile
>> @@ -1,6 +1,9 @@
>> +export SRC_PATH
> 
> I'm not sure I like this.  Could we avoid this with:
> 
> cd $(dirname "$0")/tests/qemu-iotests
> 
> instead of using SRC_PATH in qemu-iotests-quick.sh?

Do you really like this better? To me it doesn't feel cleaner.

Kevin

Re: [Qemu-devel] [PATCH v2 3/3] Add 'make check-block'

[Andreas Färber]

Am 09.03.2012 15:05, schrieb Kevin Wolf:
> Am 09.03.2012 14:26, schrieb Andreas Färber:
>> Am 09.03.2012 13:46, schrieb Kevin Wolf:
>>>  check: $(CHECKS)
>>>  	$(call quiet-command, gtester $(CHECKS), "  CHECK")
>>> +
>>> +check-block:
>>> +	$(call quiet-command, $(SRC_PATH)/tests/check-block.sh , "  CHECK")
>>
>> Please use $(SHELL) here.
> 
> Ok, I can do that. But what's the difference?

The difference is that /bin/sh can be dash or some non-POSIX-compliant
shell. There's an open bug about Solaris 10 needing some oddly named
shell instead of /bin/sh, for instance. By executing ./check-block.sh
via shebang, a shell different from the one the user has explicitly used
for configure gets used. There's an example of $(SHELL) use in ../Makefile.

An alternative would be inlining your script into the Makefile.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH v2 2/3] make check: Add qemu-iotests subset

[Stefan Hajnoczi]

On Fri, Mar 9, 2012 at 2:09 PM, Kevin Wolf <kwolf@redhat.com> wrote:
> Am 09.03.2012 14:20, schrieb Stefan Hajnoczi:
>> On Fri, Mar 9, 2012 at 12:46 PM, Kevin Wolf <kwolf@redhat.com> wrote:
>>> Run the 'quick' group from qemu-iotests during 'make check'.
>>>
>>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>>> ---
>>>  tests/Makefile              |    5 +++++
>>>  tests/qemu-iotests-quick.sh |   17 +++++++++++++++++
>>>  2 files changed, 22 insertions(+), 0 deletions(-)
>>>  create mode 100755 tests/qemu-iotests-quick.sh
>>>
>>> diff --git a/tests/Makefile b/tests/Makefile
>>> index 74b29dc..571ad42 100644
>>> --- a/tests/Makefile
>>> +++ b/tests/Makefile
>>> @@ -1,6 +1,9 @@
>>> +export SRC_PATH
>>
>> I'm not sure I like this.  Could we avoid this with:
>>
>> cd $(dirname "$0")/tests/qemu-iotests
>>
>> instead of using SRC_PATH in qemu-iotests-quick.sh?
>
> Do you really like this better? To me it doesn't feel cleaner.

It makes qemu-iotests-quick.sh self-contained instead of relying on a
magic environment variable.  That's why I like it better.

But it's not critical :).

Stefan

Re: [Qemu-devel] [PATCH v2 2/3] make check: Add qemu-iotests subset

[Anthony Liguori]

On 03/09/2012 06:46 AM, Kevin Wolf wrote:
> Run the 'quick' group from qemu-iotests during 'make check'.
>
> Signed-off-by: Kevin Wolf<kwolf@redhat.com>
> ---
>   tests/Makefile              |    5 +++++
>   tests/qemu-iotests-quick.sh |   17 +++++++++++++++++
>   2 files changed, 22 insertions(+), 0 deletions(-)
>   create mode 100755 tests/qemu-iotests-quick.sh
>
> diff --git a/tests/Makefile b/tests/Makefile
> index 74b29dc..571ad42 100644
> --- a/tests/Makefile
> +++ b/tests/Makefile
> @@ -1,6 +1,9 @@
> +export SRC_PATH
> +
>   CHECKS = check-qdict check-qfloat check-qint check-qstring check-qlist
>   CHECKS += check-qjson test-qmp-output-visitor test-qmp-input-visitor
>   CHECKS += test-string-input-visitor test-string-output-visitor test-coroutine
> +CHECKS += $(SRC_PATH)/tests/qemu-iotests-quick.sh

Not to suggest that we don't start here, but I think we should look into how to 
make qemu-iotest use gtester in the near future.

That will allow the qemu-iotest to be part of the make check-report output and 
will provide an easy way for other tools (like autotest) to run these tests.

Regards,

Anthony Liguori

>
>   check-qint.o check-qstring.o check-qdict.o check-qlist.o check-qfloat.o check-qjson.o test-coroutine.o: $(GENERATED_HEADERS)
>
> @@ -42,6 +45,8 @@ test-qmp-input-visitor: test-qmp-input-visitor.o $(qobject-obj-y) $(qapi-obj-y)
>   test-qmp-commands.o: $(addprefix $(qapi-dir)/, test-qapi-types.c test-qapi-types.h test-qapi-visit.c test-qapi-visit.h test-qmp-marshal.c test-qmp-commands.h) $(qapi-obj-y)
>   test-qmp-commands: test-qmp-commands.o $(qobject-obj-y) $(qapi-obj-y) $(tools-obj-y) $(qapi-dir)/test-qapi-visit.o $(qapi-dir)/test-qapi-types.o $(qapi-dir)/test-qmp-marshal.o module.o
>
> +$(SRC_PATH)/tests/qemu-iotests-quick.sh: qemu-img qemu-io
> +
>   .PHONY: check
>   check: $(CHECKS)
>   	$(call quiet-command, gtester $(CHECKS), "  CHECK")
> diff --git a/tests/qemu-iotests-quick.sh b/tests/qemu-iotests-quick.sh
> new file mode 100755
> index 0000000..cf90de0
> --- /dev/null
> +++ b/tests/qemu-iotests-quick.sh
> @@ -0,0 +1,17 @@
> +#!/bin/sh
> +
> +# We don't know which of the system emulator binaries there is (or if there is
> +# any at all), so the 'quick' group doesn't contain any tests that require
> +# running qemu proper. Assign a fake binary name so that qemu-iotests doesn't
> +# complain about the missing binary.
> +export QEMU_PROG="this_should_be_unused"
> +
> +export QEMU_IMG_PROG="$(pwd)/qemu-img"
> +export QEMU_IO_PROG="$(pwd)/qemu-io"
> +
> +cd $SRC_PATH/tests/qemu-iotests
> +
> +ret=0
> +./check -T -nocache -qcow2 -g quick || ret=1
> +
> +exit $ret

Re: [Qemu-devel] [PATCH v2 2/3] make check: Add qemu-iotests subset

[Kevin Wolf]

Am 09.03.2012 20:31, schrieb Anthony Liguori:
> On 03/09/2012 06:46 AM, Kevin Wolf wrote:
>> Run the 'quick' group from qemu-iotests during 'make check'.
>>
>> Signed-off-by: Kevin Wolf<kwolf@redhat.com>
>> ---
>>   tests/Makefile              |    5 +++++
>>   tests/qemu-iotests-quick.sh |   17 +++++++++++++++++
>>   2 files changed, 22 insertions(+), 0 deletions(-)
>>   create mode 100755 tests/qemu-iotests-quick.sh
>>
>> diff --git a/tests/Makefile b/tests/Makefile
>> index 74b29dc..571ad42 100644
>> --- a/tests/Makefile
>> +++ b/tests/Makefile
>> @@ -1,6 +1,9 @@
>> +export SRC_PATH
>> +
>>   CHECKS = check-qdict check-qfloat check-qint check-qstring check-qlist
>>   CHECKS += check-qjson test-qmp-output-visitor test-qmp-input-visitor
>>   CHECKS += test-string-input-visitor test-string-output-visitor test-coroutine
>> +CHECKS += $(SRC_PATH)/tests/qemu-iotests-quick.sh
> 
> Not to suggest that we don't start here, but I think we should look into how to 
> make qemu-iotest use gtester in the near future.
> 
> That will allow the qemu-iotest to be part of the make check-report output and 
> will provide an easy way for other tools (like autotest) to run these tests.

Yes, we need to figure out how to integrate shell scripts with gtester.
If we can't, we need to rethink gtester as our framework and put
something more flexible on top.

Kevin