[Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

Hi,

I've published a set of tests I wrote over the weekend on qemu.org.  My 
motivations were 1) to prevent regressions like the libguestfs one and 2) to 
have an easier way to do development testing as I work on QEMU Object Model.

Now before sending the obligatory, "What about using KVM autotest" reply, note 
that this is significantly different than KVM autotest and really occupies a 
different use-case.

It has the following characteristics:

1) It builds a custom kernel and initramfs based on busybox.  This is fairly 
important to ensure that we can run tests with no device pre-requisites.

2) Tests are scripts that are launched in the initramfs

3) The test controls exactly how QEMU is launched which allows easy testing of 
various QEMU options

4) Tests can interact with the monitor via QMP while the test is running

5) The tests execute very quickly, can be run stand alone, and do not require 
root privileges

6) They are random by nature with the ability to fix the seed in order to be 
used in git-bisect.

I think Gerd had been looking at doing something similar with a custom initrd.

I've tried to consider other architectures and had hoped that we could commit 
the vmlinuz and initramfs into git so that it was easy to test other 
architectures without having a full build environment.  Unfortunately, busybox 
doesn't link statically with glibc and I can't see an obvious way to commit 
binaries while respecting the GPL since we need to pull glibc into the initramfs.

I know buildroot exists specifically to deal with this but in my experience, 
buildroot is very unreliable and extremely heavy weight since it rebuilds gcc 
multiple times in order to bootstrap a ulibc environment.

Anyway, the code is available at:

http://git.qemu.org/qemu-test.git

See the README for instructions on how to use it.

Regards,

Anthony Liguori

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/19/2011 07:13 PM, Anthony Liguori wrote:
> Hi,
>
> I've published a set of tests I wrote over the weekend on qemu.org. 
> My motivations were 1) to prevent regressions like the libguestfs one
> and 2) to have an easier way to do development testing as I work on
> QEMU Object Model.
>
> Now before sending the obligatory, "What about using KVM autotest"
> reply, note that this is significantly different than KVM autotest and
> really occupies a different use-case.
>

The consensus of that future thread is that kvm-autotest needs to be
able to driver qemu-test, with longer repeat counts where appropriate.

This is similar to kvm-unit-tests btw, that too is a standalone project,
with some glue magic in kvm-autotest.  And they lived happily ever after.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/19/2011 11:39 AM, Avi Kivity wrote:
> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>> Hi,
>>
>> I've published a set of tests I wrote over the weekend on qemu.org.
>> My motivations were 1) to prevent regressions like the libguestfs one
>> and 2) to have an easier way to do development testing as I work on
>> QEMU Object Model.
>>
>> Now before sending the obligatory, "What about using KVM autotest"
>> reply, note that this is significantly different than KVM autotest and
>> really occupies a different use-case.
>>
>
> The consensus of that future thread is that kvm-autotest needs to be
> able to driver qemu-test, with longer repeat counts where appropriate.

Ack.  I'm happy to help make the driver work too.

Regards,

Anthony Liguori

>
> This is similar to kvm-unit-tests btw, that too is a standalone project,
> with some glue magic in kvm-autotest.  And they lived happily ever after.
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Lucas Meneghel Rodrigues]

On 12/19/2011 03:55 PM, Anthony Liguori wrote:
> On 12/19/2011 11:39 AM, Avi Kivity wrote:
>> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>>> Hi,
>>>
>>> I've published a set of tests I wrote over the weekend on qemu.org.
>>> My motivations were 1) to prevent regressions like the libguestfs one
>>> and 2) to have an easier way to do development testing as I work on
>>> QEMU Object Model.
>>>
>>> Now before sending the obligatory, "What about using KVM autotest"
>>> reply, note that this is significantly different than KVM autotest and
>>> really occupies a different use-case.
>>>
>>
>> The consensus of that future thread is that kvm-autotest needs to be
>> able to driver qemu-test, with longer repeat counts where appropriate.
>
> Ack. I'm happy to help make the driver work too.
>
> Regards,
>
> Anthony Liguori
>
>>
>> This is similar to kvm-unit-tests btw, that too is a standalone project,
>> with some glue magic in kvm-autotest. And they lived happily ever after.

I'm happy to see we already wrapped up the discussion. I'll gladly 
review the set of tests and make sure autotest can drive it. Once that 
is done, we'll execute the latest suite against the latest master branch 
daily.

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Dor Laor]

[-- Attachment #1: Type: text/plain, Size: 3673 bytes --]

On 12/19/2011 07:13 PM, Anthony Liguori wrote:
> Hi,
>
> I've published a set of tests I wrote over the weekend on qemu.org. My
> motivations were 1) to prevent regressions like the libguestfs one and
> 2) to have an easier way to do development testing as I work on QEMU
> Object Model.
>
> Now before sending the obligatory, "What about using KVM autotest"
> reply, note that this is significantly different than KVM autotest and
> really occupies a different use-case.

Well, I'm still not convinced that a new standalone package should 
handle these cases instead of kvm autotest. I'll be happy to integrate 
the tests to kvm autotest anyway and the more the merrier but imho it's 
a duplicate.

kvm-unit-test is a case for unit tests since it replaces the guest 
kernel itself and test for really low level tests like task switch, 
instruction emulation, etc.

>
> It has the following characteristics:
>
> 1) It builds a custom kernel and initramfs based on busybox. This is
> fairly important to ensure that we can run tests with no device
> pre-requisites.

This can be done easily w/ autotest too.

>
> 2) Tests are scripts that are launched in the initramfs
>
> 3) The test controls exactly how QEMU is launched which allows easy
> testing of various QEMU options

afaik kvm autotest do the same. It's true that kvm autotest might look 
less friendly but its may only be this way since it has lots of options.

>
> 4) Tests can interact with the monitor via QMP while the test is running

ditto for kvm autotest

>
> 5) The tests execute very quickly, can be run stand alone, and do not
> require root privileges

ditto for kvm auotest. It's possible to configure it w/o root too which 
is not a huge issue.

Please compare your own virtio-serial test w/ the autotest version of 
it: 
https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py

This single file tests functionality, limits, console, live migration 
and performance. Of course one can add a very basic 'hello world' sanity 
test too that will run quickly and will identify basic breakage fast.

Noways we abstract kvm autotest so libvirt will be optionally tested too 
w/ the same tests.

Again, I'm not advocating that kvm autotest is a solution for anything 
but for plain guest-host communication, monitor commands, etc it's a 
really good tool.

I agree that kvm autotest may be less friendly for developer users since 
it carries allot of options for testing a huge matrix. Lucas and Cleber 
are working these days to add a make kvmautotest target to qemu so one 
would be able to quickly execute autotest and we can think of additional 
parameters like sanity-set, migration-set, etc.

>
> 6) They are random by nature with the ability to fix the seed in order
> to be used in git-bisect.
>
> I think Gerd had been looking at doing something similar with a custom
> initrd.
>
> I've tried to consider other architectures and had hoped that we could
> commit the vmlinuz and initramfs into git so that it was easy to test
> other architectures without having a full build environment.
> Unfortunately, busybox doesn't link statically with glibc and I can't
> see an obvious way to commit binaries while respecting the GPL since we
> need to pull glibc into the initramfs.
>
> I know buildroot exists specifically to deal with this but in my
> experience, buildroot is very unreliable and extremely heavy weight
> since it rebuilds gcc multiple times in order to bootstrap a ulibc
> environment.
>
> Anyway, the code is available at:
>
> http://git.qemu.org/qemu-test.git
>
> See the README for instructions on how to use it.
>
> Regards,
>
> Anthony Liguori
>


[-- Attachment #2: Attached Message --]
[-- Type: message/rfc822, Size: 5433 bytes --]

From: Anthony Liguori <aliguori@us.ibm.com>
To: qemu-devel@nongnu.org
Cc: Anthony Liguori <aliguori@us.ibm.com>
Subject: [Qemu-devel] [PATCH] qemu-test: add virtio-serial test
Date: Thu, 22 Dec 2011 08:23:25 -0600
Message-ID: <1324563805-4285-1-git-send-email-aliguori@us.ibm.com>

This is a pretty simple test that just confirms that virtio-serial shows up and
is writable.  It also tests the alias for virtio-serial-pci.

Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
---
 tests/virtio-serial.sh |   52 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 52 insertions(+), 0 deletions(-)
 create mode 100755 tests/virtio-serial.sh

diff --git a/tests/virtio-serial.sh b/tests/virtio-serial.sh
new file mode 100755
index 0000000..e95ae6e
--- /dev/null
+++ b/tests/virtio-serial.sh
@@ -0,0 +1,52 @@
+#!/bin/sh
+
+canary="** waiting for... **"
+
+in_host() {
+    tmpchr=$tmpdir/chr.log
+
+    # Also test alias
+    devname=`choose virtio-serial virtio-serial-pci`
+
+    qemu -nographic -enable-kvm -device $devname \
+        -device virtserialport,name=org.qemu.test,chardev=chr0 \
+        -chardev file,path=$tmpchr,id=chr0
+    rc=$?
+
+    if test $rc = 0; then
+	if ! grep "$canary" $tmpchr >/dev/null; then
+	    echo "Failed to see output from guest!"
+	    rc=1
+	fi
+    fi
+
+    rm -f $tmpchr
+
+    return $rc
+}
+
+in_guest() {
+    sysfspath=/sys/bus/virtio/devices/virtio0/virtio-ports/vport0p1
+    if ! test -e $sysfspath/name; then
+	echo "Device not visible!"
+	return 1
+    fi
+
+    name=`cat $sysfspath/name`
+
+    if test "$name" != "org.qemu.test"; then
+	echo "Device has wrong name!"
+	echo "Expected 'org.qemu.test', got '$name'"
+	return 2
+    fi
+
+    echo "$canary" > /dev/vport0p1
+
+    return 0
+}
+
+if test $QEMU_TEST; then
+    in_host
+else
+    in_guest
+fi
-- 
1.7.4.1

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

Hi Dor,

On 12/25/2011 09:19 AM, Dor Laor wrote:
> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>
> Well, I'm still not convinced that a new standalone package should handle these
> cases instead of kvm autotest. I'll be happy to integrate the tests to kvm
> autotest anyway and the more the merrier but imho it's a duplicate.

I'm sure kvm autotest could be taught to do exactly what qemu-test is doing. 
But why does kvm autotest have to do everything?  I doubt there would be much 
code reuse.

I think it's not a bad thing to have multiple test suites when there isn't 
considerable overlap.

>> It has the following characteristics:
>>
>> 1) It builds a custom kernel and initramfs based on busybox. This is
>> fairly important to ensure that we can run tests with no device
>> pre-requisites.
>
> This can be done easily w/ autotest too.

Okay, please demonstrate :-)  The procedure to do this with qemu test is:

$ git clone git://git.qemu.org/qemu-test.git
$ cd qemu-test
$ git submodule update --init
$ make

How would one do this with kvm autotest?

>> 2) Tests are scripts that are launched in the initramfs
>>
>> 3) The test controls exactly how QEMU is launched which allows easy
>> testing of various QEMU options
>
> afaik kvm autotest do the same. It's true that kvm autotest might look less
> friendly but its may only be this way since it has lots of options.

Actually, kvm-autotest has various layers of abstraction in how QEMU ends up 
being launched.  As you mention below, those layers are there to allow for 
things like using libvirt.

That's desirable when you're doing "virt testing", but not so desirably when 
you're trying to write specific unit tests against QEMU.

>> 5) The tests execute very quickly, can be run stand alone, and do not
>> require root privileges
>
> ditto for kvm auotest. It's possible to configure it w/o root too which is not a
> huge issue.

When I say, "run quickly", I mean, they execute very quickly.

$ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 
tests/virtio-serial.sh

real	0m4.385s
user	0m1.460s
sys	0m1.860s

I've used kvm-autotest a lot, there is no test in kvm-autotest that is even 
close to completing in 4 seconds start to finish.  I don't think kvm-autotest 
can even run without installing a guest first which puts puts a simple test more 
at the 30 minute mark.  If we're talking about TCG testing, then we're in hours 
territory.

If there's a way to run kvm-autotest as a non-privileged user, I've not figured 
it out yet.

Of course, kvm-autotest can test a lot more things than qemu-test can since the 
tests are guest agnostic.  That is the primary architectural difference.  By 
writing the tests to one specific guest (busybox initrd), they can be extremely 
fast in execution, but the tests are less general purpose.

That makes qemu-test very useful for development testing (such as long automated 
bisect runs) but less useful for acceptance testing.

> Please compare your own virtio-serial test w/ the autotest version of it:
> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
>
>
> This single file tests functionality, limits, console, live migration and
> performance. Of course one can add a very basic 'hello world' sanity test too
> that will run quickly and will identify basic breakage fast.

Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.

There is a lot of value in being able to write simple tests in 50 lines of code. 
  In fact, the largest qemu-test at the moment is only ~70 lines of code.  And 
besides finding regressions in my own code (which is the primary use atm), I've 
found and fixed a few real bugs in upstream QEMU.

For a couple hundred lines of bash script, how can it be anything but a good 
thing :-)

> Noways we abstract kvm autotest so libvirt will be optionally tested too w/ the
> same tests.
>
> Again, I'm not advocating that kvm autotest is a solution for anything but for
> plain guest-host communication, monitor commands, etc it's a really good tool.
>
> I agree that kvm autotest may be less friendly for developer users since it
> carries allot of options for testing a huge matrix. Lucas and Cleber are working
> these days to add a make kvmautotest target to qemu so one would be able to
> quickly execute autotest and we can think of additional parameters like
> sanity-set, migration-set, etc.

We (the QEMU project) need to get more serious about testing.  We need to 
(quickly) get to a point where we can mandate that features come with test 
cases.  It is extremely hard to implement touch everything features with no good 
way to test all of the different things we support.

kvm autotest cannot fill that role because it's too complicated and too many 
layers removed from QEMU.

virtio-console/virtio-serial-bus in about 1400 LOC in QEMU.  I'm not about to 
mandate that someone writes 2.2k LOC in a python test framework in order to get 
1.4k of code merged in QEMU.

But 50 lines of bash seems like a more than reasonable requirement.

Regards,

Anthony Liguori

>>
>> 6) They are random by nature with the ability to fix the seed in order
>> to be used in git-bisect.
>>
>> I think Gerd had been looking at doing something similar with a custom
>> initrd.
>>
>> I've tried to consider other architectures and had hoped that we could
>> commit the vmlinuz and initramfs into git so that it was easy to test
>> other architectures without having a full build environment.
>> Unfortunately, busybox doesn't link statically with glibc and I can't
>> see an obvious way to commit binaries while respecting the GPL since we
>> need to pull glibc into the initramfs.
>>
>> I know buildroot exists specifically to deal with this but in my
>> experience, buildroot is very unreliable and extremely heavy weight
>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>> environment.
>>
>> Anyway, the code is available at:
>>
>> http://git.qemu.org/qemu-test.git
>>
>> See the README for instructions on how to use it.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Dor Laor]

On 12/26/2011 05:12 PM, Anthony Liguori wrote:
> Hi Dor,
>

Merry Christmas Anthony,

> On 12/25/2011 09:19 AM, Dor Laor wrote:
>> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>>
>> Well, I'm still not convinced that a new standalone package should
>> handle these
>> cases instead of kvm autotest. I'll be happy to integrate the tests to
>> kvm
>> autotest anyway and the more the merrier but imho it's a duplicate.
>
> I'm sure kvm autotest could be taught to do exactly what qemu-test is
> doing. But why does kvm autotest have to do everything? I doubt there
> would be much code reuse.
>
> I think it's not a bad thing to have multiple test suites when there
> isn't considerable overlap.

I agree but in this case, it loos to me that qemu-test is likely to do a 
subset of what kvm autotest do in a potentially less elegant way but due 
to the influence of a key maintainer may gain traction and dilute kvm 
autotest..

>
>>> It has the following characteristics:
>>>
>>> 1) It builds a custom kernel and initramfs based on busybox. This is
>>> fairly important to ensure that we can run tests with no device
>>> pre-requisites.
>>
>> This can be done easily w/ autotest too.
>
> Okay, please demonstrate :-) The procedure to do this with qemu test is:
>
> $ git clone git://git.qemu.org/qemu-test.git
> $ cd qemu-test
> $ git submodule update --init
> $ make
>
> How would one do this with kvm autotest?

git clone git://github.com/autotest/autotest.git
cd autotest/client/tests/kvm
sudo ./get_started.py

One command shorter and the above downloads F15 automatically and runs it.

(from 
https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript)

>>> 2) Tests are scripts that are launched in the initramfs
>>>
>>> 3) The test controls exactly how QEMU is launched which allows easy
>>> testing of various QEMU options
>>
>> afaik kvm autotest do the same. It's true that kvm autotest might look
>> less
>> friendly but its may only be this way since it has lots of options.
>
> Actually, kvm-autotest has various layers of abstraction in how QEMU
> ends up being launched. As you mention below, those layers are there to
> allow for things like using libvirt.

It goes beyond that, since it also related to the monitor interface as well.

> That's desirable when you're doing "virt testing", but not so desirably
> when you're trying to write specific unit tests against QEMU.

True, one may not need it at all but it's nice that a test for 
migration/stress/hotplug will be tested directly w/ qemu and libvirt w/ 
the same effort.

>
>>> 5) The tests execute very quickly, can be run stand alone, and do not
>>> require root privileges
>>
>> ditto for kvm auotest. It's possible to configure it w/o root too
>> which is not a
>> huge issue.
>
> When I say, "run quickly", I mean, they execute very quickly.

/me too

>
> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
> tests/virtio-serial.sh
>
> real 0m4.385s
> user 0m1.460s
> sys 0m1.860s

That's impressive but it's more of a function of the guest being used - 
if instead of running a full Fedora install, you'll chose your busybox 
image w/ -kernel/initrd you'll get a similar result.

>
> I've used kvm-autotest a lot, there is no test in kvm-autotest that is
> even close to completing in 4 seconds start to finish. I don't think
> kvm-autotest can even run without installing a guest first which puts
> puts a simple test more at the 30 minute mark. If we're talking about
> TCG testing, then we're in hours territory.

Autotest today may be too verbose by default and will call
echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way 
and Lucas can easily create a slim mode that does not even need to be root.

>
> If there's a way to run kvm-autotest as a non-privileged user, I've not
> figured it out yet.

Again, there is no magic in a test suite so if required we can wave it 
(maybe we need to do that regardless)

>
> Of course, kvm-autotest can test a lot more things than qemu-test can
> since the tests are guest agnostic. That is the primary architectural
> difference. By writing the tests to one specific guest (busybox initrd),
> they can be extremely fast in execution, but the tests are less general
> purpose.
>
> That makes qemu-test very useful for development testing (such as long
> automated bisect runs) but less useful for acceptance testing.

Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and 
the tests will be slimmer or each test will have such a mode.

>
>> Please compare your own virtio-serial test w/ the autotest version of it:
>> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
>>
>>
>>
>> This single file tests functionality, limits, console, live migration and
>> performance. Of course one can add a very basic 'hello world' sanity
>> test too
>> that will run quickly and will identify basic breakage fast.
>
> Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.

I bet that 
virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will be 
less than 52 lines. It depends what you like to test. as noted, the 
above test fully covers all aspects of virtio-serial. It does not have 
to be this way and I do agree it is nicer to split it to multiple sub files.

>
> There is a lot of value in being able to write simple tests in 50 lines
> of code. In fact, the largest qemu-test at the moment is only ~70 lines
> of code. And besides finding regressions in my own code (which is the
> primary use atm), I've found and fixed a few real bugs in upstream QEMU.

Now imagine what virtio_console.py buys you :)

>
> For a couple hundred lines of bash script, how can it be anything but a
> good thing :-)
>
>> Noways we abstract kvm autotest so libvirt will be optionally tested
>> too w/ the
>> same tests.
>>
>> Again, I'm not advocating that kvm autotest is a solution for anything
>> but for
>> plain guest-host communication, monitor commands, etc it's a really
>> good tool.
>>
>> I agree that kvm autotest may be less friendly for developer users
>> since it
>> carries allot of options for testing a huge matrix. Lucas and Cleber
>> are working
>> these days to add a make kvmautotest target to qemu so one would be
>> able to
>> quickly execute autotest and we can think of additional parameters like
>> sanity-set, migration-set, etc.
>
> We (the QEMU project) need to get more serious about testing. We need to

Like

> (quickly) get to a point where we can mandate that features come with
> test cases. It is extremely hard to implement touch everything features
> with no good way to test all of the different things we support.
>
> kvm autotest cannot fill that role because it's too complicated and too
> many layers removed from QEMU.
>
> virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not
> about to mandate that someone writes 2.2k LOC in a python test framework
> in order to get 1.4k of code merged in QEMU.
>
> But 50 lines of bash seems like a more than reasonable requirement.

Hmm, what's the coverage of these 50 LOC?
Maybe when your maintainer hat is on, it is indeed a huge task to keep 
all of qemu platforms/architectures to compile and run so that's enough 
but when I wear my Red Hat manager role, I wish that it will not only 
compile and boot but will test as much functionality as possible and 
will be easily re-used by R&D, QA, libvirt, while all supported guests 
are _potentially_ exercised, not just busybox.

Since kvm autotest will be there anyway, I wish that developers will 
contribute 50 LOC to autotest (and not 2.2kloc) and hope the ROI will 
justify it.

I agree autotest is not perfect but it likes to be such.
If you wish, you can challenge Lucas and Cleber w/ these type of 
requirements and we'll all improve as a result.

Cheers,
Dor

>
> Regards,
>
> Anthony Liguori
>
>>>
>>> 6) They are random by nature with the ability to fix the seed in order
>>> to be used in git-bisect.
>>>
>>> I think Gerd had been looking at doing something similar with a custom
>>> initrd.
>>>
>>> I've tried to consider other architectures and had hoped that we could
>>> commit the vmlinuz and initramfs into git so that it was easy to test
>>> other architectures without having a full build environment.
>>> Unfortunately, busybox doesn't link statically with glibc and I can't
>>> see an obvious way to commit binaries while respecting the GPL since we
>>> need to pull glibc into the initramfs.
>>>
>>> I know buildroot exists specifically to deal with this but in my
>>> experience, buildroot is very unreliable and extremely heavy weight
>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>>> environment.
>>>
>>> Anyway, the code is available at:
>>>
>>> http://git.qemu.org/qemu-test.git
>>>
>>> See the README for instructions on how to use it.
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>>
>>
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/26/2011 05:00 PM, Dor Laor wrote:
> On 12/26/2011 05:12 PM, Anthony Liguori wrote:
>> Hi Dor,
>>
>
> Merry Christmas Anthony,

Happy Hanukkah!

>> I think it's not a bad thing to have multiple test suites when there
>> isn't considerable overlap.
>
> I agree but in this case, it loos to me that qemu-test is likely to do a subset
> of what kvm autotest do in a potentially less elegant way but due to the
> influence of a key maintainer may gain traction and dilute kvm autotest..

I'm worried that if we take an attitude of doing what we've always done, we'll 
get the same results we always have.

kvm-autotest is a great test suite, but I'm not happy about the number of 
regressions that were present in 1.0 even after a month of release candidates. 
We need to write a lot more tests than we are right now that cover a much 
broader set of functionality.

>>
>>>> It has the following characteristics:
>>>>
>>>> 1) It builds a custom kernel and initramfs based on busybox. This is
>>>> fairly important to ensure that we can run tests with no device
>>>> pre-requisites.
>>>
>>> This can be done easily w/ autotest too.
>>
>> Okay, please demonstrate :-) The procedure to do this with qemu test is:
>>
>> $ git clone git://git.qemu.org/qemu-test.git
>> $ cd qemu-test
>> $ git submodule update --init
>> $ make
>>
>> How would one do this with kvm autotest?
>
> git clone git://github.com/autotest/autotest.git
> cd autotest/client/tests/kvm
> sudo ./get_started.py
>
> One command shorter and the above downloads F15 automatically and runs it.
>
> (from https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript)

I know how to install kvm-autotest :-)  You claimed that autotest could easily 
"build a custom kernel and initramfs based on busybox."  I've studied 
kvm-autotest extensively to determine if it could do what I want it to do and 
came to the conclusion that it couldn't without significant effort.

The infrastructure assumes that you have a full OS available in the guest.  The 
tests are written in Python and make a variety of assumptions.  To my knowledge, 
it's not very practical to build a busybox environment with Python embedded in it.

This means you need to install a distro and have a disk based install.  Not 
requiring this was one of my primary requirements in creating qemu-test mainly 
since the resulting images are very large and installing a guest with TCG is 
extremely slow.

OTOH, you can build a kernel and initrd with a cross compiler and booting a 
kernel, while not fast under TCG, is not all that slow.

>>
>>>> 5) The tests execute very quickly, can be run stand alone, and do not
>>>> require root privileges
>>>
>>> ditto for kvm auotest. It's possible to configure it w/o root too
>>> which is not a
>>> huge issue.
>>
>> When I say, "run quickly", I mean, they execute very quickly.
>
> /me too
>
>>
>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
>> tests/virtio-serial.sh
>>
>> real 0m4.385s
>> user 0m1.460s
>> sys 0m1.860s
>
> That's impressive but it's more of a function of the guest being used - if
> instead of running a full Fedora install, you'll chose your busybox image w/
> -kernel/initrd you'll get a similar result.

I'm fairly certain that this is not easy to do with kvm-autotest.  I'm happy to 
be proven wrong but I didn't just create qemu-test in a vacuum.

>> I've used kvm-autotest a lot, there is no test in kvm-autotest that is
>> even close to completing in 4 seconds start to finish. I don't think
>> kvm-autotest can even run without installing a guest first which puts
>> puts a simple test more at the 30 minute mark. If we're talking about
>> TCG testing, then we're in hours territory.
>
> Autotest today may be too verbose by default and will call
> echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way and Lucas
> can easily create a slim mode that does not even need to be root.

If it's easy, then by all means, please demonstrate.  If qemu-test only serves 
to show what my requirements for kvm-autotest are and Lucas can implement it all 
very easily, then I would be happy to discontinue qemu-test immediately.

But as I mentioned above, I don't think it's that simple.  You basically can't 
have python-based guest tests without having a full guest distribution.  That 
means in the very least, every kvm-autotest test will need to be rewritten.

>> If there's a way to run kvm-autotest as a non-privileged user, I've not
>> figured it out yet.
>
> Again, there is no magic in a test suite so if required we can wave it (maybe we
> need to do that regardless)

I cannot stress enough how important this is.  The entire process from git fetch 
to running the test should not require root privileges.  If you are targeting 
developers, you can't assume that the box is dedicated to testing as you would 
if you're doing QE testing.  That means you need to co-exist with the existing 
environment and not muck around with network setting and stuff like that.

FWIW, I gave this feedback during Lucas' talk at KVM Forum 2011 so this 
shouldn't be new news to anyone.

>> Of course, kvm-autotest can test a lot more things than qemu-test can
>> since the tests are guest agnostic. That is the primary architectural
>> difference. By writing the tests to one specific guest (busybox initrd),
>> they can be extremely fast in execution, but the tests are less general
>> purpose.
>>
>> That makes qemu-test very useful for development testing (such as long
>> automated bisect runs) but less useful for acceptance testing.
>
> Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and the tests
> will be slimmer or each test will have such a mode.

I've requested a simple config for autotest in the past and even offered to 
include it in qemu.git as a standard profile.

The simple config that I use for regression testing takes about two hours to 
complete.  I'd prefer to have something that took a few minutes to complete...

>>> Please compare your own virtio-serial test w/ the autotest version of it:
>>> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
>>>
>>>
>>>
>>>
>>> This single file tests functionality, limits, console, live migration and
>>> performance. Of course one can add a very basic 'hello world' sanity
>>> test too
>>> that will run quickly and will identify basic breakage fast.
>>
>> Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.
>
> I bet that virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will
> be less than 52 lines.

Okay, then please do demonstrate.  I don't mean to be coy here but as I 
mentioned earlier, I've studied kvm-autotest and based on my understanding of 
it, I don't think it's possible.

I'd be thrilled to be proven wrong though.

> It depends what you like to test. as noted, the above
> test fully covers all aspects of virtio-serial. It does not have to be this way
> and I do agree it is nicer to split it to multiple sub files.

Looking closely, I think the main difference is that the kvm-autotest test also 
tests guest semantics.  It tests the behavior of module unload, SIGIO, poll, etc.

While those tests are important for distro QE testing, they are not useful for 
QEMU testing as there's almost nothing that can change in QEMU that's going to 
affect that behavior.  Those are really Linux kernel tests, not QEMU tests.

>> There is a lot of value in being able to write simple tests in 50 lines
>> of code. In fact, the largest qemu-test at the moment is only ~70 lines
>> of code. And besides finding regressions in my own code (which is the
>> primary use atm), I've found and fixed a few real bugs in upstream QEMU.
>
> Now imagine what virtio_console.py buys you :)

Actually, the only thing that it tests that's a QEMU function is having multiple 
ports per PCI device.  That's something I could add to the qemu-test test with a 
couple dozen lines of code I suspect.

>> (quickly) get to a point where we can mandate that features come with
>> test cases. It is extremely hard to implement touch everything features
>> with no good way to test all of the different things we support.
>>
>> kvm autotest cannot fill that role because it's too complicated and too
>> many layers removed from QEMU.
>>
>> virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not
>> about to mandate that someone writes 2.2k LOC in a python test framework
>> in order to get 1.4k of code merged in QEMU.
>>
>> But 50 lines of bash seems like a more than reasonable requirement.
>
> Hmm, what's the coverage of these 50 LOC?

It tests device creation (including via alias), device fingerprinting (although 
that's a different test), and basic I/O functionality.

> Maybe when your maintainer hat is on, it is indeed a huge task to keep all of
> qemu platforms/architectures to compile and run so that's enough but when I wear
> my Red Hat manager role, I wish that it will not only compile and boot but will
> test as much functionality as possible and will be easily re-used by R&D, QA,
> libvirt, while all supported guests are _potentially_ exercised, not just busybox.

Right, there are two different use cases here:

1) light, QEMU specific testing, that can be used as a QEMU regression suite. 
Essentially, the problem to solve is that when doing changes like the memory 
API, just trying to figure out how to boot a cris guest to confirm that it 
didn't fall apart is surprisingly time consuming.  This is the kind of test 
suite that a developer would run more or less after every local commit.

2) Thorough integration tests that test the whole stack including the kernel, 
libvirt, QEMU, and guest software.  This is the sort of suite that a 
distribution would run as part of their QE procedures.

I think these goals are different enough that they deserve to be separate 
projects.  I don't think kvm-autotest would be improved by also taking on (1) as 
it would force lots of change for very little benefit to (2).

> Since kvm autotest will be there anyway, I wish that developers will contribute
> 50 LOC to autotest (and not 2.2kloc) and hope the ROI will justify it.
>
> I agree autotest is not perfect but it likes to be such.
> If you wish, you can challenge Lucas and Cleber w/ these type of requirements
> and we'll all improve as a result.

Well consider qemu-test the challenge.  It's an existence proof that we can have 
a very easy to use framework for testing that runs extremely fast and is very 
easy to write tests for.

I don't think qemu-test would ever be appropriate for distro QE testing as 
kvm-autotest does an excellent job of this.   I really think that this are two 
fundamentally different use-cases that require specialized tools.  Of course, 
I'm happy to be proven wrong.

Regards,

Anthony Liguori

>
> Cheers,
> Dor
>
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>>>
>>>> 6) They are random by nature with the ability to fix the seed in order
>>>> to be used in git-bisect.
>>>>
>>>> I think Gerd had been looking at doing something similar with a custom
>>>> initrd.
>>>>
>>>> I've tried to consider other architectures and had hoped that we could
>>>> commit the vmlinuz and initramfs into git so that it was easy to test
>>>> other architectures without having a full build environment.
>>>> Unfortunately, busybox doesn't link statically with glibc and I can't
>>>> see an obvious way to commit binaries while respecting the GPL since we
>>>> need to pull glibc into the initramfs.
>>>>
>>>> I know buildroot exists specifically to deal with this but in my
>>>> experience, buildroot is very unreliable and extremely heavy weight
>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>>>> environment.
>>>>
>>>> Anyway, the code is available at:
>>>>
>>>> http://git.qemu.org/qemu-test.git
>>>>
>>>> See the README for instructions on how to use it.
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>
>>
>
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/27/2011 05:22 PM, Anthony Liguori wrote:
>
> The infrastructure assumes that you have a full OS available in the
> guest.  The tests are written in Python and make a variety of
> assumptions.  To my knowledge, it's not very practical to build a
> busybox environment with Python embedded in it.

You could move whatever infrastructure qemu-test uses to kvm-autotest,
at which point kvm-autotest will know everything qemu-test knows.  But
there's zero reason to do that, autotest is designed to drive external
tests and in fact most of the tests it supports are not in the autotest
repository.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/27/2011 09:58 AM, Avi Kivity wrote:
> On 12/27/2011 05:22 PM, Anthony Liguori wrote:
>>
>> The infrastructure assumes that you have a full OS available in the
>> guest.  The tests are written in Python and make a variety of
>> assumptions.  To my knowledge, it's not very practical to build a
>> busybox environment with Python embedded in it.
>
> You could move whatever infrastructure qemu-test uses to kvm-autotest,
> at which point kvm-autotest will know everything qemu-test knows.  But
> there's zero reason to do that, autotest is designed to drive external
> tests and in fact most of the tests it supports are not in the autotest
> repository.

Yes.  I think having a qemu-test driver for kvm-autotest that knows enough to 
invoke the qemu-tests and integrate the results in autotest results reporting is 
the right thing to do.

Regards,

Anthony Liguori

>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Lucas Meneghel Rodrigues]

On 12/27/2011 02:40 PM, Anthony Liguori wrote:
> On 12/27/2011 09:58 AM, Avi Kivity wrote:
>> On 12/27/2011 05:22 PM, Anthony Liguori wrote:
>>>
>>> The infrastructure assumes that you have a full OS available in the
>>> guest. The tests are written in Python and make a variety of
>>> assumptions. To my knowledge, it's not very practical to build a
>>> busybox environment with Python embedded in it.
>>
>> You could move whatever infrastructure qemu-test uses to kvm-autotest,
>> at which point kvm-autotest will know everything qemu-test knows. But
>> there's zero reason to do that, autotest is designed to drive external
>> tests and in fact most of the tests it supports are not in the autotest
>> repository.
>
> Yes. I think having a qemu-test driver for kvm-autotest that knows
> enough to invoke the qemu-tests and integrate the results in autotest
> results reporting is the right thing to do.

I am happy with that too.

There's a slight inaccuracy when you've mentioned that KVM autotest 
mandates guest OS installs to perform tests. It's possible to use 
pre-installed guest images, making the time of execution of a test job 
shorter.

But anyway, your reasoning is sound. Considering that's not a lot of 
code overlap between the 2 infrastructures, as long as we can make them 
interact well, all is good. Plus, the way qemu-test is structured 
integrates well with the workflow most qemu developers are used to. In 
my point of view, it's a win-win situation, as I expect more developers 
to invest some time writing test automation.

Ideally it would be nice to have more people contributing tests to both 
qemu-test *and* kvm autotest, no doubt, but this is something we (kvm 
autotest team) still have to figure how to accomplish.

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Cleber Rosa]

On 12/26/2011 08:00 PM, Dor Laor wrote:
> On 12/26/2011 05:12 PM, Anthony Liguori wrote:
>> Hi Dor,
>>
>
> Merry Christmas Anthony,
>
>> On 12/25/2011 09:19 AM, Dor Laor wrote:
>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>>>
>>> Well, I'm still not convinced that a new standalone package should
>>> handle these
>>> cases instead of kvm autotest. I'll be happy to integrate the tests to
>>> kvm
>>> autotest anyway and the more the merrier but imho it's a duplicate.
>>
>> I'm sure kvm autotest could be taught to do exactly what qemu-test is
>> doing. But why does kvm autotest have to do everything? I doubt there
>> would be much code reuse.
>>
>> I think it's not a bad thing to have multiple test suites when there
>> isn't considerable overlap.

I think the main goal of qemu-tests (may be implicit) is to be quick and 
simple. That is indeed great, but if one thinks that's all we'll ever 
going to need, that thought is pretty naive. And it may be true that 
there's room for both test suites... or that, as busy developers, we're 
refusing to deal with the added complexity (qemu alone accounts for a 
lot) and delaying to fix the fixable. I believe on the later.

One example: kvm-autotest has a complex configuration file format with a 
steep learning curve, while a test such as 
qemu-tests/tests/simple-ping.sh would have to be tweaked if somehow the 
kernel detects the first ethernet interface as em1 (such as recent 
Fedora systems do). Simple, but not scalable.

Other example: kvm-autotest can easily implement (copy?) the 
functionality to build a busybox initramfs. We already have code to 
build from sources such as git, local source directories, remote 
tarballs, you named it. And the recipe to build a given package is also 
configurable/extensible so we could even leverage that. qemu-tests, on 
the other hand, would need lots of code to test the same range of guests 
that kvm-autotest tests. Simple, but limited.

So, it looks like, if we add the missing features to both test suites, 
we'll end up with two implementations, using different 
languages/approaches, but with very little added value.

>
> I agree but in this case, it loos to me that qemu-test is likely to do 
> a subset of what kvm autotest do in a potentially less elegant way but 
> due to the influence of a key maintainer may gain traction and dilute 
> kvm autotest..
>
>>
>>>> It has the following characteristics:
>>>>
>>>> 1) It builds a custom kernel and initramfs based on busybox. This is
>>>> fairly important to ensure that we can run tests with no device
>>>> pre-requisites.
>>>
>>> This can be done easily w/ autotest too.

The Python requirement inside the guest is true *if* we want to run 
regular autotest tests inside the guest (see 
autotest/client/virt/tests/autotest.py) and this accounts for very very 
little of kvm autotest usage. All other tests interact with the monitor 
directly and with the guest via ssh/telnet/serial.

So, I see no reason for not using a more expressive language, having an 
API, etc, to do that's done in in_host(). In the guest side, what 
changes is that we send one command at a time, checking/collecting 
results, instead of pushing the complete tests files into the initramfs.

>>
>> Okay, please demonstrate :-) The procedure to do this with qemu test is:
>>
>> $ git clone git://git.qemu.org/qemu-test.git
>> $ cd qemu-test
>> $ git submodule update --init
>> $ make
>>
>> How would one do this with kvm autotest?
>
> git clone git://github.com/autotest/autotest.git
> cd autotest/client/tests/kvm
> sudo ./get_started.py
>
> One command shorter and the above downloads F15 automatically and runs 
> it.
>
> (from 
> https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript)
>
>>>> 2) Tests are scripts that are launched in the initramfs
>>>>
>>>> 3) The test controls exactly how QEMU is launched which allows easy
>>>> testing of various QEMU options
>>>
>>> afaik kvm autotest do the same. It's true that kvm autotest might look
>>> less
>>> friendly but its may only be this way since it has lots of options.
>>
>> Actually, kvm-autotest has various layers of abstraction in how QEMU
>> ends up being launched. As you mention below, those layers are there to
>> allow for things like using libvirt.

Indeed the qemu command line parameters gets generated depending on many 
configuration parameters. It'd be *really* simple to add a configuration 
parameters that overwrites the qemu command with an static one.

>
> It goes beyond that, since it also related to the monitor interface as 
> well.
>
>> That's desirable when you're doing "virt testing", but not so desirably
>> when you're trying to write specific unit tests against QEMU.
>
> True, one may not need it at all but it's nice that a test for 
> migration/stress/hotplug will be tested directly w/ qemu and libvirt 
> w/ the same effort.
>
>>
>>>> 5) The tests execute very quickly, can be run stand alone, and do not
>>>> require root privileges
>>>
>>> ditto for kvm auotest. It's possible to configure it w/o root too
>>> which is not a
>>> huge issue.
>>
>> When I say, "run quickly", I mean, they execute very quickly.
>
> /me too
>
>>
>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
>> tests/virtio-serial.sh
>>
>> real 0m4.385s
>> user 0m1.460s
>> sys 0m1.860s
>
> That's impressive but it's more of a function of the guest being used 
> - if instead of running a full Fedora install, you'll chose your 
> busybox image w/ -kernel/initrd you'll get a similar result.

I also think so. Maybe kvm-autotest would take a little more time 
because of the different approach we take when communicating with the 
guest, but I bet it'd be irrelevant.

>
>>
>> I've used kvm-autotest a lot, there is no test in kvm-autotest that is
>> even close to completing in 4 seconds start to finish. I don't think
>> kvm-autotest can even run without installing a guest first which puts
>> puts a simple test more at the 30 minute mark. If we're talking about
>> TCG testing, then we're in hours territory.
>
> Autotest today may be too verbose by default and will call
> echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way 
> and Lucas can easily create a slim mode that does not even need to be 
> root.

The drop_caches thing has been configurable for ages.

>
>>
>> If there's a way to run kvm-autotest as a non-privileged user, I've not
>> figured it out yet.
>
> Again, there is no magic in a test suite so if required we can wave it 
> (maybe we need to do that regardless)

Some of the things we do in kvm autotest (and are not currently done in 
qemu-tests) indeed need root privileges, but we could add granularity to 
that.

>
>>
>> Of course, kvm-autotest can test a lot more things than qemu-test can
>> since the tests are guest agnostic. That is the primary architectural
>> difference. By writing the tests to one specific guest (busybox initrd),
>> they can be extremely fast in execution, but the tests are less general
>> purpose.
>>
>> That makes qemu-test very useful for development testing (such as long
>> automated bisect runs) but less useful for acceptance testing. 
>
>
> Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and 
> the tests will be slimmer or each test will have such a mode.
>
>>
>>> Please compare your own virtio-serial test w/ the autotest version 
>>> of it:
>>> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py 
>>>
>>>
>>>
>>>
>>> This single file tests functionality, limits, console, live 
>>> migration and
>>> performance. Of course one can add a very basic 'hello world' sanity
>>> test too
>>> that will run quickly and will identify basic breakage fast.
>>
>> Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.
>
> I bet that 
> virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will be 
> less than 52 lines. It depends what you like to test. as noted, the 
> above test fully covers all aspects of virtio-serial. It does not have 
> to be this way and I do agree it is nicer to split it to multiple sub 
> files.
>
>>
>> There is a lot of value in being able to write simple tests in 50 lines
>> of code. In fact, the largest qemu-test at the moment is only ~70 lines
>> of code. And besides finding regressions in my own code (which is the
>> primary use atm), I've found and fixed a few real bugs in upstream QEMU.
>
> Now imagine what virtio_console.py buys you :)
>
>>
>> For a couple hundred lines of bash script, how can it be anything but a
>> good thing :-)
>>
>>> Noways we abstract kvm autotest so libvirt will be optionally tested
>>> too w/ the
>>> same tests.
>>>
>>> Again, I'm not advocating that kvm autotest is a solution for anything
>>> but for
>>> plain guest-host communication, monitor commands, etc it's a really
>>> good tool.
>>>
>>> I agree that kvm autotest may be less friendly for developer users
>>> since it
>>> carries allot of options for testing a huge matrix. Lucas and Cleber
>>> are working
>>> these days to add a make kvmautotest target to qemu so one would be
>>> able to
>>> quickly execute autotest and we can think of additional parameters like
>>> sanity-set, migration-set, etc.
>>
>> We (the QEMU project) need to get more serious about testing. We need to
>
> Like
>
>> (quickly) get to a point where we can mandate that features come with
>> test cases. It is extremely hard to implement touch everything features
>> with no good way to test all of the different things we support.
>>
>> kvm autotest cannot fill that role because it's too complicated and too
>> many layers removed from QEMU.
>>
>> virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not
>> about to mandate that someone writes 2.2k LOC in a python test framework
>> in order to get 1.4k of code merged in QEMU.
>>
>> But 50 lines of bash seems like a more than reasonable requirement.
>
> Hmm, what's the coverage of these 50 LOC?
> Maybe when your maintainer hat is on, it is indeed a huge task to keep 
> all of qemu platforms/architectures to compile and run so that's 
> enough but when I wear my Red Hat manager role, I wish that it will 
> not only compile and boot but will test as much functionality as 
> possible and will be easily re-used by R&D, QA, libvirt, while all 
> supported guests are _potentially_ exercised, not just busybox.
>
> Since kvm autotest will be there anyway, I wish that developers will 
> contribute 50 LOC to autotest (and not 2.2kloc) and hope the ROI will 
> justify it.
>
> I agree autotest is not perfect but it likes to be such.
> If you wish, you can challenge Lucas and Cleber w/ these type of 
> requirements and we'll all improve as a result.

Yes, I believe it'd be a nice exercise for all of us.

The only thing I ask is that we bear at least with some of the 
complexity that kvm-autotest inherently holds... the last requests we 
had was to get rid of all the complexity, while retaining all the other 
nice characteristics. Pretty hard, so I think we failed, or maybe only 
half-succeeded at it.

>
> Cheers,
> Dor
>
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>>>
>>>> 6) They are random by nature with the ability to fix the seed in order
>>>> to be used in git-bisect.
>>>>
>>>> I think Gerd had been looking at doing something similar with a custom
>>>> initrd.
>>>>
>>>> I've tried to consider other architectures and had hoped that we could
>>>> commit the vmlinuz and initramfs into git so that it was easy to test
>>>> other architectures without having a full build environment.
>>>> Unfortunately, busybox doesn't link statically with glibc and I can't
>>>> see an obvious way to commit binaries while respecting the GPL 
>>>> since we
>>>> need to pull glibc into the initramfs.
>>>>
>>>> I know buildroot exists specifically to deal with this but in my
>>>> experience, buildroot is very unreliable and extremely heavy weight
>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>>>> environment.
>>>>
>>>> Anyway, the code is available at:
>>>>
>>>> http://git.qemu.org/qemu-test.git
>>>>
>>>> See the README for instructions on how to use it.
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>
>>
>
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/27/2011 04:35 PM, Cleber Rosa wrote:
> On 12/26/2011 08:00 PM, Dor Laor wrote:
>> On 12/26/2011 05:12 PM, Anthony Liguori wrote:
>>> Hi Dor,
>>>
>>
>> Merry Christmas Anthony,
>>
>>> On 12/25/2011 09:19 AM, Dor Laor wrote:
>>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>>>>
>>>> Well, I'm still not convinced that a new standalone package should
>>>> handle these
>>>> cases instead of kvm autotest. I'll be happy to integrate the tests to
>>>> kvm
>>>> autotest anyway and the more the merrier but imho it's a duplicate.
>>>
>>> I'm sure kvm autotest could be taught to do exactly what qemu-test is
>>> doing. But why does kvm autotest have to do everything? I doubt there
>>> would be much code reuse.
>>>
>>> I think it's not a bad thing to have multiple test suites when there
>>> isn't considerable overlap.
>
> I think the main goal of qemu-tests (may be implicit) is to be quick and simple.

qemu-test doesn't have a main goal.  My goal is to improve QEMU's quality. 
qemu-test is just a tool to help achieve that goal.

> That is indeed great, but if one thinks that's all we'll ever going to need,
> that thought is pretty naive.

I don't know who "we" is, but I can tell you that qemu-test is exactly what *I* 
need.  Consider that I spent a good portion of every single day testing QEMU 
with either my own or other people's patches, making that job easier and more 
automated is fairly important to me.

I'm sharing it because I suspect that a lot of other developers have a similar need.

> And it may be true that there's room for both test
> suites... or that, as busy developers, we're refusing to deal with the added
> complexity (qemu alone accounts for a lot) and delaying to fix the fixable. I
> believe on the later.
>
> One example: kvm-autotest has a complex configuration file format with a steep
> learning curve, while a test such as qemu-tests/tests/simple-ping.sh would have
> to be tweaked if somehow the kernel detects the first ethernet interface as em1
> (such as recent Fedora systems do). Simple, but not scalable.

I can tell by this comment that you don't actually understand how qemu-test 
works.  Please take a look at it before jumping to conclusions about whether it 
should or shouldn't be part of kvm-autotest.

Hint: qemu-test always uses the same kernel because it builds it as part of the 
test suite.  The behavior of how a nic test will never change unless someone 
explicitly changes the kernel.

>>>>> 1) It builds a custom kernel and initramfs based on busybox. This is
>>>>> fairly important to ensure that we can run tests with no device
>>>>> pre-requisites.
>>>>
>>>> This can be done easily w/ autotest too.
>
> The Python requirement inside the guest is true *if* we want to run regular
> autotest tests inside the guest (see autotest/client/virt/tests/autotest.py) and
> this accounts for very very little of kvm autotest usage. All other tests
> interact with the monitor directly and with the guest via ssh/telnet/serial.

qemu-test does not require any specific hardware to be used in the guest which 
lets it test a wider variety of scenarios in QEMU.  So you cannot assume there 
is ssh/telnet/serial available.

>
> So, I see no reason for not using a more expressive language,

I seriously doubt you can build a useful initramfs that contains python without 
doing something crazy like livecd-tools does....

>>> Actually, kvm-autotest has various layers of abstraction in how QEMU
>>> ends up being launched. As you mention below, those layers are there to
>>> allow for things like using libvirt.
>
> Indeed the qemu command line parameters gets generated depending on many
> configuration parameters. It'd be *really* simple to add a configuration
> parameters that overwrites the qemu command with an static one.

But if you're a QEMU developer, you want to have as much control of the command 
line as possible.  For instance, one of the tests in qemu-test makes sure to 
test invocations without -device as this triggers a different code path (there 
was a recent regression in this too).  You can't just add arguments to reproduce 
this behavior.

>>
>> It goes beyond that, since it also related to the monitor interface as well.
>>
>>> That's desirable when you're doing "virt testing", but not so desirably
>>> when you're trying to write specific unit tests against QEMU.
>>
>> True, one may not need it at all but it's nice that a test for
>> migration/stress/hotplug will be tested directly w/ qemu and libvirt w/ the
>> same effort.
>>
>>>
>>>>> 5) The tests execute very quickly, can be run stand alone, and do not
>>>>> require root privileges
>>>>
>>>> ditto for kvm auotest. It's possible to configure it w/o root too
>>>> which is not a
>>>> huge issue.
>>>
>>> When I say, "run quickly", I mean, they execute very quickly.
>>
>> /me too
>>
>>>
>>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
>>> tests/virtio-serial.sh
>>>
>>> real 0m4.385s
>>> user 0m1.460s
>>> sys 0m1.860s
>>
>> That's impressive but it's more of a function of the guest being used - if
>> instead of running a full Fedora install, you'll chose your busybox image w/
>> -kernel/initrd you'll get a similar result.
>
> I also think so. Maybe kvm-autotest would take a little more time because of the
> different approach we take when communicating with the guest, but I bet it'd be
> irrelevant.

I don't see any reason why everything needs to live in kvm-autotest... but if 
you really feel that way, please provide patches that demonstrate how this would 
work.  We could argue indefinitely about how things could work, it's much better 
to compare how things actually do work :-)

>> I agree autotest is not perfect but it likes to be such.
>> If you wish, you can challenge Lucas and Cleber w/ these type of requirements
>> and we'll all improve as a result.
>
> Yes, I believe it'd be a nice exercise for all of us.
>
> The only thing I ask is that we bear at least with some of the complexity that
> kvm-autotest inherently holds...

I think there's something of a knee jerk reaction here.  The existence of 
qemu-test does not take anything away from kvm-autotest.  It's just another tool 
in our arsenal to achieve our joint goal of making QEMU (and KVM) higher quality.

autotest is made to invoke third party tests so the two tools can co-exist in a 
complimentary way.

Regards,

Anthony Liguori

> the last requests we had was to get rid of all
> the complexity, while retaining all the other nice characteristics. Pretty hard,
> so I think we failed, or maybe only half-succeeded at it.
>
>>
>> Cheers,
>> Dor
>>
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>>
>>>>>
>>>>> 6) They are random by nature with the ability to fix the seed in order
>>>>> to be used in git-bisect.
>>>>>
>>>>> I think Gerd had been looking at doing something similar with a custom
>>>>> initrd.
>>>>>
>>>>> I've tried to consider other architectures and had hoped that we could
>>>>> commit the vmlinuz and initramfs into git so that it was easy to test
>>>>> other architectures without having a full build environment.
>>>>> Unfortunately, busybox doesn't link statically with glibc and I can't
>>>>> see an obvious way to commit binaries while respecting the GPL since we
>>>>> need to pull glibc into the initramfs.
>>>>>
>>>>> I know buildroot exists specifically to deal with this but in my
>>>>> experience, buildroot is very unreliable and extremely heavy weight
>>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>>>>> environment.
>>>>>
>>>>> Anyway, the code is available at:
>>>>>
>>>>> http://git.qemu.org/qemu-test.git
>>>>>
>>>>> See the README for instructions on how to use it.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Anthony Liguori
>>>>>
>>>>
>>>
>>
>>
>
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Cleber Rosa]

On 12/27/2011 11:37 PM, Anthony Liguori wrote:
> On 12/27/2011 04:35 PM, Cleber Rosa wrote:
>> On 12/26/2011 08:00 PM, Dor Laor wrote:
>>> On 12/26/2011 05:12 PM, Anthony Liguori wrote:
>>>> Hi Dor,
>>>>
>>>
>>> Merry Christmas Anthony,
>>>
>>>> On 12/25/2011 09:19 AM, Dor Laor wrote:
>>>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>>>>>
>>>>> Well, I'm still not convinced that a new standalone package should
>>>>> handle these
>>>>> cases instead of kvm autotest. I'll be happy to integrate the 
>>>>> tests to
>>>>> kvm
>>>>> autotest anyway and the more the merrier but imho it's a duplicate.
>>>>
>>>> I'm sure kvm autotest could be taught to do exactly what qemu-test is
>>>> doing. But why does kvm autotest have to do everything? I doubt there
>>>> would be much code reuse.
>>>>
>>>> I think it's not a bad thing to have multiple test suites when there
>>>> isn't considerable overlap.
>>
>> I think the main goal of qemu-tests (may be implicit) is to be quick 
>> and simple.
>
> qemu-test doesn't have a main goal.  My goal is to improve QEMU's 
> quality. qemu-test is just a tool to help achieve that goal.

Maybe I've used the wrong wording. Besides the different approach 
(simpler requirements, initramfs with busybox, etc), it looks to me that 
keeping it simple played an important role in your decision to write 
qemu-tests, which is indeed a sign of good design. But read on...

>
>> That is indeed great, but if one thinks that's all we'll ever going 
>> to need,
>> that thought is pretty naive.
>
> I don't know who "we" is, but I can tell you that qemu-test is exactly 
> what *I* need.  Consider that I spent a good portion of every single 
> day testing QEMU with either my own or other people's patches, making 
> that job easier and more automated is fairly important to me.

"We" is everyone that contributes one way or another to qemu. So if you 
only concerned with your needs, you're definitely on the right track.

If not, I think (trying to think as a community) it'd be beneficial to 
concentrate all efforts, unless it's not possible to do so. It's as 
simple as that. All my reasoning here revolves around this.

>
> I'm sharing it because I suspect that a lot of other developers have a 
> similar need.
>
>> And it may be true that there's room for both test
>> suites... or that, as busy developers, we're refusing to deal with 
>> the added
>> complexity (qemu alone accounts for a lot) and delaying to fix the 
>> fixable. I
>> believe on the later.
>>
>> One example: kvm-autotest has a complex configuration file format 
>> with a steep
>> learning curve, while a test such as qemu-tests/tests/simple-ping.sh 
>> would have
>> to be tweaked if somehow the kernel detects the first ethernet 
>> interface as em1
>> (such as recent Fedora systems do). Simple, but not scalable.
>
> I can tell by this comment that you don't actually understand how 
> qemu-test works.  Please take a look at it before jumping to 
> conclusions about whether it should or shouldn't be part of kvm-autotest.

I can tell you did not grasp my point: kvm autotest is more complex, but 
more capable and flexible. And I did *not* come to a conclusion, I'm 
giving examples in an attempt to enrich the discussion.

>
> Hint: qemu-test always uses the same kernel because it builds it as 
> part of the test suite.  The behavior of how a nic test will never 
> change unless someone explicitly changes the kernel.

Hopefully you understand by now that I'm giving some reasons of why 
kvm-autotest does some things the way it does (usually more complex), 
and how qemu-tests, because of its approach, does not have to deal with 
that.

>
>>>>>> 1) It builds a custom kernel and initramfs based on busybox. This is
>>>>>> fairly important to ensure that we can run tests with no device
>>>>>> pre-requisites.
>>>>>
>>>>> This can be done easily w/ autotest too.
>>
>> The Python requirement inside the guest is true *if* we want to run 
>> regular
>> autotest tests inside the guest (see 
>> autotest/client/virt/tests/autotest.py) and
>> this accounts for very very little of kvm autotest usage. All other 
>> tests
>> interact with the monitor directly and with the guest via 
>> ssh/telnet/serial.
>
> qemu-test does not require any specific hardware to be used in the 
> guest which lets it test a wider variety of scenarios in QEMU.  So you 
> cannot assume there is ssh/telnet/serial available.

I was assuming that we could count on at least a serial port in the 
guest. If not, then current kvm-autotest can not absorb the same 
functionality of qemu-tests without re-writing it. A

>
>>
>> So, I see no reason for not using a more expressive language,
>
> I seriously doubt you can build a useful initramfs that contains 
> python without doing something crazy like livecd-tools does....
>
>>>> Actually, kvm-autotest has various layers of abstraction in how QEMU
>>>> ends up being launched. As you mention below, those layers are 
>>>> there to
>>>> allow for things like using libvirt.
>>
>> Indeed the qemu command line parameters gets generated depending on many
>> configuration parameters. It'd be *really* simple to add a configuration
>> parameters that overwrites the qemu command with an static one.
>
> But if you're a QEMU developer, you want to have as much control of 
> the command line as possible.  For instance, one of the tests in 
> qemu-test makes sure to test invocations without -device as this 
> triggers a different code path (there was a recent regression in this 
> too).  You can't just add arguments to reproduce this behavior.
>
>>>
>>> It goes beyond that, since it also related to the monitor interface 
>>> as well.
>>>
>>>> That's desirable when you're doing "virt testing", but not so 
>>>> desirably
>>>> when you're trying to write specific unit tests against QEMU.
>>>
>>> True, one may not need it at all but it's nice that a test for
>>> migration/stress/hotplug will be tested directly w/ qemu and libvirt 
>>> w/ the
>>> same effort.
>>>
>>>>
>>>>>> 5) The tests execute very quickly, can be run stand alone, and do 
>>>>>> not
>>>>>> require root privileges
>>>>>
>>>>> ditto for kvm auotest. It's possible to configure it w/o root too
>>>>> which is not a
>>>>> huge issue.
>>>>
>>>> When I say, "run quickly", I mean, they execute very quickly.
>>>
>>> /me too
>>>
>>>>
>>>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
>>>> tests/virtio-serial.sh
>>>>
>>>> real 0m4.385s
>>>> user 0m1.460s
>>>> sys 0m1.860s
>>>
>>> That's impressive but it's more of a function of the guest being 
>>> used - if
>>> instead of running a full Fedora install, you'll chose your busybox 
>>> image w/
>>> -kernel/initrd you'll get a similar result.
>>
>> I also think so. Maybe kvm-autotest would take a little more time 
>> because of the
>> different approach we take when communicating with the guest, but I 
>> bet it'd be
>> irrelevant.
>
> I don't see any reason why everything needs to live in kvm-autotest... 
> but if you really feel that way, please provide patches that 
> demonstrate how this would work.  We could argue indefinitely about 
> how things could work, it's much better to compare how things actually 
> do work :-)
>
>>> I agree autotest is not perfect but it likes to be such.
>>> If you wish, you can challenge Lucas and Cleber w/ these type of 
>>> requirements
>>> and we'll all improve as a result.
>>
>> Yes, I believe it'd be a nice exercise for all of us.
>>
>> The only thing I ask is that we bear at least with some of the 
>> complexity that
>> kvm-autotest inherently holds...
>
> I think there's something of a knee jerk reaction here.  The existence 
> of qemu-test does not take anything away from kvm-autotest.  It's just 
> another tool in our arsenal to achieve our joint goal of making QEMU 
> (and KVM) higher quality.
>
> autotest is made to invoke third party tests so the two tools can 
> co-exist in a complimentary way.
>
> Regards,
>
> Anthony Liguori
>
>> the last requests we had was to get rid of all
>> the complexity, while retaining all the other nice characteristics. 
>> Pretty hard,
>> so I think we failed, or maybe only half-succeeded at it.
>>
>>>
>>> Cheers,
>>> Dor
>>>
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>>>>
>>>>>> 6) They are random by nature with the ability to fix the seed in 
>>>>>> order
>>>>>> to be used in git-bisect.
>>>>>>
>>>>>> I think Gerd had been looking at doing something similar with a 
>>>>>> custom
>>>>>> initrd.
>>>>>>
>>>>>> I've tried to consider other architectures and had hoped that we 
>>>>>> could
>>>>>> commit the vmlinuz and initramfs into git so that it was easy to 
>>>>>> test
>>>>>> other architectures without having a full build environment.
>>>>>> Unfortunately, busybox doesn't link statically with glibc and I 
>>>>>> can't
>>>>>> see an obvious way to commit binaries while respecting the GPL 
>>>>>> since we
>>>>>> need to pull glibc into the initramfs.
>>>>>>
>>>>>> I know buildroot exists specifically to deal with this but in my
>>>>>> experience, buildroot is very unreliable and extremely heavy weight
>>>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>>>>>> environment.
>>>>>>
>>>>>> Anyway, the code is available at:
>>>>>>
>>>>>> http://git.qemu.org/qemu-test.git
>>>>>>
>>>>>> See the README for instructions on how to use it.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Anthony Liguori
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Cleber Rosa]

On 12/27/2011 11:37 PM, Anthony Liguori wrote:
> On 12/27/2011 04:35 PM, Cleber Rosa wrote:
>> On 12/26/2011 08:00 PM, Dor Laor wrote:
>>> On 12/26/2011 05:12 PM, Anthony Liguori wrote:
>>>> Hi Dor,
>>>>
>>>
>>> Merry Christmas Anthony,
>>>
>>>> On 12/25/2011 09:19 AM, Dor Laor wrote:
>>>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote:
>>>>>
>>>>> Well, I'm still not convinced that a new standalone package should
>>>>> handle these
>>>>> cases instead of kvm autotest. I'll be happy to integrate the 
>>>>> tests to
>>>>> kvm
>>>>> autotest anyway and the more the merrier but imho it's a duplicate.
>>>>
>>>> I'm sure kvm autotest could be taught to do exactly what qemu-test is
>>>> doing. But why does kvm autotest have to do everything? I doubt there
>>>> would be much code reuse.
>>>>
>>>> I think it's not a bad thing to have multiple test suites when there
>>>> isn't considerable overlap.
>>
>> I think the main goal of qemu-tests (may be implicit) is to be quick 
>> and simple.
>
> qemu-test doesn't have a main goal.  My goal is to improve QEMU's 
> quality. qemu-test is just a tool to help achieve that goal.

Maybe I've used the wrong wording. I got the feeling that, besides 
testing qemu the way you need it, keeping qemu-test simple was really 
important. If not, I'd imagine you'd try to re-use something that 
already exists (yes, kvm autotest).

>
>> That is indeed great, but if one thinks that's all we'll ever going 
>> to need,
>> that thought is pretty naive.
>
> I don't know who "we" is, but I can tell you that qemu-test is exactly 
> what *I* need.  Consider that I spent a good portion of every single 
> day testing QEMU with either my own or other people's patches, making 
> that job easier and more automated is fairly important to me.

"We" is everyone that somehow contributes to QEMU, that is, the QEMU 
community. If you're only concerned about what *you* need, then you're 
on the right track. If, besides that, you feel it'd be nice to *try to* 
concentrate our efforts, then we're all on the same track.

>
> I'm sharing it because I suspect that a lot of other developers have a 
> similar need.
>
>> And it may be true that there's room for both test
>> suites... or that, as busy developers, we're refusing to deal with 
>> the added
>> complexity (qemu alone accounts for a lot) and delaying to fix the 
>> fixable. I
>> believe on the later.
>>
>> One example: kvm-autotest has a complex configuration file format 
>> with a steep
>> learning curve, while a test such as qemu-tests/tests/simple-ping.sh 
>> would have
>> to be tweaked if somehow the kernel detects the first ethernet 
>> interface as em1
>> (such as recent Fedora systems do). Simple, but not scalable.
>
> I can tell by this comment that you don't actually understand how 
> qemu-test works.  Please take a look at it before jumping to 
> conclusions about whether it should or shouldn't be part of kvm-autotest.
>
> Hint: qemu-test always uses the same kernel because it builds it as 
> part of the test suite.  The behavior of how a nic test will never 
> change unless someone explicitly changes the kernel.

I can tell you did not get my point: I'm giving some reasons of why 
current kvm autotest is somehow complex, and how qemu-tests gets away 
and keeps it simple.

BTW, I did not jumped at any conclusion. I'm just trying to enrich the 
discussion, which may end up proving that there's no other way to have 
what qemu-tests does.

>
>>>>>> 1) It builds a custom kernel and initramfs based on busybox. This is
>>>>>> fairly important to ensure that we can run tests with no device
>>>>>> pre-requisites.
>>>>>
>>>>> This can be done easily w/ autotest too.
>>
>> The Python requirement inside the guest is true *if* we want to run 
>> regular
>> autotest tests inside the guest (see 
>> autotest/client/virt/tests/autotest.py) and
>> this accounts for very very little of kvm autotest usage. All other 
>> tests
>> interact with the monitor directly and with the guest via 
>> ssh/telnet/serial.
>
> qemu-test does not require any specific hardware to be used in the 
> guest which lets it test a wider variety of scenarios in QEMU.  So you 
> cannot assume there is ssh/telnet/serial available.

I really thought we could rely on, at least, a serial connection. If 
not, then indeed the current kvm autotest approach is not compatible 
with that test environment. That is not to say that kvm autotest 
couldn't incorporate the qemu-tests approach/functionality.

BTW, I just don't like the idea of having lots of functionalities/tests 
implemented on two test suites for a single piece of software, unless 
proven that there's no way around it. To me, this is the whole point of 
this discussion.

>
>>
>> So, I see no reason for not using a more expressive language,
>
> I seriously doubt you can build a useful initramfs that contains 
> python without doing something crazy like livecd-tools does....

You're right. Again, I was thinking we could rely at least on a serial 
connection. Can we not?

>
>>>> Actually, kvm-autotest has various layers of abstraction in how QEMU
>>>> ends up being launched. As you mention below, those layers are 
>>>> there to
>>>> allow for things like using libvirt.
>>
>> Indeed the qemu command line parameters gets generated depending on many
>> configuration parameters. It'd be *really* simple to add a configuration
>> parameters that overwrites the qemu command with an static one.
>
> But if you're a QEMU developer, you want to have as much control of 
> the command line as possible.  For instance, one of the tests in 
> qemu-test makes sure to test invocations without -device as this 
> triggers a different code path (there was a recent regression in this 
> too).  You can't just add arguments to reproduce this behavior.
>
>>>
>>> It goes beyond that, since it also related to the monitor interface 
>>> as well.
>>>
>>>> That's desirable when you're doing "virt testing", but not so 
>>>> desirably
>>>> when you're trying to write specific unit tests against QEMU.
>>>
>>> True, one may not need it at all but it's nice that a test for
>>> migration/stress/hotplug will be tested directly w/ qemu and libvirt 
>>> w/ the
>>> same effort.
>>>
>>>>
>>>>>> 5) The tests execute very quickly, can be run stand alone, and do 
>>>>>> not
>>>>>> require root privileges
>>>>>
>>>>> ditto for kvm auotest. It's possible to configure it w/o root too
>>>>> which is not a
>>>>> huge issue.
>>>>
>>>> When I say, "run quickly", I mean, they execute very quickly.
>>>
>>> /me too
>>>
>>>>
>>>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
>>>> tests/virtio-serial.sh
>>>>
>>>> real 0m4.385s
>>>> user 0m1.460s
>>>> sys 0m1.860s
>>>
>>> That's impressive but it's more of a function of the guest being 
>>> used - if
>>> instead of running a full Fedora install, you'll chose your busybox 
>>> image w/
>>> -kernel/initrd you'll get a similar result.
>>
>> I also think so. Maybe kvm-autotest would take a little more time 
>> because of the
>> different approach we take when communicating with the guest, but I 
>> bet it'd be
>> irrelevant.
>
> I don't see any reason why everything needs to live in kvm-autotest... 
> but if you really feel that way, please provide patches that 
> demonstrate how this would work. 

If it's technically viable, I think that having it as part of kvm 
autotest, shows that the project is more cohesive, attracts more 
contributions, and makes better use of our efforts.

> We could argue indefinitely about how things could work, it's much 
> better to compare how things actually do work :-)

Sure, that's also what I suggested when I mentioned it'd be a "nice 
exercise for all of us".

>
>>> I agree autotest is not perfect but it likes to be such.
>>> If you wish, you can challenge Lucas and Cleber w/ these type of 
>>> requirements
>>> and we'll all improve as a result.
>>
>> Yes, I believe it'd be a nice exercise for all of us.
>>
>> The only thing I ask is that we bear at least with some of the 
>> complexity that
>> kvm-autotest inherently holds...
>
> I think there's something of a knee jerk reaction here.  The existence 
> of qemu-test does not take anything away from kvm-autotest.  It's just 
> another tool in our arsenal to achieve our joint goal of making QEMU 
> (and KVM) higher quality.

You're right, It does not take anything away from kvm autotest today. 
But suppose we can prove that kvm autotest can indeed absorve all of 
qemu-tests functionalities, it'd be itself a reason for doing so. It'd 
avoid finding ourselves with two evolved test tools that do some of the 
same things, but are separate implementations.

>
> autotest is made to invoke third party tests so the two tools can 
> co-exist in a complimentary way.

That's a no brainier and everyone so far agrees on that. The delicate 
issue is whether qemu-tests functionality could live in kvm autotest in 
a nice way.

>
> Regards,
>
> Anthony Liguori
>
>> the last requests we had was to get rid of all
>> the complexity, while retaining all the other nice characteristics. 
>> Pretty hard,
>> so I think we failed, or maybe only half-succeeded at it.
>>
>>>
>>> Cheers,
>>> Dor
>>>
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>>>>
>>>>>> 6) They are random by nature with the ability to fix the seed in 
>>>>>> order
>>>>>> to be used in git-bisect.
>>>>>>
>>>>>> I think Gerd had been looking at doing something similar with a 
>>>>>> custom
>>>>>> initrd.
>>>>>>
>>>>>> I've tried to consider other architectures and had hoped that we 
>>>>>> could
>>>>>> commit the vmlinuz and initramfs into git so that it was easy to 
>>>>>> test
>>>>>> other architectures without having a full build environment.
>>>>>> Unfortunately, busybox doesn't link statically with glibc and I 
>>>>>> can't
>>>>>> see an obvious way to commit binaries while respecting the GPL 
>>>>>> since we
>>>>>> need to pull glibc into the initramfs.
>>>>>>
>>>>>> I know buildroot exists specifically to deal with this but in my
>>>>>> experience, buildroot is very unreliable and extremely heavy weight
>>>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc
>>>>>> environment.
>>>>>>
>>>>>> Anyway, the code is available at:
>>>>>>
>>>>>> http://git.qemu.org/qemu-test.git
>>>>>>
>>>>>> See the README for instructions on how to use it.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Anthony Liguori
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/27/2011 11:01 PM, Cleber Rosa wrote:
> On 12/27/2011 11:37 PM, Anthony Liguori wrote:
>>> I think the main goal of qemu-tests (may be implicit) is to be quick and simple.
>>
>> qemu-test doesn't have a main goal. My goal is to improve QEMU's quality.
>> qemu-test is just a tool to help achieve that goal.
>
> Maybe I've used the wrong wording. I got the feeling that, besides testing qemu
> the way you need it, keeping qemu-test simple was really important.

Simple is always important.  In the case of qemu-test, there are some important 
trade offs made to keep things simple and fast.  It doesn't try to work with 
arbitrary guests which means the tests need to handle only one environment.  The 
guest is pre-made to have exactly what is needed for the tests so there is no 
setup required.  The guest is as small as possible such that the test can run as 
quickly as possible.

>>> That is indeed great, but if one thinks that's all we'll ever going to need,
>>> that thought is pretty naive.
>>
>> I don't know who "we" is, but I can tell you that qemu-test is exactly what
>> *I* need. Consider that I spent a good portion of every single day testing
>> QEMU with either my own or other people's patches, making that job easier and
>> more automated is fairly important to me.
>
> "We" is everyone that somehow contributes to QEMU, that is, the QEMU community.
> If you're only concerned about what *you* need, then you're on the right track.
> If, besides that, you feel it'd be nice to *try to* concentrate our efforts,
> then we're all on the same track.

There is no need to have a single tool that meets every possible need.  In fact, 
the Unix tradition is to have separate single purposed tools.

Having two test tools it not a bad thing provided that the overlap isn't 
significant.  We shouldn't be discussing whether it's possible to merge the two 
tools, but rather what the technical benefits of doing so would be.

Since at this point, there is almost no overlap between the two, I don't see any 
actual technical benefit to merging them.  I see benefit to autotest executing 
qemu-test, of course.

>> I'm sharing it because I suspect that a lot of other developers have a similar
>> need.
>>
>>> And it may be true that there's room for both test
>>> suites... or that, as busy developers, we're refusing to deal with the added
>>> complexity (qemu alone accounts for a lot) and delaying to fix the fixable. I
>>> believe on the later.
>>>
>>> One example: kvm-autotest has a complex configuration file format with a steep
>>> learning curve, while a test such as qemu-tests/tests/simple-ping.sh would have
>>> to be tweaked if somehow the kernel detects the first ethernet interface as em1
>>> (such as recent Fedora systems do). Simple, but not scalable.
>>
>> I can tell by this comment that you don't actually understand how qemu-test
>> works. Please take a look at it before jumping to conclusions about whether it
>> should or shouldn't be part of kvm-autotest.
>>
>> Hint: qemu-test always uses the same kernel because it builds it as part of
>> the test suite. The behavior of how a nic test will never change unless
>> someone explicitly changes the kernel.
>
> I can tell you did not get my point: I'm giving some reasons of why current kvm
> autotest is somehow complex, and how qemu-tests gets away and keeps it simple.

You're claiming that "we're refusing to deal with the added complexity (qemu 
alone accounts for a lot) and delaying to fix the fixable".

There is no way that qemu-test would ever need to deal with how Fedora 
configures udev to name ethernet devices without becoming something totally 
different than it is.  So there's no "delaying to fix the fixable" here.

qemu-test makes a simplifying assumption.  By restricting the guest to a fixed 
environment (initramfs w/busybox), things inherently become much, much simpler.

Of course, this is not an adequate assumption to make if it were our only test 
tool but fortunately, we have an existing test suite that does a very good job 
at testing a wide variety of guests :-)

>>>
>>> The Python requirement inside the guest is true *if* we want to run regular
>>> autotest tests inside the guest (see autotest/client/virt/tests/autotest.py) and
>>> this accounts for very very little of kvm autotest usage. All other tests
>>> interact with the monitor directly and with the guest via ssh/telnet/serial.
>>
>> qemu-test does not require any specific hardware to be used in the guest which
>> lets it test a wider variety of scenarios in QEMU. So you cannot assume there
>> is ssh/telnet/serial available.
>
> I really thought we could rely on, at least, a serial connection. If not, then
> indeed the current kvm autotest approach is not compatible with that test
> environment. That is not to say that kvm autotest couldn't incorporate the
> qemu-tests approach/functionality.

With the scope that qemu-test has, it cannot assume any hardware is present 
because it wants to test every piece of hardware.

> BTW, I just don't like the idea of having lots of functionalities/tests
> implemented on two test suites for a single piece of software, unless proven
> that there's no way around it. To me, this is the whole point of this discussion.

I actually disagree on principle.  Two tools shouldn't be combined unless 
there's a proven reason that they benefit from being together.  You get more 
flexibility at the end of the day having strong, independent components that can 
be combined together.

>>> So, I see no reason for not using a more expressive language,
>>
>> I seriously doubt you can build a useful initramfs that contains python
>> without doing something crazy like livecd-tools does....
>
> You're right. Again, I was thinking we could rely at least on a serial
> connection. Can we not?

No.  How would you test creating a guest with no serial devices?

The nice thing about delivering a test via the initramfs is that you're only 
injecting software into the guest (usually via some non guest visible mechanism).

>> I don't see any reason why everything needs to live in kvm-autotest... but if
>> you really feel that way, please provide patches that demonstrate how this
>> would work.
>
> If it's technically viable, I think that having it as part of kvm autotest,
> shows that the project is more cohesive, attracts more contributions, and makes
> better use of our efforts.

Just putting the code in kvm-autotest.git in a directory doesn't make sense to 
me.  Beyond the lack of a technical reason to do so, logistically, it makes it 
harder for me to ask people to submit test cases with a patch series if I can't 
actually apply the test case when I'm applying the patches to qemu.git.

If qemu-test didn't use large submodules (like linux.git), I would have made 
qemu-test part of qemu.git.  As far as I'm concerned, qemu-test.git is just an 
extension to qemu.git.

>> I think there's something of a knee jerk reaction here. The existence of
>> qemu-test does not take anything away from kvm-autotest. It's just another
>> tool in our arsenal to achieve our joint goal of making QEMU (and KVM) higher
>> quality.
>
> You're right, It does not take anything away from kvm autotest today. But
> suppose we can prove that kvm autotest can indeed absorve all of qemu-tests
> functionalities, it'd be itself a reason for doing so.

That's not obvious to me.

> It'd avoid finding
> ourselves with two evolved test tools that do some of the same things, but are
> separate implementations.

It's just too easy to argue about abstract things like this.  This discussion 
about testing has come up many times and we've talked to death how much we need 
to do better testing and get to test driven development.

Yet, not much has materialized.  I'm not interested in talking about things in 
abstract anymore.  That's why I announced qemu-test.  If folks think it should 
behave differently, then patches are welcome.

If you think that kvm-autotest can do everything that qemu-test does, just make 
the changes to kvm-autotest to make it behave the same way and show me.  If it 
makes sense, I'll happily embrace it.

Regards,

Anthony Liguori

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/28/2011 04:27 PM, Anthony Liguori wrote:
>> Maybe I've used the wrong wording. I got the feeling that, besides
>> testing qemu
>> the way you need it, keeping qemu-test simple was really important.
>
>
> Simple is always important.  In the case of qemu-test, there are some
> important trade offs made to keep things simple and fast.  It doesn't
> try to work with arbitrary guests which means the tests need to handle
> only one environment.  The guest is pre-made to have exactly what is
> needed for the tests so there is no setup required.  The guest is as
> small as possible such that the test can run as quickly as possible.

In fact using linux as a guest negates that.  First of all, which linux
version? if it's fixed, you'll eventually miss functionality and need to
migrate.  If it keeps changing, so does your test, and it will keep
breaking.

Using Linux as a guest allows you to do system tests (ping was an
example) but doesn't allow you to do unit tests (test regression where
if this bit in that register was set, but if them bits in thar registers
were clear, then the packet would be encoded with ROT-26 before being
sent out).

I think qemu-test needs to use its own drivers which allow full control
over what you do with the hardware.  Otherwise, a regression that only
shows up in non-Linux guests will not be testable with qemu-test.

> There is no need to have a single tool that meets every possible
> need.  In fact, the Unix tradition is to have separate single purposed
> tools.

Those single purpose tools, if successful, tend to grow more purposes
(cf. qemu), and if unsuccessful, tend to lose purpose.

> Having two test tools it not a bad thing provided that the overlap
> isn't significant.  

This is important, if the boundary isn't clear, then it will grow more
fuzzy in time.

I suggest the following:

 - qemu-test: qemu unit tests
 - kvm-unit-tests: kvm unit tests
 - kvm-autotest: unit test drivers + system tests

> We shouldn't be discussing whether it's possible to merge the two
> tools, but rather what the technical benefits of doing so would be.
>
> Since at this point, there is almost no overlap between the two, I
> don't see any actual technical benefit to merging them.  I see benefit
> to autotest executing qemu-test, of course.

I'd say that running a ping test is a weak version of kvm-autotest's
system tests.  Running a synthetic test that pokes values into memory
and mmio and sees a packet coming out is a unit test (the latter can in
fact be executed without a guest at all, just a table driving calls to
the memory and irq APIs).


> Just putting the code in kvm-autotest.git in a directory doesn't make
> sense to me.  Beyond the lack of a technical reason to do so,
> logistically, it makes it harder for me to ask people to submit test
> cases with a patch series if I can't actually apply the test case when
> I'm applying the patches to qemu.git.
>
> If qemu-test didn't use large submodules (like linux.git), I would
> have made qemu-test part of qemu.git.  As far as I'm concerned,
> qemu-test.git is just an extension to qemu.git.

Why not just put it in qemu.git?

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/28/2011 05:01 PM, Avi Kivity wrote:
> I'd say that running a ping test is a weak version of kvm-autotest's
> system tests.  Running a synthetic test that pokes values into memory
> and mmio and sees a packet coming out is a unit test (the latter can in
> fact be executed without a guest at all, just a table driving calls to
> the memory and irq APIs).
>

Consider
  98d23704138e0
  7b4252e83f6f7d
  f7e80adf3cc4
  c16ada980f43
  4abf12f4ea8

(found by looking for 'fix' in the commit log and filtering out the
commits that don't support my case)

how can you reject such patches on the grounds that they're not
accompanied by unit tests? only by making it easy to add tests for
them.  I think it would be hard/impossible to test them with
linux-as-a-guest, since they fix edge cases that linux doesn't invoke. 
But by having our own driver (often just using qmp to poke at memory),
we can easily generate the sequence that triggers the error.

We'd probably need a library to support setting up a pci device's BARs,
but that's easy with qmp/python integration.  You can even poke a small
executable into memory and execute it directly, if you really need guest
cpu interaction.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/28/2011 09:28 AM, Avi Kivity wrote:
> On 12/28/2011 05:01 PM, Avi Kivity wrote:
>> I'd say that running a ping test is a weak version of kvm-autotest's
>> system tests.  Running a synthetic test that pokes values into memory
>> and mmio and sees a packet coming out is a unit test (the latter can in
>> fact be executed without a guest at all, just a table driving calls to
>> the memory and irq APIs).
>>
>
> Consider
>    98d23704138e0
>    7b4252e83f6f7d
>    f7e80adf3cc4
>    c16ada980f43
>    4abf12f4ea8
>
> (found by looking for 'fix' in the commit log and filtering out the
> commits that don't support my case)
>
> how can you reject such patches on the grounds that they're not
> accompanied by unit tests?

That's why I've also proposed qtest.  But having written quite a few qtest unit 
tests by now, you hit the limits of this type of testing pretty quickly.

> only by making it easy to add tests for
> them.  I think it would be hard/impossible to test them with
> linux-as-a-guest, since they fix edge cases that linux doesn't invoke.
> But by having our own driver (often just using qmp to poke at memory),
> we can easily generate the sequence that triggers the error.
>
> We'd probably need a library to support setting up a pci device's BARs,
> but that's easy with qmp/python integration.  You can even poke a small
> executable into memory and execute it directly, if you really need guest
> cpu interaction.

Please review the qtest series.  I think it offers a pretty good approach to 
writing this style of test.  But as I mentioned, you hit the limits pretty quickly.

Regards,

Anthony Liguori

>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/28/2011 06:44 PM, Anthony Liguori wrote:
> On 12/28/2011 09:28 AM, Avi Kivity wrote:
>> On 12/28/2011 05:01 PM, Avi Kivity wrote:
>>> I'd say that running a ping test is a weak version of kvm-autotest's
>>> system tests.  Running a synthetic test that pokes values into memory
>>> and mmio and sees a packet coming out is a unit test (the latter can in
>>> fact be executed without a guest at all, just a table driving calls to
>>> the memory and irq APIs).
>>>
>>
>> Consider
>>    98d23704138e0
>>    7b4252e83f6f7d
>>    f7e80adf3cc4
>>    c16ada980f43
>>    4abf12f4ea8
>>
>> (found by looking for 'fix' in the commit log and filtering out the
>> commits that don't support my case)
>>
>> how can you reject such patches on the grounds that they're not
>> accompanied by unit tests?
>
> That's why I've also proposed qtest.  But having written quite a few
> qtest unit tests by now, you hit the limits of this type of testing
> pretty quickly.

Can you describe those limits?

>
>> only by making it easy to add tests for
>> them.  I think it would be hard/impossible to test them with
>> linux-as-a-guest, since they fix edge cases that linux doesn't invoke.
>> But by having our own driver (often just using qmp to poke at memory),
>> we can easily generate the sequence that triggers the error.
>>
>> We'd probably need a library to support setting up a pci device's BARs,
>> but that's easy with qmp/python integration.  You can even poke a small
>> executable into memory and execute it directly, if you really need guest
>> cpu interaction.
>
> Please review the qtest series.  I think it offers a pretty good
> approach to writing this style of test.  But as I mentioned, you hit
> the limits pretty quickly.

I think it's great, it looks like exactly what I wanted, except it's
been delivered on time.  I'd really like to see it integrated quickly
with some flesh around it, then replying -ENOTEST to all patches.  This
will improve qemu's quality a lot more than guest boot/ping tests, which
we do regularly with kvm-autotest anyway.

Think of how new instruction emulations are always accompanied by new
kvm-unit-tests tests, I often don't even have to ask for them.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/28/2011 11:26 AM, Avi Kivity wrote:
> On 12/28/2011 06:44 PM, Anthony Liguori wrote:
>> On 12/28/2011 09:28 AM, Avi Kivity wrote:
>>> On 12/28/2011 05:01 PM, Avi Kivity wrote:
>>>> I'd say that running a ping test is a weak version of kvm-autotest's
>>>> system tests.  Running a synthetic test that pokes values into memory
>>>> and mmio and sees a packet coming out is a unit test (the latter can in
>>>> fact be executed without a guest at all, just a table driving calls to
>>>> the memory and irq APIs).
>>>>
>>>
>>> Consider
>>>     98d23704138e0
>>>     7b4252e83f6f7d
>>>     f7e80adf3cc4
>>>     c16ada980f43
>>>     4abf12f4ea8
>>>
>>> (found by looking for 'fix' in the commit log and filtering out the
>>> commits that don't support my case)
>>>
>>> how can you reject such patches on the grounds that they're not
>>> accompanied by unit tests?
>>
>> That's why I've also proposed qtest.  But having written quite a few
>> qtest unit tests by now, you hit the limits of this type of testing
>> pretty quickly.
>
> Can you describe those limits?

I started writing a finger print test.  While it's easy to do PCI enumeration, 
it gets pretty nasty if you want to actually access BARs (to read virtio 
registers) and forget about trying to get SMBIOS information as that involves 
mucking around with ACPI.

OTOH, it was very simple to write with qemu-test:

http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/finger-print.sh;h=fc715378a3bbae0b458cc419981c2493d98f5c3d;hb=HEAD

And I ended up finding some long standing bugs in the process:

http://mid.gmane.org/1324305949-20960-2-git-send-email-aliguori@us.ibm.com

It's fairly nasty as it would only show up if doing migration from a new QEMU 
with a new guest to a old QEMU.

I think it's a good example of the type of test that I'm targeting at qemu-test. 
  It's really not interesting to run it across multiple distro types.  But doing 
it with qtest would be prohibitively hard.

>>> only by making it easy to add tests for
>>> them.  I think it would be hard/impossible to test them with
>>> linux-as-a-guest, since they fix edge cases that linux doesn't invoke.
>>> But by having our own driver (often just using qmp to poke at memory),
>>> we can easily generate the sequence that triggers the error.
>>>
>>> We'd probably need a library to support setting up a pci device's BARs,
>>> but that's easy with qmp/python integration.  You can even poke a small
>>> executable into memory and execute it directly, if you really need guest
>>> cpu interaction.
>>
>> Please review the qtest series.  I think it offers a pretty good
>> approach to writing this style of test.  But as I mentioned, you hit
>> the limits pretty quickly.
>
> I think it's great, it looks like exactly what I wanted, except it's
> been delivered on time.

Can you take a look at how interrupts are handled?  From what I gather, the only 
real limitation of this approach is that we won't be able to simulate MSI 
vectors but I don't think that's a huge problem.

I looked at integrating interrupt handling at CPUs itself and that turned out to 
be fairly invasive.

> I'd really like to see it integrated quickly
> with some flesh around it, then replying -ENOTEST to all patches.  This
> will improve qemu's quality a lot more than guest boot/ping tests, which
> we do regularly with kvm-autotest anyway.
>
> Think of how new instruction emulations are always accompanied by new
> kvm-unit-tests tests, I often don't even have to ask for them.

Yes, this is exactly where I'm heading with all of this.

Regards,

Anthony Liguori

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 06:12 PM, Anthony Liguori wrote:
>>> That's why I've also proposed qtest.  But having written quite a few
>>> qtest unit tests by now, you hit the limits of this type of testing
>>> pretty quickly.
>>
>> Can you describe those limits?
>
>
> I started writing a finger print test.  While it's easy to do PCI
> enumeration, it gets pretty nasty if you want to actually access BARs
> (to read virtio registers) 

Why is that? it should be pretty easy to poke values into the BAR registers.

Something else we can do is access the BARs directly.  We have APIs to
poke values into mmio, add an api to poke a value into a device's BAR. 
Now that each BAR is represented by exactly one memory region, known to
the pci core even if it is not mapped, it should be easy.

> and forget about trying to get SMBIOS information as that involves
> mucking around with ACPI.

SMBIOS is built by seabios, yes?  So fingerprinting it should be done
from seabios-test.git.  Let's divide it into two problems:

- seabios pokes qemu to get information.  We should fingerprint this
information to make sure different qemu version can interoperate with
different seabios versions (needed since we migrate the bios along with
the rest of the guest).  I'm guessing most of this info is from fwcfg? 
We can easily fingerprint it like any other device.
- we need to make sure seabios generates the same tables when using -M. 
Here, we're not testing a device, rather we're testing guest code, so it
makes sense to use a guest for this.

However, I don't think it's even necessary.  From a quick read of the
manual, SMBIOS is just a set of static tables in memory which are picked
up using a signature.  So all we need to do is boot an empty guest, read
its memory, and look for the tables.

>
> OTOH, it was very simple to write with qemu-test:
>
> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/finger-print.sh;h=fc715378a3bbae0b458cc419981c2493d98f5c3d;hb=HEAD
>

Yes; but using Linux limits you to what it exposes (of course Linux
exposes quite a lot, so that's mostly a non issue; but we'll have
counterexamples).

>
> And I ended up finding some long standing bugs in the process:
>
> http://mid.gmane.org/1324305949-20960-2-git-send-email-aliguori@us.ibm.com
>
>
> It's fairly nasty as it would only show up if doing migration from a
> new QEMU with a new guest to a old QEMU.

Yes, good catch.

>
> I think it's a good example of the type of test that I'm targeting at
> qemu-test.  It's really not interesting to run it across multiple
> distro types.  But doing it with qtest would be prohibitively hard.

I don't see why.  A python library to read pci config should be a couple
of dozens of lines long.  Then you iterate over all functions and print
selected registers.

>>> Please review the qtest series.  I think it offers a pretty good
>>> approach to writing this style of test.  But as I mentioned, you hit
>>> the limits pretty quickly.
>>
>> I think it's great, it looks like exactly what I wanted, except it's
>> been delivered on time.
>
>
> Can you take a look at how interrupts are handled?  From what I
> gather, the only real limitation of this approach is that we won't be
> able to simulate MSI vectors but I don't think that's a huge problem.
>
> I looked at integrating interrupt handling at CPUs itself and that
> turned out to be fairly invasive.

You mention in the changelog replacing the APIC.  Why can't you do that?

>
>> I'd really like to see it integrated quickly
>> with some flesh around it, then replying -ENOTEST to all patches.  This
>> will improve qemu's quality a lot more than guest boot/ping tests, which
>> we do regularly with kvm-autotest anyway.
>>
>> Think of how new instruction emulations are always accompanied by new
>> kvm-unit-tests tests, I often don't even have to ask for them.
>
> Yes, this is exactly where I'm heading with all of this.

It looks great.  One thing I'd change is to use the qmp protocol
(perhaps not the monitor, just the schema/codegen).  Does something
prohibit this?

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 10:36 AM, Avi Kivity wrote:
> On 12/29/2011 06:12 PM, Anthony Liguori wrote:
>>>> That's why I've also proposed qtest.  But having written quite a few
>>>> qtest unit tests by now, you hit the limits of this type of testing
>>>> pretty quickly.
>>>
>>> Can you describe those limits?
>>
>>
>> I started writing a finger print test.  While it's easy to do PCI
>> enumeration, it gets pretty nasty if you want to actually access BARs
>> (to read virtio registers)
>
> Why is that? it should be pretty easy to poke values into the BAR registers.

You have to map them which means you need to figure out what the memory layout 
looks like.  Since there's no BIOS, you need to poke fw_cfg to figure all of 
this out and then program the BARs appropriately.

It's definitely non-trivial.

> Something else we can do is access the BARs directly.  We have APIs to
> poke values into mmio, add an api to poke a value into a device's BAR.
> Now that each BAR is represented by exactly one memory region, known to
> the pci core even if it is not mapped, it should be easy.
>
>> and forget about trying to get SMBIOS information as that involves
>> mucking around with ACPI.
>
> SMBIOS is built by seabios, yes?

I think QEMU actually builds the blob of information and passes the full blob to 
SeaBIOS.  You could read and parse the blob out of fw_cfg I guess but that's 
still pretty sucky.  It's a lot nicer to just poke sysfs.

> So fingerprinting it should be done
> from seabios-test.git.  Let's divide it into two problems:

That seems awkward especially since fingerprinting Just Works with qemu-test 
already (and it was only 50 lines of shell code!).

> - seabios pokes qemu to get information.  We should fingerprint this
> information to make sure different qemu version can interoperate with
> different seabios versions (needed since we migrate the bios along with
> the rest of the guest).  I'm guessing most of this info is from fwcfg?
> We can easily fingerprint it like any other device.
> - we need to make sure seabios generates the same tables when using -M.
> Here, we're not testing a device, rather we're testing guest code, so it
> makes sense to use a guest for this.
>
> However, I don't think it's even necessary.  From a quick read of the
> manual, SMBIOS is just a set of static tables in memory which are picked
> up using a signature.  So all we need to do is boot an empty guest, read
> its memory, and look for the tables.

Doesn't it sound a whole nicer use linux.git to parse this information for us in 
a friendly, easy to consume fashion?

>> OTOH, it was very simple to write with qemu-test:
>>
>> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/finger-print.sh;h=fc715378a3bbae0b458cc419981c2493d98f5c3d;hb=HEAD
>>
>
> Yes; but using Linux limits you to what it exposes (of course Linux
> exposes quite a lot, so that's mostly a non issue; but we'll have
> counterexamples).

Maybe.  And for those counter examples, we can drill down to qtest.  But just 
because we may need to do fancy things, it doesn't mean we shouldn't take the 
easy approach 95% of the time.

>> And I ended up finding some long standing bugs in the process:
>>
>> http://mid.gmane.org/1324305949-20960-2-git-send-email-aliguori@us.ibm.com
>>
>>
>> It's fairly nasty as it would only show up if doing migration from a
>> new QEMU with a new guest to a old QEMU.
>
> Yes, good catch.
>
>>
>> I think it's a good example of the type of test that I'm targeting at
>> qemu-test.  It's really not interesting to run it across multiple
>> distro types.  But doing it with qtest would be prohibitively hard.
>
> I don't see why.  A python library to read pci config should be a couple
> of dozens of lines long.  Then you iterate over all functions and print
> selected registers.

PCI enumeration is easy.  It's mapping the bars and then poking other sources of 
information that's more challenging.

Everything is doable by writing a libOS but I'd rather just use Linux than 
invent a libOS just for testing.

>>>> Please review the qtest series.  I think it offers a pretty good
>>>> approach to writing this style of test.  But as I mentioned, you hit
>>>> the limits pretty quickly.
>>>
>>> I think it's great, it looks like exactly what I wanted, except it's
>>> been delivered on time.
>>
>>
>> Can you take a look at how interrupts are handled?  From what I
>> gather, the only real limitation of this approach is that we won't be
>> able to simulate MSI vectors but I don't think that's a huge problem.
>>
>> I looked at integrating interrupt handling at CPUs itself and that
>> turned out to be fairly invasive.
>
> You mention in the changelog replacing the APIC.  Why can't you do that?

I currently replace the I/O APIC which seems to be limited to 16 IRQs.  I think 
MSI takes a side channel directly to the local APIC, no?

>>
>>> I'd really like to see it integrated quickly
>>> with some flesh around it, then replying -ENOTEST to all patches.  This
>>> will improve qemu's quality a lot more than guest boot/ping tests, which
>>> we do regularly with kvm-autotest anyway.
>>>
>>> Think of how new instruction emulations are always accompanied by new
>>> kvm-unit-tests tests, I often don't even have to ask for them.
>>
>> Yes, this is exactly where I'm heading with all of this.
>
> It looks great.  One thing I'd change is to use the qmp protocol
> (perhaps not the monitor, just the schema/codegen).  Does something
> prohibit this?

Yeah, I thought about using QMP.  But events are posted in QMP which means that 
you never get an explicit ACK.  That may not be a problem but it's something I 
had in mind.

It also seemed to be reasonably complicated without a clear advantage.  qtest 
isn't going to be a supported protocol so we can certainly change it down the 
road if we want to.

Regards,

Anthony Liguori

>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 06:49 PM, Anthony Liguori wrote:
> On 12/29/2011 10:36 AM, Avi Kivity wrote:
>> On 12/29/2011 06:12 PM, Anthony Liguori wrote:
>>>>> That's why I've also proposed qtest.  But having written quite a few
>>>>> qtest unit tests by now, you hit the limits of this type of testing
>>>>> pretty quickly.
>>>>
>>>> Can you describe those limits?
>>>
>>>
>>> I started writing a finger print test.  While it's easy to do PCI
>>> enumeration, it gets pretty nasty if you want to actually access BARs
>>> (to read virtio registers)
>>
>> Why is that? it should be pretty easy to poke values into the BAR
>> registers.
>
> You have to map them which means you need to figure out what the
> memory layout looks like.  Since there's no BIOS, you need to poke
> fw_cfg to figure all of this out and then program the BARs appropriately.
>
> It's definitely non-trivial.

Or just map sequentially at 0xe0000000 ascending.

>
>> Something else we can do is access the BARs directly.  We have APIs to
>> poke values into mmio, add an api to poke a value into a device's BAR.
>> Now that each BAR is represented by exactly one memory region, known to
>> the pci core even if it is not mapped, it should be easy.
>>
>>> and forget about trying to get SMBIOS information as that involves
>>> mucking around with ACPI.
>>
>> SMBIOS is built by seabios, yes?
>
> I think QEMU actually builds the blob of information and passes the
> full blob to SeaBIOS.  You could read and parse the blob out of fw_cfg
> I guess but that's still pretty sucky.  It's a lot nicer to just poke
> sysfs.

Seems like two lines of code, either way.

>
>> So fingerprinting it should be done
>> from seabios-test.git.  Let's divide it into two problems:
>
> That seems awkward especially since fingerprinting Just Works with
> qemu-test already (and it was only 50 lines of shell code!).
>
>> - seabios pokes qemu to get information.  We should fingerprint this
>> information to make sure different qemu version can interoperate with
>> different seabios versions (needed since we migrate the bios along with
>> the rest of the guest).  I'm guessing most of this info is from fwcfg?
>> We can easily fingerprint it like any other device.
>> - we need to make sure seabios generates the same tables when using -M.
>> Here, we're not testing a device, rather we're testing guest code, so it
>> makes sense to use a guest for this.
>>
>> However, I don't think it's even necessary.  From a quick read of the
>> manual, SMBIOS is just a set of static tables in memory which are picked
>> up using a signature.  So all we need to do is boot an empty guest, read
>> its memory, and look for the tables.
>
> Doesn't it sound a whole nicer use linux.git to parse this information
> for us in a friendly, easy to consume fashion?

Run 'dmidecode -d /path/to/memory/dump', if you must.

I don't think the qemu-test approach is bad, per se, it's just that
qtest is better.  It gives you full control over what to fingerprint and
is not reliant on Linux not changing.

>>
>> Yes; but using Linux limits you to what it exposes (of course Linux
>> exposes quite a lot, so that's mostly a non issue; but we'll have
>> counterexamples).
>
>
> Maybe.  And for those counter examples, we can drill down to qtest. 
> But just because we may need to do fancy things, it doesn't mean we
> shouldn't take the easy approach 95% of the time.

I guess we have to see how difficult it will be with qtest.  My feeling
is that it will be very easy.

>> I don't see why.  A python library to read pci config should be a couple
>> of dozens of lines long.  Then you iterate over all functions and print
>> selected registers.
>
>
> PCI enumeration is easy.  It's mapping the bars and then poking other
> sources of information that's more challenging.
>
> Everything is doable by writing a libOS but I'd rather just use Linux
> than invent a libOS just for testing.

We will have to write a libOS to test devices.  To test, say, virtio-net
with indirect buffers, you have to map BARs (or poke unmapped BARs
directly).

>>
>> You mention in the changelog replacing the APIC.  Why can't you do that?
>  
>
> I currently replace the I/O APIC which seems to be limited to 16
> IRQs.  I think MSI takes a side channel directly to the local APIC, no?

Yes, writes to the 1MB @ 0xfee00000 gets translated to MSIs.  So you
just translate them to events.

>> It looks great.  One thing I'd change is to use the qmp protocol
>> (perhaps not the monitor, just the schema/codegen).  Does something
>> prohibit this?
>
>
> Yeah, I thought about using QMP.  But events are posted in QMP which
> means that you never get an explicit ACK.  That may not be a problem
> but it's something I had in mind.
>
> It also seemed to be reasonably complicated without a clear
> advantage.  qtest isn't going to be a supported protocol so we can
> certainly change it down the road if we want to.

It's a pity not to reuse all the tooling?  Seems like self-NIH.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 11:03 AM, Avi Kivity wrote:
> On 12/29/2011 06:49 PM, Anthony Liguori wrote:
>>> However, I don't think it's even necessary.  From a quick read of the
>>> manual, SMBIOS is just a set of static tables in memory which are picked
>>> up using a signature.  So all we need to do is boot an empty guest, read
>>> its memory, and look for the tables.
>>
>> Doesn't it sound a whole nicer use linux.git to parse this information
>> for us in a friendly, easy to consume fashion?
>
> Run 'dmidecode -d /path/to/memory/dump', if you must.
>
> I don't think the qemu-test approach is bad, per se, it's just that
> qtest is better.  It gives you full control over what to fingerprint and
> is not reliant on Linux not changing.

Maybe.  But I feel a lot more comfortable asking people to write qemu-test's 
than writing low level logic to do this stuff in qtest.

Maybe over time, we'll have a good enough libOS for qtest that qemu-test won't 
be that useful anymore.  But in the short term, I think we'll get more test 
coverage by having both.

>>>
>>> You mention in the changelog replacing the APIC.  Why can't you do that?
>>
>>
>> I currently replace the I/O APIC which seems to be limited to 16
>> IRQs.  I think MSI takes a side channel directly to the local APIC, no?
>
> Yes, writes to the 1MB @ 0xfee00000 gets translated to MSIs.  So you
> just translate them to events.

Ah, okay, I wasn't thinking of that.  I'll clean qtest up and resubmit next week.

>>> It looks great.  One thing I'd change is to use the qmp protocol
>>> (perhaps not the monitor, just the schema/codegen).  Does something
>>> prohibit this?
>>
>>
>> Yeah, I thought about using QMP.  But events are posted in QMP which
>> means that you never get an explicit ACK.  That may not be a problem
>> but it's something I had in mind.
>>
>> It also seemed to be reasonably complicated without a clear
>> advantage.  qtest isn't going to be a supported protocol so we can
>> certainly change it down the road if we want to.
>
> It's a pity not to reuse all the tooling?  Seems like self-NIH.

No, it's just going to be more work to reuse things.   I was careful to make 
sure that it was possible down the road.

We have a bit of way to go before the QAPI infrastructure is generic enough to 
be used for something like this.

Regards,

Anthony Liguori

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 07:10 PM, Anthony Liguori wrote:
> On 12/29/2011 11:03 AM, Avi Kivity wrote:
>> On 12/29/2011 06:49 PM, Anthony Liguori wrote:
>>>> However, I don't think it's even necessary.  From a quick read of the
>>>> manual, SMBIOS is just a set of static tables in memory which are
>>>> picked
>>>> up using a signature.  So all we need to do is boot an empty guest,
>>>> read
>>>> its memory, and look for the tables.
>>>
>>> Doesn't it sound a whole nicer use linux.git to parse this information
>>> for us in a friendly, easy to consume fashion?
>>
>> Run 'dmidecode -d /path/to/memory/dump', if you must.
>>
>> I don't think the qemu-test approach is bad, per se, it's just that
>> qtest is better.  It gives you full control over what to fingerprint and
>> is not reliant on Linux not changing.
>
> Maybe.  But I feel a lot more comfortable asking people to write
> qemu-test's than writing low level logic to do this stuff in qtest.

Writing the libOS is definitely a job for the (sub-)maintainers.  The
work to code a unit test for patch X should be O(X).

>>> IRQs.  I think MSI takes a side channel directly to the local APIC, no?
>>
>> Yes, writes to the 1MB @ 0xfee00000 gets translated to MSIs.  So you
>> just translate them to events.
> I currently replace the I/O APIC which seems to be limited to 16
>
> Ah, okay, I wasn't thinking of that.  I'll clean qtest up and resubmit
> next week.

Great.  I'll be happy to help writing libOS.

>>> It also seemed to be reasonably complicated without a clear
>>> advantage.  qtest isn't going to be a supported protocol so we can
>>> certainly change it down the road if we want to.
>>
>> It's a pity not to reuse all the tooling?  Seems like self-NIH.
>
>
> No, it's just going to be more work to reuse things.   I was careful
> to make sure that it was possible down the road.
>
> We have a bit of way to go before the QAPI infrastructure is generic
> enough to be used for something like this.

I've started to simplify the build procedure, maybe it will help a tiny bit.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Peter Maydell]

On 29 December 2011 16:36, Avi Kivity <avi@redhat.com> wrote:
> Yes; but using Linux limits you to what it exposes (of course Linux
> exposes quite a lot, so that's mostly a non issue; but we'll have
> counterexamples).

Actually IME Linux is pretty conservative about how it uses devices.
A lot of the ARM device models have gaping holes in their emulation
which we've never needed to fix because Linux simply doesn't use
those features (eg the PL080 DMA controller which doesn't actually
implement DMA!)

I think for devices what would be particularly useful would be
if you can write a (simple) test for something at the register
level, which generates an image which you can run on the real
hardware as well as on QEMU. Then you can confirm that your test
case is correct. Otherwise the tests are just going to bake in the
same assumptions/misconceptions about what the hardware does as
the QEMU model.

My guess is that a serious attempt at tests covering all the
functionality of a device is probably approximately doubling
the effort required for a device model, incidentally. A
half-hearted attempt probably doesn't buy you much over
automating "boot the guest OS and prod its driver".

-- PMM

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 07:22 PM, Peter Maydell wrote:
> On 29 December 2011 16:36, Avi Kivity <avi@redhat.com> wrote:
> > Yes; but using Linux limits you to what it exposes (of course Linux
> > exposes quite a lot, so that's mostly a non issue; but we'll have
> > counterexamples).
>
> Actually IME Linux is pretty conservative about how it uses devices.
> A lot of the ARM device models have gaping holes in their emulation
> which we've never needed to fix because Linux simply doesn't use
> those features (eg the PL080 DMA controller which doesn't actually
> implement DMA!)

We were discussing fingerprinting, not actually driving the device.  For
that, I think we're all agreed that qtest is vastly superior to using an
OS driver.

> I think for devices what would be particularly useful would be
> if you can write a (simple) test for something at the register
> level, which generates an image which you can run on the real
> hardware as well as on QEMU. Then you can confirm that your test
> case is correct. Otherwise the tests are just going to bake in the
> same assumptions/misconceptions about what the hardware does as
> the QEMU model.

That is exactly qtest.

> My guess is that a serious attempt at tests covering all the
> functionality of a device is probably approximately doubling
> the effort required for a device model, incidentally. A
> half-hearted attempt probably doesn't buy you much over
> automating "boot the guest OS and prod its driver".

Agreed.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Peter Maydell]

On 29 December 2011 17:26, Avi Kivity <avi@redhat.com> wrote:
> On 12/29/2011 07:22 PM, Peter Maydell wrote:
>> I think for devices what would be particularly useful would be
>> if you can write a (simple) test for something at the register
>> level, which generates an image which you can run on the real
>> hardware as well as on QEMU. Then you can confirm that your test
>> case is correct. Otherwise the tests are just going to bake in the
>> same assumptions/misconceptions about what the hardware does as
>> the QEMU model.
>
> That is exactly qtest.

Am I missing something? I looked at the qtest patches and they seem
to work by just prodding the device through qemu function calls,
not by running a guest CPU binary that prods the device.

-- PMM

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 07:36 PM, Peter Maydell wrote:
> On 29 December 2011 17:26, Avi Kivity <avi@redhat.com> wrote:
> > On 12/29/2011 07:22 PM, Peter Maydell wrote:
> >> I think for devices what would be particularly useful would be
> >> if you can write a (simple) test for something at the register
> >> level, which generates an image which you can run on the real
> >> hardware as well as on QEMU. Then you can confirm that your test
> >> case is correct. Otherwise the tests are just going to bake in the
> >> same assumptions/misconceptions about what the hardware does as
> >> the QEMU model.
> >
> > That is exactly qtest.
>
> Am I missing something? I looked at the qtest patches and they seem
> to work by just prodding the device through qemu function calls,
> not by running a guest CPU binary that prods the device.

Sorry, I misread your post.

I guess we can point a qtest test at real hardware too, if something
like vfio is available to provide access to registers and to hook
interrupts.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Peter Maydell]

On 29 December 2011 17:26, Avi Kivity <avi@redhat.com> wrote:
> On 12/29/2011 07:22 PM, Peter Maydell wrote:
>> My guess is that a serious attempt at tests covering all the
>> functionality of a device is probably approximately doubling
>> the effort required for a device model, incidentally. A
>> half-hearted attempt probably doesn't buy you much over
>> automating "boot the guest OS and prod its driver".
>
> Agreed.

The next obvious question is: are we going to make a serious attempt?
(For instance, in a hypothetical tests-required world, would we
tell those nice folks from Samsung "no you can't land your
Exynos patches unless you write 9000+ lines of test cases" ?
I suspect that if we set the bar for new board and device models
that high then the result will largely be that we don't in fact
get new board or device models.)

-- PMM

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 07:49 PM, Peter Maydell wrote:
> On 29 December 2011 17:26, Avi Kivity <avi@redhat.com> wrote:
> > On 12/29/2011 07:22 PM, Peter Maydell wrote:
> >> My guess is that a serious attempt at tests covering all the
> >> functionality of a device is probably approximately doubling
> >> the effort required for a device model, incidentally. A
> >> half-hearted attempt probably doesn't buy you much over
> >> automating "boot the guest OS and prod its driver".
> >
> > Agreed.
>
> The next obvious question is: are we going to make a serious attempt?
> (For instance, in a hypothetical tests-required world, would we
> tell those nice folks from Samsung "no you can't land your
> Exynos patches unless you write 9000+ lines of test cases" ?

Yes.

> I suspect that if we set the bar for new board and device models
> that high then the result will largely be that we don't in fact
> get new board or device models.)

If just doubles the effort, I disagree.

Even if that turns out to be the case, it's fine.  Better to have a few
good devices than dozens of bad ones.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Peter Maydell]

On 29 December 2011 17:56, Avi Kivity <avi@redhat.com> wrote:
> On 12/29/2011 07:49 PM, Peter Maydell wrote:
>> I suspect that if we set the bar for new board and device models
>> that high then the result will largely be that we don't in fact
>> get new board or device models.)
>
> If just doubles the effort, I disagree.

If you turn a six month project into a twelve month project
I suspect that will push it beyond the "worth doing" limit
in many cases. (Or more likely, people just won't bother to
send their models upstream, and we'll have more forks out there.)

> Even if that turns out to be the case, it's fine.  Better to
> have a few good devices than dozens of bad ones.

This is easier to say on the x86 side of the fence, since
most of the devices you need are already in the codebase and
nobody is going to throw them out again if tests don't get
written for them :-)

There are lots of things where I'd rather have a "tested by
booting a guest OS" implementation than none at all (like audio
support for the versatile/etc boards, which went in recently) or
TrustZone support (not in yet but may be along later). At least
then we have something that works for most people and something
we can fix bugs in, rather than a gaping hole in capability.

-- PMM

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 11:10 PM, Peter Maydell wrote:

>
> > Even if that turns out to be the case, it's fine.  Better to
> > have a few good devices than dozens of bad ones.
>
> This is easier to say on the x86 side of the fence, since
> most of the devices you need are already in the codebase and
> nobody is going to throw them out again if tests don't get
> written for them :-)
>
> There are lots of things where I'd rather have a "tested by
> booting a guest OS" implementation than none at all (like audio
> support for the versatile/etc boards, which went in recently) or
> TrustZone support (not in yet but may be along later). At least
> then we have something that works for most people and something
> we can fix bugs in, rather than a gaping hole in capability.
>

We can have different criteria for different parts of the tree. 
Undesirable, but tradeoffs have to be made.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 11:49 AM, Peter Maydell wrote:
> On 29 December 2011 17:26, Avi Kivity<avi@redhat.com>  wrote:
>> On 12/29/2011 07:22 PM, Peter Maydell wrote:
>>> My guess is that a serious attempt at tests covering all the
>>> functionality of a device is probably approximately doubling
>>> the effort required for a device model, incidentally. A
>>> half-hearted attempt probably doesn't buy you much over
>>> automating "boot the guest OS and prod its driver".
>>
>> Agreed.
>
> The next obvious question is: are we going to make a serious attempt?
> (For instance, in a hypothetical tests-required world, would we
> tell those nice folks from Samsung "no you can't land your
> Exynos patches unless you write 9000+ lines of test cases" ?

The virtio-serial test case I posted was 50 lines in qemu-test.  The 
virtio-serial driver is about ~1500 LOC.  That's about 3%.

I would expect that we at least have some sort of test that could verify that 
the Exynos platform more or less worked as expected.  If that was just booting a 
Linux kernel, that would be fine by me.

> I suspect that if we set the bar for new board and device models
> that high then the result will largely be that we don't in fact
> get new board or device models.)

This is why the barrier needs to be as low as possible.

Regards,

Anthony Liguori

> -- PMM
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Peter Maydell]

On 29 December 2011 18:35, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On 12/29/2011 11:49 AM, Peter Maydell wrote:
>> The next obvious question is: are we going to make a serious attempt?
>> (For instance, in a hypothetical tests-required world, would we
>> tell those nice folks from Samsung "no you can't land your
>> Exynos patches unless you write 9000+ lines of test cases" ?
>
> The virtio-serial test case I posted was 50 lines in qemu-test.  The
> virtio-serial driver is about ~1500 LOC.  That's about 3%.

This just means it's testing only a fraction of what virtio-serial
actually implements (as you note yourself in the comments for the
test case). For serious coverage you'd also need to cover reading(!),
larger quantities of data than single lines, what happens when one end
feeds in data but the other end isn't reading, vmstate save/load,
multiple simultaneous ports, behaviour on close-and-reopen,
whether it works on bigendian targets, benchmarking to identify
possible performance regressions, etc etc. I think that by the time
you've done all that you'll be closer to 1500 lines than 50.

> I would expect that we at least have some sort of test that could verify
> that the Exynos platform more or less worked as expected.  If that was just
> booting a Linux kernel, that would be fine by me.

Yes. There's a large range between "no tests required at all" (essentially
what we have now) and "an equivalent level of device testing to what you
would carry out before sending your hardware design out to be fabbed into
silicon" (which I hope we'd all agree would be ludicrously high for QEMU);
I'm trying to establish where we're attempting to set our bar.

I agree that we'd get a lot of bang-for-the-buck out of basic automated
"boot the guest and prod it" testing that covered most of our platforms.

-- PMM

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Blue Swirl]

On Thu, Dec 29, 2011 at 19:04, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 29 December 2011 18:35, Anthony Liguori <anthony@codemonkey.ws> wrote:
>> On 12/29/2011 11:49 AM, Peter Maydell wrote:
>>> The next obvious question is: are we going to make a serious attempt?
>>> (For instance, in a hypothetical tests-required world, would we
>>> tell those nice folks from Samsung "no you can't land your
>>> Exynos patches unless you write 9000+ lines of test cases" ?
>>
>> The virtio-serial test case I posted was 50 lines in qemu-test.  The
>> virtio-serial driver is about ~1500 LOC.  That's about 3%.
>
> This just means it's testing only a fraction of what virtio-serial
> actually implements (as you note yourself in the comments for the
> test case). For serious coverage you'd also need to cover reading(!),
> larger quantities of data than single lines, what happens when one end
> feeds in data but the other end isn't reading, vmstate save/load,
> multiple simultaneous ports, behaviour on close-and-reopen,
> whether it works on bigendian targets, benchmarking to identify
> possible performance regressions, etc etc. I think that by the time
> you've done all that you'll be closer to 1500 lines than 50.

The coverage point of view is interesting. I'd expect the coverage
rate to rise as one moves from qtest via qemu-test to autotest.
However, I'd also expect that having qtest and qemu-test would both
increase chances that patches also get test coverage.

Gathering test coverage data could be a start.

>> I would expect that we at least have some sort of test that could verify
>> that the Exynos platform more or less worked as expected.  If that was just
>> booting a Linux kernel, that would be fine by me.
>
> Yes. There's a large range between "no tests required at all" (essentially
> what we have now) and "an equivalent level of device testing to what you
> would carry out before sending your hardware design out to be fabbed into
> silicon" (which I hope we'd all agree would be ludicrously high for QEMU);
> I'm trying to establish where we're attempting to set our bar.
>
> I agree that we'd get a lot of bang-for-the-buck out of basic automated
> "boot the guest and prod it" testing that covered most of our platforms.

That may also be too simplistic. For example, sparc-test (boot Sparc32
Linux with busybox) does not catch doubleword access problems or
certain bugs in floating point point emulation. For those, I need a
boot from an install CD which starts X.

Overall, I'd think that all of qtest, qemu-test and autotest have
their place. If one brilliant day autotest becomes so lean and mean
that qemu-test is unnecessary, it can be deprecated.

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 01:04 PM, Peter Maydell wrote:
> On 29 December 2011 18:35, Anthony Liguori<anthony@codemonkey.ws>  wrote:
>> On 12/29/2011 11:49 AM, Peter Maydell wrote:
>>> The next obvious question is: are we going to make a serious attempt?
>>> (For instance, in a hypothetical tests-required world, would we
>>> tell those nice folks from Samsung "no you can't land your
>>> Exynos patches unless you write 9000+ lines of test cases" ?
>>
>> The virtio-serial test case I posted was 50 lines in qemu-test.  The
>> virtio-serial driver is about ~1500 LOC.  That's about 3%.
>
> This just means it's testing only a fraction of what virtio-serial
> actually implements (as you note yourself in the comments for the
> test case). For serious coverage you'd also need to cover reading(!),
> larger quantities of data than single lines, what happens when one end
> feeds in data but the other end isn't reading, vmstate save/load,
> multiple simultaneous ports, behaviour on close-and-reopen,
> whether it works on bigendian targets, benchmarking to identify
> possible performance regressions, etc etc. I think that by the time
> you've done all that you'll be closer to 1500 lines than 50.

At the end of the day, the right level depth of tests is something we'll need to 
work out.  Realistically, the level of test coverage is directly proportional to 
how upset will get when someone breaks that particular function.

But 0% test coverage is absolutely not enough.  So the first step is to get an 
infrastructure together that we can all live with and start writing tests.

Regards,

Anthony Liguori

>
>> I would expect that we at least have some sort of test that could verify
>> that the Exynos platform more or less worked as expected.  If that was just
>> booting a Linux kernel, that would be fine by me.
>
> Yes. There's a large range between "no tests required at all" (essentially
> what we have now) and "an equivalent level of device testing to what you
> would carry out before sending your hardware design out to be fabbed into
> silicon" (which I hope we'd all agree would be ludicrously high for QEMU);
> I'm trying to establish where we're attempting to set our bar.
>
> I agree that we'd get a lot of bang-for-the-buck out of basic automated
> "boot the guest and prod it" testing that covered most of our platforms.
>
> -- PMM
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Peter Maydell]

On 29 December 2011 21:46, Anthony Liguori <anthony@codemonkey.ws> wrote:
> But 0% test coverage is absolutely not enough.  So the first step is to get
> an infrastructure together that we can all live with and start writing
> tests.

Agreed.

How does your framework deal with non-x86 targets? Finding and
installing a working crosscompiler can be painful, especially if
you wanted to crosscompile userspace rather than just the kernel...

-- PMM

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 04:10 PM, Peter Maydell wrote:
> On 29 December 2011 21:46, Anthony Liguori<anthony@codemonkey.ws>  wrote:
>> But 0% test coverage is absolutely not enough.  So the first step is to get
>> an infrastructure together that we can all live with and start writing
>> tests.
>
> Agreed.
>
> How does your framework deal with non-x86 targets?

http://git.qemu.org/qemu-jeos.git

I've already got ppc32 support working.  Adding a new arch is just a matter of 
adding a kernel config and uClibc config to configs/

 > Finding and
> installing a working crosscompiler can be painful, especially if
> you wanted to crosscompile userspace rather than just the kernel...

I spent a couple days researching what to do here and ended up settling on:

1) build binutils for desired target

2) build GCC using (1) as a cross compiler.  This is a limited form of GCC (no 
thread support) targeted as uClibc

3) build kernel using (2) and installer headers into our new sysroot

4) build uClibc using (2) and (3)

5) build busybox using (2) and (4)

The whole process takes about 30 minutes on my laptop using -j1 which isn't all 
that bad.  It's certainly faster than doing a distro install with TCG :-)

qemu-jeos.git contains git submodules for binutils, GCC, linux, uClibc, and 
busybox plus miscellaneous scripts needed to make a working initramfs.

Once we get more ARCHs working, I'll add a cronjob to qemu.org to build weekly 
binary packages so that for most people, all you have to do is grab an RPM/DEB 
with prebuilt binaries.

Regards,

Anthony Liguori

>
> -- PMM
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Andreas Färber]

Am 29.12.2011 23:30, schrieb Anthony Liguori:
> On 12/29/2011 04:10 PM, Peter Maydell wrote:
>> How does your framework deal with non-x86 targets?
> 
> http://git.qemu.org/qemu-jeos.git
> 
> I've already got ppc32 support working.  Adding a new arch is just a
> matter of adding a kernel config and uClibc config to configs/
> 
>> Finding and
>> installing a working crosscompiler can be painful, especially if
>> you wanted to crosscompile userspace rather than just the kernel...
> 
> I spent a couple days researching what to do here and ended up settling on:
> 
> 1) build binutils for desired target
> 
> 2) build GCC using (1) as a cross compiler.  This is a limited form of
> GCC (no thread support) targeted as uClibc
> 
> 3) build kernel using (2) and installer headers into our new sysroot
> 
> 4) build uClibc using (2) and (3)
> 
> 5) build busybox using (2) and (4)
> 
> The whole process takes about 30 minutes on my laptop using -j1 which
> isn't all that bad.  It's certainly faster than doing a distro install
> with TCG :-)
> 
> qemu-jeos.git contains git submodules for binutils, GCC, linux, uClibc,
> and busybox plus miscellaneous scripts needed to make a working initramfs.

"One ring to rule them all ... and in the darkness bind them"? ;)

Seriously, what you describe may work for mainstream targets. But for
new targets the trouble starts with 1) already: which binutils? The
latest stable may not work for all architectures yet, so a generic
submodule approach is doomed to fail.

Will uClibc work for all targets? There's glibc, eglibc, newlib, ...

Not all targets have a softmmu at this time: unicore32
(Fwiw this also means no testing outside Linux hosts.)

There's no requirement that the Linux kernel must have been ported to a
QEMU target yet or that it is still in mainline or that what is in
mainline is complete enough for our testing.

So, I'm fine if you come up with a cool testing framework, with or
without Q in the name. Just please don't imply from testing two targets
that this is a one-size-fits-all solution for all current and future
targets. The qtest approach seems more promising in that regard.

> Once we get more ARCHs working, I'll add a cronjob to qemu.org to build
> weekly binary packages so that for most people, all you have to do is
> grab an RPM/DEB with prebuilt binaries.

I have a feeling OBS, Koji, etc. are more suited for this than trying to
set up a build service of your own. Let's leave qrpm for April 1st.

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/30/2011 09:43 AM, Andreas Färber wrote:
> Am 29.12.2011 23:30, schrieb Anthony Liguori:
>> On 12/29/2011 04:10 PM, Peter Maydell wrote:
>>> How does your framework deal with non-x86 targets?
>>
>> http://git.qemu.org/qemu-jeos.git
>>
>> I've already got ppc32 support working.  Adding a new arch is just a
>> matter of adding a kernel config and uClibc config to configs/
>>
>>> Finding and
>>> installing a working crosscompiler can be painful, especially if
>>> you wanted to crosscompile userspace rather than just the kernel...
>>
>> I spent a couple days researching what to do here and ended up settling on:
>>
>> 1) build binutils for desired target
>>
>> 2) build GCC using (1) as a cross compiler.  This is a limited form of
>> GCC (no thread support) targeted as uClibc
>>
>> 3) build kernel using (2) and installer headers into our new sysroot
>>
>> 4) build uClibc using (2) and (3)
>>
>> 5) build busybox using (2) and (4)
>>
>> The whole process takes about 30 minutes on my laptop using -j1 which
>> isn't all that bad.  It's certainly faster than doing a distro install
>> with TCG :-)
>>
>> qemu-jeos.git contains git submodules for binutils, GCC, linux, uClibc,
>> and busybox plus miscellaneous scripts needed to make a working initramfs.
>
> "One ring to rule them all ... and in the darkness bind them"? ;)
>
> Seriously, what you describe may work for mainstream targets. But for
> new targets the trouble starts with 1) already: which binutils? The
> latest stable may not work for all architectures yet, so a generic
> submodule approach is doomed to fail.

Let's not speculate here..  There's already an existence proof that this 
approach works for most of the targets we support (Aboriginal Linux).

> Will uClibc work for all targets? There's glibc, eglibc, newlib, ...
>
> Not all targets have a softmmu at this time: unicore32
> (Fwiw this also means no testing outside Linux hosts.)
>
> There's no requirement that the Linux kernel must have been ported to a
> QEMU target yet or that it is still in mainline or that what is in
> mainline is complete enough for our testing.

Yes, but this just becomes the exception to prove the rule.

Regards,

Anthony Liguori

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Andreas Färber]

Am 03.01.2012 14:42, schrieb Anthony Liguori:
> On 12/30/2011 09:43 AM, Andreas Färber wrote:
>> Am 29.12.2011 23:30, schrieb Anthony Liguori:
>>> 1) build binutils for desired target
>>>
>>> 2) build GCC using (1) as a cross compiler.  This is a limited form of
>>> GCC (no thread support) targeted as uClibc
>>>
>>> 3) build kernel using (2) and installer headers into our new sysroot
>>>
>>> 4) build uClibc using (2) and (3)
>>>
>>> 5) build busybox using (2) and (4)
>>>
>>> The whole process takes about 30 minutes on my laptop using -j1 which
>>> isn't all that bad.  It's certainly faster than doing a distro install
>>> with TCG :-)
>>>
>>> qemu-jeos.git contains git submodules for binutils, GCC, linux, uClibc,
>>> and busybox plus miscellaneous scripts needed to make a working
>>> initramfs.
>>
>> "One ring to rule them all ... and in the darkness bind them"? ;)
>>
>> Seriously, what you describe may work for mainstream targets. But for
>> new targets the trouble starts with 1) already: which binutils? The
>> latest stable may not work for all architectures yet, so a generic
>> submodule approach is doomed to fail.
> 
> Let's not speculate here..  There's already an existence proof that this
> approach works for most of the targets we support (Aboriginal Linux).
> 
>> Will uClibc work for all targets? There's glibc, eglibc, newlib, ...
>>
>> Not all targets have a softmmu at this time: unicore32
>> (Fwiw this also means no testing outside Linux hosts.)
>>
>> There's no requirement that the Linux kernel must have been ported to a
>> QEMU target yet or that it is still in mainline or that what is in
>> mainline is complete enough for our testing.
> 
> Yes, but this just becomes the exception to prove the rule.

That depends. The way I understood you (and that I would like to get
clarified on the call) is that you wanted to require qemu-test test
cases for patches. That would require it to work for everything without
exception.

If however you just want this as an optional "I am more confident of my
changes if they work with qemu-test on x86, ppc, myfavorite", sort of an
alternative to downloading test images from qemu.org and running them
manually, then of course it doesn't really matter if target xyz or
feature foo cannot be tested in the same automated manner.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Lucas Meneghel Rodrigues]

On 12/29/2011 05:04 PM, Peter Maydell wrote:
> On 29 December 2011 18:35, Anthony Liguori<anthony@codemonkey.ws>  wrote:
>> On 12/29/2011 11:49 AM, Peter Maydell wrote:
>>> The next obvious question is: are we going to make a serious attempt?
>>> (For instance, in a hypothetical tests-required world, would we
>>> tell those nice folks from Samsung "no you can't land your
>>> Exynos patches unless you write 9000+ lines of test cases" ?
>>
>> The virtio-serial test case I posted was 50 lines in qemu-test.  The
>> virtio-serial driver is about ~1500 LOC.  That's about 3%.
>
> This just means it's testing only a fraction of what virtio-serial
> actually implements (as you note yourself in the comments for the
> test case). For serious coverage you'd also need to cover reading(!),
> larger quantities of data than single lines, what happens when one end
> feeds in data but the other end isn't reading, vmstate save/load,
> multiple simultaneous ports, behaviour on close-and-reopen,
> whether it works on bigendian targets, benchmarking to identify
> possible performance regressions, etc etc. I think that by the time
> you've done all that you'll be closer to 1500 lines than 50.

Agreed. There's a reason why virtio-serial testing in kvm autotest is 
2.1 k LOC for the test itself and 0.9 k LOC for the auxiliary guest 
script. Most of the scenarios you've described are covered on kvm 
autotest version.

>> I would expect that we at least have some sort of test that could verify
>> that the Exynos platform more or less worked as expected.  If that was just
>> booting a Linux kernel, that would be fine by me.
>
> Yes. There's a large range between "no tests required at all" (essentially
> what we have now) and "an equivalent level of device testing to what you
> would carry out before sending your hardware design out to be fabbed into
> silicon" (which I hope we'd all agree would be ludicrously high for QEMU);
> I'm trying to establish where we're attempting to set our bar.
>
> I agree that we'd get a lot of bang-for-the-buck out of basic automated
> "boot the guest and prod it" testing that covered most of our platforms.

And that sort of automated testing can be done right now with KVM 
autotest. Granted, I never had time to make it work with ARM, but if 
people are interested, we are happily accepting patches :)

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 11:22 AM, Peter Maydell wrote:
> On 29 December 2011 16:36, Avi Kivity<avi@redhat.com>  wrote:
>> Yes; but using Linux limits you to what it exposes (of course Linux
>> exposes quite a lot, so that's mostly a non issue; but we'll have
>> counterexamples).
>
> Actually IME Linux is pretty conservative about how it uses devices.
> A lot of the ARM device models have gaping holes in their emulation
> which we've never needed to fix because Linux simply doesn't use
> those features (eg the PL080 DMA controller which doesn't actually
> implement DMA!)
>
> I think for devices what would be particularly useful would be
> if you can write a (simple) test for something at the register
> level, which generates an image which you can run on the real
> hardware as well as on QEMU. Then you can confirm that your test
> case is correct. Otherwise the tests are just going to bake in the
> same assumptions/misconceptions about what the hardware does as
> the QEMU model.

I think that could be useful, but it orthogonal to what we need for Test Driven 
Development.

The problem today is that it is very hard to make broad changes in QEMU simply 
because it's so difficult to know whether you're breaking another target.  Look 
at the challenges around merging the memory API.

We need a simple way that a developer can sanity check that they've not broken 
other targets and all of the various devices that we support.

>
> My guess is that a serious attempt at tests covering all the
> functionality of a device is probably approximately doubling
> the effort required for a device model, incidentally. A
> half-hearted attempt probably doesn't buy you much over
> automating "boot the guest OS and prod its driver".

Right, but "boot the guest OS and prod its driver" is very useful particularly 
if it can cover all of the various targets we support.

I don't think we should focus much on qtest for non-x86 targets.  I mean, if you 
are interested in it for ARM, fantastic, but I don't think we would mandate it.

For x86, it's reasonable to have stricter requirements on testing, particularly 
for new devices.

FWIW, I expect virtio-scsi to be the first guinea pig here...  I believe Stefan 
has already started looking at writing some qtest based tests for virtio-scsi.

Regards,

Anthony Liguori

> -- PMM
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Andreas Färber]

Am 29.12.2011 19:33, schrieb Anthony Liguori:
> I don't think we should focus much on qtest for non-x86 targets.  I
> mean, if you are interested in it for ARM, fantastic, but I don't think
> we would mandate it.

I'm actually very interested in having a qtest framework for non-x86 for

a) unit testing TCG (execute instruction, check register/memory results)
b) unit testing device emulation

Your v2 qtest RFC is still missing the libqtest C implementation. We
code in C so I'd like to test b) in C, not Python or some random other
language that someone happens to like for whatever reason.

Andreas

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Paolo Bonzini]

On 12/29/2011 07:33 PM, Anthony Liguori wrote:
>
> FWIW, I expect virtio-scsi to be the first guinea pig here...  I believe
> Stefan has already started looking at writing some qtest based tests for
> virtio-scsi.

I'm curious, because a SCSI HBA makes it as hard as it can be to write 
unit tests.  Pretty much everything you do will test the interactions 
between the SCSI devices and the block layer, more than the HBA itself. 
  This is especially for such a simple HBA; unit testing is much more 
"interesting" for something like the LSI device.

So, to test the HBA you need write either a mock SCSI device, or a mock 
block device.  (I hate mocks).  Now that the client is asynchronous, NBD 
is a pretty effective way to write a mock block device, though not the 
simplest.

Paolo

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Stefan Hajnoczi]

On Mon, Jan 02, 2012 at 03:07:08PM +0100, Paolo Bonzini wrote:
> On 12/29/2011 07:33 PM, Anthony Liguori wrote:
> >
> >FWIW, I expect virtio-scsi to be the first guinea pig here...  I believe
> >Stefan has already started looking at writing some qtest based tests for
> >virtio-scsi.
> 
> I'm curious, because a SCSI HBA makes it as hard as it can be to
> write unit tests.  Pretty much everything you do will test the
> interactions between the SCSI devices and the block layer, more than
> the HBA itself.  This is especially for such a simple HBA; unit
> testing is much more "interesting" for something like the LSI
> device.
> 
> So, to test the HBA you need write either a mock SCSI device, or a
> mock block device.  (I hate mocks).  Now that the client is
> asynchronous, NBD is a pretty effective way to write a mock block
> device, though not the simplest.

I haven't started yet but I am planning to write qtest virtio PCI tests
and will definitely be taking a look at virtio-scsi.

We don't necessarily need mock devices, just really simply temporary raw
files or similar.  On the other hand mock devices can be a good solution
for programmatically verifying that our intended action was performed on
the host device.

Stefan

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Paolo Bonzini]

On 01/03/2012 09:19 AM, Stefan Hajnoczi wrote:
> I haven't started yet but I am planning to write qtest virtio PCI tests
> and will definitely be taking a look at virtio-scsi.
>
> We don't necessarily need mock devices, just really simply temporary raw
> files or similar.  On the other hand mock devices can be a good solution
> for programmatically verifying that our intended action was performed on
> the host device.

Mocks, no matter whether they're mock SCSI devices or mock block 
devices, are needed if you want to test TMFs.  Cancellation was sorta 
broken by the coroutine conversion, but I think it can be fixed.

Paolo

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/28/2011 09:01 AM, Avi Kivity wrote:
> On 12/28/2011 04:27 PM, Anthony Liguori wrote:
>>> Maybe I've used the wrong wording. I got the feeling that, besides
>>> testing qemu
>>> the way you need it, keeping qemu-test simple was really important.
>>
>>
>> Simple is always important.  In the case of qemu-test, there are some
>> important trade offs made to keep things simple and fast.  It doesn't
>> try to work with arbitrary guests which means the tests need to handle
>> only one environment.  The guest is pre-made to have exactly what is
>> needed for the tests so there is no setup required.  The guest is as
>> small as possible such that the test can run as quickly as possible.
>
> In fact using linux as a guest negates that.  First of all, which linux
> version? if it's fixed, you'll eventually miss functionality and need to
> migrate.  If it keeps changing, so does your test, and it will keep
> breaking.

The kernel is a git submodule so it's a very specific version.  Yes, we may need 
to bump the version down the road and obviously, if we have to change any tests 
in the process, we can.

>
> Using Linux as a guest allows you to do system tests (ping was an
> example) but doesn't allow you to do unit tests (test regression where
> if this bit in that register was set, but if them bits in thar registers
> were clear, then the packet would be encoded with ROT-26 before being
> sent out).
>
> I think qemu-test needs to use its own drivers which allow full control
> over what you do with the hardware.  Otherwise, a regression that only
> shows up in non-Linux guests will not be testable with qemu-test.

I think you're advocating for qtest.  This is another important part of my 
testing strategy.  I haven't received a lot of input on that RFC...

http://mid.gmane.org/1322765012-3164-1-git-send-email-aliguori@us.ibm.com

But there's certain things that I still consider to be unit testing (like basic 
networking tests) that I don't want to have to write with qtest.  I'm not up for 
writing a TCP/IP stack in Python...

>> There is no need to have a single tool that meets every possible
>> need.  In fact, the Unix tradition is to have separate single purposed
>> tools.
>
> Those single purpose tools, if successful, tend to grow more purposes
> (cf. qemu), and if unsuccessful, tend to lose purpose.
>
>> Having two test tools it not a bad thing provided that the overlap
>> isn't significant.
>
> This is important, if the boundary isn't clear, then it will grow more
> fuzzy in time.
>
> I suggest the following:
>
>   - qemu-test: qemu unit tests
>   - kvm-unit-tests: kvm unit tests
>   - kvm-autotest: unit test drivers + system tests

I would counter with:

- gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest)
- qtest: low level, single device functional tests
- kvm-unit-tests: low level, instruction-set level functional tests
- qemu-test: higher level functional/coverage tests (multiple device
              interaction)
- kvm-autotest: unit/functional test drivers + acceptance testing

Note that the line I'm drawing is acceptance vs. functional testing, not unit 
vs. integration testing.  Technically, our unit tests are things like 
test-visitor.  Everything else is an integration test.

But the separation between kvm-autotest is acceptance testing vs. functional 
testing.

Acceptance testing is, "does Windows boot", "can I create three virtio-serial 
devices".

Obviously, part of acceptance testing is, "does this set of functional tests pass".

>> We shouldn't be discussing whether it's possible to merge the two
>> tools, but rather what the technical benefits of doing so would be.
>>
>> Since at this point, there is almost no overlap between the two, I
>> don't see any actual technical benefit to merging them.  I see benefit
>> to autotest executing qemu-test, of course.
>
> I'd say that running a ping test is a weak version of kvm-autotest's
> system tests.

Consider the Linux kernel to be our library of functionality to write our unit 
tests.  I don't want to write a TCP/IP stack.  We aren't just grabbing a random 
distro kernel.  We're building one from scratch configured in a specific way.

In theory, we could even host the kernel source on git.qemu.org and fork it to 
add more interesting things in the kernel (although I'd prefer not to do this, 
obviously).

>  Running a synthetic test that pokes values into memory
> and mmio and sees a packet coming out is a unit test (the latter can in
> fact be executed without a guest at all, just a table driving calls to
> the memory and irq APIs).
>
>
>> Just putting the code in kvm-autotest.git in a directory doesn't make
>> sense to me.  Beyond the lack of a technical reason to do so,
>> logistically, it makes it harder for me to ask people to submit test
>> cases with a patch series if I can't actually apply the test case when
>> I'm applying the patches to qemu.git.
>>
>> If qemu-test didn't use large submodules (like linux.git), I would
>> have made qemu-test part of qemu.git.  As far as I'm concerned,
>> qemu-test.git is just an extension to qemu.git.
>
> Why not just put it in qemu.git?

Actually, having thought a bit about it, I think we can move qemu-test into 
qemu.git and just build the initramfs in a separate repository.  That will make 
it easier to host binary packages of the later on qemu.org.

Regards,

Anthony Liguori

>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/28/2011 06:42 PM, Anthony Liguori wrote:
>> In fact using linux as a guest negates that.  First of all, which linux
>> version? if it's fixed, you'll eventually miss functionality and need to
>> migrate.  If it keeps changing, so does your test, and it will keep
>> breaking.
>
>
> The kernel is a git submodule so it's a very specific version.  Yes,
> we may need to bump the version down the road and obviously, if we
> have to change any tests in the process, we can.

Having a full linux source as part of the build process detracts
somewhat from the advantages here.

>
>>
>> Using Linux as a guest allows you to do system tests (ping was an
>> example) but doesn't allow you to do unit tests (test regression where
>> if this bit in that register was set, but if them bits in thar registers
>> were clear, then the packet would be encoded with ROT-26 before being
>> sent out).
>>
>> I think qemu-test needs to use its own drivers which allow full control
>> over what you do with the hardware.  Otherwise, a regression that only
>> shows up in non-Linux guests will not be testable with qemu-test.
>
> I think you're advocating for qtest.  This is another important part
> of my testing strategy.  I haven't received a lot of input on that RFC...
>
> http://mid.gmane.org/1322765012-3164-1-git-send-email-aliguori@us.ibm.com
>
> But there's certain things that I still consider to be unit testing
> (like basic networking tests) that I don't want to have to write with
> qtest.  I'm not up for writing a TCP/IP stack in Python...

A ping test is not a unit test.

The ping test is already covered by kvm-autotest; just set up a config
that runs just that; after the initial run you'll have a guest installed
so it'll be quick.  If we have a DSL guest it'll even be very quick.

To test the various NIC emulations, you don't need a full TCP stack,
just like you didn't need to write an NTP implementation for qtest's rtc
test.  Instead you just poke values until it sends out a packet.  If you
want to test virtio-net with both direct and indirect buffers, you can
only do that with a unit test, you can't do it using a full linux guest
since it has its own ideas of when to use indirects and when to avoid
them (for example, it may choose to always avoid them).

>> I suggest the following:
>>
>>   - qemu-test: qemu unit tests
>>   - kvm-unit-tests: kvm unit tests
>>   - kvm-autotest: unit test drivers + system tests
>
>
> I would counter with:
>
> - gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest)
> - qtest: low level, single device functional tests
> - kvm-unit-tests: low level, instruction-set level functional tests

Not really.  kvm-unit-tests tests things specific to kvm.

> - qemu-test: higher level functional/coverage tests (multiple device
>              interaction)
> - kvm-autotest: unit/functional test drivers + acceptance testing
>
> Note that the line I'm drawing is acceptance vs. functional testing,
> not unit vs. integration testing.  Technically, our unit tests are
> things like test-visitor.  Everything else is an integration test.
>
> But the separation between kvm-autotest is acceptance testing vs.
> functional testing.
>
> Acceptance testing is, "does Windows boot", "can I create three
> virtio-serial devices".
>
> Obviously, part of acceptance testing is, "does this set of functional
> tests pass".

Seems like a very blurry line.  Especially as the functional test is
weaker than either qtest and kvm-autotest.  I now have to agree with the
others that it's duplicate functionality.  Does it really matter whether
you're creating an image by compiling Linux and assembling and
initramfs, or by downloading Fedora.iso and installing it?  It's testing
exactly the same thing, guest boot and functionality.

Would you add live migration testing to qemu-test?  If yes, you're
duplicating some more.  If not, you're not doing functional or coverage
tests for that functionality.

>
>>> We shouldn't be discussing whether it's possible to merge the two
>>> tools, but rather what the technical benefits of doing so would be.
>>>
>>> Since at this point, there is almost no overlap between the two, I
>>> don't see any actual technical benefit to merging them.  I see benefit
>>> to autotest executing qemu-test, of course.
>>
>> I'd say that running a ping test is a weak version of kvm-autotest's
>> system tests.
>
> Consider the Linux kernel to be our library of functionality to write
> our unit tests.  

I was about to say the same thing, but with a negative implication. 
Using Linux restricts your tests to what Linux does with the devices.

> I don't want to write a TCP/IP stack.  We aren't just grabbing a
> random distro kernel.  We're building one from scratch configured in a
> specific way.

How does that help?

> In theory, we could even host the kernel source on git.qemu.org and
> fork it to add more interesting things in the kernel (although I'd
> prefer not to do this, obviously).

That way lies madness, though it could be argued that it lies all around us.


-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Dor Laor]

On 12/28/2011 07:21 PM, Avi Kivity wrote:
>> I think you're advocating for qtest.  This is another important part
>> >  of my testing strategy.  I haven't received a lot of input on that RFC...
>> >
>> >  http://mid.gmane.org/1322765012-3164-1-git-send-email-aliguori@us.ibm.com
>> >
>> >  But there's certain things that I still consider to be unit testing
>> >  (like basic networking tests) that I don't want to have to write with
>> >  qtest.  I'm not up for writing a TCP/IP stack in Python...
> A ping test is not a unit test.
>
> The ping test is already covered by kvm-autotest; just set up a config
> that runs just that; after the initial run you'll have a guest installed
> so it'll be quick.  If we have a DSL guest it'll even be very quick.
>
> To test the various NIC emulations, you don't need a full TCP stack,
> just like you didn't need to write an NTP implementation for qtest's rtc
> test.  Instead you just poke values until it sends out a packet.  If you
> want to test virtio-net with both direct and indirect buffers, you can
> only do that with a unit test, you can't do it using a full linux guest
> since it has its own ideas of when to use indirects and when to avoid
> them (for example, it may choose to always avoid them).
>
>>> >>  I suggest the following:
>>> >>
>>> >>     - qemu-test: qemu unit tests
>>> >>     - kvm-unit-tests: kvm unit tests
>>> >>     - kvm-autotest: unit test drivers + system tests
>> >
>> >
>> >  I would counter with:
>> >
>> >  - gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest)

+1 for qtester

>> >  - qtest: low level, single device functional tests

+1 for it

>> >  - kvm-unit-tests: low level, instruction-set level functional tests
> Not really.  kvm-unit-tests tests things specific to kvm.
>
>> >  - qemu-test: higher level functional/coverage tests (multiple device
>> >                interaction)
>> >  - kvm-autotest: unit/functional test drivers + acceptance testing
>> >
>> >  Note that the line I'm drawing is acceptance vs. functional testing,
>> >  not unit vs. integration testing.  Technically, our unit tests are
>> >  things like test-visitor.  Everything else is an integration test.
>> >
>> >  But the separation between kvm-autotest is acceptance testing vs.
>> >  functional testing.
>> >
>> >  Acceptance testing is, "does Windows boot", "can I create three
>> >  virtio-serial devices".
>> >
>> >  Obviously, part of acceptance testing is, "does this set of functional
>> >  tests pass".
> Seems like a very blurry line.  Especially as the functional test is
> weaker than either qtest and kvm-autotest.  I now have to agree with the
> others that it's duplicate functionality.  Does it really matter whether
> you're creating an image by compiling Linux and assembling and
> initramfs, or by downloading Fedora.iso and installing it?  It's testing
> exactly the same thing, guest boot and functionality.
>
> Would you add live migration testing to qemu-test?  If yes, you're
> duplicating some more.  If not, you're not doing functional or coverage
> tests for that functionality.

 From the recent threads it looks to me that the 2 advantages of 
qemu-test over kvm-autotest are:

1. python is not used within the guest
2. qemu-test is smaller and simpler

Except for the above 2, kvm autotest is superior. My main motivation to 
merge qemu-test and kvm autotest is that I fear that qemu-test will be 
the only requirement for committing stuff for qemu and developers will 
be excused from their 'duty' by writing a 50 LOC shell script and assume 
they done w/ testing. In addition to that, the 50 LOC will be too basic 
and only provide value for basic functionality tests and kvm autotest 
folks will need to rewrite all of the matching content and beyond.

Those above 2 advantages can be solve:

1. python is not used within the guest
    - One option is just to use the same shell code in kvm autotest w/o
      no python code and use the same kernel/initramfs.
    - Another way is to use a plain linux distro w/ python and boot it
      into single user test mode and potentially use a S4 guest snapshot
      or external snapshot to shorten its boot time.
      You'll get faster boot time and friendlier code.

2. qemu-test is smaller and simpler
    kvm autotest will have a fast, out of the box mode especially
    designed to answer your requirements.
    It's mainly a matter of preparing a fastTestMode.py that will setup
    all of the above in a similar way that today's 
https://github.com/autotest/autotest/blob/master/client/tests/kvm/get_started.py

Lastly, each test can have a fast|basic|sanity overloaded function that 
will be called by this.

>> >
>>>> >>>  We shouldn't be discussing whether it's possible to merge the two
>>>> >>>  tools, but rather what the technical benefits of doing so would be.
>>>> >>>
>>>> >>>  Since at this point, there is almost no overlap between the two, I
>>>> >>>  don't see any actual technical benefit to merging them.  I see benefit
>>>> >>>  to autotest executing qemu-test, of course.
>>> >>
>>> >>  I'd say that running a ping test is a weak version of kvm-autotest's
>>> >>  system tests.
>> >
>> >  Consider the Linux kernel to be our library of functionality to write
>> >  our unit tests.
> I was about to say the same thing, but with a negative implication.
> Using Linux restricts your tests to what Linux does with the devices.
>
>> >  I don't want to write a TCP/IP stack.  We aren't just grabbing a
>> >  random distro kernel.  We're building one from scratch configured in a
>> >  specific way.
> How does that help?
>
>> >  In theory, we could even host the kernel source on git.qemu.org and
>> >  fork it to add more interesting things in the kernel (although I'd
>> >  prefer not to do this, obviously).
> That way lies madness, though it could be argued that it lies all around us.

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 08:38 AM, Dor Laor wrote:
> On 12/28/2011 07:21 PM, Avi Kivity wrote:
>> Would you add live migration testing to qemu-test? If yes, you're
>> duplicating some more. If not, you're not doing functional or coverage
>> tests for that functionality.
>
>  From the recent threads it looks to me that the 2 advantages of qemu-test over
> kvm-autotest are:
>
> 1. python is not used within the guest
> 2. qemu-test is smaller and simpler
>
> Except for the above 2, kvm autotest is superior.

Sorry, but that's non-sense.

qemu-test builds a custom guest, entirely from source.  This makes it possible 
to efficiently test non-native architectures.  The tests are written for this 
minimal environment (which is not straight forward to do).  This tests will 
never run on Windows as they are 100% tied to their environment.

autotest (let's be clear, there is no such thing as kvm autotest anymore) is a 
framework for executing third party tests.  It's got fancy magic to execute in 
client mode vs. server mode.  There is a suite of tests that are integrated in 
autotest that exercise various types of virtualization functionality.  This 
suite of tests use a special config format to determine what they do.

Included in this, is the ability to create a guest from an ISO either using step 
files or via a guest-specific mechanism (kickstart, etc.).  The tests are 
written to be mostly guest agnostic and are therefore written in Python in a 
cross platform way.

There is essentially no overlap between the two and calling kvm autotest 
superior is like calling the Sun superior to the Moon.

> My main motivation to merge
> qemu-test and kvm autotest is that I fear that qemu-test will be the only
> requirement for committing stuff for qemu and developers will be excused from
> their 'duty' by writing a 50 LOC shell script and assume they done w/ testing.

There is no requirement to write autotest tests to get things merged into QEMU. 
  That's is how things are today.

And I don't think there will ever a requirement to write anything that isn't 
merged directly into qemu.git.  I'm not going to hold up merging a feature until 
another project merges something into their tree.

So unless we're going to merge KVM autotest into qemu.git, I don't think there's 
every going to be a requirement to write a KVM autotest test in order to get 
something merged into qemu.git.

But since autotest is a framework for running third-party tests, it seems to 
make sense for qemu.git to have a test framework that autotest can execute.

And since what we call KVM autotest actually tests a heck of a lot more than 
just QEMU, it makes sense for KVM autotest to continue to focus on full stack 
testing where QEMU is but one of the many components that it tests.

> In addition to that, the 50 LOC will be too basic and only provide value for
> basic functionality tests and kvm autotest folks will need to rewrite all of the
> matching content and beyond.
>
> Those above 2 advantages can be solve:
>
> 1. python is not used within the guest
> - One option is just to use the same shell code in kvm autotest w/o
> no python code and use the same kernel/initramfs.

Yes, you can make a directory in kvm autotest that just imports qemu-test, but 
what's the point of doing that?  Isn't it better for this to live in qemu.git?

> - Another way is to use a plain linux distro w/ python and boot it
> into single user test mode and potentially use a S4 guest snapshot
> or external snapshot to shorten its boot time.

You cannot easily create a "plain linux distro" for an ARM target.  If you don't 
believe me, add ARM guest support to KVM autotest and see how well it works out :-)

> You'll get faster boot time and friendlier code.

Adding S4 resume for "simple tests" seems to be a bit odd to me...

> 2. qemu-test is smaller and simpler
> kvm autotest will have a fast, out of the box mode especially
> designed to answer your requirements.
> It's mainly a matter of preparing a fastTestMode.py that will setup
> all of the above in a similar way that today's
> https://github.com/autotest/autotest/blob/master/client/tests/kvm/get_started.py

I hope you see from the above that this isn't just about having something new 
that has fewer features and is therefore simpler for the time being.

The approach is fundamentally different.

Regards,

Anthony Liguori

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 06:39 PM, Anthony Liguori wrote:
>
> qemu-test builds a custom guest, entirely from source.  This makes it
> possible to efficiently test non-native architectures.  The tests are
> written for this minimal environment (which is not straight forward to
> do).  This tests will never run on Windows as they are 100% tied to
> their environment.
>
> autotest (let's be clear, there is no such thing as kvm autotest
> anymore) is a framework for executing third party tests.  It's got
> fancy magic to execute in client mode vs. server mode.  There is a
> suite of tests that are integrated in autotest that exercise various
> types of virtualization functionality.  This suite of tests use a
> special config format to determine what they do.
>
> Included in this, is the ability to create a guest from an ISO either
> using step files or via a guest-specific mechanism (kickstart, etc.). 
> The tests are written to be mostly guest agnostic and are therefore
> written in Python in a cross platform way.
>
> There is essentially no overlap between the two and calling kvm
> autotest superior is like calling the Sun superior to the Moon.

Why not extend autotest do the same thing.  Instead of downloading a
fedora iso it would download a kernel tarball and (cross-)build it.

>
>> My main motivation to merge
>> qemu-test and kvm autotest is that I fear that qemu-test will be the
>> only
>> requirement for committing stuff for qemu and developers will be
>> excused from
>> their 'duty' by writing a 50 LOC shell script and assume they done w/
>> testing.
>
> There is no requirement to write autotest tests to get things merged
> into QEMU.  That's is how things are today.
>
> And I don't think there will ever a requirement to write anything that
> isn't merged directly into qemu.git.  I'm not going to hold up merging
> a feature until another project merges something into their tree.

What about virtio features (we used to depend on the kernel, now on the
spec)?  Seabios?

>
> So unless we're going to merge KVM autotest into qemu.git, I don't
> think there's every going to be a requirement to write a KVM autotest
> test in order to get something merged into qemu.git.

Let's consider postcopy live migration being proposed.  In addition to
various unit tests, it also wants an integration test.  So now we need
to write a qemu-test postcopy live migration test, and also autotest
test that tests non-Linux guests, migration under load, with memory
hotplug, etc.

>
> But since autotest is a framework for running third-party tests, it
> seems to make sense for qemu.git to have a test framework that
> autotest can execute.
>
> And since what we call KVM autotest actually tests a heck of a lot
> more than just QEMU, it makes sense for KVM autotest to continue to
> focus on full stack testing where QEMU is but one of the many
> components that it tests.

It might have made sense to split the kvm-testing functionality of
autotest, and have autotest drive that.  We could even have called it
qemu-test.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 10:53 AM, Avi Kivity wrote:
> On 12/29/2011 06:39 PM, Anthony Liguori wrote:
>>
>> qemu-test builds a custom guest, entirely from source.  This makes it
>> possible to efficiently test non-native architectures.  The tests are
>> written for this minimal environment (which is not straight forward to
>> do).  This tests will never run on Windows as they are 100% tied to
>> their environment.
>>
>> autotest (let's be clear, there is no such thing as kvm autotest
>> anymore) is a framework for executing third party tests.  It's got
>> fancy magic to execute in client mode vs. server mode.  There is a
>> suite of tests that are integrated in autotest that exercise various
>> types of virtualization functionality.  This suite of tests use a
>> special config format to determine what they do.
>>
>> Included in this, is the ability to create a guest from an ISO either
>> using step files or via a guest-specific mechanism (kickstart, etc.).
>> The tests are written to be mostly guest agnostic and are therefore
>> written in Python in a cross platform way.
>>
>> There is essentially no overlap between the two and calling kvm
>> autotest superior is like calling the Sun superior to the Moon.
>
> Why not extend autotest do the same thing.  Instead of downloading a
> fedora iso it would download a kernel tarball and (cross-)build it.

My plan is to post a qemu-jeos repo that has the build bits for creating the 
kernel/initramfs pair.

qemu-test will then live in qemu.git and just depend on the results of the 
qemu-jeos.git.  I'd maybe even go as far as including the binaries in qemu.git 
but I don't think that would work so well...

>>> My main motivation to merge
>>> qemu-test and kvm autotest is that I fear that qemu-test will be the
>>> only
>>> requirement for committing stuff for qemu and developers will be
>>> excused from
>>> their 'duty' by writing a 50 LOC shell script and assume they done w/
>>> testing.
>>
>> There is no requirement to write autotest tests to get things merged
>> into QEMU.  That's is how things are today.
>>
>> And I don't think there will ever a requirement to write anything that
>> isn't merged directly into qemu.git.  I'm not going to hold up merging
>> a feature until another project merges something into their tree.
>
> What about virtio features (we used to depend on the kernel, now on the
> spec)?  Seabios?

Those are rare occurrences and have always been a bit awkward.  OTOH, we're 
talking about the critical path of essentially every single feature that gets 
merged into QEMU.

>> So unless we're going to merge KVM autotest into qemu.git, I don't
>> think there's every going to be a requirement to write a KVM autotest
>> test in order to get something merged into qemu.git.
>
> Let's consider postcopy live migration being proposed.  In addition to
> various unit tests, it also wants an integration test.  So now we need
> to write a qemu-test postcopy live migration test, and also autotest
> test that tests non-Linux guests, migration under load, with memory
> hotplug, etc.

But the integration test is also going to depend on libvirt support for the 
feature.  So how does this all get orchestrated?  kvm-autotest merges the test 
case first, then QEMU merges the support, then libvirt?  What about VDSM and 
ovirt-engine?  Once kvm-autotest starts supporting VDSM that's yet another 
component to deal with.

Realistically, kvm-autotest is going to have to wait for the bits to get merged 
in all of the necessary pieces and then build a test for it on their own schedule.

>> But since autotest is a framework for running third-party tests, it
>> seems to make sense for qemu.git to have a test framework that
>> autotest can execute.
>>
>> And since what we call KVM autotest actually tests a heck of a lot
>> more than just QEMU, it makes sense for KVM autotest to continue to
>> focus on full stack testing where QEMU is but one of the many
>> components that it tests.
>
> It might have made sense to split the kvm-testing functionality of
> autotest, and have autotest drive that.  We could even have called it
> qemu-test.

I specifically advocated this during Lucas' KVM Forum talk and he was strongly 
opposed to it.

I think kvm autotest would get a lot more interest if the test cases were pulled 
out of autotest, made more stand alone.  They also should be more Unix like 
being divided into individual executables with independent command line options 
over a single do-everything configuration file.

But that's all independent of qemu-test.  qemu-test is really qtest + 
Linux/busybox as a libOS.  Even if the above happened, I would still think 
qemu-test needed to exist.

Regards,

Anthony Liguori

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 07:02 PM, Anthony Liguori wrote:
>
> Those are rare occurrences and have always been a bit awkward.  OTOH,
> we're talking about the critical path of essentially every single
> feature that gets merged into QEMU.
>

Won't that be qtest?

Major features are rare.  Small changes are common.

>> Let's consider postcopy live migration being proposed.  In addition to
>> various unit tests, it also wants an integration test.  So now we need
>> to write a qemu-test postcopy live migration test, and also autotest
>> test that tests non-Linux guests, migration under load, with memory
>> hotplug, etc.
>
> But the integration test is also going to depend on libvirt support
> for the feature.  So how does this all get orchestrated?  kvm-autotest
> merges the test case first, then QEMU merges the support, then
> libvirt?  What about VDSM and ovirt-engine?  Once kvm-autotest starts
> supporting VDSM that's yet another component to deal with.
>
> Realistically, kvm-autotest is going to have to wait for the bits to
> get merged in all of the necessary pieces and then build a test for it
> on their own schedule.

It just has to be posted and reviewed, it doesn't all have to be merged
at the same time (and the qemu merge has to precede the autotest merge)

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 11:06 AM, Avi Kivity wrote:
> On 12/29/2011 07:02 PM, Anthony Liguori wrote:
>>
>> Those are rare occurrences and have always been a bit awkward.  OTOH,
>> we're talking about the critical path of essentially every single
>> feature that gets merged into QEMU.
>>
>
> Won't that be qtest?

The difference is that the qtest test can be patch N/N in a patch series and be 
merged at exactly the same time.

I think having an in-tree test suite is the only practical way to do TDD.

Regards,

Anthony Liguori

>>> Let's consider postcopy live migration being proposed.  In addition to
>>> various unit tests, it also wants an integration test.  So now we need
>>> to write a qemu-test postcopy live migration test, and also autotest
>>> test that tests non-Linux guests, migration under load, with memory
>>> hotplug, etc.
>>
>> But the integration test is also going to depend on libvirt support
>> for the feature.  So how does this all get orchestrated?  kvm-autotest
>> merges the test case first, then QEMU merges the support, then
>> libvirt?  What about VDSM and ovirt-engine?  Once kvm-autotest starts
>> supporting VDSM that's yet another component to deal with.
>>
>> Realistically, kvm-autotest is going to have to wait for the bits to
>> get merged in all of the necessary pieces and then build a test for it
>> on their own schedule.
>
> It just has to be posted and reviewed, it doesn't all have to be merged
> at the same time (and the qemu merge has to precede the autotest merge)
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Lucas Meneghel Rodrigues]

On 12/29/2011 03:02 PM, Anthony Liguori wrote:
> On 12/29/2011 10:53 AM, Avi Kivity wrote:
>> On 12/29/2011 06:39 PM, Anthony Liguori wrote:
>>
>> It might have made sense to split the kvm-testing functionality of
>> autotest, and have autotest drive that. We could even have called it
>> qemu-test.
>
> I specifically advocated this during Lucas' KVM Forum talk and he was
> strongly opposed to it.

Ok, this is a point where I have failed in communicating things.

I've watched the presentation on video just to see if my memory was not 
betraying me, and I wouldn't say I was 'strongly opposed', it's just 
that there are some practical problems, not impossible to overcome of 
course, but worth thinking.

  * We rely on a fair bit of autotest libraries/API, so by extracting 
the APIs we rely on to start a new project we'd be creating some code 
duplication, and updates/bugfixes on the autotest API wouldn't be easily 
carried on to the new project.
  * I'm not sure that this per se would fix the perceived problems with 
the tests. It'd be nearly the same code, just outside the autotest tree, 
I fail to see how that would be much better, and therefore, justify the 
work on doing this. Of course I might be wrong.

> I think kvm autotest would get a lot more interest if the test cases
> were pulled out of autotest, made more stand alone. They also should be
> more Unix like being divided into individual executables with
> independent command line options over a single do-everything
> configuration file.

You have a point with regards to having the test cases more stand alone, 
but it's not as easy as one would think mainly because of the large 
amount of user configurable options that we have to pass to the tests. 
This is a place where the config file format shines, since we just have 
to override some parameters on different variants to have a test working 
among Windows and Linux guests, for example, all expressed in a concise 
way. The config files might look enormous, but they are actually pretty 
small compared with the amount of test combinations we can get out of 
it. This is nothing I'd call insurmountable as well.

If I get more feedback of people saying "Ok, this would make things 
better for me and make me more interested", then we'll go that route. 
I'm sorry if I gave the impression I was strongly opposed to your idea.

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 05:17 PM, Lucas Meneghel Rodrigues wrote:
> On 12/29/2011 03:02 PM, Anthony Liguori wrote:
>> On 12/29/2011 10:53 AM, Avi Kivity wrote:
>>> On 12/29/2011 06:39 PM, Anthony Liguori wrote:
>>>
>>> It might have made sense to split the kvm-testing functionality of
>>> autotest, and have autotest drive that. We could even have called it
>>> qemu-test.
>>
>> I specifically advocated this during Lucas' KVM Forum talk and he was
>> strongly opposed to it.
>
> Ok, this is a point where I have failed in communicating things.
>
> I've watched the presentation on video just to see if my memory was not
> betraying me, and I wouldn't say I was 'strongly opposed', it's just that there
> are some practical problems, not impossible to overcome of course, but worth
> thinking.

So I decided to do some snooping.  Here are some stats:

anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py
    150 balloon_check.py
     68 boot_savevm.py
    190 cdrom.py
   1875 cgroup.py
    111 cpu_hotplug.py
    170 enospc.py
     71 floppy.py
     72 getfd.py
     89 hdparm.py
      0 __init__.py
    615 ksm_overcommit.py
    107 migration_multi_host.py
    117 migration.py
     85 migration_with_file_transfer.py
     43 migration_with_reboot.py
    138 multi_disk.py
     62 nic_bonding.py
    104 nic_hotplug.py
     60 nmi_watchdog.py
    203 pci_hotplug.py
    182 physical_resources_check.py
    439 qemu_img.py
     48 qemu_iotests.py
    407 qmp_basic.py
    389 qmp_basic_rhel6.py
     54 set_link.py
     67 smbios_table.py
    356 stepmaker.py
    247 steps.py
     43 stop_continue.py
     31 system_reset_bootable.py
    181 timedrift.py
     96 timedrift_with_migration.py
     91 timedrift_with_reboot.py
    103 timedrift_with_stop.py
     28 unittest_kvmctl.py
    121 unittest.py
     90 usb.py
   2174 virtio_console.py
     85 vmstop.py
   9562 total

anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>" kvm | sort -u

Amos Kong <akong@redhat.com>
Chris Evich <cevich@redhat.com>
Cleber Rosa <crosa@redhat.com>
Jerry Tang <jtang@suse.com>
lmr <lmr@592f7852-d20e-0410-864c-8624ca9c26a4>
Lucas Meneghel Rodrigues <lmr@redhat.com>
Lucas Meneghel Rodrigues <lookkas@gmail.com>
Lukas Doktor <ldoktor@redhat.com>
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4>
Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com>
pradeep <psuriset@linux.vnet.ibm.com>
Qingtang Zhou <qzhou@redhat.com>
Thomas Jarosch <thomas.jarosch@intra2net.com>
Yiqiao Pu <ypu@redhat.com>

Which leads me to the following conclusions:

1) No one outside of autotest developers is actually contributing tests to 
kvm-autotest.

2) Most of the tests are relatively small and simple enough that they could be 
trivially ported to a stand alone utility.  For instance, looking at the 
pci_hotplug.py, it just executes monitor commands and does basic pci enumeration 
in the guest.

I don't see a lot of pitfalls here.  At the end of the day, we're not talking 
about all that much code.

>
> * We rely on a fair bit of autotest libraries/API, so by extracting the APIs we
> rely on to start a new project we'd be creating some code duplication, and
> updates/bugfixes on the autotest API wouldn't be easily carried on to the new
> project.
> * I'm not sure that this per se would fix the perceived problems with the tests.
> It'd be nearly the same code, just outside the autotest tree, I fail to see how
> that would be much better, and therefore, justify the work on doing this. Of
> course I might be wrong.

It's not about where they live, it's about the ability to execute them in a 
simple and direct fashion.

>
>> I think kvm autotest would get a lot more interest if the test cases
>> were pulled out of autotest, made more stand alone. They also should be
>> more Unix like being divided into individual executables with
>> independent command line options over a single do-everything
>> configuration file.
>
> You have a point with regards to having the test cases more stand alone, but
> it's not as easy as one would think mainly because of the large amount of user
> configurable options that we have to pass to the tests.

Configuration options are really the devil when it comes to testing.  As a 
developer, I want my test suite to just run and tell me if I have bugs.  I have 
no interest in configuring it to run a certain way.  I don't want to think that 
much about what I'm testing, I just want the tests to run and tell me if I 
screwed something up.

Regards,

Anthony Liguori

> This is a place where
> the config file format shines, since we just have to override some parameters on
> different variants to have a test working among Windows and Linux guests, for
> example, all expressed in a concise way. The config files might look enormous,
> but they are actually pretty small compared with the amount of test combinations
> we can get out of it. This is nothing I'd call insurmountable as well.
>
> If I get more feedback of people saying "Ok, this would make things better for
> me and make me more interested", then we'll go that route. I'm sorry if I gave
> the impression I was strongly opposed to your idea.
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Lucas Meneghel Rodrigues]

On 12/29/2011 10:33 PM, Anthony Liguori wrote:
> So I decided to do some snooping. Here are some stats:
>
> anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py
> 150 balloon_check.py
> 68 boot_savevm.py
> 190 cdrom.py
> 1875 cgroup.py
> 111 cpu_hotplug.py
> 170 enospc.py
> 71 floppy.py
> 72 getfd.py
> 89 hdparm.py
> 0 __init__.py
> 615 ksm_overcommit.py
> 107 migration_multi_host.py
> 117 migration.py
> 85 migration_with_file_transfer.py
> 43 migration_with_reboot.py
> 138 multi_disk.py
> 62 nic_bonding.py
> 104 nic_hotplug.py
> 60 nmi_watchdog.py
> 203 pci_hotplug.py
> 182 physical_resources_check.py
> 439 qemu_img.py
> 48 qemu_iotests.py
> 407 qmp_basic.py
> 389 qmp_basic_rhel6.py
> 54 set_link.py
> 67 smbios_table.py
> 356 stepmaker.py
> 247 steps.py
> 43 stop_continue.py
> 31 system_reset_bootable.py
> 181 timedrift.py
> 96 timedrift_with_migration.py
> 91 timedrift_with_reboot.py
> 103 timedrift_with_stop.py
> 28 unittest_kvmctl.py
> 121 unittest.py
> 90 usb.py
> 2174 virtio_console.py
> 85 vmstop.py
> 9562 total

There are more tests under client/virt/tests:

[lmr@freedom tests]$ wc -l *.py
     25 autotest.py
    100 boot.py
     42 build.py
     32 clock_getres.py
    235 ethtool.py
     84 file_transfer.py
     53 fillup_disk.py
     76 guest_s4.py
     80 guest_test.py
     45 image_copy.py
      0 __init__.py
    136 iofuzz.py
     31 ioquit.py
     40 iozone_windows.py
    127 jumbo.py
     85 kdump.py
     41 linux_s3.py
     84 lvm.py
     60 mac_change.py
     46 module_probe.py
     90 multicast.py
    106 netperf.py
    149 netstress_kill_guest.py
    255 nfs_corrupt.py
     63 nicdriver_unload.py
     39 nic_promisc.py
     73 ping.py
     29 pxe.py
     42 shutdown.py
    147 softlockup.py
     53 stress_boot.py
    164 trans_hugepage_defrag.py
    127 trans_hugepage.py
    113 trans_hugepage_swapping.py
    852 unattended_install.py
    175 vlan.py
     46 watchdog.py
    136 whql_client_install.py
    275 whql_submission.py
     49 yum_update.py
   4405 total

Not to mention the infrastructure libraries:

[lmr@freedom virt]$ wc -l *.py
   1351 aexpect.py
    705 base_installer.py
      9 common.py
      0 __init__.py
    195 installer.py
     64 installer_unittest.py
    240 kvm_installer.py
    921 kvm_monitor.py
   1635 kvm_vm.py
    324 libvirt_monitor.py
   1271 libvirt_vm.py
    110 passfd.py
    237 ppm_utils.py
    519 rss_client.py
    527 virt_env_process.py
    124 virt_http_server.py
     51 virt_passfd_setup.py
    229 virt_scheduler.py
   1401 virt_step_editor.py
    131 virt_test.py
    315 virt_test_setup.py
    813 virt_test_utils.py
   3556 virt_utils.py
    196 virt_utils_unittest.py
    224 virt_video_maker.py
   1032 virt_vm.py
  16180 total

There's a bit more outside this dir, not to mention the auxiliary data 
files. The support libraries account for roughly for the same number of 
lines of test code written, so we can say in very rough terms we have 
around 33 k of python code for kvm/libvirt tests. I agree it is more 
than an order of magnitude smaller than the qemu code base, but still it 
demands effort that has to be justified.

> anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>"
> kvm | sort -u
>
> Amos Kong <akong@redhat.com>
> Chris Evich <cevich@redhat.com>
> Cleber Rosa <crosa@redhat.com>
> Jerry Tang <jtang@suse.com>
> lmr <lmr@592f7852-d20e-0410-864c-8624ca9c26a4>
> Lucas Meneghel Rodrigues <lmr@redhat.com>
> Lucas Meneghel Rodrigues <lookkas@gmail.com>
> Lukas Doktor <ldoktor@redhat.com>
> mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4>
> Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com>
> pradeep <psuriset@linux.vnet.ibm.com>
> Qingtang Zhou <qzhou@redhat.com>
> Thomas Jarosch <thomas.jarosch@intra2net.com>
> Yiqiao Pu <ypu@redhat.com>
>
> Which leads me to the following conclusions:
>
> 1) No one outside of autotest developers is actually contributing tests
> to kvm-autotest.

Autotest was maintained using subversion until recently, all commits 
from people outside autotest were commited by autotest developers, the 
git tree was just a mirror until recently. Grepping for Signed-off-by: 
lines will give a much better idea of the contributors:

[lmr@freedom autotest.git]$ git log client/virt | grep "Signed-off-by:" 
| sort -u
     Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com>
     Signed-off-by: Alon Levy <alevy@redhat.com>
     Signed-off-by: Amos Kong <akong@redhat.com>
     Signed-off-by: Chen Cao <kcao@redhat.com>
     Signed-off-by: Chris Evich <cevich@redhat.com>
     Signed-off-by: Cleber Rosa <crosa@redhat.com>
     Signed-off-by: Feng Yang <fyang@redhat.com>
     Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
     Signed-off-by: Jason D. Gaston <jason.d.gaston@intel.com>
     Signed-off-by: Jason Wang <jasowang@redhat.com>
     Signed-off-by: Jiri Zupka <jzupka@redhat.com>
     Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
     Signed-off-by: Lukas Doktor <ldoktor@redhat.com>
     Signed-off-by: Martin Jenner <mjenner@redhat.com>
     Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com>
     Signed-off-by: Pradeep Kumar Surisetty <psuriset@linux.vnet.ibm.com>
     Signed-off-by: Qingtang Zhou <qzhou@redhat.com>
     Signed-off-by: Quentin Deldycke <quentindeldycke@gmail.com>
     Signed-off-by: Ray Chauvin <rchauvin@redhat.com>
     Signed-off-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
     Signed-off-by: Suqin Huang <shuang@redhat.com>
     Signed-off-by: Wei Yang <weiyang@linux.vnet.ibm.com>
     Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com>
     Signed-off-by: Yiqiao Pu <ypu@redhat.com>
     Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com>
     Signed-off-by: Yolkfull Chow <yzhou@redhat.com>

[lmr@freedom autotest.git]$ git log client/tests/kvm | grep 
"Signed-off-by:" | sort -u
     Signed-off-by: Alon Levy <alevy@redhat.com>
     Signed-off-by: Amos Kong <akong@redhat.com>
     Signed-off-by: Avi Kivity <avi@redhat.com>
     Signed-off-by: Bear Yang <byang@redhat.com>
     Signed-off-by: Cao, Chen <kcao@redhat.com>
     Signed-off-by: Chen Cao <kcao@redhat.com>
     Signed-off-by: Chris Evich <cevich@redhat.com>
     Signed-off-by: Cleber Rosa <crosa@redhat.com>
     Signed-off-by: David Huff <dhuff@redhat.com>
     Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
     Signed-off-by: Feng Yang <fyang@redhat.com>
     Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
     Signed-off-by: Jason Wang <jasowang@redhat.com>
     Signed-off-by: Jerry Tang <jtang@suse.com>
     Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
     Signed-off-by: Jin Bing Guo <guojb@cn.ibm.com>
     Signed-off-by: Jiri Zupka <jzupka@redhat.com>
     Signed-off-by: Ken Cao <kcao@redhat.com>
     Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com
     Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
     Signed-off-by: Lukas Doktor <ldoktor@redhat.com>
     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
     Signed-off-by: Martin Jenner <mjenner@redhat.com>
     Signed-off-by: Michael Goldish <mgoldish@redhat.com>
     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
     Signed-off-by: Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com>
     Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com>
     Signed-off-by: Qingtang Zhou <qzhou@redhat.com>
     Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
     Signed-off-by: Shuxi Shang <sshang@redhat.com>
     Signed-off-by: Sudhir Kumar <skumar@linux.vnet.ibm.com>
     Signed-off-by: Supriya Kannery <supriyak@in.ibm.com>
     Signed-off-by: Suqin Huang <shuang@redhat.com>
     Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
     Signed-off-by: Thomas Jarosch <thomas.jarosch@intra2net.com>
     Signed-off-by: Uri Lublin <uril@redhat.com>
     Signed-off-by: Yiqiao Pu <ypu@redhat.com>
     Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com>
     Signed-off-by: Yolkfull Chow <yzhou@redhat.com>

So, not quite what you wanted to demonstrate.

> 2) Most of the tests are relatively small and simple enough that they
> could be trivially ported to a stand alone utility. For instance,
> looking at the pci_hotplug.py, it just executes monitor commands and
> does basic pci enumeration in the guest.

They do depend on infrastructure code to be simple and small. The 
infrastructure code was written using autotest libraries, so although it 
is possible to make it stand alone, still there's the problem of having 
these library/classes out of sync with autotest evolving.

> I don't see a lot of pitfalls here. At the end of the day, we're not
> talking about all that much code.

Even so (you see it's not so small), it demands effort that has to be 
well justified, as I pointed out. I'm starting to believe it might be 
worth the effort.

> It's not about where they live, it's about the ability to execute them
> in a simple and direct fashion.

Ok, fair enough, we'll see what we can do, although I wish it were a 
simpler problem, see below.

>> You have a point with regards to having the test cases more stand
>> alone, but
>> it's not as easy as one would think mainly because of the large amount
>> of user
>> configurable options that we have to pass to the tests.
>
> Configuration options are really the devil when it comes to testing. As
> a developer, I want my test suite to just run and tell me if I have
> bugs. I have no interest in configuring it to run a certain way. I don't
> want to think that much about what I'm testing, I just want the tests to
> run and tell me if I screwed something up.

Still, you need to tell your tests, say, the format of the terminal 
prompt for Fedora might be slightly different of the 
[insert-distro-here], that the src and destination paths on a file 
transfer test are different for Windows and Linux guests, that the 
output of a given command when you are running a test with a virtio card 
differs from the output when you are using rtl8139, that you want to 
build qemu from git repo X, or from brew build Y, using a custom version 
of seabios, and so on and so forth.

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Cleber Rosa]

On 12/29/2011 10:20 PM, Lucas Meneghel Rodrigues wrote:
> On 12/29/2011 10:33 PM, Anthony Liguori wrote:
>> So I decided to do some snooping. Here are some stats:
>>
>> anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py
>> 150 balloon_check.py
>> 68 boot_savevm.py
>> 190 cdrom.py
>> 1875 cgroup.py
>> 111 cpu_hotplug.py
>> 170 enospc.py
>> 71 floppy.py
>> 72 getfd.py
>> 89 hdparm.py
>> 0 __init__.py
>> 615 ksm_overcommit.py
>> 107 migration_multi_host.py
>> 117 migration.py
>> 85 migration_with_file_transfer.py
>> 43 migration_with_reboot.py
>> 138 multi_disk.py
>> 62 nic_bonding.py
>> 104 nic_hotplug.py
>> 60 nmi_watchdog.py
>> 203 pci_hotplug.py
>> 182 physical_resources_check.py
>> 439 qemu_img.py
>> 48 qemu_iotests.py
>> 407 qmp_basic.py
>> 389 qmp_basic_rhel6.py
>> 54 set_link.py
>> 67 smbios_table.py
>> 356 stepmaker.py
>> 247 steps.py
>> 43 stop_continue.py
>> 31 system_reset_bootable.py
>> 181 timedrift.py
>> 96 timedrift_with_migration.py
>> 91 timedrift_with_reboot.py
>> 103 timedrift_with_stop.py
>> 28 unittest_kvmctl.py
>> 121 unittest.py
>> 90 usb.py
>> 2174 virtio_console.py
>> 85 vmstop.py
>> 9562 total
>
> There are more tests under client/virt/tests:
>
> [lmr@freedom tests]$ wc -l *.py
>     25 autotest.py
>    100 boot.py
>     42 build.py
>     32 clock_getres.py
>    235 ethtool.py
>     84 file_transfer.py
>     53 fillup_disk.py
>     76 guest_s4.py
>     80 guest_test.py
>     45 image_copy.py
>      0 __init__.py
>    136 iofuzz.py
>     31 ioquit.py
>     40 iozone_windows.py
>    127 jumbo.py
>     85 kdump.py
>     41 linux_s3.py
>     84 lvm.py
>     60 mac_change.py
>     46 module_probe.py
>     90 multicast.py
>    106 netperf.py
>    149 netstress_kill_guest.py
>    255 nfs_corrupt.py
>     63 nicdriver_unload.py
>     39 nic_promisc.py
>     73 ping.py
>     29 pxe.py
>     42 shutdown.py
>    147 softlockup.py
>     53 stress_boot.py
>    164 trans_hugepage_defrag.py
>    127 trans_hugepage.py
>    113 trans_hugepage_swapping.py
>    852 unattended_install.py
>    175 vlan.py
>     46 watchdog.py
>    136 whql_client_install.py
>    275 whql_submission.py
>     49 yum_update.py
>   4405 total
>
> Not to mention the infrastructure libraries:
>
> [lmr@freedom virt]$ wc -l *.py
>   1351 aexpect.py
>    705 base_installer.py
>      9 common.py
>      0 __init__.py
>    195 installer.py
>     64 installer_unittest.py
>    240 kvm_installer.py
>    921 kvm_monitor.py
>   1635 kvm_vm.py
>    324 libvirt_monitor.py
>   1271 libvirt_vm.py
>    110 passfd.py
>    237 ppm_utils.py
>    519 rss_client.py
>    527 virt_env_process.py
>    124 virt_http_server.py
>     51 virt_passfd_setup.py
>    229 virt_scheduler.py
>   1401 virt_step_editor.py
>    131 virt_test.py
>    315 virt_test_setup.py
>    813 virt_test_utils.py
>   3556 virt_utils.py
>    196 virt_utils_unittest.py
>    224 virt_video_maker.py
>   1032 virt_vm.py
>  16180 total
>
> There's a bit more outside this dir, not to mention the auxiliary data 
> files. The support libraries account for roughly for the same number 
> of lines of test code written, so we can say in very rough terms we 
> have around 33 k of python code for kvm/libvirt tests. I agree it is 
> more than an order of magnitude smaller than the qemu code base, but 
> still it demands effort that has to be justified.
>
>> anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>"
>> kvm | sort -u
>>
>> Amos Kong <akong@redhat.com>
>> Chris Evich <cevich@redhat.com>
>> Cleber Rosa <crosa@redhat.com>
>> Jerry Tang <jtang@suse.com>
>> lmr <lmr@592f7852-d20e-0410-864c-8624ca9c26a4>
>> Lucas Meneghel Rodrigues <lmr@redhat.com>
>> Lucas Meneghel Rodrigues <lookkas@gmail.com>
>> Lukas Doktor <ldoktor@redhat.com>
>> mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4>
>> Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com>
>> pradeep <psuriset@linux.vnet.ibm.com>
>> Qingtang Zhou <qzhou@redhat.com>
>> Thomas Jarosch <thomas.jarosch@intra2net.com>
>> Yiqiao Pu <ypu@redhat.com>
>>
>> Which leads me to the following conclusions:
>>
>> 1) No one outside of autotest developers is actually contributing tests
>> to kvm-autotest.
>
> Autotest was maintained using subversion until recently, all commits 
> from people outside autotest were commited by autotest developers, the 
> git tree was just a mirror until recently. Grepping for Signed-off-by: 
> lines will give a much better idea of the contributors:
>
> [lmr@freedom autotest.git]$ git log client/virt | grep 
> "Signed-off-by:" | sort -u
>     Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com>
>     Signed-off-by: Alon Levy <alevy@redhat.com>
>     Signed-off-by: Amos Kong <akong@redhat.com>
>     Signed-off-by: Chen Cao <kcao@redhat.com>
>     Signed-off-by: Chris Evich <cevich@redhat.com>
>     Signed-off-by: Cleber Rosa <crosa@redhat.com>
>     Signed-off-by: Feng Yang <fyang@redhat.com>
>     Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
>     Signed-off-by: Jason D. Gaston <jason.d.gaston@intel.com>
>     Signed-off-by: Jason Wang <jasowang@redhat.com>
>     Signed-off-by: Jiri Zupka <jzupka@redhat.com>
>     Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
>     Signed-off-by: Lukas Doktor <ldoktor@redhat.com>
>     Signed-off-by: Martin Jenner <mjenner@redhat.com>
>     Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com>
>     Signed-off-by: Pradeep Kumar Surisetty <psuriset@linux.vnet.ibm.com>
>     Signed-off-by: Qingtang Zhou <qzhou@redhat.com>
>     Signed-off-by: Quentin Deldycke <quentindeldycke@gmail.com>
>     Signed-off-by: Ray Chauvin <rchauvin@redhat.com>
>     Signed-off-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
>     Signed-off-by: Suqin Huang <shuang@redhat.com>
>     Signed-off-by: Wei Yang <weiyang@linux.vnet.ibm.com>
>     Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com>
>     Signed-off-by: Yiqiao Pu <ypu@redhat.com>
>     Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com>
>     Signed-off-by: Yolkfull Chow <yzhou@redhat.com>
>
> [lmr@freedom autotest.git]$ git log client/tests/kvm | grep 
> "Signed-off-by:" | sort -u
>     Signed-off-by: Alon Levy <alevy@redhat.com>
>     Signed-off-by: Amos Kong <akong@redhat.com>
>     Signed-off-by: Avi Kivity <avi@redhat.com>
>     Signed-off-by: Bear Yang <byang@redhat.com>
>     Signed-off-by: Cao, Chen <kcao@redhat.com>
>     Signed-off-by: Chen Cao <kcao@redhat.com>
>     Signed-off-by: Chris Evich <cevich@redhat.com>
>     Signed-off-by: Cleber Rosa <crosa@redhat.com>
>     Signed-off-by: David Huff <dhuff@redhat.com>
>     Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>     Signed-off-by: Feng Yang <fyang@redhat.com>
>     Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
>     Signed-off-by: Jason Wang <jasowang@redhat.com>
>     Signed-off-by: Jerry Tang <jtang@suse.com>
>     Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
>     Signed-off-by: Jin Bing Guo <guojb@cn.ibm.com>
>     Signed-off-by: Jiri Zupka <jzupka@redhat.com>
>     Signed-off-by: Ken Cao <kcao@redhat.com>
>     Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com
>     Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
>     Signed-off-by: Lukas Doktor <ldoktor@redhat.com>
>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>     Signed-off-by: Martin Jenner <mjenner@redhat.com>
>     Signed-off-by: Michael Goldish <mgoldish@redhat.com>
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>     Signed-off-by: Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com>
>     Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com>
>     Signed-off-by: Qingtang Zhou <qzhou@redhat.com>
>     Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
>     Signed-off-by: Shuxi Shang <sshang@redhat.com>
>     Signed-off-by: Sudhir Kumar <skumar@linux.vnet.ibm.com>
>     Signed-off-by: Supriya Kannery <supriyak@in.ibm.com>
>     Signed-off-by: Suqin Huang <shuang@redhat.com>
>     Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
>     Signed-off-by: Thomas Jarosch <thomas.jarosch@intra2net.com>
>     Signed-off-by: Uri Lublin <uril@redhat.com>
>     Signed-off-by: Yiqiao Pu <ypu@redhat.com>
>     Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com>
>     Signed-off-by: Yolkfull Chow <yzhou@redhat.com>
>
> So, not quite what you wanted to demonstrate.
>
>> 2) Most of the tests are relatively small and simple enough that they
>> could be trivially ported to a stand alone utility. For instance,
>> looking at the pci_hotplug.py, it just executes monitor commands and
>> does basic pci enumeration in the guest.
>
> They do depend on infrastructure code to be simple and small. The 
> infrastructure code was written using autotest libraries, so although 
> it is possible to make it stand alone, still there's the problem of 
> having these library/classes out of sync with autotest evolving.
>
>> I don't see a lot of pitfalls here. At the end of the day, we're not
>> talking about all that much code.
>
> Even so (you see it's not so small), it demands effort that has to be 
> well justified, as I pointed out. I'm starting to believe it might be 
> worth the effort.
>
>> It's not about where they live, it's about the ability to execute them
>> in a simple and direct fashion.
>
> Ok, fair enough, we'll see what we can do, although I wish it were a 
> simpler problem, see below.
>
>>> You have a point with regards to having the test cases more stand
>>> alone, but
>>> it's not as easy as one would think mainly because of the large amount
>>> of user
>>> configurable options that we have to pass to the tests.
>>
>> Configuration options are really the devil when it comes to testing. As
>> a developer, I want my test suite to just run and tell me if I have
>> bugs. I have no interest in configuring it to run a certain way. I don't
>> want to think that much about what I'm testing, I just want the tests to
>> run and tell me if I screwed something up.
>
> Still, you need to tell your tests, say, the format of the terminal 
> prompt for Fedora might be slightly different of the 
> [insert-distro-here], that the src and destination paths on a file 
> transfer test are different for Windows and Linux guests, that the 
> output of a given command when you are running a test with a virtio 
> card differs from the output when you are using rtl8139, that you want 
> to build qemu from git repo X, or from brew build Y, using a custom 
> version of seabios, and so on and so forth.

This can be solved in a pretty easy way. We can have (kvm autotest) 
config files suited for developers, together with a helper script that 
will run the tests (or test set) given on the command line. Ideally this 
would live inside the qemu git repo, together with a make target.

This config could be targeted to a DSL(-like) guest image that can be 
downloaded, instead of installed. A ~50...100MB download would be hard 
to even notice. Maybe could even be copied pristine from backup between 
each test run.

Of course the maintenance of these config files and guest images would 
be on us.

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 07:20 PM, Lucas Meneghel Rodrigues wrote:
> On 12/29/2011 10:33 PM, Anthony Liguori wrote:
>> So I decided to do some snooping. Here are some stats:
>>
>> anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py
>> 150 balloon_check.py
>> 68 boot_savevm.py
>> 190 cdrom.py
>> 1875 cgroup.py
>> 111 cpu_hotplug.py
>> 170 enospc.py
>> 71 floppy.py
>> 72 getfd.py
>> 89 hdparm.py
>> 0 __init__.py
>> 615 ksm_overcommit.py
>> 107 migration_multi_host.py
>> 117 migration.py
>> 85 migration_with_file_transfer.py
>> 43 migration_with_reboot.py
>> 138 multi_disk.py
>> 62 nic_bonding.py
>> 104 nic_hotplug.py
>> 60 nmi_watchdog.py
>> 203 pci_hotplug.py
>> 182 physical_resources_check.py
>> 439 qemu_img.py
>> 48 qemu_iotests.py
>> 407 qmp_basic.py
>> 389 qmp_basic_rhel6.py
>> 54 set_link.py
>> 67 smbios_table.py
>> 356 stepmaker.py
>> 247 steps.py
>> 43 stop_continue.py
>> 31 system_reset_bootable.py
>> 181 timedrift.py
>> 96 timedrift_with_migration.py
>> 91 timedrift_with_reboot.py
>> 103 timedrift_with_stop.py
>> 28 unittest_kvmctl.py
>> 121 unittest.py
>> 90 usb.py
>> 2174 virtio_console.py
>> 85 vmstop.py
>> 9562 total
>
> There are more tests under client/virt/tests:
>
> [lmr@freedom tests]$ wc -l *.py
> 25 autotest.py
> 100 boot.py
> 42 build.py
> 32 clock_getres.py
> 235 ethtool.py
> 84 file_transfer.py
> 53 fillup_disk.py
> 76 guest_s4.py
> 80 guest_test.py
> 45 image_copy.py
> 0 __init__.py
> 136 iofuzz.py
> 31 ioquit.py
> 40 iozone_windows.py
> 127 jumbo.py
> 85 kdump.py
> 41 linux_s3.py
> 84 lvm.py
> 60 mac_change.py
> 46 module_probe.py
> 90 multicast.py
> 106 netperf.py
> 149 netstress_kill_guest.py
> 255 nfs_corrupt.py
> 63 nicdriver_unload.py
> 39 nic_promisc.py
> 73 ping.py
> 29 pxe.py
> 42 shutdown.py
> 147 softlockup.py
> 53 stress_boot.py
> 164 trans_hugepage_defrag.py
> 127 trans_hugepage.py
> 113 trans_hugepage_swapping.py
> 852 unattended_install.py
> 175 vlan.py
> 46 watchdog.py
> 136 whql_client_install.py
> 275 whql_submission.py
> 49 yum_update.py
> 4405 total

Hrm, what's the significance of the two separate directories?

At any rate, my point still stands.  13k LOC is still small enough that making 
large changes to the infrastructure shouldn't be that big of a deal.  I think 
the important observation is that each test is pretty small which means they can 
be reasonably moved 1-by-1 to a new stand alone infrastructure.

> Not to mention the infrastructure libraries:
>
> [lmr@freedom virt]$ wc -l *.py
> 1351 aexpect.py
> 705 base_installer.py
> 9 common.py
> 0 __init__.py
> 195 installer.py
> 64 installer_unittest.py
> 240 kvm_installer.py
> 921 kvm_monitor.py
> 1635 kvm_vm.py
> 324 libvirt_monitor.py
> 1271 libvirt_vm.py
> 110 passfd.py
> 237 ppm_utils.py
> 519 rss_client.py
> 527 virt_env_process.py
> 124 virt_http_server.py
> 51 virt_passfd_setup.py
> 229 virt_scheduler.py
> 1401 virt_step_editor.py
> 131 virt_test.py
> 315 virt_test_setup.py
> 813 virt_test_utils.py
> 3556 virt_utils.py
> 196 virt_utils_unittest.py
> 224 virt_video_maker.py
> 1032 virt_vm.py
> 16180 total
>
> There's a bit more outside this dir, not to mention the auxiliary data files.
> The support libraries account for roughly for the same number of lines of test
> code written, so we can say in very rough terms we have around 33 k of python
> code for kvm/libvirt tests. I agree it is more than an order of magnitude
> smaller than the qemu code base, but still it demands effort that has to be
> justified.
>
>> anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>"
>> kvm | sort -u
>>
>> Amos Kong <akong@redhat.com>
>> Chris Evich <cevich@redhat.com>
>> Cleber Rosa <crosa@redhat.com>
>> Jerry Tang <jtang@suse.com>
>> lmr <lmr@592f7852-d20e-0410-864c-8624ca9c26a4>
>> Lucas Meneghel Rodrigues <lmr@redhat.com>
>> Lucas Meneghel Rodrigues <lookkas@gmail.com>
>> Lukas Doktor <ldoktor@redhat.com>
>> mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4>
>> Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com>
>> pradeep <psuriset@linux.vnet.ibm.com>
>> Qingtang Zhou <qzhou@redhat.com>
>> Thomas Jarosch <thomas.jarosch@intra2net.com>
>> Yiqiao Pu <ypu@redhat.com>
>>
>> Which leads me to the following conclusions:
>>
>> 1) No one outside of autotest developers is actually contributing tests
>> to kvm-autotest.
>
> Autotest was maintained using subversion until recently, all commits from people
> outside autotest were commited by autotest developers, the git tree was just a
> mirror until recently. Grepping for Signed-off-by: lines will give a much better
> idea of the contributors:
>
> [lmr@freedom autotest.git]$ git log client/virt | grep "Signed-off-by:" | sort -u
> Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com>
> Signed-off-by: Alon Levy <alevy@redhat.com>
> Signed-off-by: Amos Kong <akong@redhat.com>
> Signed-off-by: Chen Cao <kcao@redhat.com>
> Signed-off-by: Chris Evich <cevich@redhat.com>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Signed-off-by: Feng Yang <fyang@redhat.com>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> Signed-off-by: Jason D. Gaston <jason.d.gaston@intel.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Jiri Zupka <jzupka@redhat.com>
> Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
> Signed-off-by: Lukas Doktor <ldoktor@redhat.com>
> Signed-off-by: Martin Jenner <mjenner@redhat.com>
> Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com>
> Signed-off-by: Pradeep Kumar Surisetty <psuriset@linux.vnet.ibm.com>
> Signed-off-by: Qingtang Zhou <qzhou@redhat.com>
> Signed-off-by: Quentin Deldycke <quentindeldycke@gmail.com>
> Signed-off-by: Ray Chauvin <rchauvin@redhat.com>
> Signed-off-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
> Signed-off-by: Suqin Huang <shuang@redhat.com>
> Signed-off-by: Wei Yang <weiyang@linux.vnet.ibm.com>
> Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com>
> Signed-off-by: Yiqiao Pu <ypu@redhat.com>
> Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com>
> Signed-off-by: Yolkfull Chow <yzhou@redhat.com>
>
> [lmr@freedom autotest.git]$ git log client/tests/kvm | grep "Signed-off-by:" |
> sort -u
> Signed-off-by: Alon Levy <alevy@redhat.com>
> Signed-off-by: Amos Kong <akong@redhat.com>
> Signed-off-by: Avi Kivity <avi@redhat.com>
> Signed-off-by: Bear Yang <byang@redhat.com>
> Signed-off-by: Cao, Chen <kcao@redhat.com>
> Signed-off-by: Chen Cao <kcao@redhat.com>
> Signed-off-by: Chris Evich <cevich@redhat.com>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Signed-off-by: David Huff <dhuff@redhat.com>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> Signed-off-by: Feng Yang <fyang@redhat.com>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Jerry Tang <jtang@suse.com>
> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
> Signed-off-by: Jin Bing Guo <guojb@cn.ibm.com>
> Signed-off-by: Jiri Zupka <jzupka@redhat.com>
> Signed-off-by: Ken Cao <kcao@redhat.com>
> Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com
> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
> Signed-off-by: Lukas Doktor <ldoktor@redhat.com>
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> Signed-off-by: Martin Jenner <mjenner@redhat.com>
> Signed-off-by: Michael Goldish <mgoldish@redhat.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com>
> Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com>
> Signed-off-by: Qingtang Zhou <qzhou@redhat.com>
> Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
> Signed-off-by: Shuxi Shang <sshang@redhat.com>
> Signed-off-by: Sudhir Kumar <skumar@linux.vnet.ibm.com>
> Signed-off-by: Supriya Kannery <supriyak@in.ibm.com>
> Signed-off-by: Suqin Huang <shuang@redhat.com>
> Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
> Signed-off-by: Thomas Jarosch <thomas.jarosch@intra2net.com>
> Signed-off-by: Uri Lublin <uril@redhat.com>
> Signed-off-by: Yiqiao Pu <ypu@redhat.com>
> Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com>
> Signed-off-by: Yolkfull Chow <yzhou@redhat.com>
>
> So, not quite what you wanted to demonstrate.

Well it actually still is.  There isn't a tremendous overlap between that list 
and the list of QEMU contributors.

>> 2) Most of the tests are relatively small and simple enough that they
>> could be trivially ported to a stand alone utility. For instance,
>> looking at the pci_hotplug.py, it just executes monitor commands and
>> does basic pci enumeration in the guest.
>
> They do depend on infrastructure code to be simple and small. The infrastructure
> code was written using autotest libraries, so although it is possible to make it
> stand alone, still there's the problem of having these library/classes out of
> sync with autotest evolving.
>
>> I don't see a lot of pitfalls here. At the end of the day, we're not
>> talking about all that much code.
>
> Even so (you see it's not so small), it demands effort that has to be well
> justified, as I pointed out. I'm starting to believe it might be worth the effort.

There are two discussions tied into one here so just to summarize:

1) I think we can get more people working on kvm autotest and by extracting and 
making them stand alone.  We can potentially even start merging tests into QEMU 
getting more developers working on kvm autotest tests and mandating new ones for 
certain types of features.

2) kvm autotest focuses on producing guest-OS agnostic tests which require 
infrastructure (Python, ssh, etc.).  qemu-test essentially uses Linux as a libOS 
for writing unit tests.  (1) does not change the need for qemu-test IMHO.

>> It's not about where they live, it's about the ability to execute them
>> in a simple and direct fashion.
>
> Ok, fair enough, we'll see what we can do, although I wish it were a simpler
> problem, see below.
>
>>> You have a point with regards to having the test cases more stand
>>> alone, but
>>> it's not as easy as one would think mainly because of the large amount
>>> of user
>>> configurable options that we have to pass to the tests.
>>
>> Configuration options are really the devil when it comes to testing. As
>> a developer, I want my test suite to just run and tell me if I have
>> bugs. I have no interest in configuring it to run a certain way. I don't
>> want to think that much about what I'm testing, I just want the tests to
>> run and tell me if I screwed something up.
>
> Still, you need to tell your tests, say, the format of the terminal prompt for
> Fedora might be slightly different of the [insert-distro-here], that the src and
> destination paths on a file transfer test are different for Windows and Linux
> guests,

There are various clever things we can do.  Essentially, the problem is that we 
want to deliver a test payload to a guest, right?  Why not create a simple PCI 
device and expose the payload through the bar or something?  There's no need to 
screen scrap a terminal session when we can do something more direct.

> that the output of a given command when you are running a test with a
> virtio card differs from the output when you are using rtl8139, that you want to
> build qemu from git repo X, or from brew build Y, using a custom version of
> seabios, and so on and so forth.

Building qemu should not be tied into the tests themselves.  That's useful for 
regression testing but that just complicates things for development testing.

The building part is somethign that probably should continue to live in autotest.

Regards,

Anthony Liguori

>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Lucas Meneghel Rodrigues]

On 12/29/2011 12:38 PM, Dor Laor wrote:
>  From the recent threads it looks to me that the 2 advantages of
> qemu-test over kvm-autotest are:
>
> 1. python is not used within the guest

^ Just a (relatively small) subset of KVM autotest tests do require 
python in the guest (the ones that execute the autotest client inside 
the guest, virtio-console, and ksm overcommit), so if you don't want py 
on the guest, just execute the tests that do not require it. For 
example, we don't rely on any python for the Windows tests.

> 2. qemu-test is smaller and simpler
>
> Except for the above 2, kvm autotest is superior. My main motivation to
> merge qemu-test and kvm autotest is that I fear that qemu-test will be
> the only requirement for committing stuff for qemu and developers will
> be excused from their 'duty' by writing a 50 LOC shell script and assume
> they done w/ testing. In addition to that, the 50 LOC will be too basic
> and only provide value for basic functionality tests and kvm autotest
> folks will need to rewrite all of the matching content and beyond.

Requiring people to writing tests for qemu-test is already a improvement 
over the current situation. It might make people more interested and 
engaged on test driven development, some of them might want to take the 
time and learn how to use and develop tests for autotest. We gotta start 
with something :)

> Those above 2 advantages can be solve:
>
> 1. python is not used within the guest
> - One option is just to use the same shell code in kvm autotest w/o
> no python code and use the same kernel/initramfs.
> - Another way is to use a plain linux distro w/ python and boot it
> into single user test mode and potentially use a S4 guest snapshot
> or external snapshot to shorten its boot time.
> You'll get faster boot time and friendlier code.

Yes, slowly we are making autotest and the kvm tests faster by disabling 
by default some settings that make more sense to be enabled on automated 
test grids (drop memory caches, take a list of all packages installed on 
the test machine *between* tests, things like that). Hopefully it'll 
improve the experience of using it quite a lot.

Then, on the virt test side, we can find ways to cut the time to start 
the functional tests like you suggested (small, pre-installed images, 
snapshots, you name it). In any case, I believe that qemu-test is 
positive, since it lowers the barrier and gets people interested in 
writing tests. As long as we are careful and avoid overlapping features 
as much as possible, it's fine.

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/28/2011 11:21 AM, Avi Kivity wrote:
> On 12/28/2011 06:42 PM, Anthony Liguori wrote:
>>> In fact using linux as a guest negates that.  First of all, which linux
>>> version? if it's fixed, you'll eventually miss functionality and need to
>>> migrate.  If it keeps changing, so does your test, and it will keep
>>> breaking.
>>
>>
>> The kernel is a git submodule so it's a very specific version.  Yes,
>> we may need to bump the version down the road and obviously, if we
>> have to change any tests in the process, we can.
>
> Having a full linux source as part of the build process detracts
> somewhat from the advantages here.

Except that binaries can be made available.  The initramfs is currently 512k 
while the kernel is about 5MB.

OTOH, a Fedora install runs in the multiple GB.  Having kernels/initramfs for 
the dozen target types that QEMU supports is reasonable.  Having Fedora installs 
for all of them probably isn't for most people.

>> But there's certain things that I still consider to be unit testing
>> (like basic networking tests) that I don't want to have to write with
>> qtest.  I'm not up for writing a TCP/IP stack in Python...
>
> A ping test is not a unit test.
>
> The ping test is already covered by kvm-autotest; just set up a config
> that runs just that; after the initial run you'll have a guest installed
> so it'll be quick.  If we have a DSL guest it'll even be very quick.

Achieving this with kvm-autotest is not easy.  I doubt that DSL has an automated 
install mechanism and the step file based guest installation is fincky at best.

When we had the kvm-autotest day, none of the folks that tried to set it up who 
had never set it up before had something working by the end of the day.  And 
that was with lmr available on IRC to answer questions.

There is a huge setup cost with kvm-autotest.

> To test the various NIC emulations, you don't need a full TCP stack,
> just like you didn't need to write an NTP implementation for qtest's rtc
> test.  Instead you just poke values until it sends out a packet.  If you
> want to test virtio-net with both direct and indirect buffers, you can
> only do that with a unit test, you can't do it using a full linux guest
> since it has its own ideas of when to use indirects and when to avoid
> them (for example, it may choose to always avoid them).

I am not advocating that we don't write proper unit tests...

>> Acceptance testing is, "does Windows boot", "can I create three
>> virtio-serial devices".
>>
>> Obviously, part of acceptance testing is, "does this set of functional
>> tests pass".
>
> Seems like a very blurry line.  Especially as the functional test is
> weaker than either qtest and kvm-autotest.  I now have to agree with the
> others that it's duplicate functionality.  Does it really matter whether
> you're creating an image by compiling Linux and assembling and
> initramfs, or by downloading Fedora.iso and installing it?

Yes.  One of them is built entirely from source and the other is doing something 
meant to be done on bare metal.

How do you test ARM guests with kvm-autotest?  There is no such thing as an ISO 
installer for ARM.  Are we going to download random images from the internet?

Even if there is an ISO, installing a guest with TCG will take many hours.  My 
local qemu-test repo can now fully bootstrap an initramfs for non-x86 targets 
(which involves building gcc and uClibc).  Start to finish it takes about 30 
minutes regardless of the target (since the build runs natively).

> Would you add live migration testing to qemu-test?  If yes, you're
> duplicating some more.  If not, you're not doing functional or coverage
> tests for that functionality.

I would add live migration schema testing, absolutely.  I wouldn't add an 
acceptance test though.

An acceptance test with live migration would be creating a guest, running it, 
and then seeing it live migration worked.

>> I don't want to write a TCP/IP stack.  We aren't just grabbing a
>> random distro kernel.  We're building one from scratch configured in a
>> specific way.
>
> How does that help?

Not sure I understand the question.

Regards,

Anthony Liguori

>> In theory, we could even host the kernel source on git.qemu.org and
>> fork it to add more interesting things in the kernel (although I'd
>> prefer not to do this, obviously).
>
> That way lies madness, though it could be argued that it lies all around us.
>
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 06:26 PM, Anthony Liguori wrote:
> On 12/28/2011 11:21 AM, Avi Kivity wrote:
>> On 12/28/2011 06:42 PM, Anthony Liguori wrote:
>>>> In fact using linux as a guest negates that.  First of all, which
>>>> linux
>>>> version? if it's fixed, you'll eventually miss functionality and
>>>> need to
>>>> migrate.  If it keeps changing, so does your test, and it will keep
>>>> breaking.
>>>
>>>
>>> The kernel is a git submodule so it's a very specific version.  Yes,
>>> we may need to bump the version down the road and obviously, if we
>>> have to change any tests in the process, we can.
>>
>> Having a full linux source as part of the build process detracts
>> somewhat from the advantages here.
>
> Except that binaries can be made available.  The initramfs is
> currently 512k while the kernel is about 5MB.
>
> OTOH, a Fedora install runs in the multiple GB.  Having
> kernels/initramfs for the dozen target types that QEMU supports is
> reasonable.  Having Fedora installs for all of them probably isn't for
> most people.

Then use DSL for the ping test.

>
>>> But there's certain things that I still consider to be unit testing
>>> (like basic networking tests) that I don't want to have to write with
>>> qtest.  I'm not up for writing a TCP/IP stack in Python...
>>
>> A ping test is not a unit test.
>>
>> The ping test is already covered by kvm-autotest; just set up a config
>> that runs just that; after the initial run you'll have a guest installed
>> so it'll be quick.  If we have a DSL guest it'll even be very quick.
>
> Achieving this with kvm-autotest is not easy.  I doubt that DSL has an
> automated install mechanism and the step file based guest installation
> is fincky at best.
>
> When we had the kvm-autotest day, none of the folks that tried to set
> it up who had never set it up before had something working by the end
> of the day.  And that was with lmr available on IRC to answer questions.
>
> There is a huge setup cost with kvm-autotest.

That needs to be fixed, not workarounded.  I previously advised Lucas
not to spend too much time on this, since the audience is limited and it
would be hard to amortize the effort.  Now I see that this was wrong.

>
>> To test the various NIC emulations, you don't need a full TCP stack,
>> just like you didn't need to write an NTP implementation for qtest's rtc
>> test.  Instead you just poke values until it sends out a packet.  If you
>> want to test virtio-net with both direct and indirect buffers, you can
>> only do that with a unit test, you can't do it using a full linux guest
>> since it has its own ideas of when to use indirects and when to avoid
>> them (for example, it may choose to always avoid them).
>
> I am not advocating that we don't write proper unit tests...

Right, and qtest is dangerously close to making writing tests fun.  But
qemu-test is a wierd cross between qtest and kvm-autotest.

>
>>> Acceptance testing is, "does Windows boot", "can I create three
>>> virtio-serial devices".
>>>
>>> Obviously, part of acceptance testing is, "does this set of functional
>>> tests pass".
>>
>> Seems like a very blurry line.  Especially as the functional test is
>> weaker than either qtest and kvm-autotest.  I now have to agree with the
>> others that it's duplicate functionality.  Does it really matter whether
>> you're creating an image by compiling Linux and assembling and
>> initramfs, or by downloading Fedora.iso and installing it?
>
> Yes.  One of them is built entirely from source and the other is doing
> something meant to be done on bare metal.
>
> How do you test ARM guests with kvm-autotest?  There is no such thing
> as an ISO installer for ARM.  Are we going to download random images
> from the internet?

Do exactly what you plan to do with qemu-test.

>
> Even if there is an ISO, installing a guest with TCG will take many
> hours.  My local qemu-test repo can now fully bootstrap an initramfs
> for non-x86 targets (which involves building gcc and uClibc).  Start
> to finish it takes about 30 minutes regardless of the target (since
> the build runs natively).

You can skip the install test and only do the boot test.  Seed your disk
with images from qemu.org or by running the install.  kvm-autotest does
this.

>
>> Would you add live migration testing to qemu-test?  If yes, you're
>> duplicating some more.  If not, you're not doing functional or coverage
>> tests for that functionality.
>
> I would add live migration schema testing, absolutely.  I wouldn't add
> an acceptance test though.
>
> An acceptance test with live migration would be creating a guest,
> running it, and then seeing it live migration worked.

Then it's duplicate functionality.  It's already been done, but the
setup procedure is hard, so let's do it again.

>
>>> I don't want to write a TCP/IP stack.  We aren't just grabbing a
>>> random distro kernel.  We're building one from scratch configured in a
>>> specific way.
>>
>> How does that help?
>
> Not sure I understand the question.
>

In what way is your specifically configured kernel's TCP stack better
than the random distro's kernel's?

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 10:46 AM, Avi Kivity wrote:
> On 12/29/2011 06:26 PM, Anthony Liguori wrote:
>>>> I don't want to write a TCP/IP stack.  We aren't just grabbing a
>>>> random distro kernel.  We're building one from scratch configured in a
>>>> specific way.
>>>
>>> How does that help?
>>
>> Not sure I understand the question.
>>
>
> In what way is your specifically configured kernel's TCP stack better
> than the random distro's kernel's?

I firmly believe that with qtest we'll end up eventually building a libOS to 
make it easier to write qtest tests.

Overtime, that libOS will become increasingly complex up until the point where 
it approaches something that feels like an actual OS.  Effort spent developing 
libOS is a cost to building test cases.

By using Linux and a minimal userspace as our libOS, we can avoid spending a lot 
of time building a sophisticated libOS.  If we need advanced libOS features, we 
just use qemu-test.  If it's just a matter of poking some registers on a device 
along, we just use qtest.

Guest neutral tests that are meant to run on Linux, Windows, etc. are in a 
completely different ballpark.

Regards,

Anthony Liguori

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 06:53 PM, Anthony Liguori wrote:
>> In what way is your specifically configured kernel's TCP stack better
>> than the random distro's kernel's?
>
>
> I firmly believe that with qtest we'll end up eventually building a
> libOS to make it easier to write qtest tests.
>
> Overtime, that libOS will become increasingly complex up until the
> point where it approaches something that feels like an actual OS. 
> Effort spent developing libOS is a cost to building test cases.
>
> By using Linux and a minimal userspace as our libOS, we can avoid
> spending a lot of time building a sophisticated libOS.  If we need
> advanced libOS features, we just use qemu-test.  If it's just a matter
> of poking some registers on a device along, we just use qtest.

Would there be device-level tests in qemu-test?

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 11:08 AM, Avi Kivity wrote:
> On 12/29/2011 06:53 PM, Anthony Liguori wrote:
>>> In what way is your specifically configured kernel's TCP stack better
>>> than the random distro's kernel's?
>>
>>
>> I firmly believe that with qtest we'll end up eventually building a
>> libOS to make it easier to write qtest tests.
>>
>> Overtime, that libOS will become increasingly complex up until the
>> point where it approaches something that feels like an actual OS.
>> Effort spent developing libOS is a cost to building test cases.
>>
>> By using Linux and a minimal userspace as our libOS, we can avoid
>> spending a lot of time building a sophisticated libOS.  If we need
>> advanced libOS features, we just use qemu-test.  If it's just a matter
>> of poking some registers on a device along, we just use qtest.
>
> Would there be device-level tests in qemu-test?

What is a "device-level" test?

Take a look at:

http://git.qemu.org/?p=qemu-test.git;a=blob;f=scripts/bin/fingerprint;h=2d4202a826917b16856a2acb4617f623fdc4c0d3;hb=HEAD

It's reading BAR0 from any virtio devices to determine the guest features 
exposed.  Yes, it's written in sh :-)  It uses the BAR mappings that sysfs exposes.

Regards,

Anthony Liguori

>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 07:14 PM, Anthony Liguori wrote:
> On 12/29/2011 11:08 AM, Avi Kivity wrote:
>> On 12/29/2011 06:53 PM, Anthony Liguori wrote:
>>>> In what way is your specifically configured kernel's TCP stack better
>>>> than the random distro's kernel's?
>>>
>>>
>>> I firmly believe that with qtest we'll end up eventually building a
>>> libOS to make it easier to write qtest tests.
>>>
>>> Overtime, that libOS will become increasingly complex up until the
>>> point where it approaches something that feels like an actual OS.
>>> Effort spent developing libOS is a cost to building test cases.
>>>
>>> By using Linux and a minimal userspace as our libOS, we can avoid
>>> spending a lot of time building a sophisticated libOS.  If we need
>>> advanced libOS features, we just use qemu-test.  If it's just a matter
>>> of poking some registers on a device along, we just use qtest.
>>
>> Would there be device-level tests in qemu-test?
>
> What is a "device-level" test?
>

A test that tests just one device (or a patch to a single device's
device emulation).

> Take a look at:
>
> http://git.qemu.org/?p=qemu-test.git;a=blob;f=scripts/bin/fingerprint;h=2d4202a826917b16856a2acb4617f623fdc4c0d3;hb=HEAD
>
>
> It's reading BAR0 from any virtio devices to determine the guest
> features exposed.  Yes, it's written in sh :-)  

It would be trivial once libos exists.  And we need libos so we can
-ENOTEST device patches, yes?

> It uses the BAR mappings that sysfs exposes.



-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 11:22 AM, Avi Kivity wrote:
> On 12/29/2011 07:14 PM, Anthony Liguori wrote:
>> On 12/29/2011 11:08 AM, Avi Kivity wrote:
>>> On 12/29/2011 06:53 PM, Anthony Liguori wrote:
>>>>> In what way is your specifically configured kernel's TCP stack better
>>>>> than the random distro's kernel's?
>>>>
>>>>
>>>> I firmly believe that with qtest we'll end up eventually building a
>>>> libOS to make it easier to write qtest tests.
>>>>
>>>> Overtime, that libOS will become increasingly complex up until the
>>>> point where it approaches something that feels like an actual OS.
>>>> Effort spent developing libOS is a cost to building test cases.
>>>>
>>>> By using Linux and a minimal userspace as our libOS, we can avoid
>>>> spending a lot of time building a sophisticated libOS.  If we need
>>>> advanced libOS features, we just use qemu-test.  If it's just a matter
>>>> of poking some registers on a device along, we just use qtest.
>>>
>>> Would there be device-level tests in qemu-test?
>>
>> What is a "device-level" test?
>>
>
> A test that tests just one device (or a patch to a single device's
> device emulation).
>
>> Take a look at:
>>
>> http://git.qemu.org/?p=qemu-test.git;a=blob;f=scripts/bin/fingerprint;h=2d4202a826917b16856a2acb4617f623fdc4c0d3;hb=HEAD
>>
>>
>> It's reading BAR0 from any virtio devices to determine the guest
>> features exposed.  Yes, it's written in sh :-)
>
> It would be trivial once libos exists.  And we need libos so we can
> -ENOTEST device patches, yes?

Or qemu-test.  I'd be happy with a test like the virtio-serial test that I 
posted for qemu-test.

I expect we'll have a solid libos for x86 but I doubt we'll ever have a solid 
libos for other targets.

Regards,

Anthony Liguori

>
>> It uses the BAR mappings that sysfs exposes.
>
>
>

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/29/2011 11:08 AM, Avi Kivity wrote:
> On 12/29/2011 06:53 PM, Anthony Liguori wrote:
>>> In what way is your specifically configured kernel's TCP stack better
>>> than the random distro's kernel's?
>>
>>
>> I firmly believe that with qtest we'll end up eventually building a
>> libOS to make it easier to write qtest tests.
>>
>> Overtime, that libOS will become increasingly complex up until the
>> point where it approaches something that feels like an actual OS.
>> Effort spent developing libOS is a cost to building test cases.
>>
>> By using Linux and a minimal userspace as our libOS, we can avoid
>> spending a lot of time building a sophisticated libOS.  If we need
>> advanced libOS features, we just use qemu-test.  If it's just a matter
>> of poking some registers on a device along, we just use qtest.
>
> Would there be device-level tests in qemu-test?

qemu-jeos has a kernel build environment for the target so it is also possible 
to build userspace C programs and/or kernel modules so you could also write 
guest tests in C.

So it may actually be reasonable to write a special virtio-test kernel module 
actually.

Regards,

Anthony Liguori

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 07:16 PM, Anthony Liguori wrote:
>> Would there be device-level tests in qemu-test?
>
>
> qemu-jeos has a kernel build environment for the target so it is also
> possible to build userspace C programs and/or kernel modules so you
> could also write guest tests in C.
>
> So it may actually be reasonable to write a special virtio-test kernel
> module actually.

Better to use vfio or uio for that, IMO.  But qtest is better still.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Christoph Hellwig]

On Mon, Dec 19, 2011 at 11:13:24AM -0600, Anthony Liguori wrote:
> Hi,
>
> I've published a set of tests I wrote over the weekend on qemu.org.  My 
> motivations were 1) to prevent regressions like the libguestfs one and 2) 
> to have an easier way to do development testing as I work on QEMU Object 
> Model.
>
> Now before sending the obligatory, "What about using KVM autotest" reply, 
> note that this is significantly different than KVM autotest and really 
> occupies a different use-case.

Different question:  Do you want to merge qemu-iotests into it?

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/28/2011 08:49 AM, Christoph Hellwig wrote:
> On Mon, Dec 19, 2011 at 11:13:24AM -0600, Anthony Liguori wrote:
>> Hi,
>>
>> I've published a set of tests I wrote over the weekend on qemu.org.  My
>> motivations were 1) to prevent regressions like the libguestfs one and 2)
>> to have an easier way to do development testing as I work on QEMU Object
>> Model.
>>
>> Now before sending the obligatory, "What about using KVM autotest" reply,
>> note that this is significantly different than KVM autotest and really
>> occupies a different use-case.
>
> Different question:  Do you want to merge qemu-iotests into it?

I'd rather just merge it into qemu.git since qemu-io already lives there.

Regards,

Anthony Liguori

>
>