* Re: [Qemu-devel] The state of testing
2013-03-04 16:22 ` Gerd Hoffmann
@ 2013-03-04 17:26 ` mdroth
2013-03-05 9:46 ` Kevin Wolf
` (2 subsequent siblings)
3 siblings, 0 replies; 19+ messages in thread
From: mdroth @ 2013-03-04 17:26 UTC (permalink / raw)
To: Gerd Hoffmann
Cc: Kevin Wolf, Anthony Liguori, Juan Quintela, Stefan Hajnoczi,
qemu-devel, Markus Armbruster, Amit Shah
On Mon, Mar 04, 2013 at 05:22:34PM +0100, Gerd Hoffmann wrote:
> Hi,
>
> > How are things looking with device emulation, migration, monitor, char, etc?
>
> At the moment autotest/virt-test is pretty much the only workable thing
> for non-trivial devices because libqtest lacks infrastructure for pci
> and anything building on top of pci.
>
> usb has no in-tree tests, but has autotest coverage.
>
> chardevs have some autotest coverage, /me wrote a test for
> chardev-{add,remove} qmp commands. Still need to rebase + polish +
> submit that one though.
>
> Migration coverage is pretty bad overall I think.
I've been working on a tool to try to do build-bot-like coverage for
cross-version migration. It's basically a fork of Anthony's qemu-test
suite that uses the same qemu-jeos/initrd system and a single test
harness with pre-migration/post-migration/post-migration reboots tests
case.
The plan is to get it in-tree along with qemu-test, with a wrapper
that'll test current->current migration as part of `make check`, and set up
a test bot with older qemu builds to test cross-version compatibility and
generate a report along these lines:
http://wiki.qemu.org/Migration/Compatibility
I can throw the code up somewhere if anyone is interested, but it's
mostly for internal testing atm.
> What is the state of the idl patch series btw?
The IDL to generate serialization routines for device state is fairly
well-fleshed out, but there was a lot of flux in the code-parser to
process them in later stages of getting the initial code upstreamed, so
I wanted to hold off a bit to refactor the parser and consider some
alternatives (namely, Peter's suggestion of generating device structs
from a higher-level IDL, or leveraging other tools to generate the ASTs)
I should have some cycles to work on it more after the 1.4.1 release.
>
> cheers,
> Gerd
>
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] The state of testing
2013-03-04 16:22 ` Gerd Hoffmann
2013-03-04 17:26 ` mdroth
@ 2013-03-05 9:46 ` Kevin Wolf
2013-03-05 10:46 ` Stefan Hajnoczi
2013-03-05 16:09 ` Lucas Meneghel Rodrigues
3 siblings, 0 replies; 19+ messages in thread
From: Kevin Wolf @ 2013-03-05 9:46 UTC (permalink / raw)
To: Gerd Hoffmann
Cc: Anthony Liguori, Juan Quintela, Stefan Hajnoczi, Michael Roth,
qemu-devel, Amit Shah, Markus Armbruster
Am 04.03.2013 um 17:22 hat Gerd Hoffmann geschrieben:
> Hi,
>
> > How are things looking with device emulation, migration, monitor, char, etc?
>
> At the moment autotest/virt-test is pretty much the only workable thing
> for non-trivial devices because libqtest lacks infrastructure for pci
> and anything building on top of pci.
Yes, this is so true. Anthony when is the current "tomorrow" for libqos
that you told me to wait for when I asked a few months ago? I'd rather
commit _something_ now instead of waiting another year.
My tests that are waiting for libqos currently use some old patches from
Stefan for PCI, and as they have a proper SoB maybe I should just take
them into the block branch if libqos doesn't make progress very soon,
even if they aren't what you want. You can still replace them later.
Kevin
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] The state of testing
2013-03-04 16:22 ` Gerd Hoffmann
2013-03-04 17:26 ` mdroth
2013-03-05 9:46 ` Kevin Wolf
@ 2013-03-05 10:46 ` Stefan Hajnoczi
2013-03-05 11:17 ` Gerd Hoffmann
2013-03-05 16:09 ` Lucas Meneghel Rodrigues
3 siblings, 1 reply; 19+ messages in thread
From: Stefan Hajnoczi @ 2013-03-05 10:46 UTC (permalink / raw)
To: Gerd Hoffmann
Cc: Kevin Wolf, Anthony Liguori, Juan Quintela, Markus Armbruster,
Michael Roth, qemu-devel, Amit Shah
On Mon, Mar 04, 2013 at 05:22:34PM +0100, Gerd Hoffmann wrote:
> Hi,
>
> > How are things looking with device emulation, migration, monitor, char, etc?
>
> At the moment autotest/virt-test is pretty much the only workable thing
> for non-trivial devices because libqtest lacks infrastructure for pci
> and anything building on top of pci.
>
> usb has no in-tree tests, but has autotest coverage.
>
> chardevs have some autotest coverage, /me wrote a test for
> chardev-{add,remove} qmp commands. Still need to rebase + polish +
> submit that one though.
Do the USB and chardev tests run regularly against qemu.git/master?
Stefan
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] The state of testing
2013-03-05 10:46 ` Stefan Hajnoczi
@ 2013-03-05 11:17 ` Gerd Hoffmann
2013-03-05 15:59 ` Lucas Meneghel Rodrigues
0 siblings, 1 reply; 19+ messages in thread
From: Gerd Hoffmann @ 2013-03-05 11:17 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Kevin Wolf, Anthony Liguori, Juan Quintela, Markus Armbruster,
Michael Roth, qemu-devel, Amit Shah
On 03/05/13 11:46, Stefan Hajnoczi wrote:
> On Mon, Mar 04, 2013 at 05:22:34PM +0100, Gerd Hoffmann wrote:
>> Hi,
>>
>>> How are things looking with device emulation, migration, monitor, char, etc?
>>
>> At the moment autotest/virt-test is pretty much the only workable thing
>> for non-trivial devices because libqtest lacks infrastructure for pci
>> and anything building on top of pci.
>>
>> usb has no in-tree tests, but has autotest coverage.
>>
>> chardevs have some autotest coverage, /me wrote a test for
>> chardev-{add,remove} qmp commands. Still need to rebase + polish +
>> submit that one though.
>
> Do the USB and chardev tests run regularly against qemu.git/master?
I don't think we have *any* regular autotest coverage for master, have we?
I run the usb tests locally now and then.
The chardev test was used as development aid, as mentioned I still have
to get it upstream so it can be used for regression testing.
cheers,
Gerd
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] The state of testing
2013-03-05 11:17 ` Gerd Hoffmann
@ 2013-03-05 15:59 ` Lucas Meneghel Rodrigues
2013-03-05 16:14 ` Gerd Hoffmann
0 siblings, 1 reply; 19+ messages in thread
From: Lucas Meneghel Rodrigues @ 2013-03-05 15:59 UTC (permalink / raw)
To: Gerd Hoffmann
Cc: Kevin Wolf, Anthony Liguori, Juan Quintela, qemu-devel,
Stefan Hajnoczi, Michael Roth, Markus Armbruster, Amit Shah
On 03/05/2013 08:17 AM, Gerd Hoffmann wrote:
> On 03/05/13 11:46, Stefan Hajnoczi wrote:
>> On Mon, Mar 04, 2013 at 05:22:34PM +0100, Gerd Hoffmann wrote:
>>> Hi,
>>>
>>>> How are things looking with device emulation, migration, monitor, char, etc?
>>>
>>> At the moment autotest/virt-test is pretty much the only workable thing
>>> for non-trivial devices because libqtest lacks infrastructure for pci
>>> and anything building on top of pci.
>>>
>>> usb has no in-tree tests, but has autotest coverage.
>>>
>>> chardevs have some autotest coverage, /me wrote a test for
>>> chardev-{add,remove} qmp commands. Still need to rebase + polish +
>>> submit that one though.
>>
>> Do the USB and chardev tests run regularly against qemu.git/master?
^ No. We still didn't add usb tests to the sanity/stable test sets, but
it's something doable, we will add them.
> I don't think we have *any* regular autotest coverage for master, have we?
^ Yes, there is coverage. There's a daily job called sanity that takes
about 2.5 hours, that builds the latest kvm.git and installs it in the
host, then reboots the box, builds the latest qemu master and runs:
1) Linux guest install (RHEL 6.3) with the latest virtio drivers build
2) Migration testing
3) Some more basic checks
4) Linux guest install (Windows 7 SP1) with the latest virtio drivers build
5) Migration testing
6) Some more basic checks
There are more extensive test sets that take about 8 hours, that
exercise things like migration with file transfer, timedrift, among others.
This not only runs on master, but RHEL 5.x, 6.x and Fedora 17, 18.
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] The state of testing
2013-03-05 15:59 ` Lucas Meneghel Rodrigues
@ 2013-03-05 16:14 ` Gerd Hoffmann
2013-03-05 16:21 ` Lucas Meneghel Rodrigues
0 siblings, 1 reply; 19+ messages in thread
From: Gerd Hoffmann @ 2013-03-05 16:14 UTC (permalink / raw)
To: Lucas Meneghel Rodrigues
Cc: Kevin Wolf, Anthony Liguori, Juan Quintela, qemu-devel,
Stefan Hajnoczi, Michael Roth, Markus Armbruster, Amit Shah
Hi,
>> I don't think we have *any* regular autotest coverage for master, have
>> we?
>
> ^ Yes, there is coverage. There's a daily job called sanity that takes
> about 2.5 hours, that builds the latest kvm.git and installs it in the
> host, then reboots the box, builds the latest qemu master and runs:
Oh, good. I guess it would be a good idea to send reports to
qemu-devel, maybe a weekly status or mails on regressions simliar to the
buildbot build failures. So people (a) aware that the service exists in
the first place and (b) get informed that something broke which should
be fixed.
cheers,
Gerd
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] The state of testing
2013-03-05 16:14 ` Gerd Hoffmann
@ 2013-03-05 16:21 ` Lucas Meneghel Rodrigues
2013-03-06 9:53 ` Stefan Hajnoczi
0 siblings, 1 reply; 19+ messages in thread
From: Lucas Meneghel Rodrigues @ 2013-03-05 16:21 UTC (permalink / raw)
To: Gerd Hoffmann
Cc: Kevin Wolf, Anthony Liguori, Juan Quintela, Stefan Hajnoczi,
Michael Roth, qemu-devel, Amit Shah, Markus Armbruster
On 03/05/2013 01:14 PM, Gerd Hoffmann wrote:
> Hi,
>
>>> I don't think we have *any* regular autotest coverage for master, have
>>> we?
>>
>> ^ Yes, there is coverage. There's a daily job called sanity that takes
>> about 2.5 hours, that builds the latest kvm.git and installs it in the
>> host, then reboots the box, builds the latest qemu master and runs:
>
> Oh, good. I guess it would be a good idea to send reports to
> qemu-devel, maybe a weekly status or mails on regressions simliar to the
> buildbot build failures. So people (a) aware that the service exists in
> the first place and (b) get informed that something broke which should
> be fixed.
No doubt, I did try that before, but from my experience, the reports are
likely to be ignored and people will complain about SPAM.
So we ended up creating an internal mailing list where Gleb and Marcelo
are subscribed to, in case they want to look at the reports, and I'd
look at the reports and report on the ML in case something bad happens.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] The state of testing
2013-03-05 16:21 ` Lucas Meneghel Rodrigues
@ 2013-03-06 9:53 ` Stefan Hajnoczi
2013-03-06 12:00 ` Markus Armbruster
0 siblings, 1 reply; 19+ messages in thread
From: Stefan Hajnoczi @ 2013-03-06 9:53 UTC (permalink / raw)
To: Lucas Meneghel Rodrigues
Cc: Kevin Wolf, Anthony Liguori, Juan Quintela, Markus Armbruster,
Michael Roth, qemu-devel, Gerd Hoffmann, Amit Shah
On Tue, Mar 05, 2013 at 01:21:18PM -0300, Lucas Meneghel Rodrigues wrote:
> On 03/05/2013 01:14 PM, Gerd Hoffmann wrote:
> > Hi,
> >
> >>>I don't think we have *any* regular autotest coverage for master, have
> >>>we?
> >>
> >>^ Yes, there is coverage. There's a daily job called sanity that takes
> >>about 2.5 hours, that builds the latest kvm.git and installs it in the
> >>host, then reboots the box, builds the latest qemu master and runs:
> >
> >Oh, good. I guess it would be a good idea to send reports to
> >qemu-devel, maybe a weekly status or mails on regressions simliar to the
> >buildbot build failures. So people (a) aware that the service exists in
> >the first place and (b) get informed that something broke which should
> >be fixed.
>
> No doubt, I did try that before, but from my experience, the reports
> are likely to be ignored and people will complain about SPAM.
>
> So we ended up creating an internal mailing list where Gleb and
> Marcelo are subscribed to, in case they want to look at the reports,
> and I'd look at the reports and report on the ML in case something
> bad happens.
I understand that SPAM issue. Maybe only send failures to the list?
The reason why it's important to include qemu-devel@nongnu.org is that
in the current mode you are essentially just reporting bugs. This is
different from providing maintainers test results and failure
notifications so the tests become part of the QEMU development and
release process.
That's an important distinction. Maintainers need to see the test
status (e.g. "99/102 tests passing") and have the ability to dig deeper
- especially during the hard freeze period where we shake out bugs. If
maintainers have no visibility then the tests are not part of the
development process.
The way that buildbot does this is and I think the same is possible for
kvm autotest:
1. Send failure notifications to qemu-devel.
2. Provide a website to check the status
http://buildbot.b1-systems.de/qemu/.
If you want help, please let me know. qemu.org could host
web-accessible test results. We can check with Anthony and I can help
you set it up.
Stefan
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] The state of testing
2013-03-04 16:22 ` Gerd Hoffmann
` (2 preceding siblings ...)
2013-03-05 10:46 ` Stefan Hajnoczi
@ 2013-03-05 16:09 ` Lucas Meneghel Rodrigues
2013-03-05 16:23 ` Gerd Hoffmann
3 siblings, 1 reply; 19+ messages in thread
From: Lucas Meneghel Rodrigues @ 2013-03-05 16:09 UTC (permalink / raw)
To: Gerd Hoffmann
Cc: Kevin Wolf, Anthony Liguori, Juan Quintela, qemu-devel,
Stefan Hajnoczi, Michael Roth, Markus Armbruster, Amit Shah,
Paolo Bonzini
On 03/04/2013 01:22 PM, Gerd Hoffmann wrote:
> Hi,
>
>> How are things looking with device emulation, migration, monitor, char, etc?
>
> At the moment autotest/virt-test is pretty much the only workable thing
> for non-trivial devices because libqtest lacks infrastructure for pci
> and anything building on top of pci.
>
> usb has no in-tree tests, but has autotest coverage.
>
> chardevs have some autotest coverage, /me wrote a test for
> chardev-{add,remove} qmp commands. Still need to rebase + polish +
> submit that one though.
>
> Migration coverage is pretty bad overall I think.
On virt-tests there are at least 48 tests that are easy to run and
involve migration. A subset of them is executed regularly in the daily jobs:
./run -t qemu -g Fedora.18.x86_64 --list-tests | grep migrat | grep -v
multi_host
21 balloon_check.balloon-migrate
51 cpuflags.stress_guest.qemu_test_migration_with_additional_flags
65 migrate.default.tcp
66 migrate.default.unix
67 migrate.default.exec
68 migrate.default.fd
69 migrate.default.mig_cancel
70 migrate.with_speed_measurement.tcp
71 migrate.with_speed_measurement.unix
72 migrate.with_speed_measurement.exec
73 migrate.with_speed_measurement.fd
74 migrate.with_set_speed.tcp
75 migrate.with_set_speed.unix
76 migrate.with_set_speed.exec
77 migrate.with_set_speed.fd
78 migrate.with_reboot.tcp
79 migrate.with_reboot.unix
80 migrate.with_reboot.exec
81 migrate.with_reboot.fd
82 migrate.with_file_transfer.tcp
83 migrate.with_file_transfer.unix
84 migrate.with_file_transfer.exec
85 migrate.with_file_transfer.fd
86 migrate.with_autotest.dbench.tcp
87 migrate.with_autotest.dbench.unix
88 migrate.with_autotest.dbench.exec
89 migrate.with_autotest.dbench.fd
90 migrate.with_autotest.stress.tcp
91 migrate.with_autotest.stress.unix
92 migrate.with_autotest.stress.exec
93 migrate.with_autotest.stress.fd
94 migrate.with_autotest.monotonic_time.tcp
95 migrate.with_autotest.monotonic_time.unix
96 migrate.with_autotest.monotonic_time.exec
97 migrate.with_autotest.monotonic_time.fd
2146 nic_hotplug.migration.nic_8139 (requires root)
2147 nic_hotplug.migration.nic_virtio (requires root)
2148 nic_hotplug.migration.nic_e1000 (requires root)
2483
virtio_console.spread_linear.specifiable.virtserialport.with_vm.migration.offline
2484
virtio_console.spread_linear.specifiable.virtserialport.with_vm.migration.online
2515
virtio_console.spread_linear.specifiable.virtconsole.with_vm.migration.offline
2516
virtio_console.spread_linear.specifiable.virtconsole.with_vm.migration.online
2561
virtio_console.spread_2.specifiable.virtserialport.with_vm.migration.offline
2562
virtio_console.spread_2.specifiable.virtserialport.with_vm.migration.online
2584
virtio_console.spread_2.specifiable.virtconsole.with_vm.migration.offline
2585
virtio_console.spread_2.specifiable.virtconsole.with_vm.migration.online
2672 timedrift.ntp.with_migration
2676 timedrift.date.with_migration
Those can be run with relative ease on every developer laptop, not
counting the many multi host tests involving migration, that are still
not being executed on a regular basis on our test farm. So I don't think
migration coverage is that bad.
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] The state of testing
2013-03-05 16:09 ` Lucas Meneghel Rodrigues
@ 2013-03-05 16:23 ` Gerd Hoffmann
2013-03-05 16:41 ` Lucas Meneghel Rodrigues
0 siblings, 1 reply; 19+ messages in thread
From: Gerd Hoffmann @ 2013-03-05 16:23 UTC (permalink / raw)
To: Lucas Meneghel Rodrigues
Cc: Kevin Wolf, Anthony Liguori, Juan Quintela, qemu-devel,
Stefan Hajnoczi, Michael Roth, Markus Armbruster, Amit Shah,
Paolo Bonzini
Hi,
> On virt-tests there are at least 48 tests that are easy to run and
> involve migration. A subset of them is executed regularly in the daily
> jobs:
Sure, there are a bunch of tests already, which is good.
But there are also important areas which are not covered at all yet.
For example there are no cross-version migration tests, which is an area
where we often have bugs, they often went unnoticed for quite a while
and they tend to be nasty to fix when noticed after release ...
cheers,
Gerd
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] The state of testing
2013-03-05 16:23 ` Gerd Hoffmann
@ 2013-03-05 16:41 ` Lucas Meneghel Rodrigues
0 siblings, 0 replies; 19+ messages in thread
From: Lucas Meneghel Rodrigues @ 2013-03-05 16:41 UTC (permalink / raw)
To: Gerd Hoffmann
Cc: Kevin Wolf, Anthony Liguori, Juan Quintela, Stefan Hajnoczi,
Michael Roth, qemu-devel, Jiri Zupka, Amit Shah, Paolo Bonzini,
Markus Armbruster
On 03/05/2013 01:23 PM, Gerd Hoffmann wrote:
> Hi,
>
>> On virt-tests there are at least 48 tests that are easy to run and
>> involve migration. A subset of them is executed regularly in the daily
>> jobs:
>
> Sure, there are a bunch of tests already, which is good.
> But there are also important areas which are not covered at all yet.
>
> For example there are no cross-version migration tests, which is an area
> where we often have bugs, they often went unnoticed for quite a while
> and they tend to be nasty to fix when noticed after release ...
Ok, I see your point. There are some provisions to implement tests for
cross-version migration due to jzupka's work, but myself and Cleber
could not get around to create multi host jobs in the test grid yet.
^ permalink raw reply [flat|nested] 19+ messages in thread