[Qemu-devel] The state of testing

[Qemu-devel] The state of testing

[Stefan Hajnoczi]

People are working on testing for various parts of QEMU.  I started
this thread to gather an update on the state of testing and see where
we still need help.

I'm not sure yet if we should propose Google Summer of Code projects
to help with the testing infrastructure.  Perhaps working on libqtest
support for PCI, USB, virtio, etc would be a cool project.

Buildbot
------------
The buildbot performs nightly builds and "make check".  Test coverage
is still minimal but it's a starting point where we can add more tests
that will run every day.

Block
--------
tests/qemu-iotests/ contains 49 tests for the block layer.  These
tests mostly cover image format features like backing files,
snapshots, resize, and basic I/O verify tests.

The test suite has a few areas for improvement:

1. aio is not well-supported.  We need to be able to control the order
in which asynchronous requests complete in order to test all possible
code paths.  This issue also means the current aio tests are not
deterministic and can fail randomly due to reordered output.

2. make check-block is not run by the buildbot.  We should do so to
get nightly coverage of basic qcow2 tests.  I will send a buildbot
config patch to fix this.

Net
---
No tests!

How are things looking with device emulation, migration, monitor, char, etc?

Stefan

Re: [Qemu-devel] The state of testing

[Gerd Hoffmann]

  Hi,

> How are things looking with device emulation, migration, monitor, char, etc?

At the moment autotest/virt-test is pretty much the only workable thing
for non-trivial devices because libqtest lacks infrastructure for pci
and anything building on top of pci.

usb has no in-tree tests, but has autotest coverage.

chardevs have some autotest coverage, /me wrote a test for
chardev-{add,remove} qmp commands.  Still need to rebase + polish +
submit that one though.

Migration coverage is pretty bad overall I think.
What is the state of the idl patch series btw?

cheers,
  Gerd

Re: [Qemu-devel] The state of testing

[mdroth]

On Mon, Mar 04, 2013 at 05:22:34PM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > How are things looking with device emulation, migration, monitor, char, etc?
> 
> At the moment autotest/virt-test is pretty much the only workable thing
> for non-trivial devices because libqtest lacks infrastructure for pci
> and anything building on top of pci.
> 
> usb has no in-tree tests, but has autotest coverage.
> 
> chardevs have some autotest coverage, /me wrote a test for
> chardev-{add,remove} qmp commands.  Still need to rebase + polish +
> submit that one though.
> 
> Migration coverage is pretty bad overall I think.

I've been working on a tool to try to do build-bot-like coverage for
cross-version migration. It's basically a fork of Anthony's qemu-test
suite that uses the same qemu-jeos/initrd system and a single test
harness with pre-migration/post-migration/post-migration reboots tests
case.

The plan is to get it in-tree along with qemu-test, with a wrapper
that'll test current->current migration as part of `make check`, and set up
a test bot with older qemu builds to test cross-version compatibility and
generate a report along these lines:

http://wiki.qemu.org/Migration/Compatibility

I can throw the code up somewhere if anyone is interested, but it's
mostly for internal testing atm.

> What is the state of the idl patch series btw?

The IDL to generate serialization routines for device state is fairly
well-fleshed out, but there was a lot of flux in the code-parser to
process them in later stages of getting the initial code upstreamed, so
I wanted to hold off a bit to refactor the parser and consider some
alternatives (namely, Peter's suggestion of generating device structs
from a higher-level IDL, or leveraging other tools to generate the ASTs)

I should have some cycles to work on it more after the 1.4.1 release.

> 
> cheers,
>   Gerd
> 

Re: [Qemu-devel] The state of testing

[Anthony Liguori]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> People are working on testing for various parts of QEMU.  I started
> this thread to gather an update on the state of testing and see where
> we still need help.
>
> I'm not sure yet if we should propose Google Summer of Code projects
> to help with the testing infrastructure.  Perhaps working on libqtest
> support for PCI, USB, virtio, etc would be a cool project.
>
> Buildbot
> ------------
> The buildbot performs nightly builds and "make check".  Test coverage
> is still minimal but it's a starting point where we can add more tests
> that will run every day.
>
> Block
> --------
> tests/qemu-iotests/ contains 49 tests for the block layer.  These
> tests mostly cover image format features like backing files,
> snapshots, resize, and basic I/O verify tests.
>
> The test suite has a few areas for improvement:
>
> 1. aio is not well-supported.  We need to be able to control the order
> in which asynchronous requests complete in order to test all possible
> code paths.  This issue also means the current aio tests are not
> deterministic and can fail randomly due to reordered output.
>
> 2. make check-block is not run by the buildbot.  We should do so to
> get nightly coverage of basic qcow2 tests.  I will send a buildbot
> config patch to fix this.
>
> Net
> ---
> No tests!

qemu-test has a ping test that runs against rtl8139, e1000, virtio,
ne2k_pci, i82551, i82557b, and i82559er.

It also tests hotplug add/remove of network adapters with a variety of
command line combinations.

There's quite a few more things in qemu-test too.

Regards,

Anthony Liguori

>
> How are things looking with device emulation, migration, monitor, char, etc?
>
> Stefan

Re: [Qemu-devel] The state of testing

[Kevin Wolf]

Am 04.03.2013 um 17:22 hat Gerd Hoffmann geschrieben:
>   Hi,
> 
> > How are things looking with device emulation, migration, monitor, char, etc?
> 
> At the moment autotest/virt-test is pretty much the only workable thing
> for non-trivial devices because libqtest lacks infrastructure for pci
> and anything building on top of pci.

Yes, this is so true. Anthony when is the current "tomorrow" for libqos
that you told me to wait for when I asked a few months ago? I'd rather
commit _something_ now instead of waiting another year.

My tests that are waiting for libqos currently use some old patches from
Stefan for PCI, and as they have a proper SoB maybe I should just take
them into the block branch if libqos doesn't make progress very soon,
even if they aren't what you want. You can still replace them later.

Kevin

Re: [Qemu-devel] The state of testing

[Amit Shah]

On (Mon) 04 Mar 2013 [16:53:55], Stefan Hajnoczi wrote:
> People are working on testing for various parts of QEMU.  I started
> this thread to gather an update on the state of testing and see where
> we still need help.
> 
> I'm not sure yet if we should propose Google Summer of Code projects
> to help with the testing infrastructure.  Perhaps working on libqtest
> support for PCI, USB, virtio, etc would be a cool project.
> 
> Buildbot
> ------------
> The buildbot performs nightly builds and "make check".  Test coverage
> is still minimal but it's a starting point where we can add more tests
> that will run every day.
> 
> Block
> --------
> tests/qemu-iotests/ contains 49 tests for the block layer.  These
> tests mostly cover image format features like backing files,
> snapshots, resize, and basic I/O verify tests.
> 
> The test suite has a few areas for improvement:
> 
> 1. aio is not well-supported.  We need to be able to control the order
> in which asynchronous requests complete in order to test all possible
> code paths.  This issue also means the current aio tests are not
> deterministic and can fail randomly due to reordered output.
> 
> 2. make check-block is not run by the buildbot.  We should do so to
> get nightly coverage of basic qcow2 tests.  I will send a buildbot
> config patch to fix this.
> 
> Net
> ---
> No tests!
> 
> How are things looking with device emulation, migration, monitor, char, etc?

I have a few tests coded up for virtio-serial at

http://fedorapeople.org/cgit/amitshah/public_git/test-virtserial.git/

which indirectly test chardevs.

Autotest (virt-tests) also has virtio-serial tests which end up
testing chardevs.

chardevs, being an external interface to qemu, do get tested whenever
anyone starts qemu with e.g. libvirt.

Also, autotest has libvirt-based tests, which exercise the monitor.
However, I don't think there's anything comprehensive anywhere.


		Amit

Re: [Qemu-devel] The state of testing

[Stefan Hajnoczi]

On Mon, Mar 04, 2013 at 01:20:03PM -0600, Anthony Liguori wrote:
> Stefan Hajnoczi <stefanha@gmail.com> writes:
> 
> > People are working on testing for various parts of QEMU.  I started
> > this thread to gather an update on the state of testing and see where
> > we still need help.
> >
> > I'm not sure yet if we should propose Google Summer of Code projects
> > to help with the testing infrastructure.  Perhaps working on libqtest
> > support for PCI, USB, virtio, etc would be a cool project.
> >
> > Buildbot
> > ------------
> > The buildbot performs nightly builds and "make check".  Test coverage
> > is still minimal but it's a starting point where we can add more tests
> > that will run every day.
> >
> > Block
> > --------
> > tests/qemu-iotests/ contains 49 tests for the block layer.  These
> > tests mostly cover image format features like backing files,
> > snapshots, resize, and basic I/O verify tests.
> >
> > The test suite has a few areas for improvement:
> >
> > 1. aio is not well-supported.  We need to be able to control the order
> > in which asynchronous requests complete in order to test all possible
> > code paths.  This issue also means the current aio tests are not
> > deterministic and can fail randomly due to reordered output.
> >
> > 2. make check-block is not run by the buildbot.  We should do so to
> > get nightly coverage of basic qcow2 tests.  I will send a buildbot
> > config patch to fix this.
> >
> > Net
> > ---
> > No tests!
> 
> qemu-test has a ping test that runs against rtl8139, e1000, virtio,
> ne2k_pci, i82551, i82557b, and i82559er.
> 
> It also tests hotplug add/remove of network adapters with a variety of
> command line combinations.
> 
> There's quite a few more things in qemu-test too.

Great, would it help to integrate qemu-test into the buildbot or do you
already have infrastructure to run it regularly on qemu.git/master?

Stefan

Re: [Qemu-devel] The state of testing

[Stefan Hajnoczi]

On Mon, Mar 04, 2013 at 05:22:34PM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > How are things looking with device emulation, migration, monitor, char, etc?
> 
> At the moment autotest/virt-test is pretty much the only workable thing
> for non-trivial devices because libqtest lacks infrastructure for pci
> and anything building on top of pci.
> 
> usb has no in-tree tests, but has autotest coverage.
> 
> chardevs have some autotest coverage, /me wrote a test for
> chardev-{add,remove} qmp commands.  Still need to rebase + polish +
> submit that one though.

Do the USB and chardev tests run regularly against qemu.git/master?

Stefan

Re: [Qemu-devel] The state of testing

[Gerd Hoffmann]

On 03/05/13 11:46, Stefan Hajnoczi wrote:
> On Mon, Mar 04, 2013 at 05:22:34PM +0100, Gerd Hoffmann wrote:
>>   Hi,
>>
>>> How are things looking with device emulation, migration, monitor, char, etc?
>>
>> At the moment autotest/virt-test is pretty much the only workable thing
>> for non-trivial devices because libqtest lacks infrastructure for pci
>> and anything building on top of pci.
>>
>> usb has no in-tree tests, but has autotest coverage.
>>
>> chardevs have some autotest coverage, /me wrote a test for
>> chardev-{add,remove} qmp commands.  Still need to rebase + polish +
>> submit that one though.
> 
> Do the USB and chardev tests run regularly against qemu.git/master?

I don't think we have *any* regular autotest coverage for master, have we?

I run the usb tests locally now and then.

The chardev test was used as development aid, as mentioned I still have
to get it upstream so it can be used for regression testing.

cheers,
  Gerd

Re: [Qemu-devel] The state of testing

[Lucas Meneghel Rodrigues]

On 03/05/2013 07:11 AM, Amit Shah wrote:
> On (Mon) 04 Mar 2013 [16:53:55], Stefan Hajnoczi wrote:
>> People are working on testing for various parts of QEMU.  I started
>> this thread to gather an update on the state of testing and see where
>> we still need help.
>>
>> I'm not sure yet if we should propose Google Summer of Code projects
>> to help with the testing infrastructure.  Perhaps working on libqtest
>> support for PCI, USB, virtio, etc would be a cool project.
>>
>> Buildbot
>> ------------
>> The buildbot performs nightly builds and "make check".  Test coverage
>> is still minimal but it's a starting point where we can add more tests
>> that will run every day.
>>
>> Block
>> --------
>> tests/qemu-iotests/ contains 49 tests for the block layer.  These
>> tests mostly cover image format features like backing files,
>> snapshots, resize, and basic I/O verify tests.
>>
>> The test suite has a few areas for improvement:
>>
>> 1. aio is not well-supported.  We need to be able to control the order
>> in which asynchronous requests complete in order to test all possible
>> code paths.  This issue also means the current aio tests are not
>> deterministic and can fail randomly due to reordered output.
>>
>> 2. make check-block is not run by the buildbot.  We should do so to
>> get nightly coverage of basic qcow2 tests.  I will send a buildbot
>> config patch to fix this.
>>
>> Net
>> ---
>> No tests!
>>
>> How are things looking with device emulation, migration, monitor, char, etc?
>
> I have a few tests coded up for virtio-serial at
>
> http://fedorapeople.org/cgit/amitshah/public_git/test-virtserial.git/
>
> which indirectly test chardevs.
>
> Autotest (virt-tests) also has virtio-serial tests which end up
> testing chardevs.

> chardevs, being an external interface to qemu, do get tested whenever
> anyone starts qemu with e.g. libvirt.
>
> Also, autotest has libvirt-based tests, which exercise the monitor.
> However, I don't think there's anything comprehensive anywhere.

The virtio console tests span over 130 tests, and take ~4 hours to run, 
I'd consider they are comprehensive.

Re: [Qemu-devel] The state of testing

[Lucas Meneghel Rodrigues]

On 03/05/2013 08:17 AM, Gerd Hoffmann wrote:
> On 03/05/13 11:46, Stefan Hajnoczi wrote:
>> On Mon, Mar 04, 2013 at 05:22:34PM +0100, Gerd Hoffmann wrote:
>>>    Hi,
>>>
>>>> How are things looking with device emulation, migration, monitor, char, etc?
>>>
>>> At the moment autotest/virt-test is pretty much the only workable thing
>>> for non-trivial devices because libqtest lacks infrastructure for pci
>>> and anything building on top of pci.
>>>
>>> usb has no in-tree tests, but has autotest coverage.
>>>
>>> chardevs have some autotest coverage, /me wrote a test for
>>> chardev-{add,remove} qmp commands.  Still need to rebase + polish +
>>> submit that one though.
>>
>> Do the USB and chardev tests run regularly against qemu.git/master?

^ No. We still didn't add usb tests to the sanity/stable test sets, but 
it's something doable, we will add them.

> I don't think we have *any* regular autotest coverage for master, have we?

^ Yes, there is coverage. There's a daily job called sanity that takes 
about 2.5 hours, that builds the latest kvm.git and installs it in the 
host, then reboots the box, builds the latest qemu master and runs:

1) Linux guest install (RHEL 6.3) with the latest virtio drivers build
2) Migration testing
3) Some more basic checks

4) Linux guest install (Windows 7 SP1) with the latest virtio drivers build
5) Migration testing
6) Some more basic checks

There are more extensive test sets that take about 8 hours, that 
exercise things like migration with file transfer, timedrift, among others.

This not only runs on master, but RHEL 5.x, 6.x and Fedora 17, 18.

Re: [Qemu-devel] The state of testing

[Lucas Meneghel Rodrigues]

On 03/04/2013 01:22 PM, Gerd Hoffmann wrote:
>    Hi,
>
>> How are things looking with device emulation, migration, monitor, char, etc?
>
> At the moment autotest/virt-test is pretty much the only workable thing
> for non-trivial devices because libqtest lacks infrastructure for pci
> and anything building on top of pci.
>
> usb has no in-tree tests, but has autotest coverage.
>
> chardevs have some autotest coverage, /me wrote a test for
> chardev-{add,remove} qmp commands.  Still need to rebase + polish +
> submit that one though.
>
> Migration coverage is pretty bad overall I think.

On virt-tests there are at least 48 tests that are easy to run and 
involve migration. A subset of them is executed regularly in the daily jobs:

./run -t qemu -g Fedora.18.x86_64 --list-tests | grep migrat | grep -v 
multi_host

21 balloon_check.balloon-migrate
51 cpuflags.stress_guest.qemu_test_migration_with_additional_flags
65 migrate.default.tcp
66 migrate.default.unix
67 migrate.default.exec
68 migrate.default.fd
69 migrate.default.mig_cancel
70 migrate.with_speed_measurement.tcp
71 migrate.with_speed_measurement.unix
72 migrate.with_speed_measurement.exec
73 migrate.with_speed_measurement.fd
74 migrate.with_set_speed.tcp
75 migrate.with_set_speed.unix
76 migrate.with_set_speed.exec
77 migrate.with_set_speed.fd
78 migrate.with_reboot.tcp
79 migrate.with_reboot.unix
80 migrate.with_reboot.exec
81 migrate.with_reboot.fd
82 migrate.with_file_transfer.tcp
83 migrate.with_file_transfer.unix
84 migrate.with_file_transfer.exec
85 migrate.with_file_transfer.fd
86 migrate.with_autotest.dbench.tcp
87 migrate.with_autotest.dbench.unix
88 migrate.with_autotest.dbench.exec
89 migrate.with_autotest.dbench.fd
90 migrate.with_autotest.stress.tcp
91 migrate.with_autotest.stress.unix
92 migrate.with_autotest.stress.exec
93 migrate.with_autotest.stress.fd
94 migrate.with_autotest.monotonic_time.tcp
95 migrate.with_autotest.monotonic_time.unix
96 migrate.with_autotest.monotonic_time.exec
97 migrate.with_autotest.monotonic_time.fd
2146 nic_hotplug.migration.nic_8139 (requires root)
2147 nic_hotplug.migration.nic_virtio (requires root)
2148 nic_hotplug.migration.nic_e1000 (requires root)
2483 
virtio_console.spread_linear.specifiable.virtserialport.with_vm.migration.offline
2484 
virtio_console.spread_linear.specifiable.virtserialport.with_vm.migration.online
2515 
virtio_console.spread_linear.specifiable.virtconsole.with_vm.migration.offline
2516 
virtio_console.spread_linear.specifiable.virtconsole.with_vm.migration.online
2561 
virtio_console.spread_2.specifiable.virtserialport.with_vm.migration.offline
2562 
virtio_console.spread_2.specifiable.virtserialport.with_vm.migration.online
2584 
virtio_console.spread_2.specifiable.virtconsole.with_vm.migration.offline
2585 
virtio_console.spread_2.specifiable.virtconsole.with_vm.migration.online
2672 timedrift.ntp.with_migration
2676 timedrift.date.with_migration

Those can be run with relative ease on every developer laptop, not 
counting the many multi host tests involving migration, that are still 
not being executed on a regular basis on our test farm. So I don't think 
migration coverage is that bad.

Re: [Qemu-devel] The state of testing

[Gerd Hoffmann]

  Hi,

>> I don't think we have *any* regular autotest coverage for master, have
>> we?
> 
> ^ Yes, there is coverage. There's a daily job called sanity that takes
> about 2.5 hours, that builds the latest kvm.git and installs it in the
> host, then reboots the box, builds the latest qemu master and runs:

Oh, good.  I guess it would be a good idea to send reports to
qemu-devel, maybe a weekly status or mails on regressions simliar to the
buildbot build failures.  So people (a) aware that the service exists in
the first place and (b) get informed that something broke which should
be fixed.

cheers,
  Gerd

Re: [Qemu-devel] The state of testing

[Lucas Meneghel Rodrigues]

On 03/05/2013 01:14 PM, Gerd Hoffmann wrote:
>    Hi,
>
>>> I don't think we have *any* regular autotest coverage for master, have
>>> we?
>>
>> ^ Yes, there is coverage. There's a daily job called sanity that takes
>> about 2.5 hours, that builds the latest kvm.git and installs it in the
>> host, then reboots the box, builds the latest qemu master and runs:
>
> Oh, good.  I guess it would be a good idea to send reports to
> qemu-devel, maybe a weekly status or mails on regressions simliar to the
> buildbot build failures.  So people (a) aware that the service exists in
> the first place and (b) get informed that something broke which should
> be fixed.

No doubt, I did try that before, but from my experience, the reports are 
likely to be ignored and people will complain about SPAM.

So we ended up creating an internal mailing list where Gleb and Marcelo 
are subscribed to, in case they want to look at the reports, and I'd 
look at the reports and report on the ML in case something bad happens.

Re: [Qemu-devel] The state of testing

[Gerd Hoffmann]

  Hi,

> On virt-tests there are at least 48 tests that are easy to run and
> involve migration. A subset of them is executed regularly in the daily
> jobs:

Sure, there are a bunch of tests already, which is good.
But there are also important areas which are not covered at all yet.

For example there are no cross-version migration tests, which is an area
where we often have bugs, they often went unnoticed for quite a while
and they tend to be nasty to fix when noticed after release ...

cheers,
  Gerd

Re: [Qemu-devel] The state of testing

[Lucas Meneghel Rodrigues]

On 03/05/2013 01:23 PM, Gerd Hoffmann wrote:
>    Hi,
>
>> On virt-tests there are at least 48 tests that are easy to run and
>> involve migration. A subset of them is executed regularly in the daily
>> jobs:
>
> Sure, there are a bunch of tests already, which is good.
> But there are also important areas which are not covered at all yet.
>
> For example there are no cross-version migration tests, which is an area
> where we often have bugs, they often went unnoticed for quite a while
> and they tend to be nasty to fix when noticed after release ...

Ok, I see your point. There are some provisions to implement tests for 
cross-version migration due to jzupka's work, but myself and Cleber 
could not get around to create multi host jobs in the test grid yet.

Re: [Qemu-devel] The state of testing

[Stefan Hajnoczi]

On Tue, Mar 05, 2013 at 01:21:18PM -0300, Lucas Meneghel Rodrigues wrote:
> On 03/05/2013 01:14 PM, Gerd Hoffmann wrote:
> >   Hi,
> >
> >>>I don't think we have *any* regular autotest coverage for master, have
> >>>we?
> >>
> >>^ Yes, there is coverage. There's a daily job called sanity that takes
> >>about 2.5 hours, that builds the latest kvm.git and installs it in the
> >>host, then reboots the box, builds the latest qemu master and runs:
> >
> >Oh, good.  I guess it would be a good idea to send reports to
> >qemu-devel, maybe a weekly status or mails on regressions simliar to the
> >buildbot build failures.  So people (a) aware that the service exists in
> >the first place and (b) get informed that something broke which should
> >be fixed.
> 
> No doubt, I did try that before, but from my experience, the reports
> are likely to be ignored and people will complain about SPAM.
> 
> So we ended up creating an internal mailing list where Gleb and
> Marcelo are subscribed to, in case they want to look at the reports,
> and I'd look at the reports and report on the ML in case something
> bad happens.

I understand that SPAM issue.  Maybe only send failures to the list?

The reason why it's important to include qemu-devel@nongnu.org is that
in the current mode you are essentially just reporting bugs.  This is
different from providing maintainers test results and failure
notifications so the tests become part of the QEMU development and
release process.

That's an important distinction.  Maintainers need to see the test
status (e.g. "99/102 tests passing") and have the ability to dig deeper
- especially during the hard freeze period where we shake out bugs.  If
maintainers have no visibility then the tests are not part of the
development process.

The way that buildbot does this is and I think the same is possible for
kvm autotest:
1. Send failure notifications to qemu-devel.
2. Provide a website to check the status
http://buildbot.b1-systems.de/qemu/.

If you want help, please let me know.  qemu.org could host
web-accessible test results.  We can check with Anthony and I can help
you set it up.

Stefan

Re: [Qemu-devel] The state of testing

[Markus Armbruster]

Stefan Hajnoczi <stefanha@gmail.com> writes:

[...]
> The way that buildbot does this is and I think the same is possible for
> kvm autotest:
> 1. Send failure notifications to qemu-devel.
> 2. Provide a website to check the status
> http://buildbot.b1-systems.de/qemu/.

+1

[...]

Re: [Qemu-devel] The state of testing

[Amit Shah]

On (Tue) 05 Mar 2013 [12:54:06], Lucas Meneghel Rodrigues wrote:
> On 03/05/2013 07:11 AM, Amit Shah wrote:
> >On (Mon) 04 Mar 2013 [16:53:55], Stefan Hajnoczi wrote:

> >>How are things looking with device emulation, migration, monitor, char, etc?
> >
> >I have a few tests coded up for virtio-serial at
> >
> >http://fedorapeople.org/cgit/amitshah/public_git/test-virtserial.git/
> >
> >which indirectly test chardevs.
> >
> >Autotest (virt-tests) also has virtio-serial tests which end up
> >testing chardevs.
> 
> >chardevs, being an external interface to qemu, do get tested whenever
> >anyone starts qemu with e.g. libvirt.
> >
> >Also, autotest has libvirt-based tests, which exercise the monitor.
> >However, I don't think there's anything comprehensive anywhere.
> 
> The virtio console tests span over 130 tests, and take ~4 hours to
> run, I'd consider they are comprehensive.

It's a commendable job the folks who've put together the tests have
done.

However, it's still not comprehensive for the chardevs themselves.
For example, guest freezes for lack of flow control would've been
flagged by autotest otherwise.  Though I'm happy to now note that will
no longer be an issue on qemu.git, and we must now add tests for
checking flow control.

		Amit