spice in kvm-autotest [was: Re: KVM call minutes for Apr 5]

[Alon Levy <alevy@redhat.com>]

On Tue, Apr 05, 2011 at 08:07:03AM -0700, Chris Wright wrote:
[snip]
> kvm-autotest
> - roadmap...refactor to centralize testing (handle the xen-autotest split off)
> - internally at RH, lmr and cleber maintain autotest server to test
>   branches (testing qemu.git daily)
>   - have good automation for installs and testing
> - seems more QA focused than developers
>   - plenty of benefit for developers, so lack of developer use partly
>     cultural/visibility...
>   - kvm-autotest team always looking for feedback to improve for
>     developer use case
> - kvm-autotest day to have folks use it, write test, give feedback?
>   - startup cost is/was steep, the day might be too much handholding
>   - install-fest? (to get it installed and up and running)
> - buildbot or autotest for testing patches to verify building and working
> - one goal is to reduce mailing list load (patch resubmission because
>   they haven't handled basic cases that buildbot or autotest would have
>   caught)
> - fedora-virt test day coming up on April 14th.  lucas will be on hand and
>   we can piggy back on that to include kvm-autotest install and virt testing
> - kvm autotest run before qemu pull request and post merge to track
>   regressions, more frequent testing helps developers see breakage
>   quickly
>   - qemu.git daily testing already, only the "sanity" test subset 
>     - run more comprehensive "stable" set of tests on weekends
> - one issue is the large number of known failures, need to make these
>   easier to identify (and fix the failures one way or another)
> - create database and verify (regressions) against that
>   - red/yellow/green (yellow shows area was already broken)
> - autotest can be run against server, not just on laptop
> - how to do remote client display testing (e.g. spice client)
>   - dogtail and LDTP
>   - graphics could be tested w/ screenshot compares
> - WHQL testing automated as well

screenshots are already there, and they are a great start. But you can't
really do testing if you aren't recreating the same environment, and having
a client server where there is no client, while being a good test, doesn't
cover the case where the client is connected :)

So I was basically talking about the added requirement of creating a client
connection (one or more) to a single vm.

Note that I wasn't asking anyone to develop this - I'm just asking if patches
in that direction would be accepted/interesting. We (well, Swapna, cc-ed) are
still working on deciding exactly how to do automated testing as a project.

Regarding the dogtail/LDTP issue, they are about specific tests run inside the
guest, and they are certainly something we would leverage. But like I mentioned
on the call, there is a whole suite of whql tests that are display specific,
and don't require anything new. In fact, a few months ago I added support for
autotest to run one of them, resizing to all the possible modes - so I know I
don't need dogtail for significant portions of our testing. (sorry, no git link
- I'll clean it up and post, it's been done 10 months ago so probably won't
cleanly apply :)

For some ideas about what we are interested in see
http://spice-space.org/page/SpiceAutomatedTesting. (just the Requirements section).

> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html