Re: [ANNOUNCE] kvm-autotest

[Marcelo Tosatti <mtosatti@redhat.com>]

On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
> Hello,
>
>     We are happy to announce the availability of kvm-autotest, a test 
> framework for KVM based on autotest. Naturally, the purpose is to make 
> KVM stable, find/fix bugs faster and prevent regressions. It is to serve 
> developers, to test if their new code breaks anything, as well as users, 
> to verify that the KVM they are using is stable in their own environment.
>
>     There are many things to improve. Currently supported are only a few tests
> (mainly reboot/migration) and a single guest (Fedora 8 i386). This will 
> change soon as we add tests and guests to the test matrix.
>
>     We've tried to be very flexible with configuration of tests, so there are
> quite a few options to choose from (check the documentation). For example
> one may choose to download and install the latest (or a specific) KVM release,
> the latest (or a specific) daily snapshot, use git, or use other source.
> Almost all configuration is located in kvm.cfg (under client directory).
>
>     By default kvm-autotest creates a machine-local subnet with a
> local dhcp server. The dhcp server is configured to ignore unknown clients and
> only serves KVM guests. One may disable it by changing kvm.cfg (See documentation)
>
>     The test framework is based on autotest ( http://test.kernel.org/autotest ).
> Currently we only using client tests, later we may want to use server tests
> for more complicated tests.

This is looking great. Easy to use and fast. A few comments:

- As you mention, it should reuse the server/client model for running
  tests inside guests. I hacked up a "kvm_autotest" test that
  basically does:

tests = ["linus_stress", "bash_shared_mapping", "rmaptest", "tsc",
"scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]

vm.ssh.scp_to_remote(autotest_tarball, '/root')
(s,o) = vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')
for i in range(0, len(tests)):
    (s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
                       'kvm-autotest/client/tests/' + tests[i] +
                       '/control')
    print(o)

Which poorly replicates what the client/server infrastructure already
provides. IMO its a waste of time to write specialized client
tests (other than virt specific ones).

- On top of the client tests, we want to be able to force "special
stress conditions", such as:
    - host memory pressure (to test mmu shrink and mmu notifiers)
    - CPU starvation (lower the priority of the guest process with 
     higher prio CPU hungry tasks running on the host).
    - synthetic cpu migration stress. ChrisW wrote this script sometime
      ago to test clock stability:

     PID=$1
     while :
     do
        for x in 0x01 0x02 0x04 0x08 0x10 0x20 0x40 0x80
        do
            taskset -p $x $PID > /dev/null 2>&1
            usleep 5
        done
     done
    - migration (as it stands now there is no transparent way to schedule 
      client tests to run _while_ migration is in progress, one needs to
      manually change the migration test for that).

- Currently its difficult to debug client test failure inside guests,
  since the VM and its image are destroyed. Perhaps the client/server
  model handles error handling/reporting much more nicely.


Great work!