[ANNOUNCE] kvm-autotest

[ANNOUNCE] kvm-autotest

[Uri Lublin]

Hello,

     We are happy to announce the availability of kvm-autotest, a test framework 
for KVM based on autotest. Naturally, the purpose is to make KVM stable, 
find/fix bugs faster and prevent regressions. It is to serve developers, to test 
if their new code breaks anything, as well as users, to verify that the KVM they 
are using is stable in their own environment.

     There are many things to improve. Currently supported are only a few tests
(mainly reboot/migration) and a single guest (Fedora 8 i386). This will change 
soon as we add tests and guests to the test matrix.

     We've tried to be very flexible with configuration of tests, so there are
quite a few options to choose from (check the documentation). For example
one may choose to download and install the latest (or a specific) KVM release,
the latest (or a specific) daily snapshot, use git, or use other source.
Almost all configuration is located in kvm.cfg (under client directory).

     By default kvm-autotest creates a machine-local subnet with a
local dhcp server. The dhcp server is configured to ignore unknown clients and
only serves KVM guests. One may disable it by changing kvm.cfg (See documentation)

     The test framework is based on autotest ( http://test.kernel.org/autotest ).
Currently we only using client tests, later we may want to use server tests
for more complicated tests.

To download the source:
     cd /usr/local ( or another root-writeable directory )
     git clone git://git.kernel.org/pub/scm/virt/kvm/kvm-autotest.git
     cd kvm-autotest/client
To read documentation look at http://kvm.qumranet.com/kvmwiki/KVM_RegressionTest

The simplest way to use it is:
0. login as root (not necessary but simpler, as tests must be run as root)
1. Download kvm-autotest (see a few lines above)
2. Change the following in kvm.cfg according to your preference (or leave the 
default settings):
        local_kvm_dir   -- "base"   directory
        local_guest_dir -- "images" directroy (may take a lot of disk space)
        local_iso_dir   -- "isos"   directroy (may take a lot of disk space)
        bridge          -- change if your local network conflicts with
                               192.168.1/24 subnet
                           change bridge name if conflicts with a currently used
                               bridge.
        dhcp_net        -- change accordingly (e.g. use 10.0.1.2)
3. Make sure you have enough disk space for isos and images (df -h /var/local)
4. The default guest-installation test requires Fedora-8-i386-DVD.iso.
    It is configured to fetch it automatically from the web unless it already
      exists in the local_iso_dir (default path is
      /var/local/kvm_regression/isos/linux/Fedora-8-i386-DVD.iso).
    If you already have that iso image locally, create a link or change
      local_iso_dir.
    Alternatively, in tests/kvm_guest_install/Fedora-8-i386/Fedora-8-i386.ini
      set iso=/path/to/Fedora-8-i386-DVD.iso.
5. Default machine-local subnet is 192.168.1/24. If it conflicts with your
    network, you need change network configuration in kvm.cfg
6. Run the tests as root (./bin/autotest tests/kvm_runtest/control)
7. Checkout the results at results/default/

Caveats/Issues:
- Upon failure check results/defualt/debug/error-*
- After first download of KVM sources (and building + loading the modules),
     one needs to either manually remove /var/local/kvm_regression/{src,build}
     or set kvm_install=no in kvm.cfg. Otherwise the test will fail.
     The reason for it is that we do not want to remove directories
     automatically.
- The message "Guest installation completed but guest is not responsive",
     usually means that the guest installation failed.
- The tests must be run as root.
- When a guest is installing, do not move its mouse or type its keyboard.
   It confuses the guest installation algorithm (when watching its progress, e.g.
   with vncviewer).
- Guest installation is currently very sensitive to timing. If installation
   fails, sometimes it's helpfull to enlarge sleep time before the reboot and
   between steps. Do it by modifying
   tests/kvm_guest_install/Fedora-8-i386/Fedora-8-i386.steps
- After the guest is installed and you want to rerun the other tests set
     guest_install=no in kvm.cfg
   Also if you want to test an existing image, create a new vm-class (guest
   configuration) and set guest_install=no in kvm.cfg

     We hope many will test KVM on their systems and report results to the
community. We plan to have a web page with results. We hope many will add tests 
and improve KVM regression tests.

Thanks,
     Uri.

Re: [ANNOUNCE] kvm-autotest

[Marcelo Tosatti]

On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
> Hello,
>
>     We are happy to announce the availability of kvm-autotest, a test 
> framework for KVM based on autotest. Naturally, the purpose is to make 
> KVM stable, find/fix bugs faster and prevent regressions. It is to serve 
> developers, to test if their new code breaks anything, as well as users, 
> to verify that the KVM they are using is stable in their own environment.
>
>     There are many things to improve. Currently supported are only a few tests
> (mainly reboot/migration) and a single guest (Fedora 8 i386). This will 
> change soon as we add tests and guests to the test matrix.
>
>     We've tried to be very flexible with configuration of tests, so there are
> quite a few options to choose from (check the documentation). For example
> one may choose to download and install the latest (or a specific) KVM release,
> the latest (or a specific) daily snapshot, use git, or use other source.
> Almost all configuration is located in kvm.cfg (under client directory).
>
>     By default kvm-autotest creates a machine-local subnet with a
> local dhcp server. The dhcp server is configured to ignore unknown clients and
> only serves KVM guests. One may disable it by changing kvm.cfg (See documentation)
>
>     The test framework is based on autotest ( http://test.kernel.org/autotest ).
> Currently we only using client tests, later we may want to use server tests
> for more complicated tests.

This is looking great. Easy to use and fast. A few comments:

- As you mention, it should reuse the server/client model for running
  tests inside guests. I hacked up a "kvm_autotest" test that
  basically does:

tests = ["linus_stress", "bash_shared_mapping", "rmaptest", "tsc",
"scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]

vm.ssh.scp_to_remote(autotest_tarball, '/root')
(s,o) = vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')
for i in range(0, len(tests)):
    (s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
                       'kvm-autotest/client/tests/' + tests[i] +
                       '/control')
    print(o)

Which poorly replicates what the client/server infrastructure already
provides. IMO its a waste of time to write specialized client
tests (other than virt specific ones).

- On top of the client tests, we want to be able to force "special
stress conditions", such as:
    - host memory pressure (to test mmu shrink and mmu notifiers)
    - CPU starvation (lower the priority of the guest process with 
     higher prio CPU hungry tasks running on the host).
    - synthetic cpu migration stress. ChrisW wrote this script sometime
      ago to test clock stability:

     PID=$1
     while :
     do
        for x in 0x01 0x02 0x04 0x08 0x10 0x20 0x40 0x80
        do
            taskset -p $x $PID > /dev/null 2>&1
            usleep 5
        done
     done
    - migration (as it stands now there is no transparent way to schedule 
      client tests to run _while_ migration is in progress, one needs to
      manually change the migration test for that).

- Currently its difficult to debug client test failure inside guests,
  since the VM and its image are destroyed. Perhaps the client/server
  model handles error handling/reporting much more nicely.


Great work!

Re: [ANNOUNCE] kvm-autotest

[Uri Lublin]

Marcelo Tosatti wrote:
> On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
>>
>>     The test framework is based on autotest ( http://test.kernel.org/autotest ).
>> Currently we only using client tests, later we may want to use server tests
>> for more complicated tests.
> 
> This is looking great. Easy to use and fast. A few comments:
> 
> - As you mention, it should reuse the server/client model for running
>   tests inside guests. I hacked up a "kvm_autotest" test that
>   basically does:
> 
> tests = ["linus_stress", "bash_shared_mapping", "rmaptest", "tsc",
> "scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
> 
> vm.ssh.scp_to_remote(autotest_tarball, '/root')
> (s,o) = vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')
> for i in range(0, len(tests)):
>     (s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
>                        'kvm-autotest/client/tests/' + tests[i] +
>                        '/control')
>     print(o)
> 
> Which poorly replicates what the client/server infrastructure already
> provides. IMO its a waste of time to write specialized client
> tests (other than virt specific ones).
> 

You see guests as clients and the host as the server.
We were thinking of the host as a client and multi-host operations to be done by 
a server. guest-operations would be done using ssh (for linux guests) as your 
example above. You make a good point that we can use server/client 
infrastructure for guest operations. As it is simpler to write autotest client 
tests, and we thought most of the tests would be run as client tests, we want to 
postpone the server tests and focus on adding tests and guests to the matrix.

> - On top of the client tests, we want to be able to force "special
> stress conditions", such as:
>     - host memory pressure (to test mmu shrink and mmu notifiers)
Similar is how many guests (or what to memory over-subscription percentage) we 
can start while all of them are responsive.
>     - CPU starvation (lower the priority of the guest process with 
>      higher prio CPU hungry tasks running on the host).
>     - synthetic cpu migration stress. ChrisW wrote this script sometime
>       ago to test clock stability:
> 
>      PID=$1
>      while :
>      do
>         for x in 0x01 0x02 0x04 0x08 0x10 0x20 0x40 0x80
>         do
>             taskset -p $x $PID > /dev/null 2>&1
>             usleep 5
>         done
>      done
>     - migration (as it stands now there is no transparent way to schedule 
>       client tests to run _while_ migration is in progress, one needs to
>       manually change the migration test for that).

You probably do not need to change the migration test. I think we need to run 
some load on the guest (we'd probably have many load options, and the user will 
choose/configure which one to use) before migration starts and keep it running 
during the migration process.

> 
> - Currently its difficult to debug client test failure inside guests,
>   since the VM and its image are destroyed. Perhaps the client/server
>   model handles error handling/reporting much more nicely.

Agreed, we thought of that, but it's not cooked yet. Currently we always 
cleanup. We should not remove the (temporary) image if the test fails. Should we 
keep the guest running upon failure ? Currently we continue to test the next 
guest. We should probably have a configuration flag for that too.

Thanks,
Uri.

Re: [ANNOUNCE] kvm-autotest

[Ryan Harper]

* Uri Lublin <uril@qumranet.com> [2008-07-10 07:42]:
> Marcelo Tosatti wrote:
> >On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
> >>
> >>    The test framework is based on autotest ( 
> >>    http://test.kernel.org/autotest ).
> >>Currently we only using client tests, later we may want to use server 
> >>tests
> >>for more complicated tests.
> >
> >This is looking great. Easy to use and fast. A few comments:
> >
> >- As you mention, it should reuse the server/client model for running
> >  tests inside guests. I hacked up a "kvm_autotest" test that
> >  basically does:
> >
> >tests = ["linus_stress", "bash_shared_mapping", "rmaptest", "tsc",
> >"scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
> >
> >vm.ssh.scp_to_remote(autotest_tarball, '/root')
> >(s,o) = vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')
> >for i in range(0, len(tests)):
> >    (s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
> >                       'kvm-autotest/client/tests/' + tests[i] +
> >                       '/control')
> >    print(o)
> >
> >Which poorly replicates what the client/server infrastructure already
> >provides. IMO its a waste of time to write specialized client
> >tests (other than virt specific ones).
> >
> 
> You see guests as clients and the host as the server.
> We were thinking of the host as a client and multi-host operations to be 
> done by a server. guest-operations would be done using ssh (for linux 
> guests) as your example above. You make a good point that we can use 
> server/client infrastructure for guest operations. As it is simpler to 
> write autotest client tests, and we thought most of the tests would be run 
> as client tests, we want to postpone the server tests and focus on adding 
> tests and guests to the matrix.

It's definitely worth looking at the autotest server code/samples.
There exists code in-tree already to build an deploy kvm via autotest
server mode which a single machine can drive the building, installing,
creation of guests on N number of clients, directing each guest
image to run various autotest client tests, collecting all of the
results.

See autotest/server/samples/*kvm*

A proper server setup is a little involved[1] but much more streamlined
these days.

> You probably do not need to change the migration test. I think we need to 
> run some load on the guest (we'd probably have many load options, and the 
> user will choose/configure which one to use) before migration starts and 
> keep it running during the migration process.
> 
> >
> >- Currently its difficult to debug client test failure inside guests,
> >  since the VM and its image are destroyed. Perhaps the client/server
> >  model handles error handling/reporting much more nicely.
> 
> Agreed, we thought of that, but it's not cooked yet. Currently we always 
> cleanup. We should not remove the (temporary) image if the test fails. 
> Should we keep the guest running upon failure ? Currently we continue to 
> test the next guest. We should probably have a configuration flag for that 
> too.

I suppose it depends on the failure, if we're capturing the boot log
then for boot failures, I don't see any reason to keep running.  If we
succeed in booting but fail at some further point, then I think it makes
sense to keep them running.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [ANNOUNCE] kvm-autotest

[Uri Lublin]

Ryan Harper wrote:
> * Uri Lublin <uril@qumranet.com> [2008-07-10 07:42]:
>> Marcelo Tosatti wrote:
>>> On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
>>>
>>> - As you mention, it should reuse the server/client model for running
>>>  tests inside guests. I hacked up a "kvm_autotest" test that
>>>  basically does:
>>>
>>> tests = ["linus_stress", "bash_shared_mapping", "rmaptest", "tsc",
>>> "scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
>>>
>>> vm.ssh.scp_to_remote(autotest_tarball, '/root')
>>> (s,o) = vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')
>>> for i in range(0, len(tests)):
>>>    (s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
>>>                       'kvm-autotest/client/tests/' + tests[i] +
>>>                       '/control')
>>>    print(o)
>>>
>>> Which poorly replicates what the client/server infrastructure already
>>> provides. IMO its a waste of time to write specialized client
>>> tests (other than virt specific ones).
>>>
>> You see guests as clients and the host as the server.
>> We were thinking of the host as a client and multi-host operations to be 
>> done by a server. guest-operations would be done using ssh (for linux 
>> guests) as your example above. You make a good point that we can use 
>> server/client infrastructure for guest operations. As it is simpler to 
>> write autotest client tests, and we thought most of the tests would be run 
>> as client tests, we want to postpone the server tests and focus on adding 
>> tests and guests to the matrix.
> 
> It's definitely worth looking at the autotest server code/samples.
> There exists code in-tree already to build an deploy kvm via autotest
> server mode which a single machine can drive the building, installing,
> creation of guests on N number of clients, directing each guest
> image to run various autotest client tests, collecting all of the
> results.
> 
> See autotest/server/samples/*kvm*
> 
> A proper server setup is a little involved[1] but much more streamlined
> these days.
> 

Let's think of a guest-installation test. Would you implement it on the server 
or on the client ?
What do you plan for non-linux guests ?

We'll try this little exercise of writing a kvm-test on the server side and on 
the client side and compare complexity.

Thanks,
Uri.

Re: [ANNOUNCE] kvm-autotest

[Dor Laor]

>>
>> It's definitely worth looking at the autotest server code/samples.
>> There exists code in-tree already to build an deploy kvm via autotest
>> server mode which a single machine can drive the building, installing,
>> creation of guests on N number of clients, directing each guest
>> image to run various autotest client tests, collecting all of the
>> results.
>>
>> See autotest/server/samples/*kvm*
>>
>> A proper server setup is a little involved[1] but much more streamlined
>> these days.
>>
>
> Let's think of a guest-installation test. Would you implement it on 
> the server or on the client ?
> What do you plan for non-linux guests ?
>
> We'll try this little exercise of writing a kvm-test on the server 
> side and on the client side and compare complexity.
>
> Thanks,
> Uri.
IMHO we need a mixture:
- kvm/environment setup
  autoserve tests/deploy
- Internal guest tests
  Implemented as client test, executed from the server. Composed of 
benchmarks, standard functionality, applications, unit tests, etc.
- guest installation, guest boot
  client  test that  execute on the kvm host

Regards,
Dor

Re: [ANNOUNCE] kvm-autotest

[Ryan Harper]

* Uri Lublin <uril@qumranet.com> [2008-07-14 20:26]:
> Ryan Harper wrote:
> >* Uri Lublin <uril@qumranet.com> [2008-07-10 07:42]:
> >>Marcelo Tosatti wrote:
> >>>On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
> >>>
> >>>- As you mention, it should reuse the server/client model for running
> >>> tests inside guests. I hacked up a "kvm_autotest" test that
> >>> basically does:
> >>>
> >>>tests = ["linus_stress", "bash_shared_mapping", "rmaptest", "tsc",
> >>>"scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
> >>>
> >>>vm.ssh.scp_to_remote(autotest_tarball, '/root')
> >>>(s,o) = vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')
> >>>for i in range(0, len(tests)):
> >>>   (s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
> >>>                      'kvm-autotest/client/tests/' + tests[i] +
> >>>                      '/control')
> >>>   print(o)
> >>>
> >>>Which poorly replicates what the client/server infrastructure already
> >>>provides. IMO its a waste of time to write specialized client
> >>>tests (other than virt specific ones).
> >>>
> >>You see guests as clients and the host as the server.
> >>We were thinking of the host as a client and multi-host operations to be 
> >>done by a server. guest-operations would be done using ssh (for linux 
> >>guests) as your example above. You make a good point that we can use 
> >>server/client infrastructure for guest operations. As it is simpler to 
> >>write autotest client tests, and we thought most of the tests would be 
> >>run as client tests, we want to postpone the server tests and focus on 
> >>adding tests and guests to the matrix.
> >
> >It's definitely worth looking at the autotest server code/samples.
> >There exists code in-tree already to build an deploy kvm via autotest
> >server mode which a single machine can drive the building, installing,
> >creation of guests on N number of clients, directing each guest
> >image to run various autotest client tests, collecting all of the
> >results.
> >
> >See autotest/server/samples/*kvm*
> >
> >A proper server setup is a little involved[1] but much more streamlined
> >these days.
> >
> 
> Let's think of a guest-installation test. Would you implement it on the 
> server or on the client ?
> What do you plan for non-linux guests ?

Client side, for installation, we already have a solution that works for
all types of guests:

http://kvm.qumranet.com/kvmwiki/KVMTest


which is already integrated as a client test in autotest.  Once you
record your installation via kvmtest, then it is just  matter for
keeping the iso and an empty disk image around and replaying the
installation with -snapshot.

Now, I'm actually more interested in doing the following:

use kvmtest to replay an installation of a guest and instead of throwing
the guest away once (running with -snapshot) it has passes the install,
it is now ready to be used to execute autotest client tests or something
else.

> 
> We'll try this little exercise of writing a kvm-test on the server side and 
> on the client side and compare complexity.

That's a bit vague, what sort of test are you talking about?  If you
mean installation, i'm not interested since that's been handled by
KVMTest.  As to complexity, I urge you to look at the existing kvm
examples[2] in the autotest server dir, those look pretty darn simple to me
and already include all of the infrastructure for capturing console
logs, results and errors.


Oh, I forgot my pointer to the server setup last time:

1. http://test.kernel.org/autotest/AutotestServerInstall
2. autotest/server/samples/kvm.srv


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [ANNOUNCE] kvm-autotest

[Uri Lublin]

Ryan Harper wrote:
> * Uri Lublin <uril@qumranet.com> [2008-07-14 20:26]:
>> Ryan Harper wrote:
>>> * Uri Lublin <uril@qumranet.com> [2008-07-10 07:42]:
>>>> Marcelo Tosatti wrote:
>>>>> On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
>>>>>
>>>>> - As you mention, it should reuse the server/client model for running
>>>>> tests inside guests. I hacked up a "kvm_autotest" test that
>>>>> basically does:
>>>>>
>>>>> tests = ["linus_stress", "bash_shared_mapping", "rmaptest", "tsc",
>>>>> "scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
>>>>>
>>>>> vm.ssh.scp_to_remote(autotest_tarball, '/root')
>>>>> (s,o) = vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')
>>>>> for i in range(0, len(tests)):
>>>>>   (s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
>>>>>                      'kvm-autotest/client/tests/' + tests[i] +
>>>>>                      '/control')
>>>>>   print(o)
>>>>>
>>>>> Which poorly replicates what the client/server infrastructure already
>>>>> provides. IMO its a waste of time to write specialized client
>>>>> tests (other than virt specific ones).
>>>>>
>>>> You see guests as clients and the host as the server.
>>>> We were thinking of the host as a client and multi-host operations to be 
>>>> done by a server. guest-operations would be done using ssh (for linux 
>>>> guests) as your example above. You make a good point that we can use 
>>>> server/client infrastructure for guest operations. As it is simpler to 
>>>> write autotest client tests, and we thought most of the tests would be 
>>>> run as client tests, we want to postpone the server tests and focus on 
>>>> adding tests and guests to the matrix.
>>> It's definitely worth looking at the autotest server code/samples.
>>> There exists code in-tree already to build an deploy kvm via autotest
>>> server mode which a single machine can drive the building, installing,
>>> creation of guests on N number of clients, directing each guest
>>> image to run various autotest client tests, collecting all of the
>>> results.
>>>
>>> See autotest/server/samples/*kvm*
>>>
>>> A proper server setup is a little involved[1] but much more streamlined
>>> these days.
>>>
>> Let's think of a guest-installation test. Would you implement it on the 
>> server or on the client ?
>> What do you plan for non-linux guests ?
> 
> Client side, for installation, we already have a solution that works for
> all types of guests:
> 
> http://kvm.qumranet.com/kvmwiki/KVMTest
>

> which is already integrated as a client test in autotest.  Once you
> record your installation via kvmtest, then it is just  matter for
> keeping the iso and an empty disk image around and replaying the
> installation with -snapshot.
> 

So guest installation is a client test. KVMtest has its own way of
managing/booting/communicating-with guests, and naturally does not use
KVM/KVMGuest classes of autotest server.

How about the test, suggested by Marcelo/Chris, of changing physical cpu of a VM
using taskset. Would that be a client test or a server test ?
What about stop/cont, save/restore ?

I think most kvm-tests will be client tests.

More complex tests, which involve multiple hosts, such as migration
between two hosts, should be server tests.

Managing all those tests, can be done by autoserv.

> 
> 
> Now, I'm actually more interested in doing the following:
> 
> use kvmtest to replay an installation of a guest and instead of throwing
> the guest away once (running with -snapshot) it has passes the install,
> it is now ready to be used to execute autotest client tests or something
> else.

autotest client tests can be consider as "guest load" which usually does is 
orthogonal to the "real" kvm-test that is running (e.g. migration-test while 
watching a movie, or migration-test while building the kernel). Also what would 
you do for non-linux guest ?

> 
>> We'll try this little exercise of writing a kvm-test on the server side and 
>> on the client side and compare complexity.
> 
> That's a bit vague, what sort of test are you talking about?  If you
> mean installation, i'm not interested since that's been handled by
> KVMTest.
Are you actually running autotest tests with KVMTest installed guests? Do you 
have to manually exchange ssh-keys.

> As to complexity, I urge you to look at the existing kvm
> examples[2] in the autotest server dir, those look pretty darn simple to me
> and already include all of the infrastructure for capturing console
> logs, results and errors.

It does look simple. It was written for a different purpose though, which is to 
run autotest tests on guests, not to run kvm tests.

> 
> 
> Oh, I forgot my pointer to the server setup last time:
> 
> 1. http://test.kernel.org/autotest/AutotestServerInstall
> 2. autotest/server/samples/kvm.srv
> 
> 


So what do you propose ?
We've were thinking today how we can move things to the server and want to get 
your (or anyone's) opinion. Does the server always starts (boots) guests ? Do 
most tests run on the server-side (similar to [2]) or may client tests start 
guests (similar to KVMTest and migration)? If both how would you synchronize 
between them. If some tests run on the client side and only the server starts 
guests, how do those tests access the guest. Do you assume the guest images 
always exist before running the tests or do you create them automatically if 
needed/requested.

Thanks,
Uri

Re: [ANNOUNCE] kvm-autotest

[Ryan Harper]

* Uri Lublin <uril@qumranet.com> [2008-07-16 18:15]:
> >Client side, for installation, we already have a solution that works for
> >all types of guests:
> >
> >http://kvm.qumranet.com/kvmwiki/KVMTest
> >
> 
> >which is already integrated as a client test in autotest.  Once you
> >record your installation via kvmtest, then it is just  matter for
> >keeping the iso and an empty disk image around and replaying the
> >installation with -snapshot.
> >
> 
> So guest installation is a client test. KVMtest has its own way of
> managing/booting/communicating-with guests, and naturally does not use
> KVM/KVMGuest classes of autotest server.

It does not use the KVM/KVMGuest classes in autotest server precisely
because it is a client test, there is no need to do anything
server-side w.r.t KVMTest.  We ensure that we've built and installed
whatever version of kvm on the target machine and that the target
machine has access to require inputs (iso,disk image) and invoke
KVMTest.

> 
> How about the test, suggested by Marcelo/Chris, of changing physical cpu of 
> a VM
> using taskset. Would that be a client test or a server test ?
> What about stop/cont, save/restore ?
> 
> I think most kvm-tests will be client tests.

Agreed, the above examples will be client tests.  Autotest client can do
parallel execution, or even step-wise.  Between those, we should be able
to ensure we get proper coverage for the envisioned scenarios.

> 
> More complex tests, which involve multiple hosts, such as migration
> between two hosts, should be server tests.
> 
> Managing all those tests, can be done by autoserv.

Agreed.

> 
> >
> >
> >Now, I'm actually more interested in doing the following:
> >
> >use kvmtest to replay an installation of a guest and instead of throwing
> >the guest away once (running with -snapshot) it has passes the install,
> >it is now ready to be used to execute autotest client tests or something
> >else.
> 
> autotest client tests can be consider as "guest load" which usually does is 
> orthogonal to the "real" kvm-test that is running (e.g. migration-test 

Definitely.  I just want to utilize autotest to drive both guest
creation and test orchestration which includes installation as well as
testing known working guests in various scenarios and yes autotest
simplifies generating "guest load".

> while watching a movie, or migration-test while building the kernel). Also 
> what would you do for non-linux guest ?

I'm not quite sure what to do here since non-linux guests isn't really
in my scope beyond simple tests (installation, shutdown, reboot,
configuration variance).  

> 
> >
> >>We'll try this little exercise of writing a kvm-test on the server side 
> >>and on the client side and compare complexity.
> >
> >That's a bit vague, what sort of test are you talking about?  If you
> >mean installation, i'm not interested since that's been handled by
> >KVMTest.
> Are you actually running autotest tests with KVMTest installed guests? Do 

Yes, but the transition between using KVMtest to install aguest and
running autotest inside isn't completely automated -- yet.

> you have to manually exchange ssh-keys.

Yeah, we'll need a solution, but it should be pretty simple to automate,
Mostly likely pregenerate a key and serve it up to the guest via cdrom
image and then install that into the guest. 


> >As to complexity, I urge you to look at the existing kvm
u
> >examples[2] in the autotest server dir, those look pretty darn simple to me
> >and already include all of the infrastructure for capturing console
> >logs, results and errors.
> 
> It does look simple. It was written for a different purpose though, which 
> is to run autotest tests on guests, not to run kvm tests.

Sure, but that doesn't mean it isn't a good infrastructure on top of
which we can build kvm testing.

> 
> >
> >
> >Oh, I forgot my pointer to the server setup last time:
> >
> >1. http://test.kernel.org/autotest/AutotestServerInstall
> >2. autotest/server/samples/kvm.srv
> >
> >
> 
> 
> So what do you propose ?
> We've were thinking today how we can move things to the server and want to 
> get your (or anyone's) opinion. Does the server always starts (boots) 
> guests ? Do most tests run on the server-side (similar to [2]) or may 

Yeah, let me explain a bit more about how the server and clients work:

You'll have one master server which runs the job monitor, it will look
for autotest server files (.srv) and that file will run on the server,
but you will write your srv file to execute on a set of machines that
your master server manages.  Autotest server maintains a db of machines.

Looking at autotest/server/samples/sleeptest.srv:


def run(machine):
       host = hosts.SSHHost(machine)
       at = autotest.Autotest(host)
       at.run_test('sleeptest')

job.parallel_simple(run, machines)

We're defining a run function, and then running that across all the
machines in the grid.  

The other interesting one to look at is netperf-guest-to-host-far.srv.

That file demonstrates installing kvm to differnt host machines (not the
server where the .srv file is running); on each host machine,
create a KVM guest, and then on each host, creating a KVM guest, in
each guest, running a netperf test between each guest.

> client tests start guests (similar to KVMTest and migration)? If both how 
> would you synchronize between them. If some tests run on the client side 

Synchronization is done via barriers, see
autotest/client/tests/netperf2/netperf2.py and
autotest/client/common_lib/barrier.py for more details.

> and only the server starts guests, how do those tests access the guest. Do 
> you assume the guest images always exist before running the tests or do you 
> create them automatically if needed/requested.

For multi-machine setup, either we'd need to create the image as needed,
or you'll want some sort of shared storage mounted on each host that
you want to run kvm tests upon.

And obviously, shared storage is need as soon as we put migration into
the mix.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [ANNOUNCE] kvm-autotest

[Uri Lublin]

Ryan Harper wrote:
> * Uri Lublin <uril@qumranet.com> [2008-07-16 18:15]:
>>
>> I think most kvm-tests will be client tests.
> 
> Agreed, the above examples will be client tests.  Autotest client can do
> parallel execution, or even step-wise.  Between those, we should be able
> to ensure we get proper coverage for the envisioned scenarios.
> 
>> More complex tests, which involve multiple hosts, such as migration
>> between two hosts, should be server tests.
>>
>> Managing all those tests, can be done by autoserv.
> 
> Agreed.
> 

We all agree that most kvm tests are client-side tests. The server manages 
multiple hosts and complex tests (such as multi-host migration). The server also 
may run autotest tests on linux guests. We consider kvm-install, kvm-boot, 
kvm-guest-install as client tests.

Following your comments our plan is to move the control to the server size:
The server executes a .srv file (or a server side control). It calls 
client-tests (including simple building blocks such as kvm-installation, 
boot/destroy a KVM-VM, etc), providing guest id and some specific tests 
parameters. Guests are always created on the client (similar to current 
kvm-autotest behaviour) and a client-side class that knows how to map guest-id 
with a running guest (including  guest-ip) will be used by all kvm client tests.

Accessing a linux guest (e.g. running autotest tests) can be done from the 
server simply by using hosts.SSHHost just like any other linux host.
This will only work for global dhcp (no local host-only bridge) or if server and 
host are the same.

Client tests can also be started from the client side using client-side control 
files.

We do not know yet what we'll do with non-linux guests, but we sure want to test 
them too.

Thanks,
Uri

Re: [ANNOUNCE] kvm-autotest

[Marcelo Tosatti]

Hi Uri,

On Thu, Jul 10, 2008 at 03:09:36PM +0300, Uri Lublin wrote:
> You see guests as clients and the host as the server.
> We were thinking of the host as a client and multi-host operations to be 
> done by a server. guest-operations would be done using ssh (for linux 
> guests) as your example above. You make a good point that we can use 
> server/client infrastructure for guest operations. As it is simpler to 
> write autotest client tests, and we thought most of the tests would be 
> run as client tests, we want to postpone the server tests and focus on 
> adding tests and guests to the matrix.

Perhaps you need a main server on top for that, to coordinate various
hosts.

The big problem with writing client tests is that you have to rewrite
what autotest already provides, diverging from mainline. That means work
to be done everytime a new test is added in autotest, which could come
for free if guests were treated as clients and the host treated as a
server.

And the other way around is also true: new OS tests added to
kvm-autotest will not make their way naturally into autotest mainline.

Other than Windows tests, of course, but most Linux OS tests should run
fine in *NIX.

>> - On top of the client tests, we want to be able to force "special
>> stress conditions", such as:
>>     - host memory pressure (to test mmu shrink and mmu notifiers)
> Similar is how many guests (or what to memory over-subscription 
> percentage) we can start while all of them are responsive.
>>     - CPU starvation (lower the priority of the guest process with      
>> higher prio CPU hungry tasks running on the host).
>>     - synthetic cpu migration stress. ChrisW wrote this script sometime
>>       ago to test clock stability:
>>
>>      PID=$1
>>      while :
>>      do
>>         for x in 0x01 0x02 0x04 0x08 0x10 0x20 0x40 0x80
>>         do
>>             taskset -p $x $PID > /dev/null 2>&1
>>             usleep 5
>>         done
>>      done
>>     - migration (as it stands now there is no transparent way to 
>> schedule       client tests to run _while_ migration is in progress, 
>> one needs to
>>       manually change the migration test for that).
>
> You probably do not need to change the migration test. I think we need to 
> run some load on the guest (we'd probably have many load options, and the 
> user will choose/configure which one to use) before migration starts and 
> keep it running during the migration process.

Right.

>> - Currently its difficult to debug client test failure inside guests,
>>   since the VM and its image are destroyed. Perhaps the client/server
>>   model handles error handling/reporting much more nicely.
>
> Agreed, we thought of that, but it's not cooked yet. Currently we always  
> cleanup. We should not remove the (temporary) image if the test fails. 
> Should we keep the guest running upon failure ? Currently we continue to 
> test the next guest. We should probably have a configuration flag for 
> that too.

Yes, it should be dependant on a configuration flag. For developer
use, stopping the run makes sense so the failure can be analyzed. For
stress/coverage, you want to know results of the entire test battery.

Re: [ANNOUNCE] kvm-autotest

[Uri Lublin]

Marcelo Tosatti wrote:
> On Thu, Jul 10, 2008 at 03:09:36PM +0300, Uri Lublin wrote:
> 
> The big problem with writing client tests is that you have to rewrite
> what autotest already provides, diverging from mainline. That means work
> to be done everytime a new test is added in autotest, which could come
> for free if guests were treated as clients and the host treated as a
> server.

We can have a generic function like
run-on-guest(vm, run_path_on_guest, hdir, gdir, hresult, gresult) which
- copies a directory from host to guest
- runs a script on the guest (we may add an option to pass params to the script)
- copies results directory from guest to host

That would work for any autotest test (the run script will run bin/autotest, 
newly added tests will be added to the run script, or be passed as args) and 
should work for non-Linux guests.

Look at server/tests and client/tests; Most of the tests reside in the client side.

> 
> And the other way around is also true: new OS tests added to
> kvm-autotest will not make their way naturally into autotest mainline.

They may be accepted as client test(s). If we put all tests under client/kvm_* 
or even better all tests under client/kvm_runtest, than it would be just another 
(massive) client test.

> 
> Other than Windows tests, of course, but most Linux OS tests should run
> fine in *NIX.
> 
I'm not yet sure how we are going to implement access to windows-kvm-guest but 
we sure want to have windows guests in our test matrix.


Basically I don't mind trying the server-side solution.
We can have a little exercise, trying to write a kvm-test on the server and on 
the client and compare the effort.
I think we should mostly focus on adding tests and guests.


>>> - Currently its difficult to debug client test failure inside guests,
>>>   since the VM and its image are destroyed. Perhaps the client/server
>>>   model handles error handling/reporting much more nicely.
>> Agreed, we thought of that, but it's not cooked yet. Currently we always  
>> cleanup. We should not remove the (temporary) image if the test fails. 
>> Should we keep the guest running upon failure ? Currently we continue to 
>> test the next guest. We should probably have a configuration flag for 
>> that too.
> 
> Yes, it should be dependant on a configuration flag. For developer
> use, stopping the run makes sense so the failure can be analyzed. For
> stress/coverage, you want to know results of the entire test battery.

We'll add such a flag.

Thanks,
Uri