Re: [ANNOUNCE] kvm-autotest

[Uri Lublin <uril@qumranet.com>]

Ryan Harper wrote:
> * Uri Lublin <uril@qumranet.com> [2008-07-14 20:26]:
>> Ryan Harper wrote:
>>> * Uri Lublin <uril@qumranet.com> [2008-07-10 07:42]:
>>>> Marcelo Tosatti wrote:
>>>>> On Sun, Jul 06, 2008 at 01:16:13PM +0300, Uri Lublin wrote:
>>>>>
>>>>> - As you mention, it should reuse the server/client model for running
>>>>> tests inside guests. I hacked up a "kvm_autotest" test that
>>>>> basically does:
>>>>>
>>>>> tests = ["linus_stress", "bash_shared_mapping", "rmaptest", "tsc",
>>>>> "scrashme", "isic", "sleeptest", "libhugetlbfs", "..."]
>>>>>
>>>>> vm.ssh.scp_to_remote(autotest_tarball, '/root')
>>>>> (s,o) = vm.ssh.ssh('tar zvxf kvm-autotest.tar.gz')
>>>>> for i in range(0, len(tests)):
>>>>>   (s,o) = vm.ssh.ssh('kvm-autotest/client/bin/autotest ' +
>>>>>                      'kvm-autotest/client/tests/' + tests[i] +
>>>>>                      '/control')
>>>>>   print(o)
>>>>>
>>>>> Which poorly replicates what the client/server infrastructure already
>>>>> provides. IMO its a waste of time to write specialized client
>>>>> tests (other than virt specific ones).
>>>>>
>>>> You see guests as clients and the host as the server.
>>>> We were thinking of the host as a client and multi-host operations to be 
>>>> done by a server. guest-operations would be done using ssh (for linux 
>>>> guests) as your example above. You make a good point that we can use 
>>>> server/client infrastructure for guest operations. As it is simpler to 
>>>> write autotest client tests, and we thought most of the tests would be 
>>>> run as client tests, we want to postpone the server tests and focus on 
>>>> adding tests and guests to the matrix.
>>> It's definitely worth looking at the autotest server code/samples.
>>> There exists code in-tree already to build an deploy kvm via autotest
>>> server mode which a single machine can drive the building, installing,
>>> creation of guests on N number of clients, directing each guest
>>> image to run various autotest client tests, collecting all of the
>>> results.
>>>
>>> See autotest/server/samples/*kvm*
>>>
>>> A proper server setup is a little involved[1] but much more streamlined
>>> these days.
>>>
>> Let's think of a guest-installation test. Would you implement it on the 
>> server or on the client ?
>> What do you plan for non-linux guests ?
> 
> Client side, for installation, we already have a solution that works for
> all types of guests:
> 
> http://kvm.qumranet.com/kvmwiki/KVMTest
>

> which is already integrated as a client test in autotest.  Once you
> record your installation via kvmtest, then it is just  matter for
> keeping the iso and an empty disk image around and replaying the
> installation with -snapshot.
> 

So guest installation is a client test. KVMtest has its own way of
managing/booting/communicating-with guests, and naturally does not use
KVM/KVMGuest classes of autotest server.

How about the test, suggested by Marcelo/Chris, of changing physical cpu of a VM
using taskset. Would that be a client test or a server test ?
What about stop/cont, save/restore ?

I think most kvm-tests will be client tests.

More complex tests, which involve multiple hosts, such as migration
between two hosts, should be server tests.

Managing all those tests, can be done by autoserv.

> 
> 
> Now, I'm actually more interested in doing the following:
> 
> use kvmtest to replay an installation of a guest and instead of throwing
> the guest away once (running with -snapshot) it has passes the install,
> it is now ready to be used to execute autotest client tests or something
> else.

autotest client tests can be consider as "guest load" which usually does is 
orthogonal to the "real" kvm-test that is running (e.g. migration-test while 
watching a movie, or migration-test while building the kernel). Also what would 
you do for non-linux guest ?

> 
>> We'll try this little exercise of writing a kvm-test on the server side and 
>> on the client side and compare complexity.
> 
> That's a bit vague, what sort of test are you talking about?  If you
> mean installation, i'm not interested since that's been handled by
> KVMTest.
Are you actually running autotest tests with KVMTest installed guests? Do you 
have to manually exchange ssh-keys.

> As to complexity, I urge you to look at the existing kvm
> examples[2] in the autotest server dir, those look pretty darn simple to me
> and already include all of the infrastructure for capturing console
> logs, results and errors.

It does look simple. It was written for a different purpose though, which is to 
run autotest tests on guests, not to run kvm tests.

> 
> 
> Oh, I forgot my pointer to the server setup last time:
> 
> 1. http://test.kernel.org/autotest/AutotestServerInstall
> 2. autotest/server/samples/kvm.srv
> 
> 


So what do you propose ?
We've were thinking today how we can move things to the server and want to get 
your (or anyone's) opinion. Does the server always starts (boots) guests ? Do 
most tests run on the server-side (similar to [2]) or may client tests start 
guests (similar to KVMTest and migration)? If both how would you synchronize 
between them. If some tests run on the client side and only the server starts 
guests, how do those tests access the guest. Do you assume the guest images 
always exist before running the tests or do you create them automatically if 
needed/requested.

Thanks,
Uri