kvm-autotest: The automation plans?

kvm-autotest: The automation plans?

[sudhir kumar]

Hi Uri/Lucas,

Do you have any plans for enhancing kvm-autotest?
I was looking mainly on the following 2 aspects:

(1).
we have standalone migration only. Is there any plans of enhancing
kvm-autotest so that we can trigger migration while a workload is
running?
Something like this:
Start a workload(may be n instances of it).
let the test execute for some time.
Trigger migration.
Log into the target.
Check if the migration is succesful
Check if the test results are consistent.

(2).
How can we run N parallel instances of a test? Will the current
configuration  be easily able to support it?

Please provide your thoughts on the above features.

-- 
Sudhir Kumar

Re: kvm-autotest: The automation plans?

[Michael Goldish]

----- "sudhir kumar" <smalikphy@gmail.com> wrote:

> Hi Uri/Lucas,
> 
> Do you have any plans for enhancing kvm-autotest?
> I was looking mainly on the following 2 aspects:
> 
> (1).
> we have standalone migration only. Is there any plans of enhancing
> kvm-autotest so that we can trigger migration while a workload is
> running?
> Something like this:
> Start a workload(may be n instances of it).
> let the test execute for some time.
> Trigger migration.
> Log into the target.
> Check if the migration is succesful
> Check if the test results are consistent.

Yes, we have plans to implement such functionality. It shouldn't be
hard, but we need to give it some thought in order to implement it as
elegantly as possible.

> (2).
> How can we run N parallel instances of a test? Will the current
> configuration  be easily able to support it?

I currently have some experimental patches that allow running of
several parallel queues of tests. But what exactly do you mean by
"N parallel instances of a test"? Do you mean N queues? Please provide
an example so I can get a better idea.

Thanks,
Michael

Re: kvm-autotest: The automation plans?

[sudhir kumar]

On Wed, May 13, 2009 at 11:30 PM, Michael Goldish <mgoldish@redhat.com> wrote:
>
> ----- "sudhir kumar" <smalikphy@gmail.com> wrote:
>
>> Hi Uri/Lucas,
>>
>> Do you have any plans for enhancing kvm-autotest?
>> I was looking mainly on the following 2 aspects:
>>
>> (1).
>> we have standalone migration only. Is there any plans of enhancing
>> kvm-autotest so that we can trigger migration while a workload is
>> running?
>> Something like this:
>> Start a workload(may be n instances of it).
>> let the test execute for some time.
>> Trigger migration.
>> Log into the target.
>> Check if the migration is succesful
>> Check if the test results are consistent.
>
> Yes, we have plans to implement such functionality. It shouldn't be
> hard, but we need to give it some thought in order to implement it as
> elegantly as possible.
I completely agree here.
>
>> (2).
>> How can we run N parallel instances of a test? Will the current
>> configuration  be easily able to support it?
>
> I currently have some experimental patches that allow running of
> several parallel queues of tests. But what exactly do you mean by
Please post them.
> "N parallel instances of a test"? Do you mean N queues? Please provide
> an example so I can get a better idea.
I wanted a parallelism in 2 degrees. Let me try with an example.
The following test
 only raw.*ide.*default.*smp2.*RHEL5.3.i386.*migrate.dbench
is just one instance and will create one VM with given specifications
and execute migrate and dbench. So I am thinking how can we trigger n
similar tests execution in parallel. I feel job.parallel() is meant
for that but is kvm_tests.cfg good enough to be used under such a
scenario? However we have most of the stuff non static(as getting the
free vnc port, etc) but still we have some variables which are static.
For ex. vm name, migration port etc. So what are your thoughts on it.
In this scenario my system will be having N VMs, all running the same
set of testcases.

On the other hand I was looking for something like this as well.
 only raw.*ide.*default.*smp2.*RHEL5.3.i386.*migrate.dbench.dbench_instancesN.bonnie
Thus all the tests will be executed in normal way except dbench. There
should be running N instances of dbench and when over simply run
bonnie and exit.

I hope my demand to kvm-autotest is not too much but for an effective
and rigorous testing of kvm such a framework is necessary. I am bit
new to autotest framework and have very little knowledge of the server
side. I will start spending some time on looking at the available
features.

Hope I was clear this time.



-- 
Sudhir Kumar

Re: kvm-autotest: The automation plans?

[Lucas Meneghel Rodrigues]

On Wed, 2009-05-13 at 23:21 +0530, sudhir kumar wrote:
> Hi Uri/Lucas,
> 
> Do you have any plans for enhancing kvm-autotest?
> I was looking mainly on the following 2 aspects:

Hi Sudhir, about the two questions you've made, Michael has answered
them a lot better than I possibly could. So please keep in touch and
send your ideas so we can consider implementing them on our tests!

Thank you very much,

-- 
Lucas Meneghel Rodrigues
Software Engineer (QE)
Red Hat - Emerging Technologies

Re: kvm-autotest: The automation plans?

[jason wang]

sudhir kumar 写道:
> Hi Uri/Lucas,
>
> Do you have any plans for enhancing kvm-autotest?
> I was looking mainly on the following 2 aspects:
>
> (1).
> we have standalone migration only. Is there any plans of enhancing
> kvm-autotest so that we can trigger migration while a workload is
> running?
> Something like this:
> Start a workload(may be n instances of it).
> let the test execute for some time.
> Trigger migration.
> Log into the target.
> Check if the migration is succesful
> Check if the test results are consistent.
>   
We have some patches of ping pong migration and workload adding. The 
migration is based on public bridge and workload adding is based on 
running benchmark in the background of guest.
> (2).
> How can we run N parallel instances of a test? Will the current
> configuration  be easily able to support it?
>
> Please provide your thoughts on the above features.
>
>   
The parallelized instances could be easily achieved through 
job.parallel() of autotest framework, and that is what we have used in 
our tests. We have make some helper routines such as get_free_port to be 
reentrant through file lock.
We've implemented following test cases: timedrift(already sent here), 
savevm/loadvm, suspend/resume, jumboframe, migration between two 
machines and others. We will sent it here for review in the following weeks.
There are some other things could be improved:
1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest 
server UI. This would make it hard to configure the tests in the server 
side. During our test, we have merged it into control and make it could 
be configured by "editing control file" function of autotest server side 
web UI.
2) Public bridge support: I've sent a patch(TAP network support in 
kvm-autotest), this patch needs external DHCP server and requires nmap 
support. I don't know whether the method of original kvm_runtes_old(DHCP 
server of private bridge) is preferable.

Re: kvm-autotest: The automation plans?

[sudhir kumar]

On Thu, May 14, 2009 at 12:22 PM, jason wang <jasowang@redhat.com> wrote:
> sudhir kumar 写道:
>>
>> Hi Uri/Lucas,
>>
>> Do you have any plans for enhancing kvm-autotest?
>> I was looking mainly on the following 2 aspects:
>>
>> (1).
>> we have standalone migration only. Is there any plans of enhancing
>> kvm-autotest so that we can trigger migration while a workload is
>> running?
>> Something like this:
>> Start a workload(may be n instances of it).
>> let the test execute for some time.
>> Trigger migration.
>> Log into the target.
>> Check if the migration is succesful
>> Check if the test results are consistent.
>>
>
> We have some patches of ping pong migration and workload adding. The
> migration is based on public bridge and workload adding is based on running
> benchmark in the background of guest.
Cool. I would like to have look on them. So how do you manage the
background process/thread?

>>
>> (2).
>> How can we run N parallel instances of a test? Will the current
>> configuration  be easily able to support it?
>>
>> Please provide your thoughts on the above features.
>>
>>
>
> The parallelized instances could be easily achieved through job.parallel()
> of autotest framework, and that is what we have used in our tests. We have
> make some helper routines such as get_free_port to be reentrant through file
> lock.
> We've implemented following test cases: timedrift(already sent here),
> savevm/loadvm, suspend/resume, jumboframe, migration between two machines
> and others. We will sent it here for review in the following weeks.
> There are some other things could be improved:
> 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest
> server UI. This would make it hard to configure the tests in the server
> side. During our test, we have merged it into control and make it could be
> configured by "editing control file" function of autotest server side web
> UI.
Not much clue here. But I would like to keep the control file as
simple as possible and as much independent of test scenarios as
possible. kvm_tests.cfg should be the right file untill and unless it
is impossible to do by using it.
> 2) Public bridge support: I've sent a patch(TAP network support in
> kvm-autotest), this patch needs external DHCP server and requires nmap
> support. I don't know whether the method of original kvm_runtes_old(DHCP
> server of private bridge) is preferable.
The old approach is better. All might not be able to run an external
DHCP server for running the test. I do not see any issue with the old
approach.
>
>



-- 
Sudhir Kumar

Re: kvm-autotest: The automation plans?

[Michael Goldish]

----- "sudhir kumar" <smalikphy@gmail.com> wrote:

> On Thu, May 14, 2009 at 12:22 PM, jason wang <jasowang@redhat.com>
> wrote:
> > sudhir kumar 写道:
> >>
> >> Hi Uri/Lucas,
> >>
> >> Do you have any plans for enhancing kvm-autotest?
> >> I was looking mainly on the following 2 aspects:
> >>
> >> (1).
> >> we have standalone migration only. Is there any plans of enhancing
> >> kvm-autotest so that we can trigger migration while a workload is
> >> running?
> >> Something like this:
> >> Start a workload(may be n instances of it).
> >> let the test execute for some time.
> >> Trigger migration.
> >> Log into the target.
> >> Check if the migration is succesful
> >> Check if the test results are consistent.
> >>
> >
> > We have some patches of ping pong migration and workload adding.
> The
> > migration is based on public bridge and workload adding is based on
> running
> > benchmark in the background of guest.
> Cool. I would like to have look on them. So how do you manage the
> background process/thread?
> 
> >>
> >> (2).
> >> How can we run N parallel instances of a test? Will the current
> >> configuration  be easily able to support it?
> >>
> >> Please provide your thoughts on the above features.
> >>
> >>
> >
> > The parallelized instances could be easily achieved through
> job.parallel()
> > of autotest framework, and that is what we have used in our tests.
> We have
> > make some helper routines such as get_free_port to be reentrant
> through file
> > lock.
> > We've implemented following test cases: timedrift(already sent
> here),
> > savevm/loadvm, suspend/resume, jumboframe, migration between two
> machines
> > and others. We will sent it here for review in the following weeks.
> > There are some other things could be improved:
> > 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to
> autotest
> > server UI. This would make it hard to configure the tests in the
> server
> > side. During our test, we have merged it into control and make it
> could be
> > configured by "editing control file" function of autotest server
> side web
> > UI.
> Not much clue here. But I would like to keep the control file as
> simple as possible and as much independent of test scenarios as
> possible. kvm_tests.cfg should be the right file untill and unless it
> is impossible to do by using it.
> > 2) Public bridge support: I've sent a patch(TAP network support in
> > kvm-autotest), this patch needs external DHCP server and requires
> nmap
> > support. I don't know whether the method of original
> kvm_runtes_old(DHCP
> > server of private bridge) is preferable.
> The old approach is better. All might not be able to run an external
> DHCP server for running the test. I do not see any issue with the old
> approach.

We're taking more of a minimalist approach in kvm_runtest_2: the
framework should handle only the things directly related to testing.
Configuring and running a DHCP server is and should be beyond the scope
of the KVM-Autotest framework. To emulate the old behavior, you can just
start the DHCP server yourself locally. If you wish, maybe we can
bundle example scripts with the framework that will do this for the user,
but they should not be an integral part of the framework in my opinion.

> 
> 
> 
> -- 
> Sudhir Kumar
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kvm-autotest: The automation plans?

[jason wang]

Michael Goldish 写道:
> ----- "sudhir kumar" <smalikphy@gmail.com> wrote:
>
>   
>> On Thu, May 14, 2009 at 12:22 PM, jason wang <jasowang@redhat.com>
>> wrote:
>>     
>>> sudhir kumar 写道:
>>>       
>>>> Hi Uri/Lucas,
>>>>
>>>> Do you have any plans for enhancing kvm-autotest?
>>>> I was looking mainly on the following 2 aspects:
>>>>
>>>> (1).
>>>> we have standalone migration only. Is there any plans of enhancing
>>>> kvm-autotest so that we can trigger migration while a workload is
>>>> running?
>>>> Something like this:
>>>> Start a workload(may be n instances of it).
>>>> let the test execute for some time.
>>>> Trigger migration.
>>>> Log into the target.
>>>> Check if the migration is succesful
>>>> Check if the test results are consistent.
>>>>
>>>>         
>>> We have some patches of ping pong migration and workload adding.
>>>       
>> The
>>     
>>> migration is based on public bridge and workload adding is based on
>>>       
>> running
>>     
>>> benchmark in the background of guest.
>>>       
>> Cool. I would like to have look on them. So how do you manage the
>> background process/thread?
>>     
Yes, we would try to sent it here as soon as possible. The background 
workload could be added through various methods. We could an simple 
algorithm as follows:

run_migration2():
pid = run_autotest_background(test,params,env,"dbench","control.60")

Do ping-pong migration ...

wait_autoteset_background(pid)

run_autotest_background() would fork a subprocess to run function 
run_autotest() and catch its exception.
wait_autotest_background(pid) would wait until the background benchmark 
complete and analyse the result through the return value of the subprocess.
The child process could work well depends the fact that the ssh 
connection should alive during migration.
I believe this could be also achieved through job.parallel()
>>>> (2).
>>>> How can we run N parallel instances of a test? Will the current
>>>> configuration  be easily able to support it?
>>>>
>>>> Please provide your thoughts on the above features.
>>>>
>>>>
>>>>         
>>> The parallelized instances could be easily achieved through
>>>       
>> job.parallel()
>>     
>>> of autotest framework, and that is what we have used in our tests.
>>>       
>> We have
>>     
>>> make some helper routines such as get_free_port to be reentrant
>>>       
>> through file
>>     
>>> lock.
>>> We've implemented following test cases: timedrift(already sent
>>>       
>> here),
>>     
>>> savevm/loadvm, suspend/resume, jumboframe, migration between two
>>>       
>> machines
>>     
>>> and others. We will sent it here for review in the following weeks.
>>> There are some other things could be improved:
>>> 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to
>>>       
>> autotest
>>     
>>> server UI. This would make it hard to configure the tests in the
>>>       
>> server
>>     
>>> side. During our test, we have merged it into control and make it
>>>       
>> could be
>>     
>>> configured by "editing control file" function of autotest server
>>>       
>> side web
>>     
>>> UI.
>>>       
>> Not much clue here. But I would like to keep the control file as
>> simple as possible and as much independent of test scenarios as
>> possible. kvm_tests.cfg should be the right file untill and unless it
>> is impossible to do by using it.
>>     
>>> 2) Public bridge support: I've sent a patch(TAP network support in
>>> kvm-autotest), this patch needs external DHCP server and requires
>>>       
>> nmap
>>     
>>> support. I don't know whether the method of original
>>>       
>> kvm_runtes_old(DHCP
>>     
>>> server of private bridge) is preferable.
>>>       
>> The old approach is better. All might not be able to run an external
>> DHCP server for running the test. I do not see any issue with the old
>> approach.
>>     
>
> We're taking more of a minimalist approach in kvm_runtest_2: the
> framework should handle only the things directly related to testing.
> Configuring and running a DHCP server is and should be beyond the scope
> of the KVM-Autotest framework. To emulate the old behavior, you can just
> start the DHCP server yourself locally. If you wish, maybe we can
> bundle example scripts with the framework that will do this for the user,
> but they should not be an integral part of the framework in my opinion.
>
>   
>>
>> -- 
>> Sudhir Kumar
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>     

Re: kvm-autotest: The automation plans?

[sudhir kumar]

2009/5/14 sudhir kumar <smalikphy@gmail.com>:
> On Thu, May 14, 2009 at 12:22 PM, jason wang <jasowang@redhat.com> wrote:
>> sudhir kumar 写道:
>>>
>>> Hi Uri/Lucas,
>>>
>>> Do you have any plans for enhancing kvm-autotest?
>>> I was looking mainly on the following 2 aspects:
>>>
>>> (1).
>>> we have standalone migration only. Is there any plans of enhancing
>>> kvm-autotest so that we can trigger migration while a workload is
>>> running?
>>> Something like this:
>>> Start a workload(may be n instances of it).
>>> let the test execute for some time.
>>> Trigger migration.
>>> Log into the target.
>>> Check if the migration is succesful
>>> Check if the test results are consistent.
>>>
>>
>> We have some patches of ping pong migration and workload adding. The
>> migration is based on public bridge and workload adding is based on running
>> benchmark in the background of guest.
> Cool. I would like to have look on them. So how do you manage the
> background process/thread?
>
>>>
>>> (2).
>>> How can we run N parallel instances of a test? Will the current
>>> configuration  be easily able to support it?
>>>
>>> Please provide your thoughts on the above features.
>>>
>>>
>>
>> The parallelized instances could be easily achieved through job.parallel()
>> of autotest framework, and that is what we have used in our tests. We have
>> make some helper routines such as get_free_port to be reentrant through file
>> lock.
>> We've implemented following test cases: timedrift(already sent here),
>> savevm/loadvm, suspend/resume, jumboframe, migration between two machines
>> and others. We will sent it here for review in the following weeks.
>> There are some other things could be improved:
>> 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest
>> server UI. This would make it hard to configure the tests in the server
>> side. During our test, we have merged it into control and make it could be
>> configured by "editing control file" function of autotest server side web
>> UI.
> Not much clue here. But I would like to keep the control file as
> simple as possible and as much independent of test scenarios as
> possible. kvm_tests.cfg should be the right file untill and unless it
> is impossible to do by using it.
>> 2) Public bridge support: I've sent a patch(TAP network support in
Did you send the patch on the mailing list? Can you please point me to
it. In my knowledge -redir tcp *:* breaks with -net tap. If running an
external dhcp/dns server is the solution then the patch should go in.
Thanks in advance!!

>> kvm-autotest), this patch needs external DHCP server and requires nmap
>> support. I don't know whether the method of original kvm_runtes_old(DHCP
>> server of private bridge) is preferable.
> The old approach is better. All might not be able to run an external
> DHCP server for running the test. I do not see any issue with the old
> approach.
>>
>>
>
>
>
> --
> Sudhir Kumar
>



-- 
Sudhir Kumar

Re: kvm-autotest: The automation plans?

[jason wang]

sudhir kumar 写道:
> 2009/5/14 sudhir kumar <smalikphy@gmail.com>:
>   
>> On Thu, May 14, 2009 at 12:22 PM, jason wang <jasowang@redhat.com> wrote:
>>     
>>> sudhir kumar 写道:
>>>       
>>>> Hi Uri/Lucas,
>>>>
>>>> Do you have any plans for enhancing kvm-autotest?
>>>> I was looking mainly on the following 2 aspects:
>>>>
>>>> (1).
>>>> we have standalone migration only. Is there any plans of enhancing
>>>> kvm-autotest so that we can trigger migration while a workload is
>>>> running?
>>>> Something like this:
>>>> Start a workload(may be n instances of it).
>>>> let the test execute for some time.
>>>> Trigger migration.
>>>> Log into the target.
>>>> Check if the migration is succesful
>>>> Check if the test results are consistent.
>>>>
>>>>         
>>> We have some patches of ping pong migration and workload adding. The
>>> migration is based on public bridge and workload adding is based on running
>>> benchmark in the background of guest.
>>>       
>> Cool. I would like to have look on them. So how do you manage the
>> background process/thread?
>>
>>     
>>>> (2).
>>>> How can we run N parallel instances of a test? Will the current
>>>> configuration  be easily able to support it?
>>>>
>>>> Please provide your thoughts on the above features.
>>>>
>>>>
>>>>         
>>> The parallelized instances could be easily achieved through job.parallel()
>>> of autotest framework, and that is what we have used in our tests. We have
>>> make some helper routines such as get_free_port to be reentrant through file
>>> lock.
>>> We've implemented following test cases: timedrift(already sent here),
>>> savevm/loadvm, suspend/resume, jumboframe, migration between two machines
>>> and others. We will sent it here for review in the following weeks.
>>> There are some other things could be improved:
>>> 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest
>>> server UI. This would make it hard to configure the tests in the server
>>> side. During our test, we have merged it into control and make it could be
>>> configured by "editing control file" function of autotest server side web
>>> UI.
>>>       
>> Not much clue here. But I would like to keep the control file as
>> simple as possible and as much independent of test scenarios as
>> possible. kvm_tests.cfg should be the right file untill and unless it
>> is impossible to do by using it.
>>     
>>> 2) Public bridge support: I've sent a patch(TAP network support in
>>>       
> Did you send the patch on the mailing list? Can you please point me to
> it. In my knowledge -redir tcp *:* breaks with -net tap. If running an
> external dhcp/dns server is the solution then the patch should go in.
> Thanks in advance!!
>   
Thanks for pointing out this problem. The redirections indeed breaks the
tap networking. TAP patch could be seen at
http://marc.info/?l=kvm&m=124221382604554&w=2. The redirections also
leads the problems of getting the ip address of a user mode nic. So
maybe we could make the tap/user to be an exclusive option.
>>> kvm-autotest), this patch needs external DHCP server and requires nmap
>>> support. I don't know whether the method of original kvm_runtes_old(DHCP
>>> server of private bridge) is preferable.
>>>       
>> The old approach is better. All might not be able to run an external
>> DHCP server for running the test. I do not see any issue with the old
>> approach.
>>     
>>>       
>>
>> --
>> Sudhir Kumar
>>
>>     
>
>
>
>   

Re: kvm-autotest: The automation plans?

[Uri Lublin]

On 05/14/2009 01:29 PM, sudhir kumar wrote:
> On Thu, May 14, 2009 at 12:22 PM, jason wang<jasowang@redhat.com>  wrote:
>> 2) Public bridge support: I've sent a patch(TAP network support in
>> kvm-autotest), this patch needs external DHCP server and requires nmap
>> support. I don't know whether the method of original kvm_runtes_old(DHCP
>> server of private bridge) is preferable.
> The old approach is better. All might not be able to run an external
> DHCP server for running the test. I do not see any issue with the old
> approach.

Some drawbacks of the local dhcp server were:
1. The guests are not connected to the network. That mean we can not access the
internet from a guest (unless we implement a proxy on the host). We've done it
that way to make sure our local dhcp server only serves VMs, and will not
interfere with an existing dhcp server, if exists.

2. We implemented it only for specific Linux hosts Fedora/RHEL and Ubuntu.
cleanup was not best too.

Some advantages were:
1. Use the same configuration (e.g. guest MAC addresses) on many different host
with no conflicts.
2. Support users with no external dhcp server (or no access to its configuration).
3. Preconfigured, works out of the box.

Note that it was flexible. Local dhcp server was the default but one could
define the framework to use external dhcp server too.

The current approach is that the user should setup whatever (s)he needs before
running the tests, including setup of a dhcp server (to be used with TAP).

Regards,
    Uri.

Re: kvm-autotest: The automation plans?

[Michael Goldish]

----- "jason wang" <jasowang@redhat.com> wrote:

> sudhir kumar 写道:
> > Hi Uri/Lucas,
> >
> > Do you have any plans for enhancing kvm-autotest?
> > I was looking mainly on the following 2 aspects:
> >
> > (1).
> > we have standalone migration only. Is there any plans of enhancing
> > kvm-autotest so that we can trigger migration while a workload is
> > running?
> > Something like this:
> > Start a workload(may be n instances of it).
> > let the test execute for some time.
> > Trigger migration.
> > Log into the target.
> > Check if the migration is succesful
> > Check if the test results are consistent.
> >   
> We have some patches of ping pong migration and workload adding. The 
> migration is based on public bridge and workload adding is based on 
> running benchmark in the background of guest.
> > (2).
> > How can we run N parallel instances of a test? Will the current
> > configuration  be easily able to support it?
> >
> > Please provide your thoughts on the above features.
> >
> >   
> The parallelized instances could be easily achieved through 
> job.parallel() of autotest framework, and that is what we have used in
> our tests. We have make some helper routines such as get_free_port to
> be reentrant through file lock.

We'll probably have to use file locks anyway when we work with TAP, but in
VM.create(), not in get_free_port(), because we also want to prevent parallel
qemu instances from choosing the same TAP device. I'm not sure how qemu
handles this internally, and I'd rather be on the safe side.

Do you release the file lock inside get_free_port or only after running qemu?

> We've implemented following test cases: timedrift(already sent here),
> savevm/loadvm, suspend/resume, jumboframe, migration between two 
> machines and others. We will sent it here for review in the following
> weeks.
> There are some other things could be improved:
> 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest
> server UI. This would make it hard to configure the tests in the
> server 
> side. During our test, we have merged it into control and make it
> could 
> be configured by "editing control file" function of autotest server
> side web UI.

Would it not suffice to just modify the configuration, instead of completely
define it, inside the control file? This is possible using parse_string().
For example:

cfg = kvm_config.config("kvm_tests.cfg")
cfg.parse_string("only weekly")
cfg.parse_string("only Fedora RHEL Windows")
cfg.parse_string("""
variants:
    - 1:
        only ide
    - 2:
        Fedora:
            no rtl8139
""")
list = cfg.get_list()

(get_list() returns the test dictionaries.)

The advantage here is that we can have a standard kvm_tests.cfg that we all
agree on and only rather small environment-specific modifications are made
in the control file.

Re: kvm-autotest: The automation plans?

[jason wang]

Michael Goldish 写道:
> ----- "jason wang" <jasowang@redhat.com> wrote:
>
>   
>> sudhir kumar 写道:
>>     
>>> Hi Uri/Lucas,
>>>
>>> Do you have any plans for enhancing kvm-autotest?
>>> I was looking mainly on the following 2 aspects:
>>>
>>> (1).
>>> we have standalone migration only. Is there any plans of enhancing
>>> kvm-autotest so that we can trigger migration while a workload is
>>> running?
>>> Something like this:
>>> Start a workload(may be n instances of it).
>>> let the test execute for some time.
>>> Trigger migration.
>>> Log into the target.
>>> Check if the migration is succesful
>>> Check if the test results are consistent.
>>>   
>>>       
>> We have some patches of ping pong migration and workload adding. The 
>> migration is based on public bridge and workload adding is based on 
>> running benchmark in the background of guest.
>>     
>>> (2).
>>> How can we run N parallel instances of a test? Will the current
>>> configuration  be easily able to support it?
>>>
>>> Please provide your thoughts on the above features.
>>>
>>>   
>>>       
>> The parallelized instances could be easily achieved through 
>> job.parallel() of autotest framework, and that is what we have used in
>> our tests. We have make some helper routines such as get_free_port to
>> be reentrant through file lock.
>>     
>
> We'll probably have to use file locks anyway when we work with TAP, but in
> VM.create(), not in get_free_port(), because we also want to prevent parallel
> qemu instances from choosing the same TAP device. I'm not sure how qemu
> handles this internally, and I'd rather be on the safe side.
>
> Do you release the file lock inside get_free_port or only after running qemu?
>   
We record the port usage and release the file lock inside 
get_free_port(). I agree with you that it's better to get/release the 
file lock in VM.create() because it is easier and it also eliminates the 
effort of doing lock in every helper function.
For the TAP device, maybe we could give each TAP device used by qemu-kvm 
an random generated ifname to prevent qemu-kvm from choosing the same 
TAP devices. This method works well in our test.
>> We've implemented following test cases: timedrift(already sent here),
>> savevm/loadvm, suspend/resume, jumboframe, migration between two 
>> machines and others. We will sent it here for review in the following
>> weeks.
>> There are some other things could be improved:
>> 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest
>> server UI. This would make it hard to configure the tests in the
>> server 
>> side. During our test, we have merged it into control and make it
>> could 
>> be configured by "editing control file" function of autotest server
>> side web UI.
>>     
>
> Would it not suffice to just modify the configuration, instead of completely
> define it, inside the control file? This is possible using parse_string().
> For example:
>
> cfg = kvm_config.config("kvm_tests.cfg")
> cfg.parse_string("only weekly")
> cfg.parse_string("only Fedora RHEL Windows")
> cfg.parse_string("""
> variants:
>     - 1:
>         only ide
>     - 2:
>         Fedora:
>             no rtl8139
> """)
> list = cfg.get_list()
>
> (get_list() returns the test dictionaries.)
>
> The advantage here is that we can have a standard kvm_tests.cfg that we all
> agree on and only rather small environment-specific modifications are made
> in the control file.
>   
Thanks, this way makes the things easier.

Re: kvm-autotest: The automation plans?

[Michael Goldish]

----- "sudhir kumar" <smalikphy@gmail.com> wrote:

> On Wed, May 13, 2009 at 11:30 PM, Michael Goldish
> <mgoldish@redhat.com> wrote:
> >
> > ----- "sudhir kumar" <smalikphy@gmail.com> wrote:
> >
> >> Hi Uri/Lucas,
> >>
> >> Do you have any plans for enhancing kvm-autotest?
> >> I was looking mainly on the following 2 aspects:
> >>
> >> (1).
> >> we have standalone migration only. Is there any plans of enhancing
> >> kvm-autotest so that we can trigger migration while a workload is
> >> running?
> >> Something like this:
> >> Start a workload(may be n instances of it).
> >> let the test execute for some time.
> >> Trigger migration.
> >> Log into the target.
> >> Check if the migration is succesful
> >> Check if the test results are consistent.
> >
> > Yes, we have plans to implement such functionality. It shouldn't be
> > hard, but we need to give it some thought in order to implement it
> as
> > elegantly as possible.
> I completely agree here.
> >
> >> (2).
> >> How can we run N parallel instances of a test? Will the current
> >> configuration  be easily able to support it?
> >
> > I currently have some experimental patches that allow running of
> > several parallel queues of tests. But what exactly do you mean by
> Please post them.
> > "N parallel instances of a test"? Do you mean N queues? Please
> provide
> > an example so I can get a better idea.
> I wanted a parallelism in 2 degrees. Let me try with an example.
> The following test
>  only raw.*ide.*default.*smp2.*RHEL5.3.i386.*migrate.dbench
> is just one instance and will create one VM with given specifications
> and execute migrate and dbench. So I am thinking how can we trigger n
> similar tests execution in parallel. I feel job.parallel() is meant
> for that but is kvm_tests.cfg good enough to be used under such a
> scenario? However we have most of the stuff non static(as getting the
> free vnc port, etc) but still we have some variables which are
> static.
> For ex. vm name, migration port etc. So what are your thoughts on it.

I think generally kvm_tests.cfg is flexible enough, and can easily be
modified to define whatever you like.

Note, however, that the config file parser module is only responsible
for producing a list of dictionaries which define the tests to run.
It doesn't care much about parallelism -- this is up to the control file
and the rest of the framework. If you're not familiar with the format
of config files, please refer to
http://www.linux-kvm.org/page/KVM-Autotest/Test_Config_File
and
http://www.linux-kvm.org/page/KVM-Autotest/Parameters

> In this scenario my system will be having N VMs, all running the same
> set of testcases.

I thought you said one VM running migrate and dbench in parallel. I'm
not sure I follow.

> On the other hand I was looking for something like this as well.
>  only
> raw.*ide.*default.*smp2.*RHEL5.3.i386.*migrate.dbench.dbench_instancesN.bonnie
> Thus all the tests will be executed in normal way except dbench.
> There
> should be running N instances of dbench and when over simply run
> bonnie and exit.

This seems like two tests to me: dbench with dbench (several instances),
and then another unrelated bonnie test.

Also note that the variants you select with 'only' must be defined before
they can be selected. Look at the examples in the wiki as well as real
config files.

> I hope my demand to kvm-autotest is not too much but for an effective
> and rigorous testing of kvm such a framework is necessary. I am bit
> new to autotest framework and have very little knowledge of the
> server
> side. I will start spending some time on looking at the available
> features.
> 
> Hope I was clear this time.

Regarding parallelism:
Generally two types can be implemented.

1. Several independent test execution queues: in this case there are several
queues that don't interfere with each other. Each queue works with its own
VMs. This is useful for saving time by running tests in parallel on capable
hosts. This can be implemented using job.parallel() and is already running
in TLV. I will try to post the patches soon.

This can probably also be implemented from the server, if it can treat a
single physical host as if it were several, thus running several independent
copies of the Autotest client on it.

2. Several tests on a single VM, which is what you were referring to, if I
understood correctly: in this case several threads work with the same VMs
and abuse them in parallel -- one thread can run dbench while the other
runs migration on the same VM. This is possible using threads, and the
syntax in the config file can be something like 'types = dbench migration'
instead of what we currently use -- 'type = dbench'.
However, we have to think whether we really just want to run tests in
parallel. In the migration-dbench case, for example, we'd like to make sure
dbench starts running before we migrate. So maybe it's wiser to just run some
load inside the migration test, instead of the dbench test. We should
carefully consider all options.

Thanks,
Michael