Re: [RFC] KVM test: Refactoring the kvm control file and the config file

Re: [RFC] KVM test: Refactoring the kvm control file and the config file

[Michael Goldish]

----- "David Huff" <dhuff@redhat.com> wrote:

> Michael Goldish wrote:
> > ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
> > 
> >> Currently we have our kvm test control file and configuration
> file,
> >> having them split like this makes it harder for users to edit it,
> >> let's
> >> say, using the web frontend.
> >>
> >> So it might be good to merge the control file and the config file,
> >> and
> >> make a refactor on the control file code. Do you think this would
> be
> >> a valid approach? Any comments are welcome.
> >>
> >> Lucas
> > 
> > What exactly do you mean by merge? Embed the entire config file in
> > the control file as a python string?
> > 
> > A few comments:
> > 
> > 1. The bulk of the config file usually doesn't need to be modified
> > from the web frontend, IMO. It actually doesn't need to be modified
> > very often -- once everything is defined, only minor changes are
> > required.
> > 
> > 2. Changes to the config can be made in the control file rather
> easily
> > using kvm_config methods that are implemented but not currently
> used.
> > Instead of the short form:
> > 
> > list = kvm_config.config(filename).get_list()
> > 
> > we can use:
> > 
> > cfg = kvm_config.config(filename)
> > 
> > # parse any one-liner like this:
> > cfg.parse_string("only nightly")
> > 
> > # parse anything the parser understands like this:
> > cfg.parse_string("""
> > install:
> >     steps = blah
> >     foo = bar
> > only qcow2.*Windows
> > """)
> > 
> > # we can parse several times and the effect is cumulative
> > cfg.parse_string("""
> > variants:
> >     - foo:
> >         only scsi
> >     - bar:
> >         only WinVista.32
> >         variants:
> >             - 1:
> >             - 2:
> > """)
> > 
> > # we can also parse additional files:
> > cfg.parse_file("windows_cdkeys.cfg")
> > 
> > # finally, get the resulting list
> > list = cfg.get_list()
> > 
> > 3. We may want to consider something in between having the control
> and
> > config completely separated (what we have today), and having them
> both
> > in the same file. For example, we can define the test sets
> (nightly,
> > weekly, fc8_quick, custom) in the config file, and select the test
> set
> > (e.g. "only nightly") in the control file by convention.
> Alternatively
> > we can omit the test sets from the config file, and just define a
> single
> > test set (the one we'll be using) in the control file, or define
> several
> > test sets in the control file, and select one of them.
> > We can actually do both things at the same time, by defining the
> test
> > sets in the config file, and defining a "full" test set among them
> (I
> > think it's already there), which doesn't modify anything. If we want
> to
> > use a standard test set from the config file, we can do "only
> nightly"
> > in the control, and if we want to use a custom test set, we can do:
> > cfg.parse_string("""
> > only full
> > # define the test set below (no need for variants)
> > only RHEL
> > only qcow2
> > only autotest.dbench
> > """)
> > 
> > 4. It could be a good idea to make a "windows_cdkeys.cfg" file,
> that
> > contains mainly single-line exceptions, such as:
> > WinXP.32: cdkey = REPLACE_ME
> > WinXP.64: cdkey = REPLACE_ME
> > Win2003.32: cdkey = REPLACE_ME
> > ...
> > The real cdkeys should be entered by the user. Then the file will
> be
> > parsed after kvm_tests.cfg, using the parse_file() method (in the
> > control). This way the user won't have to enter the cdkeys into the
> > long config file every time it gets replaced by a newer version.
> The
> > cdkeys file won't be replaced because it's specific to the test
> > environment (we'll only supply a sample like we do with
> kvm_tests.cfg).
> > 
> > Maybe we can generalize this idea and call the file
> local_prefs.cfg,
> > and decide that the file should contain any environment-specific
> > changes that the user wants to make to the config. The file will
> > contain mainly exceptions (single or multi-line). But I'm not sure
> > there are many environment specific things other than cdkeys, so
> maybe
> > this isn't necessary.
> > 
> > 
> > Let me know what you think.
> 
> Michael all very good comments, I specifically like the windows
> config
> file idea.
> 
> The way I always envisioned it was something like this......
> 
> The config file specifies the whole test matrix, ie all variants that
> you could run each test on, ie all os's, all archs, all disk types,
> all
> cpu/mem configurations.
> 
> The control file would be more of a test specific config file,
> setting
> any local or environmental vars for each test, and like Michael said
> can
> override "stuff" fromt he main confg file...
> 
> I also really like the idea of creating a generic kvm_test that all
> kvm
>  tests would inherent from, ie.
> $AUTOTEST/client/common_lib/kvm_test.py
> 
> All helper classes ie. kvm.py, kvm_utils.py, kvm_config.py, and even
> the
> config file itself could then go into
> $AUTOTEST/client/common_lib/test_utils/ or even maybe something like
> $AUTOTEST/client/common_lib/kvm_test_utils/
> 
> All kvm specific tests would inherent form the generic kvm_test, and
> then go into either $AUTOTEST/client/tests/ or
> $AUTOTEST/client/kvm_tests/ directorys, each having their own sub dir
> like the current autotest tests.  In this dir there would be a
> control
> file specific for each test, that can override the full test matrix
> descried inthe the generic kvm_tests.cfg, as well as any additional
> file
> required by the test.
> 
> Anyway just some of my thoughts, I know its great in theory however
> may
> have some implementation short falls, like interdependence between
> tests
> and such...
> 
> 
> Comments..

I think I understand your suggestion, but let me make sure:

- If there's a global config file that is shared by all tests, I suppose
it'll run all the tests one by one, right?

- Where will test sets be defined -- in the global config file?

- If each individual test inherits from the global config file, it'll
also inherit dictionaries describing other tests, right?
e.g. the configuration of the boot test must explicitly state "only boot",
or it'll run install, migration and autotest as well?

- If you run the control file of a specific test, what happens -- does
that specific test run in many configurations (many guests, cpu options,
network, ide/scsi), or does it run just once with a single configuration?
I suppose the "normal" behavior would be to run in many configurations, but
I'm not sure what your intention was.

- Will the global config file look like the config files we have today?

I think this should be possible to implement, but I haven't given it much
thought so I'm not sure. The more interesting question is whether it's a
good idea. What are the advantages over the current approach?

Re: [RFC] KVM test: Refactoring the kvm control file and the config file

[David Huff]

Michael Goldish wrote:
> ----- "David Huff" <dhuff@redhat.com> wrote:
>> The way I always envisioned it was something like this......
>>
>> The config file specifies the whole test matrix, ie all variants that
>> you could run each test on, ie all os's, all archs, all disk types,
>> all
>> cpu/mem configurations.
>>
>> The control file would be more of a test specific config file,
>> setting
>> any local or environmental vars for each test, and like Michael said
>> can
>> override "stuff" fromt he main confg file...
>>
>> I also really like the idea of creating a generic kvm_test that all
>> kvm
>>  tests would inherent from, ie.
>> $AUTOTEST/client/common_lib/kvm_test.py
>>
>> All helper classes ie. kvm.py, kvm_utils.py, kvm_config.py, and even
>> the
>> config file itself could then go into
>> $AUTOTEST/client/common_lib/test_utils/ or even maybe something like
>> $AUTOTEST/client/common_lib/kvm_test_utils/
>>
>> All kvm specific tests would inherent form the generic kvm_test, and
>> then go into either $AUTOTEST/client/tests/ or
>> $AUTOTEST/client/kvm_tests/ directorys, each having their own sub dir
>> like the current autotest tests.  In this dir there would be a
>> control
>> file specific for each test, that can override the full test matrix
>> descried inthe the generic kvm_tests.cfg, as well as any additional
>> file
>> required by the test.
>>
>> Anyway just some of my thoughts, I know its great in theory however
>> may
>> have some implementation short falls, like interdependence between
>> tests
>> and such...
>>
>>
>> Comments..
> 
> I think I understand your suggestion, but let me make sure:
> 
> - If there's a global config file that is shared by all tests, I suppose
> it'll run all the tests one by one, right?
> 
> - Where will test sets be defined -- in the global config file?
> 
> - If each individual test inherits from the global config file, it'll
> also inherit dictionaries describing other tests, right?
> e.g. the configuration of the boot test must explicitly state "only boot",
> or it'll run install, migration and autotest as well?
> 
> - If you run the control file of a specific test, what happens -- does
> that specific test run in many configurations (many guests, cpu options,
> network, ide/scsi), or does it run just once with a single configuration?
> I suppose the "normal" behavior would be to run in many configurations, but
> I'm not sure what your intention was.
> 
> - Will the global config file look like the config files we have today?
> 
> I think this should be possible to implement, but I haven't given it much
> thought so I'm not sure. The more interesting question is whether it's a
> good idea. What are the advantages over the current approach?

The advantages I see are: 1. it more closely follows the current
autotest structure/layout, 2. solves the problem of separating each test
out of the ever growing kvm_test.py and gives a sub dir of each test for
better structure (something we have been talking about) and 3. addresses
the config vs. control file ? that this thread originally brought up.

I think the issue is in how the "kvm test" is viewed.  Is it one test
that gets run against several configurations, or is it several different
tests with different configurations?.  I have been looking at it as the
later however I do also see it the other way as well.

So maybe the solution is a little different than my first thought....

- all kvm tests are in $AUTOTEST/client/kvm_tests/
- all kvm tests inherent form $AUTOTEST/client/common_lib/kvm_test.py
- common functionality is in $AUTOTEST/client/common_lib/kvm_test_utils/
  - does *not* include generic kvm_test.cfg
- we keep the $AUTOTEST/client/kvm/ test dir which defines the test runs
and houses kvm_test.cfg file and a master control.
  - we could then define a couple sample test runs: full, quick, and
others  or implement something like your kvm_tests.common file that
other test runs can build on.

So in the end its pretty similar to what we currently have except that
the AUTOTEST/client/kvm/ dir only defines the test runs, all common
functionality and the tests them selves are moved out.

The major advantages I see, aside from the three mention above, is that
it allows us to simplify the kvm_tests.cfg file.  We can move the test
specific config to each test dir,
$AUTOTEST/client/kvm_tests/install/install.cfg which includes all the
install test parameters.  Which combined with splitting up the config
file in $AUTOTEST/client/kvm/ would make the config file shorter and
easier to read.

Again not sure if all this is worth it however some of my thoughts on
how to improve the current status.

-D

Re: [Autotest] [RFC] KVM test: Refactoring the kvm control file and the config file

[Martin Bligh]

> The advantages I see are: 1. it more closely follows the current
> autotest structure/layout, 2. solves the problem of separating each test
> out of the ever growing kvm_test.py and gives a sub dir of each test for
> better structure (something we have been talking about) and 3. addresses
> the config vs. control file ? that this thread originally brought up.
>
> I think the issue is in how the "kvm test" is viewed.  Is it one test
> that gets run against several configurations, or is it several different
> tests with different configurations?.  I have been looking at it as the
> later however I do also see it the other way as well.

I think if you try to force everything you do into one test, you'll lose
a lot of the power and flexibility of the system. I can't claim to have
entirely figured out what you're doing, but it seems somewhat like
you're reinventing some stuff with the current approach?

Some of the general design premises:
   1) Anything the user might want to configure should be in the control file
   2) Anything in test should be really pretty static.
   3) The way we get around a lot of the conflicts is by passing parameters
       to run_test, though leaving sensible defaults in for them makes things
       much easier to use.
   4) The frontend and cli are designed to allow you to edit control files,
       and/or save custom versions - that's the single object we throw
       to machines under test ... there's no passing of cfg files to clients?

We often end up with longer control files that contain a pre-canned set of
tests, and even "meta-control files" that kick off a multitude of jobs across
thousands of machines, using frontend.py. That can include control flow -
for example our internal kernel testing uses a waterfall model with several
steps:

1. Compile the kernel from source
2. Test on a bunch of single machines with a smoketest that takes an
hour or so.
3. Test on small groups of machines with cut down simulations of
cluster tests
4. Test on full clusters.

If any of those tests fails (with some built in fault tolerance for a small
hardware fallout rate), we stop the testing. All of that control flow
is governed by a control file. It sounds complex, but it's really not
if you build your "building blocks" carefully, and it's extremely powerful

> So maybe the solution is a little different than my first thought....
>
> - all kvm tests are in $AUTOTEST/client/kvm_tests/
> - all kvm tests inherent form $AUTOTEST/client/common_lib/kvm_test.py
> - common functionality is in $AUTOTEST/client/common_lib/kvm_test_utils/
>  - does *not* include generic kvm_test.cfg
> - we keep the $AUTOTEST/client/kvm/ test dir which defines the test runs
> and houses kvm_test.cfg file and a master control.
>  - we could then define a couple sample test runs: full, quick, and
> others  or implement something like your kvm_tests.common file that
> other test runs can build on.

Are all of your tests exclusive to KVM? I would think you'd want to be able
to run any "normal" test inside a KVM environment too?

Re: [Autotest] [RFC] KVM test: Refactoring the kvm control file and the config file

[Dor Laor]

On 07/21/2009 08:21 PM, Martin Bligh wrote:
>> The advantages I see are: 1. it more closely follows the current
>> autotest structure/layout, 2. solves the problem of separating each test
>> out of the ever growing kvm_test.py and gives a sub dir of each test for
>> better structure (something we have been talking about) and 3. addresses
>> the config vs. control file ? that this thread originally brought up.
>>
>> I think the issue is in how the "kvm test" is viewed.  Is it one test
>> that gets run against several configurations, or is it several different
>> tests with different configurations?.  I have been looking at it as the
>> later however I do also see it the other way as well.
>
> I think if you try to force everything you do into one test, you'll lose
> a lot of the power and flexibility of the system. I can't claim to have
> entirely figured out what you're doing, but it seems somewhat like
> you're reinventing some stuff with the current approach?
>
> Some of the general design premises:
>     1) Anything the user might want to configure should be in the control file
>     2) Anything in test should be really pretty static.
>     3) The way we get around a lot of the conflicts is by passing parameters
>         to run_test, though leaving sensible defaults in for them makes things
>         much easier to use.
>     4) The frontend and cli are designed to allow you to edit control files,
>         and/or save custom versions - that's the single object we throw
>         to machines under test ... there's no passing of cfg files to clients?
>
> We often end up with longer control files that contain a pre-canned set of
> tests, and even "meta-control files" that kick off a multitude of jobs across
> thousands of machines, using frontend.py. That can include control flow -
> for example our internal kernel testing uses a waterfall model with several
> steps:
>
> 1. Compile the kernel from source
> 2. Test on a bunch of single machines with a smoketest that takes an
> hour or so.
> 3. Test on small groups of machines with cut down simulations of
> cluster tests
> 4. Test on full clusters.
>
> If any of those tests fails (with some built in fault tolerance for a small
> hardware fallout rate), we stop the testing. All of that control flow
> is governed by a control file. It sounds complex, but it's really not
> if you build your "building blocks" carefully, and it's extremely powerful

+1

The highly flexible config file currently serves client mode tests.
We need to slowly shift functionality into the server while keeping the 
current advantages and simplicity of the client.

Martin, can you give some links to the above meta control?

>
>> So maybe the solution is a little different than my first thought....
>>
>> - all kvm tests are in $AUTOTEST/client/kvm_tests/
>> - all kvm tests inherent form $AUTOTEST/client/common_lib/kvm_test.py
>> - common functionality is in $AUTOTEST/client/common_lib/kvm_test_utils/
>>   - does *not* include generic kvm_test.cfg
>> - we keep the $AUTOTEST/client/kvm/ test dir which defines the test runs
>> and houses kvm_test.cfg file and a master control.
>>   - we could then define a couple sample test runs: full, quick, and
>> others  or implement something like your kvm_tests.common file that
>> other test runs can build on.
>
> Are all of your tests exclusive to KVM? I would think you'd want to be able
> to run any "normal" test inside a KVM environment too?

There are several autotest tests that run inside the guest today too.
Today the config file controls their execution. It would be nice if 
we'll create dependency using the server tests, that first installs VM, 
boot it and then runs various 'normal' tests inside of it.

Re: [Autotest] [RFC] KVM test: Refactoring the kvm control file and the config file

[Martin Bligh]

>> If any of those tests fails (with some built in fault tolerance for a
>> small
>> hardware fallout rate), we stop the testing. All of that control flow
>> is governed by a control file. It sounds complex, but it's really not
>> if you build your "building blocks" carefully, and it's extremely powerful
>
> +1
>
> The highly flexible config file currently serves client mode tests.
> We need to slowly shift functionality into the server while keeping the
> current advantages and simplicity of the client.
>
> Martin, can you give some links to the above meta control?

The control files themselves aren't published external to google, but
nearly all of the logic they use is - in server/frontend.py.

I really need to refactor that file, it has both flow logic in it, and
the basics for how to submit jobs to the frontend. The basic idea is
to create pairings of (test, machine label), then kick off a job to all
machines within that pairing and poll for the result. We typically use
3 to 5 of each machine type (platform) - if more than one of any
platform fails, we call that a failure.

The main entry point is run_test_suites(), which takes a list of
such pairings, along with a kernel to test, etc. We will probably need
do some work to generalize it, but the concept is all there, and it
works well for us.

>> Are all of your tests exclusive to KVM? I would think you'd want to be
>> able to run any "normal" test inside a KVM environment too?
>
> There are several autotest tests that run inside the guest today too.
> Today the config file controls their execution. It would be nice if we'll
> create dependency using the server tests, that first installs VM, boot it
> and then runs various 'normal' tests inside of it.

I'm assuming you want to be able to both run tests inside the guests
and on the raw host itself, at the same time? To that end, what we've
planned to do (but not completed yet) is to merge the client and server
code - so you can have autotest running on the scheduler, kicking off
jobs to the host. That host can then autonomously control it's own
guests, creating and destroying them, and kicking off tests inside of it.

Re: [RFC] KVM test: Refactoring the kvm control file and the config file

[Michael Goldish]

----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:

> Currently we have our kvm test control file and configuration file,
> having them split like this makes it harder for users to edit it,
> let's
> say, using the web frontend.
> 
> So it might be good to merge the control file and the config file,
> and
> make a refactor on the control file code. Do you think this would be
> a valid approach? Any comments are welcome.
> 
> Lucas

What exactly do you mean by merge? Embed the entire config file in
the control file as a python string?

A few comments:

1. The bulk of the config file usually doesn't need to be modified
from the web frontend, IMO. It actually doesn't need to be modified
very often -- once everything is defined, only minor changes are
required.

2. Changes to the config can be made in the control file rather easily
using kvm_config methods that are implemented but not currently used.
Instead of the short form:

list = kvm_config.config(filename).get_list()

we can use:

cfg = kvm_config.config(filename)

# parse any one-liner like this:
cfg.parse_string("only nightly")

# parse anything the parser understands like this:
cfg.parse_string("""
install:
    steps = blah
    foo = bar
only qcow2.*Windows
""")

# we can parse several times and the effect is cumulative
cfg.parse_string("""
variants:
    - foo:
        only scsi
    - bar:
        only WinVista.32
        variants:
            - 1:
            - 2:
""")

# we can also parse additional files:
cfg.parse_file("windows_cdkeys.cfg")

# finally, get the resulting list
list = cfg.get_list()

3. We may want to consider something in between having the control and
config completely separated (what we have today), and having them both
in the same file. For example, we can define the test sets (nightly,
weekly, fc8_quick, custom) in the config file, and select the test set
(e.g. "only nightly") in the control file by convention. Alternatively
we can omit the test sets from the config file, and just define a single
test set (the one we'll be using) in the control file, or define several
test sets in the control file, and select one of them.
We can actually do both things at the same time, by defining the test
sets in the config file, and defining a "full" test set among them (I
think it's already there), which doesn't modify anything. If we want to
use a standard test set from the config file, we can do "only nightly"
in the control, and if we want to use a custom test set, we can do:
cfg.parse_string("""
only full
# define the test set below (no need for variants)
only RHEL
only qcow2
only autotest.dbench
""")

4. It could be a good idea to make a "windows_cdkeys.cfg" file, that
contains mainly single-line exceptions, such as:
WinXP.32: cdkey = REPLACE_ME
WinXP.64: cdkey = REPLACE_ME
Win2003.32: cdkey = REPLACE_ME
...
The real cdkeys should be entered by the user. Then the file will be
parsed after kvm_tests.cfg, using the parse_file() method (in the
control). This way the user won't have to enter the cdkeys into the
long config file every time it gets replaced by a newer version. The
cdkeys file won't be replaced because it's specific to the test
environment (we'll only supply a sample like we do with kvm_tests.cfg).

Maybe we can generalize this idea and call the file local_prefs.cfg,
and decide that the file should contain any environment-specific
changes that the user wants to make to the config. The file will
contain mainly exceptions (single or multi-line). But I'm not sure
there are many environment specific things other than cdkeys, so maybe
this isn't necessary.


Let me know what you think.

Thanks,
Michael

Re: [RFC] KVM test: Refactoring the kvm control file and the config file

[Ryan Harper]

* Michael Goldish <mgoldish@redhat.com> [2009-07-21 07:38]:
> 
> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
> 
> > Currently we have our kvm test control file and configuration file,
> > having them split like this makes it harder for users to edit it,
> > let's
> > say, using the web frontend.
> > 
> > So it might be good to merge the control file and the config file,
> > and
> > make a refactor on the control file code. Do you think this would be
> > a valid approach? Any comments are welcome.
> > 
> > Lucas
> 
> What exactly do you mean by merge? Embed the entire config file in
> the control file as a python string?
> 
> A few comments:
> 
> 1. The bulk of the config file usually doesn't need to be modified
> from the web frontend, IMO. It actually doesn't need to be modified
> very often -- once everything is defined, only minor changes are
> required.

Agreed.  In fact, I have a kvm_tests.common file that has all of the
guest and parameter definitions, and then I have separate "test" files
that are appended to the common file to create a kvm_tests.cfg for the
specific tests I want to run.

> 
> 2. Changes to the config can be made in the control file rather easily
> using kvm_config methods that are implemented but not currently used.
> Instead of the short form:
> 
> list = kvm_config.config(filename).get_list()
> 
> we can use:
> 
> cfg = kvm_config.config(filename)
> 
> # parse any one-liner like this:
> cfg.parse_string("only nightly")
> 
> # parse anything the parser understands like this:
> cfg.parse_string("""
> install:
>     steps = blah
>     foo = bar
> only qcow2.*Windows
> """)
> 
> # we can parse several times and the effect is cumulative
> cfg.parse_string("""
> variants:
>     - foo:
>         only scsi
>     - bar:
>         only WinVista.32
>         variants:
>             - 1:
>             - 2:
> """)
> 
> # we can also parse additional files:
> cfg.parse_file("windows_cdkeys.cfg")
> 
> # finally, get the resulting list
> list = cfg.get_list()
> 
> 3. We may want to consider something in between having the control and
> config completely separated (what we have today), and having them both
> in the same file. For example, we can define the test sets (nightly,
> weekly, fc8_quick, custom) in the config file, and select the test set
> (e.g. "only nightly") in the control file by convention. Alternatively
> we can omit the test sets from the config file, and just define a single
> test set (the one we'll be using) in the control file, or define several
> test sets in the control file, and select one of them.

Yeah, this models what I'm doing today; common config file, and then a
separate test selector mechanism.  I'd actually prefer to not have to
touch the control file at all since it already has a bunch of logic and
other info in it; and just be able to specify my test selector file.  

I think your above examples imply we can do this with the code today:

cfg = kvm_config.config(kvm_tests_common)
 
# parse any one-liner like this:
cfg.parse_string("only nightly")

> We can actually do both things at the same time, by defining the test
> sets in the config file, and defining a "full" test set among them (I
> think it's already there), which doesn't modify anything. If we want to
> use a standard test set from the config file, we can do "only nightly"
> in the control, and if we want to use a custom test set, we can do:
> cfg.parse_string("""
> only full
> # define the test set below (no need for variants)
> only RHEL
> only qcow2
> only autotest.dbench
> """)

Yep.

> 
> 4. It could be a good idea to make a "windows_cdkeys.cfg" file, that
> contains mainly single-line exceptions, such as:
> WinXP.32: cdkey = REPLACE_ME
> WinXP.64: cdkey = REPLACE_ME
> Win2003.32: cdkey = REPLACE_ME
> ...
> The real cdkeys should be entered by the user. Then the file will be
> parsed after kvm_tests.cfg, using the parse_file() method (in the
> control). This way the user won't have to enter the cdkeys into the
> long config file every time it gets replaced by a newer version. The
> cdkeys file won't be replaced because it's specific to the test
> environment (we'll only supply a sample like we do with kvm_tests.cfg).

Yep, I like that as well.

> 
> Maybe we can generalize this idea and call the file local_prefs.cfg,
> and decide that the file should contain any environment-specific
> changes that the user wants to make to the config. The file will
> contain mainly exceptions (single or multi-line). But I'm not sure
> there are many environment specific things other than cdkeys, so maybe
> this isn't necessary.
> 
> 
> Let me know what you think.

I think have a common kvm_tests.cfg file that is automatically loaded
along with the additional one-liner/custom test selector mechanism go a
long way to providing what Lucas was asking for.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: [Autotest] [RFC] KVM test: Refactoring the kvm control file and the config file

[Lucas Meneghel Rodrigues]

On Tue, Jul 21, 2009 at 11:05 AM, Ryan Harper<ryanh@us.ibm.com> wrote:
> * Michael Goldish <mgoldish@redhat.com> [2009-07-21 07:38]:
>>
>> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
>>
>> > Currently we have our kvm test control file and configuration file,
>> > having them split like this makes it harder for users to edit it,
>> > let's
>> > say, using the web frontend.
>> >
>> > So it might be good to merge the control file and the config file,
>> > and
>> > make a refactor on the control file code. Do you think this would be
>> > a valid approach? Any comments are welcome.
>> >
>> > Lucas
>>
>> What exactly do you mean by merge? Embed the entire config file in
>> the control file as a python string?
>>
>> A few comments:
>>
>> 1. The bulk of the config file usually doesn't need to be modified
>> from the web frontend, IMO. It actually doesn't need to be modified
>> very often -- once everything is defined, only minor changes are
>> required.
>
> Agreed.  In fact, I have a kvm_tests.common file that has all of the
> guest and parameter definitions, and then I have separate "test" files
> that are appended to the common file to create a kvm_tests.cfg for the
> specific tests I want to run.
>
>>
>> 2. Changes to the config can be made in the control file rather easily
>> using kvm_config methods that are implemented but not currently used.
>> Instead of the short form:
>>
>> list = kvm_config.config(filename).get_list()
>>
>> we can use:
>>
>> cfg = kvm_config.config(filename)
>>
>> # parse any one-liner like this:
>> cfg.parse_string("only nightly")
>>
>> # parse anything the parser understands like this:
>> cfg.parse_string("""
>> install:
>>     steps = blah
>>     foo = bar
>> only qcow2.*Windows
>> """)
>>
>> # we can parse several times and the effect is cumulative
>> cfg.parse_string("""
>> variants:
>>     - foo:
>>         only scsi
>>     - bar:
>>         only WinVista.32
>>         variants:
>>             - 1:
>>             - 2:
>> """)
>>
>> # we can also parse additional files:
>> cfg.parse_file("windows_cdkeys.cfg")
>>
>> # finally, get the resulting list
>> list = cfg.get_list()
>>
>> 3. We may want to consider something in between having the control and
>> config completely separated (what we have today), and having them both
>> in the same file. For example, we can define the test sets (nightly,
>> weekly, fc8_quick, custom) in the config file, and select the test set
>> (e.g. "only nightly") in the control file by convention. Alternatively
>> we can omit the test sets from the config file, and just define a single
>> test set (the one we'll be using) in the control file, or define several
>> test sets in the control file, and select one of them.
>
> Yeah, this models what I'm doing today; common config file, and then a
> separate test selector mechanism.  I'd actually prefer to not have to
> touch the control file at all since it already has a bunch of logic and
> other info in it; and just be able to specify my test selector file.

Not editing the control file at all kinda subverts the original idea
of having the control as 'the' description of the test job. This is
one of the problems I was seeing with this approach. And when I mean
'refactor', I mean that the logic contained inside the control file
could be refactored to be made simpler.

That said, I don't think it is a *huge* problem right now, just
something that needs to be addressed at some point in time.

> I think your above examples imply we can do this with the code today:
>
> cfg = kvm_config.config(kvm_tests_common)
>
> # parse any one-liner like this:
> cfg.parse_string("only nightly")
>
>> We can actually do both things at the same time, by defining the test
>> sets in the config file, and defining a "full" test set among them (I
>> think it's already there), which doesn't modify anything. If we want to
>> use a standard test set from the config file, we can do "only nightly"
>> in the control, and if we want to use a custom test set, we can do:
>> cfg.parse_string("""
>> only full
>> # define the test set below (no need for variants)
>> only RHEL
>> only qcow2
>> only autotest.dbench
>> """)
>
> Yep.
>
>>
>> 4. It could be a good idea to make a "windows_cdkeys.cfg" file, that
>> contains mainly single-line exceptions, such as:
>> WinXP.32: cdkey = REPLACE_ME
>> WinXP.64: cdkey = REPLACE_ME
>> Win2003.32: cdkey = REPLACE_ME
>> ...
>> The real cdkeys should be entered by the user. Then the file will be
>> parsed after kvm_tests.cfg, using the parse_file() method (in the
>> control). This way the user won't have to enter the cdkeys into the
>> long config file every time it gets replaced by a newer version. The
>> cdkeys file won't be replaced because it's specific to the test
>> environment (we'll only supply a sample like we do with kvm_tests.cfg).
>
> Yep, I like that as well.
>
>>
>> Maybe we can generalize this idea and call the file local_prefs.cfg,
>> and decide that the file should contain any environment-specific
>> changes that the user wants to make to the config. The file will
>> contain mainly exceptions (single or multi-line). But I'm not sure
>> there are many environment specific things other than cdkeys, so maybe
>> this isn't necessary.
>>
>>
>> Let me know what you think.
>
> I think have a common kvm_tests.cfg file that is automatically loaded
> along with the additional one-liner/custom test selector mechanism go a
> long way to providing what Lucas was asking for.

Yes, that would be a valid approach. Thanks for sharing your ideas, Ryan.

Re: [RFC] KVM test: Refactoring the kvm control file and the config file

[David Huff]

Michael Goldish wrote:
> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
> 
>> Currently we have our kvm test control file and configuration file,
>> having them split like this makes it harder for users to edit it,
>> let's
>> say, using the web frontend.
>>
>> So it might be good to merge the control file and the config file,
>> and
>> make a refactor on the control file code. Do you think this would be
>> a valid approach? Any comments are welcome.
>>
>> Lucas
> 
> What exactly do you mean by merge? Embed the entire config file in
> the control file as a python string?
> 
> A few comments:
> 
> 1. The bulk of the config file usually doesn't need to be modified
> from the web frontend, IMO. It actually doesn't need to be modified
> very often -- once everything is defined, only minor changes are
> required.
> 
> 2. Changes to the config can be made in the control file rather easily
> using kvm_config methods that are implemented but not currently used.
> Instead of the short form:
> 
> list = kvm_config.config(filename).get_list()
> 
> we can use:
> 
> cfg = kvm_config.config(filename)
> 
> # parse any one-liner like this:
> cfg.parse_string("only nightly")
> 
> # parse anything the parser understands like this:
> cfg.parse_string("""
> install:
>     steps = blah
>     foo = bar
> only qcow2.*Windows
> """)
> 
> # we can parse several times and the effect is cumulative
> cfg.parse_string("""
> variants:
>     - foo:
>         only scsi
>     - bar:
>         only WinVista.32
>         variants:
>             - 1:
>             - 2:
> """)
> 
> # we can also parse additional files:
> cfg.parse_file("windows_cdkeys.cfg")
> 
> # finally, get the resulting list
> list = cfg.get_list()
> 
> 3. We may want to consider something in between having the control and
> config completely separated (what we have today), and having them both
> in the same file. For example, we can define the test sets (nightly,
> weekly, fc8_quick, custom) in the config file, and select the test set
> (e.g. "only nightly") in the control file by convention. Alternatively
> we can omit the test sets from the config file, and just define a single
> test set (the one we'll be using) in the control file, or define several
> test sets in the control file, and select one of them.
> We can actually do both things at the same time, by defining the test
> sets in the config file, and defining a "full" test set among them (I
> think it's already there), which doesn't modify anything. If we want to
> use a standard test set from the config file, we can do "only nightly"
> in the control, and if we want to use a custom test set, we can do:
> cfg.parse_string("""
> only full
> # define the test set below (no need for variants)
> only RHEL
> only qcow2
> only autotest.dbench
> """)
> 
> 4. It could be a good idea to make a "windows_cdkeys.cfg" file, that
> contains mainly single-line exceptions, such as:
> WinXP.32: cdkey = REPLACE_ME
> WinXP.64: cdkey = REPLACE_ME
> Win2003.32: cdkey = REPLACE_ME
> ...
> The real cdkeys should be entered by the user. Then the file will be
> parsed after kvm_tests.cfg, using the parse_file() method (in the
> control). This way the user won't have to enter the cdkeys into the
> long config file every time it gets replaced by a newer version. The
> cdkeys file won't be replaced because it's specific to the test
> environment (we'll only supply a sample like we do with kvm_tests.cfg).
> 
> Maybe we can generalize this idea and call the file local_prefs.cfg,
> and decide that the file should contain any environment-specific
> changes that the user wants to make to the config. The file will
> contain mainly exceptions (single or multi-line). But I'm not sure
> there are many environment specific things other than cdkeys, so maybe
> this isn't necessary.
> 
> 
> Let me know what you think.

Michael all very good comments, I specifically like the windows config
file idea.

The way I always envisioned it was something like this......

The config file specifies the whole test matrix, ie all variants that
you could run each test on, ie all os's, all archs, all disk types, all
cpu/mem configurations.

The control file would be more of a test specific config file, setting
any local or environmental vars for each test, and like Michael said can
override "stuff" fromt he main confg file...

I also really like the idea of creating a generic kvm_test that all kvm
 tests would inherent from, ie. $AUTOTEST/client/common_lib/kvm_test.py

All helper classes ie. kvm.py, kvm_utils.py, kvm_config.py, and even the
config file itself could then go into
$AUTOTEST/client/common_lib/test_utils/ or even maybe something like
$AUTOTEST/client/common_lib/kvm_test_utils/

All kvm specific tests would inherent form the generic kvm_test, and
then go into either $AUTOTEST/client/tests/ or
$AUTOTEST/client/kvm_tests/ directorys, each having their own sub dir
like the current autotest tests.  In this dir there would be a control
file specific for each test, that can override the full test matrix
descried inthe the generic kvm_tests.cfg, as well as any additional file
required by the test.

Anyway just some of my thoughts, I know its great in theory however may
have some implementation short falls, like interdependence between tests
and such...


Comments..

Re: [Autotest] [RFC] KVM test: Refactoring the kvm control file and the config file

[Lucas Meneghel Rodrigues]

On Tue, Jul 21, 2009 at 11:37 AM, David Huff<dhuff@redhat.com> wrote:
> Michael all very good comments, I specifically like the windows config
> file idea.
>
> The way I always envisioned it was something like this......
>
> The config file specifies the whole test matrix, ie all variants that
> you could run each test on, ie all os's, all archs, all disk types, all
> cpu/mem configurations.
>
> The control file would be more of a test specific config file, setting
> any local or environmental vars for each test, and like Michael said can
> override "stuff" fromt he main confg file...

It's not to say that separating the test matrix definition from the
test job logic is a bad idea organization-wise, however like I pointed
out on Michael's and Ryan's comments, it sort of defeats the original
autotest design goals (that was pointed out by Martin later on this
thread, I'll get to it later).

> I also really like the idea of creating a generic kvm_test that all kvm
>  tests would inherent from, ie. $AUTOTEST/client/common_lib/kvm_test.py
>
> All helper classes ie. kvm.py, kvm_utils.py, kvm_config.py, and even the
> config file itself could then go into
> $AUTOTEST/client/common_lib/test_utils/ or even maybe something like
> $AUTOTEST/client/common_lib/kvm_test_utils/

Yes, moving the large amount of infrastructure developed for the kvm
test into the autotest library namespace is something I also want to
do at some point in time. Good point.

> All kvm specific tests would inherent form the generic kvm_test, and
> then go into either $AUTOTEST/client/tests/ or
> $AUTOTEST/client/kvm_tests/ directorys, each having their own sub dir
> like the current autotest tests.  In this dir there would be a control
> file specific for each test, that can override the full test matrix
> descried inthe the generic kvm_tests.cfg, as well as any additional file
> required by the test.

Yes, moving tests out of the kvm_test realm is good, since it keeps
things more manageable on the kvm test itself. However, we need to
take into account non-linux guests, where we can't just yet use raw
autotest.

> Anyway just some of my thoughts, I know its great in theory however may
> have some implementation short falls, like interdependence between tests
> and such...

Yes, at some point, I want to take more advantage of server based
testing. We would have a server control file that:

1) A test that sets up a test client (A physical box running linux) to
be a virtualization host (KvmHost) and creates guests. The result of
this test must be a number of KvmGuests, that can be later used to
perform tests on it.
2) Just instantiate autotest inside the KvmGuests and run tests on
them using the server side control API.

Easier said than done, because we will deal with windows guests, test
dependency and controlling hypervisor parameters. So I am not saying
that this is something we need to jump in right now. My main concerns
right now are:

1) Make sure the infrastructure we've got now works really well
2) Grow the amount of tests we have so we can increase KVM testing coverage

Working out architectural changes would come later, at a more
appropriate time. Refactoring the control file and the test config
file is one of the improvements we can make on the short term using
the client model, so that's why I started the discussion.

Thanks for your thoughts on that David.