Re: [RFC] KVM test: Refactoring the kvm control file and the config file

[David Huff <dhuff@redhat.com>]

Michael Goldish wrote:
> ----- "David Huff" <dhuff@redhat.com> wrote:
>> The way I always envisioned it was something like this......
>>
>> The config file specifies the whole test matrix, ie all variants that
>> you could run each test on, ie all os's, all archs, all disk types,
>> all
>> cpu/mem configurations.
>>
>> The control file would be more of a test specific config file,
>> setting
>> any local or environmental vars for each test, and like Michael said
>> can
>> override "stuff" fromt he main confg file...
>>
>> I also really like the idea of creating a generic kvm_test that all
>> kvm
>>  tests would inherent from, ie.
>> $AUTOTEST/client/common_lib/kvm_test.py
>>
>> All helper classes ie. kvm.py, kvm_utils.py, kvm_config.py, and even
>> the
>> config file itself could then go into
>> $AUTOTEST/client/common_lib/test_utils/ or even maybe something like
>> $AUTOTEST/client/common_lib/kvm_test_utils/
>>
>> All kvm specific tests would inherent form the generic kvm_test, and
>> then go into either $AUTOTEST/client/tests/ or
>> $AUTOTEST/client/kvm_tests/ directorys, each having their own sub dir
>> like the current autotest tests.  In this dir there would be a
>> control
>> file specific for each test, that can override the full test matrix
>> descried inthe the generic kvm_tests.cfg, as well as any additional
>> file
>> required by the test.
>>
>> Anyway just some of my thoughts, I know its great in theory however
>> may
>> have some implementation short falls, like interdependence between
>> tests
>> and such...
>>
>>
>> Comments..
> 
> I think I understand your suggestion, but let me make sure:
> 
> - If there's a global config file that is shared by all tests, I suppose
> it'll run all the tests one by one, right?
> 
> - Where will test sets be defined -- in the global config file?
> 
> - If each individual test inherits from the global config file, it'll
> also inherit dictionaries describing other tests, right?
> e.g. the configuration of the boot test must explicitly state "only boot",
> or it'll run install, migration and autotest as well?
> 
> - If you run the control file of a specific test, what happens -- does
> that specific test run in many configurations (many guests, cpu options,
> network, ide/scsi), or does it run just once with a single configuration?
> I suppose the "normal" behavior would be to run in many configurations, but
> I'm not sure what your intention was.
> 
> - Will the global config file look like the config files we have today?
> 
> I think this should be possible to implement, but I haven't given it much
> thought so I'm not sure. The more interesting question is whether it's a
> good idea. What are the advantages over the current approach?

The advantages I see are: 1. it more closely follows the current
autotest structure/layout, 2. solves the problem of separating each test
out of the ever growing kvm_test.py and gives a sub dir of each test for
better structure (something we have been talking about) and 3. addresses
the config vs. control file ? that this thread originally brought up.

I think the issue is in how the "kvm test" is viewed.  Is it one test
that gets run against several configurations, or is it several different
tests with different configurations?.  I have been looking at it as the
later however I do also see it the other way as well.

So maybe the solution is a little different than my first thought....

- all kvm tests are in $AUTOTEST/client/kvm_tests/
- all kvm tests inherent form $AUTOTEST/client/common_lib/kvm_test.py
- common functionality is in $AUTOTEST/client/common_lib/kvm_test_utils/
  - does *not* include generic kvm_test.cfg
- we keep the $AUTOTEST/client/kvm/ test dir which defines the test runs
and houses kvm_test.cfg file and a master control.
  - we could then define a couple sample test runs: full, quick, and
others  or implement something like your kvm_tests.common file that
other test runs can build on.

So in the end its pretty similar to what we currently have except that
the AUTOTEST/client/kvm/ dir only defines the test runs, all common
functionality and the tests them selves are moved out.

The major advantages I see, aside from the three mention above, is that
it allows us to simplify the kvm_tests.cfg file.  We can move the test
specific config to each test dir,
$AUTOTEST/client/kvm_tests/install/install.cfg which includes all the
install test parameters.  Which combined with splitting up the config
file in $AUTOTEST/client/kvm/ would make the config file shorter and
easier to read.

Again not sure if all this is worth it however some of my thoughts on
how to improve the current status.

-D