Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish <mgoldish@redhat.com> wrote:
> Sounds great, except it won't allow you to debug your configuration
> using kvm_config.py.  So the question now is what's more important --
> the ability to debug or ease of use when running from the server.

Here we have 2 use cases:

1) Users of the web interface, that (hopefully) have canned test sets
that work reliably. Ability to debug stuff is less important on this
scenario.
2) People developing tests, and in this case ability to debug config
is very important

I see the following options:

1) Document as part of the "test development guide" that, in order to
be able to debug stuff, that all the test sets are to be written to
the config file and then, can be parsed using kvm_config.
2) If we write all dictionaries generated by that particular
configuration on files inside the job results directory, we still have
debug ability for all use cases (I am starting to like this idea very
much, as I type).

So I'd then implement option 2) and refactor the control file with the
test sets defined inside strings in the control file, then you can see
how it looks? How about that?

-- 
Lucas

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

On Wed, Oct 28, 2009 at 1:57 PM, Michael Goldish <mgoldish@redhat.com> wrote:
> One more comment on this patch:
>
> If you plan to make 'noinstall' the default build mode, then let's
> not run the build test at all by default.  If we run it with 'noinstall',
> it will just run the preprocessor and postprocessor, and those will kill
> any living VMs because no VMs are required for the build test.  So a user
> won't be able to run a reboot test, keep the VM alive, modify the config
> file and then run a different test on the same living VM, because the
> preprocessor of the second 'build' test will kill the VM.
>
> To avoid this, we can just remove the 'noinstall' variant from
> kvm_build.cfg, and keep the 'only noinstall' statement.  Because there's
> no 'noinstall' variant, the resulting test list will be empty, so nothing
> will run.  We should probably add a comment reminding the user that it's
> possible to disable installation by stating 'only noinstall' or 'only
> nothing' (in case the user replaced 'noinstall' with something else).
>
> Alternatively, we can disable installation in the control file by commenting
> out some line.

Great idea, thanks :)

-
Lucas

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Michael Goldish]

----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:

> On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish <mgoldish@redhat.com>
> wrote:
> > Sounds great, except it won't allow you to debug your configuration
> > using kvm_config.py.  So the question now is what's more important
> --
> > the ability to debug or ease of use when running from the server.
> 
> Here we have 2 use cases:
> 
> 1) Users of the web interface, that (hopefully) have canned test sets
> that work reliably. Ability to debug stuff is less important on this
> scenario.
> 2) People developing tests, and in this case ability to debug config
> is very important
> 
> I see the following options:
> 
> 1) Document as part of the "test development guide" that, in order to
> be able to debug stuff, that all the test sets are to be written to
> the config file and then, can be parsed using kvm_config.
> 2) If we write all dictionaries generated by that particular
> configuration on files inside the job results directory, we still
> have
> debug ability for all use cases (I am starting to like this idea very
> much, as I type).
> 
> So I'd then implement option 2) and refactor the control file with
> the
> test sets defined inside strings in the control file, then you can
> see
> how it looks? How about that?

Sounds fine.
- Where exactly will the test list appear?
- We should also allow printing of verbose debug output ("parsing variants
block, 9000 dicts in current context...") by passing something to the
constructor of the config object.
- We should make it clear to the user that he/she must rename the control
file (to control.lucas for example) or else it may be overwritten on the
next git-fetch or -pull.

I'm still not sure it's a great idea to make config debugging harder, so
if anyone other than Lucas who uses the KVM test is reading this, please
let us know if you ever use kvm_config.py and if you think the ability to
print the list of test dicts is important.

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

Ryan, Michael:

I absolutely agree that the ability to debug stuff is important, but
the ability to make things straightforward to use from the web
interface or cli is also important. A longer term goal is to have our
test farm and make any developer able to schedule a job on the test
farm easily and conveniently.

Having the dictionaries generated on the job debug directory seems
like a good compromise to me. Also we can come up with a smart way of
parsing the config file generated by a given control file in a similar
way we do today with kvm_config.py, it shouldn't be that hard to do
it... (I hope I won't burn my tongue with this statement).

Thanks for your input!

On Wed, Oct 28, 2009 at 4:18 PM, Ryan Harper <ryanh@us.ibm.com> wrote:
> * Michael Goldish <mgoldish@redhat.com> [2009-10-28 10:43]:
>>
>> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
>>
>> > One thing that just occurred to me is, if we have a test config
>> > "library" as you said, it's perfectly possible to put the actual test
>> > set definitions and other config files inside the control file as
>> > strings. This way one can control configuration inside the control
>> > file,
>> > making it more convenient for usage, let's say, inside the autotest
>> > web
>> > interface. Since the control file would be reduced in size, the
>> > configuration snippets being in the control file would not be a huge
>> > problem, while keeping the original autotest philosophy of keeping
>> > stuff inside the control file... What do you think?
>>
>> Sounds great, except it won't allow you to debug your configuration
>> using kvm_config.py.  So the question now is what's more important --
>> the ability to debug or ease of use when running from the server.
>
> +1 debug
>
> When creating new test scenarios I make *heavy* use of kvm_config.py;
> I'd be lost without being able to debug test configuration files.
>
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> ryanh@us.ibm.com
> _______________________________________________
> Autotest mailing list
> Autotest@test.kernel.org
> http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
>



-- 
Lucas

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Michael Goldish]

It's OK to log the debug output when running the framework, but
Ryan says he makes heavy use of kvm_config.py, which means he may
not find this comfortable enough.  Ryan, what do you think?

Another option is to allow editing any file from the web GUI in
addition to the control file.

I don't think it'll be hard to parse the control file and
automatically extract config code from it in order to debug it,
but it will almost certainly be very ugly.

We might want to consider writing a little program that will emulate
the client/bin/autotest.py program, running the control file and
providing it with a fake job object whose run_test() method will
just print the params dict instead of running the test.

----- Original Message -----
From: Lucas Meneghel Rodrigues <lmr@redhat.com>
To: Ryan Harper <ryanh@us.ibm.com>
Cc: Michael Goldish <mgoldish@redhat.com>, autotest@test.kernel.org, uril@redhat.com, kvm@vger.kernel.org
Sent: Wed, 28 Oct 2009 15:47:54 -0400 (EDT)
Subject: Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

Ryan, Michael:

I absolutely agree that the ability to debug stuff is important, but
the ability to make things straightforward to use from the web
interface or cli is also important. A longer term goal is to have our
test farm and make any developer able to schedule a job on the test
farm easily and conveniently.

Having the dictionaries generated on the job debug directory seems
like a good compromise to me. Also we can come up with a smart way of
parsing the config file generated by a given control file in a similar
way we do today with kvm_config.py, it shouldn't be that hard to do
it... (I hope I won't burn my tongue with this statement).

Thanks for your input!

On Wed, Oct 28, 2009 at 4:18 PM, Ryan Harper <ryanh@us.ibm.com> wrote:
> * Michael Goldish <mgoldish@redhat.com> [2009-10-28 10:43]:
>>
>> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
>>
>> > One thing that just occurred to me is, if we have a test config
>> > "library" as you said, it's perfectly possible to put the actual test
>> > set definitions and other config files inside the control file as
>> > strings. This way one can control configuration inside the control
>> > file,
>> > making it more convenient for usage, let's say, inside the autotest
>> > web
>> > interface. Since the control file would be reduced in size, the
>> > configuration snippets being in the control file would not be a huge
>> > problem, while keeping the original autotest philosophy of keeping
>> > stuff inside the control file... What do you think?
>>
>> Sounds great, except it won't allow you to debug your configuration
>> using kvm_config.py.  So the question now is what's more important --
>> the ability to debug or ease of use when running from the server.
>
> +1 debug
>
> When creating new test scenarios I make *heavy* use of kvm_config.py;
> I'd be lost without being able to debug test configuration files.
>
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> ryanh@us.ibm.com
> _______________________________________________
> Autotest mailing list
> Autotest@test.kernel.org
> http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
>



-- 
Lucas

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Ryan Harper]

* Lucas Meneghel Rodrigues <lmr@redhat.com> [2009-10-28 14:48]:
> Ryan, Michael:
> 
> I absolutely agree that the ability to debug stuff is important, but
> the ability to make things straightforward to use from the web
> interface or cli is also important. A longer term goal is to have our
> test farm and make any developer able to schedule a job on the test
> farm easily and conveniently.
> 
> Having the dictionaries generated on the job debug directory seems
> like a good compromise to me. Also we can come up with a smart way of
> parsing the config file generated by a given control file in a similar
> way we do today with kvm_config.py, it shouldn't be that hard to do
> it... (I hope I won't burn my tongue with this statement).

If I'm understanding things, we are talking about moving the large body
of kvm_tests.cfg test definitions, guest definitions into a "library",
and then moving the requested test config (bottom on kvm_tests.cfg) into
the control file itself which means the autotest webui would be able to
control which tests get run;  I like this idea very well.  My concern
that I mentioned is that as you edit the "library" it can be difficult
to ensure you described exactly which set of tests on which guests you
want to run and kvm_config.py is invaluable in the process of getting it
right.

Why not have kvm_config.py , or some other wrapper generate a
"kvm_tests.cfg" file dynamically from the "library" and the strings from
the control file?  That way we could still debug configuration via
kvm_config.py?  I much perfer this over queueing up jobs in the webiu,
waiting for it to run, checking the results in the DEBUG dir, adjusting,
repeat.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

On Fri, 2009-10-30 at 08:54 -0500, Ryan Harper wrote:
> * Lucas Meneghel Rodrigues <lmr@redhat.com> [2009-10-28 14:48]:
> > Ryan, Michael:
> > 
> > I absolutely agree that the ability to debug stuff is important, but
> > the ability to make things straightforward to use from the web
> > interface or cli is also important. A longer term goal is to have our
> > test farm and make any developer able to schedule a job on the test
> > farm easily and conveniently.
> > 
> > Having the dictionaries generated on the job debug directory seems
> > like a good compromise to me. Also we can come up with a smart way of
> > parsing the config file generated by a given control file in a similar
> > way we do today with kvm_config.py, it shouldn't be that hard to do
> > it... (I hope I won't burn my tongue with this statement).
> 
> If I'm understanding things, we are talking about moving the large body
> of kvm_tests.cfg test definitions, guest definitions into a "library",
> and then moving the requested test config (bottom on kvm_tests.cfg) into
> the control file itself which means the autotest webui would be able to
> control which tests get run;

That is precisely my idea!

> I like this idea very well.  My concern
> that I mentioned is that as you edit the "library" it can be difficult
> to ensure you described exactly which set of tests on which guests you
> want to run and kvm_config.py is invaluable in the process of getting it
> right.

Yes, very valid concern. Let's try to address it.

> Why not have kvm_config.py , or some other wrapper generate a
> "kvm_tests.cfg" file dynamically from the "library" and the strings from
> the control file? 

That's one idea, the concern is that the code to do that might become
ugly. But I will give it a try.

> That way we could still debug configuration via
> kvm_config.py?  I much perfer this over queueing up jobs in the webiu,
> waiting for it to run, checking the results in the DEBUG dir, adjusting,
> repeat.

Ok, I will go after this solution.

Thanks for your comments Ryan,

Lucas

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Michael Goldish]

----- "Ryan Harper" <ryanh@us.ibm.com> wrote:

> * Lucas Meneghel Rodrigues <lmr@redhat.com> [2009-10-28 14:48]:
> > Ryan, Michael:
> > 
> > I absolutely agree that the ability to debug stuff is important,
> but
> > the ability to make things straightforward to use from the web
> > interface or cli is also important. A longer term goal is to have
> our
> > test farm and make any developer able to schedule a job on the test
> > farm easily and conveniently.
> > 
> > Having the dictionaries generated on the job debug directory seems
> > like a good compromise to me. Also we can come up with a smart way
> of
> > parsing the config file generated by a given control file in a
> similar
> > way we do today with kvm_config.py, it shouldn't be that hard to do
> > it... (I hope I won't burn my tongue with this statement).
> 
> If I'm understanding things, we are talking about moving the large
> body
> of kvm_tests.cfg test definitions, guest definitions into a
> "library",
> and then moving the requested test config (bottom on kvm_tests.cfg)
> into
> the control file itself which means the autotest webui would be able
> to
> control which tests get run;  I like this idea very well.  My concern
> that I mentioned is that as you edit the "library" it can be
> difficult
> to ensure you described exactly which set of tests on which guests
> you
> want to run and kvm_config.py is invaluable in the process of getting
> it
> right.
> 
> Why not have kvm_config.py , or some other wrapper generate a
> "kvm_tests.cfg" file dynamically from the "library" and the strings
> from
> the control file?  That way we could still debug configuration via
> kvm_config.py?  I much perfer this over queueing up jobs in the
> webiu,
> waiting for it to run, checking the results in the DEBUG dir,
> adjusting,
> repeat.

I'm not sure I understand your idea: you want some program to read the
control file and generate a new file (kvm_tests.cfg or something) from
the control file and the library file, so that this file can be debugged
with kvm_config.py?

IMO this solution is "dirty" because the control file is python code, not
our own format, so it's not nice to automatically extract stuff from it.
It would be nice to do something that eases debugging, but if you ask me,
I'd rather have something as clean as possible.

Here's another idea, which I suggested but haven't received any feedback
on: let's write a little proggie that runs the control file just like
client/bin/autotest does.  The proggie will supply the control file with
a fake job object that has nothing but a run_test() method, but instead
of running a test, that method will simply nicely print out the test
params, like kvm_config.py does.  So the user will be able to do something
like './dry_run.py control.mine' which will list all the tests to be
executed.  We might also want to implement job.parallel() in addition to
job.run_test() but that should be very easy (it doesn't really have to be
parallel at all).
Does that make any sense?

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Ryan Harper]

* Michael Goldish <mgoldish@redhat.com> [2009-10-31 16:02]:
> > 
> > If I'm understanding things, we are talking about moving the large
> > body
> > of kvm_tests.cfg test definitions, guest definitions into a
> > "library",
> > and then moving the requested test config (bottom on kvm_tests.cfg)
> > into
> > the control file itself which means the autotest webui would be able
> > to
> > control which tests get run;  I like this idea very well.  My concern
> > that I mentioned is that as you edit the "library" it can be
> > difficult
> > to ensure you described exactly which set of tests on which guests
> > you
> > want to run and kvm_config.py is invaluable in the process of getting
> > it
> > right.
> > 
> > Why not have kvm_config.py , or some other wrapper generate a
> > "kvm_tests.cfg" file dynamically from the "library" and the strings
> > from
> > the control file?  That way we could still debug configuration via
> > kvm_config.py?  I much perfer this over queueing up jobs in the
> > webiu,
> > waiting for it to run, checking the results in the DEBUG dir,
> > adjusting,
> > repeat.
> 
> I'm not sure I understand your idea: you want some program to read the
> control file and generate a new file (kvm_tests.cfg or something) from
> the control file and the library file, so that this file can be debugged
> with kvm_config.py?

I wanted something that would take the test description string from the
control file, run that through the library such that it can print out
the tests that will run -- the equivalent of running kvm_config.py on a
kvm_tests.cfg file... 

> 
> IMO this solution is "dirty" because the control file is python code, not
> our own format, so it's not nice to automatically extract stuff from it.
> It would be nice to do something that eases debugging, but if you ask me,
> I'd rather have something as clean as possible.
> 
> Here's another idea, which I suggested but haven't received any feedback
> on: let's write a little proggie that runs the control file just like
> client/bin/autotest does.  The proggie will supply the control file with
> a fake job object that has nothing but a run_test() method, but instead
> of running a test, that method will simply nicely print out the test
> params, like kvm_config.py does.  So the user will be able to do something
> like './dry_run.py control.mine' which will list all the tests to be
> executed.  We might also want to implement job.parallel() in addition to
> job.run_test() but that should be very easy (it doesn't really have to be
> parallel at all).
> Does that make any sense?

It does, though I don't see how it's different then what I suggested; I
don't really care how we do it;  I still need something to debug test
config files outside of cycling through job output via the webui;


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Yolkfull Chow]

On Wed, Oct 28, 2009 at 02:04:59PM -0400, Michael Goldish wrote:
> 
> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
> 
> > On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish <mgoldish@redhat.com>
> > wrote:
> > > Sounds great, except it won't allow you to debug your configuration
> > > using kvm_config.py.  So the question now is what's more important
> > --
> > > the ability to debug or ease of use when running from the server.
> > 
> > Here we have 2 use cases:
> > 
> > 1) Users of the web interface, that (hopefully) have canned test sets
> > that work reliably. Ability to debug stuff is less important on this
> > scenario.
> > 2) People developing tests, and in this case ability to debug config
> > is very important
> > 
> > I see the following options:
> > 
> > 1) Document as part of the "test development guide" that, in order to
> > be able to debug stuff, that all the test sets are to be written to
> > the config file and then, can be parsed using kvm_config.
> > 2) If we write all dictionaries generated by that particular
> > configuration on files inside the job results directory, we still
> > have
> > debug ability for all use cases (I am starting to like this idea very
> > much, as I type).
> > 
> > So I'd then implement option 2) and refactor the control file with
> > the
> > test sets defined inside strings in the control file, then you can
> > see
> > how it looks? How about that?
> 
> Sounds fine.
> - Where exactly will the test list appear?
> - We should also allow printing of verbose debug output ("parsing variants
> block, 9000 dicts in current context...") by passing something to the
> constructor of the config object.
> - We should make it clear to the user that he/she must rename the control
> file (to control.lucas for example) or else it may be overwritten on the
> next git-fetch or -pull.
> 
> I'm still not sure it's a great idea to make config debugging harder, so
> if anyone other than Lucas who uses the KVM test is reading this, please
> let us know if you ever use kvm_config.py and if you think the ability to
> print the list of test dicts is important.

Hi Michael,

I had used kvm_config.py for printing lists of selected test dicts
often. And I think it's necessary to keep this feature. IMHO, option 2) Lucas
proposed is a good idea. What do you think? Hope I haven't missed
something. :)

> _______________________________________________
> Autotest mailing list
> Autotest@test.kernel.org
> http://test.kernel.org/cgi-bin/mailman/listinfo/autotest

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

On Wed, Nov 4, 2009 at 5:25 AM, Yolkfull Chow <yzhou@redhat.com> wrote:
> Hi Michael,
>
> I had used kvm_config.py for printing lists of selected test dicts
> often. And I think it's necessary to keep this feature. IMHO, option 2) Lucas
> proposed is a good idea. What do you think? Hope I haven't missed
> something. :)

2) is a compromise solution, but we find it's cumbersome, so we are
going to do something that can print all test dicts from a control
file, in addition to 2).

Thanks for your input!

-- 
Lucas