Re: [RFC] KVM-Autotest: basic parallel test execution

[Mike Burns <mburns@redhat.com>]

Ryan Harper wrote:
> * Michael Goldish <mgoldish@redhat.com> [2009-05-17 09:50]:
>   
>> Hi all,
>>
>> We've recently implemented a very simple form of parallel test
>> execution into KVM-Autotest and we'd like some feedback on it. This
>> suggestion allows the user to manually assign tests to hosts/queues.
>> It also takes care of assigning different MAC address ranges to
>> hosts/queues. By 'queues' I mean parallel execution pipelines. Each
>> host has one or more queues. The number of queues is defined by the
>> user, and should reflect the capabilities of the host.
>>
>> This implementation involves only minor modifications to the code
>> itself; most of the work is done in a new config file kvm_hosts.cfg,
>> which has the exact same format as kvm_tests.cfg. The new file
>> provides the framework with information about hosts/queues. The new
>> file is parsed after kvm_tests.cfg. The test sets (such as
>> 'nightly' and 'weekly'), previously defined at the end of
>> kvm_tests.cfg, should now be defined last, after kvm_hosts.cfg.
>> Test sets no longer select only the tests to execute, but also
>> where each test should be executed (i.e. on what host/queue).
>>
>> The final result of parsing the config files is a list of tests, each
>> with its own 'hostname' and 'queue' parameters. Each host executes
>> only the tests whose 'hostname' parameter matches the current host,
>> and puts tests with different 'queue' values in parallel pipelines
>> of execution.
>>
>> Ideally, the Autotest server should take care of assigning tests to
>> hosts automatically, but there are still a few technical difficulties
>> to be resolved before we can implement that. We're considering the
>> current suggestion as a temporary solution until a better one is
>> found.
>>
>> Basically, the advantages are:
>> - allows the user full control over what tests run, and where/how they run
>> - takes care of assigning MAC address ranges to different hosts/queues (required for TAP networking)
>> - can be used from the server or with the client, which makes it relevant also for users who don't have an Autotest server installed
>> - involves only minor code changes (except for the config files)
>> - is pretty much the simplest possible solution (and simple is good)
>>
>> Drawbacks:
>> - requires some initial work to be done by the user -- the user has to define exactly where each test should run
>> - test sets need to be modified when tests or hosts are added/removed, to include/exclude them
>>     
>
> I took a slightly different approach.  The kvm_tests.cfg file already
> provides a dependency relationship between different tests.  I modified
> the main loop in the control file to walk the entire list of jobs and
> pull out any jobs that don't have any dependencies (ie, install tests).
> And then run N jobs in parallel from that list until it is exhausted;
> then store the results.  Then loop the over the remaining list of jobs
> again finding the jobs that can be run.
>
> On a larger multi core system, one might set the number of parallel jobs
> equal to the number of cores.
>
> I think this works well with using autoserv to farm out different
> kvm_tests.cfg to different machines.
>
> Attaching my stale patch just for comment.  Needs to be updated since I
> sat on this for a while.  There were a number of issues:
>
> - kvm_log is a shared resource, fixed it up so parallel jobs can both
>   call it
> - vnc, redir and other network resources are shared, so, in
> kvm_tests.cfg file each job needs a parallel offset.
> - in kvm_tests.cfg file need to define additional vm and nic objects,
>    one for each parallel threads.
>
> Advantages:
>     - works a lot like the single threaded model does, and if threads=1
>     then it runs the same path
>     - config files don't change significantly, just some additional
>     VM objects at the top and some offset values
>     - transparent to an autoserv setup, autoserv would just need to
>     specify the kvm_tests.cfg file to each host.
>
> Disadvantages:
>     - the main loop waits for each group of parallel jobs to complete
>     before starting any more.  If somehow an install is mixed with a
>     reboot test, we'll wait around before starting more jobs
>     - probably a few more here, but I don't have them on the top of my
>     head.
>
>   
I haven't looked through the code at all yet, but the design seems more
in line with my thoughts of parallel execution.  Just a couple really
quick thoughts on this:

There is an effort being made to merge kvm-autotest with autotest.  I
think part of that includes a change to remove kvm_log.  I don't know if
the upstream logging system will support multi-threading, but I wouldn't
spend any time working on kvm_log.

I haven't dug through all the execution code yet.  I've mostly played
around in the kvm_install area and stepsfile stuff.  Couldn't we do
something like this:

define a param called num_threads
have base names for things like vms, images, nics, etc like vm, image, nic
each thread picks up a test to run and uses the base names + thead num
We could then use the thread number all the offsets as well.

Mike