Re: [RFC] KVM-Autotest: basic parallel test execution

[Ryan Harper <ryanh@us.ibm.com>]

* Mike Burns <mburns@redhat.com> [2009-05-20 16:13]:
> Ryan Harper wrote:
> > * Michael Goldish <mgoldish@redhat.com> [2009-05-17 09:50]:
> >   
> >> Hi all,
> >>
> >> We've recently implemented a very simple form of parallel test
> >> execution into KVM-Autotest and we'd like some feedback on it. This
> >> suggestion allows the user to manually assign tests to hosts/queues.
> >> It also takes care of assigning different MAC address ranges to
> >> hosts/queues. By 'queues' I mean parallel execution pipelines. Each
> >> host has one or more queues. The number of queues is defined by the
> >> user, and should reflect the capabilities of the host.
> >>
> >> This implementation involves only minor modifications to the code
> >> itself; most of the work is done in a new config file kvm_hosts.cfg,
> >> which has the exact same format as kvm_tests.cfg. The new file
> >> provides the framework with information about hosts/queues. The new
> >> file is parsed after kvm_tests.cfg. The test sets (such as
> >> 'nightly' and 'weekly'), previously defined at the end of
> >> kvm_tests.cfg, should now be defined last, after kvm_hosts.cfg.
> >> Test sets no longer select only the tests to execute, but also
> >> where each test should be executed (i.e. on what host/queue).
> >>
> >> The final result of parsing the config files is a list of tests, each
> >> with its own 'hostname' and 'queue' parameters. Each host executes
> >> only the tests whose 'hostname' parameter matches the current host,
> >> and puts tests with different 'queue' values in parallel pipelines
> >> of execution.
> >>
> >> Ideally, the Autotest server should take care of assigning tests to
> >> hosts automatically, but there are still a few technical difficulties
> >> to be resolved before we can implement that. We're considering the
> >> current suggestion as a temporary solution until a better one is
> >> found.
> >>
> >> Basically, the advantages are:
> >> - allows the user full control over what tests run, and where/how they run
> >> - takes care of assigning MAC address ranges to different hosts/queues (required for TAP networking)
> >> - can be used from the server or with the client, which makes it relevant also for users who don't have an Autotest server installed
> >> - involves only minor code changes (except for the config files)
> >> - is pretty much the simplest possible solution (and simple is good)
> >>
> >> Drawbacks:
> >> - requires some initial work to be done by the user -- the user has to define exactly where each test should run
> >> - test sets need to be modified when tests or hosts are added/removed, to include/exclude them
> >>     
> >
> > I took a slightly different approach.  The kvm_tests.cfg file already
> > provides a dependency relationship between different tests.  I modified
> > the main loop in the control file to walk the entire list of jobs and
> > pull out any jobs that don't have any dependencies (ie, install tests).
> > And then run N jobs in parallel from that list until it is exhausted;
> > then store the results.  Then loop the over the remaining list of jobs
> > again finding the jobs that can be run.
> >
> > On a larger multi core system, one might set the number of parallel jobs
> > equal to the number of cores.
> >
> > I think this works well with using autoserv to farm out different
> > kvm_tests.cfg to different machines.
> >
> > Attaching my stale patch just for comment.  Needs to be updated since I
> > sat on this for a while.  There were a number of issues:
> >
> > - kvm_log is a shared resource, fixed it up so parallel jobs can both
> >   call it
> > - vnc, redir and other network resources are shared, so, in
> > kvm_tests.cfg file each job needs a parallel offset.
> > - in kvm_tests.cfg file need to define additional vm and nic objects,
> >    one for each parallel threads.
> >
> > Advantages:
> >     - works a lot like the single threaded model does, and if threads=1
> >     then it runs the same path
> >     - config files don't change significantly, just some additional
> >     VM objects at the top and some offset values
> >     - transparent to an autoserv setup, autoserv would just need to
> >     specify the kvm_tests.cfg file to each host.
> >
> > Disadvantages:
> >     - the main loop waits for each group of parallel jobs to complete
> >     before starting any more.  If somehow an install is mixed with a
> >     reboot test, we'll wait around before starting more jobs
> >     - probably a few more here, but I don't have them on the top of my
> >     head.
> >
> >   
> I haven't looked through the code at all yet, but the design seems more
> in line with my thoughts of parallel execution.  Just a couple really
> quick thoughts on this:
> 
> There is an effort being made to merge kvm-autotest with autotest.  I
> think part of that includes a change to remove kvm_log.  I don't know if
> the upstream logging system will support multi-threading, but I wouldn't
> spend any time working on kvm_log.

Indeed, I spent as little time on it as it took to get it working with 2
parallel installs.  I don't have any plans for it other than if it is
present to not be in the way of parallel execution.

> 
> I haven't dug through all the execution code yet.  I've mostly played
> around in the kvm_install area and stepsfile stuff.  Couldn't we do
> something like this:
> 
> define a param called num_threads
> have base names for things like vms, images, nics, etc like vm, image, nic
> each thread picks up a test to run and uses the base names + thead num
> We could then use the thread number all the offsets as well.

Yep.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com