[RFC] KVM-Autotest: basic parallel test execution

[RFC] KVM-Autotest: basic parallel test execution

[Michael Goldish]

Hi all,

We've recently implemented a very simple form of parallel test execution into KVM-Autotest and we'd like some feedback on it. This suggestion allows the user to manually assign tests to hosts/queues. It also takes care of assigning different MAC address ranges to hosts/queues. By 'queues' I mean parallel execution pipelines. Each host has one or more queues. The number of queues is defined by the user, and should reflect the capabilities of the host.

This implementation involves only minor modifications to the code itself; most of the work is done in a new config file kvm_hosts.cfg, which has the exact same format as kvm_tests.cfg. The new file provides the framework with information about hosts/queues. The new file is parsed after kvm_tests.cfg. The test sets (such as 'nightly' and 'weekly'), previously defined at the end of kvm_tests.cfg, should now be defined last, after kvm_hosts.cfg. Test sets no longer select only the tests to execute, but also where each test should be executed (i.e. on what host/queue).

The final result of parsing the config files is a list of tests, each with its own 'hostname' and 'queue' parameters. Each host executes only the tests whose 'hostname' parameter matches the current host, and puts tests with different 'queue' values in parallel pipelines of execution.

Ideally, the Autotest server should take care of assigning tests to hosts automatically, but there are still a few technical difficulties to be resolved before we can implement that. We're considering the current suggestion as a temporary solution until a better one is found.

Basically, the advantages are:
- allows the user full control over what tests run, and where/how they run
- takes care of assigning MAC address ranges to different hosts/queues (required for TAP networking)
- can be used from the server or with the client, which makes it relevant also for users who don't have an Autotest server installed
- involves only minor code changes (except for the config files)
- is pretty much the simplest possible solution (and simple is good)

Drawbacks:
- requires some initial work to be done by the user -- the user has to define exactly where each test should run
- test sets need to be modified when tests or hosts are added/removed, to include/exclude them

We'd like to get some feedback on this -- is it usable, comfortable, do you have any suggestions, etc. We'll also be happy to answer questions.
There are no actual patches attached to this message because the implementation is trivial and we want to focus on the user's point of view.


The following is a sample kvm_hosts.cfg file.
Note that kvm_hosts.cfg can be structured in many different ways that will achieve the same results.


# Start by parsing the regular config file
include kvm_tests.cfg

# Now define MAC address ranges for VMs
# (this is environment specific information)

base_mac_r0 = 00:00:00:00:00:00
base_ip_r0 = 10.20.0.0
range_size_r0 = 256

base_mac_r1 = 00:00:00:00:01:00
base_ip_r1 = 10.20.1.0
range_size_r1 = 256

base_mac_r2 = 00:00:00:00:02:00
base_ip_r2 = 10.20.2.0
range_size_r2 = 256

# (define more MAC address ranges here...)

# Define 4 parallel queues (or more)
variants:
    - @queue1:
        queue = 1
        # Each queue uses a different environment file
        env = env
    - @queue2:
        queue = 2
        env = env2
    - @queue3:
        queue = 3
        env = env3
    - @queue4:
        queue = 4
        env = env4

# Define the hosts to be used
# (this is where you provide host-specific information)
variants:
    - @hostname1:
        # 'hostname1' should be a real hostname
        hostname = hostname1
        # Let hostname1 run 2 parallel test queues
        only queue1 queue2
        queue1:
            # Assign range r0 to queue1 on hostname1
            mac_ranges = r0
        queue2:
            # Assign range r1 to queue2 on hostname1
            mac_ranges = r1
    - @hostname2:
        # Let hostname2 run a single test queue
        hostname = hostname2
        only queue1
        # Assign range r2 to hostname2
        mac_ranges = r2
    # (define more hosts here...)


The following is a sample test set definition:


variants:
    # This is the 'nightly' test set
    - @nightly:
        # Select the hosts to run tests on
        only hostname1 hostname2
        # Select the tests to run
        only qcow2
        only ide
        only RHEL|Windows
        # Assign tests to individual hosts
        hostname1:
            # hostname1 will run Windows tests
            only Windows
            # Select the tests to run on each queue
            # (the selections should be mutually-exclusive)
            queue1:
                only Win2000 WinXP Win2003
            queue2:
                only WinVista Win2008
        hostname2:
            # hostname2 has a single queue,
            # and it will run RHEL tests
            only RHEL

    # This is the 'weekly' test set
    - @weekly:
        ...

Finally, one should select the desired test set (e.g. 'only nightly') somewhere -- at the end of a config file or in the control file (using config.parse_string()).


Thanks,
Michael

Re: [RFC] KVM-Autotest: basic parallel test execution

[Avi Kivity]

Michael Goldish wrote:
> Drawbacks:
> - requires some initial work to be done by the user -- the user has to define exactly where each test should run
>   

For me, this is a major drawback.  I'd really like a fire-and-forget 
solution.  If I have to spend my own time getting this to work, vs. 
waiting longer for the tests to run on their own, I'll just be lazy.

> - test sets need to be modified when tests or hosts are added/removed, to include/exclude them
>   

This is also annoying -- and likely to stop me from updating.

> We'd like to get some feedback on this -- is it usable, comfortable, do you have any suggestions, etc. We'll also be happy to answer questions.
> There are no actual patches attached to this message because the implementation is trivial and we want to focus on the user's point of view.
>
>   


I'd really like this to be automated, just specify a set of machines and 
have the jobs distributed.  Furthermore, it is very important to utilize 
the existing hosts better.  A 4-core 4GB server can easily run a 2x smp 
1GB guest and 2 other uniprocessor 1GB guests.  It's wasteful to add 
more servers when the existing servers are underutilized.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [RFC] KVM-Autotest: basic parallel test execution

[Mike Burns]

Avi Kivity wrote:
> Michael Goldish wrote:
>> Drawbacks:
>> - requires some initial work to be done by the user -- the user has
>> to define exactly where each test should run
>>   
>
> For me, this is a major drawback.  I'd really like a fire-and-forget
> solution.  If I have to spend my own time getting this to work, vs.
> waiting longer for the tests to run on their own, I'll just be lazy.
It seems like this is adding another layer of user interaction without
really getting that much additional functionality.  An administrator of
kvm-autotest is going to know enough to be able to create control and
kvm_tests.cfg files that can mimic this already and then just create 2
jobs in the server to run on different hosts. 
>
>> - test sets need to be modified when tests or hosts are
>> added/removed, to include/exclude them
>>   
>
> This is also annoying -- and likely to stop me from updating.
Host definitions already happen on the server and it seems like it would
be confusing to have to make sure that the host you choose when setting
up a test is setup correctly in your config files. 
>
>> We'd like to get some feedback on this -- is it usable, comfortable,
>> do you have any suggestions, etc. We'll also be happy to answer
>> questions.
>> There are no actual patches attached to this message because the
>> implementation is trivial and we want to focus on the user's point of
>> view.
>>
>>   
>
>
> I'd really like this to be automated, just specify a set of machines
> and have the jobs distributed.  Furthermore, it is very important to
> utilize the existing hosts better.  A 4-core 4GB server can easily run
> a 2x smp 1GB guest and 2 other uniprocessor 1GB guests.  It's wasteful
> to add more servers when the existing servers are underutilized.
>
Overall, the way I envisioned parallel test execution was having
multiple tests running on a single machine.  It seems that the server
already provides most (if not all) the functionality needed to spread
tests across multiple machines.  I really think this is putting too much
on the user for very small gains in functionality. 

Re: [RFC] KVM-Autotest: basic parallel test execution

[Ryan Harper]

[-- Attachment #1: Type: text/plain, Size: 4289 bytes --]

* Michael Goldish <mgoldish@redhat.com> [2009-05-17 09:50]:
> Hi all,
> 
> We've recently implemented a very simple form of parallel test
> execution into KVM-Autotest and we'd like some feedback on it. This
> suggestion allows the user to manually assign tests to hosts/queues.
> It also takes care of assigning different MAC address ranges to
> hosts/queues. By 'queues' I mean parallel execution pipelines. Each
> host has one or more queues. The number of queues is defined by the
> user, and should reflect the capabilities of the host.
> 
> This implementation involves only minor modifications to the code
> itself; most of the work is done in a new config file kvm_hosts.cfg,
> which has the exact same format as kvm_tests.cfg. The new file
> provides the framework with information about hosts/queues. The new
> file is parsed after kvm_tests.cfg. The test sets (such as
> 'nightly' and 'weekly'), previously defined at the end of
> kvm_tests.cfg, should now be defined last, after kvm_hosts.cfg.
> Test sets no longer select only the tests to execute, but also
> where each test should be executed (i.e. on what host/queue).
> 
> The final result of parsing the config files is a list of tests, each
> with its own 'hostname' and 'queue' parameters. Each host executes
> only the tests whose 'hostname' parameter matches the current host,
> and puts tests with different 'queue' values in parallel pipelines
> of execution.
> 
> Ideally, the Autotest server should take care of assigning tests to
> hosts automatically, but there are still a few technical difficulties
> to be resolved before we can implement that. We're considering the
> current suggestion as a temporary solution until a better one is
> found.
> 
> Basically, the advantages are:
> - allows the user full control over what tests run, and where/how they run
> - takes care of assigning MAC address ranges to different hosts/queues (required for TAP networking)
> - can be used from the server or with the client, which makes it relevant also for users who don't have an Autotest server installed
> - involves only minor code changes (except for the config files)
> - is pretty much the simplest possible solution (and simple is good)
> 
> Drawbacks:
> - requires some initial work to be done by the user -- the user has to define exactly where each test should run
> - test sets need to be modified when tests or hosts are added/removed, to include/exclude them

I took a slightly different approach.  The kvm_tests.cfg file already
provides a dependency relationship between different tests.  I modified
the main loop in the control file to walk the entire list of jobs and
pull out any jobs that don't have any dependencies (ie, install tests).
And then run N jobs in parallel from that list until it is exhausted;
then store the results.  Then loop the over the remaining list of jobs
again finding the jobs that can be run.

On a larger multi core system, one might set the number of parallel jobs
equal to the number of cores.

I think this works well with using autoserv to farm out different
kvm_tests.cfg to different machines.

Attaching my stale patch just for comment.  Needs to be updated since I
sat on this for a while.  There were a number of issues:

- kvm_log is a shared resource, fixed it up so parallel jobs can both
  call it
- vnc, redir and other network resources are shared, so, in
kvm_tests.cfg file each job needs a parallel offset.
- in kvm_tests.cfg file need to define additional vm and nic objects,
   one for each parallel threads.

Advantages:
    - works a lot like the single threaded model does, and if threads=1
    then it runs the same path
    - config files don't change significantly, just some additional
    VM objects at the top and some offset values
    - transparent to an autoserv setup, autoserv would just need to
    specify the kvm_tests.cfg file to each host.

Disadvantages:
    - the main loop waits for each group of parallel jobs to complete
    before starting any more.  If somehow an install is mixed with a
    reboot test, we'll wait around before starting more jobs
    - probably a few more here, but I don't have them on the top of my
    head.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

[-- Attachment #2: kvm_runtest_2_parallel_jobs.patch --]
[-- Type: text/plain, Size: 11468 bytes --]

diff --git a/client/bin/job.py b/client/bin/job.py
index 6ed5b36..d772488 100755
--- a/client/bin/job.py
+++ b/client/bin/job.py
@@ -751,21 +751,27 @@ class base_job(object):
         """Run tasks in parallel"""
 
         pids = []
+        pid_to_task = {}
+        return_vals = {}
         old_log_filename = self.log_filename
         for i, task in enumerate(tasklist):
             assert isinstance(task, (tuple, list))
             self.log_filename = old_log_filename + (".%d" % i)
             task_func = lambda: task[0](*task[1:])
-            pids.append(parallel.fork_start(self.resultdir, task_func))
+            pid = parallel.fork_start(self.resultdir, task_func)
+            pids.append(pid)
+            pid_to_task[pid] = i
 
         old_log_path = os.path.join(self.resultdir, old_log_filename)
         old_log = open(old_log_path, "a")
         exceptions = []
         for i, pid in enumerate(pids):
             # wait for the task to finish
+            task = pid_to_task[pid]
             try:
-                parallel.fork_waitfor(self.resultdir, pid)
+                return_vals[task] = parallel.fork_waitfor(self.resultdir, pid)
             except Exception, e:
+                return_vals[task] = 1
                 exceptions.append(e)
             # copy the logs from the subtask into the main log
             new_log_path = old_log_path + (".%d" % i)
@@ -784,6 +790,9 @@ class base_job(object):
             msg = "%d task(s) failed in job.parallel" % len(exceptions)
             raise error.JobError(msg)
 
+        # send back array True|False on success or failure
+        return map(lambda x: x == 0, return_vals.values())
+
 
     def quit(self):
         # XXX: should have a better name.
diff --git a/client/bin/parallel.py b/client/bin/parallel.py
index e8749db..e381c89 100644
--- a/client/bin/parallel.py
+++ b/client/bin/parallel.py
@@ -45,3 +45,4 @@ def fork_waitfor(tmp, pid):
 
     if (status != 0):
         raise error.TestError("test failed rc=%d" % (status))
+    return status
diff --git a/client/tests/kvm_runtest_2/control b/client/tests/kvm_runtest_2/control
index cb27821..1251e94 100644
--- a/client/tests/kvm_runtest_2/control
+++ b/client/tests/kvm_runtest_2/control
@@ -1,4 +1,4 @@
-import sys, os
+import sys, os, copy
 
 # enable modules import from current directory (tests/kvm_runtest_2)
 pwd = os.path.join(os.environ['AUTODIR'],'tests/kvm_runtest_2')
@@ -82,24 +82,80 @@ list = kvm_config.config(filename).get_list()
 #  -------------
 #  Run the tests
 #  -------------
+threads = 1
 status_dict = {}
-
-for dict in list:
-    if dict.get("skip") == "yes":
-        continue
-    dependencies_satisfied = True
-    for dep in dict.get("depend"):
-        for test_name in status_dict.keys():
-            if not dep in test_name:
-                continue
-            if not status_dict[test_name]:
+current_list = copy.deepcopy(list)
+
+while len(current_list) > 0:
+    parallel_jobs = []
+    for dict in current_list:
+        name = dict['name']
+        if dict.get("skip") == "yes":
+            continue
+        dependencies_satisfied = True
+        # it's ok to ignore deps that we won't run, but not ok to ignore 
+        # deps that have failed
+        for dep in dict.get("depend"):
+            # if we're waiting for the job to be run, deps aren't met
+            if dep in map(lambda x: x['name'], parallel_jobs):
                 dependencies_satisfied = False
                 break
-    if dependencies_satisfied:
-        current_status = job.run_test("kvm_runtest_2", params=dict, tag=dict.get("shortname"))
-    else:
-        current_status = False
-    status_dict[dict.get("name")] = current_status
+            for test_name in status_dict.keys():
+                if not dep in test_name:
+                    continue
+                if not status_dict[test_name]:
+                    dependencies_satisfied = False
+                    break
+
+        # add the job to parallel list if we can run it *now*
+        if dependencies_satisfied == True:
+            parallel_jobs.append(dict)
+            current_list = filter(lambda x: x['name'] != name, current_list)
+
+
+    print '**************WARK************* %d jobs to run in parallel' %(len(parallel_jobs))
+    # now, we can run the current parallel jobs in sets of N tests where
+    # N is the number of parallel jobs allowed.
+    for g in range(0,len(parallel_jobs), threads):
+        group = parallel_jobs[g:g+threads]
+        names = map(lambda x: x['name'], group)
+
+        # HACK - my python fu is not great enough to get around exec/eval to
+        # dynamically define functions which are needed for job.parallel
+        j = []
+        k = []
+        for i,dict in enumerate(group):
+                # reset vm and main_vm list into each dict
+                dict['vms'] = "vm%s" % i
+                dict['main_vm'] = "vm%s" % i
+                # FIXME: should be # of threads * index
+                dict['parallel_offset'] = 2*i
+                print dict
+                # create unique dict name and make a copy of dict
+                pdict_name = "dict_vm%s"%(i)
+                print pdict_name
+                exec "%s = copy.deepcopy(dict)" %(pdict_name)
+                tag = dict.get("shortname")
+                sdtag = 'parallel_%s' % i
+                defstr = 'def j%s():\n  job.run_test("kvm_runtest_2", params=%s, tag="%s", subdir_tag="%s")' % (i,pdict_name, tag,sdtag)
+                # define dummy function for this job                   
+                exec defstr
+                eval("j.append(j%s)" % i)
+                k.append(dict.get("shortname"))
+
+        print "***************** WARK ************* parallel: %s"%(j)
+        print "***************** WARK ************* parallel: %s"%(map(lambda x: [x], k))
+
+        # need to wrap each function in the array with [], then use * to unpack the list
+        # as variable arg list to job.parallel
+        current_status = job.parallel(*map(lambda x: [x], j))
+        #current_status = job.parallel([j[0]], [j[1]])
+        print "status: %s"%(current_status)
+
+        # update status of each job from parallel run
+        for i,dict in enumerate(group):
+            status_dict[dict.get("name")] = current_status[i]
+    
 
 # create the html report in result dir
 reporter = os.path.join(pwd, 'make_html_report.py')
diff --git a/client/tests/kvm_runtest_2/kvm_log.py b/client/tests/kvm_runtest_2/kvm_log.py
index ff1dfe9..06dc47d 100644
--- a/client/tests/kvm_runtest_2/kvm_log.py
+++ b/client/tests/kvm_runtest_2/kvm_log.py
@@ -5,10 +5,9 @@ version = "20081205"
 
 import time
 import sys
+import os
 
-prefix_stack = [""]
-log_level = 4
-
+pids = {}
 
 def name_of_caller():
     return sys._getframe(2).f_code.co_name
@@ -28,24 +27,50 @@ def format_stack(min_depth, max_depth, stop_func):
     list.reverse()
     return "|".join(list)
 
-def set_prefix(prefix):
-    global prefix_stack
-    prefix_stack.append(" %s" % prefix)
+def get_prefix(pid):
+    global pids
+    if (pids.has_key(pid) and not pids[pid].has_key('prefix')) or not pids.has_key(pid):
+        pids[pid] = {}
+        default_prefix = " [%s]" % pid
+        pids[pid]['prefix'] = [default_prefix]
+    return pids[pid]['prefix']
+
+def set_prefix(pid, prefix):
+    global pids
+    p = get_prefix(pid)
+    p.append(" %s" % prefix)
+    pids[pid]['prefix'] = p
+
+def restore_prefix(pid):
+    global pids
+    if pid in pids.keys():
+        pids[pid]['prefix'].pop()
 
-def restore_prefix():
-    global prefix_stack
-    prefix_stack.pop()
+def get_level(pid):
+    global pids
+    if (pids.has_key(pid) and not pids[pid].has_key('loglevel')) or not pids.has_key(pid):
+        pids[pid] = {}
+        pids[pid]['loglevel'] = 4
 
-def set_level(level):
-    global log_level
-    log_level = level
+    return pids[pid]['loglevel']
+
+def set_level(pid, level):
+    global pids
+    l = get_level(pid)
+    l = level
+    pids[pid]['loglevel'] = l
 
 def log(level, message, prefix=True):
+    global pids
+    mypid = os.getpid()
+    prefix = get_prefix(mypid)
+    loglevel = get_level(mypid)
+
     if message.endswith("\n"):
         message = message[:-1]
-    if level <= log_level:
+    if level <= loglevel:
         if prefix:
-            print "%s%s: %s" % (time.strftime("%Y%m%d-%H%M%S"), prefix_stack[-1], message)
+            print "%s%s: %s" % (time.strftime("%Y%m%d-%H%M%S"), prefix[-1], message)
         else:
             print message
 
diff --git a/client/tests/kvm_runtest_2/kvm_runtest_2.py b/client/tests/kvm_runtest_2/kvm_runtest_2.py
index c53877f..c6504ba 100644
--- a/client/tests/kvm_runtest_2/kvm_runtest_2.py
+++ b/client/tests/kvm_runtest_2/kvm_runtest_2.py
@@ -53,7 +53,10 @@ class kvm_runtest_2(test.test):
         resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
 
         # Set the logging prefix
-        kvm_log.set_prefix(params.get("shortname"))
+        tag = params.get("shortname")
+        mypid = os.getpid()
+        print '[%s] ************************** WARK ******************* i am tag=%s\n' %(mypid, tag)
+        kvm_log.set_prefix(mypid, tag)
 
         # Report the parameters we've received and write them as keyvals
         kvm_log.debug("Test parameters:")
@@ -65,6 +68,7 @@ class kvm_runtest_2(test.test):
 
         # Open the environment file
         env_filename = os.path.join(self.bindir, "env")
+        kvm_log.debug("using env file: %s" %(env_filename))
         env = shelve.open(env_filename, writeback=True)
         kvm_log.debug("Contents of environment: %s" % str(env))
 
diff --git a/client/tests/kvm_runtest_2/kvm_vm.py b/client/tests/kvm_runtest_2/kvm_vm.py
index fa68ce4..90ca21a 100644
--- a/client/tests/kvm_runtest_2/kvm_vm.py
+++ b/client/tests/kvm_runtest_2/kvm_vm.py
@@ -271,6 +271,9 @@ class VM:
                     kvm_log.error("Actual MD5 sum differs from expected one")
                     return False
 
+        # store parallel offset
+        self.port_offset = int(params.get("parallel_offset", 0))
+
         # Find available monitor filename
         while True:
             # The monitor filename should be unique
@@ -281,7 +284,7 @@ class VM:
 
         # Handle port redirections
         redir_names = kvm_utils.get_sub_dict_names(params, "redirs")
-        host_ports = kvm_utils.find_free_ports(5000, 6000, len(redir_names))
+        host_ports = kvm_utils.find_free_ports(self.port_offset+5000, 6000, len(redir_names))
         self.redirs = {}
         for i in range(len(redir_names)):
             redir_params = kvm_utils.get_sub_dict(params, redir_names[i])
@@ -290,7 +293,7 @@ class VM:
 
         # Find available VNC port, if needed
         if params.get("display") == "vnc":
-            self.vnc_port = kvm_utils.find_free_port(5900, 6000)
+            self.vnc_port = kvm_utils.find_free_port(self.port_offset+5900, 6000)
 
         # Make qemu command
         qemu_command = self.make_qemu_command()
@@ -298,7 +301,7 @@ class VM:
         # Is this VM supposed to accept incoming migrations?
         if for_migration:
             # Find available migration port
-            self.migration_port = kvm_utils.find_free_port(5200, 6000)
+            self.migration_port = kvm_utils.find_free_port(self.port_offset+5200, 6000)
             # Add -incoming option to the qemu command
             qemu_command += " -incoming tcp:0:%d" % self.migration_port
 

Re: [RFC] KVM-Autotest: basic parallel test execution

[Mike Burns]

Ryan Harper wrote:
> * Michael Goldish <mgoldish@redhat.com> [2009-05-17 09:50]:
>   
>> Hi all,
>>
>> We've recently implemented a very simple form of parallel test
>> execution into KVM-Autotest and we'd like some feedback on it. This
>> suggestion allows the user to manually assign tests to hosts/queues.
>> It also takes care of assigning different MAC address ranges to
>> hosts/queues. By 'queues' I mean parallel execution pipelines. Each
>> host has one or more queues. The number of queues is defined by the
>> user, and should reflect the capabilities of the host.
>>
>> This implementation involves only minor modifications to the code
>> itself; most of the work is done in a new config file kvm_hosts.cfg,
>> which has the exact same format as kvm_tests.cfg. The new file
>> provides the framework with information about hosts/queues. The new
>> file is parsed after kvm_tests.cfg. The test sets (such as
>> 'nightly' and 'weekly'), previously defined at the end of
>> kvm_tests.cfg, should now be defined last, after kvm_hosts.cfg.
>> Test sets no longer select only the tests to execute, but also
>> where each test should be executed (i.e. on what host/queue).
>>
>> The final result of parsing the config files is a list of tests, each
>> with its own 'hostname' and 'queue' parameters. Each host executes
>> only the tests whose 'hostname' parameter matches the current host,
>> and puts tests with different 'queue' values in parallel pipelines
>> of execution.
>>
>> Ideally, the Autotest server should take care of assigning tests to
>> hosts automatically, but there are still a few technical difficulties
>> to be resolved before we can implement that. We're considering the
>> current suggestion as a temporary solution until a better one is
>> found.
>>
>> Basically, the advantages are:
>> - allows the user full control over what tests run, and where/how they run
>> - takes care of assigning MAC address ranges to different hosts/queues (required for TAP networking)
>> - can be used from the server or with the client, which makes it relevant also for users who don't have an Autotest server installed
>> - involves only minor code changes (except for the config files)
>> - is pretty much the simplest possible solution (and simple is good)
>>
>> Drawbacks:
>> - requires some initial work to be done by the user -- the user has to define exactly where each test should run
>> - test sets need to be modified when tests or hosts are added/removed, to include/exclude them
>>     
>
> I took a slightly different approach.  The kvm_tests.cfg file already
> provides a dependency relationship between different tests.  I modified
> the main loop in the control file to walk the entire list of jobs and
> pull out any jobs that don't have any dependencies (ie, install tests).
> And then run N jobs in parallel from that list until it is exhausted;
> then store the results.  Then loop the over the remaining list of jobs
> again finding the jobs that can be run.
>
> On a larger multi core system, one might set the number of parallel jobs
> equal to the number of cores.
>
> I think this works well with using autoserv to farm out different
> kvm_tests.cfg to different machines.
>
> Attaching my stale patch just for comment.  Needs to be updated since I
> sat on this for a while.  There were a number of issues:
>
> - kvm_log is a shared resource, fixed it up so parallel jobs can both
>   call it
> - vnc, redir and other network resources are shared, so, in
> kvm_tests.cfg file each job needs a parallel offset.
> - in kvm_tests.cfg file need to define additional vm and nic objects,
>    one for each parallel threads.
>
> Advantages:
>     - works a lot like the single threaded model does, and if threads=1
>     then it runs the same path
>     - config files don't change significantly, just some additional
>     VM objects at the top and some offset values
>     - transparent to an autoserv setup, autoserv would just need to
>     specify the kvm_tests.cfg file to each host.
>
> Disadvantages:
>     - the main loop waits for each group of parallel jobs to complete
>     before starting any more.  If somehow an install is mixed with a
>     reboot test, we'll wait around before starting more jobs
>     - probably a few more here, but I don't have them on the top of my
>     head.
>
>   
I haven't looked through the code at all yet, but the design seems more
in line with my thoughts of parallel execution.  Just a couple really
quick thoughts on this:

There is an effort being made to merge kvm-autotest with autotest.  I
think part of that includes a change to remove kvm_log.  I don't know if
the upstream logging system will support multi-threading, but I wouldn't
spend any time working on kvm_log.

I haven't dug through all the execution code yet.  I've mostly played
around in the kvm_install area and stepsfile stuff.  Couldn't we do
something like this:

define a param called num_threads
have base names for things like vms, images, nics, etc like vm, image, nic
each thread picks up a test to run and uses the base names + thead num
We could then use the thread number all the offsets as well.

Mike

Re: [RFC] KVM-Autotest: basic parallel test execution

[Michael Goldish]

----- "Mike Burns" <mburns@redhat.com> wrote:

> Avi Kivity wrote:
> > Michael Goldish wrote:
> >> Drawbacks:
> >> - requires some initial work to be done by the user -- the user
> has
> >> to define exactly where each test should run
> >>   
> >
> > For me, this is a major drawback.  I'd really like a
> fire-and-forget
> > solution.  If I have to spend my own time getting this to work, vs.
> > waiting longer for the tests to run on their own, I'll just be
> lazy.
> It seems like this is adding another layer of user interaction
> without
> really getting that much additional functionality.  An administrator
> of
> kvm-autotest is going to know enough to be able to create control and
> kvm_tests.cfg files that can mimic this already and then just create
> 2
> jobs in the server to run on different hosts. 
> >
> >> - test sets need to be modified when tests or hosts are
> >> added/removed, to include/exclude them
> >>   
> >
> > This is also annoying -- and likely to stop me from updating.
> Host definitions already happen on the server and it seems like it
> would
> be confusing to have to make sure that the host you choose when
> setting
> up a test is setup correctly in your config files. 
> >
> >> We'd like to get some feedback on this -- is it usable,
> comfortable,
> >> do you have any suggestions, etc. We'll also be happy to answer
> >> questions.
> >> There are no actual patches attached to this message because the
> >> implementation is trivial and we want to focus on the user's point
> of
> >> view.
> >>
> >>   
> >
> >
> > I'd really like this to be automated, just specify a set of
> machines
> > and have the jobs distributed.  Furthermore, it is very important
> to
> > utilize the existing hosts better.  A 4-core 4GB server can easily
> run
> > a 2x smp 1GB guest and 2 other uniprocessor 1GB guests.  It's
> wasteful
> > to add more servers when the existing servers are underutilized.
> >
> Overall, the way I envisioned parallel test execution was having
> multiple tests running on a single machine. It seems that the server
> already provides most (if not all) the functionality needed to spread
> tests across multiple machines.  I really think this is putting too
> much on the user for very small gains in functionality.

This suggestion does allow running multiple tests in parallel on a
single machine.
host1:
  only queue1 queue2
means there will be 2 queues running in parallel on host1, and
queue1:
  only Fedora
means queue1 will run Fedora tests (in parallel to whatever queue2 will
be running).

Other than that, the suggestion doesn't add much new functionality.
However, since it's backwards compatible, in the sense that regular
test sets (like fc8_quick) can still be defined the usual way alongside
the new test sets, the cost may not be that high. The user is required
to define hosts only if he/she intends to specify test sets in the new
format.
Backwards compatibility is achieved by defining a host variant 'anyhost'
with no hostname. Then we just need to add 'only anyhost' to our
fc8_quick, and it becomes compatible with the new format. We can also use
just client parallelization, by using multiple queues with anyhost, e.g.

fc8_quick:
  only anyhost
  only queue1 queue2 queue3
  queue1:
    only something
  queue2:
    only something_else

(this will run fc8_quick on all hosts, with multiple queues on each host.)

In other words, there may not be much new functionality, but it's nearly
'free'. Also, it is somewhat appealing that implementing this involves as
few as 1-2 extra lines of code (in addition to the kvm_hosts.cfg file),
because it relies on existing capabilities of the config file parser.
It's not that I'm lazy -- it's just that it leaves very little room for
mistakes to be made.


On the other hand, Lucas says the technical problems we've encountered
while experimenting with server control files are most likely bugs that
have been resolved recently, so maybe we should just go ahead and implement
fully automatic test dispatching from the server. That was the original
plan anyway.

I suggested the simpler option just to see if people happen to find it
appealing. So far this doesn't seem to be the case, so we might as well
skip it. Let me know if this message changes your mind though (from your
message I got the impression that you weren't aware that this suggestion
enables running multiple tests on a single machine).

Thanks,
Michael

Re: [RFC] KVM-Autotest: basic parallel test execution

[Michael Goldish]

----- "Ryan Harper" <ryanh@us.ibm.com> wrote:

> * Michael Goldish <mgoldish@redhat.com> [2009-05-17 09:50]:
> > Hi all,
> > 
> > We've recently implemented a very simple form of parallel test
> > execution into KVM-Autotest and we'd like some feedback on it. This
> > suggestion allows the user to manually assign tests to
> hosts/queues.
> > It also takes care of assigning different MAC address ranges to
> > hosts/queues. By 'queues' I mean parallel execution pipelines. Each
> > host has one or more queues. The number of queues is defined by the
> > user, and should reflect the capabilities of the host.
> > 
> > This implementation involves only minor modifications to the code
> > itself; most of the work is done in a new config file
> kvm_hosts.cfg,
> > which has the exact same format as kvm_tests.cfg. The new file
> > provides the framework with information about hosts/queues. The new
> > file is parsed after kvm_tests.cfg. The test sets (such as
> > 'nightly' and 'weekly'), previously defined at the end of
> > kvm_tests.cfg, should now be defined last, after kvm_hosts.cfg.
> > Test sets no longer select only the tests to execute, but also
> > where each test should be executed (i.e. on what host/queue).
> > 
> > The final result of parsing the config files is a list of tests,
> each
> > with its own 'hostname' and 'queue' parameters. Each host executes
> > only the tests whose 'hostname' parameter matches the current host,
> > and puts tests with different 'queue' values in parallel pipelines
> > of execution.
> > 
> > Ideally, the Autotest server should take care of assigning tests to
> > hosts automatically, but there are still a few technical
> difficulties
> > to be resolved before we can implement that. We're considering the
> > current suggestion as a temporary solution until a better one is
> > found.
> > 
> > Basically, the advantages are:
> > - allows the user full control over what tests run, and where/how
> they run
> > - takes care of assigning MAC address ranges to different
> hosts/queues (required for TAP networking)
> > - can be used from the server or with the client, which makes it
> relevant also for users who don't have an Autotest server installed
> > - involves only minor code changes (except for the config files)
> > - is pretty much the simplest possible solution (and simple is
> good)
> > 
> > Drawbacks:
> > - requires some initial work to be done by the user -- the user has
> to define exactly where each test should run
> > - test sets need to be modified when tests or hosts are
> added/removed, to include/exclude them
> 
> I took a slightly different approach.  The kvm_tests.cfg file already
> provides a dependency relationship between different tests.  I
> modified
> the main loop in the control file to walk the entire list of jobs and
> pull out any jobs that don't have any dependencies (ie, install
> tests).
> And then run N jobs in parallel from that list until it is exhausted;
> then store the results.  Then loop the over the remaining list of
> jobs again finding the jobs that can be run.

I like this approach. Looks like it's somewhere between the simple
static way and the fully automatic server way.

> On a larger multi core system, one might set the number of parallel
> jobs equal to the number of cores.

It makes sense to define 'threads' in the control file, because it's
definitely not a test param, and the control file is easiest to change from
the server. However, I wonder how different values of 'threads' can be
defined for different hosts (with different capabilities) from the server.

> I think this works well with using autoserv to farm out different
> kvm_tests.cfg to different machines.

But still we'd have to manually (or automatically) divide kvm_tests.cfg
between the hosts, right? Either that, or we send all hosts the same
kvm_tests.cfg.

> Attaching my stale patch just for comment.  Needs to be updated since
> I sat on this for a while.  There were a number of issues:
> 
> - kvm_log is a shared resource, fixed it up so parallel jobs can both
>   call it

Maybe I missed something, but I think job.parallel() forks, so why do we need
to change kvm_log? How is it shared?
I encountered no problems with it when I tried running tests in parallel, but
maybe I didn't look carefully enough.

> - vnc, redir and other network resources are shared, so, in
> kvm_tests.cfg file each job needs a parallel offset.

Or we can lock a file at the beginning of VM.create() and unlock it once the VM
is running (and has taken the ports assigned to it).

I have a patch that does this. I'll post it soon hopefully (after some more
testing).

> - in kvm_tests.cfg file need to define additional vm and nic objects,
>    one for each parallel threads.

But then how do you assign VMs to tests? The user doesn't know in advance which
thread takes each VM. Can you provide a simple example config file to illustrate
this?

I see that you reset 'vms' and 'main_vm' before running each test. I'm not sure
you're supposed to decide for the user what VMs to use. The user might need more
than one VM (for migration or some stress test), and the user may choose to set
'main_vm' to any of these VMs, not necessarily one with a name like 'vm1'.

My solution was to use separate environment files (the default one is 'env').
That way there can be several VMs with the same name, living in different
environments.
This can be achieved by passing the env filename to kvm_runtest_2 as a parameter,
e.g. job.run_test("kvm_runtest_2", params=%s, env_filename='env2', ...

Another possible solution is to give each guest a different VM name in kvm_tests.cfg.

However, I still think it's safest to avoid sharing an env file, because who knows
what happens inside python's shelve module and what sort of corruptions we might get.
In fact, the docs state that shelve doesn't handle parallel access at all -- it's up
to the user to maintain database integrity.

> Advantages:
>     - works a lot like the single threaded model does, and if threads=1
>     then it runs the same path
>     - config files don't change significantly, just some additional
>     VM objects at the top and some offset values
>     - transparent to an autoserv setup, autoserv would just need to
>     specify the kvm_tests.cfg file to each host.
>
> Disadvantages:
>     - the main loop waits for each group of parallel jobs to complete
>     before starting any more.  If somehow an install is mixed with a
>     reboot test, we'll wait around before starting more jobs
>     - probably a few more here, but I don't have them on the top of my
>     head.

I have a feeling the code can be simplified a little. I'll try to write something
to illustrate what I mean, but it might take me a while.

Also, it shouldn't be too hard to support several queues that run continuously,
without having to wait for long tests to complete. It can be done using pickle
or something similar, with a shared queue file that is accessed with locks
by the parallel tasks.

In any case, we should see if this works well with the server. If it doesn't
(possibly because if we use a server control file we'll cover this functionality anyway),
then this can be a client-only solution.

One more question -- what is the purpose of the changes you made to job.py?
I know autotest already supports parallel execution, so I wonder what functionality
was missing.

Thanks,
Michael

Re: [RFC] KVM-Autotest: basic parallel test execution

[Ryan Harper]

* Mike Burns <mburns@redhat.com> [2009-05-20 16:13]:
> Ryan Harper wrote:
> > * Michael Goldish <mgoldish@redhat.com> [2009-05-17 09:50]:
> >   
> >> Hi all,
> >>
> >> We've recently implemented a very simple form of parallel test
> >> execution into KVM-Autotest and we'd like some feedback on it. This
> >> suggestion allows the user to manually assign tests to hosts/queues.
> >> It also takes care of assigning different MAC address ranges to
> >> hosts/queues. By 'queues' I mean parallel execution pipelines. Each
> >> host has one or more queues. The number of queues is defined by the
> >> user, and should reflect the capabilities of the host.
> >>
> >> This implementation involves only minor modifications to the code
> >> itself; most of the work is done in a new config file kvm_hosts.cfg,
> >> which has the exact same format as kvm_tests.cfg. The new file
> >> provides the framework with information about hosts/queues. The new
> >> file is parsed after kvm_tests.cfg. The test sets (such as
> >> 'nightly' and 'weekly'), previously defined at the end of
> >> kvm_tests.cfg, should now be defined last, after kvm_hosts.cfg.
> >> Test sets no longer select only the tests to execute, but also
> >> where each test should be executed (i.e. on what host/queue).
> >>
> >> The final result of parsing the config files is a list of tests, each
> >> with its own 'hostname' and 'queue' parameters. Each host executes
> >> only the tests whose 'hostname' parameter matches the current host,
> >> and puts tests with different 'queue' values in parallel pipelines
> >> of execution.
> >>
> >> Ideally, the Autotest server should take care of assigning tests to
> >> hosts automatically, but there are still a few technical difficulties
> >> to be resolved before we can implement that. We're considering the
> >> current suggestion as a temporary solution until a better one is
> >> found.
> >>
> >> Basically, the advantages are:
> >> - allows the user full control over what tests run, and where/how they run
> >> - takes care of assigning MAC address ranges to different hosts/queues (required for TAP networking)
> >> - can be used from the server or with the client, which makes it relevant also for users who don't have an Autotest server installed
> >> - involves only minor code changes (except for the config files)
> >> - is pretty much the simplest possible solution (and simple is good)
> >>
> >> Drawbacks:
> >> - requires some initial work to be done by the user -- the user has to define exactly where each test should run
> >> - test sets need to be modified when tests or hosts are added/removed, to include/exclude them
> >>     
> >
> > I took a slightly different approach.  The kvm_tests.cfg file already
> > provides a dependency relationship between different tests.  I modified
> > the main loop in the control file to walk the entire list of jobs and
> > pull out any jobs that don't have any dependencies (ie, install tests).
> > And then run N jobs in parallel from that list until it is exhausted;
> > then store the results.  Then loop the over the remaining list of jobs
> > again finding the jobs that can be run.
> >
> > On a larger multi core system, one might set the number of parallel jobs
> > equal to the number of cores.
> >
> > I think this works well with using autoserv to farm out different
> > kvm_tests.cfg to different machines.
> >
> > Attaching my stale patch just for comment.  Needs to be updated since I
> > sat on this for a while.  There were a number of issues:
> >
> > - kvm_log is a shared resource, fixed it up so parallel jobs can both
> >   call it
> > - vnc, redir and other network resources are shared, so, in
> > kvm_tests.cfg file each job needs a parallel offset.
> > - in kvm_tests.cfg file need to define additional vm and nic objects,
> >    one for each parallel threads.
> >
> > Advantages:
> >     - works a lot like the single threaded model does, and if threads=1
> >     then it runs the same path
> >     - config files don't change significantly, just some additional
> >     VM objects at the top and some offset values
> >     - transparent to an autoserv setup, autoserv would just need to
> >     specify the kvm_tests.cfg file to each host.
> >
> > Disadvantages:
> >     - the main loop waits for each group of parallel jobs to complete
> >     before starting any more.  If somehow an install is mixed with a
> >     reboot test, we'll wait around before starting more jobs
> >     - probably a few more here, but I don't have them on the top of my
> >     head.
> >
> >   
> I haven't looked through the code at all yet, but the design seems more
> in line with my thoughts of parallel execution.  Just a couple really
> quick thoughts on this:
> 
> There is an effort being made to merge kvm-autotest with autotest.  I
> think part of that includes a change to remove kvm_log.  I don't know if
> the upstream logging system will support multi-threading, but I wouldn't
> spend any time working on kvm_log.

Indeed, I spent as little time on it as it took to get it working with 2
parallel installs.  I don't have any plans for it other than if it is
present to not be in the way of parallel execution.

> 
> I haven't dug through all the execution code yet.  I've mostly played
> around in the kvm_install area and stepsfile stuff.  Couldn't we do
> something like this:
> 
> define a param called num_threads
> have base names for things like vms, images, nics, etc like vm, image, nic
> each thread picks up a test to run and uses the base names + thead num
> We could then use the thread number all the offsets as well.

Yep.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: [RFC] KVM-Autotest: basic parallel test execution

[Ryan Harper]

* Michael Goldish <mgoldish@redhat.com> [2009-05-20 18:15]:
> ----- "Ryan Harper" <ryanh@us.ibm.com> wrote:
> > * Michael Goldish <mgoldish@redhat.com> [2009-05-17 09:50]:
> > 
> > I took a slightly different approach.  The kvm_tests.cfg file already
> > provides a dependency relationship between different tests.  I
> > modified
> > the main loop in the control file to walk the entire list of jobs and
> > pull out any jobs that don't have any dependencies (ie, install
> > tests).
> > And then run N jobs in parallel from that list until it is exhausted;
> > then store the results.  Then loop the over the remaining list of
> > jobs again finding the jobs that can be run.
> 
> I like this approach. Looks like it's somewhere between the simple
> static way and the fully automatic server way.
> 
> > On a larger multi core system, one might set the number of parallel
> > jobs equal to the number of cores.
> 
> It makes sense to define 'threads' in the control file, because it's
> definitely not a test param, and the control file is easiest to change from
> the server. However, I wonder how different values of 'threads' can be
> defined for different hosts (with different capabilities) from the server.

yeah, I'm pretty sure you can dynamically generate the control file via
autoserv.

> 
> > I think this works well with using autoserv to farm out different
> > kvm_tests.cfg to different machines.
> 
> But still we'd have to manually (or automatically) divide kvm_tests.cfg
> between the hosts, right? Either that, or we send all hosts the same
> kvm_tests.cfg.

Yes.  Manual is the easiest first step.

> > Attaching my stale patch just for comment.  Needs to be updated since
> > I sat on this for a while.  There were a number of issues:
> > 
> > - kvm_log is a shared resource, fixed it up so parallel jobs can both
> >   call it
> 
> Maybe I missed something, but I think job.parallel() forks, so why do we need
> to change kvm_log? How is it shared?
> I encountered no problems with it when I tried running tests in parallel, but
> maybe I didn't look carefully enough.

Looking through my modifications, I think I wanted the logger to include
the PID of the thread so one would know which job was writing what to
the log. 

> 
> > - vnc, redir and other network resources are shared, so, in
> > kvm_tests.cfg file each job needs a parallel offset.
> 
> Or we can lock a file at the beginning of VM.create() and unlock it once the VM
> is running (and has taken the ports assigned to it).
> 
> I have a patch that does this. I'll post it soon hopefully (after some more
> testing).

I think the offset method is simpler and no locks.

> 
> > - in kvm_tests.cfg file need to define additional vm and nic objects,
> >    one for each parallel threads.
> 
> But then how do you assign VMs to tests? The user doesn't know in advance which
> thread takes each VM. Can you provide a simple example config file to illustrate
> this?

I don't have my config I used around, but I doubt I was thinking that
much about this.

> 
> I see that you reset 'vms' and 'main_vm' before running each test. I'm not sure
> you're supposed to decide for the user what VMs to use. The user might need more
> than one VM (for migration or some stress test), and the user may choose to set
> 'main_vm' to any of these VMs, not necessarily one with a name like 'vm1'.

I definitely wasn't worried about allowing the user to pick these
assignments.  Mainly because I didn't see much value; probably because I
haven't written a sophosticated enough config for it to matter.  

> 
> My solution was to use separate environment files (the default one is 'env').
> That way there can be several VMs with the same name, living in different
> environments.

Yeah, I was trying to do that, but I failed to get that working.  That
should allow the same definitions to be duplicated between threads (like
main_vm name, etc) but since we'd still have to share port resources, I
think we can keep the same env and use an offset to parallelize access
to shared resources.

> This can be achieved by passing the env filename to kvm_runtest_2 as a parameter,
> e.g. job.run_test("kvm_runtest_2", params=%s, env_filename='env2', ...
> 
> Another possible solution is to give each guest a different VM name in kvm_tests.cfg.
> 
> However, I still think it's safest to avoid sharing an env file, because who knows
> what happens inside python's shelve module and what sort of corruptions we might get.
> In fact, the docs state that shelve doesn't handle parallel access at all -- it's up
> to the user to maintain database integrity.

OK.  I don't care that much that we use the same env file, I was just
looking for the simplest route to parallel jobs.  If we can safely use
the same env, I think that is the simplest method.  If not, then I think
we can work up a way to use a env per thread.

> 
> > Advantages:
> >     - works a lot like the single threaded model does, and if threads=1
> >     then it runs the same path
> >     - config files don't change significantly, just some additional
> >     VM objects at the top and some offset values
> >     - transparent to an autoserv setup, autoserv would just need to
> >     specify the kvm_tests.cfg file to each host.
> >
> > Disadvantages:
> >     - the main loop waits for each group of parallel jobs to complete
> >     before starting any more.  If somehow an install is mixed with a
> >     reboot test, we'll wait around before starting more jobs
> >     - probably a few more here, but I don't have them on the top of my
> >     head.
> 
> I have a feeling the code can be simplified a little. I'll try to write something
> to illustrate what I mean, but it might take me a while.

Indeed.  I'll see about cleaning this up and reposting in the meantime.

> 
> Also, it shouldn't be too hard to support several queues that run continuously,
> without having to wait for long tests to complete. It can be done using pickle
> or something similar, with a shared queue file that is accessed with locks
> by the parallel tasks.

Sounds fancy.  I'd like to see if we can get something simple in first.

> 
> In any case, we should see if this works well with the server. If it doesn't
> (possibly because if we use a server control file we'll cover this functionality anyway),
> then this can be a client-only solution.

Right, I don't have a proper autoserv setup at the moment so once I
repost the patch, it would be good for someone who does have this setup
to give it a spin.

> 
> One more question -- what is the purpose of the changes you made to job.py?
> I know autotest already supports parallel execution, so I wonder what functionality
> was missing.

The job.py in kvm-autotest only returns one return code for the parallel
job and in kvm_runtest_2, we need a return code for each thread so we
can mark if the tests was successful or not.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: [RFC] KVM-Autotest: basic parallel test execution

[Avi Kivity]

Ryan Harper wrote:
> I took a slightly different approach.  The kvm_tests.cfg file already
> provides a dependency relationship between different tests.  I modified
> the main loop in the control file to walk the entire list of jobs and
> pull out any jobs that don't have any dependencies (ie, install tests).
> And then run N jobs in parallel from that list until it is exhausted;
> then store the results.  Then loop the over the remaining list of jobs
> again finding the jobs that can be run.
>
> On a larger multi core system, one might set the number of parallel jobs
> equal to the number of cores.
>
> I think this works well with using autoserv to farm out different
> kvm_tests.cfg to different machines.
>   

This is my preference as well, though I think scheduling needs to be 
cleverer.  Each test should specify how much memory and how many cores 
(the -m and -smp parameters) it needs, and the scheduler needs to make 
sure we don't over commit the host for tests.

This ensures the best utilization while not interfering with test timing.

-- 
error compiling committee.c: too many arguments to function

Re: [RFC] KVM-Autotest: basic parallel test execution

[Avi Kivity]

Mike Burns wrote:
> Avi Kivity wrote:
>   
>> Michael Goldish wrote:
>>     
>>> Drawbacks:
>>> - requires some initial work to be done by the user -- the user has
>>> to define exactly where each test should run
>>>   
>>>       
>> For me, this is a major drawback.  I'd really like a fire-and-forget
>> solution.  If I have to spend my own time getting this to work, vs.
>> waiting longer for the tests to run on their own, I'll just be lazy.
>>     
> It seems like this is adding another layer of user interaction without
> really getting that much additional functionality.  An administrator of
> kvm-autotest is going to know enough to be able to create control and
> kvm_tests.cfg files that can mimic this already and then just create 2
> jobs in the server to run on different hosts. 
>   

I'm an administrator of kvm-autotest and I have no idea how to create 
control and kvm_tests.cfg files.

Also, I'm a lot more interested in spreading the load on my one host 
rather than on fictional other hosts.  And for that we need a scheduler, 
since different tests need differing amounts of memory and cores.

>>> - test sets need to be modified when tests or hosts are
>>> added/removed, to include/exclude them
>>>   
>>>       
>> This is also annoying -- and likely to stop me from updating.
>>     
> Host definitions already happen on the server and it seems like it would
> be confusing to have to make sure that the host you choose when setting
> up a test is setup correctly in your config files. 
>   

Host setups should be separated from test setup, so that tests can be 
continuously updated while the host setup is kept static.

>> I'd really like this to be automated, just specify a set of machines
>> and have the jobs distributed.  Furthermore, it is very important to
>> utilize the existing hosts better.  A 4-core 4GB server can easily run
>> a 2x smp 1GB guest and 2 other uniprocessor 1GB guests.  It's wasteful
>> to add more servers when the existing servers are underutilized.
>>
>>     
> Overall, the way I envisioned parallel test execution was having
> multiple tests running on a single machine.  It seems that the server
> already provides most (if not all) the functionality needed to spread
> tests across multiple machines.  I really think this is putting too much
> on the user for very small gains in functionality. 
>   

As a user, I disagree.  I can't calculate what resources are needed by 
each test and make sure they all fit on my host.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.