Re: [PATCH] [RFC] KVM test: Major control file cleanup

Re: [PATCH] [RFC] KVM test: Major control file cleanup

[Michael Goldish]

I think this patch is a good idea.  Comments:

- If we've decided to put all the configuration in config files
(rather than the control file) let's split up the main one into
kvm_test_library.cfg and kvm_tests.cfg.sample.  The latter should
look like this:

include kvm_test_library.cfg
include kvm_cdkeys.cfg

variants:
    - fc8_quick:
        ...
    - other_test_set:
        ...

only fc8_quick

... and the former should contain everything else.

- I think we should allow the user to modify his/her configuration
in the control file using parse_string().  This is actually useful,
especially when running from the server.  (At least I use it.)

- If kvm_build.cfg or kvm_tests.cfg are missing, we should continue
without them, or create them (blank), instead of failing due to an
unhandled exception raised by the config parser.  This should be
true at least for kvm_build.cfg, which isn't always required for the
tests to run.

- We should consider removing hardcoded references to
/tmp/kvm_autotest_root/.  We use that path locally in TLV but that's
not a good reason to put it upstream IMO.
We should also stop creating symlinks to that path.
Instead we can allow more flexibility using config files.
For example, we can put this in kvm_tests.cfg.sample, between the
'include' and the 'variants':

# Uncomment and modify the following lines if you wish to modify
# the paths of the image files, ISO files, step files or qemu binaries.
# If you keep these lines commented, all mentioned paths are expected
# to be found in the KVM test dir (tests/kvm/).  For example, all
# image files are expected under tests/kvm/images/.

# qemu_binary ?<= /tmp/kvm_autotest_root/
# qemu_img_binary ?<= /tmp/kvm_autotest_root/
# image_name.* ?<= /tmp/kvm_autotest_root/
# cdrom.* ?<= /tmp/kvm_autotest_root/
# steps ?<= /tmp/kvm_autotest_root/

The ?<= operator prepends strings to keys that match a regex, so
'image_name.* ?<= something/' will prepend 'something/' to image_name,
image_name_vm2 and so on.
In kvm_test_library.cfg (currently kvm_tests.cfg.sample) all
'image_name' parameters, for example, begin with 'images/', which means
all image files should be under tests/kvm/images.
If we prepend /tmp/kvm_autotest_root/ to that, all image files will be
expected under /tmp/kvm_autotest_root/images/.

Maybe we should leave these lines uncommented by default, in case anyone
uses /tmp/kvm_autotest_root and relies on the control file creating the
symlinks.

- kvm_utils.run_tests() should return True if all tests passed and
otherwise False.  We need this in order to quit if the build test(s)
failed.

- I suggest we use kvm_utils.parse_host_specific_cfg() instead of
kvm_utils.get_test_list().  I think 'parse_host_specific_cfg' better
describes what the function does.  I also think it should take
'kvm_address_pools.cfg' as a parameter so we don't have to hardcode it
into the function.


Please see additional comments below.


----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:

> As pointed out before, the KVM reference control files
> could use a little clean up. This patch implements severe
> cleanup of the main control file by:
> 
> * Refactoring the code present there, moving it to the
> kvm_utils.py library
> * Treat the build test exactly the same way as other
> tests, moving the config stuff that used to be in the
> control file realm out to its own configuration file,
> for the sake of consistency.
> * Since a frequent complaint amongst users is that by
> default the build test will build from release, and
> that build might be unsuccessful depending on the system
> being used (older kernel versions and other sorts of
> stuff), change the default build method to 'noinstall'.
> 
> This way the control file becomes way shorter, fairly
> well organized, and we have a consistent configuration
> schema across the board, based on configuration files.
> 
> If people are OK with this change, final path will
> change the control.parallel file as well.
> 
> Important: For the default control file, I removed
> the part of the code that parses new strings for the
> test configuration because I believe those
> modifications belong to the test config files, they
> could be just commented. For cases where keeping
> all stuff inside the config file is not possible,
> appropriate code will be added (in other example
> control files).
> 
> Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
> ---
>  client/tests/kvm/control              |  203
> +++------------------------------
>  client/tests/kvm/kvm_build.cfg.sample |   82 +++++++++++++
>  client/tests/kvm/kvm_utils.py         |  112 ++++++++++++++++++
>  3 files changed, 211 insertions(+), 186 deletions(-)
>  create mode 100644 client/tests/kvm/kvm_build.cfg.sample
> 
> diff --git a/client/tests/kvm/control b/client/tests/kvm/control
> index 431baf9..7acec14 100644
> --- a/client/tests/kvm/control
> +++ b/client/tests/kvm/control
> @@ -20,194 +20,25 @@ KVM (both kernelspace and userspace) code.
>  For online docs, please refer to
> http://www.linux-kvm.org/page/KVM-Autotest
>  """
>  
> +import sys, os, logging
> +# Add the KVM tests dir to the python path
> +kvm_test_dir = os.path.join(os.environ['AUTODIR'],'tests/kvm')
> +sys.path.append(kvm_test_dir)
> +import kvm_utils, kvm_config
>  
> -import sys, os
>  
> -#-----------------------------------------------------------------------------
> -# set English environment (command output might be localized, need to
> be safe)
> -#-----------------------------------------------------------------------------
> -os.environ['LANG'] = 'en_US.UTF-8'
> +## Prepare the environment for the KVM test.
> +kvm_utils.prepare_test_run(basedir=kvm_test_dir,
> +                           rootdir='/tmp/kvm_autotest_root')
>  
> -#---------------------------------------------------------
> -# Enable modules import from current directory (tests/kvm)
> -#---------------------------------------------------------
> -pwd = os.path.join(os.environ['AUTODIR'],'tests/kvm')
> -sys.path.append(pwd)
> +cfg_names = ['kvm_build.cfg', 'kvm_tests.cfg']
>  
> -# ------------------------
> -# create required symlinks
> -# ------------------------
> -# When dispatching tests from autotest-server the links we need do
> not exist on
> -# the host (the client). The following lines create those symlinks.
> Change
> -# 'rootdir' here and/or mount appropriate directories in it.
> -#
> -# When dispatching tests on local host (client mode) one can either
> setup kvm
> -# links, or same as server mode use rootdir and set all appropriate
> links and
> -# mount-points there. For example, guest installation tests need to
> know where
> -# to find the iso-files.
> -#
> -# We create the links only if not already exist, so if one already
> set up the
> -# links for client/local run we do not touch the links.
> -rootdir='/tmp/kvm_autotest_root'
> -iso=os.path.join(rootdir, 'iso')
> -images=os.path.join(rootdir, 'images')
> -qemu=os.path.join(rootdir, 'qemu')
> -qemu_img=os.path.join(rootdir, 'qemu-img')
> +for cfg_name in cfg_names:
> +    cfg_path = os.path.join(kvm_test_dir, cfg_name)
> +    cfg = kvm_config.config(cfg_path)
> +    list = kvm_utils.get_test_list(cfg, cfg_name, kvm_test_dir)
> +    logging.info("Running test set defined on config file %s",
> cfg_name)
> +    kvm_utils.run_tests(list, job)

I think we should unroll this loop and allow users to modify the main
and/or build configurations here.  For example:

build_cfg_path = os.path.join(kvm_test_dir, "kvm_build.cfg")
test_cfg_path = os.path.join(kvm_test_dir, "kvm_tests.cfg")
open(build_cfg_path, 'a').close()
open(test_cfg_path, 'a').close()

cfg = kvm_config.config(build_cfg_path)
cfg.parse_string("""
# Make any desired changes to the build configuration here.  For example:
# release_tag = 84
""")
if not kvm_utils.run_tests(cfg.get_list(), job.run_test):
    sys.exit(1)

cfg = kvm_config.config(test_cfg_path)
cfg.parse_string("""
# Make any desired changes to the test configuration here.  For example:
# display = sdl
# install|setup: timeout_multiplier = 3
""")
kvm_utils.parse_host_specific_cfg(cfg, os.path.join(kvm_test_dir,
                                                    "kvm_address_pools.cfg"))
kvm_utils.run_tests(cfg.get_list(), job.run_test)

Alternatively, instead of creating the files in case they don't exit,
we can put the code that parses and runs tests inside 'if' statements.
I'm not sure which option is shorter.

>  
> -
> -def link_if_not_exist(ldir, target, link_name):
> -    t = target
> -    l = os.path.join(ldir, link_name)
> -    if not os.path.exists(l):
> -        os.system('ln -s %s %s' % (t, l))
> -
> -# Create links only if not already exist
> -link_if_not_exist(pwd, '../../', 'autotest')
> -link_if_not_exist(pwd, iso, 'isos')
> -link_if_not_exist(pwd, images, 'images')
> -link_if_not_exist(pwd, qemu, 'qemu')
> -link_if_not_exist(pwd, qemu_img, 'qemu-img')
> -
> -# --------------------------------------------------------
> -# Params that will be passed to the KVM install/build test
> -# --------------------------------------------------------
> -params = {
> -    "name": "build",
> -    "shortname": "build",
> -    "type": "build",
> -    "mode": "release",
> -    #"mode": "snapshot",
> -    #"mode": "localtar",
> -    #"mode": "localsrc",
> -    #"mode": "git",
> -    #"mode": "noinstall",
> -    #"mode": "koji",
> -
> -    ## Are we going to load modules built by this test?
> -    ## Defaults to 'yes', so if you are going to provide only
> userspace code to
> -    ## be built by this test, please set load_modules to 'no', and
> make sure
> -    ## the kvm and kvm-[vendor] module is already loaded by the time
> you start
> -    ## it.
> -    "load_modules": "no",
> -
> -    ## Install from a kvm release ("mode": "release"). You can
> optionally
> -    ## specify a release tag. If you omit it, the test will get the
> latest
> -    ## release tag available.
> -    #"release_tag": '84',
> -    "release_dir": 'http://downloads.sourceforge.net/project/kvm/',
> -    # This is the place that contains the sourceforge project list of
> files
> -    "release_listing": 'http://sourceforge.net/projects/kvm/files/',
> -
> -    ## Install from a kvm snapshot location ("mode": "snapshot"). You
> can
> -    ## optionally specify a snapshot date. If you omit it, the test
> will get
> -    ## yesterday's snapshot.
> -    #"snapshot_date": '20090712'
> -    #"snapshot_dir": 'http://foo.org/kvm-snapshots/',
> -
> -    ## Install from a tarball ("mode": "localtar")
> -    #"tarball": "/tmp/kvm-84.tar.gz",
> -
> -    ## Install from a local source code dir ("mode": "localsrc")
> -    #"srcdir": "/path/to/source-dir"
> -
> -    ## Install from koji build server ("mode": "koji")
> -    ## Koji is the Fedora Project buildserver. It is possible to
> install
> -    ## packages right from Koji if you provide a release tag or a
> build.
> -    ## Tag (if available)
> -    #"koji_tag": 'dist-f11',
> -    ## Build (if available, is going to override tag).
> -    #"koji_build": 'qemu-0.10-16.fc11',
> -    ## Command to interact with the build server
> -    #"koji_cmd": '/usr/bin/koji',
> -    ## The name of the source package that's being built
> -    #"src_pkg": 'qemu',
> -    ## Name of the rpms we need installed
> -    #"pkg_list": ['qemu-kvm', 'qemu-kvm-tools', 'qemu-system-x86',
> 'qemu-common', 'qemu-img'],
> -    ## Paths of the qemu relevant executables that should be checked
> -    #"qemu_bin_paths": ['/usr/bin/qemu-kvm', '/usr/bin/qemu-img'],
> -
> -    ## Install from git ("mode": "git")
> -    ## If you provide only "git_repo" and "user_git_repo", the build
> test
> -    ## will assume it will perform all build from the userspace dir,
> building
> -    ## modules trough make -C kernel LINUX=%s sync. As of today
> (07-13-2009)
> -    ## we need 3 git repos, "git_repo" (linux sources),
> "user_git_repo" and 
> -    ## "kmod_repo" to build KVM userspace + kernel modules.
> -    #"git_repo":
> 'git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git',
> -    #"kernel_branch": 'kernel_branch_name',
> -    #"kernel_lbranch": 'kernel_lbranch_name',
> -    #"kernel_tag": 'kernel_tag_name',
> -    #"user_git_repo":
> 'git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git',
> -    #"user_branch": 'user_branch_name',
> -    #"user_lbranch": 'user_lbranch_name',
> -    #"user_tag": 'user_tag_name',
> -    #"kmod_repo":
> 'git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git',
> -    #"kmod_branch": 'kmod_branch_name',
> -    #"kmod_lbranch": 'kmod_lbranch_name',
> -    #"kmod_tag": 'kmod_tag_name',
> -}
> -
> -# If you don't want to execute the build stage, just use 'noinstall'
> as the
> -# install type. If you run the tests from autotest-server, make sure
> that
> -# /tmp/kvm-autotest-root/qemu is a link to your existing executable.
> Note that
> -# if kvm_install is chose to run, it overwrites existing qemu and
> qemu-img
> -# links to point to the newly built executables.
> -r = True
> -r = job.run_test("kvm", params=params, tag=params.get("shortname"))
> -if not r:
> -    print 'kvm_installation failed ... exiting'
> -    sys.exit(1)
> -
> -# ----------------------------------------------------------
> -# Get test set (dictionary list) from the configuration file
> -# ----------------------------------------------------------
> -import kvm_config
> -
> -filename = os.path.join(pwd, "kvm_tests.cfg")
> -cfg = kvm_config.config(filename)
> -
> -# If desirable, make changes to the test configuration here.  For
> example:
> -# cfg.parse_string("install|setup: timeout_multiplier = 2")
> -# cfg.parse_string("only fc8_quick")
> -# cfg.parse_string("display = sdl")
> -
> -filename = os.path.join(pwd, "kvm_address_pools.cfg")
> -if os.path.exists(filename):
> -    cfg.parse_file(filename)
> -    hostname = os.uname()[1].split(".")[0]
> -    if cfg.filter("^" + hostname):
> -        cfg.parse_string("only ^%s" % hostname)
> -    else:
> -        cfg.parse_string("only ^default_host")
> -
> -list = cfg.get_list()
> -
> -
> -# -------------
> -# Run the tests
> -# -------------
> -status_dict = {}
> -
> -for dict in list:
> -    if dict.get("skip") == "yes":
> -        continue
> -    dependencies_satisfied = True
> -    for dep in dict.get("depend"):
> -        for test_name in status_dict.keys():
> -            if not dep in test_name:
> -                continue
> -            if not status_dict[test_name]:
> -                dependencies_satisfied = False
> -                break
> -    if dependencies_satisfied:
> -        test_iterations = int(dict.get("iterations", 1))
> -        current_status = job.run_test("kvm", params=dict,
> -                                      tag=dict.get("shortname"),
> -                                      iterations=test_iterations)
> -    else:
> -        current_status = False
> -    status_dict[dict.get("name")] = current_status
> -
> -# create the html report in result dir
> -reporter = os.path.join(pwd, 'html_report.py')
> -html_file = os.path.join(job.resultdir,'results.html')
> -os.system('%s -r %s -f %s -R' % (reporter, job.resultdir,
> html_file))
> +## Generate a nice HTML report inside results_dir
> +kvm_utils.create_report(kvm_test_dir, job.resultsdir)
> diff --git a/client/tests/kvm/kvm_build.cfg.sample
> b/client/tests/kvm/kvm_build.cfg.sample
> new file mode 100644
> index 0000000..692a747
> --- /dev/null
> +++ b/client/tests/kvm/kvm_build.cfg.sample
> @@ -0,0 +1,82 @@
> +# This configuration file holds the KVM build test config
> parameters.
> +# The default is noinstall (won't attempt to build KVM), so if you
> stick with it
> +# please make sure:
> +# 1) You have setup symbolic links to qemu and qemu-img binaries on
> the
> +#    KVM test dir.
> +# 2) The appropriate KVM modules are already loaded on your machine.
> +
> +variants:
> +    - build:
> +        type = build
> +        # Load modules built/installed by the build test?
> +        load_modules = no
> +        variants:
> +            - noinstall:
> +                mode = noinstall
> +            - release:
> +                mode = release
> +                ## Install from a kvm release. You can optionally
> specify
> +                ## a release tag. If you omit it, the build test will
> get
> +                ## the latest release tag available at that moment.
> +                # release_tag = 84
> +                release_dir =
> http://downloads.sourceforge.net/project/kvm/
> +                release_listing =
> http://sourceforge.net/projects/kvm/files/
> +            - snapshot:
> +                mode = snapshot
> +                ## Install from a kvm snapshot location. You can
> optionally
> +                ## specify a snapshot date. If you omit it, the test
> will get
> +                ## yesterday's snapshot.
> +                # snapshot_date = 20090712
> +                snapshot_dir = http://foo.org/kvm-snapshots/
> +            - localtar:
> +                mode = localtar
> +                ## Install from tarball located on the host's
> filesystem.
> +                tarball = /tmp/kvm-84.tar.gz
> +            - localsrc:
> +                mode = localsrc
> +                ## Install from tarball located on the host's
> filesystem.
> +                srcdir = /tmp/kvm-84
> +            - git:
> +                mode = git
> +                ## Install KVM from git repositories.
> +                ## If you provide only "git_repo" and
> "user_git_repo", the
> +                ## build test will assume it will perform all build
> from the
> +                ## userspace dir, building modules trough 
> +                ## make -C kernel LINUX=%s sync. As of today
> (07-13-2009)
> +                ## upstream needs 3 git repos:
> +                ## * git_repo (linux sources)
> +                ## * user_git_repo (qemu sources)
> +                ## * kmod_repo" to build KVM userspace + kernel
> modules.
> +                git_repo =
> git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git
> +                # kernel_branch = kernel_branch_name
> +                # kernel_lbranch = kernel_lbranch_name
> +                # kernel_tag = kernel_tag_name
> +                user_git_repo =
> git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
> +                # user_branch = user_branch_name
> +                # user_lbranch = user_lbranch_name
> +                # user_tag = user_tag_name
> +                kmod_repo =
> git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git
> +                # kmod_branch = kmod_branch_name
> +                # kmod_lbranch = kmod_lbranch_name
> +                # kmod_tag = kmod_tag_name
> +            - koji:
> +                mode = koji
> +                ## Install KVM from koji (Fedora build server)
> +                ## It is possible to install packages right from Koji
> if you
> +                ## provide a release tag or a build.
> +                ## Tag (if available)
> +                koji_tag = dist-f11
> +                ## Build (if available, is going to override tag).
> +                koji_build = qemu-0.10-16.fc11
> +                ## Command to interact with the build server
> +                koji_cmd = /usr/bin/koji
> +                ## The name of the source package that will be built
> +                src_pkg = qemu
> +                ## Name of the rpms we need installed
> +                pkg_list = ['qemu-kvm', 'qemu-kvm-tools',
> 'qemu-system-x86', 'qemu-common', 'qemu-img']
> +                ## Paths of the qemu relevant executables that should
> be checked
> +                qemu_bin_paths = ['/usr/bin/qemu-kvm',
> '/usr/bin/qemu-img']
> +
> +
> +# Choose your install method here
> +only build.noinstall
> \ No newline at end of file
> diff --git a/client/tests/kvm/kvm_utils.py
> b/client/tests/kvm/kvm_utils.py
> index f72984a..90b3def 100644
> --- a/client/tests/kvm/kvm_utils.py
> +++ b/client/tests/kvm/kvm_utils.py
> @@ -871,3 +871,115 @@ def unmap_url_cache(cachedir, url,
> expected_hash, method="md5"):
>          file_path = utils.unmap_url(cachedir, src, cachedir)
>  
>      return file_path
> +
> +
> +def prepare_test_run(basedir, rootdir='/tmp/kvm_autotest_root'):
> +    """
> +    Prepare the environment for a KVM test run by linking some
> previously
> +    prepared directories to the test bindir.
> +
> +    @param basedir: Location of the KVM test directory
> +    @param rootdir: Base directory that contains the KVM support
> directory
> +            structure:
> +            * isos (holds guest install isos)
> +            * images (will hold guest disk images)
> +            * qemu (qemu binary)
> +            * qemu-img (qemu-img binary)
> +    """
> +    # set English environment (command output might be localized,
> +    # need to be safe)
> +    os.environ['LANG'] = 'en_US.UTF-8'
> +
> +    # Symbolic links we desire to see in the KVM test dir
> +    list_symlinks = ['isos', 'images', 'qemu', 'qemu-img']
> +    for symlink_basename in list_symlinks:
> +        link_src = os.path.join(rootdir, symlink_basename)
> +        link_dst = os.path.join(basedir, symlink_basename)
> +        if not os.path.islink(link_dst):
> +            os.symlink(link_src, link_dst)
> +        # Testing whether our symlinks are good and in shape
> +        link_good = os.path.exists(os.readlink(link_dst))
> +        if not link_good:
> +            if symlink_basename in ['isos', 'images']:
> +                logging.critical("Mandatory directory %s is a broken
> symlink. "
> +                                 "This job will be aborted so you can
> fix the "
> +                                 "problem. Make sure %s points to a
> valid "
> +                                 "directory before proceeding.",
> +                                 symlink_basename, link_dst)
> +                raise error.JobError("Mandatory directory %s is a
> broken "
> +                                     "symlink. Please correct it." 
> %
> +                                     (link_dst))
> +            else:
> +                logging.warning("Symlink %s is broken. You might have
> problems "
> +                                "later in the test if the KVM build
> test mode "
> +                                "selected is 'noinstall' (default).
> ", link_dst)

As mentioned before I'm not sure we need this function at all.
If the user has to modify kvm_tests.cfg.sample anyway, he/she will see our
comments regarding the qemu/images/isos/steps paths.

> +def get_test_list(cfg, cfg_name, base_dir):
> +    """
> +    Return a list of test dictionaries composed by the KVM
> configuration system.
> +    This might look like a delegate of the get_list() method of the
> +    configuration, but in fact we might want to do custom config
> processing
> +    depending on the config file name provided.
> +
> +    @param cfg: KVM test configuration object
> +    @param cfg_name: Basename of the configuration file that
> generated the
> +            configuration object
> +    @param base_dir: Directory that holds the config files
> +
> +    @return: A list with test parameters dictionaries.
> +    """
> +    if cfg_name == 'kvm_tests.cfg':
> +        pools = os.path.join(base_dir, "kvm_address_pools.cfg")
> +        if os.path.exists(pools):
> +            cfg.parse_file(pools)
> +            hostname = os.uname()[1].split(".")[0]
> +            if cfg.filter("^" + hostname):
> +                cfg.parse_string("only ^%s" % hostname)
> +            else:
> +                cfg.parse_string("only ^default_host")
> +
> +    return cfg.get_list()

Already commented on this one above.

> +def run_tests(test_list, job):
> +    """
> +    Runs the sequence of KVM tests based on the list of dictionaries
> +    generated by the configuration system, handling dependencies.
> +
> +    @param test_list: List with all dictionary test parameters.
> +    @param job: Autotest job object.
> +    """
> +    status_dict = {}
> +
> +    for dict in test_list:
> +        if dict.get("skip") == "yes":
> +            continue
> +        dependencies_satisfied = True
> +        for dep in dict.get("depend"):
> +            for test_name in status_dict.keys():
> +                if not dep in test_name:
> +                    continue
> +                if not status_dict[test_name]:
> +                    dependencies_satisfied = False
> +                    break
> +        if dependencies_satisfied:
> +            test_iterations = int(dict.get("iterations", 1))
> +            current_status = job.run_test("kvm", params=dict,
> +                                          tag=dict.get("shortname"),
> +                                         
> iterations=test_iterations)
> +        else:
> +            current_status = False
> +        status_dict[dict.get("name")] = current_status

Already commented on this one above.

> +def create_report(report_dir, results_dir):
> +    """
> +    Creates a neatly arranged HTML results report in the results
> dir.
> +
> +    @param report_dir: Directory where the report script is located.
> +    @param results_dir: Directory where the results will be output.
> +    """
> +    reporter = os.path.join(report_dir, 'html_report.py')
> +    html_file = os.path.join(results_dir, 'results.html')
> +    os.system('%s -r %s -f %s -R' % (reporter, results_dir,
> html_file))
> -- 
> 1.6.2.5

Re: [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

One thing that just occurred to me is, if we have a test config
"library" as you said, it's perfectly possible to put the actual test
set definitions and other config files inside the control file as
strings. This way one can control configuration inside the control file,
making it more convenient for usage, let's say, inside the autotest web
interface. Since the control file would be reduced in size, the
configuration snippets being in the control file would not be a huge
problem, while keeping the original autotest philosophy of keeping stuff
inside the control file... What do you think?

On Wed, 2009-10-28 at 05:48 -0400, Michael Goldish wrote:
> I think this patch is a good idea.  Comments:
> 
> - If we've decided to put all the configuration in config files
> (rather than the control file) let's split up the main one into
> kvm_test_library.cfg and kvm_tests.cfg.sample.  The latter should
> look like this:
> 
> include kvm_test_library.cfg
> include kvm_cdkeys.cfg
> 
> variants:
>     - fc8_quick:
>         ...
>     - other_test_set:
>         ...
> 
> only fc8_quick
> 
> ... and the former should contain everything else.
> 
> - I think we should allow the user to modify his/her configuration
> in the control file using parse_string().  This is actually useful,
> especially when running from the server.  (At least I use it.)
> 
> - If kvm_build.cfg or kvm_tests.cfg are missing, we should continue
> without them, or create them (blank), instead of failing due to an
> unhandled exception raised by the config parser.  This should be
> true at least for kvm_build.cfg, which isn't always required for the
> tests to run.
> 
> - We should consider removing hardcoded references to
> /tmp/kvm_autotest_root/.  We use that path locally in TLV but that's
> not a good reason to put it upstream IMO.
> We should also stop creating symlinks to that path.
> Instead we can allow more flexibility using config files.
> For example, we can put this in kvm_tests.cfg.sample, between the
> 'include' and the 'variants':
> 
> # Uncomment and modify the following lines if you wish to modify
> # the paths of the image files, ISO files, step files or qemu binaries.
> # If you keep these lines commented, all mentioned paths are expected
> # to be found in the KVM test dir (tests/kvm/).  For example, all
> # image files are expected under tests/kvm/images/.
> 
> # qemu_binary ?<= /tmp/kvm_autotest_root/
> # qemu_img_binary ?<= /tmp/kvm_autotest_root/
> # image_name.* ?<= /tmp/kvm_autotest_root/
> # cdrom.* ?<= /tmp/kvm_autotest_root/
> # steps ?<= /tmp/kvm_autotest_root/
> 
> The ?<= operator prepends strings to keys that match a regex, so
> 'image_name.* ?<= something/' will prepend 'something/' to image_name,
> image_name_vm2 and so on.
> In kvm_test_library.cfg (currently kvm_tests.cfg.sample) all
> 'image_name' parameters, for example, begin with 'images/', which means
> all image files should be under tests/kvm/images.
> If we prepend /tmp/kvm_autotest_root/ to that, all image files will be
> expected under /tmp/kvm_autotest_root/images/.
> 
> Maybe we should leave these lines uncommented by default, in case anyone
> uses /tmp/kvm_autotest_root and relies on the control file creating the
> symlinks.
> 
> - kvm_utils.run_tests() should return True if all tests passed and
> otherwise False.  We need this in order to quit if the build test(s)
> failed.
> 
> - I suggest we use kvm_utils.parse_host_specific_cfg() instead of
> kvm_utils.get_test_list().  I think 'parse_host_specific_cfg' better
> describes what the function does.  I also think it should take
> 'kvm_address_pools.cfg' as a parameter so we don't have to hardcode it
> into the function.
> 
> 
> Please see additional comments below.
> 
> 
> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
> 
> > As pointed out before, the KVM reference control files
> > could use a little clean up. This patch implements severe
> > cleanup of the main control file by:
> > 
> > * Refactoring the code present there, moving it to the
> > kvm_utils.py library
> > * Treat the build test exactly the same way as other
> > tests, moving the config stuff that used to be in the
> > control file realm out to its own configuration file,
> > for the sake of consistency.
> > * Since a frequent complaint amongst users is that by
> > default the build test will build from release, and
> > that build might be unsuccessful depending on the system
> > being used (older kernel versions and other sorts of
> > stuff), change the default build method to 'noinstall'.
> > 
> > This way the control file becomes way shorter, fairly
> > well organized, and we have a consistent configuration
> > schema across the board, based on configuration files.
> > 
> > If people are OK with this change, final path will
> > change the control.parallel file as well.
> > 
> > Important: For the default control file, I removed
> > the part of the code that parses new strings for the
> > test configuration because I believe those
> > modifications belong to the test config files, they
> > could be just commented. For cases where keeping
> > all stuff inside the config file is not possible,
> > appropriate code will be added (in other example
> > control files).
> > 
> > Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
> > ---
> >  client/tests/kvm/control              |  203
> > +++------------------------------
> >  client/tests/kvm/kvm_build.cfg.sample |   82 +++++++++++++
> >  client/tests/kvm/kvm_utils.py         |  112 ++++++++++++++++++
> >  3 files changed, 211 insertions(+), 186 deletions(-)
> >  create mode 100644 client/tests/kvm/kvm_build.cfg.sample
> > 
> > diff --git a/client/tests/kvm/control b/client/tests/kvm/control
> > index 431baf9..7acec14 100644
> > --- a/client/tests/kvm/control
> > +++ b/client/tests/kvm/control
> > @@ -20,194 +20,25 @@ KVM (both kernelspace and userspace) code.
> >  For online docs, please refer to
> > http://www.linux-kvm.org/page/KVM-Autotest
> >  """
> >  
> > +import sys, os, logging
> > +# Add the KVM tests dir to the python path
> > +kvm_test_dir = os.path.join(os.environ['AUTODIR'],'tests/kvm')
> > +sys.path.append(kvm_test_dir)
> > +import kvm_utils, kvm_config
> >  
> > -import sys, os
> >  
> > -#-----------------------------------------------------------------------------
> > -# set English environment (command output might be localized, need to
> > be safe)
> > -#-----------------------------------------------------------------------------
> > -os.environ['LANG'] = 'en_US.UTF-8'
> > +## Prepare the environment for the KVM test.
> > +kvm_utils.prepare_test_run(basedir=kvm_test_dir,
> > +                           rootdir='/tmp/kvm_autotest_root')
> >  
> > -#---------------------------------------------------------
> > -# Enable modules import from current directory (tests/kvm)
> > -#---------------------------------------------------------
> > -pwd = os.path.join(os.environ['AUTODIR'],'tests/kvm')
> > -sys.path.append(pwd)
> > +cfg_names = ['kvm_build.cfg', 'kvm_tests.cfg']
> >  
> > -# ------------------------
> > -# create required symlinks
> > -# ------------------------
> > -# When dispatching tests from autotest-server the links we need do
> > not exist on
> > -# the host (the client). The following lines create those symlinks.
> > Change
> > -# 'rootdir' here and/or mount appropriate directories in it.
> > -#
> > -# When dispatching tests on local host (client mode) one can either
> > setup kvm
> > -# links, or same as server mode use rootdir and set all appropriate
> > links and
> > -# mount-points there. For example, guest installation tests need to
> > know where
> > -# to find the iso-files.
> > -#
> > -# We create the links only if not already exist, so if one already
> > set up the
> > -# links for client/local run we do not touch the links.
> > -rootdir='/tmp/kvm_autotest_root'
> > -iso=os.path.join(rootdir, 'iso')
> > -images=os.path.join(rootdir, 'images')
> > -qemu=os.path.join(rootdir, 'qemu')
> > -qemu_img=os.path.join(rootdir, 'qemu-img')
> > +for cfg_name in cfg_names:
> > +    cfg_path = os.path.join(kvm_test_dir, cfg_name)
> > +    cfg = kvm_config.config(cfg_path)
> > +    list = kvm_utils.get_test_list(cfg, cfg_name, kvm_test_dir)
> > +    logging.info("Running test set defined on config file %s",
> > cfg_name)
> > +    kvm_utils.run_tests(list, job)
> 
> I think we should unroll this loop and allow users to modify the main
> and/or build configurations here.  For example:
> 
> build_cfg_path = os.path.join(kvm_test_dir, "kvm_build.cfg")
> test_cfg_path = os.path.join(kvm_test_dir, "kvm_tests.cfg")
> open(build_cfg_path, 'a').close()
> open(test_cfg_path, 'a').close()
> 
> cfg = kvm_config.config(build_cfg_path)
> cfg.parse_string("""
> # Make any desired changes to the build configuration here.  For example:
> # release_tag = 84
> """)
> if not kvm_utils.run_tests(cfg.get_list(), job.run_test):
>     sys.exit(1)
> 
> cfg = kvm_config.config(test_cfg_path)
> cfg.parse_string("""
> # Make any desired changes to the test configuration here.  For example:
> # display = sdl
> # install|setup: timeout_multiplier = 3
> """)
> kvm_utils.parse_host_specific_cfg(cfg, os.path.join(kvm_test_dir,
>                                                     "kvm_address_pools.cfg"))
> kvm_utils.run_tests(cfg.get_list(), job.run_test)
> 
> Alternatively, instead of creating the files in case they don't exit,
> we can put the code that parses and runs tests inside 'if' statements.
> I'm not sure which option is shorter.
> 
> >  
> > -
> > -def link_if_not_exist(ldir, target, link_name):
> > -    t = target
> > -    l = os.path.join(ldir, link_name)
> > -    if not os.path.exists(l):
> > -        os.system('ln -s %s %s' % (t, l))
> > -
> > -# Create links only if not already exist
> > -link_if_not_exist(pwd, '../../', 'autotest')
> > -link_if_not_exist(pwd, iso, 'isos')
> > -link_if_not_exist(pwd, images, 'images')
> > -link_if_not_exist(pwd, qemu, 'qemu')
> > -link_if_not_exist(pwd, qemu_img, 'qemu-img')
> > -
> > -# --------------------------------------------------------
> > -# Params that will be passed to the KVM install/build test
> > -# --------------------------------------------------------
> > -params = {
> > -    "name": "build",
> > -    "shortname": "build",
> > -    "type": "build",
> > -    "mode": "release",
> > -    #"mode": "snapshot",
> > -    #"mode": "localtar",
> > -    #"mode": "localsrc",
> > -    #"mode": "git",
> > -    #"mode": "noinstall",
> > -    #"mode": "koji",
> > -
> > -    ## Are we going to load modules built by this test?
> > -    ## Defaults to 'yes', so if you are going to provide only
> > userspace code to
> > -    ## be built by this test, please set load_modules to 'no', and
> > make sure
> > -    ## the kvm and kvm-[vendor] module is already loaded by the time
> > you start
> > -    ## it.
> > -    "load_modules": "no",
> > -
> > -    ## Install from a kvm release ("mode": "release"). You can
> > optionally
> > -    ## specify a release tag. If you omit it, the test will get the
> > latest
> > -    ## release tag available.
> > -    #"release_tag": '84',
> > -    "release_dir": 'http://downloads.sourceforge.net/project/kvm/',
> > -    # This is the place that contains the sourceforge project list of
> > files
> > -    "release_listing": 'http://sourceforge.net/projects/kvm/files/',
> > -
> > -    ## Install from a kvm snapshot location ("mode": "snapshot"). You
> > can
> > -    ## optionally specify a snapshot date. If you omit it, the test
> > will get
> > -    ## yesterday's snapshot.
> > -    #"snapshot_date": '20090712'
> > -    #"snapshot_dir": 'http://foo.org/kvm-snapshots/',
> > -
> > -    ## Install from a tarball ("mode": "localtar")
> > -    #"tarball": "/tmp/kvm-84.tar.gz",
> > -
> > -    ## Install from a local source code dir ("mode": "localsrc")
> > -    #"srcdir": "/path/to/source-dir"
> > -
> > -    ## Install from koji build server ("mode": "koji")
> > -    ## Koji is the Fedora Project buildserver. It is possible to
> > install
> > -    ## packages right from Koji if you provide a release tag or a
> > build.
> > -    ## Tag (if available)
> > -    #"koji_tag": 'dist-f11',
> > -    ## Build (if available, is going to override tag).
> > -    #"koji_build": 'qemu-0.10-16.fc11',
> > -    ## Command to interact with the build server
> > -    #"koji_cmd": '/usr/bin/koji',
> > -    ## The name of the source package that's being built
> > -    #"src_pkg": 'qemu',
> > -    ## Name of the rpms we need installed
> > -    #"pkg_list": ['qemu-kvm', 'qemu-kvm-tools', 'qemu-system-x86',
> > 'qemu-common', 'qemu-img'],
> > -    ## Paths of the qemu relevant executables that should be checked
> > -    #"qemu_bin_paths": ['/usr/bin/qemu-kvm', '/usr/bin/qemu-img'],
> > -
> > -    ## Install from git ("mode": "git")
> > -    ## If you provide only "git_repo" and "user_git_repo", the build
> > test
> > -    ## will assume it will perform all build from the userspace dir,
> > building
> > -    ## modules trough make -C kernel LINUX=%s sync. As of today
> > (07-13-2009)
> > -    ## we need 3 git repos, "git_repo" (linux sources),
> > "user_git_repo" and 
> > -    ## "kmod_repo" to build KVM userspace + kernel modules.
> > -    #"git_repo":
> > 'git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git',
> > -    #"kernel_branch": 'kernel_branch_name',
> > -    #"kernel_lbranch": 'kernel_lbranch_name',
> > -    #"kernel_tag": 'kernel_tag_name',
> > -    #"user_git_repo":
> > 'git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git',
> > -    #"user_branch": 'user_branch_name',
> > -    #"user_lbranch": 'user_lbranch_name',
> > -    #"user_tag": 'user_tag_name',
> > -    #"kmod_repo":
> > 'git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git',
> > -    #"kmod_branch": 'kmod_branch_name',
> > -    #"kmod_lbranch": 'kmod_lbranch_name',
> > -    #"kmod_tag": 'kmod_tag_name',
> > -}
> > -
> > -# If you don't want to execute the build stage, just use 'noinstall'
> > as the
> > -# install type. If you run the tests from autotest-server, make sure
> > that
> > -# /tmp/kvm-autotest-root/qemu is a link to your existing executable.
> > Note that
> > -# if kvm_install is chose to run, it overwrites existing qemu and
> > qemu-img
> > -# links to point to the newly built executables.
> > -r = True
> > -r = job.run_test("kvm", params=params, tag=params.get("shortname"))
> > -if not r:
> > -    print 'kvm_installation failed ... exiting'
> > -    sys.exit(1)
> > -
> > -# ----------------------------------------------------------
> > -# Get test set (dictionary list) from the configuration file
> > -# ----------------------------------------------------------
> > -import kvm_config
> > -
> > -filename = os.path.join(pwd, "kvm_tests.cfg")
> > -cfg = kvm_config.config(filename)
> > -
> > -# If desirable, make changes to the test configuration here.  For
> > example:
> > -# cfg.parse_string("install|setup: timeout_multiplier = 2")
> > -# cfg.parse_string("only fc8_quick")
> > -# cfg.parse_string("display = sdl")
> > -
> > -filename = os.path.join(pwd, "kvm_address_pools.cfg")
> > -if os.path.exists(filename):
> > -    cfg.parse_file(filename)
> > -    hostname = os.uname()[1].split(".")[0]
> > -    if cfg.filter("^" + hostname):
> > -        cfg.parse_string("only ^%s" % hostname)
> > -    else:
> > -        cfg.parse_string("only ^default_host")
> > -
> > -list = cfg.get_list()
> > -
> > -
> > -# -------------
> > -# Run the tests
> > -# -------------
> > -status_dict = {}
> > -
> > -for dict in list:
> > -    if dict.get("skip") == "yes":
> > -        continue
> > -    dependencies_satisfied = True
> > -    for dep in dict.get("depend"):
> > -        for test_name in status_dict.keys():
> > -            if not dep in test_name:
> > -                continue
> > -            if not status_dict[test_name]:
> > -                dependencies_satisfied = False
> > -                break
> > -    if dependencies_satisfied:
> > -        test_iterations = int(dict.get("iterations", 1))
> > -        current_status = job.run_test("kvm", params=dict,
> > -                                      tag=dict.get("shortname"),
> > -                                      iterations=test_iterations)
> > -    else:
> > -        current_status = False
> > -    status_dict[dict.get("name")] = current_status
> > -
> > -# create the html report in result dir
> > -reporter = os.path.join(pwd, 'html_report.py')
> > -html_file = os.path.join(job.resultdir,'results.html')
> > -os.system('%s -r %s -f %s -R' % (reporter, job.resultdir,
> > html_file))
> > +## Generate a nice HTML report inside results_dir
> > +kvm_utils.create_report(kvm_test_dir, job.resultsdir)
> > diff --git a/client/tests/kvm/kvm_build.cfg.sample
> > b/client/tests/kvm/kvm_build.cfg.sample
> > new file mode 100644
> > index 0000000..692a747
> > --- /dev/null
> > +++ b/client/tests/kvm/kvm_build.cfg.sample
> > @@ -0,0 +1,82 @@
> > +# This configuration file holds the KVM build test config
> > parameters.
> > +# The default is noinstall (won't attempt to build KVM), so if you
> > stick with it
> > +# please make sure:
> > +# 1) You have setup symbolic links to qemu and qemu-img binaries on
> > the
> > +#    KVM test dir.
> > +# 2) The appropriate KVM modules are already loaded on your machine.
> > +
> > +variants:
> > +    - build:
> > +        type = build
> > +        # Load modules built/installed by the build test?
> > +        load_modules = no
> > +        variants:
> > +            - noinstall:
> > +                mode = noinstall
> > +            - release:
> > +                mode = release
> > +                ## Install from a kvm release. You can optionally
> > specify
> > +                ## a release tag. If you omit it, the build test will
> > get
> > +                ## the latest release tag available at that moment.
> > +                # release_tag = 84
> > +                release_dir =
> > http://downloads.sourceforge.net/project/kvm/
> > +                release_listing =
> > http://sourceforge.net/projects/kvm/files/
> > +            - snapshot:
> > +                mode = snapshot
> > +                ## Install from a kvm snapshot location. You can
> > optionally
> > +                ## specify a snapshot date. If you omit it, the test
> > will get
> > +                ## yesterday's snapshot.
> > +                # snapshot_date = 20090712
> > +                snapshot_dir = http://foo.org/kvm-snapshots/
> > +            - localtar:
> > +                mode = localtar
> > +                ## Install from tarball located on the host's
> > filesystem.
> > +                tarball = /tmp/kvm-84.tar.gz
> > +            - localsrc:
> > +                mode = localsrc
> > +                ## Install from tarball located on the host's
> > filesystem.
> > +                srcdir = /tmp/kvm-84
> > +            - git:
> > +                mode = git
> > +                ## Install KVM from git repositories.
> > +                ## If you provide only "git_repo" and
> > "user_git_repo", the
> > +                ## build test will assume it will perform all build
> > from the
> > +                ## userspace dir, building modules trough 
> > +                ## make -C kernel LINUX=%s sync. As of today
> > (07-13-2009)
> > +                ## upstream needs 3 git repos:
> > +                ## * git_repo (linux sources)
> > +                ## * user_git_repo (qemu sources)
> > +                ## * kmod_repo" to build KVM userspace + kernel
> > modules.
> > +                git_repo =
> > git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git
> > +                # kernel_branch = kernel_branch_name
> > +                # kernel_lbranch = kernel_lbranch_name
> > +                # kernel_tag = kernel_tag_name
> > +                user_git_repo =
> > git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
> > +                # user_branch = user_branch_name
> > +                # user_lbranch = user_lbranch_name
> > +                # user_tag = user_tag_name
> > +                kmod_repo =
> > git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git
> > +                # kmod_branch = kmod_branch_name
> > +                # kmod_lbranch = kmod_lbranch_name
> > +                # kmod_tag = kmod_tag_name
> > +            - koji:
> > +                mode = koji
> > +                ## Install KVM from koji (Fedora build server)
> > +                ## It is possible to install packages right from Koji
> > if you
> > +                ## provide a release tag or a build.
> > +                ## Tag (if available)
> > +                koji_tag = dist-f11
> > +                ## Build (if available, is going to override tag).
> > +                koji_build = qemu-0.10-16.fc11
> > +                ## Command to interact with the build server
> > +                koji_cmd = /usr/bin/koji
> > +                ## The name of the source package that will be built
> > +                src_pkg = qemu
> > +                ## Name of the rpms we need installed
> > +                pkg_list = ['qemu-kvm', 'qemu-kvm-tools',
> > 'qemu-system-x86', 'qemu-common', 'qemu-img']
> > +                ## Paths of the qemu relevant executables that should
> > be checked
> > +                qemu_bin_paths = ['/usr/bin/qemu-kvm',
> > '/usr/bin/qemu-img']
> > +
> > +
> > +# Choose your install method here
> > +only build.noinstall
> > \ No newline at end of file
> > diff --git a/client/tests/kvm/kvm_utils.py
> > b/client/tests/kvm/kvm_utils.py
> > index f72984a..90b3def 100644
> > --- a/client/tests/kvm/kvm_utils.py
> > +++ b/client/tests/kvm/kvm_utils.py
> > @@ -871,3 +871,115 @@ def unmap_url_cache(cachedir, url,
> > expected_hash, method="md5"):
> >          file_path = utils.unmap_url(cachedir, src, cachedir)
> >  
> >      return file_path
> > +
> > +
> > +def prepare_test_run(basedir, rootdir='/tmp/kvm_autotest_root'):
> > +    """
> > +    Prepare the environment for a KVM test run by linking some
> > previously
> > +    prepared directories to the test bindir.
> > +
> > +    @param basedir: Location of the KVM test directory
> > +    @param rootdir: Base directory that contains the KVM support
> > directory
> > +            structure:
> > +            * isos (holds guest install isos)
> > +            * images (will hold guest disk images)
> > +            * qemu (qemu binary)
> > +            * qemu-img (qemu-img binary)
> > +    """
> > +    # set English environment (command output might be localized,
> > +    # need to be safe)
> > +    os.environ['LANG'] = 'en_US.UTF-8'
> > +
> > +    # Symbolic links we desire to see in the KVM test dir
> > +    list_symlinks = ['isos', 'images', 'qemu', 'qemu-img']
> > +    for symlink_basename in list_symlinks:
> > +        link_src = os.path.join(rootdir, symlink_basename)
> > +        link_dst = os.path.join(basedir, symlink_basename)
> > +        if not os.path.islink(link_dst):
> > +            os.symlink(link_src, link_dst)
> > +        # Testing whether our symlinks are good and in shape
> > +        link_good = os.path.exists(os.readlink(link_dst))
> > +        if not link_good:
> > +            if symlink_basename in ['isos', 'images']:
> > +                logging.critical("Mandatory directory %s is a broken
> > symlink. "
> > +                                 "This job will be aborted so you can
> > fix the "
> > +                                 "problem. Make sure %s points to a
> > valid "
> > +                                 "directory before proceeding.",
> > +                                 symlink_basename, link_dst)
> > +                raise error.JobError("Mandatory directory %s is a
> > broken "
> > +                                     "symlink. Please correct it." 
> > %
> > +                                     (link_dst))
> > +            else:
> > +                logging.warning("Symlink %s is broken. You might have
> > problems "
> > +                                "later in the test if the KVM build
> > test mode "
> > +                                "selected is 'noinstall' (default).
> > ", link_dst)
> 
> As mentioned before I'm not sure we need this function at all.
> If the user has to modify kvm_tests.cfg.sample anyway, he/she will see our
> comments regarding the qemu/images/isos/steps paths.
> 
> > +def get_test_list(cfg, cfg_name, base_dir):
> > +    """
> > +    Return a list of test dictionaries composed by the KVM
> > configuration system.
> > +    This might look like a delegate of the get_list() method of the
> > +    configuration, but in fact we might want to do custom config
> > processing
> > +    depending on the config file name provided.
> > +
> > +    @param cfg: KVM test configuration object
> > +    @param cfg_name: Basename of the configuration file that
> > generated the
> > +            configuration object
> > +    @param base_dir: Directory that holds the config files
> > +
> > +    @return: A list with test parameters dictionaries.
> > +    """
> > +    if cfg_name == 'kvm_tests.cfg':
> > +        pools = os.path.join(base_dir, "kvm_address_pools.cfg")
> > +        if os.path.exists(pools):
> > +            cfg.parse_file(pools)
> > +            hostname = os.uname()[1].split(".")[0]
> > +            if cfg.filter("^" + hostname):
> > +                cfg.parse_string("only ^%s" % hostname)
> > +            else:
> > +                cfg.parse_string("only ^default_host")
> > +
> > +    return cfg.get_list()
> 
> Already commented on this one above.
> 
> > +def run_tests(test_list, job):
> > +    """
> > +    Runs the sequence of KVM tests based on the list of dictionaries
> > +    generated by the configuration system, handling dependencies.
> > +
> > +    @param test_list: List with all dictionary test parameters.
> > +    @param job: Autotest job object.
> > +    """
> > +    status_dict = {}
> > +
> > +    for dict in test_list:
> > +        if dict.get("skip") == "yes":
> > +            continue
> > +        dependencies_satisfied = True
> > +        for dep in dict.get("depend"):
> > +            for test_name in status_dict.keys():
> > +                if not dep in test_name:
> > +                    continue
> > +                if not status_dict[test_name]:
> > +                    dependencies_satisfied = False
> > +                    break
> > +        if dependencies_satisfied:
> > +            test_iterations = int(dict.get("iterations", 1))
> > +            current_status = job.run_test("kvm", params=dict,
> > +                                          tag=dict.get("shortname"),
> > +                                         
> > iterations=test_iterations)
> > +        else:
> > +            current_status = False
> > +        status_dict[dict.get("name")] = current_status
> 
> Already commented on this one above.
> 
> > +def create_report(report_dir, results_dir):
> > +    """
> > +    Creates a neatly arranged HTML results report in the results
> > dir.
> > +
> > +    @param report_dir: Directory where the report script is located.
> > +    @param results_dir: Directory where the results will be output.
> > +    """
> > +    reporter = os.path.join(report_dir, 'html_report.py')
> > +    html_file = os.path.join(results_dir, 'results.html')
> > +    os.system('%s -r %s -f %s -R' % (reporter, results_dir,
> > html_file))
> > -- 
> > 1.6.2.5

Re: [PATCH] [RFC] KVM test: Major control file cleanup

[Michael Goldish]

----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:

> One thing that just occurred to me is, if we have a test config
> "library" as you said, it's perfectly possible to put the actual test
> set definitions and other config files inside the control file as
> strings. This way one can control configuration inside the control
> file,
> making it more convenient for usage, let's say, inside the autotest
> web
> interface. Since the control file would be reduced in size, the
> configuration snippets being in the control file would not be a huge
> problem, while keeping the original autotest philosophy of keeping
> stuff inside the control file... What do you think?

Sounds great, except it won't allow you to debug your configuration
using kvm_config.py.  So the question now is what's more important --
the ability to debug or ease of use when running from the server.

> On Wed, 2009-10-28 at 05:48 -0400, Michael Goldish wrote:
> > I think this patch is a good idea.  Comments:
> > 
> > - If we've decided to put all the configuration in config files
> > (rather than the control file) let's split up the main one into
> > kvm_test_library.cfg and kvm_tests.cfg.sample.  The latter should
> > look like this:
> > 
> > include kvm_test_library.cfg
> > include kvm_cdkeys.cfg
> > 
> > variants:
> >     - fc8_quick:
> >         ...
> >     - other_test_set:
> >         ...
> > 
> > only fc8_quick
> > 
> > ... and the former should contain everything else.
> > 
> > - I think we should allow the user to modify his/her configuration
> > in the control file using parse_string().  This is actually useful,
> > especially when running from the server.  (At least I use it.)
> > 
> > - If kvm_build.cfg or kvm_tests.cfg are missing, we should continue
> > without them, or create them (blank), instead of failing due to an
> > unhandled exception raised by the config parser.  This should be
> > true at least for kvm_build.cfg, which isn't always required for
> the
> > tests to run.
> > 
> > - We should consider removing hardcoded references to
> > /tmp/kvm_autotest_root/.  We use that path locally in TLV but
> that's
> > not a good reason to put it upstream IMO.
> > We should also stop creating symlinks to that path.
> > Instead we can allow more flexibility using config files.
> > For example, we can put this in kvm_tests.cfg.sample, between the
> > 'include' and the 'variants':
> > 
> > # Uncomment and modify the following lines if you wish to modify
> > # the paths of the image files, ISO files, step files or qemu
> binaries.
> > # If you keep these lines commented, all mentioned paths are
> expected
> > # to be found in the KVM test dir (tests/kvm/).  For example, all
> > # image files are expected under tests/kvm/images/.
> > 
> > # qemu_binary ?<= /tmp/kvm_autotest_root/
> > # qemu_img_binary ?<= /tmp/kvm_autotest_root/
> > # image_name.* ?<= /tmp/kvm_autotest_root/
> > # cdrom.* ?<= /tmp/kvm_autotest_root/
> > # steps ?<= /tmp/kvm_autotest_root/
> > 
> > The ?<= operator prepends strings to keys that match a regex, so
> > 'image_name.* ?<= something/' will prepend 'something/' to
> image_name,
> > image_name_vm2 and so on.
> > In kvm_test_library.cfg (currently kvm_tests.cfg.sample) all
> > 'image_name' parameters, for example, begin with 'images/', which
> means
> > all image files should be under tests/kvm/images.
> > If we prepend /tmp/kvm_autotest_root/ to that, all image files will
> be
> > expected under /tmp/kvm_autotest_root/images/.
> > 
> > Maybe we should leave these lines uncommented by default, in case
> anyone
> > uses /tmp/kvm_autotest_root and relies on the control file creating
> the
> > symlinks.
> > 
> > - kvm_utils.run_tests() should return True if all tests passed and
> > otherwise False.  We need this in order to quit if the build
> test(s)
> > failed.
> > 
> > - I suggest we use kvm_utils.parse_host_specific_cfg() instead of
> > kvm_utils.get_test_list().  I think 'parse_host_specific_cfg'
> better
> > describes what the function does.  I also think it should take
> > 'kvm_address_pools.cfg' as a parameter so we don't have to hardcode
> it
> > into the function.
> > 
> > 
> > Please see additional comments below.
> > 
> > 
> > ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
> > 
> > > As pointed out before, the KVM reference control files
> > > could use a little clean up. This patch implements severe
> > > cleanup of the main control file by:
> > > 
> > > * Refactoring the code present there, moving it to the
> > > kvm_utils.py library
> > > * Treat the build test exactly the same way as other
> > > tests, moving the config stuff that used to be in the
> > > control file realm out to its own configuration file,
> > > for the sake of consistency.
> > > * Since a frequent complaint amongst users is that by
> > > default the build test will build from release, and
> > > that build might be unsuccessful depending on the system
> > > being used (older kernel versions and other sorts of
> > > stuff), change the default build method to 'noinstall'.
> > > 
> > > This way the control file becomes way shorter, fairly
> > > well organized, and we have a consistent configuration
> > > schema across the board, based on configuration files.
> > > 
> > > If people are OK with this change, final path will
> > > change the control.parallel file as well.
> > > 
> > > Important: For the default control file, I removed
> > > the part of the code that parses new strings for the
> > > test configuration because I believe those
> > > modifications belong to the test config files, they
> > > could be just commented. For cases where keeping
> > > all stuff inside the config file is not possible,
> > > appropriate code will be added (in other example
> > > control files).
> > > 
> > > Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
> > > ---
> > >  client/tests/kvm/control              |  203
> > > +++------------------------------
> > >  client/tests/kvm/kvm_build.cfg.sample |   82 +++++++++++++
> > >  client/tests/kvm/kvm_utils.py         |  112 ++++++++++++++++++
> > >  3 files changed, 211 insertions(+), 186 deletions(-)
> > >  create mode 100644 client/tests/kvm/kvm_build.cfg.sample
> > > 
> > > diff --git a/client/tests/kvm/control b/client/tests/kvm/control
> > > index 431baf9..7acec14 100644
> > > --- a/client/tests/kvm/control
> > > +++ b/client/tests/kvm/control
> > > @@ -20,194 +20,25 @@ KVM (both kernelspace and userspace) code.
> > >  For online docs, please refer to
> > > http://www.linux-kvm.org/page/KVM-Autotest
> > >  """
> > >  
> > > +import sys, os, logging
> > > +# Add the KVM tests dir to the python path
> > > +kvm_test_dir = os.path.join(os.environ['AUTODIR'],'tests/kvm')
> > > +sys.path.append(kvm_test_dir)
> > > +import kvm_utils, kvm_config
> > >  
> > > -import sys, os
> > >  
> > >
> -#-----------------------------------------------------------------------------
> > > -# set English environment (command output might be localized,
> need to
> > > be safe)
> > >
> -#-----------------------------------------------------------------------------
> > > -os.environ['LANG'] = 'en_US.UTF-8'
> > > +## Prepare the environment for the KVM test.
> > > +kvm_utils.prepare_test_run(basedir=kvm_test_dir,
> > > +                           rootdir='/tmp/kvm_autotest_root')
> > >  
> > > -#---------------------------------------------------------
> > > -# Enable modules import from current directory (tests/kvm)
> > > -#---------------------------------------------------------
> > > -pwd = os.path.join(os.environ['AUTODIR'],'tests/kvm')
> > > -sys.path.append(pwd)
> > > +cfg_names = ['kvm_build.cfg', 'kvm_tests.cfg']
> > >  
> > > -# ------------------------
> > > -# create required symlinks
> > > -# ------------------------
> > > -# When dispatching tests from autotest-server the links we need
> do
> > > not exist on
> > > -# the host (the client). The following lines create those
> symlinks.
> > > Change
> > > -# 'rootdir' here and/or mount appropriate directories in it.
> > > -#
> > > -# When dispatching tests on local host (client mode) one can
> either
> > > setup kvm
> > > -# links, or same as server mode use rootdir and set all
> appropriate
> > > links and
> > > -# mount-points there. For example, guest installation tests need
> to
> > > know where
> > > -# to find the iso-files.
> > > -#
> > > -# We create the links only if not already exist, so if one
> already
> > > set up the
> > > -# links for client/local run we do not touch the links.
> > > -rootdir='/tmp/kvm_autotest_root'
> > > -iso=os.path.join(rootdir, 'iso')
> > > -images=os.path.join(rootdir, 'images')
> > > -qemu=os.path.join(rootdir, 'qemu')
> > > -qemu_img=os.path.join(rootdir, 'qemu-img')
> > > +for cfg_name in cfg_names:
> > > +    cfg_path = os.path.join(kvm_test_dir, cfg_name)
> > > +    cfg = kvm_config.config(cfg_path)
> > > +    list = kvm_utils.get_test_list(cfg, cfg_name, kvm_test_dir)
> > > +    logging.info("Running test set defined on config file %s",
> > > cfg_name)
> > > +    kvm_utils.run_tests(list, job)
> > 
> > I think we should unroll this loop and allow users to modify the
> main
> > and/or build configurations here.  For example:
> > 
> > build_cfg_path = os.path.join(kvm_test_dir, "kvm_build.cfg")
> > test_cfg_path = os.path.join(kvm_test_dir, "kvm_tests.cfg")
> > open(build_cfg_path, 'a').close()
> > open(test_cfg_path, 'a').close()
> > 
> > cfg = kvm_config.config(build_cfg_path)
> > cfg.parse_string("""
> > # Make any desired changes to the build configuration here.  For
> example:
> > # release_tag = 84
> > """)
> > if not kvm_utils.run_tests(cfg.get_list(), job.run_test):
> >     sys.exit(1)
> > 
> > cfg = kvm_config.config(test_cfg_path)
> > cfg.parse_string("""
> > # Make any desired changes to the test configuration here.  For
> example:
> > # display = sdl
> > # install|setup: timeout_multiplier = 3
> > """)
> > kvm_utils.parse_host_specific_cfg(cfg, os.path.join(kvm_test_dir,
> >                                                    
> "kvm_address_pools.cfg"))
> > kvm_utils.run_tests(cfg.get_list(), job.run_test)
> > 
> > Alternatively, instead of creating the files in case they don't
> exit,
> > we can put the code that parses and runs tests inside 'if'
> statements.
> > I'm not sure which option is shorter.
> > 
> > >  
> > > -
> > > -def link_if_not_exist(ldir, target, link_name):
> > > -    t = target
> > > -    l = os.path.join(ldir, link_name)
> > > -    if not os.path.exists(l):
> > > -        os.system('ln -s %s %s' % (t, l))
> > > -
> > > -# Create links only if not already exist
> > > -link_if_not_exist(pwd, '../../', 'autotest')
> > > -link_if_not_exist(pwd, iso, 'isos')
> > > -link_if_not_exist(pwd, images, 'images')
> > > -link_if_not_exist(pwd, qemu, 'qemu')
> > > -link_if_not_exist(pwd, qemu_img, 'qemu-img')
> > > -
> > > -# --------------------------------------------------------
> > > -# Params that will be passed to the KVM install/build test
> > > -# --------------------------------------------------------
> > > -params = {
> > > -    "name": "build",
> > > -    "shortname": "build",
> > > -    "type": "build",
> > > -    "mode": "release",
> > > -    #"mode": "snapshot",
> > > -    #"mode": "localtar",
> > > -    #"mode": "localsrc",
> > > -    #"mode": "git",
> > > -    #"mode": "noinstall",
> > > -    #"mode": "koji",
> > > -
> > > -    ## Are we going to load modules built by this test?
> > > -    ## Defaults to 'yes', so if you are going to provide only
> > > userspace code to
> > > -    ## be built by this test, please set load_modules to 'no',
> and
> > > make sure
> > > -    ## the kvm and kvm-[vendor] module is already loaded by the
> time
> > > you start
> > > -    ## it.
> > > -    "load_modules": "no",
> > > -
> > > -    ## Install from a kvm release ("mode": "release"). You can
> > > optionally
> > > -    ## specify a release tag. If you omit it, the test will get
> the
> > > latest
> > > -    ## release tag available.
> > > -    #"release_tag": '84',
> > > -    "release_dir":
> 'http://downloads.sourceforge.net/project/kvm/',
> > > -    # This is the place that contains the sourceforge project
> list of
> > > files
> > > -    "release_listing":
> 'http://sourceforge.net/projects/kvm/files/',
> > > -
> > > -    ## Install from a kvm snapshot location ("mode": "snapshot").
> You
> > > can
> > > -    ## optionally specify a snapshot date. If you omit it, the
> test
> > > will get
> > > -    ## yesterday's snapshot.
> > > -    #"snapshot_date": '20090712'
> > > -    #"snapshot_dir": 'http://foo.org/kvm-snapshots/',
> > > -
> > > -    ## Install from a tarball ("mode": "localtar")
> > > -    #"tarball": "/tmp/kvm-84.tar.gz",
> > > -
> > > -    ## Install from a local source code dir ("mode": "localsrc")
> > > -    #"srcdir": "/path/to/source-dir"
> > > -
> > > -    ## Install from koji build server ("mode": "koji")
> > > -    ## Koji is the Fedora Project buildserver. It is possible to
> > > install
> > > -    ## packages right from Koji if you provide a release tag or
> a
> > > build.
> > > -    ## Tag (if available)
> > > -    #"koji_tag": 'dist-f11',
> > > -    ## Build (if available, is going to override tag).
> > > -    #"koji_build": 'qemu-0.10-16.fc11',
> > > -    ## Command to interact with the build server
> > > -    #"koji_cmd": '/usr/bin/koji',
> > > -    ## The name of the source package that's being built
> > > -    #"src_pkg": 'qemu',
> > > -    ## Name of the rpms we need installed
> > > -    #"pkg_list": ['qemu-kvm', 'qemu-kvm-tools',
> 'qemu-system-x86',
> > > 'qemu-common', 'qemu-img'],
> > > -    ## Paths of the qemu relevant executables that should be
> checked
> > > -    #"qemu_bin_paths": ['/usr/bin/qemu-kvm',
> '/usr/bin/qemu-img'],
> > > -
> > > -    ## Install from git ("mode": "git")
> > > -    ## If you provide only "git_repo" and "user_git_repo", the
> build
> > > test
> > > -    ## will assume it will perform all build from the userspace
> dir,
> > > building
> > > -    ## modules trough make -C kernel LINUX=%s sync. As of today
> > > (07-13-2009)
> > > -    ## we need 3 git repos, "git_repo" (linux sources),
> > > "user_git_repo" and 
> > > -    ## "kmod_repo" to build KVM userspace + kernel modules.
> > > -    #"git_repo":
> > > 'git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git',
> > > -    #"kernel_branch": 'kernel_branch_name',
> > > -    #"kernel_lbranch": 'kernel_lbranch_name',
> > > -    #"kernel_tag": 'kernel_tag_name',
> > > -    #"user_git_repo":
> > > 'git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git',
> > > -    #"user_branch": 'user_branch_name',
> > > -    #"user_lbranch": 'user_lbranch_name',
> > > -    #"user_tag": 'user_tag_name',
> > > -    #"kmod_repo":
> > > 'git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git',
> > > -    #"kmod_branch": 'kmod_branch_name',
> > > -    #"kmod_lbranch": 'kmod_lbranch_name',
> > > -    #"kmod_tag": 'kmod_tag_name',
> > > -}
> > > -
> > > -# If you don't want to execute the build stage, just use
> 'noinstall'
> > > as the
> > > -# install type. If you run the tests from autotest-server, make
> sure
> > > that
> > > -# /tmp/kvm-autotest-root/qemu is a link to your existing
> executable.
> > > Note that
> > > -# if kvm_install is chose to run, it overwrites existing qemu
> and
> > > qemu-img
> > > -# links to point to the newly built executables.
> > > -r = True
> > > -r = job.run_test("kvm", params=params,
> tag=params.get("shortname"))
> > > -if not r:
> > > -    print 'kvm_installation failed ... exiting'
> > > -    sys.exit(1)
> > > -
> > > -# ----------------------------------------------------------
> > > -# Get test set (dictionary list) from the configuration file
> > > -# ----------------------------------------------------------
> > > -import kvm_config
> > > -
> > > -filename = os.path.join(pwd, "kvm_tests.cfg")
> > > -cfg = kvm_config.config(filename)
> > > -
> > > -# If desirable, make changes to the test configuration here. 
> For
> > > example:
> > > -# cfg.parse_string("install|setup: timeout_multiplier = 2")
> > > -# cfg.parse_string("only fc8_quick")
> > > -# cfg.parse_string("display = sdl")
> > > -
> > > -filename = os.path.join(pwd, "kvm_address_pools.cfg")
> > > -if os.path.exists(filename):
> > > -    cfg.parse_file(filename)
> > > -    hostname = os.uname()[1].split(".")[0]
> > > -    if cfg.filter("^" + hostname):
> > > -        cfg.parse_string("only ^%s" % hostname)
> > > -    else:
> > > -        cfg.parse_string("only ^default_host")
> > > -
> > > -list = cfg.get_list()
> > > -
> > > -
> > > -# -------------
> > > -# Run the tests
> > > -# -------------
> > > -status_dict = {}
> > > -
> > > -for dict in list:
> > > -    if dict.get("skip") == "yes":
> > > -        continue
> > > -    dependencies_satisfied = True
> > > -    for dep in dict.get("depend"):
> > > -        for test_name in status_dict.keys():
> > > -            if not dep in test_name:
> > > -                continue
> > > -            if not status_dict[test_name]:
> > > -                dependencies_satisfied = False
> > > -                break
> > > -    if dependencies_satisfied:
> > > -        test_iterations = int(dict.get("iterations", 1))
> > > -        current_status = job.run_test("kvm", params=dict,
> > > -                                      tag=dict.get("shortname"),
> > > -                                     
> iterations=test_iterations)
> > > -    else:
> > > -        current_status = False
> > > -    status_dict[dict.get("name")] = current_status
> > > -
> > > -# create the html report in result dir
> > > -reporter = os.path.join(pwd, 'html_report.py')
> > > -html_file = os.path.join(job.resultdir,'results.html')
> > > -os.system('%s -r %s -f %s -R' % (reporter, job.resultdir,
> > > html_file))
> > > +## Generate a nice HTML report inside results_dir
> > > +kvm_utils.create_report(kvm_test_dir, job.resultsdir)
> > > diff --git a/client/tests/kvm/kvm_build.cfg.sample
> > > b/client/tests/kvm/kvm_build.cfg.sample
> > > new file mode 100644
> > > index 0000000..692a747
> > > --- /dev/null
> > > +++ b/client/tests/kvm/kvm_build.cfg.sample
> > > @@ -0,0 +1,82 @@
> > > +# This configuration file holds the KVM build test config
> > > parameters.
> > > +# The default is noinstall (won't attempt to build KVM), so if
> you
> > > stick with it
> > > +# please make sure:
> > > +# 1) You have setup symbolic links to qemu and qemu-img binaries
> on
> > > the
> > > +#    KVM test dir.
> > > +# 2) The appropriate KVM modules are already loaded on your
> machine.
> > > +
> > > +variants:
> > > +    - build:
> > > +        type = build
> > > +        # Load modules built/installed by the build test?
> > > +        load_modules = no
> > > +        variants:
> > > +            - noinstall:
> > > +                mode = noinstall
> > > +            - release:
> > > +                mode = release
> > > +                ## Install from a kvm release. You can
> optionally
> > > specify
> > > +                ## a release tag. If you omit it, the build test
> will
> > > get
> > > +                ## the latest release tag available at that
> moment.
> > > +                # release_tag = 84
> > > +                release_dir =
> > > http://downloads.sourceforge.net/project/kvm/
> > > +                release_listing =
> > > http://sourceforge.net/projects/kvm/files/
> > > +            - snapshot:
> > > +                mode = snapshot
> > > +                ## Install from a kvm snapshot location. You can
> > > optionally
> > > +                ## specify a snapshot date. If you omit it, the
> test
> > > will get
> > > +                ## yesterday's snapshot.
> > > +                # snapshot_date = 20090712
> > > +                snapshot_dir = http://foo.org/kvm-snapshots/
> > > +            - localtar:
> > > +                mode = localtar
> > > +                ## Install from tarball located on the host's
> > > filesystem.
> > > +                tarball = /tmp/kvm-84.tar.gz
> > > +            - localsrc:
> > > +                mode = localsrc
> > > +                ## Install from tarball located on the host's
> > > filesystem.
> > > +                srcdir = /tmp/kvm-84
> > > +            - git:
> > > +                mode = git
> > > +                ## Install KVM from git repositories.
> > > +                ## If you provide only "git_repo" and
> > > "user_git_repo", the
> > > +                ## build test will assume it will perform all
> build
> > > from the
> > > +                ## userspace dir, building modules trough 
> > > +                ## make -C kernel LINUX=%s sync. As of today
> > > (07-13-2009)
> > > +                ## upstream needs 3 git repos:
> > > +                ## * git_repo (linux sources)
> > > +                ## * user_git_repo (qemu sources)
> > > +                ## * kmod_repo" to build KVM userspace + kernel
> > > modules.
> > > +                git_repo =
> > > git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git
> > > +                # kernel_branch = kernel_branch_name
> > > +                # kernel_lbranch = kernel_lbranch_name
> > > +                # kernel_tag = kernel_tag_name
> > > +                user_git_repo =
> > > git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
> > > +                # user_branch = user_branch_name
> > > +                # user_lbranch = user_lbranch_name
> > > +                # user_tag = user_tag_name
> > > +                kmod_repo =
> > > git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git
> > > +                # kmod_branch = kmod_branch_name
> > > +                # kmod_lbranch = kmod_lbranch_name
> > > +                # kmod_tag = kmod_tag_name
> > > +            - koji:
> > > +                mode = koji
> > > +                ## Install KVM from koji (Fedora build server)
> > > +                ## It is possible to install packages right from
> Koji
> > > if you
> > > +                ## provide a release tag or a build.
> > > +                ## Tag (if available)
> > > +                koji_tag = dist-f11
> > > +                ## Build (if available, is going to override
> tag).
> > > +                koji_build = qemu-0.10-16.fc11
> > > +                ## Command to interact with the build server
> > > +                koji_cmd = /usr/bin/koji
> > > +                ## The name of the source package that will be
> built
> > > +                src_pkg = qemu
> > > +                ## Name of the rpms we need installed
> > > +                pkg_list = ['qemu-kvm', 'qemu-kvm-tools',
> > > 'qemu-system-x86', 'qemu-common', 'qemu-img']
> > > +                ## Paths of the qemu relevant executables that
> should
> > > be checked
> > > +                qemu_bin_paths = ['/usr/bin/qemu-kvm',
> > > '/usr/bin/qemu-img']
> > > +
> > > +
> > > +# Choose your install method here
> > > +only build.noinstall
> > > \ No newline at end of file
> > > diff --git a/client/tests/kvm/kvm_utils.py
> > > b/client/tests/kvm/kvm_utils.py
> > > index f72984a..90b3def 100644
> > > --- a/client/tests/kvm/kvm_utils.py
> > > +++ b/client/tests/kvm/kvm_utils.py
> > > @@ -871,3 +871,115 @@ def unmap_url_cache(cachedir, url,
> > > expected_hash, method="md5"):
> > >          file_path = utils.unmap_url(cachedir, src, cachedir)
> > >  
> > >      return file_path
> > > +
> > > +
> > > +def prepare_test_run(basedir, rootdir='/tmp/kvm_autotest_root'):
> > > +    """
> > > +    Prepare the environment for a KVM test run by linking some
> > > previously
> > > +    prepared directories to the test bindir.
> > > +
> > > +    @param basedir: Location of the KVM test directory
> > > +    @param rootdir: Base directory that contains the KVM support
> > > directory
> > > +            structure:
> > > +            * isos (holds guest install isos)
> > > +            * images (will hold guest disk images)
> > > +            * qemu (qemu binary)
> > > +            * qemu-img (qemu-img binary)
> > > +    """
> > > +    # set English environment (command output might be
> localized,
> > > +    # need to be safe)
> > > +    os.environ['LANG'] = 'en_US.UTF-8'
> > > +
> > > +    # Symbolic links we desire to see in the KVM test dir
> > > +    list_symlinks = ['isos', 'images', 'qemu', 'qemu-img']
> > > +    for symlink_basename in list_symlinks:
> > > +        link_src = os.path.join(rootdir, symlink_basename)
> > > +        link_dst = os.path.join(basedir, symlink_basename)
> > > +        if not os.path.islink(link_dst):
> > > +            os.symlink(link_src, link_dst)
> > > +        # Testing whether our symlinks are good and in shape
> > > +        link_good = os.path.exists(os.readlink(link_dst))
> > > +        if not link_good:
> > > +            if symlink_basename in ['isos', 'images']:
> > > +                logging.critical("Mandatory directory %s is a
> broken
> > > symlink. "
> > > +                                 "This job will be aborted so you
> can
> > > fix the "
> > > +                                 "problem. Make sure %s points to
> a
> > > valid "
> > > +                                 "directory before proceeding.",
> > > +                                 symlink_basename, link_dst)
> > > +                raise error.JobError("Mandatory directory %s is
> a
> > > broken "
> > > +                                     "symlink. Please correct
> it." 
> > > %
> > > +                                     (link_dst))
> > > +            else:
> > > +                logging.warning("Symlink %s is broken. You might
> have
> > > problems "
> > > +                                "later in the test if the KVM
> build
> > > test mode "
> > > +                                "selected is 'noinstall'
> (default).
> > > ", link_dst)
> > 
> > As mentioned before I'm not sure we need this function at all.
> > If the user has to modify kvm_tests.cfg.sample anyway, he/she will
> see our
> > comments regarding the qemu/images/isos/steps paths.
> > 
> > > +def get_test_list(cfg, cfg_name, base_dir):
> > > +    """
> > > +    Return a list of test dictionaries composed by the KVM
> > > configuration system.
> > > +    This might look like a delegate of the get_list() method of
> the
> > > +    configuration, but in fact we might want to do custom config
> > > processing
> > > +    depending on the config file name provided.
> > > +
> > > +    @param cfg: KVM test configuration object
> > > +    @param cfg_name: Basename of the configuration file that
> > > generated the
> > > +            configuration object
> > > +    @param base_dir: Directory that holds the config files
> > > +
> > > +    @return: A list with test parameters dictionaries.
> > > +    """
> > > +    if cfg_name == 'kvm_tests.cfg':
> > > +        pools = os.path.join(base_dir, "kvm_address_pools.cfg")
> > > +        if os.path.exists(pools):
> > > +            cfg.parse_file(pools)
> > > +            hostname = os.uname()[1].split(".")[0]
> > > +            if cfg.filter("^" + hostname):
> > > +                cfg.parse_string("only ^%s" % hostname)
> > > +            else:
> > > +                cfg.parse_string("only ^default_host")
> > > +
> > > +    return cfg.get_list()
> > 
> > Already commented on this one above.
> > 
> > > +def run_tests(test_list, job):
> > > +    """
> > > +    Runs the sequence of KVM tests based on the list of
> dictionaries
> > > +    generated by the configuration system, handling
> dependencies.
> > > +
> > > +    @param test_list: List with all dictionary test parameters.
> > > +    @param job: Autotest job object.
> > > +    """
> > > +    status_dict = {}
> > > +
> > > +    for dict in test_list:
> > > +        if dict.get("skip") == "yes":
> > > +            continue
> > > +        dependencies_satisfied = True
> > > +        for dep in dict.get("depend"):
> > > +            for test_name in status_dict.keys():
> > > +                if not dep in test_name:
> > > +                    continue
> > > +                if not status_dict[test_name]:
> > > +                    dependencies_satisfied = False
> > > +                    break
> > > +        if dependencies_satisfied:
> > > +            test_iterations = int(dict.get("iterations", 1))
> > > +            current_status = job.run_test("kvm", params=dict,
> > > +                                         
> tag=dict.get("shortname"),
> > > +                                         
> > > iterations=test_iterations)
> > > +        else:
> > > +            current_status = False
> > > +        status_dict[dict.get("name")] = current_status
> > 
> > Already commented on this one above.
> > 
> > > +def create_report(report_dir, results_dir):
> > > +    """
> > > +    Creates a neatly arranged HTML results report in the results
> > > dir.
> > > +
> > > +    @param report_dir: Directory where the report script is
> located.
> > > +    @param results_dir: Directory where the results will be
> output.
> > > +    """
> > > +    reporter = os.path.join(report_dir, 'html_report.py')
> > > +    html_file = os.path.join(results_dir, 'results.html')
> > > +    os.system('%s -r %s -f %s -R' % (reporter, results_dir,
> > > html_file))
> > > -- 
> > > 1.6.2.5

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish <mgoldish@redhat.com> wrote:
> Sounds great, except it won't allow you to debug your configuration
> using kvm_config.py.  So the question now is what's more important --
> the ability to debug or ease of use when running from the server.

Here we have 2 use cases:

1) Users of the web interface, that (hopefully) have canned test sets
that work reliably. Ability to debug stuff is less important on this
scenario.
2) People developing tests, and in this case ability to debug config
is very important

I see the following options:

1) Document as part of the "test development guide" that, in order to
be able to debug stuff, that all the test sets are to be written to
the config file and then, can be parsed using kvm_config.
2) If we write all dictionaries generated by that particular
configuration on files inside the job results directory, we still have
debug ability for all use cases (I am starting to like this idea very
much, as I type).

So I'd then implement option 2) and refactor the control file with the
test sets defined inside strings in the control file, then you can see
how it looks? How about that?

-- 
Lucas

Re: [PATCH] [RFC] KVM test: Major control file cleanup

[Ryan Harper]

* Michael Goldish <mgoldish@redhat.com> [2009-10-28 10:43]:
> 
> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
> 
> > One thing that just occurred to me is, if we have a test config
> > "library" as you said, it's perfectly possible to put the actual test
> > set definitions and other config files inside the control file as
> > strings. This way one can control configuration inside the control
> > file,
> > making it more convenient for usage, let's say, inside the autotest
> > web
> > interface. Since the control file would be reduced in size, the
> > configuration snippets being in the control file would not be a huge
> > problem, while keeping the original autotest philosophy of keeping
> > stuff inside the control file... What do you think?
> 
> Sounds great, except it won't allow you to debug your configuration
> using kvm_config.py.  So the question now is what's more important --
> the ability to debug or ease of use when running from the server.

+1 debug

When creating new test scenarios I make *heavy* use of kvm_config.py;
I'd be lost without being able to debug test configuration files.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

Ryan, Michael:

I absolutely agree that the ability to debug stuff is important, but
the ability to make things straightforward to use from the web
interface or cli is also important. A longer term goal is to have our
test farm and make any developer able to schedule a job on the test
farm easily and conveniently.

Having the dictionaries generated on the job debug directory seems
like a good compromise to me. Also we can come up with a smart way of
parsing the config file generated by a given control file in a similar
way we do today with kvm_config.py, it shouldn't be that hard to do
it... (I hope I won't burn my tongue with this statement).

Thanks for your input!

On Wed, Oct 28, 2009 at 4:18 PM, Ryan Harper <ryanh@us.ibm.com> wrote:
> * Michael Goldish <mgoldish@redhat.com> [2009-10-28 10:43]:
>>
>> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
>>
>> > One thing that just occurred to me is, if we have a test config
>> > "library" as you said, it's perfectly possible to put the actual test
>> > set definitions and other config files inside the control file as
>> > strings. This way one can control configuration inside the control
>> > file,
>> > making it more convenient for usage, let's say, inside the autotest
>> > web
>> > interface. Since the control file would be reduced in size, the
>> > configuration snippets being in the control file would not be a huge
>> > problem, while keeping the original autotest philosophy of keeping
>> > stuff inside the control file... What do you think?
>>
>> Sounds great, except it won't allow you to debug your configuration
>> using kvm_config.py.  So the question now is what's more important --
>> the ability to debug or ease of use when running from the server.
>
> +1 debug
>
> When creating new test scenarios I make *heavy* use of kvm_config.py;
> I'd be lost without being able to debug test configuration files.
>
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> ryanh@us.ibm.com
> _______________________________________________
> Autotest mailing list
> Autotest@test.kernel.org
> http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
>



-- 
Lucas

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Michael Goldish]

It's OK to log the debug output when running the framework, but
Ryan says he makes heavy use of kvm_config.py, which means he may
not find this comfortable enough.  Ryan, what do you think?

Another option is to allow editing any file from the web GUI in
addition to the control file.

I don't think it'll be hard to parse the control file and
automatically extract config code from it in order to debug it,
but it will almost certainly be very ugly.

We might want to consider writing a little program that will emulate
the client/bin/autotest.py program, running the control file and
providing it with a fake job object whose run_test() method will
just print the params dict instead of running the test.

----- Original Message -----
From: Lucas Meneghel Rodrigues <lmr@redhat.com>
To: Ryan Harper <ryanh@us.ibm.com>
Cc: Michael Goldish <mgoldish@redhat.com>, autotest@test.kernel.org, uril@redhat.com, kvm@vger.kernel.org
Sent: Wed, 28 Oct 2009 15:47:54 -0400 (EDT)
Subject: Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

Ryan, Michael:

I absolutely agree that the ability to debug stuff is important, but
the ability to make things straightforward to use from the web
interface or cli is also important. A longer term goal is to have our
test farm and make any developer able to schedule a job on the test
farm easily and conveniently.

Having the dictionaries generated on the job debug directory seems
like a good compromise to me. Also we can come up with a smart way of
parsing the config file generated by a given control file in a similar
way we do today with kvm_config.py, it shouldn't be that hard to do
it... (I hope I won't burn my tongue with this statement).

Thanks for your input!

On Wed, Oct 28, 2009 at 4:18 PM, Ryan Harper <ryanh@us.ibm.com> wrote:
> * Michael Goldish <mgoldish@redhat.com> [2009-10-28 10:43]:
>>
>> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
>>
>> > One thing that just occurred to me is, if we have a test config
>> > "library" as you said, it's perfectly possible to put the actual test
>> > set definitions and other config files inside the control file as
>> > strings. This way one can control configuration inside the control
>> > file,
>> > making it more convenient for usage, let's say, inside the autotest
>> > web
>> > interface. Since the control file would be reduced in size, the
>> > configuration snippets being in the control file would not be a huge
>> > problem, while keeping the original autotest philosophy of keeping
>> > stuff inside the control file... What do you think?
>>
>> Sounds great, except it won't allow you to debug your configuration
>> using kvm_config.py.  So the question now is what's more important --
>> the ability to debug or ease of use when running from the server.
>
> +1 debug
>
> When creating new test scenarios I make *heavy* use of kvm_config.py;
> I'd be lost without being able to debug test configuration files.
>
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> ryanh@us.ibm.com
> _______________________________________________
> Autotest mailing list
> Autotest@test.kernel.org
> http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
>



-- 
Lucas

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Ryan Harper]

* Lucas Meneghel Rodrigues <lmr@redhat.com> [2009-10-28 14:48]:
> Ryan, Michael:
> 
> I absolutely agree that the ability to debug stuff is important, but
> the ability to make things straightforward to use from the web
> interface or cli is also important. A longer term goal is to have our
> test farm and make any developer able to schedule a job on the test
> farm easily and conveniently.
> 
> Having the dictionaries generated on the job debug directory seems
> like a good compromise to me. Also we can come up with a smart way of
> parsing the config file generated by a given control file in a similar
> way we do today with kvm_config.py, it shouldn't be that hard to do
> it... (I hope I won't burn my tongue with this statement).

If I'm understanding things, we are talking about moving the large body
of kvm_tests.cfg test definitions, guest definitions into a "library",
and then moving the requested test config (bottom on kvm_tests.cfg) into
the control file itself which means the autotest webui would be able to
control which tests get run;  I like this idea very well.  My concern
that I mentioned is that as you edit the "library" it can be difficult
to ensure you described exactly which set of tests on which guests you
want to run and kvm_config.py is invaluable in the process of getting it
right.

Why not have kvm_config.py , or some other wrapper generate a
"kvm_tests.cfg" file dynamically from the "library" and the strings from
the control file?  That way we could still debug configuration via
kvm_config.py?  I much perfer this over queueing up jobs in the webiu,
waiting for it to run, checking the results in the DEBUG dir, adjusting,
repeat.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

On Fri, 2009-10-30 at 08:54 -0500, Ryan Harper wrote:
> * Lucas Meneghel Rodrigues <lmr@redhat.com> [2009-10-28 14:48]:
> > Ryan, Michael:
> > 
> > I absolutely agree that the ability to debug stuff is important, but
> > the ability to make things straightforward to use from the web
> > interface or cli is also important. A longer term goal is to have our
> > test farm and make any developer able to schedule a job on the test
> > farm easily and conveniently.
> > 
> > Having the dictionaries generated on the job debug directory seems
> > like a good compromise to me. Also we can come up with a smart way of
> > parsing the config file generated by a given control file in a similar
> > way we do today with kvm_config.py, it shouldn't be that hard to do
> > it... (I hope I won't burn my tongue with this statement).
> 
> If I'm understanding things, we are talking about moving the large body
> of kvm_tests.cfg test definitions, guest definitions into a "library",
> and then moving the requested test config (bottom on kvm_tests.cfg) into
> the control file itself which means the autotest webui would be able to
> control which tests get run;

That is precisely my idea!

> I like this idea very well.  My concern
> that I mentioned is that as you edit the "library" it can be difficult
> to ensure you described exactly which set of tests on which guests you
> want to run and kvm_config.py is invaluable in the process of getting it
> right.

Yes, very valid concern. Let's try to address it.

> Why not have kvm_config.py , or some other wrapper generate a
> "kvm_tests.cfg" file dynamically from the "library" and the strings from
> the control file? 

That's one idea, the concern is that the code to do that might become
ugly. But I will give it a try.

> That way we could still debug configuration via
> kvm_config.py?  I much perfer this over queueing up jobs in the webiu,
> waiting for it to run, checking the results in the DEBUG dir, adjusting,
> repeat.

Ok, I will go after this solution.

Thanks for your comments Ryan,

Lucas

Re: [PATCH] [RFC] KVM test: Major control file cleanup

[Michael Goldish]

One more comment on this patch:

If you plan to make 'noinstall' the default build mode, then let's
not run the build test at all by default.  If we run it with 'noinstall',
it will just run the preprocessor and postprocessor, and those will kill
any living VMs because no VMs are required for the build test.  So a user
won't be able to run a reboot test, keep the VM alive, modify the config
file and then run a different test on the same living VM, because the
preprocessor of the second 'build' test will kill the VM.

To avoid this, we can just remove the 'noinstall' variant from
kvm_build.cfg, and keep the 'only noinstall' statement.  Because there's
no 'noinstall' variant, the resulting test list will be empty, so nothing
will run.  We should probably add a comment reminding the user that it's
possible to disable installation by stating 'only noinstall' or 'only
nothing' (in case the user replaced 'noinstall' with something else).

Alternatively, we can disable installation in the control file by commenting
out some line.

----- Original Message -----
From: "Lucas Meneghel Rodrigues" <lmr@redhat.com>
To: "Michael Goldish" <mgoldish@redhat.com>
Cc: kvm@vger.kernel.org, uril@redhat.com, autotest@test.kernel.org
Sent: Wednesday, October 28, 2009 1:14:43 PM (GMT+0200) Auto-Detected
Subject: Re: [PATCH] [RFC] KVM test: Major control file cleanup

One thing that just occurred to me is, if we have a test config
"library" as you said, it's perfectly possible to put the actual test
set definitions and other config files inside the control file as
strings. This way one can control configuration inside the control file,
making it more convenient for usage, let's say, inside the autotest web
interface. Since the control file would be reduced in size, the
configuration snippets being in the control file would not be a huge
problem, while keeping the original autotest philosophy of keeping stuff
inside the control file... What do you think?

On Wed, 2009-10-28 at 05:48 -0400, Michael Goldish wrote:
> I think this patch is a good idea.  Comments:
> 
> - If we've decided to put all the configuration in config files
> (rather than the control file) let's split up the main one into
> kvm_test_library.cfg and kvm_tests.cfg.sample.  The latter should
> look like this:
> 
> include kvm_test_library.cfg
> include kvm_cdkeys.cfg
> 
> variants:
>     - fc8_quick:
>         ...
>     - other_test_set:
>         ...
> 
> only fc8_quick
> 
> ... and the former should contain everything else.
> 
> - I think we should allow the user to modify his/her configuration
> in the control file using parse_string().  This is actually useful,
> especially when running from the server.  (At least I use it.)
> 
> - If kvm_build.cfg or kvm_tests.cfg are missing, we should continue
> without them, or create them (blank), instead of failing due to an
> unhandled exception raised by the config parser.  This should be
> true at least for kvm_build.cfg, which isn't always required for the
> tests to run.
> 
> - We should consider removing hardcoded references to
> /tmp/kvm_autotest_root/.  We use that path locally in TLV but that's
> not a good reason to put it upstream IMO.
> We should also stop creating symlinks to that path.
> Instead we can allow more flexibility using config files.
> For example, we can put this in kvm_tests.cfg.sample, between the
> 'include' and the 'variants':
> 
> # Uncomment and modify the following lines if you wish to modify
> # the paths of the image files, ISO files, step files or qemu binaries.
> # If you keep these lines commented, all mentioned paths are expected
> # to be found in the KVM test dir (tests/kvm/).  For example, all
> # image files are expected under tests/kvm/images/.
> 
> # qemu_binary ?<= /tmp/kvm_autotest_root/
> # qemu_img_binary ?<= /tmp/kvm_autotest_root/
> # image_name.* ?<= /tmp/kvm_autotest_root/
> # cdrom.* ?<= /tmp/kvm_autotest_root/
> # steps ?<= /tmp/kvm_autotest_root/
> 
> The ?<= operator prepends strings to keys that match a regex, so
> 'image_name.* ?<= something/' will prepend 'something/' to image_name,
> image_name_vm2 and so on.
> In kvm_test_library.cfg (currently kvm_tests.cfg.sample) all
> 'image_name' parameters, for example, begin with 'images/', which means
> all image files should be under tests/kvm/images.
> If we prepend /tmp/kvm_autotest_root/ to that, all image files will be
> expected under /tmp/kvm_autotest_root/images/.
> 
> Maybe we should leave these lines uncommented by default, in case anyone
> uses /tmp/kvm_autotest_root and relies on the control file creating the
> symlinks.
> 
> - kvm_utils.run_tests() should return True if all tests passed and
> otherwise False.  We need this in order to quit if the build test(s)
> failed.
> 
> - I suggest we use kvm_utils.parse_host_specific_cfg() instead of
> kvm_utils.get_test_list().  I think 'parse_host_specific_cfg' better
> describes what the function does.  I also think it should take
> 'kvm_address_pools.cfg' as a parameter so we don't have to hardcode it
> into the function.
> 
> 
> Please see additional comments below.
> 
> 
> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
> 
> > As pointed out before, the KVM reference control files
> > could use a little clean up. This patch implements severe
> > cleanup of the main control file by:
> > 
> > * Refactoring the code present there, moving it to the
> > kvm_utils.py library
> > * Treat the build test exactly the same way as other
> > tests, moving the config stuff that used to be in the
> > control file realm out to its own configuration file,
> > for the sake of consistency.
> > * Since a frequent complaint amongst users is that by
> > default the build test will build from release, and
> > that build might be unsuccessful depending on the system
> > being used (older kernel versions and other sorts of
> > stuff), change the default build method to 'noinstall'.
> > 
> > This way the control file becomes way shorter, fairly
> > well organized, and we have a consistent configuration
> > schema across the board, based on configuration files.
> > 
> > If people are OK with this change, final path will
> > change the control.parallel file as well.
> > 
> > Important: For the default control file, I removed
> > the part of the code that parses new strings for the
> > test configuration because I believe those
> > modifications belong to the test config files, they
> > could be just commented. For cases where keeping
> > all stuff inside the config file is not possible,
> > appropriate code will be added (in other example
> > control files).
> > 
> > Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
> > ---
> >  client/tests/kvm/control              |  203
> > +++------------------------------
> >  client/tests/kvm/kvm_build.cfg.sample |   82 +++++++++++++
> >  client/tests/kvm/kvm_utils.py         |  112 ++++++++++++++++++
> >  3 files changed, 211 insertions(+), 186 deletions(-)
> >  create mode 100644 client/tests/kvm/kvm_build.cfg.sample
> > 
> > diff --git a/client/tests/kvm/control b/client/tests/kvm/control
> > index 431baf9..7acec14 100644
> > --- a/client/tests/kvm/control
> > +++ b/client/tests/kvm/control
> > @@ -20,194 +20,25 @@ KVM (both kernelspace and userspace) code.
> >  For online docs, please refer to
> > http://www.linux-kvm.org/page/KVM-Autotest
> >  """
> >  
> > +import sys, os, logging
> > +# Add the KVM tests dir to the python path
> > +kvm_test_dir = os.path.join(os.environ['AUTODIR'],'tests/kvm')
> > +sys.path.append(kvm_test_dir)
> > +import kvm_utils, kvm_config
> >  
> > -import sys, os
> >  
> > -#-----------------------------------------------------------------------------
> > -# set English environment (command output might be localized, need to
> > be safe)
> > -#-----------------------------------------------------------------------------
> > -os.environ['LANG'] = 'en_US.UTF-8'
> > +## Prepare the environment for the KVM test.
> > +kvm_utils.prepare_test_run(basedir=kvm_test_dir,
> > +                           rootdir='/tmp/kvm_autotest_root')
> >  
> > -#---------------------------------------------------------
> > -# Enable modules import from current directory (tests/kvm)
> > -#---------------------------------------------------------
> > -pwd = os.path.join(os.environ['AUTODIR'],'tests/kvm')
> > -sys.path.append(pwd)
> > +cfg_names = ['kvm_build.cfg', 'kvm_tests.cfg']
> >  
> > -# ------------------------
> > -# create required symlinks
> > -# ------------------------
> > -# When dispatching tests from autotest-server the links we need do
> > not exist on
> > -# the host (the client). The following lines create those symlinks.
> > Change
> > -# 'rootdir' here and/or mount appropriate directories in it.
> > -#
> > -# When dispatching tests on local host (client mode) one can either
> > setup kvm
> > -# links, or same as server mode use rootdir and set all appropriate
> > links and
> > -# mount-points there. For example, guest installation tests need to
> > know where
> > -# to find the iso-files.
> > -#
> > -# We create the links only if not already exist, so if one already
> > set up the
> > -# links for client/local run we do not touch the links.
> > -rootdir='/tmp/kvm_autotest_root'
> > -iso=os.path.join(rootdir, 'iso')
> > -images=os.path.join(rootdir, 'images')
> > -qemu=os.path.join(rootdir, 'qemu')
> > -qemu_img=os.path.join(rootdir, 'qemu-img')
> > +for cfg_name in cfg_names:
> > +    cfg_path = os.path.join(kvm_test_dir, cfg_name)
> > +    cfg = kvm_config.config(cfg_path)
> > +    list = kvm_utils.get_test_list(cfg, cfg_name, kvm_test_dir)
> > +    logging.info("Running test set defined on config file %s",
> > cfg_name)
> > +    kvm_utils.run_tests(list, job)
> 
> I think we should unroll this loop and allow users to modify the main
> and/or build configurations here.  For example:
> 
> build_cfg_path = os.path.join(kvm_test_dir, "kvm_build.cfg")
> test_cfg_path = os.path.join(kvm_test_dir, "kvm_tests.cfg")
> open(build_cfg_path, 'a').close()
> open(test_cfg_path, 'a').close()
> 
> cfg = kvm_config.config(build_cfg_path)
> cfg.parse_string("""
> # Make any desired changes to the build configuration here.  For example:
> # release_tag = 84
> """)
> if not kvm_utils.run_tests(cfg.get_list(), job.run_test):
>     sys.exit(1)
> 
> cfg = kvm_config.config(test_cfg_path)
> cfg.parse_string("""
> # Make any desired changes to the test configuration here.  For example:
> # display = sdl
> # install|setup: timeout_multiplier = 3
> """)
> kvm_utils.parse_host_specific_cfg(cfg, os.path.join(kvm_test_dir,
>                                                     "kvm_address_pools.cfg"))
> kvm_utils.run_tests(cfg.get_list(), job.run_test)
> 
> Alternatively, instead of creating the files in case they don't exit,
> we can put the code that parses and runs tests inside 'if' statements.
> I'm not sure which option is shorter.
> 
> >  
> > -
> > -def link_if_not_exist(ldir, target, link_name):
> > -    t = target
> > -    l = os.path.join(ldir, link_name)
> > -    if not os.path.exists(l):
> > -        os.system('ln -s %s %s' % (t, l))
> > -
> > -# Create links only if not already exist
> > -link_if_not_exist(pwd, '../../', 'autotest')
> > -link_if_not_exist(pwd, iso, 'isos')
> > -link_if_not_exist(pwd, images, 'images')
> > -link_if_not_exist(pwd, qemu, 'qemu')
> > -link_if_not_exist(pwd, qemu_img, 'qemu-img')
> > -
> > -# --------------------------------------------------------
> > -# Params that will be passed to the KVM install/build test
> > -# --------------------------------------------------------
> > -params = {
> > -    "name": "build",
> > -    "shortname": "build",
> > -    "type": "build",
> > -    "mode": "release",
> > -    #"mode": "snapshot",
> > -    #"mode": "localtar",
> > -    #"mode": "localsrc",
> > -    #"mode": "git",
> > -    #"mode": "noinstall",
> > -    #"mode": "koji",
> > -
> > -    ## Are we going to load modules built by this test?
> > -    ## Defaults to 'yes', so if you are going to provide only
> > userspace code to
> > -    ## be built by this test, please set load_modules to 'no', and
> > make sure
> > -    ## the kvm and kvm-[vendor] module is already loaded by the time
> > you start
> > -    ## it.
> > -    "load_modules": "no",
> > -
> > -    ## Install from a kvm release ("mode": "release"). You can
> > optionally
> > -    ## specify a release tag. If you omit it, the test will get the
> > latest
> > -    ## release tag available.
> > -    #"release_tag": '84',
> > -    "release_dir": 'http://downloads.sourceforge.net/project/kvm/',
> > -    # This is the place that contains the sourceforge project list of
> > files
> > -    "release_listing": 'http://sourceforge.net/projects/kvm/files/',
> > -
> > -    ## Install from a kvm snapshot location ("mode": "snapshot"). You
> > can
> > -    ## optionally specify a snapshot date. If you omit it, the test
> > will get
> > -    ## yesterday's snapshot.
> > -    #"snapshot_date": '20090712'
> > -    #"snapshot_dir": 'http://foo.org/kvm-snapshots/',
> > -
> > -    ## Install from a tarball ("mode": "localtar")
> > -    #"tarball": "/tmp/kvm-84.tar.gz",
> > -
> > -    ## Install from a local source code dir ("mode": "localsrc")
> > -    #"srcdir": "/path/to/source-dir"
> > -
> > -    ## Install from koji build server ("mode": "koji")
> > -    ## Koji is the Fedora Project buildserver. It is possible to
> > install
> > -    ## packages right from Koji if you provide a release tag or a
> > build.
> > -    ## Tag (if available)
> > -    #"koji_tag": 'dist-f11',
> > -    ## Build (if available, is going to override tag).
> > -    #"koji_build": 'qemu-0.10-16.fc11',
> > -    ## Command to interact with the build server
> > -    #"koji_cmd": '/usr/bin/koji',
> > -    ## The name of the source package that's being built
> > -    #"src_pkg": 'qemu',
> > -    ## Name of the rpms we need installed
> > -    #"pkg_list": ['qemu-kvm', 'qemu-kvm-tools', 'qemu-system-x86',
> > 'qemu-common', 'qemu-img'],
> > -    ## Paths of the qemu relevant executables that should be checked
> > -    #"qemu_bin_paths": ['/usr/bin/qemu-kvm', '/usr/bin/qemu-img'],
> > -
> > -    ## Install from git ("mode": "git")
> > -    ## If you provide only "git_repo" and "user_git_repo", the build
> > test
> > -    ## will assume it will perform all build from the userspace dir,
> > building
> > -    ## modules trough make -C kernel LINUX=%s sync. As of today
> > (07-13-2009)
> > -    ## we need 3 git repos, "git_repo" (linux sources),
> > "user_git_repo" and 
> > -    ## "kmod_repo" to build KVM userspace + kernel modules.
> > -    #"git_repo":
> > 'git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git',
> > -    #"kernel_branch": 'kernel_branch_name',
> > -    #"kernel_lbranch": 'kernel_lbranch_name',
> > -    #"kernel_tag": 'kernel_tag_name',
> > -    #"user_git_repo":
> > 'git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git',
> > -    #"user_branch": 'user_branch_name',
> > -    #"user_lbranch": 'user_lbranch_name',
> > -    #"user_tag": 'user_tag_name',
> > -    #"kmod_repo":
> > 'git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git',
> > -    #"kmod_branch": 'kmod_branch_name',
> > -    #"kmod_lbranch": 'kmod_lbranch_name',
> > -    #"kmod_tag": 'kmod_tag_name',
> > -}
> > -
> > -# If you don't want to execute the build stage, just use 'noinstall'
> > as the
> > -# install type. If you run the tests from autotest-server, make sure
> > that
> > -# /tmp/kvm-autotest-root/qemu is a link to your existing executable.
> > Note that
> > -# if kvm_install is chose to run, it overwrites existing qemu and
> > qemu-img
> > -# links to point to the newly built executables.
> > -r = True
> > -r = job.run_test("kvm", params=params, tag=params.get("shortname"))
> > -if not r:
> > -    print 'kvm_installation failed ... exiting'
> > -    sys.exit(1)
> > -
> > -# ----------------------------------------------------------
> > -# Get test set (dictionary list) from the configuration file
> > -# ----------------------------------------------------------
> > -import kvm_config
> > -
> > -filename = os.path.join(pwd, "kvm_tests.cfg")
> > -cfg = kvm_config.config(filename)
> > -
> > -# If desirable, make changes to the test configuration here.  For
> > example:
> > -# cfg.parse_string("install|setup: timeout_multiplier = 2")
> > -# cfg.parse_string("only fc8_quick")
> > -# cfg.parse_string("display = sdl")
> > -
> > -filename = os.path.join(pwd, "kvm_address_pools.cfg")
> > -if os.path.exists(filename):
> > -    cfg.parse_file(filename)
> > -    hostname = os.uname()[1].split(".")[0]
> > -    if cfg.filter("^" + hostname):
> > -        cfg.parse_string("only ^%s" % hostname)
> > -    else:
> > -        cfg.parse_string("only ^default_host")
> > -
> > -list = cfg.get_list()
> > -
> > -
> > -# -------------
> > -# Run the tests
> > -# -------------
> > -status_dict = {}
> > -
> > -for dict in list:
> > -    if dict.get("skip") == "yes":
> > -        continue
> > -    dependencies_satisfied = True
> > -    for dep in dict.get("depend"):
> > -        for test_name in status_dict.keys():
> > -            if not dep in test_name:
> > -                continue
> > -            if not status_dict[test_name]:
> > -                dependencies_satisfied = False
> > -                break
> > -    if dependencies_satisfied:
> > -        test_iterations = int(dict.get("iterations", 1))
> > -        current_status = job.run_test("kvm", params=dict,
> > -                                      tag=dict.get("shortname"),
> > -                                      iterations=test_iterations)
> > -    else:
> > -        current_status = False
> > -    status_dict[dict.get("name")] = current_status
> > -
> > -# create the html report in result dir
> > -reporter = os.path.join(pwd, 'html_report.py')
> > -html_file = os.path.join(job.resultdir,'results.html')
> > -os.system('%s -r %s -f %s -R' % (reporter, job.resultdir,
> > html_file))
> > +## Generate a nice HTML report inside results_dir
> > +kvm_utils.create_report(kvm_test_dir, job.resultsdir)
> > diff --git a/client/tests/kvm/kvm_build.cfg.sample
> > b/client/tests/kvm/kvm_build.cfg.sample
> > new file mode 100644
> > index 0000000..692a747
> > --- /dev/null
> > +++ b/client/tests/kvm/kvm_build.cfg.sample
> > @@ -0,0 +1,82 @@
> > +# This configuration file holds the KVM build test config
> > parameters.
> > +# The default is noinstall (won't attempt to build KVM), so if you
> > stick with it
> > +# please make sure:
> > +# 1) You have setup symbolic links to qemu and qemu-img binaries on
> > the
> > +#    KVM test dir.
> > +# 2) The appropriate KVM modules are already loaded on your machine.
> > +
> > +variants:
> > +    - build:
> > +        type = build
> > +        # Load modules built/installed by the build test?
> > +        load_modules = no
> > +        variants:
> > +            - noinstall:
> > +                mode = noinstall
> > +            - release:
> > +                mode = release
> > +                ## Install from a kvm release. You can optionally
> > specify
> > +                ## a release tag. If you omit it, the build test will
> > get
> > +                ## the latest release tag available at that moment.
> > +                # release_tag = 84
> > +                release_dir =
> > http://downloads.sourceforge.net/project/kvm/
> > +                release_listing =
> > http://sourceforge.net/projects/kvm/files/
> > +            - snapshot:
> > +                mode = snapshot
> > +                ## Install from a kvm snapshot location. You can
> > optionally
> > +                ## specify a snapshot date. If you omit it, the test
> > will get
> > +                ## yesterday's snapshot.
> > +                # snapshot_date = 20090712
> > +                snapshot_dir = http://foo.org/kvm-snapshots/
> > +            - localtar:
> > +                mode = localtar
> > +                ## Install from tarball located on the host's
> > filesystem.
> > +                tarball = /tmp/kvm-84.tar.gz
> > +            - localsrc:
> > +                mode = localsrc
> > +                ## Install from tarball located on the host's
> > filesystem.
> > +                srcdir = /tmp/kvm-84
> > +            - git:
> > +                mode = git
> > +                ## Install KVM from git repositories.
> > +                ## If you provide only "git_repo" and
> > "user_git_repo", the
> > +                ## build test will assume it will perform all build
> > from the
> > +                ## userspace dir, building modules trough 
> > +                ## make -C kernel LINUX=%s sync. As of today
> > (07-13-2009)
> > +                ## upstream needs 3 git repos:
> > +                ## * git_repo (linux sources)
> > +                ## * user_git_repo (qemu sources)
> > +                ## * kmod_repo" to build KVM userspace + kernel
> > modules.
> > +                git_repo =
> > git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git
> > +                # kernel_branch = kernel_branch_name
> > +                # kernel_lbranch = kernel_lbranch_name
> > +                # kernel_tag = kernel_tag_name
> > +                user_git_repo =
> > git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
> > +                # user_branch = user_branch_name
> > +                # user_lbranch = user_lbranch_name
> > +                # user_tag = user_tag_name
> > +                kmod_repo =
> > git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git
> > +                # kmod_branch = kmod_branch_name
> > +                # kmod_lbranch = kmod_lbranch_name
> > +                # kmod_tag = kmod_tag_name
> > +            - koji:
> > +                mode = koji
> > +                ## Install KVM from koji (Fedora build server)
> > +                ## It is possible to install packages right from Koji
> > if you
> > +                ## provide a release tag or a build.
> > +                ## Tag (if available)
> > +                koji_tag = dist-f11
> > +                ## Build (if available, is going to override tag).
> > +                koji_build = qemu-0.10-16.fc11
> > +                ## Command to interact with the build server
> > +                koji_cmd = /usr/bin/koji
> > +                ## The name of the source package that will be built
> > +                src_pkg = qemu
> > +                ## Name of the rpms we need installed
> > +                pkg_list = ['qemu-kvm', 'qemu-kvm-tools',
> > 'qemu-system-x86', 'qemu-common', 'qemu-img']
> > +                ## Paths of the qemu relevant executables that should
> > be checked
> > +                qemu_bin_paths = ['/usr/bin/qemu-kvm',
> > '/usr/bin/qemu-img']
> > +
> > +
> > +# Choose your install method here
> > +only build.noinstall
> > \ No newline at end of file
> > diff --git a/client/tests/kvm/kvm_utils.py
> > b/client/tests/kvm/kvm_utils.py
> > index f72984a..90b3def 100644
> > --- a/client/tests/kvm/kvm_utils.py
> > +++ b/client/tests/kvm/kvm_utils.py
> > @@ -871,3 +871,115 @@ def unmap_url_cache(cachedir, url,
> > expected_hash, method="md5"):
> >          file_path = utils.unmap_url(cachedir, src, cachedir)
> >  
> >      return file_path
> > +
> > +
> > +def prepare_test_run(basedir, rootdir='/tmp/kvm_autotest_root'):
> > +    """
> > +    Prepare the environment for a KVM test run by linking some
> > previously
> > +    prepared directories to the test bindir.
> > +
> > +    @param basedir: Location of the KVM test directory
> > +    @param rootdir: Base directory that contains the KVM support
> > directory
> > +            structure:
> > +            * isos (holds guest install isos)
> > +            * images (will hold guest disk images)
> > +            * qemu (qemu binary)
> > +            * qemu-img (qemu-img binary)
> > +    """
> > +    # set English environment (command output might be localized,
> > +    # need to be safe)
> > +    os.environ['LANG'] = 'en_US.UTF-8'
> > +
> > +    # Symbolic links we desire to see in the KVM test dir
> > +    list_symlinks = ['isos', 'images', 'qemu', 'qemu-img']
> > +    for symlink_basename in list_symlinks:
> > +        link_src = os.path.join(rootdir, symlink_basename)
> > +        link_dst = os.path.join(basedir, symlink_basename)
> > +        if not os.path.islink(link_dst):
> > +            os.symlink(link_src, link_dst)
> > +        # Testing whether our symlinks are good and in shape
> > +        link_good = os.path.exists(os.readlink(link_dst))
> > +        if not link_good:
> > +            if symlink_basename in ['isos', 'images']:
> > +                logging.critical("Mandatory directory %s is a broken
> > symlink. "
> > +                                 "This job will be aborted so you can
> > fix the "
> > +                                 "problem. Make sure %s points to a
> > valid "
> > +                                 "directory before proceeding.",
> > +                                 symlink_basename, link_dst)
> > +                raise error.JobError("Mandatory directory %s is a
> > broken "
> > +                                     "symlink. Please correct it." 
> > %
> > +                                     (link_dst))
> > +            else:
> > +                logging.warning("Symlink %s is broken. You might have
> > problems "
> > +                                "later in the test if the KVM build
> > test mode "
> > +                                "selected is 'noinstall' (default).
> > ", link_dst)
> 
> As mentioned before I'm not sure we need this function at all.
> If the user has to modify kvm_tests.cfg.sample anyway, he/she will see our
> comments regarding the qemu/images/isos/steps paths.
> 
> > +def get_test_list(cfg, cfg_name, base_dir):
> > +    """
> > +    Return a list of test dictionaries composed by the KVM
> > configuration system.
> > +    This might look like a delegate of the get_list() method of the
> > +    configuration, but in fact we might want to do custom config
> > processing
> > +    depending on the config file name provided.
> > +
> > +    @param cfg: KVM test configuration object
> > +    @param cfg_name: Basename of the configuration file that
> > generated the
> > +            configuration object
> > +    @param base_dir: Directory that holds the config files
> > +
> > +    @return: A list with test parameters dictionaries.
> > +    """
> > +    if cfg_name == 'kvm_tests.cfg':
> > +        pools = os.path.join(base_dir, "kvm_address_pools.cfg")
> > +        if os.path.exists(pools):
> > +            cfg.parse_file(pools)
> > +            hostname = os.uname()[1].split(".")[0]
> > +            if cfg.filter("^" + hostname):
> > +                cfg.parse_string("only ^%s" % hostname)
> > +            else:
> > +                cfg.parse_string("only ^default_host")
> > +
> > +    return cfg.get_list()
> 
> Already commented on this one above.
> 
> > +def run_tests(test_list, job):
> > +    """
> > +    Runs the sequence of KVM tests based on the list of dictionaries
> > +    generated by the configuration system, handling dependencies.
> > +
> > +    @param test_list: List with all dictionary test parameters.
> > +    @param job: Autotest job object.
> > +    """
> > +    status_dict = {}
> > +
> > +    for dict in test_list:
> > +        if dict.get("skip") == "yes":
> > +            continue
> > +        dependencies_satisfied = True
> > +        for dep in dict.get("depend"):
> > +            for test_name in status_dict.keys():
> > +                if not dep in test_name:
> > +                    continue
> > +                if not status_dict[test_name]:
> > +                    dependencies_satisfied = False
> > +                    break
> > +        if dependencies_satisfied:
> > +            test_iterations = int(dict.get("iterations", 1))
> > +            current_status = job.run_test("kvm", params=dict,
> > +                                          tag=dict.get("shortname"),
> > +                                         
> > iterations=test_iterations)
> > +        else:
> > +            current_status = False
> > +        status_dict[dict.get("name")] = current_status
> 
> Already commented on this one above.
> 
> > +def create_report(report_dir, results_dir):
> > +    """
> > +    Creates a neatly arranged HTML results report in the results
> > dir.
> > +
> > +    @param report_dir: Directory where the report script is located.
> > +    @param results_dir: Directory where the results will be output.
> > +    """
> > +    reporter = os.path.join(report_dir, 'html_report.py')
> > +    html_file = os.path.join(results_dir, 'results.html')
> > +    os.system('%s -r %s -f %s -R' % (reporter, results_dir,
> > html_file))
> > -- 
> > 1.6.2.5

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

On Wed, Oct 28, 2009 at 1:57 PM, Michael Goldish <mgoldish@redhat.com> wrote:
> One more comment on this patch:
>
> If you plan to make 'noinstall' the default build mode, then let's
> not run the build test at all by default.  If we run it with 'noinstall',
> it will just run the preprocessor and postprocessor, and those will kill
> any living VMs because no VMs are required for the build test.  So a user
> won't be able to run a reboot test, keep the VM alive, modify the config
> file and then run a different test on the same living VM, because the
> preprocessor of the second 'build' test will kill the VM.
>
> To avoid this, we can just remove the 'noinstall' variant from
> kvm_build.cfg, and keep the 'only noinstall' statement.  Because there's
> no 'noinstall' variant, the resulting test list will be empty, so nothing
> will run.  We should probably add a comment reminding the user that it's
> possible to disable installation by stating 'only noinstall' or 'only
> nothing' (in case the user replaced 'noinstall' with something else).
>
> Alternatively, we can disable installation in the control file by commenting
> out some line.

Great idea, thanks :)

-
Lucas

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Michael Goldish]

----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:

> On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish <mgoldish@redhat.com>
> wrote:
> > Sounds great, except it won't allow you to debug your configuration
> > using kvm_config.py.  So the question now is what's more important
> --
> > the ability to debug or ease of use when running from the server.
> 
> Here we have 2 use cases:
> 
> 1) Users of the web interface, that (hopefully) have canned test sets
> that work reliably. Ability to debug stuff is less important on this
> scenario.
> 2) People developing tests, and in this case ability to debug config
> is very important
> 
> I see the following options:
> 
> 1) Document as part of the "test development guide" that, in order to
> be able to debug stuff, that all the test sets are to be written to
> the config file and then, can be parsed using kvm_config.
> 2) If we write all dictionaries generated by that particular
> configuration on files inside the job results directory, we still
> have
> debug ability for all use cases (I am starting to like this idea very
> much, as I type).
> 
> So I'd then implement option 2) and refactor the control file with
> the
> test sets defined inside strings in the control file, then you can
> see
> how it looks? How about that?

Sounds fine.
- Where exactly will the test list appear?
- We should also allow printing of verbose debug output ("parsing variants
block, 9000 dicts in current context...") by passing something to the
constructor of the config object.
- We should make it clear to the user that he/she must rename the control
file (to control.lucas for example) or else it may be overwritten on the
next git-fetch or -pull.

I'm still not sure it's a great idea to make config debugging harder, so
if anyone other than Lucas who uses the KVM test is reading this, please
let us know if you ever use kvm_config.py and if you think the ability to
print the list of test dicts is important.

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Yolkfull Chow]

On Wed, Oct 28, 2009 at 02:04:59PM -0400, Michael Goldish wrote:
> 
> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
> 
> > On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish <mgoldish@redhat.com>
> > wrote:
> > > Sounds great, except it won't allow you to debug your configuration
> > > using kvm_config.py.  So the question now is what's more important
> > --
> > > the ability to debug or ease of use when running from the server.
> > 
> > Here we have 2 use cases:
> > 
> > 1) Users of the web interface, that (hopefully) have canned test sets
> > that work reliably. Ability to debug stuff is less important on this
> > scenario.
> > 2) People developing tests, and in this case ability to debug config
> > is very important
> > 
> > I see the following options:
> > 
> > 1) Document as part of the "test development guide" that, in order to
> > be able to debug stuff, that all the test sets are to be written to
> > the config file and then, can be parsed using kvm_config.
> > 2) If we write all dictionaries generated by that particular
> > configuration on files inside the job results directory, we still
> > have
> > debug ability for all use cases (I am starting to like this idea very
> > much, as I type).
> > 
> > So I'd then implement option 2) and refactor the control file with
> > the
> > test sets defined inside strings in the control file, then you can
> > see
> > how it looks? How about that?
> 
> Sounds fine.
> - Where exactly will the test list appear?
> - We should also allow printing of verbose debug output ("parsing variants
> block, 9000 dicts in current context...") by passing something to the
> constructor of the config object.
> - We should make it clear to the user that he/she must rename the control
> file (to control.lucas for example) or else it may be overwritten on the
> next git-fetch or -pull.
> 
> I'm still not sure it's a great idea to make config debugging harder, so
> if anyone other than Lucas who uses the KVM test is reading this, please
> let us know if you ever use kvm_config.py and if you think the ability to
> print the list of test dicts is important.

Hi Michael,

I had used kvm_config.py for printing lists of selected test dicts
often. And I think it's necessary to keep this feature. IMHO, option 2) Lucas
proposed is a good idea. What do you think? Hope I haven't missed
something. :)

> _______________________________________________
> Autotest mailing list
> Autotest@test.kernel.org
> http://test.kernel.org/cgi-bin/mailman/listinfo/autotest

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Lucas Meneghel Rodrigues]

On Wed, Nov 4, 2009 at 5:25 AM, Yolkfull Chow <yzhou@redhat.com> wrote:
> Hi Michael,
>
> I had used kvm_config.py for printing lists of selected test dicts
> often. And I think it's necessary to keep this feature. IMHO, option 2) Lucas
> proposed is a good idea. What do you think? Hope I haven't missed
> something. :)

2) is a compromise solution, but we find it's cumbersome, so we are
going to do something that can print all test dicts from a control
file, in addition to 2).

Thanks for your input!

-- 
Lucas

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Michael Goldish]

----- "Ryan Harper" <ryanh@us.ibm.com> wrote:

> * Lucas Meneghel Rodrigues <lmr@redhat.com> [2009-10-28 14:48]:
> > Ryan, Michael:
> > 
> > I absolutely agree that the ability to debug stuff is important,
> but
> > the ability to make things straightforward to use from the web
> > interface or cli is also important. A longer term goal is to have
> our
> > test farm and make any developer able to schedule a job on the test
> > farm easily and conveniently.
> > 
> > Having the dictionaries generated on the job debug directory seems
> > like a good compromise to me. Also we can come up with a smart way
> of
> > parsing the config file generated by a given control file in a
> similar
> > way we do today with kvm_config.py, it shouldn't be that hard to do
> > it... (I hope I won't burn my tongue with this statement).
> 
> If I'm understanding things, we are talking about moving the large
> body
> of kvm_tests.cfg test definitions, guest definitions into a
> "library",
> and then moving the requested test config (bottom on kvm_tests.cfg)
> into
> the control file itself which means the autotest webui would be able
> to
> control which tests get run;  I like this idea very well.  My concern
> that I mentioned is that as you edit the "library" it can be
> difficult
> to ensure you described exactly which set of tests on which guests
> you
> want to run and kvm_config.py is invaluable in the process of getting
> it
> right.
> 
> Why not have kvm_config.py , or some other wrapper generate a
> "kvm_tests.cfg" file dynamically from the "library" and the strings
> from
> the control file?  That way we could still debug configuration via
> kvm_config.py?  I much perfer this over queueing up jobs in the
> webiu,
> waiting for it to run, checking the results in the DEBUG dir,
> adjusting,
> repeat.

I'm not sure I understand your idea: you want some program to read the
control file and generate a new file (kvm_tests.cfg or something) from
the control file and the library file, so that this file can be debugged
with kvm_config.py?

IMO this solution is "dirty" because the control file is python code, not
our own format, so it's not nice to automatically extract stuff from it.
It would be nice to do something that eases debugging, but if you ask me,
I'd rather have something as clean as possible.

Here's another idea, which I suggested but haven't received any feedback
on: let's write a little proggie that runs the control file just like
client/bin/autotest does.  The proggie will supply the control file with
a fake job object that has nothing but a run_test() method, but instead
of running a test, that method will simply nicely print out the test
params, like kvm_config.py does.  So the user will be able to do something
like './dry_run.py control.mine' which will list all the tests to be
executed.  We might also want to implement job.parallel() in addition to
job.run_test() but that should be very easy (it doesn't really have to be
parallel at all).
Does that make any sense?

Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup

[Ryan Harper]

* Michael Goldish <mgoldish@redhat.com> [2009-10-31 16:02]:
> > 
> > If I'm understanding things, we are talking about moving the large
> > body
> > of kvm_tests.cfg test definitions, guest definitions into a
> > "library",
> > and then moving the requested test config (bottom on kvm_tests.cfg)
> > into
> > the control file itself which means the autotest webui would be able
> > to
> > control which tests get run;  I like this idea very well.  My concern
> > that I mentioned is that as you edit the "library" it can be
> > difficult
> > to ensure you described exactly which set of tests on which guests
> > you
> > want to run and kvm_config.py is invaluable in the process of getting
> > it
> > right.
> > 
> > Why not have kvm_config.py , or some other wrapper generate a
> > "kvm_tests.cfg" file dynamically from the "library" and the strings
> > from
> > the control file?  That way we could still debug configuration via
> > kvm_config.py?  I much perfer this over queueing up jobs in the
> > webiu,
> > waiting for it to run, checking the results in the DEBUG dir,
> > adjusting,
> > repeat.
> 
> I'm not sure I understand your idea: you want some program to read the
> control file and generate a new file (kvm_tests.cfg or something) from
> the control file and the library file, so that this file can be debugged
> with kvm_config.py?

I wanted something that would take the test description string from the
control file, run that through the library such that it can print out
the tests that will run -- the equivalent of running kvm_config.py on a
kvm_tests.cfg file... 

> 
> IMO this solution is "dirty" because the control file is python code, not
> our own format, so it's not nice to automatically extract stuff from it.
> It would be nice to do something that eases debugging, but if you ask me,
> I'd rather have something as clean as possible.
> 
> Here's another idea, which I suggested but haven't received any feedback
> on: let's write a little proggie that runs the control file just like
> client/bin/autotest does.  The proggie will supply the control file with
> a fake job object that has nothing but a run_test() method, but instead
> of running a test, that method will simply nicely print out the test
> params, like kvm_config.py does.  So the user will be able to do something
> like './dry_run.py control.mine' which will list all the tests to be
> executed.  We might also want to implement job.parallel() in addition to
> job.run_test() but that should be very easy (it doesn't really have to be
> parallel at all).
> Does that make any sense?

It does, though I don't see how it's different then what I suggested; I
don't really care how we do it;  I still need something to debug test
config files outside of cycling through job output via the webui;


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com