Re: [PATCH] [RFC] KVM test: Control files automatic generation to save memory

[Dor Laor <dlaor@redhat.com>]

On 02/14/2010 07:07 PM, Michael Goldish wrote:
>
> ----- "Lucas Meneghel Rodrigues"<lmr@redhat.com>  wrote:
>
>> As our configuration system generates a list of dicts
>> with test parameters, and that list might be potentially
>> *very* large, keeping all this information in memory might
>> be a problem for smaller virtualization hosts due to
>> the memory pressure created. Tests made on my 4GB laptop
>> show that most of the memory is being used during a
>> typical kvm autotest session.
>>
>> So, instead of keeping all this information in memory,
>> let's take a different approach and unfold all the
>> tests generated by the config system and generate a
>> control file:
>>
>> job.run_test('kvm', params={param1, param2, ...}, tag='foo', ...)
>> job.run_test('kvm', params={param1, param2, ...}, tag='bar', ...)
>>
>> By dumping all the dicts that were before in the memory to
>> a control file, the memory usage of a typical kvm autotest
>> session is drastically reduced making it easier to run in smaller
>> virt hosts.
>>
>> The advantages of taking this new approach are:
>>   * You can see what tests are going to run and the dependencies
>>     between them by looking at the generated control file
>>   * The control file is all ready to use, you can for example
>>     paste it on the web interface and profit
>>   * As mentioned, a lot less memory consumption, avoiding
>>     memory pressure on virtualization hosts.
>>
>> This is a crude 1st pass at implementing this approach, so please
>> provide comments.
>>
>> Signed-off-by: Lucas Meneghel Rodrigues<lmr@redhat.com>
>> ---
>
> Interesting idea!
>
> - Personally I don't like the renaming of kvm_config.py to
> generate_control.py, and prefer to keep them separate, so that
> generate_control.py has the create_control() function and
> kvm_config.py has everything else.  It's just a matter of naming;
> kvm_config.py deals mostly with config files, not with control files,
> and it can be used for other purposes than generating control files.
>
> - I wonder why so much memory is used by the test list.  Our daily
> test sets aren't very big, so although the parser should use a huge
> amount of memory while parsing, nearly all of that memory should be
> freed by the time the parser is done, because the final 'only'
> statement reduces the number of tests to a small fraction of the total
> number in a full set.  What test set did you try with that 4 GB
> machine, and how much memory was used by the test list?  If a
> ridiculous amount of memory was used, this might indicate a bug in
> kvm_config.py (maybe it keeps references to deleted tests, forcing
> them to stay in memory).

I agree, it's worth getting to the bottom of it - I wonder how many 
objects are created on kvm unstable set. It should be a huge number.
Besides that, one can always call the python garbage collection 
interface in order to free unreferenced memory immediately.

>
> - I don't think this approach will work for control.parallel, because
> the tests have to be assigned dynamically to available queues, and
> AFAIK this can't be done by a simple static control file.
>
> - Whether or not this is a good idea probably depends on the users.
> On one hand, users will be required to run generate_control.py before
> autotest.py, and the generated control files will be very big and
> ugly; on the other hand, maybe they won't care.
>
> I probably haven't given this enough thought so I might have missed a
> few things.
>
>
>>   client/tests/kvm/control             |   64 ----
>>   client/tests/kvm/generate_control.py |  586
>> ++++++++++++++++++++++++++++++++++
>>   client/tests/kvm/kvm_config.py       |  524
>> ------------------------------
>>   3 files changed, 586 insertions(+), 588 deletions(-)
>>   delete mode 100644 client/tests/kvm/control
>>   create mode 100755 client/tests/kvm/generate_control.py
>>   delete mode 100755 client/tests/kvm/kvm_config.py
>>
>> diff --git a/client/tests/kvm/control b/client/tests/kvm/control
>> deleted file mode 100644
>> index 163286e..0000000
>> --- a/client/tests/kvm/control
>> +++ /dev/null
>> @@ -1,64 +0,0 @@
>> -AUTHOR = """
>> -uril@redhat.com (Uri Lublin)
>> -drusso@redhat.com (Dror Russo)
>> -mgoldish@redhat.com (Michael Goldish)
>> -dhuff@redhat.com (David Huff)
>> -aeromenk@redhat.com (Alexey Eromenko)
>> -mburns@redhat.com (Mike Burns)
>> -"""
>> -TIME = 'MEDIUM'
>> -NAME = 'KVM test'
>> -TEST_TYPE = 'client'
>> -TEST_CLASS = 'Virtualization'
>> -TEST_CATEGORY = 'Functional'
>> -
>> -DOC = """
>> -Executes the KVM test framework on a given host. This module is
>> separated in
>> -minor functions, that execute different tests for doing Quality
>> Assurance on
>> -KVM (both kernelspace and userspace) code.
>> -
>> -For online docs, please refer to
>> http://www.linux-kvm.org/page/KVM-Autotest
>> -"""
>> -
>> -import sys, os, logging
>> -# Add the KVM tests dir to the python path
>> -kvm_test_dir = os.path.join(os.environ['AUTODIR'],'tests/kvm')
>> -sys.path.append(kvm_test_dir)
>> -# Now we can import modules inside the KVM tests dir
>> -import kvm_utils, kvm_config
>> -
>> -# set English environment (command output might be localized, need to
>> be safe)
>> -os.environ['LANG'] = 'en_US.UTF-8'
>> -
>> -build_cfg_path = os.path.join(kvm_test_dir, "build.cfg")
>> -build_cfg = kvm_config.config(build_cfg_path)
>> -# Make any desired changes to the build configuration here. For
>> example:
>> -#build_cfg.parse_string("""
>> -#release_tag = 84
>> -#""")
>> -if not kvm_utils.run_tests(build_cfg.get_list(), job):
>> -    logging.error("KVM build step failed, exiting.")
>> -    sys.exit(1)
>> -
>> -tests_cfg_path = os.path.join(kvm_test_dir, "tests.cfg")
>> -tests_cfg = kvm_config.config(tests_cfg_path)
>> -# Make any desired changes to the test configuration here. For
>> example:
>> -#tests_cfg.parse_string("""
>> -#display = sdl
>> -#install|setup: timeout_multiplier = 3
>> -#""")
>> -
>> -pools_cfg_path = os.path.join(kvm_test_dir, "address_pools.cfg")
>> -tests_cfg.parse_file(pools_cfg_path)
>> -hostname = os.uname()[1].split(".")[0]
>> -if tests_cfg.filter("^" + hostname):
>> -    tests_cfg.parse_string("only ^%s" % hostname)
>> -else:
>> -    tests_cfg.parse_string("only ^default_host")
>> -
>> -# Run the tests
>> -kvm_utils.run_tests(tests_cfg.get_list(), job)
>> -
>> -# Generate a nice HTML report inside the job's results dir
>> -kvm_utils.create_report(kvm_test_dir, job.resultdir)
>> -
>> diff --git a/client/tests/kvm/generate_control.py
>> b/client/tests/kvm/generate_control.py
>> new file mode 100755
>> index 0000000..c64dc52
>> --- /dev/null
>> +++ b/client/tests/kvm/generate_control.py
>> @@ -0,0 +1,586 @@
>> +#!/usr/bin/python
>> +"""
>> +KVM configuration file utility functions.
>> +
>> +@copyright: Red Hat 2008-2009
>> +"""
>> +
>> +import logging, re, os, sys, StringIO, optparse
>> +import common
>> +import kvm_utils
>> +from autotest_lib.client.common_lib import error
>> +from autotest_lib.client.common_lib import logging_config,
>> logging_manager
>> +
>> +
>> +class config:
>> +    """
>> +    Parse an input file or string that follows the KVM Test Config
>> File format
>> +    and generate a list of dicts that will be later used as
>> configuration
>> +    parameters by the the KVM tests.
>> +
>> +    @see:
>> http://www.linux-kvm.org/page/KVM-Autotest/Test_Config_File
>> +    """
>> +
>> +    def __init__(self, filename=None, debug=False):
>> +        """
>> +        Initialize the list and optionally parse filename.
>> +
>> +        @param filename: Path of the file that will be taken.
>> +        @param debug: Whether to turn debugging output.
>> +        """
>> +        self.list = [{"name": "", "shortname": "", "depend": []}]
>> +        self.debug = debug
>> +        self.filename = filename
>> +        if filename:
>> +            self.parse_file(filename)
>> +
>> +
>> +    def parse_file(self, filename):
>> +        """
>> +        Parse filename, return the resulting list and store it in
>> .list. If
>> +        filename does not exist, raise an exception.
>> +
>> +        @param filename: Path of the configuration file.
>> +        """
>> +        if not os.path.exists(filename):
>> +            raise Exception, "File %s not found" % filename
>> +        self.filename = filename
>> +        file = open(filename, "r")
>> +        self.list = self.parse(file, self.list)
>> +        file.close()
>> +        return self.list
>> +
>> +
>> +    def parse_string(self, str):
>> +        """
>> +        Parse a string, return the resulting list and store it in
>> .list.
>> +
>> +        @param str: String that will be parsed.
>> +        """
>> +        file = StringIO.StringIO(str)
>> +        self.list = self.parse(file, self.list)
>> +        file.close()
>> +        return self.list
>> +
>> +
>> +    def get_list(self):
>> +        """
>> +        Return the list of dictionaries. This should probably be
>> called after
>> +        parsing something.
>> +        """
>> +        return self.list
>> +
>> +
>> +    def match(self, filter, dict):
>> +        """
>> +        Return True if dict matches filter.
>> +
>> +        @param filter: A regular expression that defines the filter.
>> +        @param dict: Dictionary that will be inspected.
>> +        """
>> +        filter = re.compile(r"(\.|^)(%s)(\.|$)" % filter)
>> +        return bool(filter.search(dict["name"]))
>> +
>> +
>> +    def filter(self, filter, list=None):
>> +        """
>> +        Filter a list of dicts.
>> +
>> +        @param filter: A regular expression that will be used as a
>> filter.
>> +        @param list: A list of dictionaries that will be filtered.
>> +        """
>> +        if list is None:
>> +            list = self.list
>> +        return [dict for dict in list if self.match(filter, dict)]
>> +
>> +
>> +    def split_and_strip(self, str, sep="="):
>> +        """
>> +        Split str and strip quotes from the resulting parts.
>> +
>> +        @param str: String that will be processed
>> +        @param sep: Separator that will be used to split the string
>> +        """
>> +        temp = str.split(sep, 1)
>> +        for i in range(len(temp)):
>> +            temp[i] = temp[i].strip()
>> +            if re.findall("^\".*\"$", temp[i]):
>> +                temp[i] = temp[i].strip("\"")
>> +            elif re.findall("^\'.*\'$", temp[i]):
>> +                temp[i] = temp[i].strip("\'")
>> +        return temp
>> +
>> +
>> +    def get_next_line(self, file):
>> +        """
>> +        Get the next non-empty, non-comment line in a file like
>> object.
>> +
>> +        @param file: File like object
>> +        @return: If no line is available, return None.
>> +        """
>> +        while True:
>> +            line = file.readline()
>> +            if line == "": return None
>> +            stripped_line = line.strip()
>> +            if len(stripped_line)>  0 \
>> +                    and not stripped_line.startswith('#') \
>> +                    and not stripped_line.startswith('//'):
>> +                return line
>> +
>> +
>> +    def get_next_line_indent(self, file):
>> +        """
>> +        Return the indent level of the next non-empty, non-comment
>> line in file.
>> +
>> +        @param file: File like object.
>> +        @return: If no line is available, return -1.
>> +        """
>> +        pos = file.tell()
>> +        line = self.get_next_line(file)
>> +        if not line:
>> +            file.seek(pos)
>> +            return -1
>> +        line = line.expandtabs()
>> +        indent = 0
>> +        while line[indent] == ' ':
>> +            indent += 1
>> +        file.seek(pos)
>> +        return indent
>> +
>> +
>> +    def add_name(self, str, name, append=False):
>> +        """
>> +        Add name to str with a separator dot and return the result.
>> +
>> +        @param str: String that will be processed
>> +        @param name: name that will be appended to the string.
>> +        @return: If append is True, append name to str.
>> +                Otherwise, pre-pend name to str.
>> +        """
>> +        if str == "":
>> +            return name
>> +        # Append?
>> +        elif append:
>> +            return str + "." + name
>> +        # Prepend?
>> +        else:
>> +            return name + "." + str
>> +
>> +
>> +    def parse_variants(self, file, list, subvariants=False,
>> prev_indent=-1):
>> +        """
>> +        Read and parse lines from file like object until a line with
>> an indent
>> +        level lower than or equal to prev_indent is encountered.
>> +
>> +        @brief: Parse a 'variants' or 'subvariants' block from a
>> file-like
>> +        object.
>> +        @param file: File-like object that will be parsed
>> +        @param list: List of dicts to operate on
>> +        @param subvariants: If True, parse in 'subvariants' mode;
>> +        otherwise parse in 'variants' mode
>> +        @param prev_indent: The indent level of the "parent" block
>> +        @return: The resulting list of dicts.
>> +        """
>> +        new_list = []
>> +
>> +        while True:
>> +            indent = self.get_next_line_indent(file)
>> +            if indent<= prev_indent:
>> +                break
>> +            indented_line = self.get_next_line(file).rstrip()
>> +            line = indented_line.strip()
>> +
>> +            # Get name and dependencies
>> +            temp = line.strip("- ").split(":")
>> +            name = temp[0]
>> +            if len(temp) == 1:
>> +                dep_list = []
>> +            else:
>> +                dep_list = temp[1].split()
>> +
>> +            # See if name should be added to the 'shortname' field
>> +            add_to_shortname = True
>> +            if name.startswith("@"):
>> +                name = name.strip("@")
>> +                add_to_shortname = False
>> +
>> +            # Make a deep copy of list
>> +            temp_list = []
>> +            for dict in list:
>> +                new_dict = dict.copy()
>> +                new_dict["depend"] = dict["depend"][:]
>> +                temp_list.append(new_dict)
>> +
>> +            if subvariants:
>> +                # If we're parsing 'subvariants', first modify the
>> list
>> +                self.__modify_list_subvariants(temp_list, name,
>> dep_list,
>> +                                               add_to_shortname)
>> +                temp_list = self.parse(file, temp_list,
>> +                        restricted=True, prev_indent=indent)
>> +            else:
>> +                # If we're parsing 'variants', parse before modifying
>> the list
>> +                if self.debug:
>> +                    self.__debug_print(indented_line,
>> +                                       "Entering variant '%s' "
>> +                                       "(variant inherits %d dicts)"
>> %
>> +                                       (name, len(list)))
>> +                temp_list = self.parse(file, temp_list,
>> +                        restricted=False, prev_indent=indent)
>> +                self.__modify_list_variants(temp_list, name,
>> dep_list,
>> +                                            add_to_shortname)
>> +
>> +            new_list += temp_list
>> +
>> +        return new_list
>> +
>> +
>> +    def parse(self, file, list, restricted=False, prev_indent=-1):
>> +        """
>> +        Read and parse lines from file until a line with an indent
>> level lower
>> +        than or equal to prev_indent is encountered.
>> +
>> +        @brief: Parse a file-like object.
>> +        @param file: A file-like object
>> +        @param list: A list of dicts to operate on (list is modified
>> in
>> +        place and should not be used after the call)
>> +        @param restricted: if True, operate in restricted mode
>> +        (prohibit 'variants')
>> +        @param prev_indent: the indent level of the "parent" block
>> +        @return: Return the resulting list of dicts.
>> +        @note: List is destroyed and should not be used after the
>> call.
>> +        Only the returned list should be used.
>> +        """
>> +        while True:
>> +            indent = self.get_next_line_indent(file)
>> +            if indent<= prev_indent:
>> +                break
>> +            indented_line = self.get_next_line(file).rstrip()
>> +            line = indented_line.strip()
>> +            words = line.split()
>> +
>> +            len_list = len(list)
>> +
>> +            # Look for a known operator in the line
>> +            operators = ["?+=", "?<=", "?=", "+=", "<=", "="]
>> +            op_found = None
>> +            op_pos = len(line)
>> +            for op in operators:
>> +                pos = line.find(op)
>> +                if pos>= 0 and pos<  op_pos:
>> +                    op_found = op
>> +                    op_pos = pos
>> +
>> +            # Found an operator?
>> +            if op_found:
>> +                if self.debug and not restricted:
>> +                    self.__debug_print(indented_line,
>> +                                       "Parsing operator (%d dicts in
>> current "
>> +                                       "context)" % len_list)
>> +                (left, value) = self.split_and_strip(line, op_found)
>> +                filters_and_key = self.split_and_strip(left, ":")
>> +                filters = filters_and_key[:-1]
>> +                key = filters_and_key[-1]
>> +                filtered_list = list
>> +                for filter in filters:
>> +                    filtered_list = self.filter(filter,
>> filtered_list)
>> +                # Apply the operation to the filtered list
>> +                if op_found == "=":
>> +                    for dict in filtered_list:
>> +                        dict[key] = value
>> +                elif op_found == "+=":
>> +                    for dict in filtered_list:
>> +                        dict[key] = dict.get(key, "") + value
>> +                elif op_found == "<=":
>> +                    for dict in filtered_list:
>> +                        dict[key] = value + dict.get(key, "")
>> +                elif op_found.startswith("?"):
>> +                    exp = re.compile("^(%s)$" % key)
>> +                    if op_found == "?=":
>> +                        for dict in filtered_list:
>> +                            for key in dict.keys():
>> +                                if exp.match(key):
>> +                                    dict[key] = value
>> +                    elif op_found == "?+=":
>> +                        for dict in filtered_list:
>> +                            for key in dict.keys():
>> +                                if exp.match(key):
>> +                                    dict[key] = dict.get(key, "") +
>> value
>> +                    elif op_found == "?<=":
>> +                        for dict in filtered_list:
>> +                            for key in dict.keys():
>> +                                if exp.match(key):
>> +                                    dict[key] = value + dict.get(key,
>> "")
>> +
>> +            # Parse 'no' and 'only' statements
>> +            elif words[0] == "no" or words[0] == "only":
>> +                if len(words)<= 1:
>> +                    continue
>> +                filters = words[1:]
>> +                filtered_list = []
>> +                if words[0] == "no":
>> +                    for dict in list:
>> +                        for filter in filters:
>> +                            if self.match(filter, dict):
>> +                                break
>> +                        else:
>> +                            filtered_list.append(dict)
>> +                if words[0] == "only":
>> +                    for dict in list:
>> +                        for filter in filters:
>> +                            if self.match(filter, dict):
>> +                                filtered_list.append(dict)
>> +                                break
>> +                list = filtered_list
>> +                if self.debug and not restricted:
>> +                    self.__debug_print(indented_line,
>> +                                       "Parsing no/only (%d dicts in
>> current "
>> +                                       "context, %d remain)" %
>> +                                       (len_list, len(list)))
>> +
>> +            # Parse 'variants'
>> +            elif line == "variants:":
>> +                # 'variants' not allowed in restricted mode
>> +                # (inside an exception or inside subvariants)
>> +                if restricted:
>> +                    e_msg = "Using variants in this context is not
>> allowed"
>> +                    raise error.AutotestError(e_msg)
>> +                if self.debug and not restricted:
>> +                    self.__debug_print(indented_line,
>> +                                       "Entering variants block (%d
>> dicts in "
>> +                                       "current context)" %
>> len_list)
>> +                list = self.parse_variants(file, list,
>> subvariants=False,
>> +                                           prev_indent=indent)
>> +
>> +            # Parse 'subvariants' (the block is parsed for each dict
>> +            # separately)
>> +            elif line == "subvariants:":
>> +                if self.debug and not restricted:
>> +                    self.__debug_print(indented_line,
>> +                                       "Entering subvariants block
>> (%d dicts in "
>> +                                       "current context)" %
>> len_list)
>> +                new_list = []
>> +                # Remember current file position
>> +                pos = file.tell()
>> +                # Read the lines in any case
>> +                self.parse_variants(file, [], subvariants=True,
>> +                                    prev_indent=indent)
>> +                # Iterate over the list...
>> +                for index in range(len(list)):
>> +                    # Revert to initial file position in this
>> 'subvariants'
>> +                    # block
>> +                    file.seek(pos)
>> +                    # Everything inside 'subvariants' should be
>> parsed in
>> +                    # restricted mode
>> +                    new_list += self.parse_variants(file,
>> list[index:index+1],
>> +
>> subvariants=True,
>> +
>> prev_indent=indent)
>> +                list = new_list
>> +
>> +            # Parse 'include' statements
>> +            elif words[0] == "include":
>> +                if len(words)<= 1:
>> +                    continue
>> +                if self.debug and not restricted:
>> +                    self.__debug_print(indented_line,
>> +                                       "Entering file %s" %
>> words[1])
>> +                if self.filename:
>> +                    filename =
>> os.path.join(os.path.dirname(self.filename),
>> +                                            words[1])
>> +                    if os.path.exists(filename):
>> +                        new_file = open(filename, "r")
>> +                        list = self.parse(new_file, list,
>> restricted)
>> +                        new_file.close()
>> +                        if self.debug and not restricted:
>> +                            self.__debug_print("", "Leaving file %s"
>> % words[1])
>> +                    else:
>> +                        logging.warning("Cannot include %s -- file
>> not found",
>> +                                        filename)
>> +                else:
>> +                    logging.warning("Cannot include %s because no
>> file is "
>> +                                    "currently open", words[1])
>> +
>> +            # Parse multi-line exceptions
>> +            # (the block is parsed for each dict separately)
>> +            elif line.endswith(":"):
>> +                if self.debug and not restricted:
>> +                    self.__debug_print(indented_line,
>> +                                       "Entering multi-line exception
>> block "
>> +                                       "(%d dicts in current context
>> outside "
>> +                                       "exception)" % len_list)
>> +                line = line.strip(":")
>> +                new_list = []
>> +                # Remember current file position
>> +                pos = file.tell()
>> +                # Read the lines in any case
>> +                self.parse(file, [], restricted=True,
>> prev_indent=indent)
>> +                # Iterate over the list...
>> +                for index in range(len(list)):
>> +                    if self.match(line, list[index]):
>> +                        # Revert to initial file position in this
>> +                        # exception block
>> +                        file.seek(pos)
>> +                        # Everything inside an exception should be
>> parsed in
>> +                        # restricted mode
>> +                        new_list += self.parse(file,
>> list[index:index+1],
>> +                                               restricted=True,
>> +                                               prev_indent=indent)
>> +                    else:
>> +                        new_list += list[index:index+1]
>> +                list = new_list
>> +
>> +        return list
>> +
>> +
>> +    def __debug_print(self, str1, str2=""):
>> +        """
>> +        Nicely print two strings and an arrow.
>> +
>> +        @param str1: First string
>> +        @param str2: Second string
>> +        """
>> +        if str2:
>> +            str = "%-50s --->  %s" % (str1, str2)
>> +        else:
>> +            str = str1
>> +        logging.debug(str)
>> +
>> +
>> +    def __modify_list_variants(self, list, name, dep_list,
>> add_to_shortname):
>> +        """
>> +        Make some modifications to list, as part of parsing a
>> 'variants' block.
>> +
>> +        @param list: List to be processed
>> +        @param name: Name to be prepended to the dictionary's 'name'
>> key
>> +        @param dep_list: List of dependencies to be added to the
>> dictionary's
>> +                'depend' key
>> +        @param add_to_shortname: Boolean indicating whether name
>> should be
>> +                prepended to the dictionary's 'shortname' key as
>> well
>> +        """
>> +        for dict in list:
>> +            # Prepend name to the dict's 'name' field
>> +            dict["name"] = self.add_name(dict["name"], name)
>> +            # Prepend name to the dict's 'shortname' field
>> +            if add_to_shortname:
>> +                dict["shortname"] = self.add_name(dict["shortname"],
>> name)
>> +            # Prepend name to each of the dict's dependencies
>> +            for i in range(len(dict["depend"])):
>> +                dict["depend"][i] = self.add_name(dict["depend"][i],
>> name)
>> +            # Add new dependencies
>> +            dict["depend"] += dep_list
>> +
>> +
>> +    def __modify_list_subvariants(self, list, name, dep_list,
>> add_to_shortname):
>> +        """
>> +        Make some modifications to list, as part of parsing a
>> 'subvariants'
>> +        block.
>> +
>> +        @param list: List to be processed
>> +        @param name: Name to be appended to the dictionary's 'name'
>> key
>> +        @param dep_list: List of dependencies to be added to the
>> dictionary's
>> +                'depend' key
>> +        @param add_to_shortname: Boolean indicating whether name
>> should be
>> +                appended to the dictionary's 'shortname' as well
>> +        """
>> +        for dict in list:
>> +            # Add new dependencies
>> +            for dep in dep_list:
>> +                dep_name = self.add_name(dict["name"], dep,
>> append=True)
>> +                dict["depend"].append(dep_name)
>> +            # Append name to the dict's 'name' field
>> +            dict["name"] = self.add_name(dict["name"], name,
>> append=True)
>> +            # Append name to the dict's 'shortname' field
>> +            if add_to_shortname:
>> +                dict["shortname"] = self.add_name(dict["shortname"],
>> name,
>> +                                                  append=True)
>> +
>> +
>> +def create_control(dict_list, control_path):
>> +    """
>> +    Creates a kvm test control file from a given test list
>> dictionary.
>> +
>> +    @param dict_list: A list with dictionaries representing kvm test
>> parameters.
>> +    @param control_path: Path to the kvm control file that will be
>> generated.
>> +    """
>> +    indent = "    "
>> +    indent_level = 0
>> +    control_file = open(control_path, "w")
>> +    control_file.write("# Control file generated by
>> create_control.py\n")
>> +    control_file.write("kvm_test_dir =
>> os.path.join(os.environ['AUTODIR'], "
>> +                       "'tests/kvm')\n")
>> +    control_file.write("sys.path.append(kvm_test_dir)\n")
>> +
>> +    while dict_list:
>> +        current_dict = dict_list[0]
>> +        test_iterations = int(current_dict.get("iterations", 1))
>> +        test_tag = current_dict.get("shortname")
>> +
>> +        if len(current_dict.get("depend")) == 0:
>> +            indent_level = 0
>> +
>> +        try:
>> +            future_dict = dict_list[1]
>> +        except IndexError:
>> +            control_file.write("%sjob.run_test('kvm', params=%s,
>> tag='%s', "
>> +                               "iterations=%s)\n" % (indent *
>> indent_level,
>> +                                                     current_dict,
>> test_tag,
>> +
>> test_iterations))
>> +            break
>> +
>> +        if current_dict.get("name") in future_dict.get("depend"):
>> +            control_file.write("%sif job.run_test('kvm', params=%s,
>> tag='%s', "
>> +                               "iterations=%s):\n" % (indent *
>> indent_level,
>> +                                                      current_dict,
>> test_tag,
>> +
>> test_iterations))
>> +            indent_level += 1
>> +        else:
>> +            control_file.write("%sjob.run_test('kvm', params=%s,
>> tag='%s', "
>> +                               "iterations=%s)\n" % (indent *
>> indent_level,
>> +                                                     current_dict,
>> test_tag,
>> +
>> test_iterations))
>> +        dict_list.pop(0)
>> +        continue
>> +
>> +    control_file.close()
>> +
>> +
>> +if __name__ == "__main__":
>> +    parser = optparse.OptionParser()
>> +    parser.add_option('-f', '--file', dest="filename",
>> action='store',
>> +                      help='path to a config file that will be
>> parsed. '
>> +                           'If not specified, will parse
>> kvm_tests.cfg '
>> +                           'located inside the kvm test dir.')
>> +    parser.add_option('-c', '--control', dest="control_path",
>> action='store',
>> +                      help='path to an output control file. If not
>> specified, '
>> +                           'will generate a file called "control" at
>> the top '
>> +                           'of the kvm test directory.')
>> +    parser.add_option('--verbose', dest="debug",
>> action='store_true',
>> +                      help='include debug messages in console
>> output')
>> +    options, args = parser.parse_args()
>> +    filename = options.filename
>> +    control_path = options.control_path
>> +    debug = options.debug
>> +
>> +    if not filename:
>> +        filename = os.path.join(os.path.dirname(sys.argv[0]),
>> "tests.cfg")
>> +
>> +    # Here we configure the stand alone program to use the autotest
>> +    # logging system.
>> +    logging_manager.configure_logging(kvm_utils.KvmLoggingConfig(),
>> verbose=debug)
>> +    list = config(filename, debug=debug).get_list()
>> +    i = 0
>> +    logging.info("List of dictionaries generated from config file
>> %s:",
>> +                 filename)
>> +    for dict in list:
>> +        logging.info("Dictionary #%d:", i)
>> +        keys = dict.keys()
>> +        keys.sort()
>> +        for key in keys:
>> +            logging.info("    %s = %s", key, dict[key])
>> +        i += 1
>> +
>> +    if not control_path:
>> +        control_path = os.path.join(os.path.dirname(sys.argv[0]),
>> "control")
>> +
>> +    logging.info("Creating control file %s from config file",
>> control_path)
>> +
>> +    create_control(list, control_path)
>> diff --git a/client/tests/kvm/kvm_config.py
>> b/client/tests/kvm/kvm_config.py
>> deleted file mode 100755
>> index 656f6b3..0000000
>> --- a/client/tests/kvm/kvm_config.py
>> +++ /dev/null
>> @@ -1,524 +0,0 @@
>> -#!/usr/bin/python
>> -"""
>> -KVM configuration file utility functions.
>> -
>> -@copyright: Red Hat 2008-2009
>> -"""
>> -
>> -import logging, re, os, sys, StringIO, optparse
>> -import common
>> -import kvm_utils
>> -from autotest_lib.client.common_lib import error
>> -from autotest_lib.client.common_lib import logging_config,
>> logging_manager
>> -
>> -
>> -class config:
>> -    """
>> -    Parse an input file or string that follows the KVM Test Config
>> File format
>> -    and generate a list of dicts that will be later used as
>> configuration
>> -    parameters by the the KVM tests.
>> -
>> -    @see:
>> http://www.linux-kvm.org/page/KVM-Autotest/Test_Config_File
>> -    """
>> -
>> -    def __init__(self, filename=None, debug=False):
>> -        """
>> -        Initialize the list and optionally parse filename.
>> -
>> -        @param filename: Path of the file that will be taken.
>> -        @param debug: Whether to turn debugging output.
>> -        """
>> -        self.list = [{"name": "", "shortname": "", "depend": []}]
>> -        self.debug = debug
>> -        self.filename = filename
>> -        if filename:
>> -            self.parse_file(filename)
>> -
>> -
>> -    def parse_file(self, filename):
>> -        """
>> -        Parse filename, return the resulting list and store it in
>> .list. If
>> -        filename does not exist, raise an exception.
>> -
>> -        @param filename: Path of the configuration file.
>> -        """
>> -        if not os.path.exists(filename):
>> -            raise Exception, "File %s not found" % filename
>> -        self.filename = filename
>> -        file = open(filename, "r")
>> -        self.list = self.parse(file, self.list)
>> -        file.close()
>> -        return self.list
>> -
>> -
>> -    def parse_string(self, str):
>> -        """
>> -        Parse a string, return the resulting list and store it in
>> .list.
>> -
>> -        @param str: String that will be parsed.
>> -        """
>> -        file = StringIO.StringIO(str)
>> -        self.list = self.parse(file, self.list)
>> -        file.close()
>> -        return self.list
>> -
>> -
>> -    def get_list(self):
>> -        """
>> -        Return the list of dictionaries. This should probably be
>> called after
>> -        parsing something.
>> -        """
>> -        return self.list
>> -
>> -
>> -    def match(self, filter, dict):
>> -        """
>> -        Return True if dict matches filter.
>> -
>> -        @param filter: A regular expression that defines the filter.
>> -        @param dict: Dictionary that will be inspected.
>> -        """
>> -        filter = re.compile(r"(\.|^)(%s)(\.|$)" % filter)
>> -        return bool(filter.search(dict["name"]))
>> -
>> -
>> -    def filter(self, filter, list=None):
>> -        """
>> -        Filter a list of dicts.
>> -
>> -        @param filter: A regular expression that will be used as a
>> filter.
>> -        @param list: A list of dictionaries that will be filtered.
>> -        """
>> -        if list is None:
>> -            list = self.list
>> -        return [dict for dict in list if self.match(filter, dict)]
>> -
>> -
>> -    def split_and_strip(self, str, sep="="):
>> -        """
>> -        Split str and strip quotes from the resulting parts.
>> -
>> -        @param str: String that will be processed
>> -        @param sep: Separator that will be used to split the string
>> -        """
>> -        temp = str.split(sep, 1)
>> -        for i in range(len(temp)):
>> -            temp[i] = temp[i].strip()
>> -            if re.findall("^\".*\"$", temp[i]):
>> -                temp[i] = temp[i].strip("\"")
>> -            elif re.findall("^\'.*\'$", temp[i]):
>> -                temp[i] = temp[i].strip("\'")
>> -        return temp
>> -
>> -
>> -    def get_next_line(self, file):
>> -        """
>> -        Get the next non-empty, non-comment line in a file like
>> object.
>> -
>> -        @param file: File like object
>> -        @return: If no line is available, return None.
>> -        """
>> -        while True:
>> -            line = file.readline()
>> -            if line == "": return None
>> -            stripped_line = line.strip()
>> -            if len(stripped_line)>  0 \
>> -                    and not stripped_line.startswith('#') \
>> -                    and not stripped_line.startswith('//'):
>> -                return line
>> -
>> -
>> -    def get_next_line_indent(self, file):
>> -        """
>> -        Return the indent level of the next non-empty, non-comment
>> line in file.
>> -
>> -        @param file: File like object.
>> -        @return: If no line is available, return -1.
>> -        """
>> -        pos = file.tell()
>> -        line = self.get_next_line(file)
>> -        if not line:
>> -            file.seek(pos)
>> -            return -1
>> -        line = line.expandtabs()
>> -        indent = 0
>> -        while line[indent] == ' ':
>> -            indent += 1
>> -        file.seek(pos)
>> -        return indent
>> -
>> -
>> -    def add_name(self, str, name, append=False):
>> -        """
>> -        Add name to str with a separator dot and return the result.
>> -
>> -        @param str: String that will be processed
>> -        @param name: name that will be appended to the string.
>> -        @return: If append is True, append name to str.
>> -                Otherwise, pre-pend name to str.
>> -        """
>> -        if str == "":
>> -            return name
>> -        # Append?
>> -        elif append:
>> -            return str + "." + name
>> -        # Prepend?
>> -        else:
>> -            return name + "." + str
>> -
>> -
>> -    def parse_variants(self, file, list, subvariants=False,
>> prev_indent=-1):
>> -        """
>> -        Read and parse lines from file like object until a line with
>> an indent
>> -        level lower than or equal to prev_indent is encountered.
>> -
>> -        @brief: Parse a 'variants' or 'subvariants' block from a
>> file-like
>> -        object.
>> -        @param file: File-like object that will be parsed
>> -        @param list: List of dicts to operate on
>> -        @param subvariants: If True, parse in 'subvariants' mode;
>> -        otherwise parse in 'variants' mode
>> -        @param prev_indent: The indent level of the "parent" block
>> -        @return: The resulting list of dicts.
>> -        """
>> -        new_list = []
>> -
>> -        while True:
>> -            indent = self.get_next_line_indent(file)
>> -            if indent<= prev_indent:
>> -                break
>> -            indented_line = self.get_next_line(file).rstrip()
>> -            line = indented_line.strip()
>> -
>> -            # Get name and dependencies
>> -            temp = line.strip("- ").split(":")
>> -            name = temp[0]
>> -            if len(temp) == 1:
>> -                dep_list = []
>> -            else:
>> -                dep_list = temp[1].split()
>> -
>> -            # See if name should be added to the 'shortname' field
>> -            add_to_shortname = True
>> -            if name.startswith("@"):
>> -                name = name.strip("@")
>> -                add_to_shortname = False
>> -
>> -            # Make a deep copy of list
>> -            temp_list = []
>> -            for dict in list:
>> -                new_dict = dict.copy()
>> -                new_dict["depend"] = dict["depend"][:]
>> -                temp_list.append(new_dict)
>> -
>> -            if subvariants:
>> -                # If we're parsing 'subvariants', first modify the
>> list
>> -                self.__modify_list_subvariants(temp_list, name,
>> dep_list,
>> -                                               add_to_shortname)
>> -                temp_list = self.parse(file, temp_list,
>> -                        restricted=True, prev_indent=indent)
>> -            else:
>> -                # If we're parsing 'variants', parse before modifying
>> the list
>> -                if self.debug:
>> -                    self.__debug_print(indented_line,
>> -                                       "Entering variant '%s' "
>> -                                       "(variant inherits %d dicts)"
>> %
>> -                                       (name, len(list)))
>> -                temp_list = self.parse(file, temp_list,
>> -                        restricted=False, prev_indent=indent)
>> -                self.__modify_list_variants(temp_list, name,
>> dep_list,
>> -                                            add_to_shortname)
>> -
>> -            new_list += temp_list
>> -
>> -        return new_list
>> -
>> -
>> -    def parse(self, file, list, restricted=False, prev_indent=-1):
>> -        """
>> -        Read and parse lines from file until a line with an indent
>> level lower
>> -        than or equal to prev_indent is encountered.
>> -
>> -        @brief: Parse a file-like object.
>> -        @param file: A file-like object
>> -        @param list: A list of dicts to operate on (list is modified
>> in
>> -        place and should not be used after the call)
>> -        @param restricted: if True, operate in restricted mode
>> -        (prohibit 'variants')
>> -        @param prev_indent: the indent level of the "parent" block
>> -        @return: Return the resulting list of dicts.
>> -        @note: List is destroyed and should not be used after the
>> call.
>> -        Only the returned list should be used.
>> -        """
>> -        while True:
>> -            indent = self.get_next_line_indent(file)
>> -            if indent<= prev_indent:
>> -                break
>> -            indented_line = self.get_next_line(file).rstrip()
>> -            line = indented_line.strip()
>> -            words = line.split()
>> -
>> -            len_list = len(list)
>> -
>> -            # Look for a known operator in the line
>> -            operators = ["?+=", "?<=", "?=", "+=", "<=", "="]
>> -            op_found = None
>> -            op_pos = len(line)
>> -            for op in operators:
>> -                pos = line.find(op)
>> -                if pos>= 0 and pos<  op_pos:
>> -                    op_found = op
>> -                    op_pos = pos
>> -
>> -            # Found an operator?
>> -            if op_found:
>> -                if self.debug and not restricted:
>> -                    self.__debug_print(indented_line,
>> -                                       "Parsing operator (%d dicts in
>> current "
>> -                                       "context)" % len_list)
>> -                (left, value) = self.split_and_strip(line, op_found)
>> -                filters_and_key = self.split_and_strip(left, ":")
>> -                filters = filters_and_key[:-1]
>> -                key = filters_and_key[-1]
>> -                filtered_list = list
>> -                for filter in filters:
>> -                    filtered_list = self.filter(filter,
>> filtered_list)
>> -                # Apply the operation to the filtered list
>> -                if op_found == "=":
>> -                    for dict in filtered_list:
>> -                        dict[key] = value
>> -                elif op_found == "+=":
>> -                    for dict in filtered_list:
>> -                        dict[key] = dict.get(key, "") + value
>> -                elif op_found == "<=":
>> -                    for dict in filtered_list:
>> -                        dict[key] = value + dict.get(key, "")
>> -                elif op_found.startswith("?"):
>> -                    exp = re.compile("^(%s)$" % key)
>> -                    if op_found == "?=":
>> -                        for dict in filtered_list:
>> -                            for key in dict.keys():
>> -                                if exp.match(key):
>> -                                    dict[key] = value
>> -                    elif op_found == "?+=":
>> -                        for dict in filtered_list:
>> -                            for key in dict.keys():
>> -                                if exp.match(key):
>> -                                    dict[key] = dict.get(key, "") +
>> value
>> -                    elif op_found == "?<=":
>> -                        for dict in filtered_list:
>> -                            for key in dict.keys():
>> -                                if exp.match(key):
>> -                                    dict[key] = value + dict.get(key,
>> "")
>> -
>> -            # Parse 'no' and 'only' statements
>> -            elif words[0] == "no" or words[0] == "only":
>> -                if len(words)<= 1:
>> -                    continue
>> -                filters = words[1:]
>> -                filtered_list = []
>> -                if words[0] == "no":
>> -                    for dict in list:
>> -                        for filter in filters:
>> -                            if self.match(filter, dict):
>> -                                break
>> -                        else:
>> -                            filtered_list.append(dict)
>> -                if words[0] == "only":
>> -                    for dict in list:
>> -                        for filter in filters:
>> -                            if self.match(filter, dict):
>> -                                filtered_list.append(dict)
>> -                                break
>> -                list = filtered_list
>> -                if self.debug and not restricted:
>> -                    self.__debug_print(indented_line,
>> -                                       "Parsing no/only (%d dicts in
>> current "
>> -                                       "context, %d remain)" %
>> -                                       (len_list, len(list)))
>> -
>> -            # Parse 'variants'
>> -            elif line == "variants:":
>> -                # 'variants' not allowed in restricted mode
>> -                # (inside an exception or inside subvariants)
>> -                if restricted:
>> -                    e_msg = "Using variants in this context is not
>> allowed"
>> -                    raise error.AutotestError(e_msg)
>> -                if self.debug and not restricted:
>> -                    self.__debug_print(indented_line,
>> -                                       "Entering variants block (%d
>> dicts in "
>> -                                       "current context)" %
>> len_list)
>> -                list = self.parse_variants(file, list,
>> subvariants=False,
>> -                                           prev_indent=indent)
>> -
>> -            # Parse 'subvariants' (the block is parsed for each dict
>> -            # separately)
>> -            elif line == "subvariants:":
>> -                if self.debug and not restricted:
>> -                    self.__debug_print(indented_line,
>> -                                       "Entering subvariants block
>> (%d dicts in "
>> -                                       "current context)" %
>> len_list)
>> -                new_list = []
>> -                # Remember current file position
>> -                pos = file.tell()
>> -                # Read the lines in any case
>> -                self.parse_variants(file, [], subvariants=True,
>> -                                    prev_indent=indent)
>> -                # Iterate over the list...
>> -                for index in range(len(list)):
>> -                    # Revert to initial file position in this
>> 'subvariants'
>> -                    # block
>> -                    file.seek(pos)
>> -                    # Everything inside 'subvariants' should be
>> parsed in
>> -                    # restricted mode
>> -                    new_list += self.parse_variants(file,
>> list[index:index+1],
>> -
>> subvariants=True,
>> -
>> prev_indent=indent)
>> -                list = new_list
>> -
>> -            # Parse 'include' statements
>> -            elif words[0] == "include":
>> -                if len(words)<= 1:
>> -                    continue
>> -                if self.debug and not restricted:
>> -                    self.__debug_print(indented_line,
>> -                                       "Entering file %s" %
>> words[1])
>> -                if self.filename:
>> -                    filename =
>> os.path.join(os.path.dirname(self.filename),
>> -                                            words[1])
>> -                    if os.path.exists(filename):
>> -                        new_file = open(filename, "r")
>> -                        list = self.parse(new_file, list,
>> restricted)
>> -                        new_file.close()
>> -                        if self.debug and not restricted:
>> -                            self.__debug_print("", "Leaving file %s"
>> % words[1])
>> -                    else:
>> -                        logging.warning("Cannot include %s -- file
>> not found",
>> -                                        filename)
>> -                else:
>> -                    logging.warning("Cannot include %s because no
>> file is "
>> -                                    "currently open", words[1])
>> -
>> -            # Parse multi-line exceptions
>> -            # (the block is parsed for each dict separately)
>> -            elif line.endswith(":"):
>> -                if self.debug and not restricted:
>> -                    self.__debug_print(indented_line,
>> -                                       "Entering multi-line exception
>> block "
>> -                                       "(%d dicts in current context
>> outside "
>> -                                       "exception)" % len_list)
>> -                line = line.strip(":")
>> -                new_list = []
>> -                # Remember current file position
>> -                pos = file.tell()
>> -                # Read the lines in any case
>> -                self.parse(file, [], restricted=True,
>> prev_indent=indent)
>> -                # Iterate over the list...
>> -                for index in range(len(list)):
>> -                    if self.match(line, list[index]):
>> -                        # Revert to initial file position in this
>> -                        # exception block
>> -                        file.seek(pos)
>> -                        # Everything inside an exception should be
>> parsed in
>> -                        # restricted mode
>> -                        new_list += self.parse(file,
>> list[index:index+1],
>> -                                               restricted=True,
>> -                                               prev_indent=indent)
>> -                    else:
>> -                        new_list += list[index:index+1]
>> -                list = new_list
>> -
>> -        return list
>> -
>> -
>> -    def __debug_print(self, str1, str2=""):
>> -        """
>> -        Nicely print two strings and an arrow.
>> -
>> -        @param str1: First string
>> -        @param str2: Second string
>> -        """
>> -        if str2:
>> -            str = "%-50s --->  %s" % (str1, str2)
>> -        else:
>> -            str = str1
>> -        logging.debug(str)
>> -
>> -
>> -    def __modify_list_variants(self, list, name, dep_list,
>> add_to_shortname):
>> -        """
>> -        Make some modifications to list, as part of parsing a
>> 'variants' block.
>> -
>> -        @param list: List to be processed
>> -        @param name: Name to be prepended to the dictionary's 'name'
>> key
>> -        @param dep_list: List of dependencies to be added to the
>> dictionary's
>> -                'depend' key
>> -        @param add_to_shortname: Boolean indicating whether name
>> should be
>> -                prepended to the dictionary's 'shortname' key as
>> well
>> -        """
>> -        for dict in list:
>> -            # Prepend name to the dict's 'name' field
>> -            dict["name"] = self.add_name(dict["name"], name)
>> -            # Prepend name to the dict's 'shortname' field
>> -            if add_to_shortname:
>> -                dict["shortname"] = self.add_name(dict["shortname"],
>> name)
>> -            # Prepend name to each of the dict's dependencies
>> -            for i in range(len(dict["depend"])):
>> -                dict["depend"][i] = self.add_name(dict["depend"][i],
>> name)
>> -            # Add new dependencies
>> -            dict["depend"] += dep_list
>> -
>> -
>> -    def __modify_list_subvariants(self, list, name, dep_list,
>> add_to_shortname):
>> -        """
>> -        Make some modifications to list, as part of parsing a
>> 'subvariants'
>> -        block.
>> -
>> -        @param list: List to be processed
>> -        @param name: Name to be appended to the dictionary's 'name'
>> key
>> -        @param dep_list: List of dependencies to be added to the
>> dictionary's
>> -                'depend' key
>> -        @param add_to_shortname: Boolean indicating whether name
>> should be
>> -                appended to the dictionary's 'shortname' as well
>> -        """
>> -        for dict in list:
>> -            # Add new dependencies
>> -            for dep in dep_list:
>> -                dep_name = self.add_name(dict["name"], dep,
>> append=True)
>> -                dict["depend"].append(dep_name)
>> -            # Append name to the dict's 'name' field
>> -            dict["name"] = self.add_name(dict["name"], name,
>> append=True)
>> -            # Append name to the dict's 'shortname' field
>> -            if add_to_shortname:
>> -                dict["shortname"] = self.add_name(dict["shortname"],
>> name,
>> -                                                  append=True)
>> -
>> -
>> -if __name__ == "__main__":
>> -    parser = optparse.OptionParser()
>> -    parser.add_option('-f', '--file', dest="filename",
>> action='store',
>> -                      help='path to a config file that will be
>> parsed. '
>> -                           'If not specified, will parse
>> kvm_tests.cfg '
>> -                           'located inside the kvm test dir.')
>> -    parser.add_option('--verbose', dest="debug",
>> action='store_true',
>> -                      help='include debug messages in console
>> output')
>> -
>> -    options, args = parser.parse_args()
>> -    filename = options.filename
>> -    debug = options.debug
>> -
>> -    if not filename:
>> -        filename = os.path.join(os.path.dirname(sys.argv[0]),
>> "tests.cfg")
>> -
>> -    # Here we configure the stand alone program to use the autotest
>> -    # logging system.
>> -    logging_manager.configure_logging(kvm_utils.KvmLoggingConfig(),
>> verbose=debug)
>> -    list = config(filename, debug=debug).get_list()
>> -    i = 0
>> -    for dict in list:
>> -        logging.info("Dictionary #%d:", i)
>> -        keys = dict.keys()
>> -        keys.sort()
>> -        for key in keys:
>> -            logging.info("    %s = %s", key, dict[key])
>> -        i += 1
>> --
>> 1.6.6.1
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html