[PATCH 1/3] KVM test: Add new utility functions to kvm_utils

[PATCH 1/3] KVM test: Add new utility functions to kvm_utils

[Lucas Meneghel Rodrigues]

Some distributors ship CD and DVD files with SHA1 hash sums instead
of MD5 hash sums, so let's extend the kvm_utils functions to
evaluate and compare SHA1 hashes:

* sha1sum_file(): Calculate SHA1 sum for file
* unmap_url_cache(): Reimplementation of a function present on
autotest utils that downloads a file and caches it. The reason
I'm keeping it is that I want more testing before I move all
needed function definitions to the autotest API
* get_hash_from_file(): Extract hash string from a file containing
hashes

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
---
 client/tests/kvm/kvm_utils.py |  106 ++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 104 insertions(+), 2 deletions(-)

diff --git a/client/tests/kvm/kvm_utils.py b/client/tests/kvm/kvm_utils.py
index 53b664a..f1a6b4b 100644
--- a/client/tests/kvm/kvm_utils.py
+++ b/client/tests/kvm/kvm_utils.py
@@ -4,8 +4,8 @@ KVM test utility functions.
 @copyright: 2008-2009 Red Hat Inc.
 """
 
-import md5, thread, subprocess, time, string, random, socket, os, signal, pty
-import select, re, logging, commands, cPickle
+import md5, sha, thread, subprocess, time, string, random, socket, os, signal
+import select, re, logging, commands, cPickle, pty
 from autotest_lib.client.bin import utils
 from autotest_lib.client.common_lib import error
 import kvm_subprocess
@@ -788,3 +788,105 @@ def md5sum_file(filename, size=None):
         size -= len(data)
     f.close()
     return o.hexdigest()
+
+
+def sha1sum_file(filename, size=None):
+    """
+    Calculate the sha1sum of filename.
+    If size is not None, limit to first size bytes.
+    Throw exception if something is wrong with filename.
+    Can be also implemented with bash one-liner (assuming size%1024==0):
+    dd if=filename bs=1024 count=size/1024 | sha1sum -
+
+    @param filename: Path of the file that will have its sha1sum calculated.
+    @param returns: sha1sum of the file.
+    """
+    chunksize = 4096
+    fsize = os.path.getsize(filename)
+    if not size or size>fsize:
+        size = fsize
+    f = open(filename, 'rb')
+    o = sha.new()
+    while size > 0:
+        if chunksize > size:
+            chunksize = size
+        data = f.read(chunksize)
+        if len(data) == 0:
+            logging.debug("Nothing left to read but size=%d" % size)
+            break
+        o.update(data)
+        size -= len(data)
+    f.close()
+    return o.hexdigest()
+
+
+def unmap_url_cache(cachedir, url, expected_hash, method="md5"):
+    """
+    Downloads a file from a URL to a cache directory. If the file is already
+    at the expected position and has the expected hash, let's not download it
+    again.
+
+    @param cachedir: Directory that might hold a copy of the file we want to
+            download.
+    @param url: URL for the file we want to download.
+    @param expected_hash: Hash string that we expect the file downloaded to
+            have.
+    @param method: Method used to calculate the hash string (md5, sha1).
+    """
+    # Let's convert cachedir to a canonical path, if it's not already
+    cachedir = os.path.realpath(cachedir)
+    if not os.path.isdir(cachedir):
+        try:
+            os.makedirs(cachedir)
+        except:
+            raise ValueError('Could not create cache directory %s' % cachedir)
+    file_from_url = os.path.basename(url)
+    file_local_path = os.path.join(cachedir, file_from_url)
+
+    file_hash = None
+    failure_counter = 0
+    while not file_hash == expected_hash:
+        if os.path.isfile(file_local_path):
+            if method == "md5":
+                file_hash = md5sum_file(file_local_path)
+            elif method == "sha1":
+                file_hash = sha1sum_file(file_local_path)
+
+            if file_hash == expected_hash:
+                # File is already at the expected position and ready to go
+                src = file_from_url
+            else:
+                # Let's download the package again, it's corrupted...
+                logging.error("Seems that file %s is corrupted, trying to "
+                              "download it again" % file_from_url)
+                src = url
+                failure_counter += 1
+        else:
+            # File is not there, let's download it
+            src = url
+        if failure_counter > 1:
+            raise EnvironmentError("Consistently failed to download the "
+                                   "package %s. Aborting further download "
+                                   "attempts. This might mean either the "
+                                   "network connection has problems or the "
+                                   "expected hash string that was determined "
+                                   "for this file is wrong" % file_from_url)
+        file_path = utils.unmap_url(cachedir, src, cachedir)
+
+    return file_path
+
+
+def get_hash_from_file(sha_path, dvd_basename):
+    """
+    Get the a hash from a given DVD image from a hash file
+    (Hash files are usually named MD5SUM or SHA1SUM and are located inside the
+    download directories of the DVDs)
+
+    @param hash_path: Local path to a hash file.
+    @param cd_image: Basename of a CD image
+    """
+    hash_file = open(sha_path, 'r')
+    for line in hash_file.readlines():
+        if dvd_basename in line:
+            return line.split()[0]
+
-- 
1.6.2.5

[PATCH 2/3] KVM test: Daily DVD test control file

[Lucas Meneghel Rodrigues]

This control file is a proof of concept of a control file
that downloads a iso DVD image produced on a daily basis
to the isos directory, gets the hash sums for the DVD
images and verifies the integrity of the download.

Makes use of the utility functions introduced on
previous patches.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
---
 client/tests/kvm/control.daily_dvd |  249 ++++++++++++++++++++++++++++++++++++
 1 files changed, 249 insertions(+), 0 deletions(-)
 create mode 100644 client/tests/kvm/control.daily_dvd

diff --git a/client/tests/kvm/control.daily_dvd b/client/tests/kvm/control.daily_dvd
new file mode 100644
index 0000000..07dade6
--- /dev/null
+++ b/client/tests/kvm/control.daily_dvd
@@ -0,0 +1,249 @@
+AUTHOR = """
+uril@redhat.com (Uri Lublin)
+drusso@redhat.com (Dror Russo)
+mgoldish@redhat.com (Michael Goldish)
+dhuff@redhat.com (David Huff)
+aeromenk@redhat.com (Alexey Eromenko)
+mburns@redhat.com (Mike Burns)
+"""
+TIME = 'SHORT'
+NAME = 'KVM test'
+TEST_TYPE = 'client'
+TEST_CLASS = 'Virtualization'
+TEST_CATEGORY = 'Functional'
+
+DOC = """
+Executes the KVM test framework on a given host. This module is separated in
+minor functions, that execute different tests for doing Quality Assurance on
+KVM (both kernelspace and userspace) code.
+
+For online docs, please refer to http://www.linux-kvm.org/page/KVM-Autotest
+
+This control file tests daily DVD/CDs for a given distro, adding MD5/SHA1
+checksums and executing unattended installs of that distro.
+"""
+
+
+import sys, os
+
+#-----------------------------------------------------------------------------
+# set English environment (command output might be localized, need to be safe)
+#-----------------------------------------------------------------------------
+os.environ['LANG'] = 'en_US.UTF-8'
+
+#---------------------------------------------------------
+# Enable modules import from current directory (tests/kvm)
+#---------------------------------------------------------
+pwd = os.path.join(os.environ['AUTODIR'],'tests/kvm')
+sys.path.append(pwd)
+
+# ------------------------
+# create required symlinks
+# ------------------------
+# When dispatching tests from autotest-server the links we need do not exist on
+# the host (the client). The following lines create those symlinks. Change
+# 'rootdir' here and/or mount appropriate directories in it.
+#
+# When dispatching tests on local host (client mode) one can either setup kvm
+# links, or same as server mode use rootdir and set all appropriate links and
+# mount-points there. For example, guest installation tests need to know where
+# to find the iso-files.
+#
+# We create the links only if not already exist, so if one already set up the
+# links for client/local run we do not touch the links.
+rootdir='/tmp/kvm_autotest_root'
+iso=os.path.join(rootdir, 'iso')
+images=os.path.join(rootdir, 'images')
+qemu=os.path.join(rootdir, 'qemu')
+qemu_img=os.path.join(rootdir, 'qemu-img')
+
+
+def link_if_not_exist(ldir, target, link_name):
+    t = target
+    l = os.path.join(ldir, link_name)
+    if not os.path.exists(l):
+        os.system('ln -s %s %s' % (t, l))
+
+# Create links only if not already exist
+link_if_not_exist(pwd, '../../', 'autotest')
+link_if_not_exist(pwd, iso, 'isos')
+link_if_not_exist(pwd, images, 'images')
+link_if_not_exist(pwd, qemu, 'qemu')
+link_if_not_exist(pwd, qemu_img, 'qemu-img')
+
+# --------------------------------------------------------
+# Params that will be passed to the KVM install/build test
+# --------------------------------------------------------
+params = {
+    "name": "build",
+    "shortname": "build",
+    "type": "build",
+    "mode": "release",
+    #"mode": "snapshot",
+    #"mode": "localtar",
+    #"mode": "localsrc",
+    #"mode": "git",
+    #"mode": "noinstall",
+    #"mode": "koji",
+
+    ## Are we going to load modules built by this test?
+    ## Defaults to 'yes', so if you are going to provide only userspace code to
+    ## be built by this test, please set load_modules to 'no', and make sure
+    ## the kvm and kvm-[vendor] module is already loaded by the time you start
+    ## it.
+    "load_modules": "no",
+
+    ## Install from a kvm release ("mode": "release"). You can optionally
+    ## specify a release tag. If you omit it, the test will get the latest
+    ## release tag available.
+    #"release_tag": '84',
+    "release_dir": 'http://downloads.sourceforge.net/project/kvm/',
+    # This is the place that contains the sourceforge project list of files
+    "release_listing": 'http://sourceforge.net/projects/kvm/files/',
+
+    ## Install from a kvm snapshot location ("mode": "snapshot"). You can
+    ## optionally specify a snapshot date. If you omit it, the test will get
+    ## yesterday's snapshot.
+    #"snapshot_date": '20090712'
+    #"snapshot_dir": 'http://foo.org/kvm-snapshots/',
+
+    ## Install from a tarball ("mode": "localtar")
+    #"tarball": "/tmp/kvm-84.tar.gz",
+
+    ## Install from a local source code dir ("mode": "localsrc")
+    #"srcdir": "/path/to/source-dir"
+
+    ## Install from koji build server ("mode": "koji")
+    ## Koji is the Fedora Project buildserver. It is possible to install
+    ## packages right from Koji if you provide a release tag or a build.
+    ## Tag (if available)
+    #"koji_tag": 'dist-f11',
+    ## Build (if available, is going to override tag).
+    #"koji_build": 'qemu-0.10-16.fc11',
+    ## Command to interact with the build server
+    #"koji_cmd": '/usr/bin/koji',
+    ## The name of the source package that's being built
+    #"src_pkg": 'qemu',
+    ## Name of the rpms we need installed
+    #"pkg_list": ['qemu-kvm', 'qemu-kvm-tools', 'qemu-system-x86', 'qemu-common', 'qemu-img'],
+    ## Paths of the qemu relevant executables that should be checked
+    #"qemu_bin_paths": ['/usr/bin/qemu-kvm', '/usr/bin/qemu-img'],
+
+    ## Install from git ("mode": "git")
+    ## If you provide only "git_repo" and "user_git_repo", the build test
+    ## will assume it will perform all build from the userspace dir, building
+    ## modules trough make -C kernel LINUX=%s sync. As of today (07-13-2009)
+    ## we need 3 git repos, "git_repo" (linux sources), "user_git_repo" and 
+    ## "kmod_repo" to build KVM userspace + kernel modules.
+    #"git_repo": 'git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git',
+    #"kernel_branch": 'kernel_branch_name',
+    #"kernel_lbranch": 'kernel_lbranch_name',
+    #"kernel_tag": 'kernel_tag_name',
+    #"user_git_repo": 'git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git',
+    #"user_branch": 'user_branch_name',
+    #"user_lbranch": 'user_lbranch_name',
+    #"user_tag": 'user_tag_name',
+    #"kmod_repo": 'git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git',
+    #"kmod_branch": 'kmod_branch_name',
+    #"kmod_lbranch": 'kmod_lbranch_name',
+    #"kmod_tag": 'kmod_tag_name',
+}
+
+# If you don't want to execute the build stage, just use 'noinstall' as the
+# install type. If you run the tests from autotest-server, make sure that
+# /tmp/kvm-autotest-root/qemu is a link to your existing executable. Note that
+# if kvm_install is chose to run, it overwrites existing qemu and qemu-img
+# links to point to the newly built executables.
+r = True
+r = job.run_test("kvm", params=params, tag=params.get("shortname"))
+if not r:
+    print 'kvm_installation failed ... exiting'
+    sys.exit(1)
+
+# ----------------------------------------------------------
+# Get test set (dictionary list) from the configuration file
+# ----------------------------------------------------------
+import time
+import kvm_config, kvm_utils
+from autotest_lib.client.bin import utils
+
+filename = os.path.join(pwd, "kvm_tests.cfg")
+cfg = kvm_config.config(filename)
+
+# ----------------------------------------------------------
+# Daily DVD testing setup
+# ----------------------------------------------------------
+repo = "http://download.fedoraproject.org/pub/fedora/linux/releases/11/Fedora/"
+dvd_32_basename = "Fedora-11-%s.0-i386-DVD.iso" % time.strftime("%Y%m%d")
+dvd_64_basename = "Fedora-11-%s.0-x86_64-DVD.iso" % time.strftime("%Y%m%d")
+hash_basename = "MD5SUM"
+hash_type = "md5"
+
+dvd_32_hash_url = os.path.join(repo, "i386/iso", hash_basename)
+dvd_64_hash_url = os.path.join(repo, "x86_64/iso", hash_basename)
+dvd_32_url = os.path.join(repo, "i386/iso", dvd_32_basename)
+dvd_64_url = os.path.join(repo, "x86_64/iso", dvd_64_basename)
+
+dvd_32_sha1_path = utils.unmap_url("/tmp", dvd_32_url, "/tmp")
+dvd_64_sha1_path = utils.unmap_url("/tmp", dvd_64_url, "/tmp")
+
+dvd_32_hash = kvm_utils.get_hash_from_file(dvd_32_sha1_path, dvd_32_basename)
+dvd_64_hash = kvm_utils.get_hash_from_file(dvd_64_sha1_path, dvd_64_basename)
+
+# Download the DVD images to the images directory, caching them
+dvd_32 = kvm_utils.unmap_url_cache(os.path.join(pwd, "isos/linux"), dvd_32_url,
+                                   dvd_32_hash, hash_type)
+dvd_64 = kvm_utils.unmap_url_cache(os.path.join(pwd, "isos/linux"), dvd_64_url,
+                                   dvd_64_hash, hash_type)
+
+# Since we are dynamically downloading the isos, we need to add their
+# hash strings and iso names to the test configuration
+cfg.parse_string("Fedora.nightly.32: cdrom = isos/linux/%s" % dvd_32_basename)
+cfg.parse_string("Fedora.nightly.64: cdrom = isos/linux/%s" % dvd_64_basename)
+cfg.parse_string("Fedora.nightly.32: %ssum = %s" % (hash_type, dvd_32_hash))
+cfg.parse_string("Fedora.nightly.64: %ssum = %s" % (hash_type, dvd_32_hash))
+
+# If desirable, make changes to the test configuration here.  For example:
+# cfg.parse_string("install|setup: timeout_multiplier = 3")
+
+filename = os.path.join(pwd, "kvm_address_pools.cfg")
+if os.path.exists(filename):
+    cfg.parse_file(filename)
+    hostname = os.uname()[1].split(".")[0]
+    if cfg.filter("^" + hostname):
+        cfg.parse_string("only ^%s" % hostname)
+    else:
+        cfg.parse_string("only ^default_host")
+
+list = cfg.get_list()
+
+
+# -------------
+# Run the tests
+# -------------
+status_dict = {}
+
+for dict in list:
+    if dict.get("skip") == "yes":
+        continue
+    dependencies_satisfied = True
+    for dep in dict.get("depend"):
+        for test_name in status_dict.keys():
+            if not dep in test_name:
+                continue
+            if not status_dict[test_name]:
+                dependencies_satisfied = False
+                break
+    if dependencies_satisfied:
+        test_iterations = int(dict.get("iterations", 1))
+        current_status = job.run_test("kvm", params=dict,
+                                      tag=dict.get("shortname"),
+                                      iterations=test_iterations)
+    else:
+        current_status = False
+    status_dict[dict.get("name")] = current_status
+
+# create the html report in result dir
+reporter = os.path.join(pwd, 'make_html_report.py')
+html_file = os.path.join(job.resultdir,'results.html')
+os.system('%s -r %s -f %s -R'%(reporter, job.resultdir, html_file))
-- 
1.6.2.5

[PATCH 3/3] KVM test: Extend VM.create() method to support SHA1 check

[Lucas Meneghel Rodrigues]

Also, change variable names and messages to be more
generic.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
---
 client/tests/kvm/kvm_vm.py |   21 ++++++++++++++-------
 1 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
index a8d96ca..3d604c4 100755
--- a/client/tests/kvm/kvm_vm.py
+++ b/client/tests/kvm/kvm_vm.py
@@ -339,20 +339,27 @@ class VM:
             if params.get("md5sum_1m"):
                 logging.debug("Comparing expected MD5 sum with MD5 sum of "
                               "first MB of ISO file...")
-                actual_md5sum = kvm_utils.md5sum_file(iso, 1048576)
-                expected_md5sum = params.get("md5sum_1m")
+                actual_hash = kvm_utils.md5sum_file(iso, 1048576)
+                expected_hash = params.get("md5sum_1m")
                 compare = True
             elif params.get("md5sum"):
                 logging.debug("Comparing expected MD5 sum with MD5 sum of ISO "
                               "file...")
-                actual_md5sum = kvm_utils.md5sum_file(iso)
-                expected_md5sum = params.get("md5sum")
+                actual_hash = kvm_utils.md5sum_file(iso)
+                expected_hash = params.get("md5sum")
+                compare = True
+            elif params.get("sha1sum"):
+                logging.debug("Comparing expected SHA1 sum with SHA1 sum of "
+                              "ISO file...")
+                actual_hash = kvm_utils.md5sum_file(iso)
+                expected_hash = params.get("md5sum")
                 compare = True
             if compare:
-                if actual_md5sum == expected_md5sum:
-                    logging.debug("MD5 sums match")
+                if actual_hash == expected_hash:
+                    logging.debug("Hashes match")
                 else:
-                    logging.error("Actual MD5 sum differs from expected one")
+                    logging.error("Actual hash %s differs from expected "
+                                  "one %s", actual_hash, expected_hash)
                     return False
 
         # Make sure the following code is not executed by more than one thread
-- 
1.6.2.5

Re: [Autotest] [PATCH 1/3] KVM test: Add new utility functions to kvm_utils

[Uri Lublin]

On 10/21/2009 11:31 PM, Lucas Meneghel Rodrigues wrote:
> Some distributors ship CD and DVD files with SHA1 hash sums instead
> of MD5 hash sums, so let's extend the kvm_utils functions to
> evaluate and compare SHA1 hashes:
>
> * sha1sum_file(): Calculate SHA1 sum for file
> +def sha1sum_file(filename, size=None):
> +    """
> +    Calculate the sha1sum of filename.
> +    If size is not None, limit to first size bytes.
> +    Throw exception if something is wrong with filename.
> +    Can be also implemented with bash one-liner (assuming size%1024==0):
> +    dd if=filename bs=1024 count=size/1024 | sha1sum -
> +
> +    @param filename: Path of the file that will have its sha1sum calculated.
> +    @param returns: sha1sum of the file.
> +    """
> +    chunksize = 4096
> +    fsize = os.path.getsize(filename)
> +    if not size or size>fsize:
> +        size = fsize
> +    f = open(filename, 'rb')
> +    o = sha.new()
> +    while size>  0:
> +        if chunksize>  size:
> +            chunksize = size
> +        data = f.read(chunksize)
> +        if len(data) == 0:
> +            logging.debug("Nothing left to read but size=%d" % size)
> +            break
> +        o.update(data)
> +        size -= len(data)
> +    f.close()
> +    return o.hexdigest()
> +

This code and md5sum_file can share code.
Just pass o (or hash_new func: sha.new/md5.new) as a parameter to the function.


> +
> +
> +def get_hash_from_file(sha_path, dvd_basename):

For consistency, replace sha_path with hash_path (as in the comment below).

> +    """
> +    Get the a hash from a given DVD image from a hash file
> +    (Hash files are usually named MD5SUM or SHA1SUM and are located inside the
> +    download directories of the DVDs)
> +
> +    @param hash_path: Local path to a hash file.
> +    @param cd_image: Basename of a CD image
> +    """
> +    hash_file = open(sha_path, 'r')
> +    for line in hash_file.readlines():
> +        if dvd_basename in line:
> +            return line.split()[0]
> +

Re: [Autotest] [PATCH 3/3] KVM test: Extend VM.create() method to support SHA1 check

[Uri Lublin]

On 10/21/2009 11:31 PM, Lucas Meneghel Rodrigues wrote:

> --- a/client/tests/kvm/kvm_vm.py
> +++ b/client/tests/kvm/kvm_vm.py
> @@ -339,20 +339,27 @@ class VM:
> +            elif params.get("sha1sum"):
> +                logging.debug("Comparing expected SHA1 sum with SHA1 sum of "
> +                              "ISO file...")
> +                actual_hash = kvm_utils.md5sum_file(iso)
> +                expected_hash = params.get("md5sum")

typo: replace "md5" with "sha" for those last 2 lines