Re: [PATCH] [PATCH V2] patch to enable installing cygwin and running autotest in windows guests

[Lucas Meneghel Rodrigues <lmr@redhat.com>]

On Mon, 2010-09-27 at 17:43 +0530, Yogananth Subramanian wrote:
> The patch creates two new files 'scripts/cyg_install.py' and
> 'tests/cyg_install'. 'scripts/cyg_install.py' is used to install cygwin in
> unattended mode. This patch also introduces a new parameter 'patch_file'to
> run_autotest() in kvm_test_utils.py file, to install patch for autotest to
> run in windows. The file 'autotest_control/autotest.patch' is base patch to
> run any autotest test cases in windows and file 'autotest_control/iozone.patch'
> is a patch to run iozone on windows.

Yoganath, I have some comments to make, would you please look into them
and provide me an updated patch? Thanks!

> [cyg_install]:
> 1)local install:
>    Folder denoted by 'cyg_path' should contain the cygwin setup file and a
>    dir called "repo" which would contain all the cygwin packages need to be
>    installed.
>   remote install:
>    'cyg_path' contains the link to download cygwin setup file
> 2)scripts/cyg_install.py creates  cyg.iso,which contains "cyg_install.cmd"
>    script needed to install cygwin and set of files pointed by the
>    'cyg_path'
> 3)login to guest using telnet and run "cyg_install.cmd" to install
>    cygwin
> 
> [cygwin autotest]
> 1)the param 'test_patch_file' contains the list of patches for autotest
> 2)scp autotest to windows guest, apply the patch and run autotest
> 
> Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com>
> ---
>  client/tests/kvm/autotest_control/autotest.patch |   15 +
>  client/tests/kvm/autotest_control/iozone.patch   |   12 +
>  client/tests/kvm/kvm_test_utils.py               |   10 +-
>  client/tests/kvm/scripts/cyg_install.py          |   98 ++
>  client/tests/kvm/tests/autotest.py               |   21 +-
>  client/tests/kvm/tests/cyg_install.py            |   36 +
>  client/tests/kvm/tests_base.cfg.sample           |   45 +-
>  client/tests/kvm/tests_base.cfg.sample.orig      | 1884 ++++++++++++++++++++++
>  8 files changed, 2117 insertions(+), 4 deletions(-)
>  create mode 100644 client/tests/kvm/autotest_control/autotest.patch
>  create mode 100644 client/tests/kvm/autotest_control/iozone.patch
>  create mode 100644 client/tests/kvm/scripts/cyg_install.py
>  create mode 100644 client/tests/kvm/tests/cyg_install.py
>  create mode 100644 client/tests/kvm/tests_base.cfg.sample.orig
> 
> diff --git a/client/tests/kvm/autotest_control/autotest.patch b/client/tests/kvm/autotest_control/autotest.patch
> new file mode 100644
> index 0000000..f562f95
> --- /dev/null
> +++ b/client/tests/kvm/autotest_control/autotest.patch
> @@ -0,0 +1,15 @@
> +diff -aurpN client/bin/job.py client-new/bin/job.py
> +--- client/bin/job.py   2010-08-25 01:42:27.000000000 -0400
> ++++ client-new/bin/job.py       2010-09-07 09:54:30.000000000 -0400
> +@@ -296,7 +296,10 @@ class base_client_job(base_job.base_job)
> +
> +         # extract console= and other args from cmdline and add them into the
> +         # base args that we use for all kernels we install
> +-        cmdline = utils.read_one_line('/proc/cmdline')
> ++        if os.path.exists('/proc/cmdline'):
> ++            cmdline = utils.read_one_line('/proc/cmdline')
> ++        else:
> ++            return

^ I am happy and surprised that this is the only autotest modification
needed to run autotest under cygwin. However it's a more proper solution
to actually integrate it in autotest, in case this does not regress
functionality for existing clients (I tend to think it won't, since
every linux will have /proc/cmdline. Those could be my last words
though).

> +         kernel_args = []
> +         for karg in cmdline.split():
> +             for param in copy_cmdline:
> diff --git a/client/tests/kvm/autotest_control/iozone.patch b/client/tests/kvm/autotest_control/iozone.patch
> new file mode 100644
> index 0000000..6229205
> --- /dev/null
> +++ b/client/tests/kvm/autotest_control/iozone.patch
> @@ -0,0 +1,12 @@
> +diff -aurpN client/tests/iozone/iozone.py client-new/tests/iozone/iozone.py
> +--- client/tests/iozone/iozone.py       2010-08-25 01:42:27.000000000 -0400
> ++++ client-new/tests/iozone/iozone.py   2010-09-02 11:38:42.000000000 -0400
> +@@ -43,7 +43,7 @@ class iozone(test.test):
> +         elif (arch == 'x86_64'):
> +             utils.system('make linux-AMD64')
> +         else:
> +-            utils.system('make linux')
> ++            utils.system('make Windows')

^ IOzone has changed, now it uses utils.make (see
client/common_lib/utils.py), so need to take this into account. Perhaps
we could get rid of all the patching altogether somehow. Looking at the
current code of IOzone:

        arch = utils.get_current_kernel_arch()
        if (arch == 'ppc'):
            utils.make('linux-powerpc')
        elif (arch == 'ppc64'):
            utils.make('linux-powerpc64')
        elif (arch == 'x86_64'):
            utils.make('linux-AMD64')
        else:
            utils.make('linux')

^ If cygwin reports some different value for
utils.get_current_kernel_arch(), we could even code an additional else
clause that allows for executing

elif (arch =='cygwin-thing'):
    utils.make('windows')


Would you please look into that?

> + 
> + 
> +     def run_once(self, dir=None, args=None):
> diff --git a/client/tests/kvm/kvm_test_utils.py b/client/tests/kvm/kvm_test_utils.py
> index 5412aac..34bccb8 100644
> --- a/client/tests/kvm/kvm_test_utils.py
> +++ b/client/tests/kvm/kvm_test_utils.py
> @@ -336,7 +336,7 @@ def get_memory_info(lvms):
>      return meminfo
>  
> 
> -def run_autotest(vm, session, control_path, timeout, outputdir):
> +def run_autotest(vm, session, control_path, timeout, outputdir, patch_file):
>      """
>      Run an autotest control file inside a guest (linux only utility).
>  
> @@ -346,6 +346,7 @@ def run_autotest(vm, session, control_path, timeout, outputdir):
>      @param timeout: Timeout under which the autotest control file must complete.
>      @param outputdir: Path on host where we should copy the guest autotest
>              results to.
> +    @param patch_file: A path to an autotest patch file.

^ I am really hoping to get rid of patch_file.

>      """
>      def copy_if_hash_differs(vm, local_path, remote_path):
>          """
> @@ -460,12 +461,19 @@ def run_autotest(vm, session, control_path, timeout, outputdir):
>                              os.path.join(autotest_path, 'control')):
>          raise error.TestFail("Could not copy the test control file to guest")
>  
> +    if not patch_file == "":
> +        if not vm.copy_files_to(patch_file,
> +                            os.path.join(autotest_path, '../test.patch')):
> +            raise error.TestFail("Could not copy the test patch file to guest")
> +
>      # Run the test
>      logging.info("Running autotest control file %s on guest, timeout %ss",
>                   os.path.basename(control_path), timeout)
>      session.get_command_output("cd %s" % autotest_path)
>      session.get_command_output("rm -f control.state")
>      session.get_command_output("rm -rf results/*")
> +    if not patch_file == "":
> +        session.get_command_output("patch -p1 <../test.patch")
>      logging.info("---------------- Test output ----------------")
>      status = session.get_command_status("bin/autotest control",
>                                          timeout=timeout,
> diff --git a/client/tests/kvm/scripts/cyg_install.py b/client/tests/kvm/scripts/cyg_install.py
> new file mode 100644
> index 0000000..7213b90
> --- /dev/null
> +++ b/client/tests/kvm/scripts/cyg_install.py
> @@ -0,0 +1,98 @@
> +#!/usr/bin/python
> +"""
> +Script to install cygwin  on KVM guests.
> +"""
> +# -*- coding: utf-8 -*-
> +import os, sys, shutil, tempfile, re, ConfigParser, glob
> +import common
> +from autotest_lib.client.bin import utils
> +
> +class CygInstall(object):
> +    """
> +    Creates an iso image that will contain cygwin setup file for unattended
> +    install. The setup and cygwin packages should be present on the host
> +    under the path specified by 'cyg_path' for local install. For remote 
> +    install the setup will be downloaded from 'cyg_path'  
> +    """
> +    def __init__(self):
> +        """
> +        Gets params from environment variables and sets class attributes.
> +        """
> +        script_dir = os.path.dirname(sys.modules[__name__].__file__)
> +        kvm_test_dir = os.path.abspath(os.path.join(script_dir, ".."))
> +        images_dir = os.path.join(kvm_test_dir, 'images')
> +        self.deps_dir = os.path.join(kvm_test_dir, 'deps')
> +        self.cachedir = os.path.join(kvm_test_dir, 'cyg_cache')
> +        self.unattended_dir = os.path.join(kvm_test_dir, 'unattended')
> +        self.cyg_type = os.environ.get('KVM_TEST_cyg_type')
> +        self.cyg_path = os.environ.get('KVM_TEST_cyg_path')
> +        self.cyg_param = os.environ.get('KVM_TEST_cyg_param')
> +        self.cyg_iso = os.environ.get('KVM_TEST_cdrom_cd1')
> +
> +
> +    def setup_local(self):
> +        """
> +        cygwin setup and packages are present locally.
> +        """
> +        self.cygsetup_file()
> +        mkiso_cmd = "mkisofs -L -l -allow-lowercase -allow-multidot -relaxed-"\
> +              "filenames  -N -v -d  -D  -o %s %s"%(self.cyg_iso, self.cyg_path)
> +        if os.system(mkiso_cmd):
> +            raise SetupError('Could not create iso')
> +
> +
> +    def setup_remote(self):
> +        """
> +        cygwin setup downloaded from 'cyg_path' 
> +        """
> +
> +        if not os.path.isdir(self.cachedir):
> +            os.makedirs(self.cachedir)
> +        self.cyg_md5 = os.environ.get('KVM_TEST_cyg_md5')
> +        dacapo_pkg = utils.unmap_url_cache(self.cachedir, self.cyg_path, self.cyg_md5)
> +        self.cyg_path = self.cachedir 
> +        self.cygsetup_file()
> +        mkiso_cmd = "mkisofs -L -l -allow-lowercase -allow-multidot -relaxed"\
> +             "-filenames  -N -v -d  -D  -o %s %s"%(self.cyg_iso,self.cachedir)
> +        if os.system(mkiso_cmd):
> +            raise SetupError('Could not create iso')
> +        shutil.rmtree(self.cachedir)
> +
> +    def cygsetup_file(self):
> +       cyg_config_file = os.path.join(self.cyg_path, 'cyg_install.cmd')
> +       cyg_config = open(cyg_config_file, 'w')
> +       cyg_config.write('d:setup.exe -q -R '
> +                              '"%systemdrive%\\cygwin"  '+self.cyg_param)
> +       cyg_config.write('\n')
> +       cyg_config.write('cd %systemdrive%\\cygwin\\bin\n')
> +       cyg_config.write("""for /f "delims=" %%A in ('cygpath.exe -u %tmp%')"""
> +                                                         """ do set a=%%A\n""")
> +       cyg_config.write('bash --login -i -c "chmod 777 %a%"\n')
> +       cyg_config.write('ash.exe /usr/bin/rebaseall \n')
> +       cyg_config.write('bash --login -i -c "ssh-host-config -y --pwd '
> +                                                              '1q2w3eP"\n')
> +       cyg_config.write('net start sshd\n')
> +       cyg_config.write('bash --login -i -c "ssh-user-config -y -p"\n')
> +       cyg_config.close()
> +
> +    def setup(self):
> +        print "Starting Cygwin install setup"
> +
> +        print "Variables set:"
> +        print "    cygwin instally type: " + str(self.cyg_type)
> +        print "    cygwin path: " + str(self.cyg_path)
> +        print "    cygwin iso: " + str(self.cyg_iso)
> +
> +        if self.cyg_type == "local":
> +            self.setup_local()
> +        elif self.cyg_type == "remote":
> +            self.setup_remote()
> +        else:
> +            raise SetupError("Unexpected installation method %s" %
> +                                   self.cyg_type)
> +        print "Cygwin install setup finished successfuly"
> +
> +
> +if __name__ == "__main__":
> +    os_install = CygInstall()
> +    os_install.setup()
> diff --git a/client/tests/kvm/tests/autotest.py b/client/tests/kvm/tests/autotest.py
> index 2916ebd..e40b26a 100644
> --- a/client/tests/kvm/tests/autotest.py
> +++ b/client/tests/kvm/tests/autotest.py
> @@ -1,4 +1,4 @@
> -import os, logging
> +import os, logging, tempfile
>  from autotest_lib.client.common_lib import error
>  from autotest_lib.client.bin import utils
>  import kvm_subprocess, kvm_utils, kvm_test_utils
> @@ -20,6 +20,23 @@ def run_autotest(test, params, env):
>      timeout = int(params.get("test_timeout", 300))
>      control_path = os.path.join(test.bindir, "autotest_control",
>                                  params.get("test_control_file"))
> +    if not params.get("test_patch_file"):
> +        patch_file = ""
> +    else:
> +        autotest_files = params.get("test_patch_file").split()
> +        temp = []
> +        for autotest_file in autotest_files:
> +            patch_path = os.path.join(test.bindir, "autotest_control",
> +                                autotest_file)
> +            f = open(patch_path, "r")
> +            temp.extend(f.readlines())
> +            f.close()
> +
> +        (fd,patch_file)=tempfile.mkstemp(prefix='patch_',dir='/tmp')
> +        f = open(patch_file, "w")
> +        f.writelines(temp)
> +        f.close()
>      outputdir = test.outputdir
>  
> -    kvm_test_utils.run_autotest(vm, session, control_path, timeout, outputdir)
> +    kvm_test_utils.run_autotest(vm, session, control_path, timeout, outputdir,
> +                                                                   patch_file)
> diff --git a/client/tests/kvm/tests/cyg_install.py b/client/tests/kvm/tests/cyg_install.py
> new file mode 100644
> index 0000000..f038b2d
> --- /dev/null
> +++ b/client/tests/kvm/tests/cyg_install.py
> @@ -0,0 +1,36 @@
> +import logging, time, os
> +from autotest_lib.client.common_lib import error
> +from autotest_lib.client.bin import utils
> +from autotest_lib.client.tests.iozone import postprocessing
> +import kvm_subprocess, kvm_test_utils, kvm_utils
> +
> +
> +def run_cyg_install(test, params, env):
> +    """
> +    Install cygwin on a windows guest:
> +    1) Log into a guest
> +    2) Install cygwin with the setup contained in the cyg.iso
> +
> +    @param test: kvm test object
> +    @param params: Dictionary with the test parameters
> +    @param env: Dictionary with test environment.
> +    """
> +    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
> +    timeout = int(params.get("login_timeout", 360))
> +    session = kvm_test_utils.wait_for_login(vm, 0,timeout,0,2)
> +    results_path = os.path.join(test.resultsdir,
> +                                'raw_output_%s' % test.iteration)
> +
> +    c = params.get("cyg_cmd")
> +    t = int(params.get("cyg_timeout"))
> +    logging.info("Installing cygwin on guest, timeout %ss", t)
> +    results = session.get_command_output(command=c, timeout=t,
> +                                         print_func=logging.debug)
> +    utils.open_write_close(results_path, results)
> +
> +    logging.info("Installation succeed")
> +    session.sendline(vm.get_params().get("shutdown_command"))
> +    if not kvm_utils.wait_for(vm.is_dead, 240, 0, 1):
> +        raise error.TestFail("Guest refuses to go down")
> +    session.close()
> +
> diff --git a/client/tests/kvm/tests_base.cfg.sample b/client/tests/kvm/tests_base.cfg.sample
> index 167e86d..70ab947 100644
> --- a/client/tests/kvm/tests_base.cfg.sample
> +++ b/client/tests/kvm/tests_base.cfg.sample
> @@ -130,6 +130,25 @@ variants:
>          kill_vm_on_error = yes
>          login_timeout = 240
>  
> +    -cyg_install:     install setup unattended_install.cdrom
> +        type = cyg_install
> +        pre_command = " scripts/cyg_install.py;"
> +        cyg_param = " -P make,patch,bash,vim,openssh,python,gcc,binutils "
> +        shutdown_method = shell
> +        cyg_cmd = "cmd /c D:\cyg_install.cmd"
> +        cyg_timeout = 3600
> +        shutdown_command = shutdown /s /f /t 0
> +        variants:
> +            - local:
> +                cyg_type = local
> +                cyg_param += " -L -l d:\\repo "
> +                cyg_path = "/root/sdb/images/cyg"
> +            - remote:
> +                cyg_type = remote
> +                cyg_param += " -s http://cygwin.basemirror.de "
> +                cyg_path = "http://www.cygwin.com/setup.exe"
> +                cyg_md5 = "a4f5eaaa1096891b9812b81f104fc4d9"
> +
>      - migrate:      install setup unattended_install.cdrom
>          type = migration
>          migration_test_command = help
> @@ -1236,7 +1255,7 @@ variants:
>  
>      # Windows section
>      - @Windows:
> -        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks)
> +        no linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks)
>          shutdown_command = shutdown /s /f /t 0
>          reboot_command = shutdown /r /f /t 0
>          status_test_command = echo %errorlevel%
> @@ -1327,6 +1346,30 @@ variants:
>              pci_test_cmd = echo select disk 1 > dt && echo online >> dt && echo detail disk >> dt && echo exit >> dt && diskpart /s dt
>          physical_resources_check:
>              catch_uuid_cmd = 
> +        cyg_install:
> +            cdrom_cd1 = windows/cyg.iso
> +            shell_prompt = "Administrator>$"
> +            username = Administrator
> +            password = 1q2w3eP
> +            shell_client = telnet
> +            shell_port = 23
> +            guest_port_remote_shell = 23
> +        autotest:
> +            shell_prompt = "Administrator>$"
> +            username = Administrator
> +            password = 1q2w3eP
> +            status_test_command = echo $?
> +            shell_prompt = "\$ "
> +            shell_client = ssh
> +            shell_linesep = "\n"
> +            shell_port = 22
> +            file_transfer_client = scp
> +            file_transfer_port = 22
> +            guest_port_remote_shell = 22
> +            guest_port_file_transfer = 23
> +            test_patch_file = autotest.patch
> +        autotest.iozone:
> +            test_patch_file += " iozone.patch"
>  
>          variants:
>              - Win2000:
> diff --git a/client/tests/kvm/tests_base.cfg.sample.orig b/client/tests/kvm/tests_base.cfg.sample.orig

^ Here, this file was added by accident. Would you please get rid of it
in a future patch? Thanks!