[PATCH 1/3] KVM Test: Introduce a helper class to run a test in the background

[PATCH 1/3] KVM Test: Introduce a helper class to run a test in the background

[Jason Wang]

Sometimes, we need to run a test when doing operations on a running VM ( such as
migrate the vm during its rebooting ). So this patch introduce a simple warpper
BackgroundTest to run a specified test in the background through a dedicated
thread, it also records the exception raised in the thead and raise it when
master call join().

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 client/tests/kvm/kvm_test_utils.py |   44 +++++++++++++++++++++++++++++++++++-
 1 files changed, 43 insertions(+), 1 deletions(-)

diff --git a/client/tests/kvm/kvm_test_utils.py b/client/tests/kvm/kvm_test_utils.py
index 5412aac..9f508b9 100644
--- a/client/tests/kvm/kvm_test_utils.py
+++ b/client/tests/kvm/kvm_test_utils.py
@@ -21,7 +21,7 @@ More specifically:
 @copyright: 2008-2009 Red Hat Inc.
 """
 
-import time, os, logging, re, commands
+import time, os, logging, re, commands, threading
 from autotest_lib.client.common_lib import error
 from autotest_lib.client.bin import utils
 import kvm_utils, kvm_vm, kvm_subprocess, scan_results
@@ -505,3 +505,45 @@ def run_autotest(vm, session, control_path, timeout, outputdir):
             e_msg = ("Tests %s failed during control file execution" %
                      " ".join(bad_results))
         raise error.TestFail(e_msg)
+
+class BackgroundTest:
+    """
+    This class would run a test in background through a dedicated thread.
+    """
+
+    def __init__(self, func, params):
+        """
+        Initialize the object and set a few attributes.
+        """
+        self.thread = threading.Thread(target=self.launch,
+                                       args=(func, params))
+        self.exception = None
+
+    def launch(self, func, params):
+        """
+        Catch and record the exception.
+        """
+        try:
+            func(*params)
+        except Exception, e:
+            self.exception = e
+
+    def start(self):
+        """
+        Run func(params) in a dedicated thread
+        """
+        self.thread.start()
+
+    def join(self):
+        """
+        Wait for the join of thread and raise its exception if any.
+        """
+        self.thread.join()
+        if self.exception:
+            raise self.exception
+
+    def is_alive(self):
+        """
+        Check whether the test is still alive.
+        """
+        return self.thread.is_alive()

[PATCH 1/3] KVM Test: Introduce a helper class to run a test in the background

[Lucas Meneghel Rodrigues]

Sometimes, we need to run a test when doing operations on a running VM
(such as migrate the vm during its rebooting ). So this patch introduces
a simple warpper BackgroundTest to run a specified test in the background
through a dedicated thread, it also records the exception raised in the
thead and raise it when master call join().

Changes from v1:
- Rebase against latest upstream

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 client/tests/kvm/kvm_test_utils.py |   49 +++++++++++++++++++++++++++++++++++-
 1 files changed, 48 insertions(+), 1 deletions(-)

diff --git a/client/tests/kvm/kvm_test_utils.py b/client/tests/kvm/kvm_test_utils.py
index e7c8d33..218bf31 100644
--- a/client/tests/kvm/kvm_test_utils.py
+++ b/client/tests/kvm/kvm_test_utils.py
@@ -21,7 +21,7 @@ More specifically:
 @copyright: 2008-2009 Red Hat Inc.
 """
 
-import time, os, logging, re, commands, signal
+import time, os, logging, re, commands, signal, threading
 from autotest_lib.client.common_lib import error
 from autotest_lib.client.bin import utils
 import kvm_utils, kvm_vm, kvm_subprocess, scan_results
@@ -564,6 +564,53 @@ def run_autotest(vm, session, control_path, timeout, outputdir):
         raise error.TestFail(e_msg)
 
 
+class BackgroundTest(object):
+    """
+    This class would run a test in background through a dedicated thread.
+    """
+
+    def __init__(self, func, params):
+        """
+        Initialize the object and set a few attributes.
+        """
+        self.thread = threading.Thread(target=self.launch,
+                                       args=(func, params))
+        self.exception = None
+
+
+    def launch(self, func, params):
+        """
+        Catch and record the exception.
+        """
+        try:
+            func(*params)
+        except Exception, e:
+            self.exception = e
+
+
+    def start(self):
+        """
+        Run func(params) in a dedicated thread
+        """
+        self.thread.start()
+
+
+    def join(self):
+        """
+        Wait for the join of thread and raise its exception if any.
+        """
+        self.thread.join()
+        if self.exception:
+            raise self.exception
+
+
+    def is_alive(self):
+        """
+        Check whether the test is still alive.
+        """
+        return self.thread.is_alive()
+
+
 def get_loss_ratio(output):
     """
     Get the packet loss ratio from the output of ping
-- 
1.7.2.3

[PATCH 2/3] KVM test: Test reboot during migration

[Lucas Meneghel Rodrigues]

From: Jason Wang <jasowang@redhat.com>

This test is simple: it's doing the migration and guest rebooting in the same
time and then check the completion of migration and verify the guest state by
loggin it again after reboot.

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 client/tests/kvm/tests/migration_with_reboot.py |   45 +++++++++++++++++++++++
 client/tests/kvm/tests_base.cfg.sample          |    5 +++
 2 files changed, 50 insertions(+), 0 deletions(-)
 create mode 100644 client/tests/kvm/tests/migration_with_reboot.py

diff --git a/client/tests/kvm/tests/migration_with_reboot.py b/client/tests/kvm/tests/migration_with_reboot.py
new file mode 100644
index 0000000..985cb3a
--- /dev/null
+++ b/client/tests/kvm/tests/migration_with_reboot.py
@@ -0,0 +1,45 @@
+import logging, time
+import threading
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils
+
+
+def run_migration_with_reboot(test, params, env):
+    """
+    KVM migration test:
+    1) Get a live VM and clone it.
+    2) Verify that the source VM supports migration.  If it does, proceed with
+            the test.
+    3) Reboot the VM
+    4) Send a migration command to the source VM and wait until it's finished.
+    5) Kill off the source VM.
+    6) Log into the destination VM after the migration is finished.
+
+    @param test: kvm test object.
+    @param params: Dictionary with test parameters.
+    @param env: Dictionary with the test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    timeout = int(params.get("login_timeout", 360))
+    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
+
+    mig_timeout = float(params.get("mig_timeout", "3600"))
+    mig_protocol = params.get("migration_protocol", "tcp")
+    mig_cancel = bool(params.get("mig_cancel"))
+    bg = None
+
+    try:
+        # reboot the VM in background
+        bg = kvm_test_utils.BackgroundTest(kvm_test_utils.reboot, (vm, session))
+        bg.start()
+
+        while bg.is_alive():
+            # Migrate the VM
+            dest_vm = kvm_test_utils.migrate(vm, env, mig_timeout, mig_protocol,
+                                             False)
+            vm = dest_vm
+
+    finally:
+        if bg:
+            bg.join()
+        session.close()
diff --git a/client/tests/kvm/tests_base.cfg.sample b/client/tests/kvm/tests_base.cfg.sample
index c96f02d..8f6984c 100644
--- a/client/tests/kvm/tests_base.cfg.sample
+++ b/client/tests/kvm/tests_base.cfg.sample
@@ -147,6 +147,11 @@ variants:
             - mig_cancel:
                 migration_protocol = "tcp"
                 mig_cancel = True
+        variants:
+            - @default:
+            - with_reboot:
+                iterations = 1
+                type = migration_with_reboot
 
     - boot_savevm: install setup unattended_install.cdrom
         type = boot_savevm
-- 
1.7.2.3

[PATCH 3/3] KVM test: Test the file transfer during migartion

[Lucas Meneghel Rodrigues]

From: Jason Wang <jasowang@redhat.com>

This test just do the file transfer from host to guest during migartion in order
to check whether the nic/block state could be saved and loaded correctly.

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 .../kvm/tests/migration_with_file_transfer.py      |   59 ++++++++++++++++++++
 client/tests/kvm/tests_base.cfg.sample             |    7 ++
 2 files changed, 66 insertions(+), 0 deletions(-)
 create mode 100644 client/tests/kvm/tests/migration_with_file_transfer.py

diff --git a/client/tests/kvm/tests/migration_with_file_transfer.py b/client/tests/kvm/tests/migration_with_file_transfer.py
new file mode 100644
index 0000000..8a316bf
--- /dev/null
+++ b/client/tests/kvm/tests/migration_with_file_transfer.py
@@ -0,0 +1,59 @@
+import logging, time
+from autotest_lib.client.common_lib import utils, error
+import kvm_subprocess, kvm_test_utils, kvm_utils
+
+
+def run_migration_with_file_transfer(test, params, env):
+    """
+    KVM migration test:
+    1) Get a live VM and clone it.
+    2) Verify that the source VM supports migration.  If it does, proceed with
+            the test.
+    3) Reboot the VM
+    4) Send a migration command to the source VM and wait until it's finished.
+    5) Kill off the source VM.
+    6) Log into the destination VM after the migration is finished.
+
+    @param test: kvm test object.
+    @param params: Dictionary with test parameters.
+    @param env: Dictionary with the test environment.
+    """
+
+    def transfer_test(vm, host_path, guest_path, timeout=120):
+        """
+        vm.copy_files_to does not raise exception, so we need a wrapper
+        in order to make it to be used by BackgroundTest.
+        """
+        if not vm.copy_files_to(host_path, guest_path, timeout=timeout):
+            raise error.TestError("Fail to do the file transfer!")
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    timeout = int(params.get("login_timeout", 360))
+    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
+
+    mig_timeout = float(params.get("mig_timeout", "3600"))
+    mig_protocol = params.get("migration_protocol", "tcp")
+
+    guest_path = params.get("guest_path", "/tmp")
+    file_size = params.get("file_size", "1000")
+    transfer_timeout = int(params.get("transfer_timeout", "240"))
+    bg = None
+
+    try:
+        utils.run("dd if=/dev/zero of=/tmp/file bs=1M count=%s" % file_size)
+
+        # Transfer file from host to guest
+        bg = kvm_test_utils.BackgroundTest(transfer_test,
+                                           (vm, "/tmp/file", guest_path,
+                                            transfer_timeout))
+        bg.start()
+
+        while bg.is_alive():
+            logging.info("File transfer is not ended, start a round of migration ...")
+            # Migrate the VM
+            dest_vm = kvm_test_utils.migrate(vm, env, mig_timeout, mig_protocol, False)
+            vm = dest_vm
+    finally:
+        if bg: bg.join()
+        session.close()
+        utils.run("rm -rf /tmp/zero")
diff --git a/client/tests/kvm/tests_base.cfg.sample b/client/tests/kvm/tests_base.cfg.sample
index 8f6984c..2e06c3b 100644
--- a/client/tests/kvm/tests_base.cfg.sample
+++ b/client/tests/kvm/tests_base.cfg.sample
@@ -152,6 +152,9 @@ variants:
             - with_reboot:
                 iterations = 1
                 type = migration_with_reboot
+            - with_file_transfer:
+                iterations = 1
+                type = migration_with_file_transfer
 
     - boot_savevm: install setup unattended_install.cdrom
         type = boot_savevm
@@ -1570,6 +1573,10 @@ variants:
             migration_bg_command = start ping -t localhost
             migration_bg_check_command = tasklist | find /I "ping.exe"
             migration_bg_kill_command = taskkill /IM ping.exe /F
+        migrate_with_file_transfer:
+            guest_path = C:\
+            rtl8139:
+                file_size = 10
         stress_boot:
             alive_test_cmd = systeminfo
         timedrift:
-- 
1.7.2.3

Re: [PATCH 1/3] KVM Test: Introduce a helper class to run a test in the background

[Lucas Meneghel Rodrigues]

On Fri, 2010-11-26 at 21:01 -0200, Lucas Meneghel Rodrigues wrote:
> Sometimes, we need to run a test when doing operations on a running VM
> (such as migrate the vm during its rebooting ). So this patch introduces
> a simple warpper BackgroundTest to run a specified test in the background
> through a dedicated thread, it also records the exception raised in the
> thead and raise it when master call join().

Since we've spent some time looking into this and noone came up with a
better solution, I'm applying Jason's series. In order to run the tests
in parallel, on your test set you can put either:

only migrate.with_file_transfer
or
only migrate.with_reboot

Provided, of course, that you already had a vm installed with autotest.
If you don't you probably want to include the unattended_install.cdrom
test before.

Cheers!

Lucas

> Changes from v1:
> - Rebase against latest upstream
> 
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  client/tests/kvm/kvm_test_utils.py |   49 +++++++++++++++++++++++++++++++++++-
>  1 files changed, 48 insertions(+), 1 deletions(-)
> 
> diff --git a/client/tests/kvm/kvm_test_utils.py b/client/tests/kvm/kvm_test_utils.py
> index e7c8d33..218bf31 100644
> --- a/client/tests/kvm/kvm_test_utils.py
> +++ b/client/tests/kvm/kvm_test_utils.py
> @@ -21,7 +21,7 @@ More specifically:
>  @copyright: 2008-2009 Red Hat Inc.
>  """
>  
> -import time, os, logging, re, commands, signal
> +import time, os, logging, re, commands, signal, threading
>  from autotest_lib.client.common_lib import error
>  from autotest_lib.client.bin import utils
>  import kvm_utils, kvm_vm, kvm_subprocess, scan_results
> @@ -564,6 +564,53 @@ def run_autotest(vm, session, control_path, timeout, outputdir):
>          raise error.TestFail(e_msg)
>  
> 
> +class BackgroundTest(object):
> +    """
> +    This class would run a test in background through a dedicated thread.
> +    """
> +
> +    def __init__(self, func, params):
> +        """
> +        Initialize the object and set a few attributes.
> +        """
> +        self.thread = threading.Thread(target=self.launch,
> +                                       args=(func, params))
> +        self.exception = None
> +
> +
> +    def launch(self, func, params):
> +        """
> +        Catch and record the exception.
> +        """
> +        try:
> +            func(*params)
> +        except Exception, e:
> +            self.exception = e
> +
> +
> +    def start(self):
> +        """
> +        Run func(params) in a dedicated thread
> +        """
> +        self.thread.start()
> +
> +
> +    def join(self):
> +        """
> +        Wait for the join of thread and raise its exception if any.
> +        """
> +        self.thread.join()
> +        if self.exception:
> +            raise self.exception
> +
> +
> +    def is_alive(self):
> +        """
> +        Check whether the test is still alive.
> +        """
> +        return self.thread.is_alive()
> +
> +
>  def get_loss_ratio(output):
>      """
>      Get the packet loss ratio from the output of ping

Re: [PATCH 1/3] KVM Test: Introduce a helper class to run a test in the background

[Michael Goldish]

On 11/27/2010 01:16 AM, Lucas Meneghel Rodrigues wrote:
> On Fri, 2010-11-26 at 21:01 -0200, Lucas Meneghel Rodrigues wrote:
>> Sometimes, we need to run a test when doing operations on a running VM
>> (such as migrate the vm during its rebooting ). So this patch introduces
>> a simple warpper BackgroundTest to run a specified test in the background
>> through a dedicated thread, it also records the exception raised in the
>> thead and raise it when master call join().
> 
> Since we've spent some time looking into this and noone came up with a
> better solution, I'm applying Jason's series. In order to run the tests
> in parallel, on your test set you can put either:
> 
> only migrate.with_file_transfer
> or
> only migrate.with_reboot
> 
> Provided, of course, that you already had a vm installed with autotest.
> If you don't you probably want to include the unattended_install.cdrom
> test before.
> 
> Cheers!
> 
> Lucas

I was planning to post a modified version of the wrapper.  I guess I'll
rebase it against this version.  Also, migrate.with_reboot doesn't work,
or at least shouldn't work, because reboot() receives a VM object that
is later destroyed.