[PATCH 0/3] Launch other test during migration

[PATCH 0/3] Launch other test during migration

[Jason Wang]

We could give a further test of migration by launch test during migartion. So
the following series implements:

- A simple class to run a specified test in the background which could be used
to launch other test during migartion. Its design is rather simple and its usage
is a little tricky, but it work well.
- Two sample tests which take advantages of the background class: Test reboot
during guest migration and test file_transfer during guest migration.

In the future, we could even lauch autotest client test during guest migation.

---

Jason Wang (3):
      KVM Test: Introduce a helper class to run a test in the background
      KVM test: Test reboot during migration
      KVM test: Test the file transfer during migartion


 client/tests/kvm/kvm_test_utils.py                 |   44 +++++++++++++++
 .../kvm/tests/migration_with_file_transfer.py      |   59 ++++++++++++++++++++
 client/tests/kvm/tests/migration_with_reboot.py    |   45 +++++++++++++++
 client/tests/kvm/tests_base.cfg.sample             |   12 ++++
 4 files changed, 159 insertions(+), 1 deletions(-)
 create mode 100644 client/tests/kvm/tests/migration_with_file_transfer.py
 create mode 100644 client/tests/kvm/tests/migration_with_reboot.py

-- 
Jason Wang

[PATCH 1/3] KVM Test: Introduce a helper class to run a test in the background

[Jason Wang]

Sometimes, we need to run a test when doing operations on a running VM ( such as
migrate the vm during its rebooting ). So this patch introduce a simple warpper
BackgroundTest to run a specified test in the background through a dedicated
thread, it also records the exception raised in the thead and raise it when
master call join().

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 client/tests/kvm/kvm_test_utils.py |   44 +++++++++++++++++++++++++++++++++++-
 1 files changed, 43 insertions(+), 1 deletions(-)

diff --git a/client/tests/kvm/kvm_test_utils.py b/client/tests/kvm/kvm_test_utils.py
index 5412aac..9f508b9 100644
--- a/client/tests/kvm/kvm_test_utils.py
+++ b/client/tests/kvm/kvm_test_utils.py
@@ -21,7 +21,7 @@ More specifically:
 @copyright: 2008-2009 Red Hat Inc.
 """
 
-import time, os, logging, re, commands
+import time, os, logging, re, commands, threading
 from autotest_lib.client.common_lib import error
 from autotest_lib.client.bin import utils
 import kvm_utils, kvm_vm, kvm_subprocess, scan_results
@@ -505,3 +505,45 @@ def run_autotest(vm, session, control_path, timeout, outputdir):
             e_msg = ("Tests %s failed during control file execution" %
                      " ".join(bad_results))
         raise error.TestFail(e_msg)
+
+class BackgroundTest:
+    """
+    This class would run a test in background through a dedicated thread.
+    """
+
+    def __init__(self, func, params):
+        """
+        Initialize the object and set a few attributes.
+        """
+        self.thread = threading.Thread(target=self.launch,
+                                       args=(func, params))
+        self.exception = None
+
+    def launch(self, func, params):
+        """
+        Catch and record the exception.
+        """
+        try:
+            func(*params)
+        except Exception, e:
+            self.exception = e
+
+    def start(self):
+        """
+        Run func(params) in a dedicated thread
+        """
+        self.thread.start()
+
+    def join(self):
+        """
+        Wait for the join of thread and raise its exception if any.
+        """
+        self.thread.join()
+        if self.exception:
+            raise self.exception
+
+    def is_alive(self):
+        """
+        Check whether the test is still alive.
+        """
+        return self.thread.is_alive()

[PATCH 2/3] KVM test: Test reboot during migration

[Jason Wang]

This test is simple: it's doing the migration and guest rebooting in the same
time and then check the completion of migration and verify the guest state by
loggin it again after reboot.

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 client/tests/kvm/tests/migration_with_reboot.py |   45 +++++++++++++++++++++++
 client/tests/kvm/tests_base.cfg.sample          |    5 +++
 2 files changed, 50 insertions(+), 0 deletions(-)
 create mode 100644 client/tests/kvm/tests/migration_with_reboot.py

diff --git a/client/tests/kvm/tests/migration_with_reboot.py b/client/tests/kvm/tests/migration_with_reboot.py
new file mode 100644
index 0000000..985cb3a
--- /dev/null
+++ b/client/tests/kvm/tests/migration_with_reboot.py
@@ -0,0 +1,45 @@
+import logging, time
+import threading
+from autotest_lib.client.common_lib import error
+import kvm_subprocess, kvm_test_utils, kvm_utils
+
+
+def run_migration_with_reboot(test, params, env):
+    """
+    KVM migration test:
+    1) Get a live VM and clone it.
+    2) Verify that the source VM supports migration.  If it does, proceed with
+            the test.
+    3) Reboot the VM
+    4) Send a migration command to the source VM and wait until it's finished.
+    5) Kill off the source VM.
+    6) Log into the destination VM after the migration is finished.
+
+    @param test: kvm test object.
+    @param params: Dictionary with test parameters.
+    @param env: Dictionary with the test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    timeout = int(params.get("login_timeout", 360))
+    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
+
+    mig_timeout = float(params.get("mig_timeout", "3600"))
+    mig_protocol = params.get("migration_protocol", "tcp")
+    mig_cancel = bool(params.get("mig_cancel"))
+    bg = None
+
+    try:
+        # reboot the VM in background
+        bg = kvm_test_utils.BackgroundTest(kvm_test_utils.reboot, (vm, session))
+        bg.start()
+
+        while bg.is_alive():
+            # Migrate the VM
+            dest_vm = kvm_test_utils.migrate(vm, env, mig_timeout, mig_protocol,
+                                             False)
+            vm = dest_vm
+
+    finally:
+        if bg:
+            bg.join()
+        session.close()
diff --git a/client/tests/kvm/tests_base.cfg.sample b/client/tests/kvm/tests_base.cfg.sample
index c8fbf50..1565dce 100644
--- a/client/tests/kvm/tests_base.cfg.sample
+++ b/client/tests/kvm/tests_base.cfg.sample
@@ -150,6 +150,11 @@ variants:
             - mig_cancel:
                 migration_protocol = "tcp"
                 mig_cancel = True
+        variants:
+            - @default:
+            - with_reboot:
+                iterations = 1
+                type = migration_with_reboot
 
     - boot_savevm: install setup unattended_install.cdrom
         type = boot_savevm

[PATCH 3/3] KVM test: Test the file transfer during migartion

[Jason Wang]

This test just do the file transfer from host to guest during migartion in order
to check whether the nic/block state could be saved and loaded correctly.

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 .../kvm/tests/migration_with_file_transfer.py      |   59 ++++++++++++++++++++
 client/tests/kvm/tests_base.cfg.sample             |    7 ++
 2 files changed, 66 insertions(+), 0 deletions(-)
 create mode 100644 client/tests/kvm/tests/migration_with_file_transfer.py

diff --git a/client/tests/kvm/tests/migration_with_file_transfer.py b/client/tests/kvm/tests/migration_with_file_transfer.py
new file mode 100644
index 0000000..8a316bf
--- /dev/null
+++ b/client/tests/kvm/tests/migration_with_file_transfer.py
@@ -0,0 +1,59 @@
+import logging, time
+from autotest_lib.client.common_lib import utils, error
+import kvm_subprocess, kvm_test_utils, kvm_utils
+
+
+def run_migration_with_file_transfer(test, params, env):
+    """
+    KVM migration test:
+    1) Get a live VM and clone it.
+    2) Verify that the source VM supports migration.  If it does, proceed with
+            the test.
+    3) Reboot the VM
+    4) Send a migration command to the source VM and wait until it's finished.
+    5) Kill off the source VM.
+    6) Log into the destination VM after the migration is finished.
+
+    @param test: kvm test object.
+    @param params: Dictionary with test parameters.
+    @param env: Dictionary with the test environment.
+    """
+
+    def transfer_test(vm, host_path, guest_path, timeout=120):
+        """
+        vm.copy_files_to does not raise exception, so we need a wrapper
+        in order to make it to be used by BackgroundTest.
+        """
+        if not vm.copy_files_to(host_path, guest_path, timeout=timeout):
+            raise error.TestError("Fail to do the file transfer!")
+
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    timeout = int(params.get("login_timeout", 360))
+    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
+
+    mig_timeout = float(params.get("mig_timeout", "3600"))
+    mig_protocol = params.get("migration_protocol", "tcp")
+
+    guest_path = params.get("guest_path", "/tmp")
+    file_size = params.get("file_size", "1000")
+    transfer_timeout = int(params.get("transfer_timeout", "240"))
+    bg = None
+
+    try:
+        utils.run("dd if=/dev/zero of=/tmp/file bs=1M count=%s" % file_size)
+
+        # Transfer file from host to guest
+        bg = kvm_test_utils.BackgroundTest(transfer_test,
+                                           (vm, "/tmp/file", guest_path,
+                                            transfer_timeout))
+        bg.start()
+
+        while bg.is_alive():
+            logging.info("File transfer is not ended, start a round of migration ...")
+            # Migrate the VM
+            dest_vm = kvm_test_utils.migrate(vm, env, mig_timeout, mig_protocol, False)
+            vm = dest_vm
+    finally:
+        if bg: bg.join()
+        session.close()
+        utils.run("rm -rf /tmp/zero")
diff --git a/client/tests/kvm/tests_base.cfg.sample b/client/tests/kvm/tests_base.cfg.sample
index 1565dce..56ea7f7 100644
--- a/client/tests/kvm/tests_base.cfg.sample
+++ b/client/tests/kvm/tests_base.cfg.sample
@@ -155,6 +155,9 @@ variants:
             - with_reboot:
                 iterations = 1
                 type = migration_with_reboot
+            - with_file_transfer:
+                iterations = 1
+                type = migration_with_file_transfer
 
     - boot_savevm: install setup unattended_install.cdrom
         type = boot_savevm
@@ -1278,6 +1281,10 @@ variants:
             migration_bg_command = start ping -t localhost
             migration_bg_check_command = tasklist | find /I "ping.exe"
             migration_bg_kill_command = taskkill /IM ping.exe /F
+        migrate_with_file_transfer:
+            guest_path = C:\
+            rtl8139:
+                file_size = 10
         stress_boot:
             alive_test_cmd = systeminfo
         timedrift:

Re: [PATCH 3/3] KVM test: Test the file transfer during migartion

[Michael Goldish]

On 09/25/2010 11:36 AM, Jason Wang wrote:
> This test just do the file transfer from host to guest during migartion in order
> to check whether the nic/block state could be saved and loaded correctly.
> 
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  .../kvm/tests/migration_with_file_transfer.py      |   59 ++++++++++++++++++++
>  client/tests/kvm/tests_base.cfg.sample             |    7 ++
>  2 files changed, 66 insertions(+), 0 deletions(-)
>  create mode 100644 client/tests/kvm/tests/migration_with_file_transfer.py
> 
> diff --git a/client/tests/kvm/tests/migration_with_file_transfer.py b/client/tests/kvm/tests/migration_with_file_transfer.py
> new file mode 100644
> index 0000000..8a316bf
> --- /dev/null
> +++ b/client/tests/kvm/tests/migration_with_file_transfer.py
> @@ -0,0 +1,59 @@
> +import logging, time
> +from autotest_lib.client.common_lib import utils, error
> +import kvm_subprocess, kvm_test_utils, kvm_utils
> +
> +
> +def run_migration_with_file_transfer(test, params, env):
> +    """
> +    KVM migration test:
> +    1) Get a live VM and clone it.
> +    2) Verify that the source VM supports migration.  If it does, proceed with
> +            the test.
> +    3) Reboot the VM
> +    4) Send a migration command to the source VM and wait until it's finished.
> +    5) Kill off the source VM.
> +    6) Log into the destination VM after the migration is finished.
> +
> +    @param test: kvm test object.
> +    @param params: Dictionary with test parameters.
> +    @param env: Dictionary with the test environment.
> +    """
> +
> +    def transfer_test(vm, host_path, guest_path, timeout=120):
> +        """
> +        vm.copy_files_to does not raise exception, so we need a wrapper
> +        in order to make it to be used by BackgroundTest.
> +        """
> +        if not vm.copy_files_to(host_path, guest_path, timeout=timeout):
> +            raise error.TestError("Fail to do the file transfer!")
> +
> +    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
> +    timeout = int(params.get("login_timeout", 360))
> +    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
> +
> +    mig_timeout = float(params.get("mig_timeout", "3600"))
> +    mig_protocol = params.get("migration_protocol", "tcp")
> +
> +    guest_path = params.get("guest_path", "/tmp")
> +    file_size = params.get("file_size", "1000")
> +    transfer_timeout = int(params.get("transfer_timeout", "240"))
> +    bg = None
> +
> +    try:
> +        utils.run("dd if=/dev/zero of=/tmp/file bs=1M count=%s" % file_size)

Wouldn't it be useful to read from /dev/urandom instead of /dev/zero,
and compare the md5 hash of the file generated on the host with the md5
hash of the file that arrives in the guest?

> +
> +        # Transfer file from host to guest
> +        bg = kvm_test_utils.BackgroundTest(transfer_test,
> +                                           (vm, "/tmp/file", guest_path,
> +                                            transfer_timeout))
> +        bg.start()
> +
> +        while bg.is_alive():
> +            logging.info("File transfer is not ended, start a round of migration ...")
> +            # Migrate the VM
> +            dest_vm = kvm_test_utils.migrate(vm, env, mig_timeout, mig_protocol, False)
> +            vm = dest_vm
> +    finally:
> +        if bg: bg.join()
> +        session.close()
> +        utils.run("rm -rf /tmp/zero")

Re: [PATCH 3/3] KVM test: Test the file transfer during migartion

[Jason Wang]

Michael Goldish writes:
 > On 09/25/2010 11:36 AM, Jason Wang wrote:
 > > This test just do the file transfer from host to guest during migartion in order
 > > to check whether the nic/block state could be saved and loaded correctly.
 > > 
 > > Signed-off-by: Jason Wang <jasowang@redhat.com>
 > > ---
 > >  .../kvm/tests/migration_with_file_transfer.py      |   59 ++++++++++++++++++++
 > >  client/tests/kvm/tests_base.cfg.sample             |    7 ++
 > >  2 files changed, 66 insertions(+), 0 deletions(-)
 > >  create mode 100644 client/tests/kvm/tests/migration_with_file_transfer.py
 > > 
 > > diff --git a/client/tests/kvm/tests/migration_with_file_transfer.py b/client/tests/kvm/tests/migration_with_file_transfer.py
 > > new file mode 100644
 > > index 0000000..8a316bf
 > > --- /dev/null
 > > +++ b/client/tests/kvm/tests/migration_with_file_transfer.py
 > > @@ -0,0 +1,59 @@
 > > +import logging, time
 > > +from autotest_lib.client.common_lib import utils, error
 > > +import kvm_subprocess, kvm_test_utils, kvm_utils
 > > +
 > > +
 > > +def run_migration_with_file_transfer(test, params, env):
 > > +    """
 > > +    KVM migration test:
 > > +    1) Get a live VM and clone it.
 > > +    2) Verify that the source VM supports migration.  If it does, proceed with
 > > +            the test.
 > > +    3) Reboot the VM
 > > +    4) Send a migration command to the source VM and wait until it's finished.
 > > +    5) Kill off the source VM.
 > > +    6) Log into the destination VM after the migration is finished.
 > > +
 > > +    @param test: kvm test object.
 > > +    @param params: Dictionary with test parameters.
 > > +    @param env: Dictionary with the test environment.
 > > +    """
 > > +
 > > +    def transfer_test(vm, host_path, guest_path, timeout=120):
 > > +        """
 > > +        vm.copy_files_to does not raise exception, so we need a wrapper
 > > +        in order to make it to be used by BackgroundTest.
 > > +        """
 > > +        if not vm.copy_files_to(host_path, guest_path, timeout=timeout):
 > > +            raise error.TestError("Fail to do the file transfer!")
 > > +
 > > +    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
 > > +    timeout = int(params.get("login_timeout", 360))
 > > +    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
 > > +
 > > +    mig_timeout = float(params.get("mig_timeout", "3600"))
 > > +    mig_protocol = params.get("migration_protocol", "tcp")
 > > +
 > > +    guest_path = params.get("guest_path", "/tmp")
 > > +    file_size = params.get("file_size", "1000")
 > > +    transfer_timeout = int(params.get("transfer_timeout", "240"))
 > > +    bg = None
 > > +
 > > +    try:
 > > +        utils.run("dd if=/dev/zero of=/tmp/file bs=1M count=%s" % file_size)
 > 
 > Wouldn't it be useful to read from /dev/urandom instead of /dev/zero,
 > and compare the md5 hash of the file generated on the host with the md5
 > hash of the file that arrives in the guest?
 > 

Yes, as the file transfer test have been applied, I would rebase this
case to use it.

 > > +
 > > +        # Transfer file from host to guest
 > > +        bg = kvm_test_utils.BackgroundTest(transfer_test,
 > > +                                           (vm, "/tmp/file", guest_path,
 > > +                                            transfer_timeout))
 > > +        bg.start()
 > > +
 > > +        while bg.is_alive():
 > > +            logging.info("File transfer is not ended, start a round of migration ...")
 > > +            # Migrate the VM
 > > +            dest_vm = kvm_test_utils.migrate(vm, env, mig_timeout, mig_protocol, False)
 > > +            vm = dest_vm
 > > +    finally:
 > > +        if bg: bg.join()
 > > +        session.close()
 > > +        utils.run("rm -rf /tmp/zero")
 > --
 > To unsubscribe from this list: send the line "unsubscribe kvm" in
 > the body of a message to majordomo@vger.kernel.org
 > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] Launch other test during migration

[Michael Goldish]

On 09/25/2010 11:36 AM, Jason Wang wrote:
> We could give a further test of migration by launch test during migartion. So
> the following series implements:
> 
> - A simple class to run a specified test in the background which could be used
> to launch other test during migartion. Its design is rather simple and its usage
> is a little tricky, but it work well.
> - Two sample tests which take advantages of the background class: Test reboot
> during guest migration and test file_transfer during guest migration.
> 
> In the future, we could even lauch autotest client test during guest migation.
> 
> ---
> 
> Jason Wang (3):
>       KVM Test: Introduce a helper class to run a test in the background
>       KVM test: Test reboot during migration
>       KVM test: Test the file transfer during migartion
> 
> 
>  client/tests/kvm/kvm_test_utils.py                 |   44 +++++++++++++++
>  .../kvm/tests/migration_with_file_transfer.py      |   59 ++++++++++++++++++++
>  client/tests/kvm/tests/migration_with_reboot.py    |   45 +++++++++++++++
>  client/tests/kvm/tests_base.cfg.sample             |   12 ++++
>  4 files changed, 159 insertions(+), 1 deletions(-)
>  create mode 100644 client/tests/kvm/tests/migration_with_file_transfer.py
>  create mode 100644 client/tests/kvm/tests/migration_with_reboot.py

It seems to me that this method is only applicable to tests/functions
that don't require a VM object (i.e. that require only a shell session
object).  kvm_test_utils.reboot() operates on a VM object, and the same
VM is destroyed by migrate() which runs in the background, so eventually
reboot() tries logging into a destroyed VM, which fails because
vm.get_address() fails.  Any monitor operation will also fail.  If the
autotest wrapper requires a VM object (currently it does) then it can't
be used either.

An alternative (somewhat ugly) way to migrate in the background is to
pass a boolean 'migrate' flag to various functions/tests, such as
reboot() and run_autotest().  If 'migrate' is True, these functions will
do something like

vm = kvm_test_utils.migrate(vm, ...)

in their waiting loops, where wait_for() is normally used.  This
guarantees that 'vm' is always a valid VM object.  For example:

# Log in after reboot
while time.time() < end_time:
    if migrate_in_bg:
        vm = kvm_test_utils.migrate(vm, ...)
    session = vm.remote_login()
    if session:
        break
    time.sleep(2)

This is much longer than the usual wait_for(...) but it does the job.
What do you think?

Re: [PATCH 0/3] Launch other test during migration

[Jason Wang]

Michael Goldish writes:
 > On 09/25/2010 11:36 AM, Jason Wang wrote:
 > > We could give a further test of migration by launch test during migartion. So
 > > the following series implements:
 > > 
 > > - A simple class to run a specified test in the background which could be used
 > > to launch other test during migartion. Its design is rather simple and its usage
 > > is a little tricky, but it work well.
 > > - Two sample tests which take advantages of the background class: Test reboot
 > > during guest migration and test file_transfer during guest migration.
 > > 
 > > In the future, we could even lauch autotest client test during guest migation.
 > > 
 > > ---
 > > 
 > > Jason Wang (3):
 > >       KVM Test: Introduce a helper class to run a test in the background
 > >       KVM test: Test reboot during migration
 > >       KVM test: Test the file transfer during migartion
 > > 
 > > 
 > >  client/tests/kvm/kvm_test_utils.py                 |   44 +++++++++++++++
 > >  .../kvm/tests/migration_with_file_transfer.py      |   59 ++++++++++++++++++++
 > >  client/tests/kvm/tests/migration_with_reboot.py    |   45 +++++++++++++++
 > >  client/tests/kvm/tests_base.cfg.sample             |   12 ++++
 > >  4 files changed, 159 insertions(+), 1 deletions(-)
 > >  create mode 100644 client/tests/kvm/tests/migration_with_file_transfer.py
 > >  create mode 100644 client/tests/kvm/tests/migration_with_reboot.py
 > 
 > It seems to me that this method is only applicable to tests/functions
 > that don't require a VM object (i.e. that require only a shell session
 > object).  kvm_test_utils.reboot() operates on a VM object, and the same
 > VM is destroyed by migrate() which runs in the background, so eventually
 > reboot() tries logging into a destroyed VM, which fails because
 > vm.get_address() fails.  Any monitor operation will also fail.  If the
 > autotest wrapper requires a VM object (currently it does) then it can't
 > be used either.
 > 

You are right and that's an issue when running test in parallel with
migration, but reboot through shell should work. The aim of this kind
of test is just to add some stress ( such as run_autotest) during
migartion, so most (probably all) of the tests only needs a
session. So I think it's not a very big issue in this situation.

 > An alternative (somewhat ugly) way to migrate in the background is to
 > pass a boolean 'migrate' flag to various functions/tests, such as
 > reboot() and run_autotest().  If 'migrate' is True, these functions will
 > do something like
 > 
 > vm = kvm_test_utils.migrate(vm, ...)
 > 
 > in their waiting loops, where wait_for() is normally used.  This
 > guarantees that 'vm' is always a valid VM object.  For example:
 > 
 > # Log in after reboot
 > while time.time() < end_time:
 >     if migrate_in_bg:
 >         vm = kvm_test_utils.migrate(vm, ...)
 >     session = vm.remote_login()
 >     if session:
 >         break
 >     time.sleep(2)
 > 
 > This is much longer than the usual wait_for(...) but it does the job.
 > What do you think?

This makes sense but it would let testcases need to care about the
migration and it's also hard to put all related codes into a wrapper
which would complicate the codes.

Despite the issue of vm object, all tests that only depends on the
shell session should works well with my method and it's more easy to
be extended. Maybe we could just warn the user of its usage and adapt
my method?

 > --
 > To unsubscribe from this list: send the line "unsubscribe kvm" in
 > the body of a message to majordomo@vger.kernel.org
 > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] Launch other test during migration

[Michael Goldish]

On 11/02/2010 07:34 AM, Jason Wang wrote:
> Michael Goldish writes:
>  > On 09/25/2010 11:36 AM, Jason Wang wrote:
>  > > We could give a further test of migration by launch test during migartion. So
>  > > the following series implements:
>  > > 
>  > > - A simple class to run a specified test in the background which could be used
>  > > to launch other test during migartion. Its design is rather simple and its usage
>  > > is a little tricky, but it work well.
>  > > - Two sample tests which take advantages of the background class: Test reboot
>  > > during guest migration and test file_transfer during guest migration.
>  > > 
>  > > In the future, we could even lauch autotest client test during guest migation.
>  > > 
>  > > ---
>  > > 
>  > > Jason Wang (3):
>  > >       KVM Test: Introduce a helper class to run a test in the background
>  > >       KVM test: Test reboot during migration
>  > >       KVM test: Test the file transfer during migartion
>  > > 
>  > > 
>  > >  client/tests/kvm/kvm_test_utils.py                 |   44 +++++++++++++++
>  > >  .../kvm/tests/migration_with_file_transfer.py      |   59 ++++++++++++++++++++
>  > >  client/tests/kvm/tests/migration_with_reboot.py    |   45 +++++++++++++++
>  > >  client/tests/kvm/tests_base.cfg.sample             |   12 ++++
>  > >  4 files changed, 159 insertions(+), 1 deletions(-)
>  > >  create mode 100644 client/tests/kvm/tests/migration_with_file_transfer.py
>  > >  create mode 100644 client/tests/kvm/tests/migration_with_reboot.py
>  > 
>  > It seems to me that this method is only applicable to tests/functions
>  > that don't require a VM object (i.e. that require only a shell session
>  > object).  kvm_test_utils.reboot() operates on a VM object, and the same
>  > VM is destroyed by migrate() which runs in the background, so eventually
>  > reboot() tries logging into a destroyed VM, which fails because
>  > vm.get_address() fails.  Any monitor operation will also fail.  If the
>  > autotest wrapper requires a VM object (currently it does) then it can't
>  > be used either.
>  > 
> 
> You are right and that's an issue when running test in parallel with
> migration, but reboot through shell should work. The aim of this kind
> of test is just to add some stress ( such as run_autotest) during
> migartion, so most (probably all) of the tests only needs a
> session. So I think it's not a very big issue in this situation.

Many tests need to be modified in order to require only a session.  I've
tried reboot and it doesn't work, and I can see that run_autotest() also
uses a VM.  Reboot needs the VM object in order to log in to make sure
the VM goes back up, and run_autotest() needs it for SCP and probably
is_alive().  I agree that some tests don't require a VM object, but the
ones that do are also interesting.

>  > An alternative (somewhat ugly) way to migrate in the background is to
>  > pass a boolean 'migrate' flag to various functions/tests, such as
>  > reboot() and run_autotest().  If 'migrate' is True, these functions will
>  > do something like
>  > 
>  > vm = kvm_test_utils.migrate(vm, ...)
>  > 
>  > in their waiting loops, where wait_for() is normally used.  This
>  > guarantees that 'vm' is always a valid VM object.  For example:
>  > 
>  > # Log in after reboot
>  > while time.time() < end_time:
>  >     if migrate_in_bg:
>  >         vm = kvm_test_utils.migrate(vm, ...)
>  >     session = vm.remote_login()
>  >     if session:
>  >         break
>  >     time.sleep(2)
>  > 
>  > This is much longer than the usual wait_for(...) but it does the job.
>  > What do you think?
> 
> This makes sense but it would let testcases need to care about the
> migration and it's also hard to put all related codes into a wrapper
> which would complicate the codes.
> 
> Despite the issue of vm object, all tests that only depends on the
> shell session should works well with my method and it's more easy to

We should also find a solution for tests that require a VM object.

Which other tests do you think it would be interesting to run during
migration, in addition to reboot(), run_autotest() and file_transfer()?

> be extended. Maybe we could just warn the user of its usage and adapt
> my method?

I think it's cleaner to just avoid passing a VM object to BackgroundTest.

Re: [PATCH 0/3] Launch other test during migration

[Jason Wang]

Michael Goldish writes:
 > On 11/02/2010 07:34 AM, Jason Wang wrote:
 > > Michael Goldish writes:
 > >  > On 09/25/2010 11:36 AM, Jason Wang wrote:
 > >  > > We could give a further test of migration by launch test during migartion. So
 > >  > > the following series implements:
 > >  > > 
 > >  > > - A simple class to run a specified test in the background which could be used
 > >  > > to launch other test during migartion. Its design is rather simple and its usage
 > >  > > is a little tricky, but it work well.
 > >  > > - Two sample tests which take advantages of the background class: Test reboot
 > >  > > during guest migration and test file_transfer during guest migration.
 > >  > > 
 > >  > > In the future, we could even lauch autotest client test during guest migation.
 > >  > > 
 > >  > > ---
 > >  > > 
 > >  > > Jason Wang (3):
 > >  > >       KVM Test: Introduce a helper class to run a test in the background
 > >  > >       KVM test: Test reboot during migration
 > >  > >       KVM test: Test the file transfer during migartion
 > >  > > 
 > >  > > 
 > >  > >  client/tests/kvm/kvm_test_utils.py                 |   44 +++++++++++++++
 > >  > >  .../kvm/tests/migration_with_file_transfer.py      |   59 ++++++++++++++++++++
 > >  > >  client/tests/kvm/tests/migration_with_reboot.py    |   45 +++++++++++++++
 > >  > >  client/tests/kvm/tests_base.cfg.sample             |   12 ++++
 > >  > >  4 files changed, 159 insertions(+), 1 deletions(-)
 > >  > >  create mode 100644 client/tests/kvm/tests/migration_with_file_transfer.py
 > >  > >  create mode 100644 client/tests/kvm/tests/migration_with_reboot.py
 > >  > 
 > >  > It seems to me that this method is only applicable to tests/functions
 > >  > that don't require a VM object (i.e. that require only a shell session
 > >  > object).  kvm_test_utils.reboot() operates on a VM object, and the same
 > >  > VM is destroyed by migrate() which runs in the background, so eventually
 > >  > reboot() tries logging into a destroyed VM, which fails because
 > >  > vm.get_address() fails.  Any monitor operation will also fail.  If the
 > >  > autotest wrapper requires a VM object (currently it does) then it can't
 > >  > be used either.
 > >  > 
 > > 
 > > You are right and that's an issue when running test in parallel with
 > > migration, but reboot through shell should work. The aim of this kind
 > > of test is just to add some stress ( such as run_autotest) during
 > > migartion, so most (probably all) of the tests only needs a
 > > session. So I think it's not a very big issue in this situation.
 > 
 > Many tests need to be modified in order to require only a session.  I've
 > tried reboot and it doesn't work, and I can see that run_autotest() also
 > uses a VM.  Reboot needs the VM object in order to log in to make sure
 > the VM goes back up, and run_autotest() needs it for SCP and probably
 > is_alive().  I agree that some tests don't require a VM object, but the
 > ones that do are also interesting.
 > 

Looks like every test need a VM object and we could not expect the
execution order among autotest threads, so if the thread created by
BackgroundTest was executed after migration, it would always fail.

I know re-use the VM object after migration involves lots of
modification, but is that possible or valuable?

And I think we can just focus on the tests which is useful to run in
parallel with migration. Your suggestions looks good but it only works
with the tests which have a step to wait for something like test
results. For the tests who does not have such step, another method is
needed.

 > >  > An alternative (somewhat ugly) way to migrate in the background is to
 > >  > pass a boolean 'migrate' flag to various functions/tests, such as
 > >  > reboot() and run_autotest().  If 'migrate' is True, these functions will
 > >  > do something like
 > >  > 
 > >  > vm = kvm_test_utils.migrate(vm, ...)
 > >  > 
 > >  > in their waiting loops, where wait_for() is normally used.  This
 > >  > guarantees that 'vm' is always a valid VM object.  For example:
 > >  > 
 > >  > # Log in after reboot
 > >  > while time.time() < end_time:
 > >  >     if migrate_in_bg:
 > >  >         vm = kvm_test_utils.migrate(vm, ...)
 > >  >     session = vm.remote_login()
 > >  >     if session:
 > >  >         break
 > >  >     time.sleep(2)
 > >  > 
 > >  > This is much longer than the usual wait_for(...) but it does the job.
 > >  > What do you think?
 > > 
 > > This makes sense but it would let testcases need to care about the
 > > migration and it's also hard to put all related codes into a wrapper
 > > which would complicate the codes.
 > > 
 > > Despite the issue of vm object, all tests that only depends on the
 > > shell session should works well with my method and it's more easy to
 > 
 > We should also find a solution for tests that require a VM object.
 > 
 > Which other tests do you think it would be interesting to run during
 > migration, in addition to reboot(), run_autotest() and
 > file_transfer()?

The most important test is run_autotst and maybe run_ping() and other
network related test.

 > 
 > > be extended. Maybe we could just warn the user of its usage and adapt
 > > my method?
 > 
 > I think it's cleaner to just avoid passing a VM object to BackgroundTest.

Re: [PATCH 0/3] Launch other test during migration

[Michael Goldish]

On 11/03/2010 04:53 AM, Jason Wang wrote:
> Michael Goldish writes:
>  > On 11/02/2010 07:34 AM, Jason Wang wrote:
>  > > Michael Goldish writes:
>  > >  > On 09/25/2010 11:36 AM, Jason Wang wrote:
>  > >  > > We could give a further test of migration by launch test during migartion. So
>  > >  > > the following series implements:
>  > >  > > 
>  > >  > > - A simple class to run a specified test in the background which could be used
>  > >  > > to launch other test during migartion. Its design is rather simple and its usage
>  > >  > > is a little tricky, but it work well.
>  > >  > > - Two sample tests which take advantages of the background class: Test reboot
>  > >  > > during guest migration and test file_transfer during guest migration.
>  > >  > > 
>  > >  > > In the future, we could even lauch autotest client test during guest migation.
>  > >  > > 
>  > >  > > ---
>  > >  > > 
>  > >  > > Jason Wang (3):
>  > >  > >       KVM Test: Introduce a helper class to run a test in the background
>  > >  > >       KVM test: Test reboot during migration
>  > >  > >       KVM test: Test the file transfer during migartion
>  > >  > > 
>  > >  > > 
>  > >  > >  client/tests/kvm/kvm_test_utils.py                 |   44 +++++++++++++++
>  > >  > >  .../kvm/tests/migration_with_file_transfer.py      |   59 ++++++++++++++++++++
>  > >  > >  client/tests/kvm/tests/migration_with_reboot.py    |   45 +++++++++++++++
>  > >  > >  client/tests/kvm/tests_base.cfg.sample             |   12 ++++
>  > >  > >  4 files changed, 159 insertions(+), 1 deletions(-)
>  > >  > >  create mode 100644 client/tests/kvm/tests/migration_with_file_transfer.py
>  > >  > >  create mode 100644 client/tests/kvm/tests/migration_with_reboot.py
>  > >  > 
>  > >  > It seems to me that this method is only applicable to tests/functions
>  > >  > that don't require a VM object (i.e. that require only a shell session
>  > >  > object).  kvm_test_utils.reboot() operates on a VM object, and the same
>  > >  > VM is destroyed by migrate() which runs in the background, so eventually
>  > >  > reboot() tries logging into a destroyed VM, which fails because
>  > >  > vm.get_address() fails.  Any monitor operation will also fail.  If the
>  > >  > autotest wrapper requires a VM object (currently it does) then it can't
>  > >  > be used either.
>  > >  > 
>  > > 
>  > > You are right and that's an issue when running test in parallel with
>  > > migration, but reboot through shell should work. The aim of this kind
>  > > of test is just to add some stress ( such as run_autotest) during
>  > > migartion, so most (probably all) of the tests only needs a
>  > > session. So I think it's not a very big issue in this situation.
>  > 
>  > Many tests need to be modified in order to require only a session.  I've
>  > tried reboot and it doesn't work, and I can see that run_autotest() also
>  > uses a VM.  Reboot needs the VM object in order to log in to make sure
>  > the VM goes back up, and run_autotest() needs it for SCP and probably
>  > is_alive().  I agree that some tests don't require a VM object, but the
>  > ones that do are also interesting.
>  > 
> 
> Looks like every test need a VM object and we could not expect the
> execution order among autotest threads, so if the thread created by
> BackgroundTest was executed after migration, it would always fail.
> 
> I know re-use the VM object after migration involves lots of
> modification, but is that possible or valuable?

I don't think it's worth it.  A qemu process waiting for migration has a
monitor of its own, so it needs to have an instance id of its own.  I
suppose we can refactor everything to get around that somehow, but it'll
be messy.

Another option is to pass env and the string 'vm1' to functions, instead
of passing the VM object itself.  migrate() replaces 'vm1' in env, so
whenever the test accesses env['vm1'] it'll get the right VM.  However,
it's still not safe to use the monitor of a VM that's going to be
destroyed and replaced, and if the monitor can't be used safely,
allowing access to the VM is a bit unclean IMO.

> And I think we can just focus on the tests which is useful to run in
> parallel with migration. Your suggestions looks good but it only works
> with the tests which have a step to wait for something like test
> results. For the tests who does not have such step, another method is
> needed.

I think most interesting tests have some sort of loop or waiting step in
them.  For run_autotest(), we should insert the calls to migrate() in a
loop that waits for the test to complete (i.e. waits for the shell
prompt to return).  The only case I can think of where another method
might be needed is the file transfer test, where the waiting loop is
deep inside some internal function.  However, even if we use
BackgroundTest for file transfer, it's still not nice to pass the VM
object (by calling vm.copy_files_to()) knowing that the same object will
be destroyed soon.

>  > >  > An alternative (somewhat ugly) way to migrate in the background is to
>  > >  > pass a boolean 'migrate' flag to various functions/tests, such as
>  > >  > reboot() and run_autotest().  If 'migrate' is True, these functions will
>  > >  > do something like
>  > >  > 
>  > >  > vm = kvm_test_utils.migrate(vm, ...)
>  > >  > 
>  > >  > in their waiting loops, where wait_for() is normally used.  This
>  > >  > guarantees that 'vm' is always a valid VM object.  For example:
>  > >  > 
>  > >  > # Log in after reboot
>  > >  > while time.time() < end_time:
>  > >  >     if migrate_in_bg:
>  > >  >         vm = kvm_test_utils.migrate(vm, ...)
>  > >  >     session = vm.remote_login()
>  > >  >     if session:
>  > >  >         break
>  > >  >     time.sleep(2)
>  > >  > 
>  > >  > This is much longer than the usual wait_for(...) but it does the job.
>  > >  > What do you think?
>  > > 
>  > > This makes sense but it would let testcases need to care about the
>  > > migration and it's also hard to put all related codes into a wrapper
>  > > which would complicate the codes.
>  > > 
>  > > Despite the issue of vm object, all tests that only depends on the
>  > > shell session should works well with my method and it's more easy to
>  > 
>  > We should also find a solution for tests that require a VM object.
>  > 
>  > Which other tests do you think it would be interesting to run during
>  > migration, in addition to reboot(), run_autotest() and
>  > file_transfer()?
> 
> The most important test is run_autotst and maybe run_ping() and other
> network related test.
> 
>  > 
>  > > be extended. Maybe we could just warn the user of its usage and adapt
>  > > my method?
>  > 
>  > I think it's cleaner to just avoid passing a VM object to BackgroundTest.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] Launch other test during migration

[Jason Wang]

Hello guys:

Any further suggestion which I need to improve those patches?

I agree that there's no much tests need to be run is this way except for
migration. In order to validate the function of migration, many tests need to be
run in parallel with migration and this series is just aimed at this.

One major advantage of this is that it could greatly simplified the test design
and could reuse existed test cases without modification. Without this, we must
split the tests cases itself or split the migration test and modification to 
existed code is also required.

One major issue is that not all tests could be run in this way, tests which
needs to do monitor operation may not work well, but ususally this kind of test
is meaningless during migration.

Another probably issue is that we can not control precisely when the migration
start, but it's not important to tests I post here because the background tests
are usually launch as a kind of stress and need much more time than just a
single migration to complete, so what we need to do here is just let the
migration run until the background tests finish.

The fact is that these tests work well and have find real issues with migration.

Any comments or suggestions?

----- "Jason Wang" <jasowang@redhat.com> wrote:

> We could give a further test of migration by launch test during
> migartion. So
> the following series implements:
> 
> - A simple class to run a specified test in the background which could
> be used
> to launch other test during migartion. Its design is rather simple and
> its usage
> is a little tricky, but it work well.
> - Two sample tests which take advantages of the background class: Test
> reboot
> during guest migration and test file_transfer during guest migration.
> 
> In the future, we could even lauch autotest client test during guest
> migation.
> 
> ---
> 
> Jason Wang (3):
>       KVM Test: Introduce a helper class to run a test in the
> background
>       KVM test: Test reboot during migration
>       KVM test: Test the file transfer during migartion
> 
> 
>  client/tests/kvm/kvm_test_utils.py                 |   44
> +++++++++++++++
>  .../kvm/tests/migration_with_file_transfer.py      |   59
> ++++++++++++++++++++
>  client/tests/kvm/tests/migration_with_reboot.py    |   45
> +++++++++++++++
>  client/tests/kvm/tests_base.cfg.sample             |   12 ++++
>  4 files changed, 159 insertions(+), 1 deletions(-)
>  create mode 100644
> client/tests/kvm/tests/migration_with_file_transfer.py
>  create mode 100644 client/tests/kvm/tests/migration_with_reboot.py
> 
> -- 
> Jason Wang
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] Launch other test during migration

[pradeep]

On Mon, 18 Oct 2010 00:59:20 -0400 (EDT)
Jason Wang <jasowang@redhat.com> wrote:

> Hello guys:
> 
> Any further suggestion which I need to improve those patches?
> 
> I agree that there's no much tests need to be run is this way except
> for migration. In order to validate the function of migration, many
> tests need to be run in parallel with migration and this series is
> just aimed at this.
> 
> One major advantage of this is that it could greatly simplified the
> test design and could reuse existed test cases without modification.
> Without this, we must split the tests cases itself or split the
> migration test and modification to existed code is also required.
> 
> One major issue is that not all tests could be run in this way, tests
> which needs to do monitor operation may not work well, but ususally
> this kind of test is meaningless during migration.
> 
> Another probably issue is that we can not control precisely when the
> migration start, but it's not important to tests I post here because
> the background tests are usually launch as a kind of stress and need
> much more time than just a single migration to complete, so what we
> need to do here is just let the migration run until the background
> tests finish.
> 
> The fact is that these tests work well and have find real issues with
> migration.
> 
> Any comments or suggestions?
> 
Good to cover different aspects of live migration using KVM 

In cloud,  When we are targetting for minimal downtime for migration, always we need to think about the performance impact of the
resources and actions.

we feel, its good to cover couple of below combinations.

 any state change of  a VM + migration

1. migration + reading data from host devices like CD, USB
2. migrate +file trasfer
3. migrate + reboot
4. internet download + migration
5. guest is given shutdown command and in parallel live migration
happens
6.hotplugging of nic, storage, memory, cpu + live migration


Thanks
Pradeep

Re: [PATCH 0/3] Launch other test during migration

[Jason Wang]

pradeep writes:
 > On Mon, 18 Oct 2010 00:59:20 -0400 (EDT)
 > Jason Wang <jasowang@redhat.com> wrote:
 > 
 > > Hello guys:
 > > 
 > > Any further suggestion which I need to improve those patches?
 > > 
 > > I agree that there's no much tests need to be run is this way except
 > > for migration. In order to validate the function of migration, many
 > > tests need to be run in parallel with migration and this series is
 > > just aimed at this.
 > > 
 > > One major advantage of this is that it could greatly simplified the
 > > test design and could reuse existed test cases without modification.
 > > Without this, we must split the tests cases itself or split the
 > > migration test and modification to existed code is also required.
 > > 
 > > One major issue is that not all tests could be run in this way, tests
 > > which needs to do monitor operation may not work well, but ususally
 > > this kind of test is meaningless during migration.
 > > 
 > > Another probably issue is that we can not control precisely when the
 > > migration start, but it's not important to tests I post here because
 > > the background tests are usually launch as a kind of stress and need
 > > much more time than just a single migration to complete, so what we
 > > need to do here is just let the migration run until the background
 > > tests finish.
 > > 
 > > The fact is that these tests work well and have find real issues with
 > > migration.
 > > 
 > > Any comments or suggestions?
 > > 
 > Good to cover different aspects of live migration using KVM 
 > 
 > In cloud,  When we are targetting for minimal downtime for migration, always we need to think about the performance impact of the
 > resources and actions.
 > 
 > we feel, its good to cover couple of below combinations.
 > 
 >  any state change of  a VM + migration
 > 
 > 1. migration + reading data from host devices like CD, USB
 > 2. migrate +file trasfer
 > 3. migrate + reboot
 > 4. internet download + migration
 > 5. guest is given shutdown command and in parallel live migration
 > happens
 > 6.hotplugging of nic, storage, memory, cpu + live migration
 > 
 > 

Thanks for these suggestions, I would try to implement these in the after we
find a solution to run a test in parallel with migrations.

 > Thanks
 > Pradeep
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 >