[PATCH 08/19] KVM test: Add basic file transfer test

[PATCH 08/19] KVM test: Add basic file transfer test

[Lucas Meneghel Rodrigues]

From: Amos Kong <akong@redhat.com>

This test is the basic test of transfering file between host and guest.
Try to transfer a large file from host to guest, and transfer it back
to host, then compare the files by calculate their md5 hash.

The default file size is 4000M, scp timeout is 1000s. It means if the
average speed is less than 4M/s, this test will be fail. We can extend
this test by using another disk later, then we can transfer larger
files without the limit of first disk size.

Changes from v4:
- Make the test work on windows
- Generalize code, make it easier for people to implement more methods
- Print throughput data
- Avoid using shell commands to do things that can be perfectly done
  using the python API, such as remove files on host

Changes from v1:
- Use md5 to verify the integrity of files
- Try to use autotest API, such as, utils.system()

Signed-off-by: Amos Kong <akong@redhat.com>
---
 client/tests/kvm/tests/file_transfer.py |   90 +++++++++++++++++++++++++++++++
 client/tests/kvm/tests_base.cfg.sample  |   16 +++++-
 2 files changed, 105 insertions(+), 1 deletions(-)
 create mode 100644 client/tests/kvm/tests/file_transfer.py

diff --git a/client/tests/kvm/tests/file_transfer.py b/client/tests/kvm/tests/file_transfer.py
new file mode 100644
index 0000000..e872bed
--- /dev/null
+++ b/client/tests/kvm/tests/file_transfer.py
@@ -0,0 +1,90 @@
+import logging, commands, re, time, os
+from autotest_lib.client.common_lib import error
+from autotest_lib.client.bin import utils
+import kvm_utils, kvm_test_utils
+
+def run_file_transfer(test, params, env):
+    """
+    Test ethrnet device function by ethtool
+
+    1) Boot up a VM.
+    2) Create a large file by dd on host.
+    3) Copy this file from host to guest.
+    4) Copy this file from guest to host.
+    5) Check if file transfers ended good.
+
+    @param test: KVM test object.
+    @param params: Dictionary with the test parameters.
+    @param env: Dictionary with test environment.
+    """
+    vm = kvm_test_utils.get_living_vm(env, params.get("main_vm"))
+    timeout=int(params.get("login_timeout", 360))
+
+    session = kvm_test_utils.wait_for_login(vm, timeout=timeout)
+    if not session:
+        raise error.TestFail("Could not log into guest '%s'" % vm.name)
+
+    dir = test.tmpdir
+    transfer_timeout = int(params.get("transfer_timeout"))
+    transfer_type = params.get("transfer_type")
+    tmp_dir = params.get("tmp_dir", "/tmp/")
+    clean_cmd = params.get("clean_cmd", "rm -f")
+    filesize = int(params.get("filesize", 4000))
+    cmd = "dd if=/dev/urandom of=%s/a.out bs=1M count=%d" % (dir, filesize)
+    guest_path = tmp_dir + "b.out"
+
+    try:
+        logging.info("Creating %dMB file on host", filesize)
+        utils.run(cmd)
+
+        if transfer_type == "remote":
+            logging.info("Transfering file host -> guest, timeout: %ss",
+                         transfer_timeout)
+            t_begin = time.time()
+            success = vm.copy_files_to("%s/a.out" % dir, guest_path,
+                                       timeout=transfer_timeout)
+            t_end = time.time()
+            throughput = filesize / (t_end - t_begin)
+            if not success:
+                raise error.TestFail("Fail to transfer file from host to guest")
+            logging.info("File transfer host -> guest succeed, "
+                         "estimated throughput: %.2fMB/s", throughput)
+
+            logging.info("Transfering file guest -> host, timeout: %ss",
+                         transfer_timeout)
+            t_begin = time.time()
+            success = vm.copy_files_from(guest_path, "%s/c.out" % dir,
+                                         timeout=transfer_timeout)
+            t_end = time.time()
+            throughput = filesize / (t_end - t_begin)
+            if not success:
+                raise error.TestFail("Fail to transfer file from guest to host")
+            logging.info("File transfer guest -> host succeed, "
+                         "estimated throughput: %.2fMB/s", throughput)
+        else:
+            raise error.TestError("Unknown test file transfer mode %s" %
+                                  transfer_type)
+
+        for f in ['a.out', 'c.out']:
+            p = os.path.join(dir, f)
+            size = os.path.getsize(p)
+            logging.debug('Size of %s: %sB', f, size)
+
+        md5_orig = utils.hash_file("%s/a.out" % dir, method="md5")
+        md5_new = utils.hash_file("%s/c.out" % dir, method="md5")
+
+        if md5_orig != md5_new:
+            raise error.TestFail("File changed after transfer host -> guest "
+                                 "and guest -> host")
+
+    finally:
+        logging.info('Cleaning temp file on guest')
+        clean_cmd += " %s" % guest_path
+        s, o = session.get_command_status_output(clean_cmd)
+        if s:
+            logging.warning("Failed to clean remote file %s, output:%s",
+                            guest_path, o)
+        logging.info('Cleaning temp files on host')
+        os.remove('%s/a.out' % dir)
+        os.remove('%s/c.out' % dir)
+        session.close()
diff --git a/client/tests/kvm/tests_base.cfg.sample b/client/tests/kvm/tests_base.cfg.sample
index cf1deaf..5003008 100644
--- a/client/tests/kvm/tests_base.cfg.sample
+++ b/client/tests/kvm/tests_base.cfg.sample
@@ -476,6 +476,14 @@ variants:
     - jumbo: install setup unattended_install.cdrom
         type = jumbo
 
+    - file_transfer: install setup unattended_install.cdrom
+        type = file_transfer
+        filesize = 4000
+        transfer_timeout = 1000
+        variants:
+            - remote:
+                transfer_type = remote
+
     - physical_resources_check: install setup unattended_install.cdrom
         type = physical_resources_check
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'
@@ -679,6 +687,9 @@ variants:
                 time_command = date +'TIME: %a %m/%d/%Y %H:%M:%S.%N'
                 time_filter_re = "(?:TIME: \w\w\w )(.{19})(?:\.\d\d)"
                 time_format = "%m/%d/%Y %H:%M:%S"
+        file_transfer:
+            tmp_dir = /tmp/
+            clean_cmd = rm -f
 
         variants:
             - Fedora:
@@ -1250,7 +1261,7 @@ variants:
 
     # Windows section
     - @Windows:
-        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo
+        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
@@ -1342,6 +1353,9 @@ variants:
         physical_resources_check:
             catch_uuid_cmd = 
 
+        file_transfer:
+            tmp_dir = C:\
+            clean_cmd = del
         variants:
             - Win2000:
                 no reboot whql
-- 
1.7.1

Re: [Autotest] [PATCH 08/19] KVM test: Add basic file transfer test

[Lucas Meneghel Rodrigues]

On Wed, 2010-09-29 at 13:48 -0400, Lucas Meneghel Rodrigues wrote: 
>      # Windows section
>      - @Windows:
> -        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo
> +        no autotest linux_s3 vlan_tag ioquit unattended_install.(url|nfs|remote_ks) jumbo file_transfer

^^^ This is fixed on the patchset, there's a last patch that takes file
transfer out of the blocked tests. I made this to avoid a painful rebase
just because of a single line. If you want to test this with windows,
please make sure file_transfer is not blocked here on your
tests_base.cfg, *please*.