Re: [KVM_AUTOTEST] add kvm hugepage variant

["Lukáš Doktor" <ldoktor@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 2535 bytes --]

Well, thank you for notifications, I'll keep them in my mind.

Also the problem with mempath vs. mem-path is solved. It was just a 
misspell in one version of KVM.

* fixed patch attached

Dne 20.7.2009 14:58, Lucas Meneghel Rodrigues napsal(a):
> On Fri, 2009-07-10 at 12:01 +0200, Lukáš Doktor wrote:
>> After discussion I split the patches.
>
> Hi Lukáš, sorry for the delay answering your patch. Looks good to me in
> general, I have some remarks to make:
>
> 1) When posting patches to the autotest kvm tests, please cross post the
> autotest mailing list (autotest@test.kernel.org) and the KVM list.
>
> 2) About scripts to prepare the environment to perform tests - we've had
> some discussion about including shell scripts on autotest. Bottom line,
> autotest has a policy of not including non python code when possible
> [1]. So, would you mind re-creating your hugepage setup code in python
> and re-sending it?
>
> Thanks for your contribution, looking forward getting it integrated to
> our tests.
>
> [1] Unless when it is not practical for testing purposes - writing tests
> in C is just fine, for example.
>
>> This patch adds kvm_hugepage variant. It prepares the host system and
>> start vm with -mem-path option. It does not clean after itself, because
>>    it's impossible to unmount and free hugepages before all guests are
>> destroyed.
>>
>> I need to ask you what to do with change of qemu parameter. Newest
>> versions are using -mempath insted of -mem-path. This is impossible to
>> fix using current config file. I can see 2 solutions:
>> 1) direct change in kvm_vm.py (parse output and try another param)
>> 2) detect qemu capabilities outside and create additional layer (better
>> for future occurrence)
>>
>> Dne 9.7.2009 11:24, Lukáš Doktor napsal(a):
>>> This patch adds kvm_hugepage variant. It prepares the host system and
>>> start vm with -mem-path option. It does not clean after itself, because
>>> it's impossible to unmount and free hugepages before all guests are
>>> destroyed.
>>>
>>> There is also added autotest.libhugetlbfs test.
>>>
>>> I need to ask you what to do with change of qemu parameter. Newest
>>> versions are using -mempath insted of -mem-path. This is impossible to
>>> fix using current config file. I can see 2 solutions:
>>> 1) direct change in kvm_vm.py (parse output and try another param)
>>> 2) detect qemu capabilities outside and create additional layer (better
>>> for future occurrence)
>>>
>>> Tested by:ldoktor@redhat.com on RHEL5.4 with kvm-83-72.el5
>


[-- Attachment #2: kvm_hugepage.patch --]
[-- Type: text/plain, Size: 3315 bytes --]

diff --git a/client/tests/kvm/kvm_tests.cfg.sample b/client/tests/kvm/kvm_tests.cfg.sample
index 5bd6eb8..70e290d 100644
--- a/client/tests/kvm/kvm_tests.cfg.sample
+++ b/client/tests/kvm/kvm_tests.cfg.sample
@@ -555,6 +555,13 @@ variants:
         only default
         image_format = raw
 
+variants:
+    - @kvm_smallpages:
+    - kvm_hugepages:
+        hugepage_path = /mnt/hugepage
+        pre_command = "/usr/bin/python scripts/hugepage.py"
+        extra_params += " -mem-path /mnt/hugepage"
+
 
 variants:
     - @basic:
@@ -568,6 +575,7 @@ variants:
         only Fedora.8.32
         only install setup boot shutdown
         only rtl8139
+        only kvm_smallpages
     - @sample1:
         only qcow2
         only ide
diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
index 48f2916..2b97ccc 100644
--- a/client/tests/kvm/kvm_vm.py
+++ b/client/tests/kvm/kvm_vm.py
@@ -412,6 +412,13 @@ class VM:
                 self.destroy()
                 return False
 
+            if output:
+                logging.debug("qemu produced some output:\n%s", output)
+                if "alloc_mem_area" in output:
+                    logging.error("Could not allocate hugepage memory"
+                                 " -- qemu command:\n%s", qemu_command)
+                    return False
+
             logging.debug("VM appears to be alive with PID %d", self.pid)
             return True
 

diff -Narup a/client/tests/kvm/scripts/hugepage.py b/client/tests/kvm/scripts/
hugepage.py
--- a/client/tests/kvm/scripts/hugepage.py 1970-01-01 01:00:00.000000000 +0100
+++ a/client/tests/kvm/scripts/hugepage.py    2009-07-21 16:47:00.000000000 +0200
@@ -0,0 +1,63 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+# Alocates enough hugepages and mount hugetlbfs
+import os, sys, time
+
+# Variables check & set
+vms = os.environ['KVM_TEST_vms'].split().__len__()
+try:
+    max_vms = int(os.environ['KVM_TEST_max_vms'])
+except KeyError:
+    max_vms = 0
+mem = int(os.environ['KVM_TEST_mem'])
+hugepage_path = os.environ['KVM_TEST_hugepage_path']
+
+fmeminfo = open("/proc/meminfo", "r")
+while fmeminfo:
+       line = fmeminfo.readline()
+       if line.startswith("Hugepagesize"):
+               dumm, hp_size, dumm = line.split()
+               break
+fmeminfo.close()
+
+if not hp_size:
+    print "Could not get Hugepagesize from /proc/meminfo file"
+    raise ValueError
+
+if vms < max_vms:
+    vms = max_vms
+
+vmsm = ((vms * mem) + (vms * 64))
+target = (vmsm * 1024 / int(hp_size)) 
+
+# Iteratively set # of hugepages
+fhp = open("/proc/sys/vm/nr_hugepages", "r+")
+hp = fhp.readline()
+while int(hp) < target:
+    hp_ = hp
+    fhp.write(target.__str__())
+    fhp.flush()
+    time.sleep(5)
+    fhp.seek(0)
+    hp = int(fhp.readline())
+    if hp_ == hp:
+        raise MemoryError
+fhp.close()
+
+# Mount hugepage filesystem, if necessarily
+fmount = open("/proc/mounts", "r")
+mount = 1
+line = fmount.readline()
+while line:
+    if line.split()[1] == os.environ['KVM_TEST_hugepage_path']:
+        mount = 0
+        break
+    line = fmount.readline()
+fmount.close()
+
+if mount:
+    if not os.path.exists(hugepage_path):
+        os.makedirs(hugepage_path)
+    cmd = "mount -t hugetlbfs none %s" % (hugepage_path)
+    if os.system(cmd):
+        raise OSError