[KVM_AUTOTEST][PATCH] KSM_overcommit: dynamic reserve calculation (2)

[KVM_AUTOTEST][PATCH] KSM_overcommit: dynamic reserve calculation (2)

[Lukas Doktor]

Hi,

thanks for nice page about git workflow. I always wanted to try it but never had the time to sit down and learn...

Booth the TMPFS and 0.055 guest_reserve constant are set empirically using various RHEL and Fedora guest/hosts. Smaller hosts can work with smaller (0.045) constant but we didn't want to make the code more complex to exactly fit the limits.

Original changelog:
* NEW: guest_reserve and host_reserve are now calculated based on used memory
* NEW: tmpfs reserve is also evaluated to fit the overhead
* NEW: VM alive check during split_guest()
* FIX: In function split_guest() we used incorrect session
* MOD: Increase number of VNC ports

Best regards,
Lukas

[PATCH 1/4] Increase maximum number of VNC ports

[Lukas Doktor]

Signed-off-by: Lukas Doktor <ldoktor@redhat.com>
---
 client/tests/kvm/kvm_vm.py |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
index d1e0246..c203e14 100755
--- a/client/tests/kvm/kvm_vm.py
+++ b/client/tests/kvm/kvm_vm.py
@@ -396,7 +396,7 @@ class VM:
 
             # Find available VNC port, if needed
             if params.get("display") == "vnc":
-                self.vnc_port = kvm_utils.find_free_port(5900, 6000)
+                self.vnc_port = kvm_utils.find_free_port(5900, 6100)
 
             # Find random UUID if specified 'uuid = random' in config file
             if params.get("uuid") == "random":
-- 
1.6.2.5

[PATCH 2/4] Add VMs alive check while spliting the guest's pages

[Lukas Doktor]

Signed-off-by: Lukas Doktor <ldoktor@redhat.com>
---
 client/tests/kvm/tests/ksm_overcommit.py |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/client/tests/kvm/tests/ksm_overcommit.py b/client/tests/kvm/tests/ksm_overcommit.py
index 4aa6deb..b3d6880 100644
--- a/client/tests/kvm/tests/ksm_overcommit.py
+++ b/client/tests/kvm/tests/ksm_overcommit.py
@@ -142,6 +142,12 @@ def run_ksm_overcommit(test, params, env):
         session = None
         vm = None
         for i in range(1, vmsc):
+            # Check VMs
+            for j in range(0, vmsc):
+                if not lvms[j].is_alive:
+                    e_msg = "VM %d died while executing static_random_fill in"\
+                            " VM %d on allocator loop" % (j, i)
+                    raise error.TestFail(e_msg)
             vm = lvms[i]
             session = lsessions[i]
             a_cmd = "mem.static_random_fill()"
@@ -154,6 +160,10 @@ def run_ksm_overcommit(test, params, env):
                 logging.debug("Watching host memory while filling vm %s memory",
                               vm.name)
                 while not out.startswith("PASS") and not out.startswith("FAIL"):
+                    if not vm.is_alive():
+                        e_msg = "VM %d died while executing static_random_fill"\
+                                " on allocator loop" % i
+                        raise error.TestFail(e_msg)
                     free_mem = int(utils.read_from_meminfo("MemFree"))
                     if (ksm_swap):
                         free_mem = (free_mem +
-- 
1.6.2.5

[PATCH 3/4] FIX: Incorrect session in function split_guest()

[Lukas Doktor]

Signed-off-by: Jiri Zupka <jzupka@redhat.com>
---
 client/tests/kvm/tests/ksm_overcommit.py |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/client/tests/kvm/tests/ksm_overcommit.py b/client/tests/kvm/tests/ksm_overcommit.py
index b3d6880..24d6643 100644
--- a/client/tests/kvm/tests/ksm_overcommit.py
+++ b/client/tests/kvm/tests/ksm_overcommit.py
@@ -212,7 +212,7 @@ def run_ksm_overcommit(test, params, env):
 
         # Verify last machine with randomly generated memory
         a_cmd = "mem.static_random_verify()"
-        _execute_allocator(a_cmd, lvms[last_vm], session,
+        _execute_allocator(a_cmd, lvms[last_vm], lsessions[last_vm],
                            (mem / 200 * 50 * perf_ratio))
         logging.debug(kvm_test_utils.get_memory_info([lvms[last_vm]]))
 
-- 
1.6.2.5

[PATCH 4/4] guest_reserve, host_reserve and tmpfs overhead automatic calculation based on used memory

[Lukas Doktor]

Signed-off-by: Lukas Doktor <ldoktor@redhat.com>
Signed-off-by: Jiri Zupka <jzupka@redhat.com>
---
 client/tests/kvm/scripts/allocator.py    |   11 +++++---
 client/tests/kvm/tests/ksm_overcommit.py |   41 +++++++++++++++++++++++++++---
 client/tests/kvm/tests_base.cfg.sample   |    6 ++--
 3 files changed, 47 insertions(+), 11 deletions(-)

diff --git a/client/tests/kvm/scripts/allocator.py b/client/tests/kvm/scripts/allocator.py
index 1036893..227745a 100755
--- a/client/tests/kvm/scripts/allocator.py
+++ b/client/tests/kvm/scripts/allocator.py
@@ -8,10 +8,12 @@ Auxiliary script used to allocate memory on guests.
 """
 
 
-import os, array, sys, struct, random, copy, inspect, tempfile, datetime
+import os, array, sys, struct, random, copy, inspect, tempfile, datetime, math
 
 PAGE_SIZE = 4096 # machine page size
 
+TMPFS_OVERHEAD = 0.0022 # overhead on 1MB of write data 
+
 
 class MemFill(object):
     """
@@ -32,7 +34,8 @@ class MemFill(object):
 
         self.tmpdp = tempfile.mkdtemp()
         ret_code = os.system("mount -o size=%dM tmpfs %s -t tmpfs" %
-                             ((mem + 25), self.tmpdp))
+                             ((mem+math.ceil(mem*TMPFS_OVERHEAD)), 
+                             self.tmpdp))
         if ret_code != 0:
             if os.getuid() != 0:
                 print ("FAIL: Unable to mount tmpfs "
@@ -42,7 +45,7 @@ class MemFill(object):
         else:
             self.f = tempfile.TemporaryFile(prefix='mem', dir=self.tmpdp)
             self.allocate_by = 'L'
-            self.npages = (mem * 1024 * 1024) / PAGE_SIZE
+            self.npages = ((mem * 1024 * 1024) / PAGE_SIZE)
             self.random_key = random_key
             self.static_value = static_value
             print "PASS: Initialization"
@@ -83,7 +86,7 @@ class MemFill(object):
         @return: return array of bytes size PAGE_SIZE.
         """
         a = array.array("B")
-        for i in range(PAGE_SIZE / a.itemsize):
+        for i in range((PAGE_SIZE / a.itemsize)):
             try:
                 a.append(value)
             except:
diff --git a/client/tests/kvm/tests/ksm_overcommit.py b/client/tests/kvm/tests/ksm_overcommit.py
index 24d6643..8dc1722 100644
--- a/client/tests/kvm/tests/ksm_overcommit.py
+++ b/client/tests/kvm/tests/ksm_overcommit.py
@@ -348,12 +348,29 @@ def run_ksm_overcommit(test, params, env):
 
     # Main test code
     logging.info("Starting phase 0: Initialization")
+
     # host_reserve: mem reserve kept for the host system to run
-    host_reserve = int(params.get("ksm_host_reserve", 512))
+    host_reserve = int(params.get("ksm_host_reserve", -1))
+    if (host_reserve == -1):
+        # default host_reserve = MemAvailable + one_minimal_guest(128MB)
+        # later we add 64MB per additional guest
+        host_reserve = ((utils.memtotal() - utils.read_from_meminfo("MemFree"))
+                        / 1024 + 128)
+        # using default reserve
+        _host_reserve = True
+    else:
+        _host_reserve = False
+
     # guest_reserve: mem reserve kept to avoid guest OS to kill processes
-    guest_reserve = int(params.get("ksm_guest_reserve", 1024))
-    logging.debug("Memory reserved for host to run: %d", host_reserve)
-    logging.debug("Memory reserved for guest to run: %d", guest_reserve)
+    guest_reserve = int(params.get("ksm_guest_reserve", -1))
+    if (guest_reserve == -1):
+        # default guest_reserve = minimal_system_mem(256MB)
+        # later we add tmpfs overhead
+        guest_reserve = 256
+        # using default reserve
+        _guest_reserve = True
+    else:
+        _guest_reserve = False
 
     max_vms = int(params.get("max_vms", 2))
     overcommit = float(params.get("ksm_overcommit_ratio", 2.0))
@@ -365,6 +382,10 @@ def run_ksm_overcommit(test, params, env):
 
     if (params['ksm_mode'] == "serial"):
         max_alloc = vmsc
+        if _host_reserve:
+            # First round of additional guest reserves
+            host_reserve += vmsc * 64
+            _host_reserve = vmsc
 
     host_mem = (int(utils.memtotal()) / 1024 - host_reserve)
 
@@ -412,6 +433,10 @@ def run_ksm_overcommit(test, params, env):
             if mem - guest_reserve - 1 > 3100:
                 vmsc = int(math.ceil((host_mem * overcommit) /
                                      (3100 + guest_reserve)))
+                if _host_reserve:
+                    host_reserve += (vmsc - _host_reserve) * 64
+                    host_mem -= (vmsc - _host_reserve) * 64
+                    _host_reserve = vmsc
                 mem = int(math.floor(host_mem * overcommit / vmsc))
 
         if os.popen("uname -i").readline().startswith("i386"):
@@ -420,8 +445,16 @@ def run_ksm_overcommit(test, params, env):
             if mem > 3100 - 64:
                 vmsc = int(math.ceil((host_mem * overcommit) /
                                      (3100 - 64.0)))
+                if _host_reserve:
+                    host_reserve += (vmsc - _host_reserve) * 64
+                    host_mem -= (vmsc - _host_reserve) * 64
+                    _host_reserve = vmsc
                 mem = int(math.floor(host_mem * overcommit / vmsc))
 
+    # 0.055 represents OS + TMPFS additional reserve per guest ram MB
+    if _guest_reserve:
+        guest_reserve += math.ceil(mem * 0.055)
+
     swap = int(utils.read_from_meminfo("SwapTotal")) / 1024
 
     logging.debug("Overcommit = %f", overcommit)
diff --git a/client/tests/kvm/tests_base.cfg.sample b/client/tests/kvm/tests_base.cfg.sample
index bb3646c..06eaa74 100644
--- a/client/tests/kvm/tests_base.cfg.sample
+++ b/client/tests/kvm/tests_base.cfg.sample
@@ -305,9 +305,9 @@ variants:
         ksm_overcommit_ratio = 3
         # Max paralel runs machine
         ksm_parallel_ratio = 4
-        # Host memory reserve
-        ksm_host_reserve = 512
-        ksm_guest_reserve = 1024
+        # Host memory reserve (default - best fit for used mem)
+        # ksm_host_reserve = 512
+        # ksm_guest_reserve = 1024
         variants:
             - ksm_serial:
                 ksm_mode = "serial"
-- 
1.6.2.5

Re: [Autotest] [KVM_AUTOTEST][PATCH] KSM_overcommit: dynamic reserve calculation (2)

[Lucas Meneghel Rodrigues]

Hi Lukas and Jiri - please hold on, cause this week I'm on vacation,
will be back next week and will finish work on your patchset.

Cheers,

On Sat, May 8, 2010 at 2:01 PM, Lukas Doktor <ldoktor@redhat.com> wrote:
> Hi,
>
> thanks for nice page about git workflow. I always wanted to try it but never had the time to sit down and learn...
>
> Booth the TMPFS and 0.055 guest_reserve constant are set empirically using various RHEL and Fedora guest/hosts. Smaller hosts can work with smaller (0.045) constant but we didn't want to make the code more complex to exactly fit the limits.
>
> Original changelog:
> * NEW: guest_reserve and host_reserve are now calculated based on used memory
> * NEW: tmpfs reserve is also evaluated to fit the overhead
> * NEW: VM alive check during split_guest()
> * FIX: In function split_guest() we used incorrect session
> * MOD: Increase number of VNC ports
>
> Best regards,
> Lukas
>
>
> _______________________________________________
> Autotest mailing list
> Autotest@test.kernel.org
> http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
>



-- 
Lucas

Re: [PATCH 1/4] Increase maximum number of VNC ports

[Lucas Meneghel Rodrigues]

On Sat, 2010-05-08 at 18:01 +0100, Lukas Doktor wrote:
> Signed-off-by: Lukas Doktor <ldoktor@redhat.com>

The new patch set looks good, commited them all, see:

http://autotest.kernel.org/changeset/4492

Thanks!

> ---
>  client/tests/kvm/kvm_vm.py |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
> index d1e0246..c203e14 100755
> --- a/client/tests/kvm/kvm_vm.py
> +++ b/client/tests/kvm/kvm_vm.py
> @@ -396,7 +396,7 @@ class VM:
>  
>              # Find available VNC port, if needed
>              if params.get("display") == "vnc":
> -                self.vnc_port = kvm_utils.find_free_port(5900, 6000)
> +                self.vnc_port = kvm_utils.find_free_port(5900, 6100)
>  
>              # Find random UUID if specified 'uuid = random' in config file
>              if params.get("uuid") == "random":