[PATCH] enforce dom0 cpus and balloon out memory

[PATCH] enforce dom0 cpus and balloon out memory

[aq]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

This patch does 2 jobs:

- Enforce the number of CPUs dom0 will take. See the new variable
"dom0-cpus" in xend-config.sxp (you will want to set this variable to
1 on SMP systems)

- Balloon out dom0 memory when creating domU, if there is not enough
free memory. The lowest level we will balloon out is configured via
the new variable "dom0-min-mem" in xend-config.sxp


I still have a doubt: where to put the code to enforce dom0-cpus. At
the moment I put it into
python/xen/xend/server/SrvDaemon.py, and hopefully that is resonable
enough. Any comment?



Signed-off-by: Nguyen Anh Quynh <aquynh@gmail.com>



$ diffstat enforce_resource.patch 
 examples/xend-config.sxp            |    8 +++++++
 python/xen/xend/XendRoot.py         |   10 +++++++++
 python/xen/xend/server/SrvDaemon.py |   30 +++++++++++++++++++++++++++-
 python/xen/xm/create.py             |   38 ++++++++++++++++++++++++++++++++++++
 4 files changed, 85 insertions(+), 1 deletion(-)

[-- Attachment #2: enforce_resource.patch --]
[-- Type: application/octet-stream, Size: 5375 bytes --]

diff -r 501a70f3ae96 tools/examples/xend-config.sxp
--- a/tools/examples/xend-config.sxp	Thu Jul 28 12:34:45 2005
+++ b/tools/examples/xend-config.sxp	Fri Jul 29 11:55:38 2005
@@ -44,3 +44,11 @@
 # Setup script for enbd-backed block devices
 (block-enbd block-enbd)
 
+# Dom0 will balloon out when needed to free memory for domU.
+# dom0-min-mem is the lowest memory level (in MB) dom0 will get down to.
+# If dom0-min-mem=0, dom0 will never balloon out.
+(dom0-min-mem 0)
+
+# In SMP system, dom0 will use only CPUs in range [1,dom0-cpus]
+# If dom0-cpus = 0, dom0 will take all cpus available
+(dom0-cpus 0)
diff -r 501a70f3ae96 tools/python/xen/xend/XendRoot.py
--- a/tools/python/xen/xend/XendRoot.py	Thu Jul 28 12:34:45 2005
+++ b/tools/python/xen/xend/XendRoot.py	Fri Jul 29 11:55:38 2005
@@ -75,6 +75,10 @@
 
     """Default port xend serves consoles at. """
     console_port_base_default = '9600'
+
+    dom0_min_mem_default = '0'
+
+    dom0_cpus_default = '0'
 
     components = {}
 
@@ -329,6 +333,12 @@
     def get_vif_antispoof(self):
         return self.get_config_bool('vif-antispoof', 'yes')
 
+    def get_dom0_min_mem(self):
+        return self.get_config_int('dom0-min-mem', self.dom0_min_mem_default)
+
+    def get_dom0_cpus(self):
+        return self.get_config_int('dom0-cpus', self.dom0_cpus_default)
+
 def instance():
     """Get an instance of XendRoot.
     Use this instead of the constructor.
diff -r 501a70f3ae96 tools/python/xen/xend/server/SrvDaemon.py
--- a/tools/python/xen/xend/server/SrvDaemon.py	Thu Jul 28 12:34:45 2005
+++ b/tools/python/xen/xend/server/SrvDaemon.py	Fri Jul 29 11:55:38 2005
@@ -5,7 +5,6 @@
 ###########################################################
 
 import os
-import os.path
 import signal
 import sys
 import threading
@@ -16,6 +15,7 @@
 import StringIO
 import traceback
 import time
+import glob
 
 from xen.lowlevel import xu
 
@@ -25,6 +25,7 @@
 from xen.xend.XendError import XendError
 from xen.xend.server import SrvServer
 from xen.xend.XendLogging import log
+from xen.xend import XendRoot; xroot = XendRoot.instance()
 
 import channel
 import controller
@@ -327,6 +328,7 @@
         return self.cleanup(kill=True)
 
     def run(self):
+        _enforce_dom0_cpus()
         try:
             log.info("Xend Daemon started")
             self.createFactories()
@@ -363,6 +365,32 @@
         #sys.exit(rc)
         os._exit(rc)
 
+def _enforce_dom0_cpus():
+    dn = xroot.get_dom0_cpus()
+
+    for d in glob.glob("/sys/devices/system/cpu/cpu*"):
+        cpu = int(os.path.basename(d)[3:])
+        if (dn == 0) or (cpu < dn):
+            v = "1"
+        else:
+            v = "0"
+        try:
+            f = open("%s/online" %d, "r+")
+            c = f.read(1)
+            if (c != v):
+                if v == "0":
+                    log.info("dom0 is trying to give back cpu %d", cpu)
+                else:
+                    log.info("dom0 is trying to take cpu %d", cpu)
+                f.seek(0)
+                f.write(v)
+                f.close()
+                log.info("dom0 successfully enforced cpu %d", cpu)
+            else:
+                f.close()
+        except:
+            pass
+
 def instance():
     global inst
     try:
diff -r 501a70f3ae96 tools/python/xen/xm/create.py
--- a/tools/python/xen/xm/create.py	Thu Jul 28 12:34:45 2005
+++ b/tools/python/xen/xm/create.py	Fri Jul 29 11:55:38 2005
@@ -1,4 +1,5 @@
 # Copyright (C) 2004 Mike Wray <mike.wray@hp.com>
+# Copyright (C) 2005 Nguyen Anh Quynh <aquynh@gmail.com>
 
 """Domain creation.
 """
@@ -7,10 +8,13 @@
 import sys
 import socket
 
+import xen.lowlevel.xc
+
 from xen.xend import sxp
 from xen.xend import PrettyPrint
 from xen.xend.XendClient import server, XendError
 from xen.xend.XendBootloader import bootloader
+from xen.xend import XendRoot; xroot = XendRoot.instance()
 from xen.util import blkif
 
 from xen.util import console_client
@@ -644,6 +648,36 @@
               % (dom, console_port))
     return (dom, console_port)
 
+def get_dom0_alloc():
+    """Return current allocation memory of dom0 (in MB). Return 0 on error"""
+    PROC_XEN_BALLOON = "/proc/xen/balloon"
+
+    f = open(PROC_XEN_BALLOON, "r")
+    line = f.readline()
+    for x in line.split():
+        for n in x:
+            if not n.isdigit():
+                break
+        else:
+            f.close()
+            return int(x)/1024
+    f.close()
+    return 0
+
+def balloon_out(dom0_min_mem, opts):
+    """Balloon out to get memory for domU, if necessarily"""
+    SLACK = 4
+
+    xc = xen.lowlevel.xc.new()
+    pinfo = xc.physinfo()
+    free_mem = pinfo['free_pages']/256
+    if free_mem < opts.vals.memory + SLACK:
+        need_mem = opts.vals.memory + SLACK - free_mem
+        cur_alloc = get_dom0_alloc()
+        if cur_alloc - need_mem >= dom0_min_mem:
+            server.xend_domain_mem_target_set(0, cur_alloc - need_mem)
+    del xc
+
 def main(argv):
     opts = gopts
     args = opts.parse(argv)
@@ -671,6 +705,10 @@
     if opts.vals.dryrun:
         PrettyPrint.prettyprint(config)
     else:
+        dom0_min_mem = xroot.get_dom0_min_mem()
+        if dom0_min_mem != 0:
+            balloon_out(dom0_min_mem, opts)
+
         (dom, console) = make_domain(opts, config)
         if opts.vals.console_autoconnect:
             path = "/var/lib/xend/console-%s" % console

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] enforce dom0 cpus and balloon out memory

[Ryan Harper]

* aq <aquynh@gmail.com> [2005-07-29 09:01]:
> This patch does 2 jobs:
> 
> - Enforce the number of CPUs dom0 will take. See the new variable
> "dom0-cpus" in xend-config.sxp (you will want to set this variable to
> 1 on SMP systems)

You could use vcpu-hotplug to drop the extra cpus from dom0 instead of
writing directly into sysfs.  The only stopper to that approach is the
fact that dom0's self.channel is not initialized when xend starts and
subsequently doesn't send the control message.  Channels are initialized
when a domain is built, but since dom0 is built by xen, the routine
which sets up channels isn't called for dom0.  Haven't found a good place
for xend to initialized the channel for dom0 yet either.

See vcpu_hotplug() in xen/xend/XendDomInfo.py

Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [PATCH] enforce dom0 cpus and balloon out memory

[Mark Williamson]

> You could use vcpu-hotplug to drop the extra cpus from dom0 instead of
> writing directly into sysfs.  The only stopper to that approach is the
> fact that dom0's self.channel is not initialized when xend starts and
> subsequently doesn't send the control message.  Channels are initialized
> when a domain is built, but since dom0 is built by xen, the routine
> which sets up channels isn't called for dom0.  Haven't found a good place
> for xend to initialized the channel for dom0 yet either.

Hmmm.  Is the channel never set up then?  It'd certainly be nice if it were: 
then you can use "xm balloon" to set dom0's memory footprint, which is 
preferable to using the /proc interface directly.

Cheers,
Mark

>
> See vcpu_hotplug() in xen/xend/XendDomInfo.py
>
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> (512) 838-9253   T/L: 678-9253
> ryanh@us.ibm.com
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH] enforce dom0 cpus and balloon out memory

[Ryan Harper]

* Mark Williamson <mark.williamson@cl.cam.ac.uk> [2005-07-29 10:25]:
> > You could use vcpu-hotplug to drop the extra cpus from dom0 instead of
> > writing directly into sysfs.  The only stopper to that approach is the
> > fact that dom0's self.channel is not initialized when xend starts and
> > subsequently doesn't send the control message.  Channels are initialized
> > when a domain is built, but since dom0 is built by xen, the routine
> > which sets up channels isn't called for dom0.  Haven't found a good place
> > for xend to initialized the channel for dom0 yet either.
> 
> Hmmm.  Is the channel never set up then?  It'd certainly be nice if it were: 

Correct.  As mentioned above, this is only a problem for dom0 since the
xend contruct_image() code isn't called for dom0 as xen is the builder.
The fix is as simple as calling create_channel() from dom0's XendDomInfo
object.  I've not determined the appropriate location to issue the call.

> then you can use "xm balloon" to set dom0's memory footprint, which is 
> preferable to using the /proc interface directly.

Indeed.  Same goes for using vcpu_hotplug() versus sysfs

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

RE: [PATCH] enforce dom0 cpus and balloon out memory

[Ian Pratt]

 > You could use vcpu-hotplug to drop the extra cpus from dom0 
> instead of writing directly into sysfs.  The only stopper to 
> that approach is the fact that dom0's self.channel is not 
> initialized when xend starts and subsequently doesn't send 
> the control message.  Channels are initialized when a domain 
> is built, but since dom0 is built by xen, the routine which 
> sets up channels isn't called for dom0.  Haven't found a good 
> place for xend to initialized the channel for dom0 yet either.
> 
> See vcpu_hotplug() in xen/xend/XendDomInfo.py

Yep, I'd much prefer this approach -- its annoying the tools don't work
on dom0. Volunteers?

Ian

Re: [PATCH] enforce dom0 cpus and balloon out memory

[Keir Fraser]

On 29 Jul 2005, at 18:26, Ian Pratt wrote:

>>
>> See vcpu_hotplug() in xen/xend/XendDomInfo.py
>
> Yep, I'd much prefer this approach -- its annoying the tools don't work
> on dom0. Volunteers?

They certainly *ought* to work. I can assure you that xend does connect 
to domain0's control interface. Otherwise you could not set up virtual 
blkif and netif connections to backend drivers running in domain0!

This should be a small bugfix: no major implementation effort required. 
:-)

  -- Keir

Re: [PATCH] enforce dom0 cpus and balloon out memory

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-07-29 13:20]:
> 
> On 29 Jul 2005, at 18:26, Ian Pratt wrote:
> 
> >>
> >>See vcpu_hotplug() in xen/xend/XendDomInfo.py
> >
> >Yep, I'd much prefer this approach -- its annoying the tools don't work
> >on dom0. Volunteers?
> 
> They certainly *ought* to work. I can assure you that xend does connect 
> to domain0's control interface. Otherwise you could not set up virtual 
> blkif and netif connections to backend drivers running in domain0!
> 
> This should be a small bugfix: no major implementation effort required. 
> :-)

Agreed.  It is a simple matter of find the right place in the xend code
to initialize the self.channel (calling crate_channel()) for dom0.

I reported this a while ago.  _something_ is wrong with self.channel in
XendDomInfo object for dom0 as I noted in this message:

http://lists.xensource.com/archives/html/xen-devel/2005-07/msg00444.html

Also in bugzilla #101, I added a patch to show that the control message
wasn't sent because the channel is not valid.

http://bugzilla.xensource.com/cgi-bin/bugzilla/show_bug.cgi?id=101


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com