[PATCH] tunable time slice for credit scheduler

[PATCH] tunable time slice for credit scheduler

[Atsushi SAKAI]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

Hi,

This patch intends to add tunablity for time slice on credit scheduler.
Each domain can change time slice. 

for example, 
xm sched-credit -d 0 -s 20
(unit is msec)

Signed-off-by: Atsushi SAKAI <sakaia@jp.fujitsu.com>

c.f.
1)libxc(pyxc) forgets help message for cap.
I add it on tools/python/xen/lowlevel/xc/xc.c

2)time slice can set from 1 to 100(msec).
I have see VNC connection timeout problem for many domain on one CPU.
For this reason, it can set time slice from 1 to 100 msec.
Of course, default value is 30msec.


Thanks
Atsushi SAKAI


[-- Attachment #2: tune_timeslice.patch --]
[-- Type: application/octet-stream, Size: 12504 bytes --]

diff -r 94cce9a51540 tools/python/xen/lowlevel/xc/xc.c
--- a/tools/python/xen/lowlevel/xc/xc.c	Mon May 14 12:54:26 2007 -0600
+++ b/tools/python/xen/lowlevel/xc/xc.c	Thu May 17 16:45:38 2007 +0900
@@ -833,18 +833,21 @@ static PyObject *pyxc_sched_credit_domai
     uint32_t domid;
     uint16_t weight;
     uint16_t cap;
-    static char *kwd_list[] = { "domid", "weight", "cap", NULL };
-    static char kwd_type[] = "I|HH";
+    uint16_t slice;
+    static char *kwd_list[] = { "domid", "weight", "cap", "slice", NULL };
+    static char kwd_type[] = "I|HHH";
     struct xen_domctl_sched_credit sdom;
     
     weight = 0;
     cap = (uint16_t)~0U;
+    slice = 0;
     if( !PyArg_ParseTupleAndKeywords(args, kwds, kwd_type, kwd_list, 
-                                     &domid, &weight, &cap) )
+                                     &domid, &weight, &cap, &slice) )
         return NULL;
 
     sdom.weight = weight;
     sdom.cap = cap;
+    sdom.slice = slice;
 
     if ( xc_sched_credit_domain_set(self->xc_handle, domid, &sdom) != 0 )
         return pyxc_error_to_exception();
@@ -864,9 +867,10 @@ static PyObject *pyxc_sched_credit_domai
     if ( xc_sched_credit_domain_get(self->xc_handle, domid, &sdom) != 0 )
         return pyxc_error_to_exception();
 
-    return Py_BuildValue("{s:H,s:H}",
+    return Py_BuildValue("{s:H,s:H,s:H}",
                          "weight",  sdom.weight,
-                         "cap",     sdom.cap);
+                         "cap",     sdom.cap,
+                         "slice",   sdom.slice);
 }
 
 static PyObject *pyxc_domain_setmaxmem(XcObject *self, PyObject *args)
@@ -1290,6 +1294,8 @@ static PyMethodDef pyxc_methods[] = {
       "SMP credit scheduler.\n"
       " domid     [int]:   domain id to set\n"
       " weight    [short]: domain's scheduling weight\n"
+      " cap       [short]: domain's scheduling cap\n"
+      " slice     [short]: domain's scheduling slice\n"
       "Returns: [int] 0 on success; -1 on error.\n" },
 
     { "sched_credit_domain_get",
@@ -1299,7 +1305,9 @@ static PyMethodDef pyxc_methods[] = {
       "SMP credit scheduler.\n"
       " domid     [int]:   domain id to get\n"
       "Returns:   [dict]\n"
-      " weight    [short]: domain's scheduling weight\n"},
+      " weight    [short]: domain's scheduling weight\n"
+      " cap       [short]: domain's scheduling cap\n"
+      " slice     [short]: domain's scheduling slice\n" },
 
     { "evtchn_alloc_unbound", 
       (PyCFunction)pyxc_evtchn_alloc_unbound,
diff -r 94cce9a51540 tools/python/xen/xend/XendAPI.py
--- a/tools/python/xen/xend/XendAPI.py	Mon May 14 12:54:26 2007 -0600
+++ b/tools/python/xen/xend/XendAPI.py	Tue May 15 19:52:10 2007 +0900
@@ -1455,10 +1455,12 @@ class XendAPI(object):
 
         #need to update sched params aswell
         if 'weight' in xeninfo.info['vcpus_params'] \
-           and 'cap' in xeninfo.info['vcpus_params']:
+           and 'cap' in xeninfo.info['vcpus_params'] \
+           and 'slice' in xeninfo.info['vcpus_params']:
             weight = xeninfo.info['vcpus_params']['weight']
             cap = xeninfo.info['vcpus_params']['cap']
-            xendom.domain_sched_credit_set(xeninfo.getDomid(), weight, cap)
+            slice = xeninfo.info['vcpus_params']['slice']
+            xendom.domain_sched_credit_set(xeninfo.getDomid(), weight, cap, slice)
 
     def VM_set_VCPUs_number_live(self, _, vm_ref, num):
         dom = XendDomain.instance().get_vm_by_uuid(vm_ref)
diff -r 94cce9a51540 tools/python/xen/xend/XendDomain.py
--- a/tools/python/xen/xend/XendDomain.py	Mon May 14 12:54:26 2007 -0600
+++ b/tools/python/xen/xend/XendDomain.py	Wed May 16 14:48:20 2007 +0900
@@ -1393,13 +1393,14 @@ class XendDomain:
         except Exception, ex:
             raise XendError(str(ex))
     
-    def domain_sched_credit_set(self, domid, weight = None, cap = None):
+    def domain_sched_credit_set(self, domid, weight = None, cap = None, slice = None):
         """Set credit scheduler parameters for a domain.
 
         @param domid: Domain ID or Name
         @type domid: int or string.
         @type weight: int
         @type cap: int
+        @type slice: int
         @rtype: 0
         """
         dominfo = self.domain_lookup_nr(domid)
@@ -1416,10 +1417,16 @@ class XendDomain:
             elif cap < 0 or cap > dominfo.getVCpuCount() * 100:
                 raise XendError("cap is out of range")
 
+            if slice is None:
+                slice = int(0)
+            elif slice < 1 or slice > 101:
+                raise XendError("slice is out of range 1-100")
+
             assert type(weight) == int
             assert type(cap) == int
-
-            return xc.sched_credit_domain_set(dominfo.getDomid(), weight, cap)
+            assert type(slice) == int
+
+            return xc.sched_credit_domain_set(dominfo.getDomid(), weight, cap, slice)
         except Exception, ex:
             log.exception(ex)
             raise XendError(str(ex))
diff -r 94cce9a51540 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Mon May 14 12:54:26 2007 -0600
+++ b/tools/python/xen/xend/XendDomainInfo.py	Wed May 16 11:22:01 2007 +0900
@@ -411,7 +411,8 @@ class XendDomainInfo:
                 if xennode.xenschedinfo() == 'credit':
                     xendomains.domain_sched_credit_set(self.getDomid(),
                                                        self.getWeight(),
-                                                       self.getCap())
+                                                       self.getCap(),
+                                                       self.getSlice())
             except:
                 log.exception('VM start failed')
                 self.destroy()
@@ -1009,6 +1010,9 @@ class XendDomainInfo:
 
     def getWeight(self):
         return self.info.get('cpu_weight', 256)
+
+    def getSlice(self):
+        return self.info.get('cpu_slice', 30)
 
     def setResume(self, state):
         self._resume = state
diff -r 94cce9a51540 tools/python/xen/xend/server/SrvDomain.py
--- a/tools/python/xen/xend/server/SrvDomain.py	Mon May 14 12:54:26 2007 -0600
+++ b/tools/python/xen/xend/server/SrvDomain.py	Wed May 16 15:39:23 2007 +0900
@@ -155,7 +155,8 @@ class SrvDomain(SrvDir):
     def op_domain_sched_credit_set(self, _, req):
         fn = FormFn(self.xd.domain_sched_credit_set,
                     [['dom', 'int'],
-                     ['weight', 'int']])
+                     ['weight', 'int']
+                     ['slice', 'int']])
         val = fn(req.args, {'dom': self.dom.domid})
         return val
 
diff -r 94cce9a51540 tools/python/xen/xm/main.py
--- a/tools/python/xen/xm/main.py	Mon May 14 12:54:26 2007 -0600
+++ b/tools/python/xen/xm/main.py	Wed May 16 16:16:48 2007 +0900
@@ -132,7 +132,7 @@ SUBCOMMAND_HELP = {
     'log'         : ('', 'Print Xend log'),
     'rename'      : ('<Domain> <NewDomainName>', 'Rename a domain.'),
     'sched-sedf'  : ('<Domain> [options]', 'Get/set EDF parameters.'),
-    'sched-credit': ('[-d <Domain> [-w[=WEIGHT]|-c[=CAP]]]',
+    'sched-credit': ('[-d <Domain> [-w[=WEIGHT]|-c[=CAP]|-s[=SLICE]]]',
                      'Get/set credit scheduler parameters.'),
     'sysrq'       : ('<Domain> <letter>', 'Send a sysrq to a domain.'),
     'debug-keys'  : ('<Keys>', 'Send debug keys to Xen.'),
@@ -207,6 +207,7 @@ SUBCOMMAND_OPTIONS = {
        ('-d DOMAIN', '--domain=DOMAIN', 'Domain to modify'),
        ('-w WEIGHT', '--weight=WEIGHT', 'Weight (int)'),
        ('-c CAP',    '--cap=CAP',       'Cap (int)'),
+       ('-s SLICE',  '--slice=SLICE',   'Slice (int)'),
     ),
     'list': (
        ('-l', '--long',         'Output all VM details in SXP'),
@@ -1467,8 +1468,8 @@ def xm_sched_credit(args):
     check_sched_type('credit')
 
     try:
-        opts, params = getopt.getopt(args, "d:w:c:",
-            ["domain=", "weight=", "cap="])
+        opts, params = getopt.getopt(args, "d:w:c:s:",
+            ["domain=", "weight=", "cap=", "slice="])
     except getopt.GetoptError, opterr:
         err(opterr)
         usage('sched-credit')
@@ -1476,6 +1477,7 @@ def xm_sched_credit(args):
     domid = None
     weight = None
     cap = None
+    slice = None
 
     for o, a in opts:
         if o == "-d":
@@ -1483,18 +1485,20 @@ def xm_sched_credit(args):
         elif o == "-w":
             weight = int(a)
         elif o == "-c":
-            cap = int(a);
+            cap = int(a)
+        elif o == "-s":
+            slice = int(a);
 
     doms = filter(lambda x : domid_match(domid, x),
                   [parse_doms_info(dom)
                   for dom in getDomains(None, 'running')])
 
-    if weight is None and cap is None:
+    if weight is None and cap is None and slice is None:
         if domid is not None and doms == []: 
             err("Domain '%s' does not exist." % domid)
             usage('sched-credit')
         # print header if we aren't setting any parameters
-        print '%-33s %-2s %-6s %-4s' % ('Name','ID','Weight','Cap')
+        print '%-33s %-2s %-6s %-4s %-6s' % ('Name','ID','Weight','Cap','Slice')
         
         for d in doms:
             try:
@@ -1507,16 +1511,17 @@ def xm_sched_credit(args):
             except xmlrpclib.Fault:
                 pass
 
-            if 'weight' not in info or 'cap' not in info:
+            if 'weight' not in info or 'cap' not in info or 'slice' not in info:
                 # domain does not support sched-credit?
-                info = {'weight': -1, 'cap': -1}
+                info = {'weight': -1, 'cap': -1, 'slice':-1}
 
             info['weight'] = int(info['weight'])
             info['cap']    = int(info['cap'])
+            info['slice']  = int(info['slice'])
             
             info['name']  = d['name']
             info['domid'] = int(d['domid'])
-            print( ("%(name)-32s %(domid)3d %(weight)6d %(cap)4d") % info)
+            print( ("%(name)-32s %(domid)3d %(weight)6d %(cap)4d %(slice)5d") % info)
     else:
         if domid is None:
             # place holder for system-wide scheduler parameters
@@ -1532,8 +1537,12 @@ def xm_sched_credit(args):
                 get_single_vm(domid),
                 "cap",
                 cap)            
+            server.xenapi.VM.add_to_VCPUs_params_live(
+                get_single_vm(domid),
+                "slice",
+                slice)            
         else:
-            result = server.xend.domain.sched_credit_set(domid, weight, cap)
+            result = server.xend.domain.sched_credit_set(domid, weight, cap, slice)
             if result != 0:
                 err(str(result))
 
diff -r 94cce9a51540 xen/common/sched_credit.c
--- a/xen/common/sched_credit.c	Mon May 14 12:54:26 2007 -0600
+++ b/xen/common/sched_credit.c	Wed May 16 17:04:47 2007 +0900
@@ -223,6 +223,7 @@ struct csched_dom {
     uint16_t active_vcpu_count;
     uint16_t weight;
     uint16_t cap;
+    uint16_t slice;
 };
 
 /*
@@ -709,6 +710,7 @@ csched_dom_cntl(
     {
         op->u.credit.weight = sdom->weight;
         op->u.credit.cap = sdom->cap;
+        op->u.credit.slice = sdom->slice;
     }
     else
     {
@@ -728,6 +730,9 @@ csched_dom_cntl(
 
         if ( op->u.credit.cap != (uint16_t)~0U )
             sdom->cap = op->u.credit.cap;
+
+        if ( op->u.credit.slice != 0 )
+            sdom->slice = op->u.credit.slice;
 
         spin_unlock_irqrestore(&csched_priv.lock, flags);
     }
@@ -756,6 +761,7 @@ csched_dom_init(struct domain *dom)
     sdom->dom = dom;
     sdom->weight = CSCHED_DEFAULT_WEIGHT;
     sdom->cap = 0U;
+    sdom->slice = CSCHED_MSECS_PER_TSLICE;
     dom->sched_priv = sdom;
 
     return 0;
@@ -1210,7 +1216,12 @@ csched_schedule(s_time_t now)
     /*
      * Return task to run next...
      */
-    ret.time = MILLISECS(CSCHED_MSECS_PER_TSLICE);
+    if(snext->sdom != NULL){
+        ret.time = MILLISECS(snext->sdom->slice);
+    }else{
+        /* for idle domain */
+        ret.time = MILLISECS(CSCHED_MSECS_PER_TSLICE);
+    }
     ret.task = snext->vcpu;
 
     CSCHED_VCPU_CHECK(ret.task);
diff -r 94cce9a51540 xen/include/public/domctl.h
--- a/xen/include/public/domctl.h	Mon May 14 12:54:26 2007 -0600
+++ b/xen/include/public/domctl.h	Tue May 15 11:08:52 2007 +0900
@@ -305,6 +305,7 @@ struct xen_domctl_scheduler_op {
         struct xen_domctl_sched_credit {
             uint16_t weight;
             uint16_t cap;
+            uint16_t slice;
         } credit;
     } u;
 };

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

question about the differences for xen in x86_32 and x86_64

[tgh]

¾
hi
I am confused about the differences betwoon xen in x86_32 and x86_64
in x86_32,Ring 1 for guest OS, ring3 for user-space, Xen lives in top 
64MB of linear addr space, Segmentation used to protect Xen as switching 
page tables too slow on standard x86, Hypercalls jump to Xen in ring 0, 
Guest OS with ‘fast trap’ handler could direct user-space to guest OS 
system calls, is it right?
while in x86_64 ,both guestos and userapplications seem to run in ring3, 
and what about other thing memtioned above for x86_64? such as the 
memory mapping for xen and guestos kernel and applications , system call 
fast trap handling


I am confused about it
could you help me
Thanks in advance

Re: [PATCH] tunable time slice for credit scheduler

[Emmanuel Ackaouy]

The sched_credit.c part of this patch looks good to me (except for
the coding style issues). Before adding per-domain tunable time
slices though, shouldn't we make the global time slice tunable?


On May 17, 2007, at 10:25, Atsushi SAKAI wrote:

> Hi,
>
> This patch intends to add tunablity for time slice on credit scheduler.
> Each domain can change time slice.
>
> for example,
> xm sched-credit -d 0 -s 20
> (unit is msec)
>
> Signed-off-by: Atsushi SAKAI <sakaia@jp.fujitsu.com>
>
> c.f.
> 1)libxc(pyxc) forgets help message for cap.
> I add it on tools/python/xen/lowlevel/xc/xc.c
>
> 2)time slice can set from 1 to 100(msec).
> I have see VNC connection timeout problem for many domain on one CPU.
> For this reason, it can set time slice from 1 to 100 msec.
> Of course, default value is 30msec.
>
>
> Thanks
> Atsushi SAKAI
>
> <tune_timeslice.patch>_______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH] tunable time slice for credit scheduler

[Keir Fraser]

I'm not sure that extra parameters are what the scheduler really needs.
Tuning for better performance 'out of the box' would be more useful. Under
what circumstances were you seeing VNC connection timeouts? If it's an
extreme overload situation then that can likely be detected and dealt with
by automatic time-slice adjustment. I can't see a time-slice option being
used sensibly in many scenarios.

 -- Keir

On 27/6/07 13:08, "Emmanuel Ackaouy" <ackaouy@gmail.com> wrote:

> The sched_credit.c part of this patch looks good to me (except for
> the coding style issues). Before adding per-domain tunable time
> slices though, shouldn't we make the global time slice tunable?
> 
> 
> On May 17, 2007, at 10:25, Atsushi SAKAI wrote:
> 
>> Hi,
>> 
>> This patch intends to add tunablity for time slice on credit scheduler.
>> Each domain can change time slice.
>> 
>> for example,
>> xm sched-credit -d 0 -s 20
>> (unit is msec)
>> 
>> Signed-off-by: Atsushi SAKAI <sakaia@jp.fujitsu.com>
>> 
>> c.f.
>> 1)libxc(pyxc) forgets help message for cap.
>> I add it on tools/python/xen/lowlevel/xc/xc.c
>> 
>> 2)time slice can set from 1 to 100(msec).
>> I have see VNC connection timeout problem for many domain on one CPU.
>> For this reason, it can set time slice from 1 to 100 msec.
>> Of course, default value is 30msec.
>> 
>> 
>> Thanks
>> Atsushi SAKAI
>> 
>> <tune_timeslice.patch>_______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH] tunable time slice for credit scheduler

[Atsushi SAKAI]

Hi, Keir and Emmanuel

 Thanks for reviewing.
I am very happy to get a comment on this.

Primary motivation of this patch is 
Network benchmark(Web server throughput) at Xen.
(Which is presented on XenSummit 2007 Spring)

In this comparison, 
network through put is not good if you use credit scheduler
http://www.xensource.com/files/xensummit_4/3schedulers-xen-summit_Cherkosova.pdf

In this presentation, SEDF and BVT selects the best parameter before 
comparison.(p.9)
On the other hand, Credit Scheduler does not have such kind of 
tunable paramter.
I think the comparison is not fare.
This patch adds this kind of parameter for fare comparison.

And I think time slice should be differenciated by 
Dom0, DriverDomain and Others.
So I make this patch defining each domain timeslice. 
But as a first step, global time slice is a good idea.


Anyway, this is not related to VNC issue.

Thanks
Atsushi SAKAI


Keir Fraser <keir@xensource.com> wrote:

> I'm not sure that extra parameters are what the scheduler really needs.
> Tuning for better performance 'out of the box' would be more useful. Under
> what circumstances were you seeing VNC connection timeouts? If it's an
> extreme overload situation then that can likely be detected and dealt with
> by automatic time-slice adjustment. I can't see a time-slice option being
> used sensibly in many scenarios.
> 
>  -- Keir
> 
> On 27/6/07 13:08, "Emmanuel Ackaouy" <ackaouy@gmail.com> wrote:
> 
> > The sched_credit.c part of this patch looks good to me (except for
> > the coding style issues). Before adding per-domain tunable time
> > slices though, shouldn't we make the global time slice tunable?
> > 
> > 
> > On May 17, 2007, at 10:25, Atsushi SAKAI wrote:
> > 
> >> Hi,
> >> 
> >> This patch intends to add tunablity for time slice on credit scheduler.
> >> Each domain can change time slice.
> >> 
> >> for example,
> >> xm sched-credit -d 0 -s 20
> >> (unit is msec)
> >> 
> >> Signed-off-by: Atsushi SAKAI <sakaia@jp.fujitsu.com>
> >> 
> >> c.f.
> >> 1)libxc(pyxc) forgets help message for cap.
> >> I add it on tools/python/xen/lowlevel/xc/xc.c
> >> 
> >> 2)time slice can set from 1 to 100(msec).
> >> I have see VNC connection timeout problem for many domain on one CPU.
> >> For this reason, it can set time slice from 1 to 100 msec.
> >> Of course, default value is 30msec.
> >> 
> >> 
> >> Thanks
> >> Atsushi SAKAI
> >> 
> >> <tune_timeslice.patch>_______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: question about the differences for xen in x86_32 and x86_64

[Mark Williamson]

> I am confused about the differences betwoon xen in x86_32 and x86_64
> in x86_32,Ring 1 for guest OS, ring3 for user-space, Xen lives in top
> 64MB of linear addr space, Segmentation used to protect Xen as switching
> page tables too slow on standard x86, Hypercalls jump to Xen in ring 0,
> Guest OS with ‘fast trap’ handler could direct user-space to guest OS
> system calls, is it right?

Yes, that's right.

> while in x86_64 ,both guestos and userapplications seem to run in ring3,
> and what about other thing memtioned above for x86_64? such as the
> memory mapping for xen and guestos kernel and applications , system call
> fast trap handling

Xen still lives at the top of the address space; I'm not sure how much it 
reserves for itself.  The guest kernel (when mapped) lives below Xen.  Guest 
userspace lives below that.

When running in guest usermode only the guest userspace and Xen are mapped 
into memory.  Xen is protected from guest userspace using the supervisor 
protection bit in the pagetables.

AFAIK, all system calls must bounce through Xen.  I don't think it's possible 
for OSes to take syscalls directly anymore.  This is because Xen has to map 
the guest kernel's pages into memory (by hooking an extra subtree into the 
current pagetable).  Xen then kicks the guest kernel into action; the kernel 
can directly access userspace pages since they are still mapped.  Xen is 
protected from the guest kernel by the supervisor protection bit in the 
pagetables.

When the kernel finishes its work it has to execute a hypercall to instruct 
Xen to transfer control back to userspace.  Xen has to flush the TLB at this 
point to remove the kernel mappings; otherwise it could be possible for the 
guest's userspace to interfere with the kernel's memory.

I believe the "global" bit is set on guest userspace pagetable entries, which 
means that the userspace mappings are not flushed at this point (performance 
optimisation).  Xen can still flush these mappings with a "global flush" if a 
switch of application running or of domain occurs.

> I am confused about it
> could you help me
> Thanks in advance

Hope that helps.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!