[PATCH] 0/2 VCPU creation and allocation

[PATCH] 0/2 VCPU creation and allocation

[Ryan Harper]

I've put together two patches.  The first introduces a new dom0_op,
set_max_vcpus, which with an associated variable and a check in the
VCPUOP handler fixes [1]bug 288.  Also included is a new VCPUOP, 
VCPUOP_create, which handles all of the vcpu creation tasks and leaves
initialization and unpausing to VCPUOP_initialize.  The separation
allows for build-time allocation of vcpus which becomes more important
when trying to allocate memory in a NUMA-aware manner (i.e. knowing
which physical cpus are being used in a domain before we allocate the
memory).

The second patch adds a new domain allocmap cpumap_t which is a bitmap
of physical cpus from which vcpus are to be allocated.  As vcpus are
created, the selection of which physical cpu is balanced across the set
of physical cpus within the map.  The patch lets us control vcpu
allocation at a high level (start all VCPUS on hyperthreads, NODE1,
CPU2).

1. http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=288

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [PATCH] 0/2 VCPU creation and allocation

[Keir Fraser]

On 10 Oct 2005, at 16:16, Ryan Harper wrote:

> Also included is a new VCPUOP,
> VCPUOP_create, which handles all of the vcpu creation tasks and leaves
> initialization and unpausing to VCPUOP_initialize.

I think two new Xen operations is one too many.

Either we should take set_max_vcpus, and have that implicitly do the 
work of VCPUOP_create, or we should take VCPUOP_create (callable only 
by domain0) and have that implicitly increase max_vcpus for the subject 
domain.

  -- Keir

Re: [PATCH] 0/2 VCPU creation and allocation

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-10 10:26]:
> 
> On 10 Oct 2005, at 16:16, Ryan Harper wrote:
> 
> >Also included is a new VCPUOP,
> >VCPUOP_create, which handles all of the vcpu creation tasks and leaves
> >initialization and unpausing to VCPUOP_initialize.
> 
> I think two new Xen operations is one too many.
> 
> Either we should take set_max_vcpus, and have that implicitly do the 
> work of VCPUOP_create, or we should take VCPUOP_create (callable only 
> by domain0) and have that implicitly increase max_vcpus for the subject 
> domain.

OK.  I'll re-work VCPUOP_create to be callable only by dom0 and
auto-increment max_vcpus.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [PATCH] 0/2 VCPU creation and allocation

[Keir Fraser]

On 10 Oct 2005, at 16:28, Ryan Harper wrote:

>> Either we should take set_max_vcpus, and have that implicitly do the
>> work of VCPUOP_create, or we should take VCPUOP_create (callable only
>> by domain0) and have that implicitly increase max_vcpus for the 
>> subject
>> domain.
>
> OK.  I'll re-work VCPUOP_create to be callable only by dom0 and
> auto-increment max_vcpus.

Do you even need a max_vcpus variable? Surely the appropriate check is 
implicit in VCPUOP_initialise detecting whether or not the relevant 
VCPU has been created?

I like small code and simple interfaces. :-)

  -- Keir

Re: [PATCH] 0/2 VCPU creation and allocation

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-10 10:56]:
> 
> On 10 Oct 2005, at 16:28, Ryan Harper wrote:
> 
> >>Either we should take set_max_vcpus, and have that implicitly do the
> >>work of VCPUOP_create, or we should take VCPUOP_create (callable only
> >>by domain0) and have that implicitly increase max_vcpus for the 
> >>subject
> >>domain.
> >
> >OK.  I'll re-work VCPUOP_create to be callable only by dom0 and
> >auto-increment max_vcpus.
> 
> Do you even need a max_vcpus variable? Surely the appropriate check is 
> implicit in VCPUOP_initialise detecting whether or not the relevant 
> VCPU has been created?

I was going to ensure ordered VCPU creation.  Without something like
vcpuid < max_vcpus+1, and increment on successful creation, one can
create vcpus in any order, 1,5,7, 10.  I don't think it *should* matter
but I've not looked elsewhere through the code to see if there are any
other areas assuming all struct vcpu* being valid between 0 and n in
the d->vcpus[] array.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [PATCH] 0/2 VCPU creation and allocation

[Keir Fraser]

On 10 Oct 2005, at 17:05, Ryan Harper wrote:

>> Do you even need a max_vcpus variable? Surely the appropriate check is
>> implicit in VCPUOP_initialise detecting whether or not the relevant
>> VCPU has been created?
>
> I was going to ensure ordered VCPU creation.  Without something like
> vcpuid < max_vcpus+1, and increment on successful creation, one can
> create vcpus in any order, 1,5,7, 10.  I don't think it *should* matter
> but I've not looked elsewhere through the code to see if there are any
> other areas assuming all struct vcpu* being valid between 0 and n in
> the d->vcpus[] array.

Then the vcpu parameter to VCPUOP_create is redundant -- there's only 
one value you will be prepared to accept! If we don't want the 
flexibility of a sparse vcpu map (and I think we don't) then perhaps we 
are better off without VCPUOP_create (which is maybe even a bit neater, 
leaving vcpu_op as a completely unpriv local hypercall) and stick with 
the set_max_vcpus dom0_op? And have that implicitly create the vcpu 
struct for vcpus 0...n-1?

  -- Keir

Re: [PATCH] 0/2 VCPU creation and allocation

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-10 11:13]:
> 
> On 10 Oct 2005, at 17:05, Ryan Harper wrote:
> 
> >>Do you even need a max_vcpus variable? Surely the appropriate check is
> >>implicit in VCPUOP_initialise detecting whether or not the relevant
> >>VCPU has been created?
> >
> >I was going to ensure ordered VCPU creation.  Without something like
> >vcpuid < max_vcpus+1, and increment on successful creation, one can
> >create vcpus in any order, 1,5,7, 10.  I don't think it *should* matter
> >but I've not looked elsewhere through the code to see if there are any
> >other areas assuming all struct vcpu* being valid between 0 and n in
> >the d->vcpus[] array.
> 
> Then the vcpu parameter to VCPUOP_create is redundant -- there's only 
> one value you will be prepared to accept! If we don't want the 
> flexibility of a sparse vcpu map (and I think we don't) then perhaps we 
> are better off without VCPUOP_create (which is maybe even a bit neater, 
> leaving vcpu_op as a completely unpriv local hypercall) and stick with 
> the set_max_vcpus dom0_op? And have that implicitly create the vcpu 
> struct for vcpus 0...n-1?

OK, that makes sense.  I'll turn VCPUOP_create into set_max_vcpus(max)
which will create vcpus 1-(max-1).  Any preference on the hypercall
name?  Does set_max_vcpus() still make sense if it is also creating
vcpus?

How about DOM0_CREATEVCPUS and
    do_createvcpus(struct domain* d, unsigned int max_vcpus).

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [PATCH] 0/2 VCPU creation and allocation

[Keir Fraser]

On 10 Oct 2005, at 17:23, Ryan Harper wrote:

> OK, that makes sense.  I'll turn VCPUOP_create into set_max_vcpus(max)
> which will create vcpus 1-(max-1).  Any preference on the hypercall
> name?  Does set_max_vcpus() still make sense if it is also creating
> vcpus?
>
> How about DOM0_CREATEVCPUS and
>     do_createvcpus(struct domain* d, unsigned int max_vcpus).

I'm not fussed. I guess I have a slight preference for the latter 
(createvcpus).

  -- Keir

[PATCH] 2nd try: 0/2 VCPU creation and allocation

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-10 11:34]:
> 
> On 10 Oct 2005, at 17:23, Ryan Harper wrote:
> 
> >OK, that makes sense.  I'll turn VCPUOP_create into set_max_vcpus(max)
> >which will create vcpus 1-(max-1).  Any preference on the hypercall
> >name?  Does set_max_vcpus() still make sense if it is also creating
> >vcpus?
> >
> >How about DOM0_CREATEVCPUS and
> >    do_createvcpus(struct domain* d, unsigned int max_vcpus).
> 
> I'm not fussed. I guess I have a slight preference for the latter 
> (createvcpus).

The first patch creates a new dom0_op, DOM0_VCPUSINCREASE, which will
create VCPU 1 to (limit-1) in the target domain.  The max_vcpus value is
gone as is the VCPUOP_create.

The second patch is the same as before.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

[PATCH] 2nd try: 1/2 VCPU creation and allocation

[Ryan Harper]

New dom0_op DOM0_VCPUSINCREASE.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com


diffstat output:
 tools/libxc/xc_domain.c                 |    9 +++++++++
 tools/libxc/xenctrl.h                   |   11 +++++++++++
 tools/python/xen/lowlevel/xc/xc.c       |   28 ++++++++++++++++++++++++++++
 tools/python/xen/xend/XendDomainInfo.py |    4 ++++
 xen/arch/x86/domain_build.c             |    6 ++++++
 xen/common/dom0_ops.c                   |   27 +++++++++++++++++++++++++++
 xen/common/domain.c                     |   25 ++++++++++---------------
 xen/include/public/dom0_ops.h           |    8 ++++++++
 xen/include/xen/domain.h                |    2 ++
 9 files changed, 105 insertions(+), 15 deletions(-)

Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
---
diff -r 015f8ae81276 tools/libxc/xc_domain.c
--- a/tools/libxc/xc_domain.c	Mon Oct 10 14:38:01 2005
+++ b/tools/libxc/xc_domain.c	Mon Oct 10 17:26:54 2005
@@ -329,6 +329,15 @@
     return err;
 }
 
+int xc_domain_vcpus_increase(int xc_handle, u32 domid, unsigned int limit)
+{
+    dom0_op_t op;
+    op.cmd = DOM0_VCPUSINCREASE;
+    op.u.vcpusincrease.domain = (domid_t)domid;
+    op.u.vcpusincrease.limit = (u8)limit;
+    return do_dom0_op(xc_handle, &op);
+}
+
 /*
  * Local variables:
  * mode: C
diff -r 015f8ae81276 tools/libxc/xenctrl.h
--- a/tools/libxc/xenctrl.h	Mon Oct 10 14:38:01 2005
+++ b/tools/libxc/xenctrl.h	Mon Oct 10 17:26:54 2005
@@ -152,6 +152,17 @@
                      u32 ssidref,
                      u32 *pdomid);
 
+/*
+ * This function increase the number vcpus in a domain to limit.
+ *
+ * @parm xc_handle a handle to an open hypervisor interface.
+ * @parm domid the domain id in which vcpus are to be created.
+ * @parm limit the total number of vcpus to run in domain.
+ * @return 0 on success, -1 on failure.
+ */
+int xc_domain_vcpus_increase(int xc_handle,
+                             u32 domid, 
+                             unsigned int limit);
 
 int xc_domain_dumpcore(int xc_handle, 
                        u32 domid,
diff -r 015f8ae81276 tools/python/xen/lowlevel/xc/xc.c
--- a/tools/python/xen/lowlevel/xc/xc.c	Mon Oct 10 14:38:01 2005
+++ b/tools/python/xen/lowlevel/xc/xc.c	Mon Oct 10 17:26:54 2005
@@ -91,6 +91,26 @@
         return PyErr_SetFromErrno(xc_error);
 
     return PyInt_FromLong(dom);
+}
+
+static PyObject *pyxc_domain_vcpus_increase(PyObject *self,
+                                            PyObject *args,
+                                            PyObject *kwds)
+{
+    XcObject *xc = (XcObject *)self;
+
+    u32 dom, limit;
+
+    static char *kwd_list[] = { "dom", "limit", NULL };
+
+    if ( !PyArg_ParseTupleAndKeywords(args, kwds, "ii", kwd_list, &dom, &limit) )
+        return NULL;
+
+    if ( xc_domain_vcpus_increase(xc->xc_handle, dom, limit) != 0 )
+        return PyErr_SetFromErrno(xc_error);
+    
+    Py_INCREF(zero);
+    return zero;
 }
 
 static PyObject *pyxc_domain_pause(PyObject *self,
@@ -783,6 +803,14 @@
       " dom    [int, 0]:        Domain identifier to use (allocated if zero).\n"
       "Returns: [int] new domain identifier; -1 on error.\n" },
 
+    { "domain_vcpus_increase", 
+      (PyCFunction)pyxc_domain_vcpus_increase,
+      METH_VARARGS | METH_KEYWORDS, "\n"
+      "Increase the number of VCPUs in domain.\n"
+      " dom       [int, 0]:      Domain identifier to use.\n"
+      " limit     [int, 0]:      New total number of VCPUs in domain.\n"
+      "Returns: [int] 0 on success; -1 on error.\n" },
+
     { "domain_dumpcore", 
       (PyCFunction)pyxc_domain_dumpcore, 
       METH_VARARGS | METH_KEYWORDS, "\n"
diff -r 015f8ae81276 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Mon Oct 10 14:38:01 2005
+++ b/tools/python/xen/xend/XendDomainInfo.py	Mon Oct 10 17:26:54 2005
@@ -1018,6 +1018,10 @@
             self.image.handleBootloading()
 
         xc.domain_setcpuweight(self.domid, self.info['cpu_weight'])
+
+        # increase the total number of vcpus in domain
+        xc.domain_vcpus_increase(self.domid, int(self.info['vcpus']));
+
         # XXX Merge with configure_maxmem?
         m = self.image.getDomainMemory(self.info['memory_KiB'])
         xc.domain_setmaxmem(self.domid, m)
diff -r 015f8ae81276 xen/arch/x86/domain_build.c
--- a/xen/arch/x86/domain_build.c	Mon Oct 10 14:38:01 2005
+++ b/xen/arch/x86/domain_build.c	Mon Oct 10 17:26:54 2005
@@ -14,6 +14,7 @@
 #include <xen/event.h>
 #include <xen/elf.h>
 #include <xen/kernel.h>
+#include <xen/domain.h>
 #include <asm/regs.h>
 #include <asm/system.h>
 #include <asm/io.h>
@@ -559,6 +560,11 @@
         d->shared_info->vcpu_data[i].evtchn_upcall_mask = 1;
     d->shared_info->n_vcpu = num_online_cpus();
 
+    /* create extra vcpus */
+    if ( (d->shared_info->n_vcpu > 1) ) 
+       for ( i = 1; i < d->shared_info->n_vcpu; i++ )
+          do_createvcpu(d, i);
+
     /* Set up monitor table */
     update_pagetables(v);
 
diff -r 015f8ae81276 xen/common/dom0_ops.c
--- a/xen/common/dom0_ops.c	Mon Oct 10 14:38:01 2005
+++ b/xen/common/dom0_ops.c	Mon Oct 10 17:26:54 2005
@@ -227,6 +227,33 @@
     }
     break;
 
+    case DOM0_VCPUSINCREASE:
+    {
+        struct domain *d;
+        unsigned int i, limit = op->u.vcpusincrease.limit;
+
+        if ( limit >= MAX_VIRT_CPUS )
+            ret = -EINVAL;
+
+        d = find_domain_by_id(op->u.vcpusincrease.domain);
+
+        ret = -ESRCH;
+        if ( d != NULL )
+        {
+            ret = -EINVAL;
+            LOCK_BIGLOCK(d);
+            /* NB: VCPU0 is allocated on domain creation */
+            for ( i=1; i<limit; i++ ) {
+                if ( d->vcpu[i] == NULL )
+                    if ( (ret = do_createvcpu(d, i)) != 0 )
+                        break;
+            }
+            UNLOCK_BIGLOCK(d);
+            put_domain(d);
+        }
+    }
+    break;
+
     case DOM0_DESTROYDOMAIN:
     {
         struct domain *d = find_domain_by_id(op->u.destroydomain.domain);
diff -r 015f8ae81276 xen/common/domain.c
--- a/xen/common/domain.c	Mon Oct 10 14:38:01 2005
+++ b/xen/common/domain.c	Mon Oct 10 17:26:54 2005
@@ -368,10 +368,9 @@
     return rc;
 }
 
-int boot_vcpu(struct domain *d, int vcpuid, struct vcpu_guest_context *ctxt) 
-{
-    struct vcpu *v;
-    int rc;
+int do_createvcpu(struct domain *d, int vcpuid)
+{
+    struct vcpu *v;
 
     ASSERT(d->vcpu[vcpuid] == NULL);
 
@@ -387,20 +386,11 @@
 
     arch_do_boot_vcpu(v);
 
-    if ( (rc = arch_set_info_guest(v, ctxt)) != 0 )
-        goto out;
-
     sched_add_domain(v);
 
     set_bit(_VCPUF_down, &v->vcpu_flags);
-    clear_bit(_VCPUF_ctrl_pause, &v->vcpu_flags);
 
     return 0;
-
- out:
-    arch_free_vcpu_struct(d->vcpu[vcpuid]);
-    d->vcpu[vcpuid] = NULL;
-    return rc;
 }
 
 long do_vcpu_op(int cmd, int vcpuid, void *arg)
@@ -413,7 +403,7 @@
     if ( (vcpuid < 0) || (vcpuid >= MAX_VIRT_CPUS) )
         return -EINVAL;
 
-    if ( ((v = d->vcpu[vcpuid]) == NULL) && (cmd != VCPUOP_initialise) )
+    if ( ((v = d->vcpu[vcpuid]) == NULL) )
         return -ENOENT;
 
     switch ( cmd )
@@ -433,7 +423,12 @@
         }
 
         LOCK_BIGLOCK(d);
-        rc = (d->vcpu[vcpuid] == NULL) ? boot_vcpu(d, vcpuid, ctxt) : -EEXIST;
+        clear_bit(_VCPUF_ctrl_pause, &v->vcpu_flags);
+        if ( (rc = arch_set_info_guest(v, ctxt)) != 0 ) {
+            sched_rem_domain(v);
+            arch_free_vcpu_struct(d->vcpu[vcpuid]);
+            d->vcpu[vcpuid] = NULL;
+        }
         UNLOCK_BIGLOCK(d);
 
         xfree(ctxt);
diff -r 015f8ae81276 xen/include/public/dom0_ops.h
--- a/xen/include/public/dom0_ops.h	Mon Oct 10 14:38:01 2005
+++ b/xen/include/public/dom0_ops.h	Mon Oct 10 17:26:54 2005
@@ -386,6 +386,13 @@
         int is_ram;
     } *memory_map;
 } dom0_physical_memory_map_t;
+
+#define DOM0_VCPUSINCREASE 41
+typedef struct {
+    domid_t domain;               /* domain to be affected */
+    u8 limit;                     /* total number of vcpus to run */
+} dom0_vcpusincrease_t;
+
 
 typedef struct {
     u32 cmd;
@@ -422,6 +429,7 @@
         dom0_getdomaininfolist_t getdomaininfolist;
         dom0_platform_quirk_t    platform_quirk;
         dom0_physical_memory_map_t physical_memory_map;
+        dom0_vcpusincrease_t     vcpusincrease;
     } u;
 } dom0_op_t;
 
diff -r 015f8ae81276 xen/include/xen/domain.h
--- a/xen/include/xen/domain.h	Mon Oct 10 14:38:01 2005
+++ b/xen/include/xen/domain.h	Mon Oct 10 17:26:54 2005
@@ -1,6 +1,8 @@
 
 #ifndef __XEN_DOMAIN_H__
 #define __XEN_DOMAIN_H__
+
+extern int do_createvcpu(struct domain *d, int vcpuid);
 
 /*
  * Arch-specifics.

[PATCH] 2nd try: 2/2 VCPU creation and allocation

[Ryan Harper]

Allocation map.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com


diffstat output:
 tools/examples/xmexample1               |    4 +
 tools/examples/xmexample2               |    3 +
 tools/python/xen/lowlevel/xc/xc.c       |    2 
 tools/python/xen/xend/XendDomainInfo.py |   16 ++++-
 tools/python/xen/xm/create.py           |    6 ++
 xen/common/dom0_ops.c                   |   90 +++++++++++++++++---------------
 xen/common/domain.c                     |   28 +++++++++
 xen/common/schedule.c                   |   11 ++-
 xen/include/public/dom0_ops.h           |    2 
 xen/include/xen/domain.h                |    2 
 xen/include/xen/sched.h                 |    2 
 11 files changed, 114 insertions(+), 52 deletions(-)

Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
---
diff -r 12d1e93653c2 tools/examples/xmexample1
--- a/tools/examples/xmexample1	Tue Oct 11 00:13:12 2005
+++ b/tools/examples/xmexample1	Mon Oct 10 19:35:55 2005
@@ -27,6 +27,10 @@
 
 # Number of Virtual CPUS to use, default is 1
 #vcpus = 1
+
+# A bitmap of which physical cpus are vcpus allowed to use.
+# ex1: 0x2 <-- bit 1 set means all vcpus will be created on CPU1
+#allocmap = 0xffffffff  # default value, vcpus can run on any cpu.
 
 #----------------------------------------------------------------------------
 # Define network interfaces.
diff -r 12d1e93653c2 tools/examples/xmexample2
--- a/tools/examples/xmexample2	Tue Oct 11 00:13:12 2005
+++ b/tools/examples/xmexample2	Mon Oct 10 19:35:55 2005
@@ -58,6 +58,9 @@
 # Number of Virtual CPUS to use, default is 1
 #vcpus = 1
 vcpus = 4 # make your domain a 4-way
+
+# A bitmap of which physical cpus are vcpus allowed to use.
+allocmap = 0x2  # start all of your VCPUs on CPU1
 
 #----------------------------------------------------------------------------
 # Define network interfaces.
diff -r 12d1e93653c2 tools/python/xen/lowlevel/xc/xc.c
--- a/tools/python/xen/lowlevel/xc/xc.c	Tue Oct 11 00:13:12 2005
+++ b/tools/python/xen/lowlevel/xc/xc.c	Mon Oct 10 19:35:55 2005
@@ -185,7 +185,7 @@
 
     static char *kwd_list[] = { "dom", "vcpu", "cpumap", NULL };
 
-    if ( !PyArg_ParseTupleAndKeywords(args, kwds, "i|ii", kwd_list, 
+    if ( !PyArg_ParseTupleAndKeywords(args, kwds, "i|iL", kwd_list, 
                                       &dom, &vcpu, &cpumap) )
         return NULL;
 
diff -r 12d1e93653c2 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Tue Oct 11 00:13:12 2005
+++ b/tools/python/xen/xend/XendDomainInfo.py	Mon Oct 10 19:35:55 2005
@@ -266,6 +266,7 @@
     result['maxmem']    = get_cfg('maxmem',    int)
     result['maxmem_kb'] = get_cfg('maxmem_kb', int)
     result['cpu']       = get_cfg('cpu',       int)
+    result['allocmap']  = get_cfg('allocmap',  int)
     result['image']     = get_cfg('image')
 
     try:
@@ -438,6 +439,7 @@
             defaultInfo('cpu_weight',   lambda: 1.0)
             defaultInfo('vcpus',        lambda: 1)
             defaultInfo('vcpu_avail',   lambda: (1 << self.info['vcpus']) - 1)
+            defaultInfo('allocmap'  ,   lambda: None)
             defaultInfo('bootloader',   lambda: None)
             defaultInfo('backend',      lambda: [])
             defaultInfo('device',       lambda: [])
@@ -1018,6 +1020,16 @@
             self.image.handleBootloading()
 
         xc.domain_setcpuweight(self.domid, self.info['cpu_weight'])
+        cpu = self.info['cpu']
+        if cpu is not None and cpu != -1:
+            xc.domain_pincpu(self.domid, 0, 1 << cpu)
+
+        # set the domain alloc map for future vcpus,
+        # repin VCPU0 according to the alloc map
+        allocmap = self.info['allocmap']
+        if self.domid and allocmap:
+            xc.domain_pincpu(self.domid, -1, allocmap)  # domain allocmap
+            xc.domain_pincpu(self.domid, 0, allocmap)   # repin VCPU0
 
         # increase the total number of vcpus in domain
         xc.domain_vcpus_increase(self.domid, int(self.info['vcpus']));
@@ -1026,10 +1038,6 @@
         m = self.image.getDomainMemory(self.info['memory_KiB'])
         xc.domain_setmaxmem(self.domid, m)
         xc.domain_memory_increase_reservation(self.domid, m, 0, 0)
-
-        cpu = self.info['cpu']
-        if cpu is not None and cpu != -1:
-            xc.domain_pincpu(self.domid, 0, 1 << cpu)
 
         self.info['start_time'] = time.time()
 
diff -r 12d1e93653c2 tools/python/xen/xm/create.py
--- a/tools/python/xen/xm/create.py	Tue Oct 11 00:13:12 2005
+++ b/tools/python/xen/xm/create.py	Mon Oct 10 19:35:55 2005
@@ -151,6 +151,10 @@
 gopts.var('cpu', val='CPU',
           fn=set_int, default=None,
           use="CPU to run the domain on.")
+
+gopts.var('allocmap', val='ALLOCMAP',
+          fn=set_int, default=None,
+          use="Set default cpumap used for allocating vcpus.")
 
 gopts.var('vcpus', val='VCPUS',
           fn=set_int, default=1,
@@ -562,6 +566,8 @@
         config.append(['maxmem', vals.maxmem])
     if vals.cpu is not None:
         config.append(['cpu', vals.cpu])
+    if vals.allocmap is not None:
+        config.append(['allocmap', vals.allocmap])
     if vals.cpu_weight is not None:
         config.append(['cpu_weight', vals.cpu_weight])
     if vals.blkif:
diff -r 12d1e93653c2 xen/common/dom0_ops.c
--- a/xen/common/dom0_ops.c	Tue Oct 11 00:13:12 2005
+++ b/xen/common/dom0_ops.c	Mon Oct 10 19:35:55 2005
@@ -275,7 +275,7 @@
     {
         domid_t dom = op->u.pincpudomain.domain;
         struct domain *d = find_domain_by_id(dom);
-        struct vcpu *v;
+        struct vcpu *v = NULL;
         cpumap_t cpumap;
 
 
@@ -285,27 +285,33 @@
             break;
         }
         
-        if ( (op->u.pincpudomain.vcpu >= MAX_VIRT_CPUS) ||
-             !d->vcpu[op->u.pincpudomain.vcpu] )
-        {
-            ret = -EINVAL;
-            put_domain(d);
-            break;
-        }
-
-        v = d->vcpu[op->u.pincpudomain.vcpu];
-        if ( v == NULL )
-        {
-            ret = -ESRCH;
-            put_domain(d);
-            break;
-        }
-
-        if ( v == current )
-        {
-            ret = -EINVAL;
-            put_domain(d);
-            break;
+        /* don't bail on vcpu = -1 as that sets domain cpumap */
+        if ( (op->u.pincpudomain.vcpu != -1) && 
+              ((op->u.pincpudomain.vcpu >= MAX_VIRT_CPUS) ||
+              !d->vcpu[op->u.pincpudomain.vcpu])
+           )
+        {
+            ret = -EINVAL;
+            put_domain(d);
+            break;
+        }
+
+        /* don't get a struct vcpu pointer for -1 op */
+        if (op->u.pincpudomain.vcpu != -1) {
+           v = d->vcpu[op->u.pincpudomain.vcpu];
+           if ( v == NULL )
+           {
+               ret = -ESRCH;
+               put_domain(d);
+               break;
+           }
+
+           if ( v == current )
+           {
+               ret = -EINVAL;
+               put_domain(d);
+               break;
+           }
         }
 
         if ( copy_from_user(&cpumap, op->u.pincpudomain.cpumap,
@@ -316,24 +322,28 @@
             break;
         }
 
-        /* update cpumap for this vcpu */
-        v->cpumap = cpumap;
-
-        if ( cpumap == CPUMAP_RUNANYWHERE )
-        {
-            clear_bit(_VCPUF_cpu_pinned, &v->vcpu_flags);
-        }
-        else
-        {
-            /* pick a new cpu from the usable map */
-            int new_cpu = (int)find_first_set_bit(cpumap) % num_online_cpus();
-
-            vcpu_pause(v);
-            vcpu_migrate_cpu(v, new_cpu);
-            set_bit(_VCPUF_cpu_pinned, &v->vcpu_flags);
-            vcpu_unpause(v);
-        }
-
+        /* update domain vcpu alloc map */
+        if ( v == NULL ) {
+            d->allocmap = cpumap;
+        } else {
+            /* update cpumap for this vcpu */
+            v->cpumap = cpumap;
+
+           if ( cpumap == CPUMAP_RUNANYWHERE )
+           {
+               clear_bit(_VCPUF_cpu_pinned, &v->vcpu_flags);
+           }
+           else
+           {
+               /* pick a new cpu from the usable map */
+               int new_cpu = get_next_processor(d, v, &cpumap);
+
+               vcpu_pause(v);
+               vcpu_migrate_cpu(v, new_cpu);
+               set_bit(_VCPUF_cpu_pinned, &v->vcpu_flags);
+               vcpu_unpause(v);
+           }
+        }
         put_domain(d);
     }
     break;
diff -r 12d1e93653c2 xen/common/domain.c
--- a/xen/common/domain.c	Tue Oct 11 00:13:12 2005
+++ b/xen/common/domain.c	Mon Oct 10 19:35:55 2005
@@ -41,6 +41,7 @@
     atomic_set(&d->refcnt, 1);
     atomic_set(&v->pausecnt, 0);
 
+    d->allocmap  = CPUMAP_RUNANYWHERE;
     d->domain_id = dom_id;
     v->processor = cpu;
 
@@ -380,7 +381,7 @@
     v = d->vcpu[vcpuid];
 
     atomic_set(&v->pausecnt, 0);
-    v->cpumap = CPUMAP_RUNANYWHERE;
+    v->cpumap = d->allocmap;
 
     memcpy(&v->arch, &idle0_vcpu.arch, sizeof(v->arch));
 
@@ -469,6 +470,31 @@
 
     return -ENOSYS;
 }
+
+/* find the least loaded processor , ignorning vcpu v, in cpumap_t *map */
+int get_next_processor(struct domain* d, struct vcpu *v, cpumap_t *map) {
+    struct vcpu  *vc = NULL;
+    int pro, i, cnt[NR_CPUS] = { 0 };
+
+    /* count the processor layout for this dom, except for vcpu v 
+     * whose processor field may not have been set yet. */
+    for_each_vcpu( d, vc ) {
+        if (vc->vcpu_id != v->vcpu_id)
+            cnt[vc->processor]++;
+    }
+
+    /* start from the first allowable cpu, guard against bogus cpus */
+    pro = (int)find_first_set_bit(*map) % num_online_cpus();
+
+    /* pick least loaded processor in the map */
+    for ( i = pro; i < num_online_cpus(); i++ ) {
+        if ( test_bit(i, &*map) && (cnt[i] <= cnt[pro]) )
+            pro = i;
+    }
+
+    return pro;
+}
+
 
 /*
  * Local variables:
diff -r 12d1e93653c2 xen/common/schedule.c
--- a/xen/common/schedule.c	Tue Oct 11 00:13:12 2005
+++ b/xen/common/schedule.c	Mon Oct 10 19:35:55 2005
@@ -125,12 +125,13 @@
         v->next_in_list  = vc->next_in_list;
         vc->next_in_list = v;
 
-        if (test_bit(_VCPUF_cpu_pinned, &vc->vcpu_flags)) {
-            v->processor = (vc->processor + 1) % num_online_cpus();
+
+        /* XXX: if previous vcpu was pinned, mark new vcpu as pinned why? */
+        if (test_bit(_VCPUF_cpu_pinned, &vc->vcpu_flags))
             set_bit(_VCPUF_cpu_pinned, &v->vcpu_flags);
-        } else {
-            v->processor = (vc->processor + 1) % num_online_cpus();
-        }
+
+        v->processor = get_next_processor(d, v, &d->allocmap);
+
     }
 
     return v;
diff -r 12d1e93653c2 xen/include/public/dom0_ops.h
--- a/xen/include/public/dom0_ops.h	Tue Oct 11 00:13:12 2005
+++ b/xen/include/public/dom0_ops.h	Mon Oct 10 19:35:55 2005
@@ -181,7 +181,7 @@
 typedef struct {
     /* IN variables. */
     domid_t      domain;
-    u16          vcpu;
+    s16          vcpu;
     cpumap_t     *cpumap;
 } dom0_pincpudomain_t;
 
diff -r 12d1e93653c2 xen/include/xen/domain.h
--- a/xen/include/xen/domain.h	Tue Oct 11 00:13:12 2005
+++ b/xen/include/xen/domain.h	Mon Oct 10 19:35:55 2005
@@ -27,4 +27,6 @@
 
 extern void dump_pageframe_info(struct domain *d);
 
+int get_next_processor(struct domain *d, struct vcpu *v, cpumap_t *map);
+
 #endif /* __XEN_DOMAIN_H__ */
diff -r 12d1e93653c2 xen/include/xen/sched.h
--- a/xen/include/xen/sched.h	Tue Oct 11 00:13:12 2005
+++ b/xen/include/xen/sched.h	Mon Oct 10 19:35:55 2005
@@ -134,6 +134,8 @@
 
     /* Bitmask of CPUs which are holding onto this domain's state. */
     cpumask_t        cpumask;
+
+    cpumap_t         allocmap;        /* vcpu allocation bitmap */
 
     struct arch_domain arch;

Re: [PATCH] 2nd try: 2/2 VCPU creation and allocation

[Keir Fraser]

On 11 Oct 2005, at 16:16, Ryan Harper wrote:

> Allocation map.

What does this patch buy us, given that we already support per-vcpu 
maps? If I understand correctly, this adds a per-domain map, which 
seems redundant to me.

  -- Keir

Re: [PATCH] 2nd try: 2/2 VCPU creation and allocation

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-12 11:05]:
> 
> On 11 Oct 2005, at 16:16, Ryan Harper wrote:
> 
> >Allocation map.
> 
> What does this patch buy us, given that we already support per-vcpu 
> maps? If I understand correctly, this adds a per-domain map, which 
> seems redundant to me.

I guess not as much as it did before we had build-time vcpu creation.  I
can easily translate the allocation map in the configuration file into a
set of pincpu ops before allocating domain memory and achieve the same
effect as the patch does without the changes to the domain structure and
pincpu dom0_op.  

I'll resend without the changes to xen.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

RE: [PATCH] 0/2 VCPU creation and allocation

[Ian Pratt]

> > Also included is a new VCPUOP,
> > VCPUOP_create, which handles all of the vcpu creation tasks 
> and leaves 
> > initialization and unpausing to VCPUOP_initialize.
> 
> I think two new Xen operations is one too many.
> 
> Either we should take set_max_vcpus, and have that implicitly 
> do the work of VCPUOP_create, or we should take VCPUOP_create 
> (callable only by domain0) and have that implicitly increase 
> max_vcpus for the subject domain.

While we're at it, let's clean up the xm commands too. I tihnk its
pointless letting the user specify the vcpu they want to enable/disable.
Let's change the command names to vcpu-add / vcpu-remove and have them
work on the lowest numbered free vcpu in the case of vcpu-add and the
highest currently active in the case of vcpu-remove.

 
Ian