RE: [PATCH 2/3] xend: Add multiple cpumasks support

RE: [PATCH 2/3] xend: Add multiple cpumasks support

[Ian Pratt]

> This patch modifies xend to accept and parse multiple cpumask strings
> from the cpus parameter.  The cpus string stays the same, but it now
> can parse either a python list of strings:

According to the example config file, we currently take a string
containing the list of CPUs that the *domain* can run on. We should be
putting this CPU mask into all of the VCPUs for the domain (rather than
one bit in each as we currently do).

Adding support to enable separate masks for each VCPU isn't a bad idea,
but we certainly don't want to break the behaviour of being able to set
the mask for a domain.

Allowing a list of masks to be passed in (as well as a single string)
isn't a bad idea.

Surely the comma separated list of vcpus isn't compatible with the
current parsing? Or are you using the quotes to try and separate the
different vcpu masks? That's horrid.

BTW: As syntactic sugar, if the string starts with an exclusion, should
it implicitly mean that mask starts off containing all 1's rather than
all 0's? [avoiding the need to do "0-255,^0"]

Ian   
 
> [ '2-5, '2-5', '2-5' ]
> 
> A regular string with ", " as the separator:
> 
> "2-5, 2-5, 2-5, 2-5"
> 
> or a mixture of both:
> 
> [ 2-5, '2-5', 2-5, '2-5' ]
> 
> all result in the same list of integers which is used to create the
> cpumask for each vcpu in the domain.
> 
> Without this patch the cpus parameter only enables one bit of the
> cpumask preventing the credit scheduler from balancing.
> 
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> (512) 838-9253   T/L: 678-9253
> ryanh@us.ibm.com
> 
> 
> diffstat output:
>  examples/xmexample.hvm            |    3 +
>  examples/xmexample.vti            |    1
>  examples/xmexample1               |    3 +
>  examples/xmexample2               |    3 +
>  examples/xmexample3               |    3 +
>  python/xen/xend/XendDomainInfo.py |   66
+++++++++++++++++++++++++--------
> -----
>  6 files changed, 53 insertions(+), 26 deletions(-)
> 
> Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
> ---
> diff -r e6de470f9287 tools/python/xen/xend/XendDomainInfo.py
> --- a/tools/python/xen/xend/XendDomainInfo.py	Sat Aug 12 08:22:45 2006
-
> 0500
> +++ b/tools/python/xen/xend/XendDomainInfo.py	Sat Aug 12 08:22:47 2006
-
> 0500
> @@ -328,27 +328,49 @@ def parseConfig(config):
>             else:
>                 result['cpus'] = str(result['cpu'])
> 
> -        # convert 'cpus' string to list of ints
> -        # 'cpus' supports a list of ranges (0-3), seperated by
> -        # commas, and negation, (^1).
> -        # Precedence is settled by  order of the string:
> -        #     "0-3,^1"   -> [0,2,3]
> -        #     "0-3,^1,1" -> [0,1,2,3]
> -        if result['cpus']:
> -            cpus = []
> -            for c in result['cpus'].split(','):
> -                if c.find('-') != -1:
> -                    (x,y) = c.split('-')
> -                    for i in range(int(x),int(y)+1):
> -                        cpus.append(int(i))
> -                else:
> -                    # remove this element from the list
> -                    if c[0] == '^':
> -                        cpus = [x for x in cpus if x != int(c[1:])]
> +        if result['cpus'] is not None:
> +            # see if cpus string specifies multiple cpumasks, or just
one
> +            # e.g: "[ '2-5', '0-1', '0-3,^1', '5' ]" vs. "0-3,5"
> +            if result['cpus'].startswith("["):
> +                # the below was tested with the following sample
string
> +                # "[4, '2-5','2-5', '1-3,6,^2', '2-6','1' 1-7,^2 ]"
and
> resulted in
> +                # ['4', '2-5', '2-5', '1-3,6,^2', '2-6', '1']
> +                result['cpus'] = filter(lambda x: len(x), map(lambda
x:
> x.strip(", "),
> +
> result['cpus'].replace('[',"").replace(']',"").split("'")))
> +
> +            # if user didn't use list bracket, convert to to list of
> strings
> +            # cpus = "4, 2-5,^4, 1-3,5,^2, 1, ^2" ->
> +            # ['4', '2-5,^4', '1-3,5,^2', '1', '^2']
> +            # also takes care of cpus = "4"
> +            else:
> +                result['cpus'] = map(lambda x: x.strip(", "),
> result['cpus'].split())
> +
> +            # convert 'cpus' list of strings into a list of list of
ints
> +            # 'cpus' supports a list of ranges (0-3), seperated by
> +            # commas, and negation, (^1).
> +            # Precedence is settled by  order of the string:
> +            #     "0-3,^1"   -> [0,2,3]
> +            #     "0-3,^1,1" -> [0,1,2,3]
> +
> +            new_cpus = []
> +            for x in result['cpus']:
> +                cpus = []
> +                for c in x.split(','):
> +                    if c.find('-') != -1:
> +                        (x,y) = c.split('-')
> +                        for i in range(int(x),int(y)+1):
> +                            cpus.append(int(i))
>                      else:
> -                        cpus.append(int(c))
> -
> -            result['cpus'] = cpus
> +                        # remove this element from the list
> +                        if c[0] == '^':
> +                            cpus = [x for x in cpus if x !=
int(c[1:])]
> +                        else:
> +                            cpus.append(int(c))
> +
> +                new_cpus.append(cpus)
> +
> +
> +            result['cpus'] = new_cpus
> 
>      except ValueError, exn:
>          raise VmError(
> @@ -1275,8 +1297,8 @@ class XendDomainInfo:
>              cpus = self.info['cpus']
>              if cpus is not None and len(cpus) > 0:
>                  for v in range(0, self.info['max_vcpu_id']+1):
> -                    # pincpu takes a list of ints
> -                    cpu = [ int( cpus[v % len(cpus)] ) ]
> +                    # pincpu takes a list of ints,
> +                    cpu = map(lambda x: int(x), cpus[v % len(cpus)])
>                      xc.vcpu_setaffinity(self.domid, v, cpu)
> 
>              # set domain maxmem in KiB
> diff -r e6de470f9287 tools/examples/xmexample.hvm
> --- a/tools/examples/xmexample.hvm	Sat Aug 12 08:22:45 2006 -0500
> +++ b/tools/examples/xmexample.hvm	Sat Aug 12 08:34:13 2006 -0500
> @@ -47,10 +47,11 @@ name = "ExampleHVMDomain"
>  # enable/disable HVM guest APIC, default=0 (disabled)
>  #apic=0
> 
> -# List of which CPUS this domain is allowed to use, default Xen picks
> +# List of which CPUS vcpus are allowed to use, default Xen picks
>  #cpus = ""         # leave to Xen to pick
>  #cpus = "0"        # all vcpus run on CPU0
>  #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
> +#cpus = "0-1, 2-3" # run VCPU0 on CPU0-1, VCPU1 on CPU2-3
> 
>  # Optionally define mac and/or bridge for the network interfaces.
>  # Random MACs are assigned if not given.
> diff -r e6de470f9287 tools/examples/xmexample.vti
> --- a/tools/examples/xmexample.vti	Sat Aug 12 08:22:45 2006 -0500
> +++ b/tools/examples/xmexample.vti	Sat Aug 12 08:39:42 2006 -0500
> @@ -34,6 +34,7 @@ name = "ExampleVTIDomain"
>  #cpus = ""         # leave to Xen to pick
>  #cpus = "0"        # all vcpus run on CPU0
>  #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
> +#cpus = "0-1, 2-3" # run VCPU0 on CPU0-1, VCPU1 on CPU2-3
> 
>  # Optionally define mac and/or bridge for the network interfaces.
>  # Random MACs are assigned if not given.
> diff -r e6de470f9287 tools/examples/xmexample1
> --- a/tools/examples/xmexample1	Sat Aug 12 08:22:45 2006 -0500
> +++ b/tools/examples/xmexample1	Sat Aug 12 08:27:02 2006 -0500
> @@ -30,10 +30,11 @@ name = "ExampleDomain"
>  # on each call to 'xm create'.
>  #uuid = "06ed00fe-1162-4fc4-b5d8-11993ee4a8b9"
> 
> -# List of which CPUS this domain is allowed to use, default Xen picks
> +# List of which CPUS vcpus are allowed to use, default Xen picks
>  #cpus = ""         # leave to Xen to pick
>  #cpus = "0"        # all vcpus run on CPU0
>  #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
> +#cpus = "0-1, 2-3" # run VCPU0 on CPU0-1, VCPU1 on CPU2-3
> 
>  # Number of Virtual CPUS to use, default is 1
>  #vcpus = 1
> diff -r e6de470f9287 tools/examples/xmexample2
> --- a/tools/examples/xmexample2	Sat Aug 12 08:22:45 2006 -0500
> +++ b/tools/examples/xmexample2	Sat Aug 12 08:27:47 2006 -0500
> @@ -59,10 +59,11 @@ name = "VM%d" % vmid
>  # on each call to 'xm create'.
>  #uuid = "06ed00fe-1162-4fc4-b5d8-11993ee4a8b9"
> 
> -# List of which CPUS this domain is allowed to use, default Xen picks
> +# List of which CPUS vcpus are allowed to use, default Xen picks
>  #cpus = ""         # leave to Xen to pick
>  #cpus = "0"        # all vcpus run on CPU0
>  #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
> +#cpus = "0-1, 2-3" # run VCPU0 on CPU0-1, VCPU1 on CPU2-3
>  #cpus = "%s" % vmid # set based on vmid (mod number of CPUs)
> 
>  # Number of Virtual CPUS to use, default is 1
> diff -r e6de470f9287 tools/examples/xmexample3
> --- a/tools/examples/xmexample3	Sat Aug 12 08:22:45 2006 -0500
> +++ b/tools/examples/xmexample3	Sat Aug 12 08:33:35 2006 -0500
> @@ -59,10 +59,11 @@ name = "VM%d" % vmid
>  # on each call to 'xm create'.
>  #uuid = "06ed00fe-1162-4fc4-b5d8-11993ee4a8b9"
> 
> -# List of which CPUS this domain is allowed to use, default Xen picks
> +# List of which CPUS vcpus are allowed to use, default Xen picks
>  #cpus = ""         # leave to Xen to pick
>  #cpus = "0"        # all vcpus run on CPU0
>  #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
> +#cpus = "0-1, 2-3" # run VCPU0 on CPU0-1, VCPU1 on CPU2-3
>  cpus = "%s" % vmid # set based on vmid (mod number of CPUs)
> 
>
#-----------------------------------------------------------------------
--
> ---
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Harper]

* Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2006-08-14 13:12]:
> > This patch modifies xend to accept and parse multiple cpumask strings
> > from the cpus parameter.  The cpus string stays the same, but it now
> > can parse either a python list of strings:
> 
> According to the example config file, we currently take a string
> containing the list of CPUs that the *domain* can run on. We should be
> putting this CPU mask into all of the VCPUs for the domain (rather than
> one bit in each as we currently do).
> 
> Adding support to enable separate masks for each VCPU isn't a bad idea,
> but we certainly don't want to break the behaviour of being able to set
> the mask for a domain.

This doesn't break the previous behavior though maybe the description or
implementation is misleading.  We may have dropped the behavior over time
as I seem to recall having a cpumap_t/cpumask in the domain structure,
but there isn't a domain-wide cpumask anymore.  Instead there is a
cpumask per vcpu.  The cpus parameter is used to restrict which
physical cpus the domains' vcpus' can use.  This is done by mapping
each vcpu to a value from the list of physical cpus the domain can
use.  The side-effect of that is that the cpumask of the vcpu has
only that cpu set, which prevents balancing when using the credit
scheduler.

Are you asking that we introduce in addition to the per-vcpu cpumask
another domain-wide mask that we would use to further restrict the vcpu
masks (think cpus_and(d->affinity, v->affinity))?  And have two config
variables like below?

vcpus=2
domain_cpus = "0-7"
vcpu_cpus = "0-3, 4-7"

> 
> Allowing a list of masks to be passed in (as well as a single string)
> isn't a bad idea.
> 
> Surely the comma separated list of vcpus isn't compatible with the
> current parsing? Or are you using the quotes to try and separate the
> different vcpu masks? That's horrid.

I didn't want the quotes either, nor the python list.  I guess I
misinterpreted your suggestion for requirements.  As for the commas, it
is certainly functional, but it sounds like we need some other cpumask
field delimiter, any takers for "|" ?

cpus = "0-3,^2|4-7|8|9-12"


> 
> BTW: As syntactic sugar, if the string starts with an exclusion, should
> it implicitly mean that mask starts off containing all 1's rather than
> all 0's? [avoiding the need to do "0-255,^0"]

That makes sense.  I'm not sure what happens right now if you lead with
an exclusion.  I can look into that after we settle on what do with
current patches.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Keir Fraser]

On 14/8/06 7:47 pm, "Ryan Harper" <ryanh@us.ibm.com> wrote:

> I didn't want the quotes either, nor the python list.  I guess I
> misinterpreted your suggestion for requirements.  As for the commas, it
> is certainly functional, but it sounds like we need some other cpumask
> field delimiter, any takers for "|" ?
> 
> cpus = "0-3,^2|4-7|8|9-12"

This is more readable than comma-space delimiters. I mainly care that we
support Python list form in the config file though.

I think Ian's right that setting the same cpumask for all VCPUs ought to be
the most common operation: Xen ought to do something sensible with that (it
probably already does with the new scheduler). So fixing the *existing*
syntax to do the sensible domain-wide thing, if it doesn't already, would be
nice.

 -- Keir

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2006-08-15 17:37]:
> 
> 
> 
> On 14/8/06 7:47 pm, "Ryan Harper" <ryanh@us.ibm.com> wrote:
> 
> > I didn't want the quotes either, nor the python list.  I guess I
> > misinterpreted your suggestion for requirements.  As for the commas, it
> > is certainly functional, but it sounds like we need some other cpumask
> > field delimiter, any takers for "|" ?
> > 
> > cpus = "0-3,^2|4-7|8|9-12"
> 
> This is more readable than comma-space delimiters. I mainly care that we
> support Python list form in the config file though.

The current code supports both.  It is trivial to switch the delimiter
for pure string form above to use "|".  The remaining question is
whether we should support both or only list.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

RE: [PATCH 2/3] xend: Add multiple cpumasks support

[Ian Pratt]

> After knocking off the dust, here it is.  Allows max_vcpus to be set
in
> the config file.  If not present, it defaults to 8.

Thanks. I think "vcpus_max" might be a better name, though.

Ian 

 
> Signed-off-by: Ryan Grimm <grimm@us.ibm.com>
> 
> diff -r ec03b24a2d83 -r 263d3eb8c182 docs/man/xmdomain.cfg.pod.5
> --- a/docs/man/xmdomain.cfg.pod.5	Tue Aug 15 19:53:55 2006 +0100
> +++ b/docs/man/xmdomain.cfg.pod.5	Wed Aug 16 13:45:35 2006 -0500
> @@ -176,6 +176,13 @@ kernel supports.  For instance:
> 
>  Will cause the domain to boot to runlevel 4.
> 
> +=item B<max_vcpus>
> +
> +The number of virtual cpus a domain can bring up in its life.  In
order
> +to use this the xen kernel must be compiled with SMP support.
> +
> +This defaults to 8, meaning the domain can bring up at most 8 vcpus.
> +
>  =item B<nfs_server>
> 
>  The IP address of the NFS server to use as the root device for the
> diff -r ec03b24a2d83 -r 263d3eb8c182 tools/examples/xmexample1
> --- a/tools/examples/xmexample1	Tue Aug 15 19:53:55 2006 +0100
> +++ b/tools/examples/xmexample1	Wed Aug 16 13:45:35 2006 -0500
> @@ -34,6 +34,9 @@ name = "ExampleDomain"
>  #cpus = ""         # leave to Xen to pick
>  #cpus = "0"        # all vcpus run on CPU0
>  #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
> +
> +# Max number of Virtual CPUS a domain can have in its life
> +#max_vcpus = 8
> 
>  # Number of Virtual CPUS to use, default is 1
>  #vcpus = 1
> diff -r ec03b24a2d83 -r 263d3eb8c182 tools/examples/xmexample2
> --- a/tools/examples/xmexample2	Tue Aug 15 19:53:55 2006 +0100
> +++ b/tools/examples/xmexample2	Wed Aug 16 13:45:35 2006 -0500
> @@ -64,6 +64,9 @@ name = "VM%d" % vmid
>  #cpus = "0"        # all vcpus run on CPU0
>  #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
>  #cpus = "%s" % vmid # set based on vmid (mod number of CPUs)
> +
> +# Max number of Virtual CPUS a domain can have in its life
> +max_vcpus = 8
> 
>  # Number of Virtual CPUS to use, default is 1
>  #vcpus = 1
> diff -r ec03b24a2d83 -r 263d3eb8c182
> tools/python/xen/xend/XendDomainInfo.py
> --- a/tools/python/xen/xend/XendDomainInfo.py	Tue Aug 15 19:53:55 2006
> +0100
> +++ b/tools/python/xen/xend/XendDomainInfo.py	Wed Aug 16 13:45:35 2006
-
> 0500
> @@ -128,6 +128,7 @@ ROUNDTRIPPING_CONFIG_ENTRIES = [
>  ROUNDTRIPPING_CONFIG_ENTRIES = [
>      ('uuid',       str),
>      ('vcpus',      int),
> +    ('max_vcpus',  int),
>      ('vcpu_avail', int),
>      ('cpu_weight', float),
>      ('memory',     int),
> @@ -567,6 +568,7 @@ class XendDomainInfo:
>                  avail = int(1)
> 
>              defaultInfo('vcpus',        lambda: avail)
> +            defaultInfo('max_vcpus',    lambda: 8)
>              defaultInfo('online_vcpus', lambda: self.info['vcpus'])
>              defaultInfo('max_vcpu_id',  lambda: self.info['vcpus']-1)
>              defaultInfo('vcpu_avail',   lambda: (1 <<
self.info['vcpus'])
> - 1)
> @@ -749,7 +751,7 @@ class XendDomainInfo:
>                  return 'offline'
> 
>          result = {}
> -        for v in range(0, self.info['vcpus']):
> +        for v in range(0, self.info['max_vcpus']):
>              result["cpu/%d/availability" % v] = availability(v)
>          return result
> 
> @@ -1231,7 +1233,7 @@ class XendDomainInfo:
>          self.recreateDom()
> 
>          # Set maximum number of vcpus in domain
> -        xc.domain_max_vcpus(self.domid, int(self.info['vcpus']))
> +        xc.domain_max_vcpus(self.domid, int(self.info['max_vcpus']))
> 
> 
>      def introduceDomain(self):
> diff -r ec03b24a2d83 -r 263d3eb8c182 tools/python/xen/xm/create.py
> --- a/tools/python/xen/xm/create.py	Tue Aug 15 19:53:55 2006 +0100
> +++ b/tools/python/xen/xm/create.py	Wed Aug 16 13:45:35 2006 -0500
> @@ -177,6 +177,10 @@ gopts.var('apic', val='APIC',
>  gopts.var('apic', val='APIC',
>            fn=set_int, default=0,
>            use="Disable or enable APIC of HVM domain.")
> +
> +gopts.var('max_vcpus', val='VCPUS',
> +          fn=set_int, default=8,
> +          use="max # of Virtual CPUS a domain will have in its
life.")
> 
>  gopts.var('vcpus', val='VCPUS',
>            fn=set_int, default=1,
> @@ -667,7 +671,7 @@ def make_config(vals):
>                  config.append([n, v])
> 
>      map(add_conf, ['name', 'memory', 'maxmem', 'restart',
'on_poweroff',
> -                   'on_reboot', 'on_crash', 'vcpus', 'features'])
> +                   'on_reboot', 'on_crash', 'vcpus', 'max_vcpus',
> 'features'])
> 
>      if vals.uuid is not None:
>          config.append(['uuid', vals.uuid])
> 
> 
> On Mon, Aug 14, 2006 at 05:46:05PM -0500, Ryan Harper wrote:
> > * Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2006-08-14 17:41]:
> > > > > Either Keir's cpu[X] = "Y" approach or my cpu = [ "A","B","C"
]
> > > approach
> > > > > seem workable.
> > > >
> > > > Your last email seemed to indicate to me that you didn't like
using
> > > > quoted values in a list to separate per-vcpu cpumask values.
Maybe I
> > > > was mistaken.
> > >
> > > If it's an honest python list I have no problem. Your example
appeared
> > > to be some quoting within a string.
> >
> > OK.
> >
> > > My approach is a list too...
> > >
> > > > > BTW: does the right thing happen in the face of vcpu hot
plugging?
> > > i.e.
> > > > > if I unplug a vcpu and put it back in do I keep the old mask?
If I
> > > add
> > > > > vcpus what mask do they get?
> > > >
> > > > unplug events only affect a vcpu's status.  The internal struct
> > > > vcpu in the hypervisor is not de-allocated/re-allocated during
> hotplug
> > > > events.
> > > >
> > > > We don't currently support a hotadd for vcpus that weren't
allocated
> > > at
> > > > domain creation time.  The current method for simulating hot-add
> would
> > > > be to start a domain with 32 VCPUS and disable all by the number
of
> > > > vcpus you currently want.  Ryan Grimm posted a patch back in
February
> > > > that had xend do this by adding a new config option, max_vcpus,
which
> > > > was used when calling xc_domain_max_vcpus() having the
hypervisor
> > > alloc
> > > > that max number of vcpus and then using the vcpus parameter to
> > > determine
> > > > how many to bring online.
> > >
> > > I like the idea of having a vcpus_max
> >
> > I'll see if Ryan Grimm can dust that one off and resend it.
> >
> > --
> > Ryan Harper
> > Software Engineer; Linux Technology Center
> > IBM Corp., Austin, Tx
> > (512) 838-9253   T/L: 678-9253
> > ryanh@us.ibm.com
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> --
> Thanks,
> Ryan Grimm
> IBM Linux Technology Center
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Grimm]

Sure, looks a bit better that way.  Here ya go:

Signed-off-by: Ryan Grimm <grimm@us.ibm.com>

diff -r ec03b24a2d83 -r a70a45471442 docs/man/xmdomain.cfg.pod.5
--- a/docs/man/xmdomain.cfg.pod.5	Tue Aug 15 19:53:55 2006 +0100
+++ b/docs/man/xmdomain.cfg.pod.5	Thu Aug 17 10:22:48 2006 -0500
@@ -176,6 +176,13 @@ kernel supports.  For instance:
 
 Will cause the domain to boot to runlevel 4.
 
+=item B<vcpus_max>
+
+The number of virtual cpus a domain can bring up in its life.  In order 
+to use this the xen kernel must be compiled with SMP support.
+
+This defaults to 8, meaning the domain can bring up at most 8 vcpus.
+
 =item B<nfs_server>
 
 The IP address of the NFS server to use as the root device for the
diff -r ec03b24a2d83 -r a70a45471442 tools/examples/xmexample1
--- a/tools/examples/xmexample1	Tue Aug 15 19:53:55 2006 +0100
+++ b/tools/examples/xmexample1	Thu Aug 17 10:22:48 2006 -0500
@@ -34,6 +34,9 @@ name = "ExampleDomain"
 #cpus = ""         # leave to Xen to pick
 #cpus = "0"        # all vcpus run on CPU0
 #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
+
+# Max number of Virtual CPUS a domain can have in its life
+#vcpus_max = 8
 
 # Number of Virtual CPUS to use, default is 1
 #vcpus = 1
diff -r ec03b24a2d83 -r a70a45471442 tools/examples/xmexample2
--- a/tools/examples/xmexample2	Tue Aug 15 19:53:55 2006 +0100
+++ b/tools/examples/xmexample2	Thu Aug 17 10:22:48 2006 -0500
@@ -64,6 +64,9 @@ name = "VM%d" % vmid
 #cpus = "0"        # all vcpus run on CPU0
 #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
 #cpus = "%s" % vmid # set based on vmid (mod number of CPUs)
+
+# Max number of Virtual CPUS a domain can have in its life
+vcpus_max = 8
 
 # Number of Virtual CPUS to use, default is 1
 #vcpus = 1
diff -r ec03b24a2d83 -r a70a45471442 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Tue Aug 15 19:53:55 2006 +0100
+++ b/tools/python/xen/xend/XendDomainInfo.py	Thu Aug 17 10:22:48 2006 -0500
@@ -128,6 +128,7 @@ ROUNDTRIPPING_CONFIG_ENTRIES = [
 ROUNDTRIPPING_CONFIG_ENTRIES = [
     ('uuid',       str),
     ('vcpus',      int),
+    ('vcpus_max',  int), 
     ('vcpu_avail', int),
     ('cpu_weight', float),
     ('memory',     int),
@@ -567,6 +568,7 @@ class XendDomainInfo:
                 avail = int(1)
 
             defaultInfo('vcpus',        lambda: avail)
+            defaultInfo('vcpus_max',    lambda: 8)
             defaultInfo('online_vcpus', lambda: self.info['vcpus'])
             defaultInfo('max_vcpu_id',  lambda: self.info['vcpus']-1)
             defaultInfo('vcpu_avail',   lambda: (1 << self.info['vcpus']) - 1)
@@ -749,7 +751,7 @@ class XendDomainInfo:
                 return 'offline'
 
         result = {}
-        for v in range(0, self.info['vcpus']):
+        for v in range(0, self.info['vcpus_max']):
             result["cpu/%d/availability" % v] = availability(v)
         return result
 
@@ -1231,7 +1233,7 @@ class XendDomainInfo:
         self.recreateDom()
 
         # Set maximum number of vcpus in domain
-        xc.domain_max_vcpus(self.domid, int(self.info['vcpus']))
+        xc.domain_max_vcpus(self.domid, int(self.info['vcpus_max']))
 
 
     def introduceDomain(self):
diff -r ec03b24a2d83 -r a70a45471442 tools/python/xen/xm/create.py
--- a/tools/python/xen/xm/create.py	Tue Aug 15 19:53:55 2006 +0100
+++ b/tools/python/xen/xm/create.py	Thu Aug 17 10:22:48 2006 -0500
@@ -177,6 +177,10 @@ gopts.var('apic', val='APIC',
 gopts.var('apic', val='APIC',
           fn=set_int, default=0,
           use="Disable or enable APIC of HVM domain.")
+
+gopts.var('vcpus_max', val='VCPUS',
+          fn=set_int, default=8,
+          use="max # of Virtual CPUS a domain will have in its life.")
 
 gopts.var('vcpus', val='VCPUS',
           fn=set_int, default=1,
@@ -667,7 +671,7 @@ def make_config(vals):
                 config.append([n, v])
 
     map(add_conf, ['name', 'memory', 'maxmem', 'restart', 'on_poweroff',
-                   'on_reboot', 'on_crash', 'vcpus', 'features'])
+                   'on_reboot', 'on_crash', 'vcpus', 'vcpus_max', 'features'])
 
     if vals.uuid is not None:
         config.append(['uuid', vals.uuid])



On Thu, Aug 17, 2006 at 03:01:34AM +0100, Ian Pratt wrote:
> > After knocking off the dust, here it is.  Allows max_vcpus to be set
> in
> > the config file.  If not present, it defaults to 8.
> 
> Thanks. I think "vcpus_max" might be a better name, though.
> 
> Ian 
> 
>  
> > Signed-off-by: Ryan Grimm <grimm@us.ibm.com>
> > 
> > diff -r ec03b24a2d83 -r 263d3eb8c182 docs/man/xmdomain.cfg.pod.5
> > --- a/docs/man/xmdomain.cfg.pod.5	Tue Aug 15 19:53:55 2006 +0100
> > +++ b/docs/man/xmdomain.cfg.pod.5	Wed Aug 16 13:45:35 2006 -0500
> > @@ -176,6 +176,13 @@ kernel supports.  For instance:
> > 
> >  Will cause the domain to boot to runlevel 4.
> > 
> > +=item B<max_vcpus>
> > +
> > +The number of virtual cpus a domain can bring up in its life.  In
> order
> > +to use this the xen kernel must be compiled with SMP support.
> > +
> > +This defaults to 8, meaning the domain can bring up at most 8 vcpus.
> > +
> >  =item B<nfs_server>
> > 
> >  The IP address of the NFS server to use as the root device for the
> > diff -r ec03b24a2d83 -r 263d3eb8c182 tools/examples/xmexample1
> > --- a/tools/examples/xmexample1	Tue Aug 15 19:53:55 2006 +0100
> > +++ b/tools/examples/xmexample1	Wed Aug 16 13:45:35 2006 -0500
> > @@ -34,6 +34,9 @@ name = "ExampleDomain"
> >  #cpus = ""         # leave to Xen to pick
> >  #cpus = "0"        # all vcpus run on CPU0
> >  #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
> > +
> > +# Max number of Virtual CPUS a domain can have in its life
> > +#max_vcpus = 8
> > 
> >  # Number of Virtual CPUS to use, default is 1
> >  #vcpus = 1
> > diff -r ec03b24a2d83 -r 263d3eb8c182 tools/examples/xmexample2
> > --- a/tools/examples/xmexample2	Tue Aug 15 19:53:55 2006 +0100
> > +++ b/tools/examples/xmexample2	Wed Aug 16 13:45:35 2006 -0500
> > @@ -64,6 +64,9 @@ name = "VM%d" % vmid
> >  #cpus = "0"        # all vcpus run on CPU0
> >  #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
> >  #cpus = "%s" % vmid # set based on vmid (mod number of CPUs)
> > +
> > +# Max number of Virtual CPUS a domain can have in its life
> > +max_vcpus = 8
> > 
> >  # Number of Virtual CPUS to use, default is 1
> >  #vcpus = 1
> > diff -r ec03b24a2d83 -r 263d3eb8c182
> > tools/python/xen/xend/XendDomainInfo.py
> > --- a/tools/python/xen/xend/XendDomainInfo.py	Tue Aug 15 19:53:55 2006
> > +0100
> > +++ b/tools/python/xen/xend/XendDomainInfo.py	Wed Aug 16 13:45:35 2006
> -
> > 0500
> > @@ -128,6 +128,7 @@ ROUNDTRIPPING_CONFIG_ENTRIES = [
> >  ROUNDTRIPPING_CONFIG_ENTRIES = [
> >      ('uuid',       str),
> >      ('vcpus',      int),
> > +    ('max_vcpus',  int),
> >      ('vcpu_avail', int),
> >      ('cpu_weight', float),
> >      ('memory',     int),
> > @@ -567,6 +568,7 @@ class XendDomainInfo:
> >                  avail = int(1)
> > 
> >              defaultInfo('vcpus',        lambda: avail)
> > +            defaultInfo('max_vcpus',    lambda: 8)
> >              defaultInfo('online_vcpus', lambda: self.info['vcpus'])
> >              defaultInfo('max_vcpu_id',  lambda: self.info['vcpus']-1)
> >              defaultInfo('vcpu_avail',   lambda: (1 <<
> self.info['vcpus'])
> > - 1)
> > @@ -749,7 +751,7 @@ class XendDomainInfo:
> >                  return 'offline'
> > 
> >          result = {}
> > -        for v in range(0, self.info['vcpus']):
> > +        for v in range(0, self.info['max_vcpus']):
> >              result["cpu/%d/availability" % v] = availability(v)
> >          return result
> > 
> > @@ -1231,7 +1233,7 @@ class XendDomainInfo:
> >          self.recreateDom()
> > 
> >          # Set maximum number of vcpus in domain
> > -        xc.domain_max_vcpus(self.domid, int(self.info['vcpus']))
> > +        xc.domain_max_vcpus(self.domid, int(self.info['max_vcpus']))
> > 
> > 
> >      def introduceDomain(self):
> > diff -r ec03b24a2d83 -r 263d3eb8c182 tools/python/xen/xm/create.py
> > --- a/tools/python/xen/xm/create.py	Tue Aug 15 19:53:55 2006 +0100
> > +++ b/tools/python/xen/xm/create.py	Wed Aug 16 13:45:35 2006 -0500
> > @@ -177,6 +177,10 @@ gopts.var('apic', val='APIC',
> >  gopts.var('apic', val='APIC',
> >            fn=set_int, default=0,
> >            use="Disable or enable APIC of HVM domain.")
> > +
> > +gopts.var('max_vcpus', val='VCPUS',
> > +          fn=set_int, default=8,
> > +          use="max # of Virtual CPUS a domain will have in its
> life.")
> > 
> >  gopts.var('vcpus', val='VCPUS',
> >            fn=set_int, default=1,
> > @@ -667,7 +671,7 @@ def make_config(vals):
> >                  config.append([n, v])
> > 
> >      map(add_conf, ['name', 'memory', 'maxmem', 'restart',
> 'on_poweroff',
> > -                   'on_reboot', 'on_crash', 'vcpus', 'features'])
> > +                   'on_reboot', 'on_crash', 'vcpus', 'max_vcpus',
> > 'features'])
> > 
> >      if vals.uuid is not None:
> >          config.append(['uuid', vals.uuid])
> > 
> > 
> > On Mon, Aug 14, 2006 at 05:46:05PM -0500, Ryan Harper wrote:
> > > * Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2006-08-14 17:41]:
> > > > > > Either Keir's cpu[X] = "Y" approach or my cpu = [ "A","B","C"
> ]
> > > > approach
> > > > > > seem workable.
> > > > >
> > > > > Your last email seemed to indicate to me that you didn't like
> using
> > > > > quoted values in a list to separate per-vcpu cpumask values.
> Maybe I
> > > > > was mistaken.
> > > >
> > > > If it's an honest python list I have no problem. Your example
> appeared
> > > > to be some quoting within a string.
> > >
> > > OK.
> > >
> > > > My approach is a list too...
> > > >
> > > > > > BTW: does the right thing happen in the face of vcpu hot
> plugging?
> > > > i.e.
> > > > > > if I unplug a vcpu and put it back in do I keep the old mask?
> If I
> > > > add
> > > > > > vcpus what mask do they get?
> > > > >
> > > > > unplug events only affect a vcpu's status.  The internal struct
> > > > > vcpu in the hypervisor is not de-allocated/re-allocated during
> > hotplug
> > > > > events.
> > > > >
> > > > > We don't currently support a hotadd for vcpus that weren't
> allocated
> > > > at
> > > > > domain creation time.  The current method for simulating hot-add
> > would
> > > > > be to start a domain with 32 VCPUS and disable all by the number
> of
> > > > > vcpus you currently want.  Ryan Grimm posted a patch back in
> February
> > > > > that had xend do this by adding a new config option, max_vcpus,
> which
> > > > > was used when calling xc_domain_max_vcpus() having the
> hypervisor
> > > > alloc
> > > > > that max number of vcpus and then using the vcpus parameter to
> > > > determine
> > > > > how many to bring online.
> > > >
> > > > I like the idea of having a vcpus_max
> > >
> > > I'll see if Ryan Grimm can dust that one off and resend it.
> > >
> > > --
> > > Ryan Harper
> > > Software Engineer; Linux Technology Center
> > > IBM Corp., Austin, Tx
> > > (512) 838-9253   T/L: 678-9253
> > > ryanh@us.ibm.com
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > 
> > --
> > Thanks,
> > Ryan Grimm
> > IBM Linux Technology Center
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Thanks,
Ryan Grimm
IBM Linux Technology Center

RE: [PATCH 2/3] xend: Add multiple cpumasks support

[Ian Pratt]

> > > cpus = "0-3,^2|4-7|8|9-12"
> >
> > This is more readable than comma-space delimiters. I mainly care
that we
> > support Python list form in the config file though.
> 
> The current code supports both.  It is trivial to switch the delimiter
> for pure string form above to use "|".  The remaining question is
> whether we should support both or only list.

I would certainly prefer to promote the list of strings syntax in docs,
but I don't really mind.

BTW: has any progress been made understanding the NUMA allocator crash
on Unisys boxen?

Thanks,
Ian

RE: [PATCH 2/3] xend: Add multiple cpumasks support

[Subrahmanian, Raj]

> BTW: has any progress been made understanding the NUMA 
> allocator crash on Unisys boxen?
I had lost the machine I was testing on to another project. Now that I
have it back, I should have some information out there soon.
Thanks
Raj

RE: [PATCH 2/3] xend: Add multiple cpumasks support

[Ian Pratt]

> > Either Keir's cpu[X] = "Y" approach or my cpu = [ "A","B","C" ]
approach
> > seem workable.
> 
> Your last email seemed to indicate to me that you didn't like using
> quoted values in a list to separate per-vcpu cpumask values.  Maybe I
> was mistaken.

If it's an honest python list I have no problem. Your example appeared
to be some quoting within a string.

> > Keir's approach is rather ill defined if someone tries using both
cpu=
> > and cpu[X]= in the same config file, but I don't see that as a big
> > problem.  Take your pick :-)
> 
> I'm leaning toward the list notation since I already have code that
> parses that properly.

My approach is a list too...

> > BTW: does the right thing happen in the face of vcpu hot plugging?
i.e.
> > if I unplug a vcpu and put it back in do I keep the old mask? If I
add
> > vcpus what mask do they get?
> 
> unplug events only affect a vcpu's status.  The internal struct
> vcpu in the hypervisor is not de-allocated/re-allocated during hotplug
> events.
> 
> We don't currently support a hotadd for vcpus that weren't allocated
at
> domain creation time.  The current method for simulating hot-add would
> be to start a domain with 32 VCPUS and disable all by the number of
> vcpus you currently want.  Ryan Grimm posted a patch back in February
> that had xend do this by adding a new config option, max_vcpus, which
> was used when calling xc_domain_max_vcpus() having the hypervisor
alloc
> that max number of vcpus and then using the vcpus parameter to
determine
> how many to bring online.

I like the idea of having a vcpus_max

> > We should probably add a 'vcpu-pin' variant that enables the mask to
be
> > set for all vcpus. Perhaps '-1' for the vcpu number? Or should we
add
> > 'vcpu-pin-all'?
> 
> vcpu-pin using -1 is probably the quickest, least intrusive method to
> get this behavior.  We could also use a keyword, all for instance:
> 
> xm vcpu-pin vm1 all 0-4,^5

Nice.

> > [secondly, what do you think about implicitly defaulting the mask to
all
> > 1's if the first item in a cpu mask is an exclusion? e.g. ^1]
> 
> That makes sense.  I'll include a patch in the set to add this
behavior.

Thanks,
Ian

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Harper]

* Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2006-08-14 17:41]:
> > > Either Keir's cpu[X] = "Y" approach or my cpu = [ "A","B","C" ]
> approach
> > > seem workable.
> > 
> > Your last email seemed to indicate to me that you didn't like using
> > quoted values in a list to separate per-vcpu cpumask values.  Maybe I
> > was mistaken.
> 
> If it's an honest python list I have no problem. Your example appeared
> to be some quoting within a string.

OK.

> My approach is a list too...
> 
> > > BTW: does the right thing happen in the face of vcpu hot plugging?
> i.e.
> > > if I unplug a vcpu and put it back in do I keep the old mask? If I
> add
> > > vcpus what mask do they get?
> > 
> > unplug events only affect a vcpu's status.  The internal struct
> > vcpu in the hypervisor is not de-allocated/re-allocated during hotplug
> > events.
> > 
> > We don't currently support a hotadd for vcpus that weren't allocated
> at
> > domain creation time.  The current method for simulating hot-add would
> > be to start a domain with 32 VCPUS and disable all by the number of
> > vcpus you currently want.  Ryan Grimm posted a patch back in February
> > that had xend do this by adding a new config option, max_vcpus, which
> > was used when calling xc_domain_max_vcpus() having the hypervisor
> alloc
> > that max number of vcpus and then using the vcpus parameter to
> determine
> > how many to bring online.
> 
> I like the idea of having a vcpus_max

I'll see if Ryan Grimm can dust that one off and resend it.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Grimm]

After knocking off the dust, here it is.  Allows max_vcpus to be set in
the config file.  If not present, it defaults to 8.

Signed-off-by: Ryan Grimm <grimm@us.ibm.com>

diff -r ec03b24a2d83 -r 263d3eb8c182 docs/man/xmdomain.cfg.pod.5
--- a/docs/man/xmdomain.cfg.pod.5	Tue Aug 15 19:53:55 2006 +0100
+++ b/docs/man/xmdomain.cfg.pod.5	Wed Aug 16 13:45:35 2006 -0500
@@ -176,6 +176,13 @@ kernel supports.  For instance:
 
 Will cause the domain to boot to runlevel 4.
 
+=item B<max_vcpus>
+
+The number of virtual cpus a domain can bring up in its life.  In order 
+to use this the xen kernel must be compiled with SMP support.
+
+This defaults to 8, meaning the domain can bring up at most 8 vcpus.
+
 =item B<nfs_server>
 
 The IP address of the NFS server to use as the root device for the
diff -r ec03b24a2d83 -r 263d3eb8c182 tools/examples/xmexample1
--- a/tools/examples/xmexample1	Tue Aug 15 19:53:55 2006 +0100
+++ b/tools/examples/xmexample1	Wed Aug 16 13:45:35 2006 -0500
@@ -34,6 +34,9 @@ name = "ExampleDomain"
 #cpus = ""         # leave to Xen to pick
 #cpus = "0"        # all vcpus run on CPU0
 #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
+
+# Max number of Virtual CPUS a domain can have in its life
+#max_vcpus = 8
 
 # Number of Virtual CPUS to use, default is 1
 #vcpus = 1
diff -r ec03b24a2d83 -r 263d3eb8c182 tools/examples/xmexample2
--- a/tools/examples/xmexample2	Tue Aug 15 19:53:55 2006 +0100
+++ b/tools/examples/xmexample2	Wed Aug 16 13:45:35 2006 -0500
@@ -64,6 +64,9 @@ name = "VM%d" % vmid
 #cpus = "0"        # all vcpus run on CPU0
 #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
 #cpus = "%s" % vmid # set based on vmid (mod number of CPUs)
+
+# Max number of Virtual CPUS a domain can have in its life
+max_vcpus = 8
 
 # Number of Virtual CPUS to use, default is 1
 #vcpus = 1
diff -r ec03b24a2d83 -r 263d3eb8c182 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Tue Aug 15 19:53:55 2006 +0100
+++ b/tools/python/xen/xend/XendDomainInfo.py	Wed Aug 16 13:45:35 2006 -0500
@@ -128,6 +128,7 @@ ROUNDTRIPPING_CONFIG_ENTRIES = [
 ROUNDTRIPPING_CONFIG_ENTRIES = [
     ('uuid',       str),
     ('vcpus',      int),
+    ('max_vcpus',  int), 
     ('vcpu_avail', int),
     ('cpu_weight', float),
     ('memory',     int),
@@ -567,6 +568,7 @@ class XendDomainInfo:
                 avail = int(1)
 
             defaultInfo('vcpus',        lambda: avail)
+            defaultInfo('max_vcpus',    lambda: 8)
             defaultInfo('online_vcpus', lambda: self.info['vcpus'])
             defaultInfo('max_vcpu_id',  lambda: self.info['vcpus']-1)
             defaultInfo('vcpu_avail',   lambda: (1 << self.info['vcpus']) - 1)
@@ -749,7 +751,7 @@ class XendDomainInfo:
                 return 'offline'
 
         result = {}
-        for v in range(0, self.info['vcpus']):
+        for v in range(0, self.info['max_vcpus']):
             result["cpu/%d/availability" % v] = availability(v)
         return result
 
@@ -1231,7 +1233,7 @@ class XendDomainInfo:
         self.recreateDom()
 
         # Set maximum number of vcpus in domain
-        xc.domain_max_vcpus(self.domid, int(self.info['vcpus']))
+        xc.domain_max_vcpus(self.domid, int(self.info['max_vcpus']))
 
 
     def introduceDomain(self):
diff -r ec03b24a2d83 -r 263d3eb8c182 tools/python/xen/xm/create.py
--- a/tools/python/xen/xm/create.py	Tue Aug 15 19:53:55 2006 +0100
+++ b/tools/python/xen/xm/create.py	Wed Aug 16 13:45:35 2006 -0500
@@ -177,6 +177,10 @@ gopts.var('apic', val='APIC',
 gopts.var('apic', val='APIC',
           fn=set_int, default=0,
           use="Disable or enable APIC of HVM domain.")
+
+gopts.var('max_vcpus', val='VCPUS',
+          fn=set_int, default=8,
+          use="max # of Virtual CPUS a domain will have in its life.")
 
 gopts.var('vcpus', val='VCPUS',
           fn=set_int, default=1,
@@ -667,7 +671,7 @@ def make_config(vals):
                 config.append([n, v])
 
     map(add_conf, ['name', 'memory', 'maxmem', 'restart', 'on_poweroff',
-                   'on_reboot', 'on_crash', 'vcpus', 'features'])
+                   'on_reboot', 'on_crash', 'vcpus', 'max_vcpus', 'features'])
 
     if vals.uuid is not None:
         config.append(['uuid', vals.uuid])


On Mon, Aug 14, 2006 at 05:46:05PM -0500, Ryan Harper wrote:
> * Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2006-08-14 17:41]:
> > > > Either Keir's cpu[X] = "Y" approach or my cpu = [ "A","B","C" ]
> > approach
> > > > seem workable.
> > > 
> > > Your last email seemed to indicate to me that you didn't like using
> > > quoted values in a list to separate per-vcpu cpumask values.  Maybe I
> > > was mistaken.
> > 
> > If it's an honest python list I have no problem. Your example appeared
> > to be some quoting within a string.
> 
> OK.
> 
> > My approach is a list too...
> > 
> > > > BTW: does the right thing happen in the face of vcpu hot plugging?
> > i.e.
> > > > if I unplug a vcpu and put it back in do I keep the old mask? If I
> > add
> > > > vcpus what mask do they get?
> > > 
> > > unplug events only affect a vcpu's status.  The internal struct
> > > vcpu in the hypervisor is not de-allocated/re-allocated during hotplug
> > > events.
> > > 
> > > We don't currently support a hotadd for vcpus that weren't allocated
> > at
> > > domain creation time.  The current method for simulating hot-add would
> > > be to start a domain with 32 VCPUS and disable all by the number of
> > > vcpus you currently want.  Ryan Grimm posted a patch back in February
> > > that had xend do this by adding a new config option, max_vcpus, which
> > > was used when calling xc_domain_max_vcpus() having the hypervisor
> > alloc
> > > that max number of vcpus and then using the vcpus parameter to
> > determine
> > > how many to bring online.
> > 
> > I like the idea of having a vcpus_max
> 
> I'll see if Ryan Grimm can dust that one off and resend it.
> 
> -- 
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> (512) 838-9253   T/L: 678-9253
> ryanh@us.ibm.com
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Thanks,
Ryan Grimm
IBM Linux Technology Center

RE: [PATCH 2/3] xend: Add multiple cpumasks support

[Ian Pratt]

> Are you interested in the multi cpumask approach?  

Yes: it certainly doesn't hurt to have that flexibility. 

> If so any thoughts on how
> you'd like to see multiple cpumasks in the config file?

Either Keir's cpu[X] = "Y" approach or my cpu = [ "A","B","C" ] approach
seem workable.

Keir's approach is rather ill defined if someone tries using both cpu=
and cpu[X]= in the same config file, but I don't see that as a big
problem.  Take your pick :-)

BTW: does the right thing happen in the face of vcpu hot plugging? i.e.
if I unplug a vcpu and put it back in do I keep the old mask? If I add
vcpus what mask do they get?

We should probably add a 'vcpu-pin' variant that enables the mask to be
set for all vcpus. Perhaps '-1' for the vcpu number? Or should we add
'vcpu-pin-all'?

[secondly, what do you think about implicitly defaulting the mask to all
1's if the first item in a cpu mask is an exclusion? e.g. ^1] 

Ian

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Harper]

* Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2006-08-14 17:04]:
> > Are you interested in the multi cpumask approach?  
> 
> Yes: it certainly doesn't hurt to have that flexibility. 
> 
> > If so any thoughts on how
> > you'd like to see multiple cpumasks in the config file?
> 
> Either Keir's cpu[X] = "Y" approach or my cpu = [ "A","B","C" ] approach
> seem workable.

Your last email seemed to indicate to me that you didn't like using
quoted values in a list to separate per-vcpu cpumask values.  Maybe I
was mistaken.

> 
> Keir's approach is rather ill defined if someone tries using both cpu=
> and cpu[X]= in the same config file, but I don't see that as a big
> problem.  Take your pick :-)

I'm leaning toward the list notation since I already have code that
parses that properly.

> 
> BTW: does the right thing happen in the face of vcpu hot plugging? i.e.
> if I unplug a vcpu and put it back in do I keep the old mask? If I add
> vcpus what mask do they get?

unplug events only affect a vcpu's status.  The internal struct
vcpu in the hypervisor is not de-allocated/re-allocated during hotplug
events.

We don't currently support a hotadd for vcpus that weren't allocated at
domain creation time.  The current method for simulating hot-add would
be to start a domain with 32 VCPUS and disable all by the number of
vcpus you currently want.  Ryan Grimm posted a patch back in February
that had xend do this by adding a new config option, max_vcpus, which
was used when calling xc_domain_max_vcpus() having the hypervisor alloc
that max number of vcpus and then using the vcpus parameter to determine
how many to bring online.

> 
> We should probably add a 'vcpu-pin' variant that enables the mask to be
> set for all vcpus. Perhaps '-1' for the vcpu number? Or should we add
> 'vcpu-pin-all'?

vcpu-pin using -1 is probably the quickest, least intrusive method to
get this behavior.  We could also use a keyword, all for instance:

xm vcpu-pin vm1 all 0-4,^5 

> 
> [secondly, what do you think about implicitly defaulting the mask to all
> 1's if the first item in a cpu mask is an exclusion? e.g. ^1] 

That makes sense.  I'll include a patch in the set to add this behavior.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Keir Fraser]

On 14/8/06 11:20 pm, "Ryan Harper" <ryanh@us.ibm.com> wrote:

>> Either Keir's cpu[X] = "Y" approach or my cpu = [ "A","B","C" ] approach
>> seem workable.
>> 
>> Keir's approach is rather ill defined if someone tries using both cpu=
>> and cpu[X]= in the same config file, but I don't see that as a big
>> problem.  Take your pick :-)
> 
> I'm leaning toward the list notation since I already have code that
> parses that properly.

Am I mistaken or aren't both the above forms basically the same (in both
cases 'cpu' is a list of strings)? I like this form, even if we cook it down
in xm into a different sxp syntax (if you want to avoid python lists in the
sxp).

 -- Keir

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2006-08-15 17:36]:
> 
> 
> 
> On 14/8/06 11:20 pm, "Ryan Harper" <ryanh@us.ibm.com> wrote:
> 
> >> Either Keir's cpu[X] = "Y" approach or my cpu = [ "A","B","C" ] approach
> >> seem workable.
> >> 
> >> Keir's approach is rather ill defined if someone tries using both cpu=
> >> and cpu[X]= in the same config file, but I don't see that as a big
> >> problem.  Take your pick :-)
> > 
> > I'm leaning toward the list notation since I already have code that
> > parses that properly.
> 
> Am I mistaken or aren't both the above forms basically the same (in both
> cases 'cpu' is a list of strings)? I like this form, even if we cook it down
> in xm into a different sxp syntax (if you want to avoid python lists in the
> sxp).

I'm not really opposed to any particular way.  IMO, I think that having a
single config variable, cpus, is simpler, and more compact, for
instance, when passing a one-time value on the command line.  Certainly
cpu[X]="Y" is easier to understand, but I don't think it is a far
stretch to list form.

If we went with cpu[X]="Y", would we do away with cpus?  If not, would
a cpus value map to cpu[0]?  And what about 'cpu', which currently is
prepended to the cpus value, which retains the behavior of pinning
VCPU0?

I'm about to resend the patches which currently parse the list form,
[ "a", "b", "c" ].  If you are set on cpu[X]="Y" and we know how we want
to handled the cases I mentioned above, I can rework to suit.  My
preference is the list form which has the benefit of working code
behind it.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

RE: [PATCH 2/3] xend: Add multiple cpumasks support

[Ian Pratt]

> > Adding support to enable separate masks for each VCPU isn't a bad
idea,
> > but we certainly don't want to break the behaviour of being able to
set
> > the mask for a domain.
> 
> This doesn't break the previous behavior though maybe the description
or
> implementation is misleading.  We may have dropped the behavior over
time
> as I seem to recall having a cpumap_t/cpumask in the domain structure,
> but there isn't a domain-wide cpumask anymore.  Instead there is a
> cpumask per vcpu.  The cpus parameter is used to restrict which
> physical cpus the domains' vcpus' can use.  This is done by mapping
> each vcpu to a value from the list of physical cpus the domain can
> use.  The side-effect of that is that the cpumask of the vcpu has
> only that cpu set, which prevents balancing when using the credit
> scheduler.

The current code doesn't do what the comment in the example config file
says. We should just fix the code to match the comment!

> Are you asking that we introduce in addition to the per-vcpu cpumask
> another domain-wide mask that we would use to further restrict the
vcpu
> masks (think cpus_and(d->affinity, v->affinity))?  And have two config
> variables like below?

There's no need: just set them all vcpus to the same mask.

Thanks,
Ian

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Harper]

* Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2006-08-14 13:56]:
> > > Adding support to enable separate masks for each VCPU isn't a bad
> idea,
> > > but we certainly don't want to break the behaviour of being able to
> set
> > > the mask for a domain.
> > 
> > This doesn't break the previous behavior though maybe the description
> or
> > implementation is misleading.  We may have dropped the behavior over
> time
> > as I seem to recall having a cpumap_t/cpumask in the domain structure,
> > but there isn't a domain-wide cpumask anymore.  Instead there is a
> > cpumask per vcpu.  The cpus parameter is used to restrict which
> > physical cpus the domains' vcpus' can use.  This is done by mapping
> > each vcpu to a value from the list of physical cpus the domain can
> > use.  The side-effect of that is that the cpumask of the vcpu has
> > only that cpu set, which prevents balancing when using the credit
> > scheduler.
> 
> The current code doesn't do what the comment in the example config file
> says. We should just fix the code to match the comment!

Certainly.  I'll sync them up.

> 
> > Are you asking that we introduce in addition to the per-vcpu cpumask
> > another domain-wide mask that we would use to further restrict the
> vcpu
> > masks (think cpus_and(d->affinity, v->affinity))?  And have two config
> > variables like below?
> 
> There's no need: just set them all vcpus to the same mask.

OK.  It seems like I went a step too far.  I'll resend the simpler patch
of just repeating the same mask for each vcpu in the domain.  Are you
interested in the multi cpumask approach?  If so any thoughts on how
you'd like to see multiple cpumasks in the config file?

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Harper]

* Ryan Harper <ryanh@us.ibm.com> [2006-08-14 14:34]:
> * Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2006-08-14 13:56]:
> > > > Adding support to enable separate masks for each VCPU isn't a bad
> > idea,
> > > > but we certainly don't want to break the behaviour of being able to
> > set
> > > > the mask for a domain.
> > > 
> > > This doesn't break the previous behavior though maybe the description
> > or
> > > implementation is misleading.  We may have dropped the behavior over
> > time
> > > as I seem to recall having a cpumap_t/cpumask in the domain structure,
> > > but there isn't a domain-wide cpumask anymore.  Instead there is a
> > > cpumask per vcpu.  The cpus parameter is used to restrict which
> > > physical cpus the domains' vcpus' can use.  This is done by mapping
> > > each vcpu to a value from the list of physical cpus the domain can
> > > use.  The side-effect of that is that the cpumask of the vcpu has
> > > only that cpu set, which prevents balancing when using the credit
> > > scheduler.
> > 
> > The current code doesn't do what the comment in the example config file
> > says. We should just fix the code to match the comment!
> 
> Certainly.  I'll sync them up.
> 
> > 
> > > Are you asking that we introduce in addition to the per-vcpu cpumask
> > > another domain-wide mask that we would use to further restrict the
> > vcpu
> > > masks (think cpus_and(d->affinity, v->affinity))?  And have two config
> > > variables like below?
> > 
> > There's no need: just set them all vcpus to the same mask.
> 
> OK.  It seems like I went a step too far.  I'll resend the simpler patch
> of just repeating the same mask for each vcpu in the domain.  Are you

Here is the simple patch that applies the specified cpumask to each
vcpu.  There is a bit of behavior change for sedf users as xen picks the
first bit in the mask when allocating vcpus leaving all of the domains'
vcpus on the first cpu in the mask and requires manual pinning to spread
them out.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com


diffstat output:
 XendDomainInfo.py |    7 ++-----
 1 files changed, 2 insertions(+), 5 deletions(-)

Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
---
# HG changeset patch
# User Ryan Harper <ryanh@us.ibm.com>
# Date 1155405919 18000
# Node ID 83fd301be2d6ea464079044406c3815fd7ae0796
# Parent  f328519053f5a444af475ec10dc8089a0b176e3f
Apply the domain cpumask to each vcpu rather than mapping vcpus to cpus in the
list.  This is more inline with the comments for the cpus parameter and also
allows the credit scheduler to balance vcpus within the domain cpumask.

diff -r f328519053f5 -r 83fd301be2d6 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Mon Aug 14 10:58:02 2006 +0100
+++ b/tools/python/xen/xend/XendDomainInfo.py	Sat Aug 12 13:05:19 2006 -0500
@@ -1272,12 +1272,9 @@ class XendDomainInfo:
             # repin domain vcpus if a restricted cpus list is provided
             # this is done prior to memory allocation to aide in memory
             # distribution for NUMA systems.
-            cpus = self.info['cpus']
-            if cpus is not None and len(cpus) > 0:
+            if self.info['cpus'] is not None and len(self.info['cpus']) > 0:
                 for v in range(0, self.info['max_vcpu_id']+1):
-                    # pincpu takes a list of ints
-                    cpu = [ int( cpus[v % len(cpus)] ) ]
-                    xc.vcpu_setaffinity(self.domid, v, cpu)
+                    xc.vcpu_setaffinity(self.domid, v, self.info['cpus'])
 
             # set domain maxmem in KiB
             xc.domain_setmaxmem(self.domid, self.info['maxmem'] * 1024)

[PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Harper]

This patch modifies xend to accept and parse multiple cpumask strings
from the cpus parameter.  The cpus string stays the same, but it now
can parse either a python list of strings:

[ '2-5, '2-5', '2-5' ]

A regular string with ", " as the separator:

"2-5, 2-5, 2-5, 2-5"

or a mixture of both:

[ 2-5, '2-5', 2-5, '2-5' ]

all result in the same list of integers which is used to create the
cpumask for each vcpu in the domain.

Without this patch the cpus parameter only enables one bit of the
cpumask preventing the credit scheduler from balancing.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com


diffstat output:
 examples/xmexample.hvm            |    3 +
 examples/xmexample.vti            |    1 
 examples/xmexample1               |    3 +
 examples/xmexample2               |    3 +
 examples/xmexample3               |    3 +
 python/xen/xend/XendDomainInfo.py |   66 +++++++++++++++++++++++++-------------
 6 files changed, 53 insertions(+), 26 deletions(-)

Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
---
diff -r e6de470f9287 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Sat Aug 12 08:22:45 2006 -0500
+++ b/tools/python/xen/xend/XendDomainInfo.py	Sat Aug 12 08:22:47 2006 -0500
@@ -328,27 +328,49 @@ def parseConfig(config):
            else:
                result['cpus'] = str(result['cpu'])
 
-        # convert 'cpus' string to list of ints
-        # 'cpus' supports a list of ranges (0-3), seperated by
-        # commas, and negation, (^1).  
-        # Precedence is settled by  order of the string:
-        #     "0-3,^1"   -> [0,2,3]
-        #     "0-3,^1,1" -> [0,1,2,3]
-        if result['cpus']:
-            cpus = []
-            for c in result['cpus'].split(','):
-                if c.find('-') != -1:             
-                    (x,y) = c.split('-')
-                    for i in range(int(x),int(y)+1):
-                        cpus.append(int(i))
-                else:
-                    # remove this element from the list 
-                    if c[0] == '^':
-                        cpus = [x for x in cpus if x != int(c[1:])]
+        if result['cpus'] is not None:
+            # see if cpus string specifies multiple cpumasks, or just one
+            # e.g: "[ '2-5', '0-1', '0-3,^1', '5' ]" vs. "0-3,5"
+            if result['cpus'].startswith("["):
+                # the below was tested with the following sample string
+                # "[4, '2-5','2-5', '1-3,6,^2', '2-6','1' 1-7,^2 ]" and resulted in
+                # ['4', '2-5', '2-5', '1-3,6,^2', '2-6', '1']
+                result['cpus'] = filter(lambda x: len(x), map(lambda x: x.strip(", "),
+                           result['cpus'].replace('[',"").replace(']',"").split("'")))
+
+            # if user didn't use list bracket, convert to to list of strings
+            # cpus = "4, 2-5,^4, 1-3,5,^2, 1, ^2" ->
+            # ['4', '2-5,^4', '1-3,5,^2', '1', '^2'] 
+            # also takes care of cpus = "4"
+            else:
+                result['cpus'] = map(lambda x: x.strip(", "), result['cpus'].split())
+
+            # convert 'cpus' list of strings into a list of list of ints
+            # 'cpus' supports a list of ranges (0-3), seperated by
+            # commas, and negation, (^1).  
+            # Precedence is settled by  order of the string:
+            #     "0-3,^1"   -> [0,2,3]
+            #     "0-3,^1,1" -> [0,1,2,3]
+
+            new_cpus = []
+            for x in result['cpus']:
+                cpus = []
+                for c in x.split(','):
+                    if c.find('-') != -1:             
+                        (x,y) = c.split('-')
+                        for i in range(int(x),int(y)+1):
+                            cpus.append(int(i))
                     else:
-                        cpus.append(int(c))
-
-            result['cpus'] = cpus
+                        # remove this element from the list 
+                        if c[0] == '^':
+                            cpus = [x for x in cpus if x != int(c[1:])]
+                        else:
+                            cpus.append(int(c))
+
+                new_cpus.append(cpus)
+             
+            
+            result['cpus'] = new_cpus
         
     except ValueError, exn:
         raise VmError(
@@ -1275,8 +1297,8 @@ class XendDomainInfo:
             cpus = self.info['cpus']
             if cpus is not None and len(cpus) > 0:
                 for v in range(0, self.info['max_vcpu_id']+1):
-                    # pincpu takes a list of ints
-                    cpu = [ int( cpus[v % len(cpus)] ) ]
+                    # pincpu takes a list of ints, 
+                    cpu = map(lambda x: int(x), cpus[v % len(cpus)])
                     xc.vcpu_setaffinity(self.domid, v, cpu)
 
             # set domain maxmem in KiB
diff -r e6de470f9287 tools/examples/xmexample.hvm
--- a/tools/examples/xmexample.hvm	Sat Aug 12 08:22:45 2006 -0500
+++ b/tools/examples/xmexample.hvm	Sat Aug 12 08:34:13 2006 -0500
@@ -47,10 +47,11 @@ name = "ExampleHVMDomain"
 # enable/disable HVM guest APIC, default=0 (disabled)
 #apic=0
 
-# List of which CPUS this domain is allowed to use, default Xen picks
+# List of which CPUS vcpus are allowed to use, default Xen picks
 #cpus = ""         # leave to Xen to pick
 #cpus = "0"        # all vcpus run on CPU0
 #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
+#cpus = "0-1, 2-3" # run VCPU0 on CPU0-1, VCPU1 on CPU2-3
 
 # Optionally define mac and/or bridge for the network interfaces.
 # Random MACs are assigned if not given.
diff -r e6de470f9287 tools/examples/xmexample.vti
--- a/tools/examples/xmexample.vti	Sat Aug 12 08:22:45 2006 -0500
+++ b/tools/examples/xmexample.vti	Sat Aug 12 08:39:42 2006 -0500
@@ -34,6 +34,7 @@ name = "ExampleVTIDomain"
 #cpus = ""         # leave to Xen to pick
 #cpus = "0"        # all vcpus run on CPU0
 #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
+#cpus = "0-1, 2-3" # run VCPU0 on CPU0-1, VCPU1 on CPU2-3
 
 # Optionally define mac and/or bridge for the network interfaces.
 # Random MACs are assigned if not given.
diff -r e6de470f9287 tools/examples/xmexample1
--- a/tools/examples/xmexample1	Sat Aug 12 08:22:45 2006 -0500
+++ b/tools/examples/xmexample1	Sat Aug 12 08:27:02 2006 -0500
@@ -30,10 +30,11 @@ name = "ExampleDomain"
 # on each call to 'xm create'.
 #uuid = "06ed00fe-1162-4fc4-b5d8-11993ee4a8b9"
 
-# List of which CPUS this domain is allowed to use, default Xen picks
+# List of which CPUS vcpus are allowed to use, default Xen picks
 #cpus = ""         # leave to Xen to pick
 #cpus = "0"        # all vcpus run on CPU0
 #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
+#cpus = "0-1, 2-3" # run VCPU0 on CPU0-1, VCPU1 on CPU2-3
 
 # Number of Virtual CPUS to use, default is 1
 #vcpus = 1
diff -r e6de470f9287 tools/examples/xmexample2
--- a/tools/examples/xmexample2	Sat Aug 12 08:22:45 2006 -0500
+++ b/tools/examples/xmexample2	Sat Aug 12 08:27:47 2006 -0500
@@ -59,10 +59,11 @@ name = "VM%d" % vmid
 # on each call to 'xm create'.
 #uuid = "06ed00fe-1162-4fc4-b5d8-11993ee4a8b9"
 
-# List of which CPUS this domain is allowed to use, default Xen picks
+# List of which CPUS vcpus are allowed to use, default Xen picks
 #cpus = ""         # leave to Xen to pick
 #cpus = "0"        # all vcpus run on CPU0
 #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
+#cpus = "0-1, 2-3" # run VCPU0 on CPU0-1, VCPU1 on CPU2-3
 #cpus = "%s" % vmid # set based on vmid (mod number of CPUs)
 
 # Number of Virtual CPUS to use, default is 1
diff -r e6de470f9287 tools/examples/xmexample3
--- a/tools/examples/xmexample3	Sat Aug 12 08:22:45 2006 -0500
+++ b/tools/examples/xmexample3	Sat Aug 12 08:33:35 2006 -0500
@@ -59,10 +59,11 @@ name = "VM%d" % vmid
 # on each call to 'xm create'.
 #uuid = "06ed00fe-1162-4fc4-b5d8-11993ee4a8b9"
 
-# List of which CPUS this domain is allowed to use, default Xen picks
+# List of which CPUS vcpus are allowed to use, default Xen picks
 #cpus = ""         # leave to Xen to pick
 #cpus = "0"        # all vcpus run on CPU0
 #cpus = "0-3,5,^1" # run on cpus 0,2,3,5
+#cpus = "0-1, 2-3" # run VCPU0 on CPU0-1, VCPU1 on CPU2-3
 cpus = "%s" % vmid # set based on vmid (mod number of CPUs)
 
 #----------------------------------------------------------------------------

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Keir Fraser]

On 14/8/06 5:57 pm, "Ryan Harper" <ryanh@us.ibm.com> wrote:

> This patch modifies xend to accept and parse multiple cpumask strings
> from the cpus parameter.  The cpus string stays the same, but it now
> can parse either a python list of strings:
> 
> [ '2-5, '2-5', '2-5' ]
> 
> A regular string with ", " as the separator:
> 
> "2-5, 2-5, 2-5, 2-5"
> 
> or a mixture of both:
> 
> [ 2-5, '2-5', 2-5, '2-5' ]

This isn't really a mixture of both, is it? It looks syntactically incorrect
(e.g., first 2-5 is unquoted so not a string).

I'm not sure about the use of ', ' as a delimiter. It would be less
confusing to strictly require the use of the list form. I would imagine it's
then clearest used as:
Cpus = []
Cpus[0] = '2-5'
Cpus[1] = '3-4'
...

Trying to read long lists of cpu constraints with spaces in will make people
go cross-eyed!

 -- Keir

> all result in the same list of integers which is used to create the
> cpumask for each vcpu in the domain.

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2006-08-14 12:37]:
> On 14/8/06 5:57 pm, "Ryan Harper" <ryanh@us.ibm.com> wrote:
> 
> > This patch modifies xend to accept and parse multiple cpumask strings
> > from the cpus parameter.  The cpus string stays the same, but it now
> > can parse either a python list of strings:
> > 
> > [ '2-5, '2-5', '2-5' ]
> > 
> > A regular string with ", " as the separator:
> > 
> > "2-5, 2-5, 2-5, 2-5"
> > 
> > or a mixture of both:
> > 
> > [ 2-5, '2-5', 2-5, '2-5' ]
> 
> This isn't really a mixture of both, is it? It looks syntactically incorrect
> (e.g., first 2-5 is unquoted so not a string).

create.py ends up passing the whole thing as a string even if you mark
it up as a list with strings embedded which is why I can mix it.  If you
like, we can drop support for dealing with the [ ... ] form and just use
cpus = ""

> 
> I'm not sure about the use of ', ' as a delimiter. It would be less
> confusing to strictly require the use of the list form. I would imagine it's
> then clearest used as:
> Cpus = []
> Cpus[0] = '2-5'
> Cpus[1] = '3-4'
> ...

I'd prefer to not have to use any language arrays.  I'm not sure how
this will map to the xml-based config files that Ewan was talking about,
but I suppose he will have to come up with something since we have
things like the disk parameter which is in python list format.

> 
> Trying to read long lists of cpu constraints with spaces in will make people
> go cross-eyed!

I agree.  The point was that I'm attempting to parse as much as I can
make sense of,  not to indicate how we should tell the user to convey this
information.  I also like being able to retain the cpu=, cpus=, previous
syntax while extending it to support multiple cpumasks.  

I think the following (which I added in the example files)
is good enough without having to resort to arrays/dictionaries.

cpus = "2-5, 2-5, 2-5, 2-5"

I'm open to other field delimiters if that is a point of contention.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: [PATCH 2/3] xend: Add multiple cpumasks support

[Keir Fraser]

On 14/8/06 6:48 pm, "Ryan Harper" <ryanh@us.ibm.com> wrote:

>> 
>> I'm not sure about the use of ', ' as a delimiter. It would be less
>> confusing to strictly require the use of the list form. I would imagine it's
>> then clearest used as:
>> Cpus = []
>> Cpus[0] = '2-5'
>> Cpus[1] = '3-4'
>> ...
> 
> I'd prefer to not have to use any language arrays.  I'm not sure how
> this will map to the xml-based config files that Ewan was talking about,
> but I suppose he will have to come up with something since we have
> things like the disk parameter which is in python list format.

I think that'll be a different format again. Perhaps a separate cpu-info
section of the file for each VCPU, within which we specify the affinity
info. I'm pretty sure we won't go for a single string with whitespace
delimiters!

Anyhow, the current patch does have flexibility in its favour, since it
supports the list format *and* the string format. So I guess it's okay
really.

 -- Keir