XEN Proposal

XEN Proposal

[Juergen Gross]

Hi,

Currently the XEN credit scheduler has its pitfalls in supporting weights of
domains together with cpu pinning (see the threads
http://lists.xensource.com/archives/html/xen-devel/2007-02/msg00006.html
http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00365.html
http://lists.xensource.com/archives/html/xen-devel/2007-07/msg00303.html
which include a rejected patch).

We are facing this problem, too. We tried the above patch, but it didn't solve
our problem completely, so we decided to start a new solution.

Our basic requirement is to limit a set of domains to a set of physical cpus
while specifying the scheduling weight for each domain. The general (and in my
opinion best) solution would be the introduction of a "pool" concept in XEN.

Each physical cpu is dedicated to exactly one pool. At start of XEN this is
pool0. A domain is member of a single pool (dom0 will always be member of
pool0), there may be several domains in one pool. Scheduling does not cross
pool boundaries, so the weight of a domain is only related to the weight of
the other domains in the same pool. So it is possible to have an own scheduler
for each pool.

What changes would be needed?
- The hypervisor must be pool-aware. It needs information about the pool
  configuration (cpu mask, scheduler) and the pool membership of a domain.
  The scheduler must restrict itself to its own pool only.
- There must be an interface to set and query the pool configuration.
- At domain creation the domain must be added to a pool.
- libxc must be expanded to support the new interfaces.
- xend and the xm command must support pools, defaulting to pool0 if no pool
  is specified

The xm commands could look like this:
xm pool-create pool1 ncpu=4              # create a pool with 4 cpus
xm pool-create pool2 cpu=1,3,5           # create a pool with 3 dedicated cpus
xm pool-list                             # show pools:
  pool      cpus          sched      domains
  pool0     0,2,4         credit     0
  pool1     6-9           credit     1,7
  pool2     1,3,5         credit     2,3
xm pool-modify pool1 ncpu=3              # set new number of cpus
xm pool-modify pool1 cpu=6,7,9           # modify cpu-pinning
xm pool-destroy pool1                    # destroy pool
xm create vm5 pool=pool1                 # start domain in pool1

There is much more potential in this approach:
- add memory to a pool? Could be interesting for NUMA
- recent discussions on xen-devel related to scheduling (credit scheduler for
  client virtualization) show some demand for further work regarding priority
  and/or grouping of domains
- this might be an interesting approach for migration of multiple related
  domains (pool migration)
- move (or migrate?) a domain to another pool
- ...

Any comments, suggestions, work already done, ...?
Otherwise we will be starting our effort soon.

Juergen

-- 
Juergen Gross                             Principal Developer
IP SW OS6                      Telephone: +49 (0) 89 636 47950
Fujitsu Siemens Computers         e-mail: juergen.gross@fujitsu-siemens.com
Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
D-81739 Muenchen         Company details: www.fujitsu-siemens.com/imprint.html

RE: XEN Proposal

[Tian, Kevin]

I remember seeing some post before about domain group scheduler.
Not sure about its progress now, and maybe you can check that
thread to see anything useful?

Thanks,
Kevin

>From:Juergen Gross
>Sent: Wednesday, December 10, 2008 9:10 PM
>
>Hi,
>
>Currently the XEN credit scheduler has its pitfalls in 
>supporting weights of
>domains together with cpu pinning (see the threads
>http://lists.xensource.com/archives/html/xen-devel/2007-02/msg0
>0006.html
>http://lists.xensource.com/archives/html/xen-devel/2006-10/msg0
>0365.html
>http://lists.xensource.com/archives/html/xen-devel/2007-07/msg0
>0303.html
>which include a rejected patch).
>
>We are facing this problem, too. We tried the above patch, but 
>it didn't solve
>our problem completely, so we decided to start a new solution.
>
>Our basic requirement is to limit a set of domains to a set of 
>physical cpus
>while specifying the scheduling weight for each domain. The 
>general (and in my
>opinion best) solution would be the introduction of a "pool" 
>concept in XEN.
>
>Each physical cpu is dedicated to exactly one pool. At start 
>of XEN this is
>pool0. A domain is member of a single pool (dom0 will always 
>be member of
>pool0), there may be several domains in one pool. Scheduling 
>does not cross
>pool boundaries, so the weight of a domain is only related to 
>the weight of
>the other domains in the same pool. So it is possible to have 
>an own scheduler
>for each pool.
>
>What changes would be needed?
>- The hypervisor must be pool-aware. It needs information 
>about the pool
>  configuration (cpu mask, scheduler) and the pool membership 
>of a domain.
>  The scheduler must restrict itself to its own pool only.
>- There must be an interface to set and query the pool configuration.
>- At domain creation the domain must be added to a pool.
>- libxc must be expanded to support the new interfaces.
>- xend and the xm command must support pools, defaulting to 
>pool0 if no pool
>  is specified
>
>The xm commands could look like this:
>xm pool-create pool1 ncpu=4              # create a pool with 4 cpus
>xm pool-create pool2 cpu=1,3,5           # create a pool with 
>3 dedicated cpus
>xm pool-list                             # show pools:
>  pool      cpus          sched      domains
>  pool0     0,2,4         credit     0
>  pool1     6-9           credit     1,7
>  pool2     1,3,5         credit     2,3
>xm pool-modify pool1 ncpu=3              # set new number of cpus
>xm pool-modify pool1 cpu=6,7,9           # modify cpu-pinning
>xm pool-destroy pool1                    # destroy pool
>xm create vm5 pool=pool1                 # start domain in pool1
>
>There is much more potential in this approach:
>- add memory to a pool? Could be interesting for NUMA
>- recent discussions on xen-devel related to scheduling 
>(credit scheduler for
>  client virtualization) show some demand for further work 
>regarding priority
>  and/or grouping of domains
>- this might be an interesting approach for migration of 
>multiple related
>  domains (pool migration)
>- move (or migrate?) a domain to another pool
>- ...
>
>Any comments, suggestions, work already done, ...?
>Otherwise we will be starting our effort soon.
>
>Juergen
>
>-- 
>Juergen Gross                             Principal Developer
>IP SW OS6                      Telephone: +49 (0) 89 636 47950
>Fujitsu Siemens Computers         e-mail: 
>juergen.gross@fujitsu-siemens.com
>Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
>D-81739 Muenchen         Company details: 
>www.fujitsu-siemens.com/imprint.html
>
>
>
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>

Re: XEN Proposal

[Keir Fraser]

That was grouping domains to directly share scheduling credits, rather than
grouping to share physical resources.

 -- Keir

On 10/12/2008 13:21, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> I remember seeing some post before about domain group scheduler.
> Not sure about its progress now, and maybe you can check that
> thread to see anything useful?
> 
> Thanks,
> Kevin
> 
>> From:Juergen Gross
>> Sent: Wednesday, December 10, 2008 9:10 PM
>> 
>> Hi,
>> 
>> Currently the XEN credit scheduler has its pitfalls in
>> supporting weights of
>> domains together with cpu pinning (see the threads
>> http://lists.xensource.com/archives/html/xen-devel/2007-02/msg0
>> 0006.html
>> http://lists.xensource.com/archives/html/xen-devel/2006-10/msg0
>> 0365.html
>> http://lists.xensource.com/archives/html/xen-devel/2007-07/msg0
>> 0303.html
>> which include a rejected patch).
>> 
>> We are facing this problem, too. We tried the above patch, but
>> it didn't solve
>> our problem completely, so we decided to start a new solution.
>> 
>> Our basic requirement is to limit a set of domains to a set of
>> physical cpus
>> while specifying the scheduling weight for each domain. The
>> general (and in my
>> opinion best) solution would be the introduction of a "pool"
>> concept in XEN.
>> 
>> Each physical cpu is dedicated to exactly one pool. At start
>> of XEN this is
>> pool0. A domain is member of a single pool (dom0 will always
>> be member of
>> pool0), there may be several domains in one pool. Scheduling
>> does not cross
>> pool boundaries, so the weight of a domain is only related to
>> the weight of
>> the other domains in the same pool. So it is possible to have
>> an own scheduler
>> for each pool.
>> 
>> What changes would be needed?
>> - The hypervisor must be pool-aware. It needs information
>> about the pool
>>  configuration (cpu mask, scheduler) and the pool membership
>> of a domain.
>>  The scheduler must restrict itself to its own pool only.
>> - There must be an interface to set and query the pool configuration.
>> - At domain creation the domain must be added to a pool.
>> - libxc must be expanded to support the new interfaces.
>> - xend and the xm command must support pools, defaulting to
>> pool0 if no pool
>>  is specified
>> 
>> The xm commands could look like this:
>> xm pool-create pool1 ncpu=4              # create a pool with 4 cpus
>> xm pool-create pool2 cpu=1,3,5           # create a pool with
>> 3 dedicated cpus
>> xm pool-list                             # show pools:
>>  pool      cpus          sched      domains
>>  pool0     0,2,4         credit     0
>>  pool1     6-9           credit     1,7
>>  pool2     1,3,5         credit     2,3
>> xm pool-modify pool1 ncpu=3              # set new number of cpus
>> xm pool-modify pool1 cpu=6,7,9           # modify cpu-pinning
>> xm pool-destroy pool1                    # destroy pool
>> xm create vm5 pool=pool1                 # start domain in pool1
>> 
>> There is much more potential in this approach:
>> - add memory to a pool? Could be interesting for NUMA
>> - recent discussions on xen-devel related to scheduling
>> (credit scheduler for
>>  client virtualization) show some demand for further work
>> regarding priority
>>  and/or grouping of domains
>> - this might be an interesting approach for migration of
>> multiple related
>>  domains (pool migration)
>> - move (or migrate?) a domain to another pool
>> - ...
>> 
>> Any comments, suggestions, work already done, ...?
>> Otherwise we will be starting our effort soon.
>> 
>> Juergen
>> 
>> -- 
>> Juergen Gross                             Principal Developer
>> IP SW OS6                      Telephone: +49 (0) 89 636 47950
>> Fujitsu Siemens Computers         e-mail:
>> juergen.gross@fujitsu-siemens.com
>> Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
>> D-81739 Muenchen         Company details:
>> www.fujitsu-siemens.com/imprint.html
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: XEN Proposal

[Juergen Gross]

Tian, Kevin wrote:
> I remember seeing some post before about domain group scheduler.
> Not sure about its progress now, and maybe you can check that
> thread to see anything useful?

Thanks for that hint. As you are mentioning it, I remember, too.
Group scheduling is a little bit different, but the patches contain very
useful stuff for hypercall, tools etc. which will make our work easier.

Juergen

-- 
Juergen Gross                             Principal Developer
IP SW OS6                      Telephone: +49 (0) 89 636 47950
Fujitsu Siemens Computers         e-mail: juergen.gross@fujitsu-siemens.com
Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
D-81739 Muenchen         Company details: www.fujitsu-siemens.com/imprint.html

Re: XEN Proposal

[Chris]

[-- Attachment #1.1: Type: text/plain, Size: 5872 bytes --]

Grouping domains for the purpose of sharing scheduler credits was the  
intent of the sched-groups project done by Mike Day at IBM:

http://xen.markmail.org/search/?q=sched-groups&x=0&y=0#query:sched- 
groups%20from%3A%22Mike%20D.%20Day%22+page:1+mid:k36x7tqjy3zquqv7+state:results) 
.

A related project called Domain Groups (domgrps for short) also exists:

http://xen.markmail.org/search/?q=%22domain+group%22#query:%22domain 
%20group%22%20from%3AChris%20date%3A200702%20+page:1+mid:b525tcfzvfgv3gzo+state:results

The primary difference is that domgrps is a general-purpose domain  
grouping framework with integration into both the VMM and userspace  
tools.  It does group migration, moving domains b/w groups, etc..

To demonstrate the flexibility of the domgrps architecture, I merged  
domgrps and schedgrps:

http://xen.markmail.org/search/?q=domgrps+schedgrps#query:domgrps  
schedgrps from%3A"Chris"+page:1+mid:rvetcmnmm7bkgexp+state:results

It should be equally straightforward to adapt domgrps to accommodate  
this newly proposed pooling concept.  I'd be happy to get involved if  
Keir suggests which grouping features are of interest.  If we first  
need a discussion to define the potential grouping features that's OK  
too, but I'll delay that effort until I'm asked.

Cheers
-Chris


On Dec 10, 2008, at 8:34 AM, Keir Fraser wrote:

> That was grouping domains to directly share scheduling credits,  
> rather than
> grouping to share physical resources.
>
> -- Keir
>
> On 10/12/2008 13:21, "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
>> I remember seeing some post before about domain group scheduler.
>> Not sure about its progress now, and maybe you can check that
>> thread to see anything useful?
>>
>> Thanks,
>> Kevin
>>
>>> From:Juergen Gross
>>> Sent: Wednesday, December 10, 2008 9:10 PM
>>>
>>> Hi,
>>>
>>> Currently the XEN credit scheduler has its pitfalls in
>>> supporting weights of
>>> domains together with cpu pinning (see the threads
>>> http://lists.xensource.com/archives/html/xen-devel/2007-02/msg0
>>> 0006.html
>>> http://lists.xensource.com/archives/html/xen-devel/2006-10/msg0
>>> 0365.html
>>> http://lists.xensource.com/archives/html/xen-devel/2007-07/msg0
>>> 0303.html
>>> which include a rejected patch).
>>>
>>> We are facing this problem, too. We tried the above patch, but
>>> it didn't solve
>>> our problem completely, so we decided to start a new solution.
>>>
>>> Our basic requirement is to limit a set of domains to a set of
>>> physical cpus
>>> while specifying the scheduling weight for each domain. The
>>> general (and in my
>>> opinion best) solution would be the introduction of a "pool"
>>> concept in XEN.
>>>
>>> Each physical cpu is dedicated to exactly one pool. At start
>>> of XEN this is
>>> pool0. A domain is member of a single pool (dom0 will always
>>> be member of
>>> pool0), there may be several domains in one pool. Scheduling
>>> does not cross
>>> pool boundaries, so the weight of a domain is only related to
>>> the weight of
>>> the other domains in the same pool. So it is possible to have
>>> an own scheduler
>>> for each pool.
>>>
>>> What changes would be needed?
>>> - The hypervisor must be pool-aware. It needs information
>>> about the pool
>>> configuration (cpu mask, scheduler) and the pool membership
>>> of a domain.
>>> The scheduler must restrict itself to its own pool only.
>>> - There must be an interface to set and query the pool  
>>> configuration.
>>> - At domain creation the domain must be added to a pool.
>>> - libxc must be expanded to support the new interfaces.
>>> - xend and the xm command must support pools, defaulting to
>>> pool0 if no pool
>>> is specified
>>>
>>> The xm commands could look like this:
>>> xm pool-create pool1 ncpu=4              # create a pool with 4 cpus
>>> xm pool-create pool2 cpu=1,3,5           # create a pool with
>>> 3 dedicated cpus
>>> xm pool-list                             # show pools:
>>> pool      cpus          sched      domains
>>> pool0     0,2,4         credit     0
>>> pool1     6-9           credit     1,7
>>> pool2     1,3,5         credit     2,3
>>> xm pool-modify pool1 ncpu=3              # set new number of cpus
>>> xm pool-modify pool1 cpu=6,7,9           # modify cpu-pinning
>>> xm pool-destroy pool1                    # destroy pool
>>> xm create vm5 pool=pool1                 # start domain in pool1
>>>
>>> There is much more potential in this approach:
>>> - add memory to a pool? Could be interesting for NUMA
>>> - recent discussions on xen-devel related to scheduling
>>> (credit scheduler for
>>> client virtualization) show some demand for further work
>>> regarding priority
>>> and/or grouping of domains
>>> - this might be an interesting approach for migration of
>>> multiple related
>>> domains (pool migration)
>>> - move (or migrate?) a domain to another pool
>>> - ...
>>>
>>> Any comments, suggestions, work already done, ...?
>>> Otherwise we will be starting our effort soon.
>>>
>>> Juergen
>>>
>>> -- 
>>> Juergen Gross                             Principal Developer
>>> IP SW OS6                      Telephone: +49 (0) 89 636 47950
>>> Fujitsu Siemens Computers         e-mail:
>>> juergen.gross@fujitsu-siemens.com
>>> Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
>>> D-81739 Muenchen         Company details:
>>> www.fujitsu-siemens.com/imprint.html
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


[-- Attachment #1.2: Type: text/html, Size: 16652 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: XEN Proposal

[Juergen Gross]

Chris wrote:
> Grouping domains for the purpose of sharing scheduler credits was the
> intent of the sched-groups project done by Mike Day at IBM:
> 
> http://xen.markmail.org/search/?q=sched-groups&x=0&y=0#query
> <http://xen.markmail.org/search/?q=sched-groups&x=0&y=0#query>:sched-groups%20from%3A%22Mike%20D.%20Day%22+page:1+mid:k36x7tqjy3zquqv7+state:results).
>  
> 
> A related project called Domain Groups (domgrps for short) also exists:
> 
> http://xen.markmail.org/search/?q=%22domain+group%22#query:%22domain%20group%22%20from%3AChris%20date%3A200702%20+page:1+mid:b525tcfzvfgv3gzo+state:results
> 
> The primary difference is that domgrps is a general-purpose domain
> grouping framework with integration into both the VMM and userspace
> tools.  It does group migration, moving domains b/w groups, etc..  
> 
> To demonstrate the flexibility of the domgrps architecture, I merged
> domgrps and schedgrps:
> 
> http://xen.markmail.org/search/?q=domgrps+schedgrps#query:domgrps
> schedgrps from%3A"Chris"+page:1+mid:rvetcmnmm7bkgexp+state:results
> 
> It should be equally straightforward to adapt domgrps to accommodate
> this newly proposed pooling concept.  I'd be happy to get involved if
> Keir suggests which grouping features are of interest.  If we first need
> a discussion to define the potential grouping features that's OK too,
> but I'll delay that effort until I'm asked.  

Chris, thanks for your support.
I think Keir's main interest is adding only features that are really needed.
So let me explain WHY we want to implement a pool concept:

We have a mainframe operating system called BS2000 originally designed for
/390 machines. As our customers often require small machines (software
prices are based on machine power) we decided some years ago to port BS2000
to other architectures as /390 machines are rather expensive.
We did ports to MIPS (we had MIPS UNIX machines in these times), to SPARC
(after our UNIX strategy switched to SPARC/Solaris), and now to x86_64.
We always had a virtual machine concept in BS2000 called VM2000 which is
quite important for our customers.
On x86_64 we decided to use XEN as base for our virtual machines to be able
to share the hardware with other systems as well.
The software prices for BS2000 will be still related to the machine power.
But often customers need only a small portion of the complete x86 machine
power for BS2000, so we added a license scheme to limit the power available
to BS2000 by pinning the domains with BS2000 to a subset of cpus.

BTW: on /390 and SPARC we already support VM-pools, so adding pools to XEN
would make this feature available to our customers who are still using it
today on other platforms.


Juergen

-- 
Juergen Gross                             Principal Developer
IP SW OS6                      Telephone: +49 (0) 89 636 47950
Fujitsu Siemens Computers         e-mail: juergen.gross@fujitsu-siemens.com
Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
D-81739 Muenchen         Company details: www.fujitsu-siemens.com/imprint.html

Re: XEN Proposal

[George Dunlap]

On Thu, Dec 11, 2008 at 6:23 AM, Juergen Gross
<juergen.gross@fujitsu-siemens.com> wrote:
>> The software prices for BS2000 will be still related to the machine power.
> But often customers need only a small portion of the complete x86 machine
> power for BS2000, so we added a license scheme to limit the power available
> to BS2000 by pinning the domains with BS2000 to a subset of cpus.

OK, so... you sell software, and the cost of it depends on how
powerful a machine you run it on.  But most people don't need even one
full core's worth of power.  You want to be able to charge people who
need a full core of processing power one price, and charge people who
need only say, 20%, less?  So to artificially limit the power
available to a given instance, you pin all instances to some subset of
cpus?

I presume then, that there are multiple instances, and people pay for
how many cores they can use total...?  And that your customers
generally have other things running on the server as well.

It seems like what you really want is to cap the number of credits the
VMs can get, and then take them offline when their credits go negative
(instead of competing for resources in a "best-effort" fashion).  :-)

However, it does seem like being able to partition up a Xen server
into "pools" of cpu resources, each with its own scheduler, that don't
compete with each other, might be generally useful.  It should be
relatively straightforward to slide in under the current scheduler
architecture, without having to change much in the schedulers
themselves.  That's how I'd prefer it done, if possible: a clean layer
underneath a scheduler.

 -George

>
> BTW: on /390 and SPARC we already support VM-pools, so adding pools to XEN
> would make this feature available to our customers who are still using it
> today on other platforms.
>
>
> Juergen
>
> --
> Juergen Gross                             Principal Developer
> IP SW OS6                      Telephone: +49 (0) 89 636 47950
> Fujitsu Siemens Computers         e-mail: juergen.gross@fujitsu-siemens.com
> Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
> D-81739 Muenchen         Company details: www.fujitsu-siemens.com/imprint.html
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: XEN Proposal

[George Dunlap]

On Thu, Dec 11, 2008 at 3:13 PM, George Dunlap <dunlapg@umich.edu> wrote:
> It seems like what you really want is to cap the number of credits the
> VMs can get, and then take them offline when their credits go negative
> (instead of competing for resources in a "best-effort" fashion).  :-)

Actually, it turns out this functionality is already present.

xm sched-credit -d [domid] -c 30

will "cap" the domain to 30% of one cpu, even if there are no other
cpus running.  You should be able to utilize that directly, no
development needed.

 -George

Re: XEN Proposal

[Juergen Gross]

George Dunlap wrote:
> On Thu, Dec 11, 2008 at 6:23 AM, Juergen Gross
> <juergen.gross@fujitsu-siemens.com> wrote:
>>> The software prices for BS2000 will be still related to the machine power.
>> But often customers need only a small portion of the complete x86 machine
>> power for BS2000, so we added a license scheme to limit the power available
>> to BS2000 by pinning the domains with BS2000 to a subset of cpus.
> 
> OK, so... you sell software, and the cost of it depends on how
> powerful a machine you run it on.  But most people don't need even one
> full core's worth of power.  You want to be able to charge people who
> need a full core of processing power one price, and charge people who
> need only say, 20%, less?  So to artificially limit the power
> available to a given instance, you pin all instances to some subset of
> cpus?

More or less.
But the processors of our servers are 6-core. A customer might want to pay
for only some cores of power.

> 
> I presume then, that there are multiple instances, and people pay for
> how many cores they can use total...?  And that your customers
> generally have other things running on the server as well.

Correct.

> 
> It seems like what you really want is to cap the number of credits the
> VMs can get, and then take them offline when their credits go negative
> (instead of competing for resources in a "best-effort" fashion).  :-)

In theory, yes.
But if a customer has a license for the power of 3 cores for BS2000, he
should be able to run for example 4 BS2000 domains which must not consume
more power in sum, but if 3 domains are more or less idle, the remaining
domain could take all 3 cores.
I don't think this is an easy job with caps.

> 
> However, it does seem like being able to partition up a Xen server
> into "pools" of cpu resources, each with its own scheduler, that don't
> compete with each other, might be generally useful.  It should be
> relatively straightforward to slide in under the current scheduler
> architecture, without having to change much in the schedulers
> themselves.  That's how I'd prefer it done, if possible: a clean layer
> underneath a scheduler.

That is the direction I would go.


Juergen

-- 
Juergen Gross                             Principal Developer
IP SW OS6                      Telephone: +49 (0) 89 636 47950
Fujitsu Siemens Computers         e-mail: juergen.gross@fujitsu-siemens.com
Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
D-81739 Muenchen         Company details: www.fujitsu-siemens.com/imprint.html

Re: XEN Proposal

[Chris]

>> However, it does seem like being able to partition up a Xen server
>> into "pools" of cpu resources, each with its own scheduler, that  
>> don't
>> compete with each other, might be generally useful.  It should be
>> relatively straightforward to slide in under the current scheduler
>> architecture, without having to change much in the schedulers
>> themselves.  That's how I'd prefer it done, if possible: a clean  
>> layer
>> underneath a scheduler.
>
> That is the direction I would go

Agreed; I do not mean to suggest otherwise.

My point is that if this project and others all implement their own  
management tools and inevitably conflicting means of storing  
information about of sets of domains, then that's not a path to a  
happy place.

I also recognize that I'm trying to head off a problem before it  
exists, which may not be the way to go.

If you decide that your needs are best served by developing a new set  
of tools, please keep the discussion alive on the list.  It's a  
potentially useful capability and I'd like to see how it progresses.

Regards
-Chris

Re: XEN Proposal

[Juergen Gross]

Chris wrote:
> Agreed; I do not mean to suggest otherwise.
> 
> My point is that if this project and others all implement their own
> management tools and inevitably conflicting means of storing information
> about of sets of domains, then that's not a path to a happy place.

You are absolutely right!

> If you decide that your needs are best served by developing a new set of
> tools, please keep the discussion alive on the list.  It's a potentially
> useful capability and I'd like to see how it progresses.

Okay, here is our proposal for the user interface. We plan to do the
implementation work in the next month(s).


Support of cpu-pools

The physical cpus of a server are grouped in cpu-pools. Each physical CPU is
assigned at most to one cpu-pool. Domains are each restricted to a single
cpu-pool. Scheduling does not cross cpu-pool boundaries, so each cpu-pool
has an own scheduler.

The cpu-pool of the domain Dom0 will be pool0, at boot-time we have to create
this cpu-pool implicitly.

The creation of a cpu-pool is similar to the creation of a domain. In a first
step the attributes of a cpu-pool have to be defined, in the second step the
cpu-pool will be activated (started) by assigning the resources (CPUs) and
initializing the scheduler. An activated cpu-pool owns at least one physical
CPU.

A physical CPU may be removed from an activated cpu-pool. Further CPUs may be
assigned to the cpu-pool. The CPU may be assigned to a cpu-pool only if that
CPU is free (not assigned to any other activated cpu-pool). The free CPUs are
internally managed.

The poolname will be added as an additional parameter to the configfile of a
domain. If no poolname is specified the poolname is defaults to pool0. At
domain start the domain is added to its cpu-pool, which must be activated.

A started domain can be moved to another activated cpu-pool.

A cpu-pool may be deactivated if there is no active domain assigned to it. A
cpu-pool is deactivated by removing the resources (CPUs).

A cpu-pool may be deleted if it is deactivated.

The administration of the cpu-pools is done by xm subcommands.

new xm subcommands:


xm pool-new  poolconfigfile [name=value]..

           define a cpu-pool.

           The pool-new sub command requires a config file and can optionally
           take a series of name value pairs that add to or override variables
           defined in the config file.  See poolconfigfile description.

           poolconfigfile can either be an absolute path to a file, or a
           relative path to a file located in /etc/xen/pool.

xm pool-start <poolname>

           Activate a Xend managed cpu-pool.
           (The cpu-pool must be defined by xm pool-new (or by API). The
           cpu-pool will be activated only if the requested CPUs are
           available.)

xm pool-create   poolconfigfile [name=value]..

           define a new cpu-pool and activate it.

xm pool-list  [options] [poolname,...]

           List information about all/some cpu-pools.

           -l|--long       Output all cpu-pool details in SXP

           output format for deactivated cpu-pools without option --long

            cpu-pool     cpus         sched    domains
            <poolname>   <# CPUs>     <sched>

           output format for activated cpu-pools without option --long

            cpu-pool     cpus            sched    domains
            <poolname>   <list of CPUs>  <sched>  <list of active domains>

           output format with option --long

            (pool
              (name    <poolname>)
              (uuid    <uuid>)
              (state   <activated|deactivated>)
              (cpun    <number of CPUs>)
              (cpus    <list of CPUs>)
              (sched   <sched>)
              (domains <list of started domains running in the cpu-pool>)
            )

xm pool-destroy <poolname>

           deactivate a cpu-pool.
           (Unassign the CPUs of the cpu-pool. Delete the cpu-pool
           if it was created by xm pool-create.
           A cpu-pool may be deactivated only if there are no active
           domains added to it.)

xm pool-delete <poolname>

           Delete the cpu-pool.
           (deletion is only possible if the cpu-pool is deactivated.)

xm pool-cpu-add <poolname> <cpu>

           Add a free cpu to the activated cpu-pool.

xm pool-cpu-remove <poolname> <cpu>

           Remove a cpu from the activated cpu-pool.
           (subsequent the removed CPU may be added to another cpu-pool.
           The last CPU of the cpu-pool cannot be removed explicitly.)

xm pool-migrate <domain> <poolname>

           move the domain to the specified cpu-pool.

extended subcommands

xm list ... [--pool=<poolname>]

           select only domains added to the cpu-pool.

xm list --long|-l

           additional info:
             (cpu_pool <poolname>)

xm info

           the line
            xen_scheduler   : <sched>
           shows the scheduler of pool0.

           additional info:
            free_cpus  : <n>
           (number of free CPUs)


xen pool configfile

       pool configuration files live in /etc/xen/pool by default.  If you
       store config files anywhere else the full path to the config file must
       be specified in the xm pool-create command.

  supported options:

    name
           A unique name for the cpu-pool.  Attempting to create two cpu-pools
           with the same name will cause an error.

    sched
           Name of scheduler (Default: credit)

    cpus
           List of CPUs to be activated                or
           Number of CPUs to be activated (Default: 1)

    other_config
           ["<param> = <value>" ...]


xen domain configfile

  additional option:

    pool
            Name of cpu-pool the domain is added to.
            xm create or xm start is rejected if the cpu-pool is not
            activated.

            For pinning only CPUs of the cpu-pool are taken into account.

Autostart of cpu-pools and domains

Before starting a domain automatically its cpu-pool has to be started.

By default all CPUs are assigned to the cpu-pool pool0.

As for automatically starting a cpu-pool free CPUs are needed, a new boot
parameter in /boot/grub/menu.lst is required to create pool0 with less CPUs.

To specify a cpu-pool is to be started at boot-time, its configfile (or a link
to it) has to be placed in /etc/xen/pool/auto/. The xendomains script for
automatically starting domains has to be extended. Before starting the domains
the autostart of the cpu-pools is done.


Juergen

-- 
Juergen Gross                             Principal Developer
IP SW OS6                      Telephone: +49 (0) 89 636 47950
Fujitsu Siemens Computers         e-mail: juergen.gross@fujitsu-siemens.com
Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
D-81739 Muenchen         Company details: www.fujitsu-siemens.com/imprint.html