XCP: dom0 scalability

XCP: dom0 scalability

[Dave Scott]

Hi,

I've added a draft of a doc called "the dom0 capacity crunch" to the wiki:

http://wiki.xensource.com/xenwiki/XCP_Overview?action=AttachFile&do=get&target=xcp_dom0_capacity_crunch.pdf

The doc contains a proposal for dealing with ever-increasing dom0 load, primarily by moving the load out of dom0 (stubdoms, helper domains etc) and, where still necessary, tweaking the number of dom0 vcpus. I think this is becoming pretty important and we'll need to work on this asap.

Comments are welcome.

Cheers,
Dave

Re: XCP: dom0 scalability

[Pasi Kärkkäinen]

On Fri, Oct 08, 2010 at 08:54:25PM +0100, Dave Scott wrote:
> Hi,
> 
> I've added a draft of a doc called "the dom0 capacity crunch" to the wiki:
> 
> http://wiki.xensource.com/xenwiki/XCP_Overview?action=AttachFile&do=get&target=xcp_dom0_capacity_crunch.pdf
> 
> The doc contains a proposal for dealing with ever-increasing dom0 load, primarily by moving the load out of dom0 (stubdoms, helper domains etc) and, where still necessary, tweaking the number of dom0 vcpus. I think this is becoming pretty important and we'll need to work on this asap.
> 
> Comments are welcome.
> 

All that makes sense to me.

I think oxenstored already has (some) support for running in stubdom,
qemu-dm already supports stubdoms, driver-domains should work..

This stuff will require a lof of testing..

-- Pasi

Re: Re: [Xen-devel] XCP: dom0 scalability

[Thomas.Gazagnaire-MZpvjPyXg2s]

> I think oxenstored already has (some) support for running in stubdom,
the difficult part for oxenstored if I remember correctly is the way xapi
communicates with it, ie. using fds. Don't know if it is easy or not to
replace them with ring buffers ... But Vincent should know :-)

> qemu-dm already supports stubdoms, driver-domains should work..
>
> This stuff will require a lof of testing..
>
> -- Pasi
>
>
> _______________________________________________
> xen-api mailing list
> xen-api-GuqFBffKawuULHF6PoxzQEEOCMrvLtNR@public.gmane.org
> http://lists.xensource.com/mailman/listinfo/xen-api
>

RE: Re: [Xen-devel] XCP: dom0 scalability

[Dave Scott]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]


Hi Thomas, Pasi,

Thomas wrote:
> I think oxenstored already has (some) support for running in stubdom,
> the difficult part for oxenstored if I remember correctly is the way
> xapi
> communicates with it, ie. using fds. Don't know if it is easy or not to
> replace them with ring buffers ... But Vincent should know :-)

IIRC there's a function 'Xc.daemon_open' which uses a Unix.file_descr and a function 'Xc.domain_open' which uses the ring buffers. I think we'd just need to redefine our 'with_xs' helper function to use 'Xc.domain_open'?

Pasi wrote:
> > qemu-dm already supports stubdoms, driver-domains should work..
> >
> > This stuff will require a lof of testing..

It'll be a lot of fun :)

Cheers,
Dave


[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [Xen-API] XCP: dom0 scalability

[George Shuklin]

This seems rational, but what about overhead? Every domain will require
own kernel with memory (main concern about memory).

I think, first step shall be splitting xapi to few separate daemons.

В Птн, 08/10/2010 в 20:54 +0100, Dave Scott пишет:
> Hi,
> 
> I've added a draft of a doc called "the dom0 capacity crunch" to the wiki:
> 
> http://wiki.xensource.com/xenwiki/XCP_Overview?action=AttachFile&do=get&target=xcp_dom0_capacity_crunch.pdf
> 
> The doc contains a proposal for dealing with ever-increasing dom0 load, primarily by moving the load out of dom0 (stubdoms, helper domains etc) and, where still necessary, tweaking the number of dom0 vcpus. I think this is becoming pretty important and we'll need to work on this asap.
> 
> Comments are welcome.
> 
> Cheers,
> Dave
> 
> _______________________________________________
> xen-api mailing list
> xen-api@lists.xensource.com
> http://lists.xensource.com/mailman/listinfo/xen-api

Re: XCP: dom0 scalability

[Jeremy Fitzhardinge]

 On 10/08/2010 12:54 PM, Dave Scott wrote:
> I've added a draft of a doc called "the dom0 capacity crunch" to the wiki:
>
> http://wiki.xensource.com/xenwiki/XCP_Overview?action=AttachFile&do=get&target=xcp_dom0_capacity_crunch.pdf
>
> The doc contains a proposal for dealing with ever-increasing dom0 load, primarily by moving the load out of dom0 (stubdoms, helper domains etc) and, where still necessary, tweaking the number of dom0 vcpus. I think this is becoming pretty important and we'll need to work on this asap.

I'm a big fan of disaggregation, but from a performance perspective
xenstored seems pretty insignificant.

    J

Re: XCP: dom0 scalability

[Tim Deegan]

At 00:16 +0100 on 13 Oct (1286928975), Jeremy Fitzhardinge wrote:
> I'm a big fan of disaggregation, but from a performance perspective
> xenstored seems pretty insignificant.

True, xenstored is more a victim of dom0 load than a cause of it.  We've
seen some pretty nasty performance when trying to boot lots of VMs with
a loaded dom0; PV drivers in the VMs could block for many seconds
waiting for xenstore to get back to them.  It still makes sense to get
it out of there.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

RE: XCP: dom0 scalability

[Dan Magenheimer]

> At 00:16 +0100 on 13 Oct (1286928975), Jeremy Fitzhardinge wrote:
> > I'm a big fan of disaggregation, but from a performance perspective
> > xenstored seems pretty insignificant.
> 
> True, xenstored is more a victim of dom0 load than a cause of it.
> We've
> seen some pretty nasty performance when trying to boot lots of VMs with
> a loaded dom0; PV drivers in the VMs could block for many seconds
> waiting for xenstore to get back to them.  It still makes sense to get
> it out of there.

Hmmm... if dom0 is getting scheduled enough and the VMs are still
blocking on xenstore (i.e. the Linux scheduler in dom0 is the problem),
getting xenstore off of dom0 makes sense.  But if the reason xenstore
is not getting back to them is because dom0 is simply not getting
enough cpu time (i.e. the Xen scheduler is the problem), moving xenstore
out of there may not only NOT help, but may make things worse.  True?

Re: XCP: dom0 scalability

[Tim Deegan]

At 16:46 +0100 on 13 Oct (1286988381), Dan Magenheimer wrote:
> Hmmm... if dom0 is getting scheduled enough and the VMs are still
> blocking on xenstore (i.e. the Linux scheduler in dom0 is the problem),
> getting xenstore off of dom0 makes sense.  But if the reason xenstore
> is not getting back to them is because dom0 is simply not getting
> enough cpu time (i.e. the Xen scheduler is the problem), moving xenstore
> out of there may not only NOT help, but may make things worse.  True?

The reason that dom0's not getting enough time in this case is that it's
running with only 1 vcpu and trying to do more than 1 vcpu worth of
work.  I'd expect things to get very slightly better if some of that
work can happen on another vpcu (in another dom).

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: XCP: dom0 scalability

[Jeremy Fitzhardinge]

 On 10/13/2010 09:07 AM, Tim Deegan wrote:
> At 16:46 +0100 on 13 Oct (1286988381), Dan Magenheimer wrote:
>> Hmmm... if dom0 is getting scheduled enough and the VMs are still
>> blocking on xenstore (i.e. the Linux scheduler in dom0 is the problem),
>> getting xenstore off of dom0 makes sense.  But if the reason xenstore
>> is not getting back to them is because dom0 is simply not getting
>> enough cpu time (i.e. the Xen scheduler is the problem), moving xenstore
>> out of there may not only NOT help, but may make things worse.  True?
> The reason that dom0's not getting enough time in this case is that it's
> running with only 1 vcpu and trying to do more than 1 vcpu worth of
> work.  I'd expect things to get very slightly better if some of that
> work can happen on another vpcu (in another dom).

Or just make dom0 multi-vcpu?

    J