Re: Re: Xen scheduler

Re: Re: Xen scheduler

[pak333]

[-- Attachment #1.1: Type: text/plain, Size: 2293 bytes --]

Thanks. A little more clarification.

Here is an example. 

I have multiple VMs, each with 2 vcpus. There is no user affinity. So i will let the vcpus run whereever the Xen scheduler chooses. My system has 2 dual core sockets.

     If all 4 pcpus are idle, then will the scheduler assign the vcpus of a VM to the same socket pcpus. 
     If while running, 2 pcpus from different sockets become available, the scheduler will assign 2 vcpus to those two pcpus. Does the scheduler do any optimization as to moving the vcpus of a vm to the same socket or just assign the vcpus as they become ready.

or if 3 pcpus are idle will the scheduler assign vcpus from a VM to the same socket.

Basically all my questions boil down to this: Does the Scheduler know about the pcpu layout(same socket) and does it do any scheduling based on that.

Thanks
Prabha



     


-------------- Original message -------------- 
From: "Mike D. Day" <ncmike@us.ibm.com> 

> On 21/04/07 06:03 +0000, pak333@comcast.net wrote: 
> > 
> > Hi, 
> > 
> > 
> > 
> > On running on a dual/quad core does the Xen scheduler take into 
> > account the physical layout of the cores. 
> > 
> > For example if a VM has two vcpus, and there are 4 physical cpus 
> > free, will it take care to assign the 2vcpus (from a VM) to 2 pcpus 
> > on the same socket. 
> 
> 
> The scheduler only knows the affinity of vcpus for physical 
> cpus. The affinity is determined by a userspace application and can 
> be modified using a domain control hypercall. Look in 
> xen/common/domctl.c around line 568 for the following: 
> 
> case XEN_DOMCTL_setvcpuaffinity: 
> case XEN_DOMCTL_getvcpuaffinity: 
> 
> 
> 
> When the credit scheduler migrates a vcpu to a pcpu, it only considers 
> pcpus for which the affinity bit is set. If the userspace application 
> sets affinity such that only the bits set for pcpus on the same 
> socket, then the vcpu will only run on pcpu's sharing the same 
> socket. 
> 
> 
> Mike 
> 
> -- 
> Mike D. Day 
> IBM LTC 
> Cell: 919 412-3900 
> Sametime: ncmike@us.ibm.com AIM: ncmikeday Yahoo: ultra.runner 
> PGP key: http://www.ncultra.org/ncmike/pubkey.asc 
> 
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xensource.com 
> http://lists.xensource.com/xen-devel 

[-- Attachment #1.2: Type: text/html, Size: 3101 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Re: Xen scheduler

[Petersson, Mats]

 

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> pak333@comcast.net
> Sent: 23 April 2007 20:34
> To: ncmike@us.ibm.com
> Cc: Mike D. Day; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Re: Xen scheduler
> 
> Thanks. A little more clarification.
>  
> Here is an example. 
>  
> I have multiple VMs, each with 2 vcpus. There is no user 
> affinity. So i will let the vcpus run whereever the Xen 
> scheduler chooses. My system has 2 dual core sockets.
>  
>      If all 4 pcpus are idle, then will the scheduler assign 
> the vcpus of a VM to the same socket pcpus. 
>      If while running, 2 pcpus from different sockets become 
> available, the scheduler will assign 2 vcpus to those two 
> pcpus. Does the scheduler do any optimization as to moving 
> the vcpus of a vm to the same socket or just assign the vcpus 
> as they become ready.

Xen's current schedulers doesn't have any clue about CPU cores and their
relationship to sockets, memory locations or any other such things. So
if you get two VCPUS from the same domain or two different domains on
one of your physical dual core CPU is entirely random, and will remain
so at any time a VCPU is rescheduled. Whichever PCPU happens to be ready
when the VCPU is schedule will be used unless you specifically restrict
the VCPU to a (set of) PCPU(s). 
>  
> or if 3 pcpus are idle will the scheduler assign vcpus from a 
> VM to the same socket.

It will assign "any VCPU to any PCPU that is allowed for that VCPU", and
it doesn't really care which VM or which socket any particular VCPU/PCPU
combination belongs to. 
>  
> Basically all my questions boil down to this: Does the 
> Scheduler know about the pcpu layout(same socket) and does it 
> do any scheduling based on that.

Not at present. There's been some discussions on this, and whilst it's
easy to solve some of the obvious cases, there are also some harder nuts
to crack. What do you do when the system is really busy and there's not
a "good" PCPU to schedule a particular VCPU on - do you wait for the
PCPU that is ideal to become available, or do you schedule it on a less
ideal PCPU? How long do you allow the wait for that ideal PCPU?

Whilst it's easy to say "Just do it right", solving the rather hairy
problems of when there's congestion and making the right "judgement" of
the situation is much harder. 

--
Mats
>  
> Thanks
> Prabha
>  
>  
>  
>      
>  
>  
> 
> 	-------------- Original message -------------- 
> 	From: "Mike D. Day" <ncmike@us.ibm.com> 
> 	
> 	> On 21/04/07 06:03 +0000, pak333@comcast.net wrote: 
> 	> > 
> 	> > Hi, 
> 	> > 
> 	> > 
> 	> > 
> 	> > On running on a dual/quad core does the Xen 
> scheduler take into 
> 	> > account the physical layout of the cores. 
> 	> > 
> 	> > For example if a VM has two vcpus, and there are 4 
> physical cpus 
> 	> > free, will it take care to assign the 2vcpus (from 
> a VM) to 2 pcpus 
> 	> > on the same socket. 
> 	> 
> 	> 
> 	> The scheduler only knows the affinity of vcpus for physical 
> 	> cpus. The affinity is determined by a userspace 
> application and can 
> 	> be modified using a domain control hypercall. Look in 
> 	> xen/common/domctl.c around line 568 for the following: 
> 	> 
> 	> case XEN_DOMCTL_setvc puaffinity: 
> 	> case XEN_DOMCTL_getvcpuaffinity: 
> 	> 
> 	> 
> 	> 
> 	> When the credit scheduler migrates a vcpu to a pcpu, 
> it only considers 
> 	> pcpus for which the affinity bit is set. If the 
> userspace application 
> 	> sets affinity such that only the bits set for pcpus 
> on the same 
> 	> socket, then the vcpu will only run on pcpu's sharing 
> the same 
> 	> socket. 
> 	> 
> 	> 
> 	> Mike 
> 	> 
> 	> -- 
> 	> Mike D. Day 
> 	> IBM LTC 
> 	> Cell: 919 412-3900 
> 	> Sametime: ncmike@us.ibm.com AIM: ncmikeday Yahoo: 
> ultra.runner 
> 	> PGP key: http://www.ncultra.org/ncmike/pubkey.asc 
> 	> 
> 	> _______________________________________________ 
> 	> Xen-devel mailing list 
> 	> Xen-devel@lists.xensource.com 
> 	> http://lists.xensource.com/xen-devel 
> 
> 

Re: Re: Xen scheduler

[Emmanuel Ackaouy]

On Apr 23, 2007, at 21:33, pak333@comcast.net wrote:
> Basically all my questions boil down to this: Does the Scheduler know 
> about the pcpu layout(same socket) and does it do any scheduling based 
> on that.

Yes but not how you suggested.

The scheduler actually tries to schedule VCPUs across multiple
sockets before it "co-schedules" a socket. The idea behind this is
to maximize the achievable memory bandwidth.

On hyperthreaded systems, the scheduler will also attempt to
schedule across cores before it co-schedules hyperthreads. This
is to maximize achievable cycles.

At this time, no attempt is made to schedule 2 VCPUs of the
same VM any differently than 2 VCPUs of distinct VMs.

If you feel two VCPUs would do better co-scheduled on a
core or socket, you'd currently have to use cpumasks -- as
mike suggested -- to restrict where they can run manually. I'd
be curious to know of real world cases where doing this
increases performance significantly.

Hope this helps.

Cheers,
Emmanuel.

RE: Re: Xen scheduler

[Petersson, Mats]

 

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Emmanuel Ackaouy
> Sent: 24 April 2007 15:35
> To: pak333@comcast.net
> Cc: ncmike@us.ibm.com; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Re: Xen scheduler
> 
> On Apr 23, 2007, at 21:33, pak333@comcast.net wrote:
> > Basically all my questions boil down to this: Does the 
> Scheduler know 
> > about the pcpu layout(same socket) and does it do any 
> scheduling based 
> > on that.
> 
> Yes but not how you suggested.
> 
> The scheduler actually tries to schedule VCPUs across multiple
> sockets before it "co-schedules" a socket. The idea behind this is
> to maximize the achievable memory bandwidth.
> 
> On hyperthreaded systems, the scheduler will also attempt to
> schedule across cores before it co-schedules hyperthreads. This
> is to maximize achievable cycles.
> 
> At this time, no attempt is made to schedule 2 VCPUs of the
> same VM any differently than 2 VCPUs of distinct VMs.
> 
> If you feel two VCPUs would do better co-scheduled on a
> core or socket, you'd currently have to use cpumasks -- as
> mike suggested -- to restrict where they can run manually. I'd
> be curious to know of real world cases where doing this
> increases performance significantly.

If you have data-sharing between the apps on the same socket, and a shared L2 or L3 cache, and the application/data fits in the cache, I could see that it would help. [And of course, the OS for example will have some data and code-sharing between CPU's - so some application where a lot of time is spent in the OS itself would be benefitting from "socket sharing"]. 

For other applications, having better memory bandwitch is most likely better. 

Of course, for ideal performance, it would also have to be taken into account which CPU owns the memory being used, as the latency of transferring memory from one CPU to another in a NUMA system can affect the performance quite noticably.

--
Mats
> 
> Hope this helps.
> 
> Cheers,
> Emmanuel.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 

Re: Re: Xen scheduler

[Emmanuel Ackaouy]

Mats,

As I've just explained in my previous post to the mailing list, what
you say is not totally correct.

On partially idle systems, the credit scheduler does schedule across
sockets and cores before co-scheduling them. At least it used to but
I've not looked at the code in a while so perhaps someone changed
this?

On a system with no idle PCPU, when time slicing becomes
necessary, what you say is true: The scheduler doesn't care
which PCPU hosts which VCPU. On NUMA systems, this
means a VCPU may not always run closest to the memory
it is using at the time.

On Apr 24, 2007, at 13:07, Petersson, Mats wrote:

>
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com
>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
>> pak333@comcast.net
>> Sent: 23 April 2007 20:34
>> To: ncmike@us.ibm.com
>> Cc: Mike D. Day; xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] Re: Xen scheduler
>>
>> Thanks. A little more clarification.
>>
>> Here is an example.
>>
>> I have multiple VMs, each with 2 vcpus. There is no user
>> affinity. So i will let the vcpus run whereever the Xen
>> scheduler chooses. My system has 2 dual core sockets.
>>
>>      If all 4 pcpus are idle, then will the scheduler assign
>> the vcpus of a VM to the same socket pcpus.
>>      If while running, 2 pcpus from different sockets become
>> available, the scheduler will assign 2 vcpus to those two
>> pcpus. Does the scheduler do any optimization as to moving
>> the vcpus of a vm to the same socket or just assign the vcpus
>> as they become ready.
>
> Xen's current schedulers doesn't have any clue about CPU cores and 
> their
> relationship to sockets, memory locations or any other such things. So
> if you get two VCPUS from the same domain or two different domains on
> one of your physical dual core CPU is entirely random, and will remain
> so at any time a VCPU is rescheduled. Whichever PCPU happens to be 
> ready
> when the VCPU is schedule will be used unless you specifically restrict
> the VCPU to a (set of) PCPU(s).
>>
>> or if 3 pcpus are idle will the scheduler assign vcpus from a
>> VM to the same socket.
>
> It will assign "any VCPU to any PCPU that is allowed for that VCPU", 
> and
> it doesn't really care which VM or which socket any particular 
> VCPU/PCPU
> combination belongs to.
>>
>> Basically all my questions boil down to this: Does the
>> Scheduler know about the pcpu layout(same socket) and does it
>> do any scheduling based on that.
>
> Not at present. There's been some discussions on this, and whilst it's
> easy to solve some of the obvious cases, there are also some harder 
> nuts
> to crack. What do you do when the system is really busy and there's not
> a "good" PCPU to schedule a particular VCPU on - do you wait for the
> PCPU that is ideal to become available, or do you schedule it on a less
> ideal PCPU? How long do you allow the wait for that ideal PCPU?
>
> Whilst it's easy to say "Just do it right", solving the rather hairy
> problems of when there's congestion and making the right "judgement" of
> the situation is much harder.
>
> --
> Mats
>>
>> Thanks
>> Prabha
>>
>>
>>
>>
>>
>>
>>
>> 	-------------- Original message --------------
>> 	From: "Mike D. Day" <ncmike@us.ibm.com>
>> 	
>> 	> On 21/04/07 06:03 +0000, pak333@comcast.net wrote:
>> 	> >
>> 	> > Hi,
>> 	> >
>> 	> >
>> 	> >
>> 	> > On running on a dual/quad core does the Xen
>> scheduler take into
>> 	> > account the physical layout of the cores.
>> 	> >
>> 	> > For example if a VM has two vcpus, and there are 4
>> physical cpus
>> 	> > free, will it take care to assign the 2vcpus (from
>> a VM) to 2 pcpus
>> 	> > on the same socket.
>> 	>
>> 	>
>> 	> The scheduler only knows the affinity of vcpus for physical
>> 	> cpus. The affinity is determined by a userspace
>> application and can
>> 	> be modified using a domain control hypercall. Look in
>> 	> xen/common/domctl.c around line 568 for the following:
>> 	>
>> 	> case XEN_DOMCTL_setvc puaffinity:
>> 	> case XEN_DOMCTL_getvcpuaffinity:
>> 	>
>> 	>
>> 	>
>> 	> When the credit scheduler migrates a vcpu to a pcpu,
>> it only considers
>> 	> pcpus for which the affinity bit is set. If the
>> userspace application
>> 	> sets affinity such that only the bits set for pcpus
>> on the same
>> 	> socket, then the vcpu will only run on pcpu's sharing
>> the same
>> 	> socket.
>> 	>
>> 	>
>> 	> Mike
>> 	>
>> 	> --
>> 	> Mike D. Day
>> 	> IBM LTC
>> 	> Cell: 919 412-3900
>> 	> Sametime: ncmike@us.ibm.com AIM: ncmikeday Yahoo:
>> ultra.runner
>> 	> PGP key: http://www.ncultra.org/ncmike/pubkey.asc
>> 	>
>> 	> _______________________________________________
>> 	> Xen-devel mailing list
>> 	> Xen-devel@lists.xensource.com
>> 	> http://lists.xensource.com/xen-devel
>>
>>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: Re: Xen scheduler

[Petersson, Mats]

 

> -----Original Message-----
> From: Emmanuel Ackaouy [mailto:ackaouy@gmail.com] 
> Sent: 24 April 2007 15:47
> To: Petersson, Mats
> Cc: pak333@comcast.net; xen-devel@lists.xensource.com; 
> ncmike@us.ibm.com
> Subject: Re: [Xen-devel] Re: Xen scheduler
> 
> Mats,
> 
> As I've just explained in my previous post to the mailing list, what
> you say is not totally correct.

Yes, I realized from your previous post that what I said was not
entirely correctly. 

> 
> On partially idle systems, the credit scheduler does schedule across
> sockets and cores before co-scheduling them. At least it used to but
> I've not looked at the code in a while so perhaps someone changed
> this?

Not that I'm aware of. I apologize for the misinformation. 

> 
> On a system with no idle PCPU, when time slicing becomes
> necessary, what you say is true: The scheduler doesn't care
> which PCPU hosts which VCPU. On NUMA systems, this
> means a VCPU may not always run closest to the memory
> it is using at the time.
> 
> On Apr 24, 2007, at 13:07, Petersson, Mats wrote:
> 
> >
> >
> >> -----Original Message-----
> >> From: xen-devel-bounces@lists.xensource.com
> >> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
> >> pak333@comcast.net
> >> Sent: 23 April 2007 20:34
> >> To: ncmike@us.ibm.com
> >> Cc: Mike D. Day; xen-devel@lists.xensource.com
> >> Subject: Re: [Xen-devel] Re: Xen scheduler
> >>
> >> Thanks. A little more clarification.
> >>
> >> Here is an example.
> >>
> >> I have multiple VMs, each with 2 vcpus. There is no user
> >> affinity. So i will let the vcpus run whereever the Xen
> >> scheduler chooses. My system has 2 dual core sockets.
> >>
> >>      If all 4 pcpus are idle, then will the scheduler assign
> >> the vcpus of a VM to the same socket pcpus.
> >>      If while running, 2 pcpus from different sockets become
> >> available, the scheduler will assign 2 vcpus to those two
> >> pcpus. Does the scheduler do any optimization as to moving
> >> the vcpus of a vm to the same socket or just assign the vcpus
> >> as they become ready.
> >
> > Xen's current schedulers doesn't have any clue about CPU cores and 
> > their
> > relationship to sockets, memory locations or any other such 
> things. So
> > if you get two VCPUS from the same domain or two different 
> domains on
> > one of your physical dual core CPU is entirely random, and 
> will remain
> > so at any time a VCPU is rescheduled. Whichever PCPU happens to be 
> > ready
> > when the VCPU is schedule will be used unless you 
> specifically restrict
> > the VCPU to a (set of) PCPU(s).
> >>
> >> or if 3 pcpus are idle will the scheduler assign vcpus from a
> >> VM to the same socket.
> >
> > It will assign "any VCPU to any PCPU that is allowed for 
> that VCPU", 
> > and
> > it doesn't really care which VM or which socket any particular 
> > VCPU/PCPU
> > combination belongs to.
> >>
> >> Basically all my questions boil down to this: Does the
> >> Scheduler know about the pcpu layout(same socket) and does it
> >> do any scheduling based on that.
> >
> > Not at present. There's been some discussions on this, and 
> whilst it's
> > easy to solve some of the obvious cases, there are also some harder 
> > nuts
> > to crack. What do you do when the system is really busy and 
> there's not
> > a "good" PCPU to schedule a particular VCPU on - do you wait for the
> > PCPU that is ideal to become available, or do you schedule 
> it on a less
> > ideal PCPU? How long do you allow the wait for that ideal PCPU?
> >
> > Whilst it's easy to say "Just do it right", solving the rather hairy
> > problems of when there's congestion and making the right 
> "judgement" of
> > the situation is much harder.
> >
> > --
> > Mats
> >>
> >> Thanks
> >> Prabha
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> 	-------------- Original message --------------
> >> 	From: "Mike D. Day" <ncmike@us.ibm.com>
> >> 	
> >> 	> On 21/04/07 06:03 +0000, pak333@comcast.net wrote:
> >> 	> >
> >> 	> > Hi,
> >> 	> >
> >> 	> >
> >> 	> >
> >> 	> > On running on a dual/quad core does the Xen
> >> scheduler take into
> >> 	> > account the physical layout of the cores.
> >> 	> >
> >> 	> > For example if a VM has two vcpus, and there are 4
> >> physical cpus
> >> 	> > free, will it take care to assign the 2vcpus (from
> >> a VM) to 2 pcpus
> >> 	> > on the same socket.
> >> 	>
> >> 	>
> >> 	> The scheduler only knows the affinity of vcpus for physical
> >> 	> cpus. The affinity is determined by a userspace
> >> application and can
> >> 	> be modified using a domain control hypercall. Look in
> >> 	> xen/common/domctl.c around line 568 for the following:
> >> 	>
> >> 	> case XEN_DOMCTL_setvc puaffinity:
> >> 	> case XEN_DOMCTL_getvcpuaffinity:
> >> 	>
> >> 	>
> >> 	>
> >> 	> When the credit scheduler migrates a vcpu to a pcpu,
> >> it only considers
> >> 	> pcpus for which the affinity bit is set. If the
> >> userspace application
> >> 	> sets affinity such that only the bits set for pcpus
> >> on the same
> >> 	> socket, then the vcpu will only run on pcpu's sharing
> >> the same
> >> 	> socket.
> >> 	>
> >> 	>
> >> 	> Mike
> >> 	>
> >> 	> --
> >> 	> Mike D. Day
> >> 	> IBM LTC
> >> 	> Cell: 919 412-3900
> >> 	> Sametime: ncmike@us.ibm.com AIM: ncmikeday Yahoo:
> >> ultra.runner
> >> 	> PGP key: http://www.ncultra.org/ncmike/pubkey.asc
> >> 	>
> >> 	> _______________________________________________
> >> 	> Xen-devel mailing list
> >> 	> Xen-devel@lists.xensource.com
> >> 	> http://lists.xensource.com/xen-devel
> >>
> >>
> >
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> 
> 
> 
> 

Re: Re: Xen scheduler

[Emmanuel Ackaouy]

On Apr 24, 2007, at 16:42, Petersson, Mats wrote:
>> If you feel two VCPUs would do better co-scheduled on a
>> core or socket, you'd currently have to use cpumasks -- as
>> mike suggested -- to restrict where they can run manually. I'd
>> be curious to know of real world cases where doing this
>> increases performance significantly.
>
> If you have data-sharing between the apps on the same socket, and a 
> shared L2 or L3 cache, and the application/data fits in the cache, I 
> could see that it would help. [And of course, the OS for example will 
> have some data and code-sharing between CPU's - so some application 
> where a lot of time is spent in the OS itself would be benefitting 
> from "socket sharing"].
>
> For other applications, having better memory bandwitch is most likely 
> better.
>
> Of course, for ideal performance, it would also have to be taken into 
> account which CPU owns the memory being used, as the latency of 
> transferring memory from one CPU to another in a NUMA system can 
> affect the performance quite noticably.

I understand in theory what would do better scheduled in either
of these ways. What I'm interested in learning about is actual
applications that people use that exhibit the type of L2/3 cache
sharing that would make it significantly better to co-schedule
the VCPUs in question on whole sockets rather than across
them.