New CPU scheduler w/ SMP load balancer

New CPU scheduler w/ SMP load balancer

[Emmanuel Ackaouy]

Executive summary
-----------------

I have just committed a change to xen-unstable.hg which adds
support for a new Xen CPU scheduler, which should greatly
enhance resource scheduling on SMP systems and provides better
QoS controls.

The new scheduler is not currently the default, so to enable
it you will need to put "sched=credit" on the Xen command line.
Please give it a go and provide us with some feedback as to how
it performs with your workloads!


Detail
------

The new CPU scheduler automatically load balances guest VCPUs
across all available physical CPUs on an SMP host. You no longer
need to manually pin VCPUs to load balance the system. However,
you can restrict which CPUs a particular VCPU may run on using
the existing "vcpu-pin" interface.

Each guest domain is assigned a "weight" and a "cap".

A domain with a weight of 512 will get twice as much CPU as a
domain with a weight of 256 on a contended host. Legal weights
range from 1 to 65535 and the default is 256.

The cap optionally fixes the maximum amount of CPU a guest will
be able to consume, even if the host system has idle CPU cycles.
The cap is expressed in percentage of one physical CPU: 100 is
1 physical CPU, 50 is half a CPU, 400 is 4 CPUs, etc... The
default, 0, means there is no upper cap.

To use the new scheduler, you need to pick up my change. You
will need to upgrade the tools as well as the hypervisor. We
intend to make this the default scheduler in the near future.
Until then, you must specify "sched=credit" on the Xen command
line in your boot loader.

Once you are running with the new SMP credit scheduler, you will
be able to check and modify your domains' weights and caps using
the "xm csched" command:

xm csched -d <domain>			lists weight and cap
xm csched -d <domain> -w <weight>	sets the weight
xm csched -d <domain> -c <cap>		sets the cap

I'm interested to hear about your experiences using this
scheduler as well as any performance results running your
favorite applications or benchmarks over it.

Enjoy,
Emmanuel.

Re: New CPU scheduler w/ SMP load balancer

[Anthony Liguori]

Hi Emmanuel,

Thanks for the heads up.  It would have been nice to float these patches 
to the list before committing.

Can you provide some more details on any results you may have seen with 
the new scheduler?  How does it affect common benchmarks?  How does the 
"load balancer" scale?  How much penalty do you pay (if any at all) on UP?

Better yet, if you have a paper you could share, that would be even 
better :-)  If you cannot share because of conference restrictions, it 
would be nice if you could a condensed version (similar to what the L4ka 
group did for their afterburning work).

Based on your description though, the new scheduler looks very promising!

Regards,

Anthony Liguori

Emmanuel Ackaouy wrote:
> Executive summary
> -----------------
>
> I have just committed a change to xen-unstable.hg which adds
> support for a new Xen CPU scheduler, which should greatly
> enhance resource scheduling on SMP systems and provides better
> QoS controls.
>
> The new scheduler is not currently the default, so to enable
> it you will need to put "sched=credit" on the Xen command line.
> Please give it a go and provide us with some feedback as to how
> it performs with your workloads!
>
>
> Detail
> ------
>
> The new CPU scheduler automatically load balances guest VCPUs
> across all available physical CPUs on an SMP host. You no longer
> need to manually pin VCPUs to load balance the system. However,
> you can restrict which CPUs a particular VCPU may run on using
> the existing "vcpu-pin" interface.
>
> Each guest domain is assigned a "weight" and a "cap".
>
> A domain with a weight of 512 will get twice as much CPU as a
> domain with a weight of 256 on a contended host. Legal weights
> range from 1 to 65535 and the default is 256.
>
> The cap optionally fixes the maximum amount of CPU a guest will
> be able to consume, even if the host system has idle CPU cycles.
> The cap is expressed in percentage of one physical CPU: 100 is
> 1 physical CPU, 50 is half a CPU, 400 is 4 CPUs, etc... The
> default, 0, means there is no upper cap.
>
> To use the new scheduler, you need to pick up my change. You
> will need to upgrade the tools as well as the hypervisor. We
> intend to make this the default scheduler in the near future.
> Until then, you must specify "sched=credit" on the Xen command
> line in your boot loader.
>
> Once you are running with the new SMP credit scheduler, you will
> be able to check and modify your domains' weights and caps using
> the "xm csched" command:
>
> xm csched -d <domain>			lists weight and cap
> xm csched -d <domain> -w <weight>	sets the weight
> xm csched -d <domain> -c <cap>		sets the cap
>
> I'm interested to hear about your experiences using this
> scheduler as well as any performance results running your
> favorite applications or benchmarks over it.
>
> Enjoy,
> Emmanuel.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   

Re: New CPU scheduler w/ SMP load balancer

[Emmanuel Ackaouy]

Hi Anthony.

Thanks for your feedback. I'll take a look at your comments
regarding the Xend python code in the patch this week end.

On Fri, May 26, 2006 at 10:42:39AM -0500, Anthony Liguori wrote:
> Can you provide some more details on any results you may have seen with 
> the new scheduler?  How does it affect common benchmarks?  How does the 
> "load balancer" scale?  How much penalty do you pay (if any at all) on UP?

It is not simple to define a set of performance benchmarks for
a VCPU scheduler. On an SMP host, the credit scheduler is a lot
better at enforcing fairness across multiple guest, some SMP
and some UP. Certainly, the VCPU scheduler has an effect on I/O
benchmarks because of the interaction between domUs and dom0.

I found that on a uni-processor, running ttcp in a domU yielded
almost twice the network bandwidth with the credit scheduler
compared to with SEDF. This probably has less to do with scheduling
algorithms than with implementation problems though.

For SMP guests, the credit scheduler enforces that all VCPUs
make equal progress. This solves a number of serious performance
problems when you are time slicing some of your physical CPUs
between multiple SMP guests.

In terms of consolidating multiple guests on one SMP host, we
are now playing in a different ballpark with the credit scheduler:
When a CPU goes idle, it immediately picks up a runnable VCPU
waiting on the runqueue on another CPU. With SEDF and BVT, you
have to manually place all the VCPUs in the system and there are
no dynamic adjustements when VCPUs go to sleep waiting for I/O.
The credit scheduler is work conserving in that it will make use
of any CPU cycles when there is runnable work. It does this as
soon as a CPU runs out of work. This is in contrast with other
load balancing algorithms that work in the background and move
things around on some type of clock tick. Being work conserving
on SMP hosts is a huge improvement over the previous scheduler
implementations.

In terms of scaling, I have taken profiles on an 8-way system
and found lock contention to be reasonable. We'll need to do
some performance work and perhaps pad some cachelines or change
a few things to run on very large NUMA type systems but by design,
the credit scheduler is designed to scale to very large systems.

The common code path (do_schedule) is designed to be extremely
fast on both UP and MP systems. Using the scientific method of
code inspection :-), these code paths are a lot shorter and faster
than the SEDF ones. The accounting work in the credit scheduler is
done every 30 milliseconds outside the common path and its
complexity is linear with the number of running VCPUs in the
system. Making accounting work overhead independant of the
number of scheduling operations is good on I/O workloads where
lots of context switches occur.

> Better yet, if you have a paper you could share, that would be even 
> better :-)  If you cannot share because of conference restrictions, it 
> would be nice if you could a condensed version (similar to what the L4ka 
> group did for their afterburning work).

Writing a paper is something I'd like to do at some point once
we've had more experience in the field.

> Based on your description though, the new scheduler looks very promising!

I am eager to hear people's experiences with the new scheduler,
especially on SMP hosts.


Cheers,
Emmanuel.

Re: New CPU scheduler w/ SMP load balancer

[Diwaker Gupta]

Hi Emmanuel, list:

Is there any documentation on the internal working of the scheduler,
beyond the source code?

Thanks,
Diwaker

On 5/26/06, Emmanuel Ackaouy <ack@xensource.com> wrote:
> Executive summary
> -----------------
>
> I have just committed a change to xen-unstable.hg which adds
> support for a new Xen CPU scheduler, which should greatly
> enhance resource scheduling on SMP systems and provides better
> QoS controls.

-- 
Web/Blog/Gallery: http://floatingsun.net/blog

Re: New CPU scheduler w/ SMP load balancer

[Emmanuel Ackaouy]

On Wed, Jun 21, 2006 at 11:28:49AM -0700, Diwaker Gupta wrote:
> Hi Emmanuel, list:
> 
> Is there any documentation on the internal working of the scheduler,
> beyond the source code?

Not really.

There are some comments in the code that should be helpful.
Also, some of the mails I sent to xen-devel should be useful
in highlighting the basic design strategy and goals.

It would be good to put all this together and on the Wiki.

Re: New CPU scheduler w/ SMP load balancer

[Anand Gupta]

[-- Attachment #1.1: Type: text/plain, Size: 1100 bytes --]

On 5/26/06, Emmanuel Ackaouy <ack@xensource.com> wrote:
>
> I have just committed a change to xen-unstable.hg which adds
> support for a new Xen CPU scheduler, which should greatly
> enhance resource scheduling on SMP systems and provides better
> QoS controls.
>
> The new scheduler is not currently the default, so to enable
> it you will need to put "sched=credit" on the Xen command line.
> Please give it a go and provide us with some feedback as to how
> it performs with your workloads!


Is the scheduler now default in the latest unstable ? How do i check which
is the scheduler ?

Once you are running with the new SMP credit scheduler, you will
> be able to check and modify your domains' weights and caps using
> the "xm csched" command:
>
> xm csched -d <domain>                   lists weight and cap
> xm csched -d <domain> -w <weight>       sets the weight
> xm csched -d <domain> -c <cap>          sets the cap
>

I tried xm csched and it gave me errors:

Error: Sub Command csched not found!

However xm sched-credit does seem to work. Was this changed ?

-- 
regards,

Anand Gupta

[-- Attachment #1.2: Type: text/html, Size: 1803 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: New CPU scheduler w/ SMP load balancer

[Emmanuel Ackaouy]

On Fri, Jun 23, 2006 at 03:19:02PM +0530, Anand Gupta wrote:
>    Is the scheduler now default in the latest unstable ? How do i check which
>    is the scheduler ?

Yes, it is now the default scheduler as of changeset a31f3bff4f76.
One way is to check the Xen boot console output:
xm dmesg | grep "Using scheduler:"

>From the console, the 'r' debug key prints runq info and will
also let you know which scheduler is running.

xm sched-credit commands will only work if the credit scheduler
is running.

In addition, there's a hypercall which you can ahold of via
xc_sched_id which returns an integer scheduler id defined in
sched_ctl.h (SCHED_CREDIT is 5).

>      xm csched -d <domain>                   lists weight and cap
>      xm csched -d <domain> -w <weight>       sets the weight
>      xm csched -d <domain> -c <cap>          sets the cap
> 
>    I tried xm csched and it gave me errors:
> 
>    Error: Sub Command csched not found!
> 
>    However xm sched-credit does seem to work. Was this changed ?

Yeah. I think Keir changed this shortly after I sent the mail.
We have sched-sedf and sched-bvt commands so it made sense to
call this sched-credit. Sorry for the confusion.

Re: [Xen-devel] New CPU scheduler w/ SMP load balancer

[Anand Gupta]

[-- Attachment #1.1: Type: text/plain, Size: 1491 bytes --]

On 6/23/06, Emmanuel Ackaouy <ack@xensource.com> wrote:
>
> On Fri, Jun 23, 2006 at 03:19:02PM +0530, Anand Gupta wrote:
> >    Is the scheduler now default in the latest unstable ? How do i check
> which
> >    is the scheduler ?
>
> Yes, it is now the default scheduler as of changeset a31f3bff4f76.
> One way is to check the Xen boot console output:
> xm dmesg | grep "Using scheduler:"


Thanks for the info. Yes i see the default scheduler is credit.

>From the console, the 'r' debug key prints runq info and will
> also let you know which scheduler is running.


Can you please explain this ?

>      xm csched -d <domain>                   lists weight and cap
> >      xm csched -d <domain> -w <weight>       sets the weight
> >      xm csched -d <domain> -c <cap>          sets the cap


Are there any valid entries for weight or cap ? Any limits etc ?

>
> >    I tried xm csched and it gave me errors:
> >
> >    Error: Sub Command csched not found!
> >
> >    However xm sched-credit does seem to work. Was this changed ?
>
> Yeah. I think Keir changed this shortly after I sent the mail.
> We have sched-sedf and sched-bvt commands so it made sense to
> call this sched-credit. Sorry for the confusion.
>

No confusion. Excellent work i should say.

I have been trying this scheduler with kernel compile on dom0 and some
activity inside domU and tried to assign various weights and caps for both
domains and the result showed up perfectly in xm top.

-- 
regards,

Anand Gupta

[-- Attachment #1.2: Type: text/html, Size: 2715 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: New CPU scheduler w/ SMP load balancer

[Atsushi SAKAI]

At first, you should see the following manual.
http://www.cl.cam.ac.uk/Research/SRG/netos/xen/readmes/user.pdf
This Users manual is daily updated.
Of course, The manual on Credit scheduler is included.

>>      xm csched -d <domain>                   lists weight and cap
>> >      xm csched -d <domain> -w <weight>       sets the weight
>> >      xm csched -d <domain> -c <cap>          sets the cap
>
>
>Are there any valid entries for weight or cap ? Any limits etc ?
>

Thanks.
Atsushi SAKAI

Re: New CPU scheduler w/ SMP load balancer

[Anand Gupta]

[-- Attachment #1.1: Type: text/plain, Size: 697 bytes --]

Thanks found all the information about the usage and the limits in the
manual.

On 6/23/06, Atsushi SAKAI <sakaia@jp.fujitsu.com> wrote:
>
> At first, you should see the following manual.
> http://www.cl.cam.ac.uk/Research/SRG/netos/xen/readmes/user.pdf
> This Users manual is daily updated.
> Of course, The manual on Credit scheduler is included.
>
> >>      xm csched -d <domain>                   lists weight and cap
> >> >      xm csched -d <domain> -w <weight>       sets the weight
> >> >      xm csched -d <domain> -c <cap>          sets the cap
> >
> >
> >Are there any valid entries for weight or cap ? Any limits etc ?
> >
>
> Thanks.
> Atsushi SAKAI
>
>
>


-- 
regards,

Anand Gupta

[-- Attachment #1.2: Type: text/html, Size: 1425 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: New CPU scheduler w/ SMP load balancer

[Apparao, Padmashree K]

On the same note: has anyone got the same information with the BVT and
sedf schedulers:
Performance, overhead and efficiency in load balancing

Thanks
- Padma



-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Anthony
Liguori
Sent: Friday, May 26, 2006 8:43 AM
To: xen-devel@lists.xensource.com; ack@xensource.com
Subject: Re: [Xen-devel] New CPU scheduler w/ SMP load balancer

Hi Emmanuel,

Thanks for the heads up.  It would have been nice to float these patches

to the list before committing.

Can you provide some more details on any results you may have seen with 
the new scheduler?  How does it affect common benchmarks?  How does the 
"load balancer" scale?  How much penalty do you pay (if any at all) on
UP?

Better yet, if you have a paper you could share, that would be even 
better :-)  If you cannot share because of conference restrictions, it 
would be nice if you could a condensed version (similar to what the L4ka

group did for their afterburning work).

Based on your description though, the new scheduler looks very
promising!

Regards,

Anthony Liguori

Emmanuel Ackaouy wrote:
> Executive summary
> -----------------
>
> I have just committed a change to xen-unstable.hg which adds
> support for a new Xen CPU scheduler, which should greatly
> enhance resource scheduling on SMP systems and provides better
> QoS controls.
>
> The new scheduler is not currently the default, so to enable
> it you will need to put "sched=credit" on the Xen command line.
> Please give it a go and provide us with some feedback as to how
> it performs with your workloads!
>
>
> Detail
> ------
>
> The new CPU scheduler automatically load balances guest VCPUs
> across all available physical CPUs on an SMP host. You no longer
> need to manually pin VCPUs to load balance the system. However,
> you can restrict which CPUs a particular VCPU may run on using
> the existing "vcpu-pin" interface.
>
> Each guest domain is assigned a "weight" and a "cap".
>
> A domain with a weight of 512 will get twice as much CPU as a
> domain with a weight of 256 on a contended host. Legal weights
> range from 1 to 65535 and the default is 256.
>
> The cap optionally fixes the maximum amount of CPU a guest will
> be able to consume, even if the host system has idle CPU cycles.
> The cap is expressed in percentage of one physical CPU: 100 is
> 1 physical CPU, 50 is half a CPU, 400 is 4 CPUs, etc... The
> default, 0, means there is no upper cap.
>
> To use the new scheduler, you need to pick up my change. You
> will need to upgrade the tools as well as the hypervisor. We
> intend to make this the default scheduler in the near future.
> Until then, you must specify "sched=credit" on the Xen command
> line in your boot loader.
>
> Once you are running with the new SMP credit scheduler, you will
> be able to check and modify your domains' weights and caps using
> the "xm csched" command:
>
> xm csched -d <domain>			lists weight and cap
> xm csched -d <domain> -w <weight>	sets the weight
> xm csched -d <domain> -c <cap>		sets the cap
>
> I'm interested to hear about your experiences using this
> scheduler as well as any performance results running your
> favorite applications or benchmarks over it.
>
> Enjoy,
> Emmanuel.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: New CPU scheduler w/ SMP load balancer

[Kamble, Nitin A]

Hi Emmanuel,
   I was looking into doing some load balancing (there was none earlier)
to the domain/vcpu scheduling inside the Xen. And I am glad to see your
patch is targeting exactly that.
   I believe the credit scheduler also has sizeable impact on the HVM
domain performance. Do you have any performance data for the HVM guests
with your scheduler?

Thanks & Regards,
Nitin
------------------------------------------------------------------------
-----------
Open Source Technology Center, Intel Corp

>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>bounces@lists.xensource.com] On Behalf Of Emmanuel Ackaouy
>Sent: Friday, May 26, 2006 12:11 PM
>To: Anthony Liguori
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] New CPU scheduler w/ SMP load balancer
>
>Hi Anthony.
>
>Thanks for your feedback. I'll take a look at your comments
>regarding the Xend python code in the patch this week end.
>
>On Fri, May 26, 2006 at 10:42:39AM -0500, Anthony Liguori wrote:
>> Can you provide some more details on any results you may have seen
with
>> the new scheduler?  How does it affect common benchmarks?  How does
the
>> "load balancer" scale?  How much penalty do you pay (if any at all)
on
>UP?
>
>It is not simple to define a set of performance benchmarks for
>a VCPU scheduler. On an SMP host, the credit scheduler is a lot
>better at enforcing fairness across multiple guest, some SMP
>and some UP. Certainly, the VCPU scheduler has an effect on I/O
>benchmarks because of the interaction between domUs and dom0.
>
>I found that on a uni-processor, running ttcp in a domU yielded
>almost twice the network bandwidth with the credit scheduler
>compared to with SEDF. This probably has less to do with scheduling
>algorithms than with implementation problems though.
>
>For SMP guests, the credit scheduler enforces that all VCPUs
>make equal progress. This solves a number of serious performance
>problems when you are time slicing some of your physical CPUs
>between multiple SMP guests.
>
>In terms of consolidating multiple guests on one SMP host, we
>are now playing in a different ballpark with the credit scheduler:
>When a CPU goes idle, it immediately picks up a runnable VCPU
>waiting on the runqueue on another CPU. With SEDF and BVT, you
>have to manually place all the VCPUs in the system and there are
>no dynamic adjustements when VCPUs go to sleep waiting for I/O.
>The credit scheduler is work conserving in that it will make use
>of any CPU cycles when there is runnable work. It does this as
>soon as a CPU runs out of work. This is in contrast with other
>load balancing algorithms that work in the background and move
>things around on some type of clock tick. Being work conserving
>on SMP hosts is a huge improvement over the previous scheduler
>implementations.
>
>In terms of scaling, I have taken profiles on an 8-way system
>and found lock contention to be reasonable. We'll need to do
>some performance work and perhaps pad some cachelines or change
>a few things to run on very large NUMA type systems but by design,
>the credit scheduler is designed to scale to very large systems.
>
>The common code path (do_schedule) is designed to be extremely
>fast on both UP and MP systems. Using the scientific method of
>code inspection :-), these code paths are a lot shorter and faster
>than the SEDF ones. The accounting work in the credit scheduler is
>done every 30 milliseconds outside the common path and its
>complexity is linear with the number of running VCPUs in the
>system. Making accounting work overhead independant of the
>number of scheduling operations is good on I/O workloads where
>lots of context switches occur.
>
>> Better yet, if you have a paper you could share, that would be even
>> better :-)  If you cannot share because of conference restrictions,
it
>> would be nice if you could a condensed version (similar to what the
L4ka
>> group did for their afterburning work).
>
>Writing a paper is something I'd like to do at some point once
>we've had more experience in the field.
>
>> Based on your description though, the new scheduler looks very
promising!
>
>I am eager to hear people's experiences with the new scheduler,
>especially on SMP hosts.
>
>
>Cheers,
>Emmanuel.
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

Re: New CPU scheduler w/ SMP load balancer

[Emmanuel Ackaouy]

On Fri, May 26, 2006 at 01:57:47PM -0700, Kamble, Nitin A wrote:
>    I was looking into doing some load balancing (there was none earlier)
> to the domain/vcpu scheduling inside the Xen. And I am glad to see your
> patch is targeting exactly that.
>    I believe the credit scheduler also has sizeable impact on the HVM
> domain performance. Do you have any performance data for the HVM guests
> with your scheduler?

The new scheduler is aggressive about moving VCPUs around to
keep the entire SMP host system busy. The operation to move a
VCPU's context from one CPU to another needs to be reasonably
efficient.

I didn't try this with an HVM guest. Can you help out with
this? I need to look at the code path that moves an HVM VCPU's
context between physical CPUs. We may need to optimize that
path.

Emmanuel.