credit scheduler

credit scheduler

[Karl Rister]

[-- Attachment #1: Type: text/plain, Size: 1478 bytes --]

While doing some performance analysis of Xen I have noticed some interesting 
behavior of the credit scheduler that I would like to see discussed.

In my most basic test I have a 4 socket dual core Intel box (Paxville) with a 
uniprocessor dom0 and 7 uniprocessor domUs.  Each of the domUs is pinned on 
its own core with the first core of the system left for dom0.  When running 
the credit scheduler the dom0 VCPU will bounce around the system, sometimes 
landing on the same thread as one of the domUs or sometimes on one of the 
sibling hyperthreads (this appears to happen a majority of the time it 
moves).  This is less than ideal when considering cache warmth and the 
sharing of CPU resources when the first core of the system is always 
available in this configuration.  Does the credit scheduler have any 
awareness of cache warmth or CPU siblings when balancing?

I have also seen similar behavior when running tests in the domUs such that 
each has its VCPU running at 100% utilization so I believe this behavior to 
be fairly uniform.

In my testing I am looking for uniform behavior so I have just been setting 
sched=sedf but I would like to move to credit scheduler since it is the new 
default. However, until I sort out this behavior I cannot.

Attached is a snapshot of "xm vcpu-list" taken every 5 seconds for 5 
measurements to show the dom0 VCPU migration pattern on a system with idle 
domains.

-- 
Karl Rister
IBM Linux Performance Team
kmr@us.ibm.com

[-- Attachment #2: xm.vcpu-list --]
[-- Type: text/plain, Size: 8590 bytes --]

Name                              ID  VCPU  CPU  State  Time(s)  CPU Affinity
Domain-0                           0     0    7   r--     402.2  any cpu
Domain-0                           0     1    -   --p       0.0  any cpu
Domain-0                           0     2    -   --p       0.0  any cpu
Domain-0                           0     3    -   --p       0.0  any cpu
Domain-0                           0     4    -   --p       0.0  any cpu
Domain-0                           0     5    -   --p       0.0  any cpu
Domain-0                           0     6    -   --p       0.0  any cpu
Domain-0                           0     7    -   --p       0.0  any cpu
Domain-0                           0     8    -   --p       0.0  any cpu
Domain-0                           0     9    -   --p       0.0  any cpu
Domain-0                           0    10    -   --p       0.0  any cpu
Domain-0                           0    11    -   --p       0.0  any cpu
Domain-0                           0    12    -   --p       0.0  any cpu
Domain-0                           0    13    -   --p       0.0  any cpu
Domain-0                           0    14    -   --p       0.0  any cpu
Domain-0                           0    15    -   --p       0.0  any cpu
dom1                               1     0    2   -b-      21.0  2
dom2                               2     0    4   -b-      21.4  4
dom3                               3     0    6   -b-      20.8  6
dom4                               4     0    8   -b-      21.0  8
dom5                               5     0   10   -b-      21.5  10
dom6                               6     0   12   -b-      21.1  12
dom7                               7     0   14   -b-      20.4  14
Name                              ID  VCPU  CPU  State  Time(s)  CPU Affinity
Domain-0                           0     0    6   r--     402.7  any cpu
Domain-0                           0     1    -   --p       0.0  any cpu
Domain-0                           0     2    -   --p       0.0  any cpu
Domain-0                           0     3    -   --p       0.0  any cpu
Domain-0                           0     4    -   --p       0.0  any cpu
Domain-0                           0     5    -   --p       0.0  any cpu
Domain-0                           0     6    -   --p       0.0  any cpu
Domain-0                           0     7    -   --p       0.0  any cpu
Domain-0                           0     8    -   --p       0.0  any cpu
Domain-0                           0     9    -   --p       0.0  any cpu
Domain-0                           0    10    -   --p       0.0  any cpu
Domain-0                           0    11    -   --p       0.0  any cpu
Domain-0                           0    12    -   --p       0.0  any cpu
Domain-0                           0    13    -   --p       0.0  any cpu
Domain-0                           0    14    -   --p       0.0  any cpu
Domain-0                           0    15    -   --p       0.0  any cpu
dom1                               1     0    2   -b-      21.0  2
dom2                               2     0    4   -b-      21.4  4
dom3                               3     0    6   -b-      20.8  6
dom4                               4     0    8   -b-      21.0  8
dom5                               5     0   10   -b-      21.5  10
dom6                               6     0   12   -b-      21.1  12
dom7                               7     0   14   -b-      20.5  14
Name                              ID  VCPU  CPU  State  Time(s)  CPU Affinity
Domain-0                           0     0    6   r--     403.0  any cpu
Domain-0                           0     1    -   --p       0.0  any cpu
Domain-0                           0     2    -   --p       0.0  any cpu
Domain-0                           0     3    -   --p       0.0  any cpu
Domain-0                           0     4    -   --p       0.0  any cpu
Domain-0                           0     5    -   --p       0.0  any cpu
Domain-0                           0     6    -   --p       0.0  any cpu
Domain-0                           0     7    -   --p       0.0  any cpu
Domain-0                           0     8    -   --p       0.0  any cpu
Domain-0                           0     9    -   --p       0.0  any cpu
Domain-0                           0    10    -   --p       0.0  any cpu
Domain-0                           0    11    -   --p       0.0  any cpu
Domain-0                           0    12    -   --p       0.0  any cpu
Domain-0                           0    13    -   --p       0.0  any cpu
Domain-0                           0    14    -   --p       0.0  any cpu
Domain-0                           0    15    -   --p       0.0  any cpu
dom1                               1     0    2   -b-      21.0  2
dom2                               2     0    4   -b-      21.4  4
dom3                               3     0    6   -b-      20.8  6
dom4                               4     0    8   -b-      21.0  8
dom5                               5     0   10   -b-      21.5  10
dom6                               6     0   12   -b-      21.1  12
dom7                               7     0   14   -b-      20.5  14
Name                              ID  VCPU  CPU  State  Time(s)  CPU Affinity
Domain-0                           0     0    6   r--     403.4  any cpu
Domain-0                           0     1    -   --p       0.0  any cpu
Domain-0                           0     2    -   --p       0.0  any cpu
Domain-0                           0     3    -   --p       0.0  any cpu
Domain-0                           0     4    -   --p       0.0  any cpu
Domain-0                           0     5    -   --p       0.0  any cpu
Domain-0                           0     6    -   --p       0.0  any cpu
Domain-0                           0     7    -   --p       0.0  any cpu
Domain-0                           0     8    -   --p       0.0  any cpu
Domain-0                           0     9    -   --p       0.0  any cpu
Domain-0                           0    10    -   --p       0.0  any cpu
Domain-0                           0    11    -   --p       0.0  any cpu
Domain-0                           0    12    -   --p       0.0  any cpu
Domain-0                           0    13    -   --p       0.0  any cpu
Domain-0                           0    14    -   --p       0.0  any cpu
Domain-0                           0    15    -   --p       0.0  any cpu
dom1                               1     0    2   -b-      21.0  2
dom2                               2     0    4   -b-      21.4  4
dom3                               3     0    6   -b-      20.8  6
dom4                               4     0    8   -b-      21.0  8
dom5                               5     0   10   -b-      21.5  10
dom6                               6     0   12   -b-      21.1  12
dom7                               7     0   14   -b-      20.5  14
Name                              ID  VCPU  CPU  State  Time(s)  CPU Affinity
Domain-0                           0     0    5   r--     403.8  any cpu
Domain-0                           0     1    -   --p       0.0  any cpu
Domain-0                           0     2    -   --p       0.0  any cpu
Domain-0                           0     3    -   --p       0.0  any cpu
Domain-0                           0     4    -   --p       0.0  any cpu
Domain-0                           0     5    -   --p       0.0  any cpu
Domain-0                           0     6    -   --p       0.0  any cpu
Domain-0                           0     7    -   --p       0.0  any cpu
Domain-0                           0     8    -   --p       0.0  any cpu
Domain-0                           0     9    -   --p       0.0  any cpu
Domain-0                           0    10    -   --p       0.0  any cpu
Domain-0                           0    11    -   --p       0.0  any cpu
Domain-0                           0    12    -   --p       0.0  any cpu
Domain-0                           0    13    -   --p       0.0  any cpu
Domain-0                           0    14    -   --p       0.0  any cpu
Domain-0                           0    15    -   --p       0.0  any cpu
dom1                               1     0    2   -b-      21.0  2
dom2                               2     0    4   -b-      21.4  4
dom3                               3     0    6   -b-      20.8  6
dom4                               4     0    8   -b-      21.0  8
dom5                               5     0   10   -b-      21.5  10
dom6                               6     0   12   -b-      21.1  12
dom7                               7     0   14   -b-      20.5  14

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: credit scheduler

[Ian Pratt]

> In my most basic test I have a 4 socket dual core Intel box (Paxville)
with
> a
> uniprocessor dom0 and 7 uniprocessor domUs.  Each of the domUs is
pinned on
> its own core with the first core of the system left for dom0.  When
running
> the credit scheduler the dom0 VCPU will bounce around the system,
sometimes
> landing on the same thread as one of the domUs or sometimes on one of
the
> sibling hyperthreads (this appears to happen a majority of the time it
> moves).  This is less than ideal when considering cache warmth and the
> sharing of CPU resources when the first core of the system is always
> available in this configuration.  Does the credit scheduler have any
> awareness of cache warmth or CPU siblings when balancing?
> 
> I have also seen similar behavior when running tests in the domUs such
that
> each has its VCPU running at 100% utilization so I believe this
behavior to
> be fairly uniform.

The sometimes suboptimal use of hyperthreading is well understood and is
on the todo list. It hasn't been a priority as the current generation of
Intel Core/Core2 and Opteron CPUs don't have HT.

Apart from the hyperthreading sibling case, a dom0 vcpu should only ever
migrate to another CPU that has been idle for a complete rebalancing
period. Hence the rate of movement should be very low and thus the
overhead of priming the cache tiny.  It's conceivable one could
construct an argument that it's not worth the special case code in the
scheduler to fix...

BTW: The PhD thesis of our very own James Bulpin has lots of useful data
on cache warming times and how to optimize for HT systems:
http://www.cl.cam.ac.uk/TechReports/UCAM-CL-TR-619.pdf

Allowing dom0 vcpus to explicitly be pinned would certainly be a good
thing though, and is slated for post 3.0.3 -- see the thread on this
topic earlier today. Actually, in the interim it would be easy to add a
xen command line parameter to pin dom0 vcpus...

Ian

Re: credit scheduler

[Keir Fraser]

On 28/8/06 11:28 pm, "Karl Rister" <kmr@us.ibm.com> wrote:

> This is less than ideal when considering cache warmth and the
> sharing of CPU resources when the first core of the system is always
> available in this configuration.  Does the credit scheduler have any
> awareness of cache warmth or CPU siblings when balancing?

It doesn't right now, but in your setup dom0 has its 'home core' all to
itself, so it's odd that it's bouncing around. Other cores should only steal
the domain if it is runnable but not running -- that should almost never be
the case if the domain is not contending for CPU.

 -- Keir

Re: credit scheduler

[George Dunlap]

I've seen something similar... I have a cpu with two HT and two
domains (single-vcpu dom0 and an HVM domain), and I'm using the
VMEXIT/VMENTER tracing.  The most natural thing would be for each to
get its own logical cpu, but under the credit scheduler, the HVM
domain is seeing the VMENTER on a different logical cpu than it took
the VMEXIT on.  This makes the RDTSC values taken not comparable.

 -George

On 8/29/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>
>
>
> On 28/8/06 11:28 pm, "Karl Rister" <kmr@us.ibm.com> wrote:
>
> > This is less than ideal when considering cache warmth and the
> > sharing of CPU resources when the first core of the system is always
> > available in this configuration.  Does the credit scheduler have any
> > awareness of cache warmth or CPU siblings when balancing?
>
> It doesn't right now, but in your setup dom0 has its 'home core' all to
> itself, so it's odd that it's bouncing around. Other cores should only steal
> the domain if it is runnable but not running -- that should almost never be
> the case if the domain is not contending for CPU.
>
>  -- Keir
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: credit scheduler

[Sean Dague]

[-- Attachment #1.1: Type: text/plain, Size: 1694 bytes --]

On Mon, Aug 28, 2006 at 05:28:08PM -0500, Karl Rister wrote:
> While doing some performance analysis of Xen I have noticed some interesting 
> behavior of the credit scheduler that I would like to see discussed.
> 
> In my most basic test I have a 4 socket dual core Intel box (Paxville) with a 
> uniprocessor dom0 and 7 uniprocessor domUs.  Each of the domUs is pinned on 
> its own core with the first core of the system left for dom0.  When running 
> the credit scheduler the dom0 VCPU will bounce around the system, sometimes 
> landing on the same thread as one of the domUs or sometimes on one of the 
> sibling hyperthreads (this appears to happen a majority of the time it 
> moves).  This is less than ideal when considering cache warmth and the 
> sharing of CPU resources when the first core of the system is always 
> available in this configuration.  Does the credit scheduler have any 
> awareness of cache warmth or CPU siblings when balancing?

A related issue I noticed the other day when doing xen testing off of
xen-unstable was that the irqbalancer process ended up consuming 100% of
dom0 cpu when running an SMP Dom0 on the credit scheduler, and a bunch of
idle domUs.  It got the the point that ssh wasn't feasable, and I had to go
to the console and shut off irqbalancer to return things to normal.

I'd follow up further, but I'm about to go "off grid" for the next 2.5
weeks.  I relayed information to other folks on the IBM team to see if
anyone else can reproduce it.

	-Sean

-- 
Sean Dague
IBM Linux Technology Center                     email: japh@us.ibm.com
Open Hypervisor Team                           alt: sldague@us.ibm.com

[-- Attachment #1.2: Type: application/pgp-signature, Size: 191 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: credit scheduler

[Emmanuel Ackaouy]

On Mon, Aug 28, 2006 at 05:28:08PM -0500, Karl Rister wrote:
> In my testing I am looking for uniform behavior so I have just been setting 
> sched=sedf but I would like to move to credit scheduler since it is the new 
> default. However, until I sort out this behavior I cannot.

One thing you could do is restrict dom0's VCPU so that
it doesn't move around to a hyperthread on a core that
you have assigned to a domU.

So figure out what CPU dom0's VCPU is on and restrict
its cpumask so that it stays on that core. Then move
any domU out of that core.

Credit scheduler

[pak333]

[-- Attachment #1.1: Type: text/plain, Size: 494 bytes --]

Hi

I have noticed that VMs are typically spending 3 microsecs or less before they are being prempted. I thought that the credit schduler time slice was 30 ms. I have 4 VMs running and they are all cpu intensive except for 1 (which is IO intensive) but having a VM spend max 3 micro secs before being kicked out seems strange.

Is there something else going on that i am not aware of. Is the time slice really 30 millisecs? I am using default parameters of the credit scheduler.

Thanks
-Prabha

[-- Attachment #1.2: Type: text/html, Size: 646 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Credit scheduler

[Keir Fraser]

[-- Attachment #1.1: Type: text/plain, Size: 857 bytes --]

How many physical CPUs do you have, and what is the IO workload that one of
the VMs is running?

 -- Keir

On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote:

> Hi
>  
> I have noticed that VMs are typically spending 3 microsecs or less before they
> are being prempted. I thought that the credit schduler time slice was 30 ms. I
> have 4 VMs running and they are all cpu intensive except for 1 (which is IO
> intensive) but having a VM spend max 3 micro secs before being kicked out
> seems strange.
>  
> Is there something else going on that i am not aware of. Is the time slice
> really 30 millisecs? I am using default parameters of the credit scheduler.
>  
> Thanks
> -Prabha
>  
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



[-- Attachment #1.2: Type: text/html, Size: 1554 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Credit scheduler

[pak333]

[-- Attachment #1: Type: text/plain, Size: 1122 bytes --]


I have 4 physical cpus (2 dual core sockets) and the IO workload is loadsim with 500 users

Thanks
-Prabha


 -------------- Original message ----------------------
From: Keir Fraser <keir@xensource.com>
> How many physical CPUs do you have, and what is the IO workload that one of
> the VMs is running?
> 
>  -- Keir
> 
> On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote:
> 
> > Hi
> >  
> > I have noticed that VMs are typically spending 3 microsecs or less before they
> > are being prempted. I thought that the credit schduler time slice was 30 ms. I
> > have 4 VMs running and they are all cpu intensive except for 1 (which is IO
> > intensive) but having a VM spend max 3 micro secs before being kicked out
> > seems strange.
> >  
> > Is there something else going on that i am not aware of. Is the time slice
> > really 30 millisecs? I am using default parameters of the credit scheduler.
> >  
> > Thanks
> > -Prabha
> >  
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 



[-- Attachment #2: Type: message/rfc822, Size: 3564 bytes --]

[-- Attachment #2.1.1.1: Type: text/plain, Size: 857 bytes --]

How many physical CPUs do you have, and what is the IO workload that one of
the VMs is running?

 -- Keir

On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote:

> Hi
>  
> I have noticed that VMs are typically spending 3 microsecs or less before they
> are being prempted. I thought that the credit schduler time slice was 30 ms. I
> have 4 VMs running and they are all cpu intensive except for 1 (which is IO
> intensive) but having a VM spend max 3 micro secs before being kicked out
> seems strange.
>  
> Is there something else going on that i am not aware of. Is the time slice
> really 30 millisecs? I am using default parameters of the credit scheduler.
>  
> Thanks
> -Prabha
>  
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



[-- Attachment #2.1.1.2: Type: text/html, Size: 1554 bytes --]

[-- Attachment #2.1.2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Credit scheduler

[Keir Fraser]

[-- Attachment #1.1: Type: text/plain, Size: 1909 bytes --]

Are these all Windows HVM guests? And do you exclude domain0 when you count
the number of VMs in your system (so it¹s actually 4 user VMs + domain0 VM)?
How did you come up with the 3us figure ‹ some kind of in-guest tracing, or
using xentrace, or by some other means?

Does the problem still happen if you run 2 cpu-intensive VMs instead of 3?

 -- Keir

On 27/6/07 02:09, "pak333@comcast.net" <pak333@comcast.net> wrote:

> 
> I have 4 physical cpus (2 dual core sockets) and the IO workload is loadsim
> with 500 users
> 
> Thanks
> -Prabha
> 
> 
>  -------------- Original message ----------------------
> From: Keir Fraser <keir@xensource.com>
>> > How many physical CPUs do you have, and what is the IO workload that one of
>> > the VMs is running?
>> > 
>> >  -- Keir
>> > 
>> > On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote:
>> > 
>>> > > Hi
>>> > >  
>>> > > I have noticed that VMs are typically spending 3 microsecs or less
>>> before they
>>> > > are being prempted. I thought that the credit schduler time slice was 30
>>> ms. I
>>> > > have 4 VMs running and they are all cpu intensive except for 1 (which is
IO
>>> > > intensive) but having a VM spend max 3 micro secs before being kicked
out
>>> > > seems strange.
>>> > >  
>>> > > Is there something else going on that i am not aware of. Is the time >>>
slice
>>> > > really 30 millisecs? I am using default parameters of the credit
>>> scheduler.
>>> > >  
>>> > > Thanks
>>> > > -Prabha
>>> > >  
>>> > > 
>>> > > 
>>> > > _______________________________________________
>>> > > Xen-devel mailing list
>>> > > Xen-devel@lists.xensource.com
>>> > > http://lists.xensource.com/xen-devel
>> > 
>> > 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



[-- Attachment #1.2: Type: text/html, Size: 2735 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Credit scheduler

[pak333]

[-- Attachment #1: Type: text/plain, Size: 3148 bytes --]



Yes all these are HVM guests. 
Yes there is dom0 and 4 other VMs. 

I added some trace statements in the schedule.c function and then postprocessed that data to come with the 3us 

example I have 16400 lines printed with xentrace during a span of 52 ms. Which means that i have entered the scheduler 16400 times during the 52ms time interval, which is roughly 3.2 us.

i have not yet tried any other combination.

my understanding of the schduler is that the scheduler kicks in on a timer interrupt or a block of any VM.
so in a all cpu intensive VMs the VM would run interrupted for 30ms(credit schduler cap). 
since we have IO, whenever a VM needs IO service, it blocks and domain 0 kicks in and services the IO.
is this correct?

What I also see is that dom0 cpu utilization is only 28% across all 4 cpus. Assuming that the scheduler was entered because of an IO request, dom0 kicks in but spends very little time servicing the request. Dom0 kicks in 6800 times of the  16400 times that the scheduler function was called. Does this make sense?

Thanks
-Prabha









 -------------- Original message ----------------------
From: Keir Fraser <keir@xensource.com>
> Are these all Windows HVM guests? And do you exclude domain0 when you count
> the number of VMs in your system (so it¹s actually 4 user VMs + domain0 VM)?
> How did you come up with the 3us figure ‹ some kind of in-guest tracing, or
> using xentrace, or by some other means?
> 
> Does the problem still happen if you run 2 cpu-intensive VMs instead of 3?
> 
>  -- Keir
> 
> On 27/6/07 02:09, "pak333@comcast.net" <pak333@comcast.net> wrote:
> 
> > 
> > I have 4 physical cpus (2 dual core sockets) and the IO workload is loadsim
> > with 500 users
> > 
> > Thanks
> > -Prabha
> > 
> > 
> >  -------------- Original message ----------------------
> > From: Keir Fraser <keir@xensource.com>
> >> > How many physical CPUs do you have, and what is the IO workload that one of
> >> > the VMs is running?
> >> > 
> >> >  -- Keir
> >> > 
> >> > On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote:
> >> > 
> >>> > > Hi
> >>> > >  
> >>> > > I have noticed that VMs are typically spending 3 microsecs or less
> >>> before they
> >>> > > are being prempted. I thought that the credit schduler time slice was 30
> >>> ms. I
> >>> > > have 4 VMs running and they are all cpu intensive except for 1 (which is
> IO
> >>> > > intensive) but having a VM spend max 3 micro secs before being kicked
> out
> >>> > > seems strange.
> >>> > >  
> >>> > > Is there something else going on that i am not aware of. Is the time >>>
> slice
> >>> > > really 30 millisecs? I am using default parameters of the credit
> >>> scheduler.
> >>> > >  
> >>> > > Thanks
> >>> > > -Prabha
> >>> > >  
> >>> > > 
> >>> > > 
> >>> > > _______________________________________________
> >>> > > Xen-devel mailing list
> >>> > > Xen-devel@lists.xensource.com
> >>> > > http://lists.xensource.com/xen-devel
> >> > 
> >> > 
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 
> 



[-- Attachment #2: Type: message/rfc822, Size: 5776 bytes --]

[-- Attachment #2.1.1.1: Type: text/plain, Size: 1909 bytes --]

Are these all Windows HVM guests? And do you exclude domain0 when you count
the number of VMs in your system (so it¹s actually 4 user VMs + domain0 VM)?
How did you come up with the 3us figure ‹ some kind of in-guest tracing, or
using xentrace, or by some other means?

Does the problem still happen if you run 2 cpu-intensive VMs instead of 3?

 -- Keir

On 27/6/07 02:09, "pak333@comcast.net" <pak333@comcast.net> wrote:

> 
> I have 4 physical cpus (2 dual core sockets) and the IO workload is loadsim
> with 500 users
> 
> Thanks
> -Prabha
> 
> 
>  -------------- Original message ----------------------
> From: Keir Fraser <keir@xensource.com>
>> > How many physical CPUs do you have, and what is the IO workload that one of
>> > the VMs is running?
>> > 
>> >  -- Keir
>> > 
>> > On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote:
>> > 
>>> > > Hi
>>> > >  
>>> > > I have noticed that VMs are typically spending 3 microsecs or less
>>> before they
>>> > > are being prempted. I thought that the credit schduler time slice was 30
>>> ms. I
>>> > > have 4 VMs running and they are all cpu intensive except for 1 (which is
IO
>>> > > intensive) but having a VM spend max 3 micro secs before being kicked
out
>>> > > seems strange.
>>> > >  
>>> > > Is there something else going on that i am not aware of. Is the time >>>
slice
>>> > > really 30 millisecs? I am using default parameters of the credit
>>> scheduler.
>>> > >  
>>> > > Thanks
>>> > > -Prabha
>>> > >  
>>> > > 
>>> > > 
>>> > > _______________________________________________
>>> > > Xen-devel mailing list
>>> > > Xen-devel@lists.xensource.com
>>> > > http://lists.xensource.com/xen-devel
>> > 
>> > 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



[-- Attachment #2.1.1.2: Type: text/html, Size: 2735 bytes --]

[-- Attachment #2.1.2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Credit scheduler

[Atsushi SAKAI]

Hi,　Prabha

　30msec is the maximum time slice.
And I guess your 3 microsecond is the response after 
SCHEDULE_SOFTIRQ is set.
As you know,
If SCHEDULE_SOFTIRQ is set, 
it waits softirq calls schedule(). (this time interval is 3microsec) 

And I/O intensive  has higher priority than CPU intensive.
So I/O intensive job is first dispatched domain in runq.
This is because latency improvement for I/O intensive guest. 

So your behavior is not strange.

Thanks
Atsushi SAKAI


pak333@comcast.net wrote:

> Hi
> 
> I have noticed that VMs are typically spending 3 microsecs or less before they are being prempted. I thought that the credit schduler time slice was 30 ms. I have 4 VMs running and they are all cpu intensive except for 1 (which is IO intensive) but having a VM spend max 3 micro secs before being kicked out seems strange.
> 
> Is there something else going on that i am not aware of. Is the time slice really 30 millisecs? I am using default parameters of the credit scheduler.
> 
> Thanks
> -Prabha

Re: Credit scheduler

[pak333]

[-- Attachment #1.1: Type: text/plain, Size: 1488 bytes --]

Thanks for the explanation, i am not sure i understood it correctly.


when does softirq get set. So when softirq is set a processor enters the scheduler every 3 micro secs? where in the source code is this handled. please could you send me some pointers

thanks
-Prabha




-------------- Original message -------------- 
From: Atsushi SAKAI <sakaia@jp.fujitsu.com> 

> Hi,^[$B!!^[(BPrabha 
> 
> ^[$B!!^[(B30msec is the maximum time slice. 
> And I guess your 3 microsecond is the response after 
> SCHEDULE_SOFTIRQ is set. 
> As you know, 
> If SCHEDULE_SOFTIRQ is set, 
> it waits softirq calls schedule(). (this time interval is 3microsec) 
> 
> And I/O intensive has higher priority than CPU intensive. 
> So I/O intensive job is first dispatched domain in runq. 
> This is because latency improvement for I/O intensive guest. 
> 
> So your behavior is not strange. 
> 
> Thanks 
> Atsushi SAKAI 
> 
> 
> pak333@comcast.net wrote: 
> 
> > Hi 
> > 
> > I have noticed that VMs are typically spending 3 microsecs or less before they 
> are being prempted. I thought that the credit schduler time slice was 30 ms. I 
> have 4 VMs running and they are all cpu intensive except for 1 (which is IO 
> intensive) but having a VM spend max 3 micro secs before being kicked out seems 
> strange. 
> > 
> > Is there something else going on that i am not aware of. Is the time slice 
> really 30 millisecs? I am using default parameters of the credit scheduler. 
> > 
> > Thanks 
> > -Prabha 
> 
> 

[-- Attachment #1.2: Type: text/html, Size: 2035 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Credit scheduler

[Atsushi SAKAI]

Hi Prabha

Please check do_softirq in assembler.
for x86 
call do_softirq 
is in.

And usually do_softirq is executed at the end of system call.

Thanks
Atsushi SAKAI



pak333@comcast.net wrote:

> Thanks for the explanation, i am not sure i understood it correctly.
> 
> 
> when does softirq get set. So when softirq is set a processor enters the scheduler every 3 micro secs? where in the source code is this handled. please could you send me some pointers
> 
> thanks
> -Prabha
> 
> 
> 
> 
> -------------- Original message -------------- 
> From: Atsushi SAKAI <sakaia@jp.fujitsu.com> 
> 
> > Hi,　Prabha 
> > 
> > 　30msec is the maximum time slice. 
> > And I guess your 3 microsecond is the response after 
> > SCHEDULE_SOFTIRQ is set. 
> > As you know, 
> > If SCHEDULE_SOFTIRQ is set, 
> > it waits softirq calls schedule(). (this time interval is 3microsec) 
> > 
> > And I/O intensive has higher priority than CPU intensive. 
> > So I/O intensive job is first dispatched domain in runq. 
> > This is because latency improvement for I/O intensive guest. 
> > 
> > So your behavior is not strange. 
> > 
> > Thanks 
> > Atsushi SAKAI 
> > 
> > 
> > pak333@comcast.net wrote: 
> > 
> > > Hi 
> > > 
> > > I have noticed that VMs are typically spending 3 microsecs or less before they 
> > are being prempted. I thought that the credit schduler time slice was 30 ms. I 
> > have 4 VMs running and they are all cpu intensive except for 1 (which is IO 
> > intensive) but having a VM spend max 3 micro secs before being kicked out seems 
> > strange. 
> > > 
> > > Is there something else going on that i am not aware of. Is the time slice 
> > really 30 millisecs? I am using default parameters of the credit scheduler. 
> > > 
> > > Thanks 
> > > -Prabha 
> > 
> > 

Re: Credit scheduler

[Keir Fraser]

[-- Attachment #1.1: Type: text/plain, Size: 4064 bytes --]

You¹ll probably do better by pinning dom0 to cpu0 and all other domains to
all other cpus.

 -- Keir

On 27/6/07 02:55, "pak333@comcast.net" <pak333@comcast.net> wrote:

> 
> 
> Yes all these are HVM guests.
> Yes there is dom0 and 4 other VMs.
> 
> I added some trace statements in the schedule.c function and then
> postprocessed that data to come with the 3us
> 
> example I have 16400 lines printed with xentrace during a span of 52 ms. Which
> means that i have entered the scheduler 16400 times during the 52ms time
> interval, which is roughly 3.2 us.
> 
> i have not yet tried any other combination.
> 
> my understanding of the schduler is that the scheduler kicks in on a timer
> interrupt or a block of any VM.
> so in a all cpu intensive VMs the VM would run interrupted for 30ms(credit
> schduler cap). 
> since we have IO, whenever a VM needs IO service, it blocks and domain 0 kicks
> in and services the IO.
> is this correct?
> 
> What I also see is that dom0 cpu utilization is only 28% across all 4 cpus.
> Assuming that the scheduler was entered because of an IO request, dom0 kicks
> in but spends very little time servicing the request. Dom0 kicks in 6800 times
> of the  16400 times that the scheduler function was called. Does this make
> sense?
> 
> Thanks
> -Prabha
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  -------------- Original message ----------------------
> From: Keir Fraser <keir@xensource.com>
>> > Are these all Windows HVM guests? And do you exclude domain0 when you count
>> > the number of VMs in your system (so it¹s actually 4 user VMs + domain0
>> VM)?
>> > How did you come up with the 3us figure ‹ some kind of in-guest tracing, or
>> > using xentrace, or by some other means?
>> > 
>> > Does the problem still happen if you run 2 cpu-intensive VMs instead of 3?
>> > 
>> >  -- Keir
>> > 
>> > On 27/6/07 02:09, "pak333@comcast.net" <pak333@comcast.net> wrote:
>> > 
>>> > > 
>>> > > I have 4 physical cpus (2 dual core sockets) and the IO workload is
>>> loadsim
>>> > > with 500 users
>>> > > 
>>> > > Thanks
>>> > > -Prabha
>>> > > 
>>> > > 
>>> > >  -------------- Original message ----------------------
>>> > > From: Keir Fraser <keir@xensource.com>
>>>>> > >> > How many physical CPUs do you have, and what is the IO workload
>>>>> that one of
>>>>> > >> > the VMs is running?
>>>>> > >> > 
>>>>> > >> >  -- Keir
>>>>> > >> > 
>>>>> > >> > On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote:
>>>>> > >> > 
>>>>>>> > >>> > > Hi
>>>>>>> > >>> > >  
>>>>>>> > >>> > > I have noticed that VMs are typically spending 3 microsecs or
less
>>>>> > >>> before they
>>>>>>> > >>> > > are being prempted. I thought that the credit schduler time
>>>>>>> slice was 30
>>>>> > >>> ms. I
>>>>>>> > >>> > > have 4 VMs running and they are all cpu intensive except for 1
(which is
>> > IO
>>>>>>> > >>> > > intensive) but having a VM spend max 3 micro secs before being
kicked
>> > out
>>>>>>> > >>> > > seems strange.
>>>>>>> > >>> > >  
>>>>>>> > >>> > > Is there something else going on that i am not aware of. Is
>>>>>>> the time >>>
>> > slice
>>>>>>> > >>> > > really 30 millisecs? I am using default parameters of the
credit
>>>>> > >>> scheduler.
>>>>>>> > >>> > >  
>>>>>>> > >>> > > Thanks
>>>>>>> > >>> > > -Prabha
>>>>>>> > >>> > >  
>>>>>>> > >>> > > 
>>>>>>> > >>> > > 
>>>>>>> > >>> > > _______________________________________________
>>>>>>> > >>> > > Xen-devel mailing list
>>>>>>> > >>> > > Xen-devel@lists.xensource.com
>>>>>>> > >>> > > http://lists.xensource.com/xen-devel
>>>>> > >> > 
>>>>> > >> > 
>>> > > 
>>> > > 
>>> > > 
>>> > > _______________________________________________
>>> > > Xen-devel mailing list
>>> > > Xen-devel@lists.xensource.com
>>> > > http://lists.xensource.com/xen-devel
>> > 
>> > 
>> > 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



[-- Attachment #1.2: Type: text/html, Size: 5405 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Credit scheduler

[pak333]

[-- Attachment #1.1: Type: text/plain, Size: 3551 bytes --]

Hi Atsushi 

I am still confused, so let me explain what i think should happen and understand from you why it may not happen.

I have 2 cpu intensive VMs and 1 IO intensive VM. My system has 4 physical cpus and 8 virtual cpus.

Now using default parameters for the credit scheduler, the cpu intersive vcpu should run for 30 millsec. It does not, it runs for less (few microsecs).

You said this could be because of softirq which are raised after a system call. Can i disable the softirqs. What happens if i do? If i cannot disable is there a way to see what VM is raising the softirq. If the cpu-VM raises the softirq it gets preempted or does it continue to run. How can I montior the softirqs raised by this VM. Can I tell if the premption is from a softirq or from something else. 

Also, if the IO VM requests an IO, it will block and dom0 wakes up and gets scheduled to run to service the IO. Will it preempt the cpu intensive VM. If so why? Shouldn't the cpuintensiveV M get its quantum of time. 
Or does the IOVM get higher priority to preempt the cpu intensive VM. How does the scheduler  pick which cpu to run dom0 on ( if all the vcpus running are cpu intensive)? If there is a mix of cpu-vcpus and Io-vcpus, which will be the victim

As you see i have a lot of questions, so please bear with me and help me understand the scheduling behavior of Xen

Thanks
Prabha




-------------- Original message -------------- 
From: Atsushi SAKAI <sakaia@jp.fujitsu.com> 

> Hi Prabha 
> 
> Please check do_softirq in assembler. 
> for x86 
> call do_softirq 
> is in. 
> 
> And usually do_softirq is executed at the end of system call. 
> 
> Thanks 
> Atsushi SAKAI 
> 
> 
> 
> pak333@comcast.net wrote: 
> 
> > Thanks for the explanation, i am not sure i understood it correctly. 
> > 
> > 
> > when does softirq get set. So when softirq is set a processor enters the 
> scheduler every 3 micro secs? where in the source code is this handled. please 
> could you send me some pointers 
> > 
> > thanks 
> > -Prabha 
> > 
> > 
> > 
> > 
> > -------------- Original message -------------- 
> > From: Atsushi SAKAI 
> > 
> > > Hi,^[$B!!^[(BPrabha 
> > > 
> > > ^[$B!!^[(B30msec is the maximum time slice. 
> > > And I guess your 3 microsecond is the response after 
> > > SCHEDULE_SOFTIRQ is set. 
> > > As you know, 
> > > If SCHEDULE_SOFTIRQ is set, 
> > > it waits softirq calls schedule(). (this time interval is 3microsec) 
> > > 
> > > And I/O intensive has higher priority than CPU intensive. 
> > > So I/O intensive job is first dispatched domain in runq. 
> > > This is because latency improvement for I/O intensive guest. 
> > > 
> > > So your behavior is not strange. 
> > > 
> > > Thanks 
> > > Atsushi SAKAI 
> > > 
> > > 
> > > pak333@comcast.net wrote: 
> > > 
> > > > Hi 
> > > > 
> > > > I have noticed that VMs are typically spending 3 microsecs or less before 
> they 
> > > are being prempted. I thought that the credit schduler time slice was 30 ms. 
> I 
> > > have 4 VMs running and they are all cpu intensive except for 1 (which is IO 
> > > intensive) but having a VM spend max 3 micro secs before being kicked out 
> seems 
> > > strange. 
> > > > 
> > > > Is there something else going on that i am not aware of. Is the time slice 
> > > really 30 millisecs? I am using default parameters of the credit scheduler. 
> > > > 
> > > > Thanks 
> > > > -Prabha 
> > > 
> > > 
> 
> 
> 
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xensource.com 
> http://lists.xensource.com/xen-devel 

[-- Attachment #1.2: Type: text/html, Size: 4786 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Credit scheduler

[Mats Petersson]

At 21:46 12/07/2007, pak333@comcast.net wrote:
>Hi Atsushi
>
>I am still confused, so let me explain what i think should happen 
>and understand from you why it may not happen.
>
>I have 2 cpu intensive VMs and 1 IO intensive VM. My system has 4 
>physical cpus and 8 virtual cpus.
>
>Now using default parameters for the credit scheduler, the cpu 
>intersive vcpu should run for 30 millsec. It does not, it runs for 
>less (few microsecs).
>
>You said this could be because of softirq which are raised after a 
>system call. Can i disable the softirqs. What happens if i do? If i 
>cannot disable is there a way to see what VM is raising the softirq. 
>If the cpu-VM raises the softirq it gets preempted or does it 
>continue to run. How can I montior the softirqs raised by this VM. 
>Can I tell if the premption is from a softirq or from something else.

I don't believe you can "disable" a softirq - you could of course 
avoid doing the call to "do_softirq", but that is most likely just 
going to completely remove any scheduling capability at all in the 
system, making the system "single-tasking". Not a recommended way, in 
my opinion.

>
>Also, if the IO VM requests an IO, it will block and dom0 wakes up 
>and gets scheduled to run to service the IO. Will it preempt the cpu 
>intensive VM. If so why? Shouldn't the cpuintensiveV M get its 
>quantum of time.
>Or does the IOVM get higher priority to preempt the cpu intensive 
>VM. How does the scheduler  pick which cpu to run dom0 on ( if all 
>the vcpus running are cpu intensive)? If there is a mix of cpu-vcpus 
>and Io-vcpus, which will be the victim


The whole idea of having separate queues for IO and CPU intensive VMs 
(or processes in a normal kernel scenario) is to allow the 
IO-intensive tasks to avoid waiting for the next time-slot when a 
CPU-hogging VM/process is running  [1]. The normal behaviour for the 
scheduler is to determine dynamically if the current VM/process is IO 
or CPU intensive.

Unless Dom0 is used for doing some silly CPU-intensive task 
(calculating PI with a million decimal points or compiling Xen + 
Linux kernel, for example), it will most likely look to the scheduler 
like a IO-intensive VM. So it will have the same priority as any 
other IO-intensive VM (assuming Dom0 has equal scheduler parameters 
as other guests, which I'm pretty sure is the default behaviour).

If, like in your example, there are a number of processors busy with 
CPU-intensive tasks, and others with IO-intensive tasks, the most 
likely scenario is that any new IO-request will go be run on a 
"previously CPU-intensive" CPU - but on the other hand, if you have a 
lot of IO-intensive processing, there should be some processor(s) 
that are "asleep" - unless all CPU's are busy running CPU-intensive tasks...

Note that scheduling is a complex subject, and there have been dozens 
of PhD dissertations written on the subject.

[1] The idea being that if we process the IO-request that just 
completed, we can set up another IO-request, and this will get better 
IO-throughput than waiting a long time (relatively speaking 30ms is 
an eternity to the processor).

--
Mats
>
>As you see i have a lot of questions, so please bear with me and 
>help me understand the scheduling behavior of Xen
>
>Thanks
>Prabha
>
>
>
>
>-------------- Original message --------------
>From: Atsushi SAKAI <sakaia@jp.fujitsu.com>
>
> > Hi Prabha
> >
> > Please check do_softirq in assembler.
> > for x86
> > call do_softirq
> > is in.
> >
> > And usually do_softirq is executed at the end of system call.
> >
> > Thanks
> > Atsushi SAKAI
> >
> >
> >
> > pak333@comcast.net wrote:
> >
> > > Thanks for the explanation, i am not sure i understood it correctly.
> > >
> > >
> > > when does softirq get set. So when softirq is set a processor enters the
> > scheduler every 3 micro secs? where in the source code is this 
> handled. please
> > could you send me some pointers
> > >
> > > thanks
> > > -Prabha
> > >
> > >
> > >
> > >
> > > -------------- Or iginal message --------------
> > > From: Atsushi SAKAI
> > >
> > > > Hi,$B!!(BPrabha
> > > >
> > > > $B!!(B30msec is the maximum time slice.
> > > > And I guess your 3 microsecond is the response after
> > > > SCHEDULE_SOFTIRQ is set.
> > > > As you know,
> > > > If SCHEDULE_SOFTIRQ is set,
> > > > it waits softirq calls schedule(). (this time interval is 3microsec)
> > > >
> > > > And I/O intensive has higher priority than CPU intensive.
> > > > So I/O intensive job is first dispatched domain in runq.
> > > > This is because latency improvement for I/O intensive guest.
> > > >
> > > > So your behavior is not strange.
> > > >
> > > > Thanks
> > > > Atsushi SAKAI
> > > >
> > > >
> > > > pak333@comcast.net wrote:
> > &g
>t; >
> > > > > Hi
> > > > >
> > > > > I have noticed that VMs are typically spending 3 microsecs 
> or less before
> > they
> > > > are being prempted. I thought that the credit schduler time 
> slice was 30 ms.
> > I
> > > > have 4 VMs running and they are all cpu intensive except for 
> 1 (which is IO
> > > > intensive) but having a VM spend max 3 micro secs before 
> being kicked out
> > seems
> > > > strange.
> > > > >
> > > > > Is there something else going on that i am not aware of. Is 
> the time slice
> > > > really 30 millisecs? I am using default parameters of the 
> credit scheduler.
> > > > >
> > > > > Thanks
> > > > > -Prabha
> > > >
> > > >
> >
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen source.com
> > http://lists.xensource.com/xen-devel
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

Re: Credit scheduler

[pak333]

[-- Attachment #1.1: Type: text/plain, Size: 6770 bytes --]

Thanks for the explanations. Some more questions

Please see my comments inline:
Thanks
Prabha

-------------- Original message -------------- 
From: Mats Petersson <mats@planetcatfish.com> 

> At 21:46 12/07/2007, pak333@comcast.net wrote: 
> >Hi Atsushi 
> > 
> >I am still confused, so let me explain what i think should happen 
> >and understand from you why it may not happen. 
> > 
> >I have 2 cpu intensive VMs and 1 IO intensive VM. My system has 4 
> >physical cpus and 8 virtual cpus. 
> > 
> >Now using default parameters for the credit scheduler, the cpu 
> >intersive vcpu should run for 30 millsec. It does not, it runs for 
> >less (few microsecs). 
> > 
> >You said this could be because of softirq which are raised after a 
> >system call. Can i disable the softirqs. What happens if i do? If i 
> >cannot disable is there a way to see what VM is raising the softirq. 
> >If the cpu-VM raises the softirq it gets preempted or does it 
> >continue to run. How can I montior the softirqs raised by this VM. 
> >Can I tell if the premption is from a softirq or from something else. 
> 
> I don't believe you can "disable" a softirq - you could of course 
> avoid doing the call to "do_softirq", but that is most likely just 
> going to completely remove any scheduling capability at all in the 
> system, making the system "single-tasking". Not a recommended way, in 
> my opinion. 
> 
Agreed
if softirq is indeed a problem, and if a cpu-intensive VM is being preempted because of a softirq, then maybe affintizing the interrupts of dom0 to a pcpu will reduce this preemption. Comments?
> > 
> >Also, if the IO VM requests an IO, it will block and dom0 wakes up 
> >and gets scheduled to run to service the IO. Will it preempt the cpu 
> >intensive VM. If so why? Shouldn't the cpuintensiveV M get its 
> >quantum of time. 
> >Or does the IOVM get higher priority to preempt the cpu intensive 
> >VM. How does the scheduler pick which cpu to run dom0 on ( if all 
> >the vcpus running are cpu intensive)? If there is a mix of cpu-vcpus 
> >and Io-vcpus, which will be the victim 
> 
> 
> The whole idea of having separate queues for IO and CPU intensive VMs 
> (or processes in a normal kernel scenario) is to allow the 
> IO-intensive tasks to avoid waiting for the next time-slot when a 
> CPU-hogging VM/process is running [1]. The normal behaviour for the 
> scheduler is to determine dynamically if the current VM/process is IO 
> or CPU intensive. 
> 
> Unless Dom0 is used for doing some silly CPU-intensive task 
> (calculating PI with a million decimal points or compiling Xen + 
> Linux kernel, for example), it will most likely look to the scheduler 
> like a IO-intensive VM. So it will have the same priority as any 
> other IO-intensive VM (assuming Dom0 has equal scheduler parameters 
> as other guests, which I'm pretty sure is the default behaviour). 
> 
Does dom0 have a higher pritority than any CPU intensive VM
> If, like in your example, there are a number of processors busy with 
> CPU-intensive tasks, and others with IO-intensive tasks, the most 
> likely scenario is that any new IO-request will go be run on a 
> "previously CPU-intensive" CPU - but on the other hand, if you have a 
> lot of IO-intensive processing, there should be some processor(s) 
> that are "asleep" - unless all CPU's are busy running CPU-intensive tasks... 
But will it preempt a CPU intensive vcpu if no cpu is available
> 
> Note that scheduling is a complex subject, and there have been dozens 
> of PhD dissertations written on the subject. 
Yes, but what does the Xen credit scheduler actually do
> 
> [1] The idea being that if we process the IO-request that just 
> completed, we can set up another IO-request, and this will get better 
> IO-throughput than waiting a long time (relatively speaking 30ms is 
> an eternity to the processor). 
Don't understand what you mean here.

> 
> -- 
> Mats 
> > 
> >As you see i have a lot of questions, so please bear with me and 
> >help me understand the scheduling behavior of Xen 
> > 
> >Thanks 
> >Prabha 
> > 
> > 
> > 
> > 
> >-------------- Original message -------------- 
> >From: Atsushi SAKAI 
> > 
> > > Hi Prabha 
> > > 
> > > Please check do_softirq in assembler. 
> > > for x86 
> > > call do_softirq 
> > > is in. 
> > > 
> > > And usually do_softirq is executed at the end of system call. 
> > > 
> > > Thanks 
> > > Atsushi SAKAI 
> > > 
> > > 
> > > 
> > > pak333@comcast.net wrote: 
> > > 
> > > > Thanks for the explanation, i am not sure i understood it correctly. 
> > > > 
> > > > 
> > > > when does softirq get set. So when softirq is set a processor enters the 
> > > scheduler every 3 micro secs? where in the source code is this 
> > handled. please 
> > > could you send me some pointers 
> > > > 
> > > > thanks 
> > > > -Prabha 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > -------------- Or iginal message -------------- 
> > > > From: Atsushi SAKAI 
> > > > 
> > > > > Hi,$B!!(BPrabha 
> > > > > 
> > > > > $B!!(B30msec is the maximum time slice. 
> > > > > And I guess your 3 microsecond is the response after 
> > > > > SCHEDULE_SOFTIRQ is set. 
> > > > > As you know, 
> > > > > If SCHEDULE_SOFTIRQ is set, 
> > > > > it waits softirq calls schedule(). (this time interval is 3microsec) 
> > > > > 
> > > > > And I/O intensive has higher priority than CPU intensive. 
> > > > > So I/O intensive job is first dispatched domain in runq. 
> > > > > This is because latency improvement for I/O intensive guest. 
> > > > > 
> > > > > So your behavior is not strange. 
> > > > > 
> > > > > Thanks 
> > > > > Atsushi SAKAI 
> > > > > 
> > > > > 
> > > > > pak333@comcast.net wrote: 
> > > &g 
> >t; > 
> > > > > > Hi 
> > > > > > 
> > > > > > I have noticed that VMs are typically spending 3 microsecs 
> > or less before 
> > > they 
> > > > > are being prempted. I thought that the credit schduler time 
> > slice was 30 ms. 
> > > I 
> > > > > have 4 VMs running and they are all cpu intensive except for 
> > 1 (which is IO 
> > > > > intensive) but having a VM spend max 3 micro secs before 
> > being kicked out 
> > > seems 
> > > > > strange. 
> > > > > > 
> > > > > > Is there something else going on that i am not aware of. Is 
> > the time slice 
> > > > > really 30 millisecs? I am using default parameters of the 
> > credit scheduler. 
> > > > > > 
> > > > > > Thanks 
> > > > > > -Prabha 
> > > > > 
> > > > > 
> > > 
> > > 
> > > 
> > > _______________________________________________ 
> > > Xen-devel mailing list 
> > > Xen-devel@lists.xen source.com 
> > > http://lists.xensource.com/xen-devel 
> > 
> >_______________________________________________ 
> >Xen-devel mailing list 
> >Xen-devel@lists.xensource.com 
> >http://lists.xensource.com/xen-devel 
> 

[-- Attachment #1.2: Type: text/html, Size: 9391 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Credit scheduler

[Atsushi SAKAI]

Hi, Prabha

pak333@comcast.net wrote:

[snip]

> > >Also, if the IO VM requests an IO, it will block and dom0 wakes up 
> > >and gets scheduled to run to service the IO. Will it preempt the cpu 
> > >intensive VM. If so why? Shouldn't the cpuintensiveV M get its 
> > >quantum of time. 
> > >Or does the IOVM get higher priority to preempt the cpu intensive 
> > >VM. How does the scheduler pick which cpu to run dom0 on ( if all 
> > >the vcpus running are cpu intensive)? If there is a mix of cpu-vcpus 
> > >and Io-vcpus, which will be the victim 
> > 
> > 
> > The whole idea of having separate queues for IO and CPU intensive VMs 
> > (or processes in a normal kernel scenario) is to allow the 
> > IO-intensive tasks to avoid waiting for the next time-slot when a 
> > CPU-hogging VM/process is running [1]. The normal behaviour for the 
> > scheduler is to determine dynamically if the current VM/process is IO 
> > or CPU intensive. 
> > 
> > Unless Dom0 is used for doing some silly CPU-intensive task 
> > (calculating PI with a million decimal points or compiling Xen + 
> > Linux kernel, for example), it will most likely look to the scheduler 
> > like a IO-intensive VM. So it will have the same priority as any 
> > other IO-intensive VM (assuming Dom0 has equal scheduler parameters 
> > as other guests, which I'm pretty sure is the default behaviour). 
> > 
> Does dom0 have a higher pritority than any CPU intensive VM

Easy way is to set higher weight to dom0.
(This makes get higher priority like I/O intensive domain for domain 
dispatch) 

If you want to set higher priority of Dom0 see follow patch.
(This is sample patch for boost Dom0 only not boost I/O intensive.)
http://lists.xensource.com/archives/html/xen-devel/2007-05/msg00529.html

Or just set  CSCHED_PRI_TS_BOOST   for domain0
in case dom0 priority is  CSCHED_PRI_TS_UNDER.


> > If, like in your example, there are a number of processors busy with 
> > CPU-intensive tasks, and others with IO-intensive tasks, the most 
> > likely scenario is that any new IO-request will go be run on a 
> > "previously CPU-intensive" CPU - but on the other hand, if you have a 
> > lot of IO-intensive processing, there should be some processor(s) 
> > that are "asleep" - unless all CPU's are busy running CPU-intensive tasks... 
> But will it preempt a CPU intensive vcpu if no cpu is available

[snip] 

> > 
> > [1] The idea being that if we process the IO-request that just 
> > completed, we can set up another IO-request, and this will get better 
> > IO-throughput than waiting a long time (relatively speaking 30ms is 
> > an eternity to the processor). 
> Don't understand what you mean here.

I guess Mats suggests to use AIO.

Thanks
Atsushi SAKAI

Re: Credit scheduler

[Mats Petersson]

At 08:30 13/07/2007, Atsushi SAKAI wrote:
>Hi, Prabha
>
>pak333@comcast.net wrote:
>
>[snip]
>
> > > >Also, if the IO VM requests an IO, it will block and dom0 wakes up
> > > >and gets scheduled to run to service the IO. Will it preempt the cpu
> > > >intensive VM. If so why? Shouldn't the cpuintensiveV M get its
> > > >quantum of time.
> > > >Or does the IOVM get higher priority to preempt the cpu intensive
> > > >VM. How does the scheduler pick which cpu to run dom0 on ( if all
> > > >the vcpus running are cpu intensive)? If there is a mix of cpu-vcpus
> > > >and Io-vcpus, which will be the victim
> > >
> > >
> > > The whole idea of having separate queues for IO and CPU intensive VMs
> > > (or processes in a normal kernel scenario) is to allow the
> > > IO-intensive tasks to avoid waiting for the next time-slot when a
> > > CPU-hogging VM/process is running [1]. The normal behaviour for the
> > > scheduler is to determine dynamically if the current VM/process is IO
> > > or CPU intensive.
> > >
> > > Unless Dom0 is used for doing some silly CPU-intensive task
> > > (calculating PI with a million decimal points or compiling Xen +
> > > Linux kernel, for example), it will most likely look to the scheduler
> > > like a IO-intensive VM. So it will have the same priority as any
> > > other IO-intensive VM (assuming Dom0 has equal scheduler parameters
> > > as other guests, which I'm pretty sure is the default behaviour).
> > >
> > Does dom0 have a higher pritority than any CPU intensive VM
>
>Easy way is to set higher weight to dom0.
>(This makes get higher priority like I/O intensive domain for domain
>dispatch)
>
>If you want to set higher priority of Dom0 see follow patch.
>(This is sample patch for boost Dom0 only not boost I/O intensive.)
>http://lists.xensource.com/archives/html/xen-devel/2007-05/msg00529.html
>
>Or just set  CSCHED_PRI_TS_BOOST   for domain0
>in case dom0 priority is  CSCHED_PRI_TS_UNDER.
>
>
> > > If, like in your example, there are a number of processors busy with
> > > CPU-intensive tasks, and others with IO-intensive tasks, the most
> > > likely scenario is that any new IO-request will go be run on a
> > > "previously CPU-intensive" CPU - but on the other hand, if you have a
> > > lot of IO-intensive processing, there should be some processor(s)
> > > that are "asleep" - unless all CPU's are busy running 
> CPU-intensive tasks...
> > But will it preempt a CPU intensive vcpu if no cpu is available
>
>[snip]
>
> > >
> > > [1] The idea being that if we process the IO-request that just
> > > completed, we can set up another IO-request, and this will get better
> > > IO-throughput than waiting a long time (relatively speaking 30ms is
> > > an eternity to the processor).
> > Don't understand what you mean here.
>
>I guess Mats suggests to use AIO.

Well, sort of. I meant the idea behind waking an IO-intensive process 
by pre-empting a CPU intensive process is that it improves the 
overall IO throughput, because very often, once one IO request is 
completed, another one will be requested. Hardware will take some 
time to proces each request, at which time it's fine to run the 
CPU-intensive process. If you prevent the IO intensive task from 
running until the CPU-intensive task has exhausted it's time, it's 
obviously going to take longer before the next IO-request is started, 
and thus longer until it gets completed. It makes little difference 
here if the APPLICATION is asynchronously or synchronously issuing 
the IO-request (assuming that under asynchronous circumstances there 
is a LIMITED amount of AIO queue - else it becomes a CPU-intensive 
task building until it's issued all of the IO-requests and fall asleep).

--
Mats


>Thanks
>Atsushi SAKAI

Credit Scheduler

[gaurav somani]

[-- Attachment #1.1: Type: text/plain, Size: 2028 bytes --]

Hi all,


I am just trying to understand the credit scheduler in xen.
please correct me in below sequence .
There are four VMs named V1 to V4. each having 256 credits. t is the CPU
time.


                        V1                    V2
V3                       V4

t=0                      256                 256
256                    256
t=10                    156                 256
256                    256
t=20                     56                  256
256                    256
t=30                    -44                  256
256                    256

t=40                    -44                 156
256                    256
t=50                    -44                 56
256                    256
t=60                     -44                -44
256                    256

t=70                    -44                  -44
156                    256
t=80                    -44                  -44
56                    256
t=90                    -44                  -44               -44
                   256

t=100                    -44                  -44
-44                    156
t=110                    -44                  -44
-44                    56
t=120                    -44                  -44               -44
                   -44

Credits will be re assigned when some of all credits becomes negative.
what I understand from the BOOST state is. According to I/O event mapping in
event channel, if the domain corresponding to that I/O event is in UNDER
state than it can be upgraded to BOOST state which is highest priority state
in xen.
So currently running domain will be preempted and BOOSTED domain will
receive the CPU.

I am having some questions
(1) How many credits are deducted from a VM in a BOOSTed state?
(2) If a VM has the work only for 10 ms, then what happens? Will it go to
idle state? after 10 secs or it will consume whole 30 secs slice?

Please provide me the direction.

Regards

Gaurav somani
Dhirubhai Ambani Institute of ICT,
Gandhinagar
Gujarat, INDIA

[-- Attachment #1.2: Type: text/html, Size: 2367 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Credit Scheduler

[Michael xin]

Hi~

Gaurav Somani wrote:
> 
> Hi all,
> 
> 
> I am just trying to understand the credit scheduler in xen.
> please correct me in below sequence .
> There are four VMs named V1 to V4. each having 256 credits. t is the CPU
> time.
> 
> 
>                         V1                    V2
> V3                       V4
> 
> t=0                      256                 256
> 256                    256
> t=10                    156                 256
> 256                    256
> t=20                     56                  256
> 256                    256
> t=30                    -44                  256
> 256                    256
> 
> t=40                    -44                 156
> 256                    256
> t=50                    -44                 56
> 256                    256
> t=60                     -44                -44
> 256                    256
> 
> t=70                    -44                  -44
> 156                    256
> t=80                    -44                  -44
> 56                    256
> t=90                    -44                  -44               -44
>                    256
> 
> t=100                    -44                  -44
> -44                    156
> t=110                    -44                  -44
> -44                    56
> t=120                    -44                  -44               -44
>                    -44
> 
> Credits will be reassigned when some of all credits becomes negative.
>                            IS WRONG. Scheduler will do it every 30ms
> what I understand from the BOOST state is. According to I/O event mapping
> in
> event channel, if the domain corresponding to that I/O event is in UNDER
> state than it can be upgraded to BOOST state which is highest priority
> state
> in xen.
> So currently running domain will be preempted and BOOSTED domain will
> receive the CPU.
> 
> I am having some questions
> (1) How many credits are deducted from a VM in a BOOSTed state?     the
> same as other state
> (2) If a VM has the work only for 10 ms, then what happens? Will it go to
> idle state? after 10 secs or it will consume whole 30 secs slice?          
> former is right , I think
> 
> Please provide me the direction.
> 
> Regards
> 
> Gaurav somani
> Dhirubhai Ambani Institute of ICT,
> Gandhinagar
> Gujarat, INDIA
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 

-- 
View this message in context: http://www.nabble.com/Credit-Scheduler-tp23874583p24161521.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

credit scheduler

[David Xu]

[-- Attachment #1.1: Type: text/plain, Size: 114 bytes --]

Hi,

What's the meaning of runq_sort in csched_privates and runq_sort_last in
csched_pcpu? Thanks.

Regards,
Cong

[-- Attachment #1.2: Type: text/html, Size: 177 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: credit scheduler

[George Dunlap]

On Fri, 2011-06-17 at 15:48 +0100, David Xu wrote:
> Hi,
> 
> 
> What's the meaning of runq_sort in csched_privates and runq_sort_last
> in csched_pcpu? Thanks.

The hint can be found in this comment (xen/common/sched_credit.c:1140):

    /* Inform each CPU that its runq needs to be sorted */
    prv->runq_sort++;

Every 30ms, the "master" cpu will increment prv->runq_sort.

Every 10ms, each individual cpu will check prv->runq_sort, to see if it
has been incremented (by comparing it to runq_sort_last).  If it has
incermented, then the cpu will sort its runqueue by priority.

 -George

credit scheduler

[Pratik shinde]

[-- Attachment #1.1: Type: text/plain, Size: 323 bytes --]

sir,
 thank you for replaying my mail
   i am reading xen credit scheduler but i am not getting how credit
scheduler distribute the credits among virtual machines.or specifically how
it decides the credit to be given.sorry sir,this would be an insane question
but i was finding answer to this question since last two days.

[-- Attachment #1.2: Type: text/html, Size: 335 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: credit scheduler

[George Dunlap]

It's hard to know what kind of answer you want.  The credits are
distributed to vcpus in xen/common/sched_credit.c:csched_acct().  Do
you have a specific question about the algorithm there?

 -George

On Sun, Sep 18, 2011 at 4:21 PM, Pratik shinde <pracshi@gmail.com> wrote:
> sir,
>  thank you for replaying my mail
>    i am reading xen credit scheduler but i am not getting how credit
> scheduler distribute the credits among virtual machines.or specifically how
> it decides the credit to be given.sorry sir,this would be an insane question
> but i was finding answer to this question since last two days.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>

Credit scheduler

[Lasya Venneti]

[-- Attachment #1.1: Type: text/plain, Size: 321 bytes --]

Hi,

Can somebody please point me to a resource that talks about scheduler
design for VMs , in contrast with a standard scheduler ( I have consulted
Remzi, Operating Systems for this which is pretty easy to understand
conceptually ). I need to understand the working of the credit scheduler.

Thanks!

Sincerely,
Lasya V

[-- Attachment #1.2: Type: text/html, Size: 453 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Credit scheduler

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 922 bytes --]

On Wed, 2015-10-21 at 23:33 +0530, Lasya Venneti wrote:
> Hi, 
> 
> Can somebody please point me to a resource that talks about scheduler
> design for VMs , in contrast with a standard scheduler ( I have
> consulted Remzi, Operating Systems for this which is pretty easy to
> understand conceptually ). I need to understand the working of the
> credit scheduler. 
> 
For the records, we chatted on IRC. This is about Lasya being
interested in OPW. So we're putting this on hold, while she takes care
of fulfilling the other requirements of the OPW application.

We'll resurface the conversation, if necessary, after the program
starts.

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel