Credit Scheduler not working correct (3.0.4-0)

Credit Scheduler not working correct (3.0.4-0)

[Timo Benk]

Hi,

on my system, it seems that the credit scheduler does not work as expected. The
problem seems to be related to HyperThreading, if i switch off HT in the BIOS
Setup, the scheduler works as expected.

I have two domains running, both pinned to the same processor:

Domain-0:~ # xm vcpu-list
Name                              ID  VCPU   CPU State   Time(s) CPU
Affinity
Domain-0                           0     0     1   ---      30.2 any cpu
Domain-0                           0     1     1   r--      12.0 any cpu
xendom1                            1     0     0   ---      98.6 0
xendom2                            2     0     0   r--     110.3 0

Both domains should consume max. 100 Percent of that processor, but
xendom1 has a weight twice as big as xendom2:

Domain-0:~ # xm sched-credit -d xendom1
{'cap': 100, 'weight': 200}
Domain-0:~ # xm sched-credit -d xendom2
{'cap': 100, 'weight': 100}

In my understanding of the credit scheduler, xendom1 should get twice as
much CPU-time as xendom2. But:

xentop - 11:05:46   Xen 3.0.4-0
Mem: 2612284k total, 2603852k used, 8432k free    CPUs: 2 @ 2992MHz
      NAME  STATE   CPU(sec) CPU(%)  [...]
  Domain-0 -----r         42    0.6  [...]
   xendom1 ------        122   49.8  [...]
   xendom2 -----r        133   50.2  [...]

As you can see, both domains got an equal amount of the CPU.

Am i missing something or do i understand something wrong?

Greetings,
-timo
-- 
Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc

Re: Credit Scheduler not working correct (3.0.4-0)

[Atsushi SAKAI]

Hi, Timo

If you want to divide CPU resources as you expected,
each vcpu weight should be same.
In your case, you should define xendom1 for 1vcpu
and xendom2 for 2vcpu(s). then after you should pin these vcpus to pcpu0.

Thanks
Atsushi SAKAI


Timo Benk <timo.benk@gmx.de> wrote:

> Hi,
> 
> on my system, it seems that the credit scheduler does not work as expected. The
> problem seems to be related to HyperThreading, if i switch off HT in the BIOS
> Setup, the scheduler works as expected.
> 
> I have two domains running, both pinned to the same processor:
> 
> Domain-0:~ # xm vcpu-list
> Name                              ID  VCPU   CPU State   Time(s) CPU
> Affinity
> Domain-0                           0     0     1   ---      30.2 any cpu
> Domain-0                           0     1     1   r--      12.0 any cpu
> xendom1                            1     0     0   ---      98.6 0
> xendom2                            2     0     0   r--     110.3 0
> 
> Both domains should consume max. 100 Percent of that processor, but
> xendom1 has a weight twice as big as xendom2:
> 
> Domain-0:~ # xm sched-credit -d xendom1
> {'cap': 100, 'weight': 200}
> Domain-0:~ # xm sched-credit -d xendom2
> {'cap': 100, 'weight': 100}
> 
> In my understanding of the credit scheduler, xendom1 should get twice as
> much CPU-time as xendom2. But:
> 
> xentop - 11:05:46   Xen 3.0.4-0
> Mem: 2612284k total, 2603852k used, 8432k free    CPUs: 2 @ 2992MHz
>       NAME  STATE   CPU(sec) CPU(%)  [...]
>   Domain-0 -----r         42    0.6  [...]
>    xendom1 ------        122   49.8  [...]
>    xendom2 -----r        133   50.2  [...]
> 
> As you can see, both domains got an equal amount of the CPU.
> 
> Am i missing something or do i understand something wrong?
> 
> Greetings,
> -timo
> -- 
> Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
> PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Credit Scheduler not working correct (3.0.4-0)

[Timo Benk]

Atsushi SAKAI wrote:
> Hi, Timo
> 
> If you want to divide CPU resources as you expected,
> each vcpu weight should be same.
> In your case, you should define xendom1 for 1vcpu
> and xendom2 for 2vcpu(s). then after you should pin these vcpus to pcpu0.

Hm, is that a bug or a feature?

I thought the crdeit-schedulers primary goal was to avoid this pinning nightmare
you have with SEDF?

Greetings,
-timo

> Timo Benk <timo.benk@gmx.de> wrote:
> 
>> Hi,
>>
>> on my system, it seems that the credit scheduler does not work as expected. The
>> problem seems to be related to HyperThreading, if i switch off HT in the BIOS
>> Setup, the scheduler works as expected.
>>
>> I have two domains running, both pinned to the same processor:
>>
>> Domain-0:~ # xm vcpu-list
>> Name                              ID  VCPU   CPU State   Time(s) CPU
>> Affinity
>> Domain-0                           0     0     1   ---      30.2 any cpu
>> Domain-0                           0     1     1   r--      12.0 any cpu
>> xendom1                            1     0     0   ---      98.6 0
>> xendom2                            2     0     0   r--     110.3 0
>>
>> Both domains should consume max. 100 Percent of that processor, but
>> xendom1 has a weight twice as big as xendom2:
>>
>> Domain-0:~ # xm sched-credit -d xendom1
>> {'cap': 100, 'weight': 200}
>> Domain-0:~ # xm sched-credit -d xendom2
>> {'cap': 100, 'weight': 100}
>>
>> In my understanding of the credit scheduler, xendom1 should get twice as
>> much CPU-time as xendom2. But:
>>
>> xentop - 11:05:46   Xen 3.0.4-0
>> Mem: 2612284k total, 2603852k used, 8432k free    CPUs: 2 @ 2992MHz
>>       NAME  STATE   CPU(sec) CPU(%)  [...]
>>   Domain-0 -----r         42    0.6  [...]
>>    xendom1 ------        122   49.8  [...]
>>    xendom2 -----r        133   50.2  [...]
>>
>> As you can see, both domains got an equal amount of the CPU.
>>
>> Am i missing something or do i understand something wrong?
>>
>> Greetings,
>> -timo
>> -- 
>> Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
>> PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 
> 


-- 
Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc

Re: Credit Scheduler not working correct (3.0.4-0)

[Atsushi SAKAI]

Hi, Timo

Primary goal of Credit Scheduler is loadbalancing multiple CPU resources.
So if you pin vcpu, this condition is violated,
since vcpu cannot migrate between pcpu.

Anyway, Credit scheduler specification is defined by Emmanuel.
So I have no comment on it.

But, I hope scheduler instruction should be simple.(like vmware weight)
Current credit scheduler is very complex 
when we uses relative weight and vcpu pin with mixing these two parameter)

c.f. vmware uses absolute weight(like CPU freqency)

Thanks
Atsushi SAKAI



Timo Benk <timo.benk@gmx.de> wrote:

> Atsushi SAKAI wrote:
> > Hi, Timo
> > 
> > If you want to divide CPU resources as you expected,
> > each vcpu weight should be same.
> > In your case, you should define xendom1 for 1vcpu
> > and xendom2 for 2vcpu(s). then after you should pin these vcpus to pcpu0.
> 
> Hm, is that a bug or a feature?
> 
> I thought the crdeit-schedulers primary goal was to avoid this pinning nightmare
> you have with SEDF?
> 
> Greetings,
> -timo
> 
> > Timo Benk <timo.benk@gmx.de> wrote:
> > 
> >> Hi,
> >>
> >> on my system, it seems that the credit scheduler does not work as expected. The
> >> problem seems to be related to HyperThreading, if i switch off HT in the BIOS
> >> Setup, the scheduler works as expected.
> >>
> >> I have two domains running, both pinned to the same processor:
> >>
> >> Domain-0:~ # xm vcpu-list
> >> Name                              ID  VCPU   CPU State   Time(s) CPU
> >> Affinity
> >> Domain-0                           0     0     1   ---      30.2 any cpu
> >> Domain-0                           0     1     1   r--      12.0 any cpu
> >> xendom1                            1     0     0   ---      98.6 0
> >> xendom2                            2     0     0   r--     110.3 0
> >>
> >> Both domains should consume max. 100 Percent of that processor, but
> >> xendom1 has a weight twice as big as xendom2:
> >>
> >> Domain-0:~ # xm sched-credit -d xendom1
> >> {'cap': 100, 'weight': 200}
> >> Domain-0:~ # xm sched-credit -d xendom2
> >> {'cap': 100, 'weight': 100}
> >>
> >> In my understanding of the credit scheduler, xendom1 should get twice as
> >> much CPU-time as xendom2. But:
> >>
> >> xentop - 11:05:46   Xen 3.0.4-0
> >> Mem: 2612284k total, 2603852k used, 8432k free    CPUs: 2 @ 2992MHz
> >>       NAME  STATE   CPU(sec) CPU(%)  [...]
> >>   Domain-0 -----r         42    0.6  [...]
> >>    xendom1 ------        122   49.8  [...]
> >>    xendom2 -----r        133   50.2  [...]
> >>
> >> As you can see, both domains got an equal amount of the CPU.
> >>
> >> Am i missing something or do i understand something wrong?
> >>
> >> Greetings,
> >> -timo
> >> -- 
> >> Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
> >> PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc
> >>
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> > 
> > 
> 
> 
> -- 
> Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
> PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc
> 

Re: Credit Scheduler not working correct (3.0.4-0)

[Emmanuel Ackaouy]

Ok so If I get this right, what you want to do is have 2 UP guests with
different weights. You also want to turn off HT and you do this by
pinning both guests on CPU 0.

Credit accounting and pinning VCPUs don't always interact well
and I suspect you're hitting such a problem here.

If you have access to the Xen console, can you send output of
the 'r' (runq) key while your domains are each getting 50% of
CPU? Just take a few snapshots a few seconds apart. That would
confirm the theory.

Also, setting cap=100 on a UP guest is pointless. Just leave it
as zero. A VCPU isn't going to get more than 100% of a physical
CPU anyway.


On Feb 1, 2007, at 10:47, Timo Benk wrote:
> Hi,
>
> on my system, it seems that the credit scheduler does not work as 
> expected. The
> problem seems to be related to HyperThreading, if i switch off HT in 
> the BIOS
> Setup, the scheduler works as expected.
>
> I have two domains running, both pinned to the same processor:
>
> Domain-0:~ # xm vcpu-list
> Name                              ID  VCPU   CPU State   Time(s) CPU
> Affinity
> Domain-0                           0     0     1   ---      30.2 any 
> cpu
> Domain-0                           0     1     1   r--      12.0 any 
> cpu
> xendom1                            1     0     0   ---      98.6 0
> xendom2                            2     0     0   r--     110.3 0
>
> Both domains should consume max. 100 Percent of that processor, but
> xendom1 has a weight twice as big as xendom2:
>
> Domain-0:~ # xm sched-credit -d xendom1
> {'cap': 100, 'weight': 200}
> Domain-0:~ # xm sched-credit -d xendom2
> {'cap': 100, 'weight': 100}
>
> In my understanding of the credit scheduler, xendom1 should get twice 
> as
> much CPU-time as xendom2. But:
>
> xentop - 11:05:46   Xen 3.0.4-0
> Mem: 2612284k total, 2603852k used, 8432k free    CPUs: 2 @ 2992MHz
>       NAME  STATE   CPU(sec) CPU(%)  [...]
>   Domain-0 -----r         42    0.6  [...]
>    xendom1 ------        122   49.8  [...]
>    xendom2 -----r        133   50.2  [...]
>
> As you can see, both domains got an equal amount of the CPU.
>
> Am i missing something or do i understand something wrong?
>
> Greetings,
> -timo
> -- 
> Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
> PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Credit Scheduler not working correct (3.0.4-0)

[Daniele Palumbo]

On Thursday 01 February 2007 11:30, Emmanuel Ackaouy wrote:
> Also, setting cap=100 on a UP guest is pointless. Just leave it
> as zero. A VCPU isn't going to get more than 100% of a physical
> CPU anyway.

is pointless if you have only 1 vcpu, right?

thanks
Daniele

Re: Credit Scheduler not working correct (3.0.4-0)

[Emmanuel Ackaouy]

On Feb 1, 2007, at 11:58, Daniele Palumbo wrote:
> On Thursday 01 February 2007 11:30, Emmanuel Ackaouy wrote:
> Also, setting cap=100 on a UP guest is pointless. Just leave it
>> as zero. A VCPU isn't going to get more than 100% of a physical
>> CPU anyway.
>
> is pointless if you have only 1 vcpu, right?

Yes, because 100 means 1 physical CPU.
100 * number of VCPUs is the upper bound anyway.

Re: Credit Scheduler not working correct (3.0.4-0)

[Timo Benk]

Emmanuel Ackaouy wrote:
> If you have access to the Xen console, can you send output of
> the 'r' (runq) key while your domains are each getting 50% of
> CPU? Just take a few snapshots a few seconds apart. That would
> confirm the theory.
Sorry, but i dunno how to do that? Can you give me a hint?

> Also, setting cap=100 on a UP guest is pointless. Just leave it
> as zero. A VCPU isn't going to get more than 100% of a physical
> CPU anyway.
Yepp, you are right. The reason i have done this was 'cause i want
to demonstrate the credit-scheduler.

If both domains have one vcpu, managed by Xen and i set the following:
Domain-0:~ # xm sched-credit -d xendom1
{'cap': 100, 'weight': 200}
Domain-0:~ # xm sched-credit -d xendom2
{'cap': 100, 'weight': 100}

Both domain are running at 100 percent cpu, but on different pcpus.
That is correct, but does not demonstrate how to use the credit scheduler.

Ok, the other advice from Atsushi was to give one domain two vcpus, the other one vcpu,
both with the same weight and all vcpus pinned to the same pcpu. That does not help
either. Both domains are still running at 50 percent.

Even if i set the weight of xendom1, resp. xendom2 to be twice as the value
of xendom2, resp. xendom1, i get the same result.

Here is what i have done:

Domain-0:~ # xm vcpu-list
Name                              ID  VCPU   CPU State   Time(s) CPU Affinity
Domain-0                           0     0     1   ---      46.1 any cpu
Domain-0                           0     1     1   r--      28.5 any cpu
xendom1                            1     0     0   r--     558.7 0
xendom2                            3     0     0   -b-      21.6 0
xendom2                            3     1     0   ---     256.0 0

Domain-0:~ # xm sched-credit -d xendom1
{'cap': 0, 'weight': 256}

Domain-0:~ # xm sched-credit -d xendom2
{'cap': 0, 'weight': 256}

3 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 2612284k total, 2603856k used, 8428k free    CPUs: 2 @ 2992MHz
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR SSID
  Domain-0 -----r         85    1.6    2048256   78.4   no limit       n/a     2    4      163      139    0        0        0        0    0
   xendom1 ------        566   49.0     262144   10.0     532480      20.4     1    1        5       29    2        0       10      539    0
   xendom2 -----r        285   51.0     262144   10.0     262144      10.0     2    1        6       17    2        0     4622     1772    0

Greetings,
-timo

> On Feb 1, 2007, at 10:47, Timo Benk wrote:
>> Hi,
>>
>> on my system, it seems that the credit scheduler does not work as
>> expected. The
>> problem seems to be related to HyperThreading, if i switch off HT in
>> the BIOS
>> Setup, the scheduler works as expected.
>>
>> I have two domains running, both pinned to the same processor:
>>
>> Domain-0:~ # xm vcpu-list
>> Name                              ID  VCPU   CPU State   Time(s) CPU
>> Affinity
>> Domain-0                           0     0     1   ---      30.2 any cpu
>> Domain-0                           0     1     1   r--      12.0 any cpu
>> xendom1                            1     0     0   ---      98.6 0
>> xendom2                            2     0     0   r--     110.3 0
>>
>> Both domains should consume max. 100 Percent of that processor, but
>> xendom1 has a weight twice as big as xendom2:
>>
>> Domain-0:~ # xm sched-credit -d xendom1
>> {'cap': 100, 'weight': 200}
>> Domain-0:~ # xm sched-credit -d xendom2
>> {'cap': 100, 'weight': 100}
>>
>> In my understanding of the credit scheduler, xendom1 should get twice as
>> much CPU-time as xendom2. But:
>>
>> xentop - 11:05:46   Xen 3.0.4-0
>> Mem: 2612284k total, 2603852k used, 8432k free    CPUs: 2 @ 2992MHz
>>       NAME  STATE   CPU(sec) CPU(%)  [...]
>>   Domain-0 -----r         42    0.6  [...]
>>    xendom1 ------        122   49.8  [...]
>>    xendom2 -----r        133   50.2  [...]
>>
>> As you can see, both domains got an equal amount of the CPU.
>>
>> Am i missing something or do i understand something wrong?
>>
>> Greetings,
>> -timo
>> -- 
>> Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
>> PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
> 


-- 
Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc

Re: Credit Scheduler not working correct (3.0.4-0)

[Atsushi SAKAI]

Hi, Emmanuel

I wrote down my guess based on previous study for Timo's cases.
(I ask you this behavior is specification or not previously.)
http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00365.html
This case is similar to case 2) 
(when pinned-vcpu credit sum is over 1pcpu capacity)


Let's calculate on this.
Dom1 for 100Weight
Dom2 for 200Weight 
But CPU resources is 200(2CPU x 100)
So these credits are Dom1:67 and Dom2:133
But These vcpus are pinned to pcpu0
pcpu0 has only 100. but credit total is 200.
This makes a problem.

These 2vcpu has same priority in credit scheduler,
 since it cannot not consume its credit.
So these vcpus are scheduled as round robin.
This makes equal consumption of pcpu0.

Is this wrong guess?

Thanks
Atsushi SAKAI


Emmanuel Ackaouy <ack@xensource.com> wrote:

> Ok so If I get this right, what you want to do is have 2 UP guests with
> different weights. You also want to turn off HT and you do this by
> pinning both guests on CPU 0.
> 
> Credit accounting and pinning VCPUs don't always interact well
> and I suspect you're hitting such a problem here.
> 
> If you have access to the Xen console, can you send output of
> the 'r' (runq) key while your domains are each getting 50% of
> CPU? Just take a few snapshots a few seconds apart. That would
> confirm the theory.
> 
> Also, setting cap=100 on a UP guest is pointless. Just leave it
> as zero. A VCPU isn't going to get more than 100% of a physical
> CPU anyway.
> 
> 
> On Feb 1, 2007, at 10:47, Timo Benk wrote:
> > Hi,
> >
> > on my system, it seems that the credit scheduler does not work as 
> > expected. The
> > problem seems to be related to HyperThreading, if i switch off HT in 
> > the BIOS
> > Setup, the scheduler works as expected.
> >
> > I have two domains running, both pinned to the same processor:
> >
> > Domain-0:~ # xm vcpu-list
> > Name                              ID  VCPU   CPU State   Time(s) CPU
> > Affinity
> > Domain-0                           0     0     1   ---      30.2 any 
> > cpu
> > Domain-0                           0     1     1   r--      12.0 any 
> > cpu
> > xendom1                            1     0     0   ---      98.6 0
> > xendom2                            2     0     0   r--     110.3 0
> >
> > Both domains should consume max. 100 Percent of that processor, but
> > xendom1 has a weight twice as big as xendom2:
> >
> > Domain-0:~ # xm sched-credit -d xendom1
> > {'cap': 100, 'weight': 200}
> > Domain-0:~ # xm sched-credit -d xendom2
> > {'cap': 100, 'weight': 100}
> >
> > In my understanding of the credit scheduler, xendom1 should get twice 
> > as
> > much CPU-time as xendom2. But:
> >
> > xentop - 11:05:46   Xen 3.0.4-0
> > Mem: 2612284k total, 2603852k used, 8432k free    CPUs: 2 @ 2992MHz
> >       NAME  STATE   CPU(sec) CPU(%)  [...]
> >   Domain-0 -----r         42    0.6  [...]
> >    xendom1 ------        122   49.8  [...]
> >    xendom2 -----r        133   50.2  [...]
> >
> > As you can see, both domains got an equal amount of the CPU.
> >
> > Am i missing something or do i understand something wrong?
> >
> > Greetings,
> > -timo
> > -- 
> > Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
> > PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Credit Scheduler not working correct (3.0.4-0)

[Emmanuel Ackaouy]

On Feb 1, 2007, at 12:32, Atsushi SAKAI wrote:
> Hi, Emmanuel
>
> I wrote down my guess based on previous study for Timo's cases.
> (I ask you this behavior is specification or not previously.)
> http://lists.xensource.com/archives/html/xen-devel/2006-10/ 
> msg00365.html
> This case is similar to case 2)
> (when pinned-vcpu credit sum is over 1pcpu capacity)
>
>
> Let's calculate on this.
> Dom1 for 100Weight
> Dom2 for 200Weight
> But CPU resources is 200(2CPU x 100)
> So these credits are Dom1:67 and Dom2:133
> But These vcpus are pinned to pcpu0
> pcpu0 has only 100. but credit total is 200.
> This makes a problem.
>
> These 2vcpu has same priority in credit scheduler,
>  since it cannot not consume its credit.
> So these vcpus are scheduled as round robin.
> This makes equal consumption of pcpu0.
>
> Is this wrong guess?

You've described the problem.

Arbitrary pinning makes system wide weighted fair share complex.
Within that problem set there are simple cases where we could easily
do better than now though.

Using cpumasks to take one physical CPU out of the system is one
such case. Arguably, we should have a notion of system partitions
to do this though but in the mean time, we can throw some code
together to not totally bail on weighted fair share when some simpler
uses of pinning are used.

I'll probably have time to look at this next week.