BUG in ht-aware scheduler/nice in 2.6.7-rc2 on dual xeon

BUG in ht-aware scheduler/nice in 2.6.7-rc2 on dual xeon

[bert hubert]

Con, Ingo, List,

I'm overjoyed with decent ht-aware scheduling in 2.6.7-rc2 and it does
mostly the right thing. However, the 'nice' work by Con shows some slight
problems.

Please find attached program 'eat-time.cc'. Make sure not to compile it with
-O which might confuse things as this program basically does nothing.

Run it without arguments to determine the speed of 1 cpu, it outputs a
number (megaloops/second). Then start it with that number as a parameter:

Sample:

$ ./eat-time
592
$ ./eat-time 592
99%
99%
100%
etc

Now starting four of these at the same time gives the desired result:

$ ./eat-time 592 & ./eat-time 592 & ./eat-time 592 & ./eat-time 592
50%
50%
50%
50% 
etc

This however:

$ ./eat-time 592 & ./eat-time 592 & 
100%
99%
In another xterm:
$ nice -n +19 ./eat-time 592 & nice -n +19 ./eat-time 592
5%
5%
5%

Fails sometimes, with all processes getting 50%. The above 'screenshot' is
from the working and expected situation, which happens most of the time.

When it goes wrong, top shows me that Cpu0 and Cpu1 are 100% user, while
Cpu2 and Cpu3 are both 100% nice.  The niced processes show up in top as
PRiority 39, the unniced ones (NI = 0) as PR 25.

I've also seen it that Cpu2 and Cpu3 are 100% busy, and 0 and 1 are 100%
nice.

I'd say this situation happens once every 5 or 8 invocations, and perhaps
somewhat more when first starting the niced processes.

Perhaps related, when running the above 'nice -n +19' line on its own, I see
all CPUs getting load over time, the two processes are wandering. After a
while they settle down, only to go on wandering again some time later, also
touching configurations where a physical cpu suddenly hosts two processes.

Without nice, two processes get firmly pegged to different physical CPUs.

Anything I can do to help resolve this, just yell.

Thanks!

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: BUG in ht-aware scheduler/nice in 2.6.7-rc2 on dual xeon

[Con Kolivas]

On Mon, 7 Jun 2004 18:56, bert hubert wrote:
> Con, Ingo, List,
>
> I'm overjoyed with decent ht-aware scheduling in 2.6.7-rc2 and it does
> mostly the right thing. However, the 'nice' work by Con shows some slight
> problems.
>
> Please find attached program 'eat-time.cc'. Make sure not to compile it
> with -O which might confuse things as this program basically does nothing.
>
> Run it without arguments to determine the speed of 1 cpu, it outputs a
> number (megaloops/second). Then start it with that number as a parameter:
>
> Sample:
>
> $ ./eat-time
> 592
> $ ./eat-time 592
> 99%
> 99%
> 100%
> etc
>
> Now starting four of these at the same time gives the desired result:
>
> $ ./eat-time 592 & ./eat-time 592 & ./eat-time 592 & ./eat-time 592
> 50%
> 50%
> 50%
> 50%
> etc
>
> This however:
>
> $ ./eat-time 592 & ./eat-time 592 &
> 100%
> 99%
> In another xterm:
> $ nice -n +19 ./eat-time 592 & nice -n +19 ./eat-time 592
> 5%
> 5%
> 5%
>
> Fails sometimes, with all processes getting 50%. The above 'screenshot' is
> from the working and expected situation, which happens most of the time.
>
> When it goes wrong, top shows me that Cpu0 and Cpu1 are 100% user, while
> Cpu2 and Cpu3 are both 100% nice.  The niced processes show up in top as
> PRiority 39, the unniced ones (NI = 0) as PR 25.

This is just because the scheduler balancing is not aware of nice and when two 
same niceness tasks are on the same physical core they get equal shares. The 
ht-aware nice only works at keeping different nice values on the same 
physical core fair. There is no more that can be done using the current ht 
aware mechanism; a far more complicated balancing algorithm that takes nice 
into account would be required.

Con

Re: BUG in ht-aware scheduler/nice in 2.6.7-rc2 on dual xeon

[Nick Piggin]

bert hubert wrote:

> Fails sometimes, with all processes getting 50%. The above 'screenshot' is
> from the working and expected situation, which happens most of the time.
> 
> When it goes wrong, top shows me that Cpu0 and Cpu1 are 100% user, while
> Cpu2 and Cpu3 are both 100% nice.  The niced processes show up in top as
> PRiority 39, the unniced ones (NI = 0) as PR 25.
> 
> I've also seen it that Cpu2 and Cpu3 are 100% busy, and 0 and 1 are 100%
> nice.
> 

Ahh, that is because we don't do nice-aware SMP balancing. It
is orthogonal to the HT-nice work.

Fixing it properly requires someone to sit down and do some careful
design of nice-aware balancing that should be general enough to handle
the HT problems as a matter of course.