CPU Load Balancer / Scheduler / CFS

CPU Load Balancer / Scheduler / CFS

[Robert W]

I'm seeing some bizarre differences between two servers (one running
2.6.32 and the other 2.6.18) with how processes are spread across
CPUs.

I created a simple test client: https://gist.github.com/2306761
which forks 4 times to create processes which just consume cpu.

On 2.6.18, the processes are distributed across CPUs as expected (all
4 processes are consuming 100%).
But on 2.6.32, the distribution varies. Sometimes it's (100%, 33%,
33%, 33%), other times it's (100%, 100%, 50%, 50%), etc.. while all
the other CPUs sit completely idle..

Both servers have many cores (more than 4). Both servers each have 2
physical processors, and are hyper-threaded.

I've done many other similar tests (tried running the "stress" program
with different parameters and I also got bizarre results. It basically
looks like the CPU Load Balancer never moves a process to a different
CPU under any circumstances). I haven't set anything regarding cpusets
/ etc. so it should all be default behavior. Also, if I manually move
the processes to other cpus (using taskset) then I can get them to all
run at 100%.

What could be going on here?

CPU Load Balancer / Scheduler / CFS

[Mulyadi Santosa]

Hi Robert...

On Fri, Apr 6, 2012 at 00:24, Robert W <robertwalters83@gmail.com> wrote:
> But on 2.6.32, the distribution varies. Sometimes it's (100%, 33%,
> 33%, 33%), other times it's (100%, 100%, 50%, 50%), etc.. while all
> the other CPUs sit completely idle..
>
> Both servers have many cores (more than 4). Both servers each have 2
> physical processors, and are hyper-threaded.

Ahem, that HT might be the cause.

To elaborate more. Linux process scheduler tend to avoid the HT
sibling, because they are not "real" processor. Or if not possible to
avoid at all, then put at least load as possible.

Also, Linux kernel tries to "stick" a process into certain core within
certain time interval. This is done to make sure the L1/L2 cache is
utilized as long as possible. Processor relocation will trigger cache
invalidation and that costs a lot.

CMIIW....

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com