Re: Scheduler(?) regression from 2.6.22 to 2.6.24 for short-lived threads

[Willy Tarreau <w@1wt.eu>]

Hi Mike,

On Sat, Feb 09, 2008 at 08:58:39AM +0100, Mike Galbraith wrote:
> 
> On Fri, 2008-02-08 at 18:04 -0600, Olof Johansson wrote:
> > Hi,
> > 
> > I ended up with a customer benchmark in my lap this week that doesn't
> > do well on recent kernels. :(
> > 
> > After cutting it down to a simple testcase/microbenchmark, it seems like
> > recent kernels don't do as well with short-lived threads competing
> > with the thread it's cloned off of. The CFS scheduler changes come to
> > mind, but I suppose it could be caused by something else as well.
> > 
> > The pared-down testcase is included below. Reported runtime for the
> > testcase has increased almost 3x between 2.6.22 and 2.6.24:
> > 
> > 2.6.22: 3332 ms
> > 2.6.23: 4397 ms
> > 2.6.24: 8953 ms
> > 2.6.24-git19: 8986 ms
> 
> My 3GHz P4 shows disjointed results.
> 
> 2.6.22.17-smp
> time 798 ms
> time 780 ms
> time 702 ms
> 
> 2.6.22.17-cfs-v24.1-smp
> time 562 ms
> time 551 ms
> time 551 ms
> 
> 2.6.23.15-smp
> time 254 ms
> time 254 ms
> time 293 ms
> 
> 2.6.23.15-cfs-v24-smp
> time 764 ms
> time 791 ms
> time 780 ms
> 
> 2.6.24.1-smp
> time 815 ms
> time 820 ms
> time 771 ms

How many CPUs do you have ? It's impressive to see such important variations.
I would guess from the numbers that you sometimes have 1 or 2 CPUs doing
nothing. I've already observed strange CPU usage patterns while building
kernels (bound to 50% usage on a dual-athlon), but I can't say for sure
that it was only on 2.6, so it may be related to make dependencies instead
(at least it's what I've been suspecting till now).

> 2.6.25-smp (git today)
> time 29 ms
> time 61 ms
> time 72 ms

These ones look rather strange. What type of workload is it ? Can you
publish the program for others to test it ?

Regards,
Willy