Re: Odd sched behaviour; It takes 5 threads or more to load 2 CPU cores during kernel build

[Peter Williams <pwil3058@bigpond.net.au>]

Jesper Juhl wrote:
> Hi everyone,
> 
> This is a continuation of the issue I initially reported in the "make
> -j with j <= 4 seems to only load a single CPU core" thread
> (http://lkml.org/lkml/2006/2/21/231).
> 
> I've now got some more data, so hopefully someone can help me get a
> handle on this thing.
> 
> In a nutshell the problem is that I need to run "make -j N" where N is
> 
>>= 5 in order to put maximum load on both cores of my Athlon 64 X2
> 
> 4400+ when building kernels and with N < 5 the build apears to be
> pretty much done serially and not at all parallel.  This baffles me to
> be honest.
> The expected behaviour is that running with "make -j 2" on an
> otherwise idle machine would schedule a CC to run on each core most of
> the time, and with "make -j 3" & "make -j 4" there should definately
> be something executing on both cores all the time.
> What I see is that unless I bump it up to -j 5 or greater only one
> core seems to be put to work and the other one mostly just sits around
> spinning its wheels doing nothing.

Out of concern that this problem may be caused by the smpnice patches 
(which modify load balancing) I've done some testing on my HT 
workstation (with 2.6.16-rc4 installed).  The short summary of the 
results of my tests is that the smpnice patches do not appear to be at 
fault and that the problem lies in the kernel build mechanism for recent 
kernels.

More detail:

Test 1: build linux-2.6.14 with "make -j 2" and observe the results 
using top with the "latest CPU ran on" column enabled.  Result: both 
channels are being used and the build appears to be running in parallel.

Test 2: do the same test with a 2.6.16-rc5 kernel source.  Result: both 
channels are not being used and the build appears to be running serially.

Hence my conclusion that the problem is caused by recent (since 2.6.14) 
changes to the build mechanism and that it is not a scheduler or load 
balancing problem.

The good news from this is that a binary search trying to find when the 
problem doesn't need to reboot with the kernels each time but just 
evaluate the build process.

Peter
-- 
Peter Williams                                   pwil3058@bigpond.net.au

"Learning, n. The kind of ignorance distinguishing the studious."
  -- Ambrose Bierce