Building the kernel on an SMP box?

Building the kernel on an SMP box?

[Brian D. McGrew]

Good morning all!

Currently I'm building my kernels on a Dell PE 1800 3.0GHz.  My dilemma
is that I build and rebuild the kernel about twenty times a day and even
though it only takes about 20 minutes, that's rapidly becoming too slow!
Today it's the 2.6.17 kernel on FC5 that I'm building with.

I see all these blurbs out there about someone being able to build a
complete kernel in under a minute or running an SMP build across
multiple CPUS and/or multiple machines.

So, to ask the group that should know the best ... What would be a
reasonable configuration to get my builds down under five minutes or so?
And then to go to the extreme, what kind of horsepower should I be
looking for if I want get the build times down to say a minute or so???

Thanks!

:b!

Brian D. McGrew { brian@visionpro.com || brian@doubledimension.com }
--
> This is a test.  This is only a test!
  Had this been an actual emergency, you would have been
  told to cancel this test and seek professional assistance!

Re: Building the kernel on an SMP box?

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

> And then to go to the extreme, what kind of horsepower should I be
> looking for if I want get the build times down to say a minute or so???

The machine where I reached the one minute limit was a 16x 2,7Ghz Xeon HT with 
32GB of RAM. The good thing is I had to pay neither the machine nor the power 
bill for it :)

Eike

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Building the kernel on an SMP box?

[Bill Davidsen]

Brian D. McGrew wrote:
> Good morning all!
> 
> Currently I'm building my kernels on a Dell PE 1800 3.0GHz.  My dilemma
> is that I build and rebuild the kernel about twenty times a day and even
> though it only takes about 20 minutes, that's rapidly becoming too slow!
> Today it's the 2.6.17 kernel on FC5 that I'm building with.
> 
> I see all these blurbs out there about someone being able to build a
> complete kernel in under a minute or running an SMP build across
> multiple CPUS and/or multiple machines.
> 
> So, to ask the group that should know the best ... What would be a
> reasonable configuration to get my builds down under five minutes or so?
> And then to go to the extreme, what kind of horsepower should I be
> looking for if I want get the build times down to say a minute or so???

I can just about make it on 2xXeons at 3.0GHz, HT enabled, 4GB RAM. But 
the new "Core 2 Duo" stuff is tons faster, is dual core but no HT yet, 
has better cache, faster memory bus... costs more. Some big NUMA 
hardware after that,

I will guess the new cheap point is o/c Core 2 Duo 6700, and will finish 
in about 7 min. Patience is a virtue.

-- 
Bill Davidsen <davidsen@tmr.com>
   Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.

Re: Building the kernel on an SMP box?

[Andi Kleen]

"Brian D. McGrew" <brian@visionpro.com> writes:
> 
> So, to ask the group that should know the best ... What would be a
> reasonable configuration to get my builds down under five minutes or so?
> And then to go to the extreme, what kind of horsepower should I be
> looking for if I want get the build times down to say a minute or so???

Depending on your build pattern you can likely speed up rebuilds by
using ccache. 

If that doesn't help get one or two (or more as needed) cheap dual
core systems and use icecream (http://en.opensuse.org/Icecream) to do
a cluster build and build with -jN (N=2*available cores/threads or so)

To combine icecream and ccache in the same build you can use 
ftp://ftp.firstfloor.org/pub/ak/smallsrc/icecache.c

For parallel builds modular builds are faster than static builds
because otherwise the linker becomes a bottleneck.

-Andi

Re: Building the kernel on an SMP box?

[Bill Davidsen]

Andi Kleen wrote:
> "Brian D. McGrew" <brian@visionpro.com> writes:
>> So, to ask the group that should know the best ... What would be a
>> reasonable configuration to get my builds down under five minutes or so?
>> And then to go to the extreme, what kind of horsepower should I be
>> looking for if I want get the build times down to say a minute or so???
> 
> Depending on your build pattern you can likely speed up rebuilds by
> using ccache. 
> 
> If that doesn't help get one or two (or more as needed) cheap dual
> core systems and use icecream (http://en.opensuse.org/Icecream) to do
> a cluster build and build with -jN (N=2*available cores/threads or so)

That sounds really useful, although I bet it assumes that the build 
environment is the same on all machines. Or at least similar. I'll have 
to try that, I have two lightly used machines to add to the build. Thanks!

-- 
Bill Davidsen <davidsen@tmr.com>
   Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.

Re: Building the kernel on an SMP box?

[Andi Kleen]

> That sounds really useful, although I bet it assumes that the build 
> environment is the same on all machines. Or at least similar. 

No it doesn't.

-Andi

Re: Building the kernel on an SMP box?

[Bill Davidsen]

Andi Kleen wrote:

>>That sounds really useful, although I bet it assumes that the build 
>>environment is the same on all machines. Or at least similar. 
>>    
>>
>
>No it doesn't.
>
>  
>
I will definitely have to look at that, then, I would have been very 
cautious about having the same version of the compiler and loader 
everywhere, but if the tasks are exported using the tool chain on the 
initiating computer or something, it should be all right.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: Building the kernel on an SMP box?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 528 bytes --]

On Sat, 29 Jul 2006 21:29:17 +0200, Andi Kleen said:
> 
> > That sounds really useful, although I bet it assumes that the build 
> > environment is the same on all machines. Or at least similar. 
> 
> No it doesn't.

What happens with the 'gcc version magic' with modules if the main kernel
and modules are compiled on separate machines with different GCC releases
installed?   Certainly seems like a requirement for a "similar" environment
to me, if compiling with gcc 4.1.0 vs 4.1.1. results in a module that won't
insmod....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Building the kernel on an SMP box?

[Andi Kleen]

On Sunday 30 July 2006 18:18, Valdis.Kletnieks@vt.edu wrote:

> What happens with the 'gcc version magic' with modules if the main kernel
> and modules are compiled on separate machines with different GCC releases
> installed?   Certainly seems like a requirement for a "similar" environment
> to me, if compiling with gcc 4.1.0 vs 4.1.1. results in a module that won't
> insmod....

Please read the icecream documentation.

-Andi

Re: Building the kernel on an SMP box?

[Jesper Juhl]

[-- Attachment #1: Type: text/plain, Size: 2164 bytes --]

On 27/07/06, Brian D. McGrew <brian@visionpro.com> wrote:
> Good morning all!
>
> Currently I'm building my kernels on a Dell PE 1800 3.0GHz.  My dilemma
> is that I build and rebuild the kernel about twenty times a day and even
> though it only takes about 20 minutes, that's rapidly becoming too slow!
> Today it's the 2.6.17 kernel on FC5 that I'm building with.
>
> I see all these blurbs out there about someone being able to build a
> complete kernel in under a minute or running an SMP build across
> multiple CPUS and/or multiple machines.
>
> So, to ask the group that should know the best ... What would be a
> reasonable configuration to get my builds down under five minutes or so?

My box builds my ordinary .config (attached) in under 5min :

$ time make -j3
real    3m58.047s
user    4m54.574s
sys     1m24.202s

Here are some more numbers for you to compare to your box :

allnoconfig :

$ make clean
$ make allnoconfig
$ time make -j3
real    0m54.544s
user    1m2.113s
sys     0m20.781s


allmodconfig :

$ make clean
$ make allmodconfig
$ time make -j3
real    28m49.748s
user    35m3.212s
sys     10m43.633s


This box uses a Athlon64 X2 4400+ CPU, has 2GB RAM and a single
Ultra160 SCSI disk.
It's currently running an i386 kernel, not x86_64, so it's probably
not performing to it's full potential atm.


> And then to go to the extreme, what kind of horsepower should I be
> looking for if I want get the build times down to say a minute or so???
>
Make sure you have a fast disk and adequate amounts of RAM so you
don't end up swapping, then add more CPU's.
A dual-core box is not too expensive these days and an additional core
really speeds things up. If you have the money for it, then a box with
two dual-core CPU's is ofcourse even better.
If you have several machines, then you can use distcc
(http://distcc.samba.org/) to distribute the build across the boxes -
that's an easy way to speed things up if you have a bunch of older
computers lying around.


-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

[-- Attachment #2: .config.gz --]
[-- Type: application/x-gzip, Size: 7542 bytes --]

Re: Building the kernel on an SMP box?

[Handle X]

On 7/29/06, Jesper Juhl <jesper.juhl@gmail.com> wrote:
> On 27/07/06, Brian D. McGrew <brian@visionpro.com> wrote:
> > Good morning all!
> > reasonable configuration to get my builds down under five minutes or so?
I got a (moderately old) Sun V20z with 2 Opteron 2.2 GHz with 7G ram.
My numbers are comparable with yours. (I don't know if they are on the
lower side for such a h/w configuration)
>
> My box builds my ordinary .config (attached) in under 5min :
>
> $ time make -j3
> real    3m58.047s
> user    4m54.574s
> sys     1m24.202s
Mine is (with your configuration file, src updated with latest linus' tree,
[om@turyxsrv /home/src/linux-2.6]$  time make -j4
...snipped...
real    4m12.316s
user    5m12.600s
sys     0m37.742s

With stdout redirected, (It helps to save a minute or so )
[om@turyxsrv /home/src/linux-2.6]$ time make -j4  >>time.txt

real    3m23.504s
user    5m14.776s
sys     0m38.138s

>
> Here are some more numbers for you to compare to your box :
>
> allnoconfig :
>
> $ make clean
> $ make allnoconfig
> $ time make -j3
> real    0m54.544s
> user    1m2.113s
> sys     0m20.781s

[om@turyxsrv /home/src/linux-2.6]$ time make -j4 > allnoconfigtime.txt
real    0m43.399s
user    1m6.256s
sys     0m9.389s
>
>
> allmodconfig :
>
> $ make clean
> $ make allmodconfig
> $ time make -j3
> real    28m49.748s
> user    35m3.212s
> sys     10m43.633s

[om@turyxsrv /home/src/linux-2.6]$ time make -j4 > allmodconfigtime.txt
real    39m59.100s
user    34m16.857s
sys     4m36.697s
> This box uses a Athlon64 X2 4400+ CPU, has 2GB RAM and a single
> Ultra160 SCSI disk.

[root@turyxsrv ~]# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   2676 MB in  2.00 seconds = 1338.79 MB/sec
 Timing buffered disk reads:  184 MB in  3.02 seconds =  61.00 MB/sec