[Linux-ia64] Parallel make

[Linux-ia64] Parallel make

[Ian Wienand]

When I try a parallel make, make seems to get it's dependencies out of
order.  

For example, here's a make oldconfig ; make -j4 output on a fresh 2.5.69

ianw@tartufi:/usr/src/ianw/linux-2.5.69$ make -j4
make -f scripts/Makefile.build obj=scripts
  gcc -Wp,-MD,scripts/.empty.o.d -D__KERNEL__ -Iinclude -Wall -Wstrict-prototypes -Wno-trigraphs -g -O2 -fno-strict-aliasing -fno-common -pipe  -ffixed-r13 -mfixed-rangeñ0-f15,f32-f127 -falign-functions2 -fomit-frame-pointer -nostdinc -iwithprefix include -mconstant-gp   -DKBUILD_BASENAME=empty
-DKBUILD_MODNAME=empty -c -o scripts/empty.o scripts/empty.c
  scripts/mk_elfconfig ia64 < scripts/empty.o > scripts/elfconfig.h
  gcc -Wp,-MD,scripts/.file2alias.o.d -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer   -c -o scripts/file2alias.o scripts/file2alias.c
  gcc -Wp,-MD,scripts/.modpost.o.d -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer   -c -o scripts/modpost.o scripts/modpost.c
  gcc  -o scripts/modpost scripts/modpost.o scripts/file2alias.o
  SPLIT   include/linux/autoconf.h -> include/config/*
  gcc -E -Wp,-MD,arch/ia64/.vmlinux.lds.s.d -D__ASSEMBLY__ -D__KERNEL__ -Iinclude -mconstant-gp -nostdinc -iwithprefix include   -P -C -Uia64   -o arch/ia64/vmlinux.lds.s arch/ia64/vmlinux.lds.S
In file included from arch/ia64/vmlinux.lds.S:4:
include/asm/ptrace.h:56:25: asm/offsets.h: No such file or directory
make: *** [arch/ia64/vmlinux.lds.s] Error 1
make: *** Waiting for unfinished jobs....
make: *** Waiting for unfinished jobs....

At first I thought it might be because the all target depends on
vmlinux, but this isn't it.  Currently I'm stumped.  'Single' make works
fine, and if you run parallel make a few times eventually offsets.h
gets generated and the build continues on fine.

Is it just me?

-i
ianw@gelato.unsw.edu.au
http://www.gelato.unsw.edu.au

Re: [Linux-ia64] Parallel make

[Ian Wienand]

On Wed, May 14, 2003 at 01:38:23PM +1000, Ian Wienand wrote:
> At first I thought it might be because the all target depends on
> vmlinux, but this isn't it.  Currently I'm stumped.  'Single' make works
> fine, and if you run parallel make a few times eventually offsets.h
> gets generated and the build continues on fine.

sorry to reply to myself but I forgot to say the path of least
resistance might be to just remove ptrace.h from vmlinux.lds.S, as it
doesn't seem to be needed.

-i

--- linux-2.5.69/arch/ia64/vmlinux.lds.S.old    2003-05-14 14:25:17.000000000 +1000
+++ linux-2.5.69/arch/ia64/vmlinux.lds.S        2003-05-14 14:25:32.000000000 +1000
@@ -1,7 +1,6 @@
 #include <linux/config.h>
  
 #include <asm/cache.h>
-#include <asm/ptrace.h>
 #include <asm/system.h>
  
 #define LOAD_OFFSET PAGE_OFFSET

Re: [Linux-ia64] Parallel make

[David Mosberger]

>>>>> On Wed, 14 May 2003 13:38:23 +1000, Ian Wienand <ianw@gelato.unsw.edu.au> said:

  Ian> When I try a parallel make, make seems to get it's dependencies
  Ian> out of order.

I see the same thing, but it's only for asm-ia64/offsets.h.  After
that, parallel makes work just fine.  I tend to be able to live with
such "minor" build issues for a loooong time, giving others ample
chance to get impatient first... ;-)

	--david

Re: [Linux-ia64] Parallel make

[Eric Piel]

David Mosberger wrote:
> 
> >>>>> On Wed, 14 May 2003 13:38:23 +1000, Ian Wienand <ianw@gelato.unsw.edu.au> said:
> 
>   Ian> When I try a parallel make, make seems to get it's dependencies
>   Ian> out of order.
> 
> I see the same thing, but it's only for asm-ia64/offsets.h.  After
> that, parallel makes work just fine.  I tend to be able to live with
> such "minor" build issues for a loooong time, giving others ample
> chance to get impatient first... ;-)
> 
Don't wait, do it now:
make prepare
make -j4 compressed

Eric

Re: [Linux-ia64] Parallel make

[David Mosberger]

>>>>> On Wed, 14 May 2003 09:27:07 +0200, Eric Piel <Eric.Piel@Bull.Net> said:

  >> I see the same thing, but it's only for asm-ia64/offsets.h.  After
  >> that, parallel makes work just fine.  I tend to be able to live with
  >> such "minor" build issues for a loooong time, giving others ample
  >> chance to get impatient first... ;-)

  Eric> Don't wait, do it now:
  Eric> make prepare
  Eric> make -j4 compressed

Ah, yes, I think I've seen that once mentioned in a mail.

BTW: have you gotten any feedback regarding the timer bug you
reported?  I think the same (rounding) bug may have broken the
profil() call in libc.

	--david

Re: [Linux-ia64] Parallel make

[Eric Piel]

David Mosberger wrote:
> 
> BTW: have you gotten any feedback regarding the timer bug you
> reported?  I think the same (rounding) bug may have broken the
> profil() call in libc.
Yes, I'm working with George Anzinger on this issue. For now a
"workaround" is to revert the functions
timespec_to_jiffies()...jiffies_to_timeval() in include/linux/time.h and
tstojiffie() in posix-timers.c to the versions of 2.5.67 .

Eric

Re: [Linux-ia64] Parallel make

[Sam Ravnborg]

On Wed, May 14, 2003 at 01:38:23PM +1000, Ian Wienand wrote:
> When I try a parallel make, make seems to get it's dependencies out of
> order.  

Could you try the following patch.

It does:
1) Remove superflous all: target, covered by the main Makefile
2) .PHONY shall only list non-file targets
3) Use make $@, $< where appropriate
4) For the offset file do not duplicate dependencies from main Makefile

	Sam

=== arch/ia64/Makefile 1.42 vs edited ==--- 1.42/arch/ia64/Makefile	Wed Mar 26 17:30:43 2003
+++ edited/arch/ia64/Makefile	Wed May 14 19:46:03 2003
@@ -60,17 +60,15 @@
 boot := arch/ia64/boot
 tools := arch/ia64/tools
 
-.PHONY: boot compressed include/asm-ia64/offsets.h
-
-all: prepare vmlinux
+.PHONY: boot compressed check
 
 compressed: vmlinux.gz
 
 vmlinux.gz: vmlinux
-	$(Q)$(MAKE) $(build)=$(boot) vmlinux.gz
+	$(Q)$(MAKE) $(build)=$(boot) $@
 
 check: vmlinux
-	arch/ia64/scripts/unwcheck.sh vmlinux
+	arch/ia64/scripts/unwcheck.sh $<
 
 archclean:
 	$(Q)$(MAKE) $(clean)=$(boot)
@@ -83,7 +81,7 @@
 boot:	lib/lib.a vmlinux
 	$(Q)$(MAKE) $(build)=$(boot) $@
 
-include/asm-ia64/offsets.h: include/asm include/linux/version.h include/config/MARKER
+include/asm-ia64/offsets.h:
 	$(Q)$(MAKE) $(build)=$(tools) $@
 
 define archhelp

Re: [Linux-ia64] Parallel make

[David Mosberger]

>>>>> On Wed, 14 May 2003 19:48:46 +0200, Sam Ravnborg <sam@ravnborg.org> said:

  Sam> On Wed, May 14, 2003 at 01:38:23PM +1000, Ian Wienand wrote:
  >> When I try a parallel make, make seems to get it's dependencies
  >> out of order.

  Sam> Could you try the following patch.

  Sam> It does: 1) Remove superflous all: target, covered by the main
  Sam> Makefile 2) .PHONY shall only list non-file targets 3) Use make
  Sam> $@, $< where appropriate 4) For the offset file do not
  Sam> duplicate dependencies from main Makefile

Very nice.  Seems to do the trick for me.

	--david

Re: [Linux-ia64] Parallel make

[David Mosberger]

>>>>> On Wed, 14 May 2003 10:42:14 +0200, Eric Piel <Eric.Piel@Bull.Net> said:

  Eric> David Mosberger wrote:

  >> BTW: have you gotten any feedback regarding the timer bug you
  >> reported?  I think the same (rounding) bug may have broken the
  >> profil() call in libc.

  Eric> Yes, I'm working with George Anzinger on this issue. For now a
  Eric> "workaround" is to revert the functions
  Eric> timespec_to_jiffies()...jiffies_to_timeval() in
  Eric> include/linux/time.h and tstojiffie() in posix-timers.c to the
  Eric> versions of 2.5.67 .

That works for me.  Undoing the time.h changes does make profil() work
again.

Thanks,

	--david

Re: [Linux-ia64] Parallel make

[Martin Pool]

On 14 May 2003, David Mosberger <davidm@napali.hpl.hp.com> wrote:
> 
>   >> I see the same thing, but it's only for asm-ia64/offsets.h.  After
>   >> that, parallel makes work just fine.  I tend to be able to live with
>   >> such "minor" build issues for a loooong time, giving others ample
>   >> chance to get impatient first... ;-)
> 
>   Eric> Don't wait, do it now:
>   Eric> make prepare
>   Eric> make -j4 compressed

(Advertorial): Now you've got make -j4, you should try ccache and
distcc.  They make development on big trees enormously more
enjoyable.  Andrew Morton and other kernel developers love them.

  http://distcc.samba.org/
  http://ccache.samba.org/

-- 
Martin 

Re: [Linux-ia64] Parallel make

[Ian Wienand]

On Wed, May 14, 2003 at 07:48:46PM +0200, Sam Ravnborg wrote:
> Could you try the following patch.

thanks, it now works perfectly.

-i

Re: [Linux-ia64] Parallel make

[David Mosberger]

>>>>> On Thu, 15 May 2003 09:59:29 +1000, Martin Pool <mbp@samba.org> said:

  Martin> On 14 May 2003, David Mosberger <davidm@napali.hpl.hp.com> wrote:
  >> 
  >> >> I see the same thing, but it's only for asm-ia64/offsets.h.  After
  >> >> that, parallel makes work just fine.  I tend to be able to live with
  >> >> such "minor" build issues for a loooong time, giving others ample
  >> >> chance to get impatient first... ;-)
  >> 
  Eric> Don't wait, do it now:
  Eric> make prepare
  Eric> make -j4 compressed

  Martin> (Advertorial): Now you've got make -j4, you should try ccache and
  Martin> distcc.  They make development on big trees enormously more
  Martin> enjoyable.  Andrew Morton and other kernel developers love them.

  Martin> http://distcc.samba.org/
  Martin> http://ccache.samba.org/

Actually, Stephane setup distcc here.  Unfortunately it doesn't help
me all that much, because linking is what takes a large part of my
time (because I'm making incremental changes and because I insist on
compiling with -g).

I have to say though I was very impressed with how easy to use distcc
is.  Nice job!

(BTW: still waiting for someone to tell me I need a 64-way SuperDome
to get my kernel compiles done... ;-)

	--david

Re: [Linux-ia64] Parallel make

[Stephane Eranian]

Martin,

On Thu, May 15, 2003 at 09:59:29AM +1000, Martin Pool wrote:
> On 14 May 2003, David Mosberger <davidm@napali.hpl.hp.com> wrote:
> > 
> >   >> I see the same thing, but it's only for asm-ia64/offsets.h.  After
> >   >> that, parallel makes work just fine.  I tend to be able to live with
> >   >> such "minor" build issues for a loooong time, giving others ample
> >   >> chance to get impatient first... ;-)
> > 
> >   Eric> Don't wait, do it now:
> >   Eric> make prepare
> >   Eric> make -j4 compressed
> 
> (Advertorial): Now you've got make -j4, you should try ccache and
> distcc.  They make development on big trees enormously more
> enjoyable.  Andrew Morton and other kernel developers love them.
> 
>   http://distcc.samba.org/
>   http://ccache.samba.org/
> 
Yes, distcc is really nice. I have been using it for a month or so
for my kernel compiles using a cluster of Itanium 2 machines. What
I like is that it is really easy to setup. The danger with the
approach is that you can easily overload your local (master) host.
As for kernel compiles for 2.5, there are a lot of ld steps
which are still all done on the master node.  Besides that you
can really feel the speed difference. I need to take a look at ccache
now that you mention it.

Very cool. 

-- 
-Stephane

Re: [Linux-ia64] Parallel make

[Grant Grundler]

On Thu, May 15, 2003 at 09:59:29AM +1000, Martin Pool wrote:
> (Advertorial): Now you've got make -j4, you should try ccache and
> distcc. 

(Endorsement): Stephane and I played around with just distcc for
2.4 ia64 kernels on a smaller cluster of rx2600 (dual CPU).
The "master" runs out of CPU around -j16 (8 hosts).
But that resulted in 66 second kernel builds.
And distcc really is trivial to setup.

grant

Re: [Linux-ia64] Parallel make

[Martin Pool]

On 15 May 2003, Stephane Eranian <eranian@hpl.hp.com> wrote:

> Yes, distcc is really nice. I have been using it for a month or so
> for my kernel compiles using a cluster of Itanium 2 machines. What
> I like is that it is really easy to setup. The danger with the
> approach is that you can easily overload your local (master) host.

It can work well to use -l as well as -j, to put an upper limit on the
load on the master host.  If you have 2 local CPUs and 8 altogether
then perhaps.

  make -l4 -j10

That ought to make concurrency ramp down as it gets into the linking
phase. 

I ought to add that to the manual/faq.

> As for kernel compiles for 2.5, there are a lot of ld steps
> which are still all done on the master node.  Besides that you
> can really feel the speed difference. I need to take a look at ccache
> now that you mention it.
> 
> Very cool. 

Thanks, I'm glad you like it.

-- 
Martin