Problems with parallel building

Problems with parallel building

[Uwe Kleine-König]

Hello,

using v2.6.24-7284-g9ef9dc6 (plus some patches that shouldn't affect
kbuild) I have problems with parallel building (make -j 3).  Sometimes
the build fails, the error isn't always the same.

The last build resulted in:

	...
	  SYSMAP  .tmp_System.map
	  OBJCOPY arch/arm/boot/Image
	  OBJCOPY arch/arm/boot/Image
	  OBJCOPY arch/arm/boot/Image
	  Kernel: arch/arm/boot/Image is ready
	  GZIP    arch/arm/boot/Image.gz
	  Kernel: arch/arm/boot/Image is ready
	  Kernel: arch/arm/boot/Image is ready
	  AS      arch/arm/boot/compressed/head.o
	  GZIP    arch/arm/boot/compressed/piggy.gz
	  AS      arch/arm/boot/compressed/head.o
	  GZIP    arch/arm/boot/compressed/piggy.gz
	  UIMAGE  arch/arm/boot/uzImage
	  CC      arch/arm/boot/compressed/misc.o
	Image Name:   Linux-2.6.24-07325-g0f3fee0-dirt
	Created:      Tue Feb  5 14:49:25 2008
	Image Type:   ARM Linux Kernel Image (gzip compressed)
	Data Size:    1240792 Bytes = 1211.71 kB = 1.18 MB
	Load Address: 0x00008000
	Entry Point:  0x00008000
	  Image arch/arm/boot/uzImage is ready
	  Kernel: arch/arm/boot/Image is ready
	  CC      arch/arm/boot/compressed/misc.o
	  AS      arch/arm/boot/compressed/piggy.o
	  LD      arch/arm/boot/compressed/vmlinux
	arch/arm/boot/compressed/head.o: In function `not_relocated':
	/home/ukleinek/gsrc/linux-2.6/arch/arm/boot/compressed/head.S:249: undefined reference to `decompress_kernel'
	arch/arm/boot/compressed/head.o: In function `wont_overwrite':
	/home/ukleinek/gsrc/linux-2.6/arch/arm/boot/compressed/head.S:286: undefined reference to `decompress_kernel'
	make[4]: *** [arch/arm/boot/compressed/vmlinux] Error 1
	make[3]: *** [arch/arm/boot/compressed/vmlinux] Error 2
	make[2]: *** [uImage] Error 2
	make[2]: *** Waiting for unfinished jobs....

The command to start the build was:

	make "ARCH=arm" "CROSS_COMPILE=arm-linux-" -j 3 uzImage uImage zImage Image

Running that command once more worked without errors.

I could reproduce that problem with current Linus' master
(v2.6.24-7284-g9ef9dc6).

I did:

	while delmake ns9xxx_defconfig && delmake -j 3 Image zImage uImage && delmake mrproper; do :; done

with delmake being a wrapper script that does

	make "ARCH=arm" "CROSS_COMPILE=arm-linux-" "$@"

after adding the toolchain to my PATH.

The result was:

	...
	  KSYM    .tmp_kallsyms2.S
	  AS      .tmp_kallsyms2.o
	  LD      vmlinux
	  SYSMAP  System.map
	  SYSMAP  .tmp_System.map
	  OBJCOPY arch/arm/boot/Image
	  Kernel: arch/arm/boot/Image is ready
	  Kernel: arch/arm/boot/Image is ready
	  AS      arch/arm/boot/compressed/head.o
	  GZIP    arch/arm/boot/compressed/piggy.gz
	  CC      arch/arm/boot/compressed/misc.o
	  AS      arch/arm/boot/compressed/piggy.o
	  Kernel: arch/arm/boot/Image is ready
	  AS      arch/arm/boot/compressed/piggy.o
	  CC      arch/arm/boot/compressed/misc.o
	fixdep: arch/arm/boot/compressed/.piggy.o.d: No such file or directory
	make[2]: *** [arch/arm/boot/compressed/piggy.o] Error 2
	make[2]: *** Waiting for unfinished jobs....
	make[1]: *** [arch/arm/boot/compressed/vmlinux] Error 2
	make: *** [zImage] Error 2
	make: *** Waiting for unfinished jobs....
	  LD      arch/arm/boot/compressed/vmlinux
	  OBJCOPY arch/arm/boot/zImage
	  Kernel: arch/arm/boot/zImage is ready
	  UIMAGE  arch/arm/boot/uImage
	Image Name:   Linux-2.6.24
	Created:      Tue Feb  5 15:05:22 2008
	Image Type:   ARM Linux Kernel Image (uncompressed)
	Data Size:    734016 Bytes = 716.81 kB = 0.70 MB
	Load Address: 0x00108000
	Entry Point:  0x00108000
	  Image arch/arm/boot/uImage is ready

Now the loop is running again with passing V=1 make.  So far without failing :-)
I will let this run over night and report tomorrow.

[...]
Oh, it failed just now, I think at the same stage:

	...
	  arm-linux-ld -EL  -p --no-undefined -X  -o .tmp_vmlinux2 -T arch/arm/kernel/vmlinux.lds arch/arm/kernel/head.o arch/arm/kernel/init_task.o  init/built-in.o --start-group  usr/built-in.o  arch/arm/kernel/built-in.o  arch/arm/mm/built-in.o  arch/arm/common/built-in.o  arch/arm/mach-ns9xxx/built-in.o  arch/arm/nwfpe/built-in.o  kernel/built-in.o  mm/built-in.o  fs/built-in.o  ipc/built-in.o  security/built-in.o  crypto/built-in.o  block/built-in.o  arch/arm/lib/lib.a  lib/lib.a  arch/arm/lib/built-in.o  lib/built-in.o  drivers/built-in.o  sound/built-in.o  net/built-in.o --end-group .tmp_kallsyms1.o
	  arm-linux-nm -n .tmp_vmlinux2 | scripts/kallsyms  > .tmp_kallsyms2.S
	  arm-linux-gcc -Wp,-MD,./..tmp_kallsyms2.o.d -D__ASSEMBLY__ -mabi=apcs-gnu -mno-thumb-interwork -D__LINUX_ARM_ARCH__=5 -march=armv5te -mtune=arm9tdmi -msoft-float -gdwarf-2   -nostdinc -isystem /home/ukleinek/csrc/LxNETES-4/bin/../lib/gcc/arm-linux-uclibc/4.0.3/include -D__KERNEL__ -Iinclude  -include include/linux/autoconf.h -mlittle-endian    -c -o .tmp_kallsyms2.o .tmp_kallsyms2.S
	  arm-linux-ld -EL  -p --no-undefined -X  -o vmlinux -T arch/arm/kernel/vmlinux.lds arch/arm/kernel/head.o arch/arm/kernel/init_task.o  init/built-in.o --start-group  usr/built-in.o  arch/arm/kernel/built-in.o  arch/arm/mm/built-in.o  arch/arm/common/built-in.o  arch/arm/mach-ns9xxx/built-in.o  arch/arm/nwfpe/built-in.o  kernel/built-in.o  mm/built-in.o  fs/built-in.o  ipc/built-in.o  security/built-in.o  crypto/built-in.o  block/built-in.o  arch/arm/lib/lib.a  lib/lib.a  arch/arm/lib/built-in.o  lib/built-in.o  drivers/built-in.o  sound/built-in.o  net/built-in.o --end-group .tmp_kallsyms2.o
	echo 'cmd_vmlinux := arm-linux-ld -EL  -p --no-undefined -X  -o vmlinux -T arch/arm/kernel/vmlinux.lds arch/arm/kernel/head.o arch/arm/kernel/init_task.o  init/built-in.o --start-group  usr/built-in.o  arch/arm/kernel/built-in.o  arch/arm/mm/built-in.o  arch/arm/common/built-in.o  arch/arm/mach-ns9xxx/built-in.o  arch/arm/nwfpe/built-in.o  kernel/built-in.o  mm/built-in.o  fs/built-in.o  ipc/built-in.o  security/built-in.o  crypto/built-in.o  block/built-in.o  arch/arm/lib/lib.a  lib/lib.a  arch/arm/lib/built-in.o  lib/built-in.o  drivers/built-in.o  sound/built-in.o  net/built-in.o --end-group .tmp_kallsyms2.o' > ./.vmlinux.cmd
	echo '  /bin/sh /home/ukleinek/gsrc/l/scripts/mksysmap  System.map' && /bin/sh /home/ukleinek/gsrc/l/scripts/mksysmap vmlinux System.map; if [ $? -ne 0 ]; then rm -f vmlinux; /bin/false; fi;
	  /bin/sh /home/ukleinek/gsrc/l/scripts/mksysmap  System.map
	echo '  /bin/sh /home/ukleinek/gsrc/l/scripts/mksysmap  .tmp_System.map' && /bin/sh /home/ukleinek/gsrc/l/scripts/mksysmap .tmp_vmlinux2 .tmp_System.map
	  /bin/sh /home/ukleinek/gsrc/l/scripts/mksysmap  .tmp_System.map
	cmp -s System.map .tmp_System.map || (echo Inconsistent kallsyms data; echo Try setting CONFIG_KALLSYMS_EXTRA_PASS; rm .tmp_kallsyms* ; /bin/false )
	rm -f .old_version
	make -f scripts/Makefile.build obj=arch/arm/boot MACHINE=arch/arm/mach-ns9xxx/ arch/arm/boot/Image
	make -f scripts/Makefile.build obj=arch/arm/boot MACHINE=arch/arm/mach-ns9xxx/ arch/arm/boot/zImage
	  arm-linux-objcopy -O binary -R .note -R .note.gnu.build-id -R .comment -S  vmlinux arch/arm/boot/Image
	  Kernel: arch/arm/boot/Image is ready
	make -f scripts/Makefile.build obj=arch/arm/boot MACHINE=arch/arm/mach-ns9xxx/ arch/arm/boot/uImage
	  Kernel: arch/arm/boot/Image is ready
	make -f scripts/Makefile.build obj=arch/arm/boot/compressed arch/arm/boot/compressed/vmlinux
	  arm-linux-gcc -Wp,-MD,arch/arm/boot/compressed/.head.o.d  -nostdinc -isystem /home/ukleinek/csrc/LxNETES-4/bin/../lib/gcc/arm-linux-uclibc/4.0.3/include -D__KERNEL__ -Iinclude  -include include/linux/autoconf.h -mlittle-endian -D__ASSEMBLY__ -mabi=apcs-gnu -mno-thumb-interwork -D__LINUX_ARM_ARCH__=5 -march=armv5te -mtune=arm9tdmi -msoft-float -gdwarf-2     -c -o arch/arm/boot/compressed/head.o arch/arm/boot/compressed/head.S
	  gzip -f -9 < arch/arm/boot/compressed/../Image > arch/arm/boot/compressed/piggy.gz
	  arm-linux-gcc -Wp,-MD,arch/arm/boot/compressed/.misc.o.d  -nostdinc -isystem /home/ukleinek/csrc/LxNETES-4/bin/../lib/gcc/arm-linux-uclibc/4.0.3/include -D__KERNEL__ -Iinclude  -include include/linux/autoconf.h -mlittle-endian -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -O2 -marm -fno-omit-frame-pointer -mapcs -mno-sched-prolog -mabi=apcs-gnu -mno-thumb-interwork -D__LINUX_ARM_ARCH__=5 -march=armv5te -mtune=arm9tdmi  -msoft-float -Uarm -fno-omit-frame-pointer -fno-optimize-sibling-calls -g  -Wdeclaration-after-statement -Wno-pointer-sign  -fpic -fno-builtin -Dstatic=  -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(misc)"  -D"KBUILD_MODNAME=KBUILD_STR(misc)" -c -o arch/arm/boot/compressed/.tmp_misc.o arch/arm/boot/compressed/misc.c
	  arm-linux-gcc -Wp,-MD,arch/arm/boot/compressed/.piggy.o.d  -nostdinc -isystem /home/ukleinek/csrc/LxNETES-4/bin/../lib/gcc/arm-linux-uclibc/4.0.3/include -D__KERNEL__ -Iinclude  -include include/linux/autoconf.h -mlittle-endian -D__ASSEMBLY__ -mabi=apcs-gnu -mno-thumb-interwork -D__LINUX_ARM_ARCH__=5 -march=armv5te -mtune=arm9tdmi -msoft-float -gdwarf-2     -c -o arch/arm/boot/compressed/piggy.o arch/arm/boot/compressed/piggy.S
	  Kernel: arch/arm/boot/Image is ready
	make -f scripts/Makefile.build obj=arch/arm/boot/compressed arch/arm/boot/compressed/vmlinux
	  arm-linux-gcc -Wp,-MD,arch/arm/boot/compressed/.piggy.o.d  -nostdinc -isystem /home/ukleinek/csrc/LxNETES-4/bin/../lib/gcc/arm-linux-uclibc/4.0.3/include -D__KERNEL__ -Iinclude  -include include/linux/autoconf.h -mlittle-endian -D__ASSEMBLY__ -mabi=apcs-gnu -mno-thumb-interwork -D__LINUX_ARM_ARCH__=5 -march=armv5te -mtune=arm9tdmi -msoft-float -gdwarf-2     -c -o arch/arm/boot/compressed/piggy.o arch/arm/boot/compressed/piggy.S
	  arm-linux-gcc -Wp,-MD,arch/arm/boot/compressed/.misc.o.d  -nostdinc -isystem /home/ukleinek/csrc/LxNETES-4/bin/../lib/gcc/arm-linux-uclibc/4.0.3/include -D__KERNEL__ -Iinclude  -include include/linux/autoconf.h -mlittle-endian -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -O2 -marm -fno-omit-frame-pointer -mapcs -mno-sched-prolog -mabi=apcs-gnu -mno-thumb-interwork -D__LINUX_ARM_ARCH__=5 -march=armv5te -mtune=arm9tdmi  -msoft-float -Uarm -fno-omit-frame-pointer -fno-optimize-sibling-calls -g  -Wdeclaration-after-statement -Wno-pointer-sign  -fpic -fno-builtin -Dstatic=  -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(misc)"  -D"KBUILD_MODNAME=KBUILD_STR(misc)" -c -o arch/arm/boot/compressed/.tmp_misc.o arch/arm/boot/compressed/misc.c
	fixdep: arch/arm/boot/compressed/.piggy.o.d: No such file or directory
	make[2]: *** [arch/arm/boot/compressed/piggy.o] Error 2
	make[1]: *** [arch/arm/boot/compressed/vmlinux] Error 2
	make: *** [zImage] Error 2
	make: *** Waiting for unfinished jobs....
	  arm-linux-ld -EL    --defsym zreladdr=0x108000 --defsym params_phys=0x100 -p --no-undefined -X /home/ukleinek/csrc/LxNETES-4/bin/../lib/gcc/arm-linux-uclibc/4.0.3/libgcc.a -T arch/arm/boot/compressed/vmlinux.lds arch/arm/boot/compressed/head.o arch/arm/boot/compressed/piggy.o arch/arm/boot/compressed/misc.o -o arch/arm/boot/compressed/vmlinux 
	  arm-linux-objcopy -O binary -R .note -R .note.gnu.build-id -R .comment -S  arch/arm/boot/compressed/vmlinux arch/arm/boot/zImage
	  Kernel: arch/arm/boot/zImage is ready
	  /bin/sh /home/ukleinek/gsrc/l/scripts/mkuboot.sh -A arm -O linux -T kernel -C none -a 0x108000 -e 0x108000 -n 'Linux-2.6.24' -d arch/arm/boot/zImage arch/arm/boot/uImage
	Image Name:   Linux-2.6.24
	Created:      Tue Feb  5 15:19:05 2008
	Image Type:   ARM Linux Kernel Image (uncompressed)
	Data Size:    733972 Bytes = 716.77 kB = 0.70 MB
	Load Address: 0x00108000
	Entry Point:  0x00108000
	  Image arch/arm/boot/uImage is ready

ukleinek@zentaur:~$ arm-linux-gcc --version
arm-linux-gcc (GCC) 4.0.3
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

ukleinek@zentaur:~$ arm-linux-as --version
GNU assembler 2.16.1
Copyright 2005 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License.  This program has absolutely no warranty.
This assembler was configured for a target of `arm-linux-uclibc'.

Best regards
Uwe

-- 
Uwe Kleine-König, Software Engineer
Digi International GmbH Branch Breisach, Küferstrasse 8, 79206 Breisach, Germany
Tax: 315/5781/0242 / VAT: DE153662976 / Reg. Amtsgericht Dortmund HRB 13962

Re: Problems with parallel building

[Sam Ravnborg]

On Tue, Feb 05, 2008 at 03:23:43PM +0100, Uwe Kleine-König wrote:
> Hello,
> 
> using v2.6.24-7284-g9ef9dc6 (plus some patches that shouldn't affect
> kbuild) I have problems with parallel building (make -j 3).  Sometimes
> the build fails, the error isn't always the same.

Hi Uwe.

Is this new behaviour?
If it is it would be nice if you could bisect when it happened.
One suspect could be:
18c32dac75b187d1a4e858f3cfdf03e844129f5e
or
0b35786d77ba4037f181982cc8ca20a7a3bf0fd2

but they should only be relevant when you use O=...

If this is as it has always been then I am not suprised.
It is non-trivial to take all the targets supplied on the commandline
and serialize them especially as we do not know them beforehand
and we do not know which one is specified.

	Sam

Re: Problems with parallel building

[Uwe Kleine-König]

Hi Sam,

Sam Ravnborg wrote:
> On Tue, Feb 05, 2008 at 03:23:43PM +0100, Uwe Kleine-König wrote:
> > Hello,
> > 
> > using v2.6.24-7284-g9ef9dc6 (plus some patches that shouldn't affect
> > kbuild) I have problems with parallel building (make -j 3).  Sometimes
> > the build fails, the error isn't always the same.
> 
> Hi Uwe.
> 
> Is this new behaviour?
I'm not sure, because I only recently created some scripts that use -j 3
and use them frequently.  I'd say it started around 2.6.23 ...

> If it is it would be nice if you could bisect when it happened.
> One suspect could be:
> 18c32dac75b187d1a4e858f3cfdf03e844129f5e
> or
> 0b35786d77ba4037f181982cc8ca20a7a3bf0fd2
... which could match the latter.

> but they should only be relevant when you use O=...
I do use O=...

I will try to bisect it and let you know.

Best regards
Uwe

-- 
Uwe Kleine-König, Software Engineer
Digi International GmbH Branch Breisach, Küferstrasse 8, 79206 Breisach, Germany
Tax: 315/5781/0242 / VAT: DE153662976 / Reg. Amtsgericht Dortmund HRB 13962

Re: Problems with parallel building

[Uwe Kleine-König]

Uwe Kleine-König wrote:
> Hi Sam,
> 
> Sam Ravnborg wrote:
> > On Tue, Feb 05, 2008 at 03:23:43PM +0100, Uwe Kleine-König wrote:
> > > Hello,
> > > 
> > > using v2.6.24-7284-g9ef9dc6 (plus some patches that shouldn't affect
> > > kbuild) I have problems with parallel building (make -j 3).  Sometimes
> > > the build fails, the error isn't always the same.
> > 
> > Hi Uwe.
> > 
> > Is this new behaviour?
> I'm not sure, because I only recently created some scripts that use -j 3
> and use them frequently.  I'd say it started around 2.6.23 ...
> 
> > If it is it would be nice if you could bisect when it happened.
> > One suspect could be:
> > 18c32dac75b187d1a4e858f3cfdf03e844129f5e
> > or
> > 0b35786d77ba4037f181982cc8ca20a7a3bf0fd2
> ... which could match the latter.
> 
> > but they should only be relevant when you use O=...
> I do use O=...
> 
> I will try to bisect it and let you know.
With e.g. v2.6.21 it's even worse.  I get for each build:

	...
	CRC32c (Castagnoli, et al) Cyclic Redundancy-Check (LIBCRC32C) [N/m/y/?] n
	#
	# configuration written to .config
	#
	  GEN     /home/ukleinek/tmp/buildtest/obj/Makefile
	  GEN     /home/ukleinek/tmp/buildtest/obj/Makefile
	  GEN     /home/ukleinek/tmp/buildtest/obj/Makefile
	scripts/kconfig/conf -s arch/arm/Kconfig
	scripts/kconfig/conf -s arch/arm/Kconfig
	scripts/kconfig/conf -s arch/arm/Kconfig
	  CHK     include/linux/version.h
	  UPD     include/linux/version.h
	  SYMLINK include/asm-arm/arch -> include/asm-arm/arch-ns9xxx
	  CHK     include/linux/version.h
	  CHK     include/linux/version.h
	  GEN     /home/ukleinek/tmp/buildtest/obj/Makefile
	  GEN     /home/ukleinek/tmp/buildtest/obj/Makefile
	  CHK     include/linux/utsrelease.h
	  UPD     include/linux/utsrelease.h
	make[2]: `/home/ukleinek/tmp/buildtest/linux-2.6/include/asm-arm/mach-types.h' is up to date.
	  Using /home/ukleinek/tmp/buildtest/linux-2.6 as source for kernel
	  /home/ukleinek/tmp/buildtest/linux-2.6 is not clean, please run 'make mrproper'
	  in the '/home/ukleinek/tmp/buildtest/linux-2.6' directory.
	make[1]: *** [prepare3] Error 1
	make: *** [uImage] Error 2
	make: *** Waiting for unfinished jobs....
	  CHK     include/linux/utsrelease.h
	make[2]: `/home/ukleinek/tmp/buildtest/linux-2.6/include/asm-arm/mach-types.h' is up to date.
	  Using /home/ukleinek/tmp/buildtest/linux-2.6 as source for kernel
	  /home/ukleinek/tmp/buildtest/linux-2.6 is not clean, please run 'make mrproper'
	  in the '/home/ukleinek/tmp/buildtest/linux-2.6' directory.
	make[1]: *** [prepare3] Error 1
	make: *** [zImage] Error 2
	  GEN     /home/ukleinek/tmp/buildtest/obj/Makefile
	  CHK     include/linux/utsrelease.h
	make[2]: `/home/ukleinek/tmp/buildtest/linux-2.6/include/asm-arm/mach-types.h' is up to date.
	  Using /home/ukleinek/tmp/buildtest/linux-2.6 as source for kernel
	  /home/ukleinek/tmp/buildtest/linux-2.6 is not clean, please run 'make mrproper'
	  in the '/home/ukleinek/tmp/buildtest/linux-2.6' directory.
	make[1]: *** [prepare3] Error 1
	make: *** [Image] Error 2
	make: INTERNAL: Exiting with 4 jobserver tokens available; should be 3!

The result is the same for v2.6.22 and v2.6.23.

Bisecting this yielded 0b35786d77ba4037f181982cc8ca20a7a3bf0fd2.

The script used for building is:

	#! /bin/bash -e

	seq -w 1 100 | while read i; do 
		echo "********************* run $i *************************";

		rm -rf ../obj;
		mkdir ../obj;

		delmake O=../obj ns9xxx_defconfig;

		if ! delmake O=../obj -j 3 uImage zImage Image; then
			break;
		fi;
	done;

Best regards
Uwe

-- 
Uwe Kleine-König, Software Engineer
Digi International GmbH Branch Breisach, Küferstrasse 8, 79206 Breisach, Germany
Tax: 315/5781/0242 / VAT: DE153662976 / Reg. Amtsgericht Dortmund HRB 13962