[U-Boot] [PATCH] Makefile: fix parallel build

[U-Boot] [PATCH] Makefile: fix parallel build

[Daniel Hobi]

During parallel build, the top Makefile spawns multiple sub-makes
for targets in cpu/$(CPU). If cpu/$(CPU)/.depend is not present, the
sub-makes may end up generating this file simultaneously which leads
to corrupted content.

A typical error message is:

.depend:39: *** multiple target patterns.  Stop.

This patch serializes the creation of cpu/$(CPU)/.depend by adding
cpu/$(CPU) to the depend target in the top Makefile.

Signed-off-by: Daniel Hobi <daniel.hobi@schmid-telecom.ch>
---
 Makefile |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/Makefile b/Makefile
index 19b5ac0..2fd22c7 100644
--- a/Makefile
+++ b/Makefile
@@ -400,7 +400,7 @@ env:
 		$(MAKE) -C tools/env all MTD_VERSION=${MTD_VERSION} || exit 1
 
 depend dep:	$(TIMESTAMP_FILE) $(VERSION_FILE) $(obj)include/autoconf.mk
-		for dir in $(SUBDIRS) ; do $(MAKE) -C $$dir _depend ; done
+		for dir in $(SUBDIRS) cpu/$(CPU); do $(MAKE) -C $$dir _depend ; done
 
 TAG_SUBDIRS += include
 TAG_SUBDIRS += lib_generic board/$(BOARDDIR)
-- 
1.6.5.4

[U-Boot] [PATCH] Makefile: fix parallel build

[Mike Frysinger]

On Thursday 10 December 2009 08:41:07 Daniel Hobi wrote:
> During parallel build, the top Makefile spawns multiple sub-makes
> for targets in cpu/$(CPU). If cpu/$(CPU)/.depend is not present, the
> sub-makes may end up generating this file simultaneously which leads
> to corrupted content.
> 
> A typical error message is:
> 
> .depend:39: *** multiple target patterns.  Stop.
> 
> This patch serializes the creation of cpu/$(CPU)/.depend by adding
> cpu/$(CPU) to the depend target in the top Makefile.

seems to happen in lib_$(ARCH)/ too, but in reviewing my logs from the last 
few months, most parallel .depend failures have indeed been in cpu/$(CPU)/.  
maybe this is just coincidence though ... perhaps the depend target should 
walk all subdirs instead of a just random few ones (use LIBS).
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20091211/38ce3d46/attachment.pgp 

[U-Boot] [PATCH] Makefile: fix parallel build

[Daniel Hobi]

On 11.12.2009 20:25, Mike Frysinger wrote:
> On Thursday 10 December 2009 08:41:07 Daniel Hobi wrote:
>> During parallel build, the top Makefile spawns multiple sub-makes
>> for targets in cpu/$(CPU). If cpu/$(CPU)/.depend is not present, the
>> sub-makes may end up generating this file simultaneously which leads
>> to corrupted content.
>>
>> A typical error message is:
>>
>> .depend:39: *** multiple target patterns.  Stop.
>>
>> This patch serializes the creation of cpu/$(CPU)/.depend by adding
>> cpu/$(CPU) to the depend target in the top Makefile.
> 
> seems to happen in lib_$(ARCH)/ too, but in reviewing my logs from the last 
> few months, most parallel .depend failures have indeed been in cpu/$(CPU)/.  
> maybe this is just coincidence though ... perhaps the depend target should 
> walk all subdirs instead of a just random few ones (use LIBS).

The problem with cpu/$(CPU)/ is that the top Makefile builds multiple
targets within this directory in parallel - at least start.o and
lib$(CPU).a. For all other directories, parallel build should work,
since there is only one target per directory.

Can you provide any commit ID where building lib_$(ARCH)/ failed?

Thus, adding all LIBS to the depend target should not make any
difference. In order to properly track such dependencies we should
switch to non-recursive Makefiles (as seen in Linux) since Recursive
Make [is] Considered Harmful[1].

[1] http://miller.emu.id.au/pmiller/books/rmch/

Best regards,
Daniel

[U-Boot] [PATCH] Makefile: fix parallel build

[Mike Frysinger]

On Tuesday 15 December 2009 04:21:02 Daniel Hobi wrote:
> On 11.12.2009 20:25, Mike Frysinger wrote:
> > On Thursday 10 December 2009 08:41:07 Daniel Hobi wrote:
> >> During parallel build, the top Makefile spawns multiple sub-makes
> >> for targets in cpu/$(CPU). If cpu/$(CPU)/.depend is not present, the
> >> sub-makes may end up generating this file simultaneously which leads
> >> to corrupted content.
> >>
> >> A typical error message is:
> >>
> >> .depend:39: *** multiple target patterns.  Stop.
> >>
> >> This patch serializes the creation of cpu/$(CPU)/.depend by adding
> >> cpu/$(CPU) to the depend target in the top Makefile.
> >
> > seems to happen in lib_$(ARCH)/ too, but in reviewing my logs from the
> > last few months, most parallel .depend failures have indeed been in
> > cpu/$(CPU)/. maybe this is just coincidence though ... perhaps the depend
> > target should walk all subdirs instead of a just random few ones (use
> > LIBS).
> 
> The problem with cpu/$(CPU)/ is that the top Makefile builds multiple
> targets within this directory in parallel - at least start.o and
> lib$(CPU).a. For all other directories, parallel build should work,
> since there is only one target per directory.

OK, that answers the next part

> Can you provide any commit ID where building lib_$(ARCH)/ failed?

it isnt specific to lib_$(ARCH).  wherever the LDSCRiPT is located will cause 
a problem.  in the Blackfin case, it's in lib_blackfin/.

so maybe to extend this workaround, add like $(dir $(LDSCRIPT)) to the list.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20091215/de33dcbb/attachment.pgp 

[U-Boot] [PATCH v2] Makefile: fix parallel build

[Daniel Hobi]

During parallel build, the top Makefile spawns multiple sub-makes for
targets in cpu/$(CPU) and $(dir $(LDSCRIPT)). If the .depend files are
not present in these directories, the sub-makes may end up generating
these files simultaneously which leads to corrupted content.

A typical error message is:

.depend:39: *** multiple target patterns.  Stop.

This patch serializes the creation of .depend in cpu/$(CPU) and
$(dir $(LDSCRIPT)) by adding these directories to the depend target
in the top Makefile.

Other directories in $(LIBS) are not affected since they contain only
one Make target and thus only one sub-make is spawned per directory.

Signed-off-by: Daniel Hobi <daniel.hobi@schmid-telecom.ch>
---
v2:
 - Break overlong line
 - Add comment about other directories to commit message
 - Also build target _depend in $(dir $(LDSCRIPT)) (as suggested by Mike)
   This requires the Makefile in $(dir $(LDSCRIPT)) to provide a
   _depend target, which is the case for all LDSCRIPT values in the
   current tree

diff --git a/Makefile b/Makefile
index 19b5ac0..e9be7a5 100644
--- a/Makefile
+++ b/Makefile
@@ -400,7 +400,8 @@ env:
 		$(MAKE) -C tools/env all MTD_VERSION=${MTD_VERSION} || exit 1
 
 depend dep:	$(TIMESTAMP_FILE) $(VERSION_FILE) $(obj)include/autoconf.mk
-		for dir in $(SUBDIRS) ; do $(MAKE) -C $$dir _depend ; done
+		for dir in $(SUBDIRS) cpu/$(CPU) $(dir $(LDSCRIPT)) ; do \
+			$(MAKE) -C $$dir _depend ; done
 
 TAG_SUBDIRS += include
 TAG_SUBDIRS += lib_generic board/$(BOARDDIR)
-- 
1.6.5.4

[U-Boot] [PATCH v2] Makefile: fix parallel build

[Mike Frysinger]

On Tuesday 15 December 2009 08:51:46 Daniel Hobi wrote:
>  depend dep:	$(TIMESTAMP_FILE) $(VERSION_FILE) $(obj)include/autoconf.mk
> -		for dir in $(SUBDIRS) ; do $(MAKE) -C $$dir _depend ; done
> +		for dir in $(SUBDIRS) cpu/$(CPU) $(dir $(LDSCRIPT)) ; do \
> +			$(MAKE) -C $$dir _depend ; done

i'd stick a short comment here explaining the reason for the specific subdirs.  
otherwise, the new additions all look good to me.  this is something that has 
annoyed me for quite a long time, but has been hard to reproduce to figure out 
what exactly was failing.  thanks !
Signed-off-by: Mike Frysinger <vapier@gentoo.org>
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20091215/14c3d5b2/attachment.pgp 

[U-Boot] [PATCH v2] Makefile: fix parallel build

[Wolfgang Denk]

Dear Daniel Hobi,

In message <1260885106-3494-1-git-send-email-daniel.hobi@schmid-telecom.ch> you wrote:
> During parallel build, the top Makefile spawns multiple sub-makes for
> targets in cpu/$(CPU) and $(dir $(LDSCRIPT)). If the .depend files are
> not present in these directories, the sub-makes may end up generating
> these files simultaneously which leads to corrupted content.
...
> diff --git a/Makefile b/Makefile
> index 19b5ac0..e9be7a5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -400,7 +400,8 @@ env:
>  		$(MAKE) -C tools/env all MTD_VERSION=${MTD_VERSION} || exit 1
>  
>  depend dep:	$(TIMESTAMP_FILE) $(VERSION_FILE) $(obj)include/autoconf.mk
> -		for dir in $(SUBDIRS) ; do $(MAKE) -C $$dir _depend ; done
> +		for dir in $(SUBDIRS) cpu/$(CPU) $(dir $(LDSCRIPT)) ; do \
> +			$(MAKE) -C $$dir _depend ; done

I ageree with Mike: please add a comment here.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
The important thing about being a leader is not being right or wrong,
but being *certain*.                    - Terry Pratchett, _Truckers_

[U-Boot] [PATCH v3] Makefile: fix parallel build

[Daniel Hobi]

During parallel build, the top Makefile spawns multiple sub-makes for
targets in cpu/$(CPU) and $(dir $(LDSCRIPT)). If the .depend files are
not present in these directories, the sub-makes may end up generating
these files simultaneously which leads to corrupted content.

A typical error message is:

.depend:39: *** multiple target patterns.  Stop.

This patch serializes the creation of .depend in cpu/$(CPU) and
$(dir $(LDSCRIPT)) by adding these directories to the depend target
in the top Makefile.

Other directories in $(LIBS) are not affected since they contain only
one Make target and thus only one sub-make is spawned per directory.

Signed-off-by: Daniel Hobi <daniel.hobi@schmid-telecom.ch>
---
v3:
 - Add a comment why make _depend is required for these two subdirs
v2:
 - Also build target depend in $(dir $(LDSCRIPT)) (suggested by Mike)
 - Break overlong line

 Makefile |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/Makefile b/Makefile
index 793fcec..1ab197b 100644
--- a/Makefile
+++ b/Makefile
@@ -398,8 +398,11 @@ updater:
 env:
 		$(MAKE) -C tools/env all MTD_VERSION=${MTD_VERSION} || exit 1
 
+# Explicitly make _depend in subdirs containing multiple targets to prevent
+# parallel sub-makes creating .depend files simultaneously.
 depend dep:	$(TIMESTAMP_FILE) $(VERSION_FILE) $(obj)include/autoconf.mk
-		for dir in $(SUBDIRS) ; do $(MAKE) -C $$dir _depend ; done
+		for dir in $(SUBDIRS) cpu/$(CPU) $(dir $(LDSCRIPT)) ; do \
+			$(MAKE) -C $$dir _depend ; done
 
 TAG_SUBDIRS = $(SUBDIRS)
 TAG_SUBDIRS += $(dir $(__LIBS))
-- 
1.6.6

[U-Boot] [PATCH v3] Makefile: fix parallel build

[Wolfgang Denk]

Dear Daniel Hobi,

In message <1263834819-15315-1-git-send-email-daniel.hobi@schmid-telecom.ch> you wrote:
> During parallel build, the top Makefile spawns multiple sub-makes for
> targets in cpu/$(CPU) and $(dir $(LDSCRIPT)). If the .depend files are
> not present in these directories, the sub-makes may end up generating
> these files simultaneously which leads to corrupted content.
> 
> A typical error message is:
> 
> .depend:39: *** multiple target patterns.  Stop.
> 
> This patch serializes the creation of .depend in cpu/$(CPU) and
> $(dir $(LDSCRIPT)) by adding these directories to the depend target
> in the top Makefile.
> 
> Other directories in $(LIBS) are not affected since they contain only
> one Make target and thus only one sub-make is spawned per directory.
> 
> Signed-off-by: Daniel Hobi <daniel.hobi@schmid-telecom.ch>
> ---
> v3:
>  - Add a comment why make _depend is required for these two subdirs
> v2:
>  - Also build target depend in $(dir $(LDSCRIPT)) (suggested by Mike)
>  - Break overlong line
> 
>  Makefile |    5 ++++-
>  1 files changed, 4 insertions(+), 1 deletions(-)

Applied, thanks.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"To take a significant step forward, you must make a series of finite
improvements." - Donald J. Atwood, General Motors