* [LTP] [PATCH] Support for parallel compilation
@ 2009-12-04 17:04 Jiri Palecek
0 siblings, 0 replies; 6+ messages in thread
From: Jiri Palecek @ 2009-12-04 17:04 UTC (permalink / raw)
Cc: ltp-list
Hello,
I tried to make ltp with the -j switch and found the following problems:
- the autotools target fail because of missing dependency on aclocal.m4 (almost all autotools targets depend on it)
- the shell loop in generic_trunk_target makes the "all" rules of the subdirectories run strictly sequentially
This patch fixes these issues by adding dependencies to the autotools targets (it also deletes some of the duplicated targets from the realtime Makefile), and by making one target for each subdirectory in generic_trunk_target.
Note that I really don't know if this works on an arbitrary make version.
Regards
Jiri Palcek
Signed-off-by: Jiri Palecek <jpalecek@web.de>
---
include/mk/automake.mk | 4 ++--
include/mk/generic_trunk_target.inc | 15 ++++++++++-----
testcases/realtime/Makefile | 16 ++++++----------
3 files changed, 18 insertions(+), 17 deletions(-)
diff --git a/include/mk/automake.mk b/include/mk/automake.mk
index 81d77c9..ba75f94 100644
--- a/include/mk/automake.mk
+++ b/include/mk/automake.mk
@@ -54,7 +54,7 @@ autoheader: include/config.h.in
include:
mkdir -p "$@"
-include/config.h.in: configure.ac $(wildcard m4/*.m4) m4/ltp-version.m4
+include/config.h.in: configure.ac $(wildcard m4/*.m4) m4/ltp-version.m4 aclocal.m4
$(AUTOHEADER)
touch $@
@@ -65,7 +65,7 @@ m4/ltp-version.m4: ChangeLog
AUTOMAKE_FILES := config.guess config.sub install-sh missing stamp-h1
automake: aclocal $(AUTOMAKE_FILES)
$(AUTOMAKE_FILES): m4/Makefile.in
-m4/Makefile.in: m4/Makefile.am
+m4/Makefile.in: m4/Makefile.am aclocal.m4
$(AUTOMAKE) -c -a
.PHONY: ac-clean ac-distclean ac-maintainer-clean
diff --git a/include/mk/generic_trunk_target.inc b/include/mk/generic_trunk_target.inc
index 7ffb9e9..deaddcc 100644
--- a/include/mk/generic_trunk_target.inc
+++ b/include/mk/generic_trunk_target.inc
@@ -52,15 +52,23 @@ RECURSIVE_TARGETS ?= all install
$(eval $(get_make_dirs))
-.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
+.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS)) $(foreach tgt,$(RECURSIVE_TARGETS),$(addprefix trunk-subdir-$(tgt)--,$(SUBDIRS)))
$(SUBDIRS): %:
mkdir -m 00755 -p "$@"
+get_trunk_target = $(firstword $(subst --, ,$(patsubst trunk-subdir-%,%,$(1))))
+get_trunk_subdir = $(patsubst trunk-subdir-$(call get_trunk_target,$(1))--%,%,$(1))
+
+$(foreach tgt,$(RECURSIVE_TARGETS),$(addprefix trunk-subdir-$(tgt)--,$(SUBDIRS))): %:
+ mkdir -m 00755 -p "$(call get_trunk_subdir,$@)"
+ $(MAKE) -C $(call get_trunk_subdir,$@) -f "$(abs_srcdir)/$(call get_trunk_subdir,$@)/Makefile" $(call get_trunk_target,$@)
+
$(MAKE_TARGETS): | $(MAKE_DEPS)
trunk-all: $(MAKE_TARGETS)
+
trunk-clean:: | $(SUBDIRS)
$(if $(strip $(CLEAN_TARGETS)),$(RM) -f $(CLEAN_TARGETS))
@@ -89,10 +97,7 @@ ifeq ($(strip $(SUBDIRS)),)
$(warning CURDIR is: $(CURDIR))
$(error SUBDIRS empty -- did you want generic_leaf_target instead?)
else
-$(RECURSIVE_TARGETS): %: | $(SUBDIRS)
- @set -e; for dir in $(SUBDIRS); do \
- $(MAKE) -C $$dir -f "$(abs_srcdir)/$$dir/Makefile" $@; \
- done
+$(RECURSIVE_TARGETS): %: $(addprefix trunk-subdir-%--,$(SUBDIRS))
endif
# vim: syntax=make
diff --git a/testcases/realtime/Makefile b/testcases/realtime/Makefile
index 9527b55..3aff429 100644
--- a/testcases/realtime/Makefile
+++ b/testcases/realtime/Makefile
@@ -54,26 +54,22 @@ ac-distclean:: ac-clean
ac-maintainer-clean:: ac-distclean
$(RM) -f aclocal.m4 configure $(AUTOMAKE_FILES) m4/Makefile.in
-aclocal:
- $(ACLOCAL) -I $(abs_srcdir)/m4
-
-autoconf:
- $(AUTOCONF)
+aclocal: aclocal.m4
-autoheader:
- $(AUTOHEADER)
+aclocal.m4:
+ $(ACLOCAL) -I $(abs_srcdir)/m4
autoconf: configure
-include/realtime_config.h.in: configure.in $(wildcard m4/*.m4)
+include/realtime_config.h.in: configure.in $(wildcard m4/*.m4) aclocal.m4
$(AUTOHEADER)
touch $@
configure: configure.in aclocal.m4
$(AUTOCONF)
-automake: aclocal $(AUTOMAKE_FILES)
+automake: $(AUTOMAKE_FILES)
$(AUTOMAKE_FILES): m4/Makefile.in
-m4/Makefile.in: m4/Makefile.am
+m4/Makefile.in: m4/Makefile.am aclocal.m4
$(AUTOMAKE) -c -a
.PHONY: autoheader
--
1.6.4.3
------------------------------------------------------------------------------
Join us December 9, 2009 for the Red Hat Virtual Experience,
a free event focused on virtualization and cloud computing.
Attend in-depth sessions from your desk. Your couch. Anywhere.
http://p.sf.net/sfu/redhat-sfdev2dev
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [LTP] [PATCH] Support for parallel compilation
[not found] <4b1944a0.8713f30a.3b2f.4a0fSMTPIN_ADDED@mx.google.com>
@ 2009-12-04 19:13 ` Mike Frysinger
2009-12-05 0:03 ` Jiří Paleček
0 siblings, 1 reply; 6+ messages in thread
From: Mike Frysinger @ 2009-12-04 19:13 UTC (permalink / raw)
To: ltp-list; +Cc: Jiri Palecek
[-- Attachment #1.1: Type: Text/Plain, Size: 393 bytes --]
On Friday 04 December 2009 12:04:06 Jiri Palecek wrote:
> -.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
> +.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
> $(foreach tgt,$(RECURSIVE_TARGETS),$(addprefix
> trunk-subdir-$(tgt)--,$(SUBDIRS)))
these foreach look like an overly complicated form of $(patsubst):
$(patsubst %,moo_%,$(DIRS))
-mike
[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
[-- Attachment #2: Type: text/plain, Size: 300 bytes --]
------------------------------------------------------------------------------
Join us December 9, 2009 for the Red Hat Virtual Experience,
a free event focused on virtualization and cloud computing.
Attend in-depth sessions from your desk. Your couch. Anywhere.
http://p.sf.net/sfu/redhat-sfdev2dev
[-- Attachment #3: Type: text/plain, Size: 155 bytes --]
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [LTP] [PATCH] Support for parallel compilation
2009-12-04 19:13 ` [LTP] [PATCH] Support for parallel compilation Mike Frysinger
@ 2009-12-05 0:03 ` Jiří Paleček
2009-12-05 8:01 ` Garrett Cooper
0 siblings, 1 reply; 6+ messages in thread
From: Jiří Paleček @ 2009-12-05 0:03 UTC (permalink / raw)
To: Mike Frysinger, ltp-list
On Fri, 04 Dec 2009 20:13:11 +0100, Mike Frysinger <vapier@gentoo.org>
wrote:
> On Friday 04 December 2009 12:04:06 Jiri Palecek wrote:
>> -.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
>> +.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
>> $(foreach tgt,$(RECURSIVE_TARGETS),$(addprefix
>> trunk-subdir-$(tgt)--,$(SUBDIRS)))
>
> these foreach look like an overly complicated form of $(patsubst):
> $(patsubst %,moo_%,$(DIRS))
It may look like it, but they aren't. What you've written is a simple
addprefix, while the foreach does some for of cartesian product
($(TARGETS)x$(DIRS)).
Jiri Palecek
------------------------------------------------------------------------------
Join us December 9, 2009 for the Red Hat Virtual Experience,
a free event focused on virtualization and cloud computing.
Attend in-depth sessions from your desk. Your couch. Anywhere.
http://p.sf.net/sfu/redhat-sfdev2dev
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [LTP] [PATCH] Support for parallel compilation
2009-12-05 0:03 ` Jiří Paleček
@ 2009-12-05 8:01 ` Garrett Cooper
2009-12-19 2:05 ` Jiří Paleček
0 siblings, 1 reply; 6+ messages in thread
From: Garrett Cooper @ 2009-12-05 8:01 UTC (permalink / raw)
To: Jiří Paleček; +Cc: ltp-list, Mike Frysinger
On Fri, Dec 4, 2009 at 4:03 PM, Jiří Paleček <jpalecek@web.de> wrote:
> On Fri, 04 Dec 2009 20:13:11 +0100, Mike Frysinger <vapier@gentoo.org>
> wrote:
>
>> On Friday 04 December 2009 12:04:06 Jiri Palecek wrote:
>>> -.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
>>> +.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
>>> $(foreach tgt,$(RECURSIVE_TARGETS),$(addprefix
>>> trunk-subdir-$(tgt)--,$(SUBDIRS)))
>>
>> these foreach look like an overly complicated form of $(patsubst):
>> $(patsubst %,moo_%,$(DIRS))
>
> It may look like it, but they aren't. What you've written is a simple
> addprefix, while the foreach does some for of cartesian product
> ($(TARGETS)x$(DIRS)).
1. I checked in the fixes for the autotools targets after verifying
them (thank you btw!).
2. Inline mkdirs are bad, because mkdir's can fail or cause failures
over NFS due to excessive I/O, which is why I quickly removed the test
-d || mkdir -p logic in the 2nd major draft of the makefile
infrastructure.
3. Although your idea for fixing the lack of recursive make'ing is
good, what about this instead?
#
# A canned set of operations for leaf makes. This creates a series of
targets, like:
#
# $(subdir)-all $(subdir)-clean $(subdir)-install:
#
# 1 - Recursive targets to act upon.
# 2 - the subdirectory to do the leaf_make on.
#
define generate_dir_dep
ifndef __$(1)_dir_dep__
__$(1)_dir_dep__ = 1
$(1):
mkdir -p "$$@"
endif
endif
define leaf_make
$$(foreach recursive_target,$(1),$$(addprefix
$(2)-$$(recursive_target))): $(1)-%: $(2)
$(MAKE) -C $* -f ""$(abs_srcdir)/$$*/Makefile" $$(patsubst $(1)-%,%,$$@)
endef
$(foreach subdir $(SUBDIRS),$(eval $(call generate_dir_dep,$(subdir))))
$(foreach subdir,$(SUBDIRS),$(eval $(call leaf_make
$(RECURSIVE_TARGETS),$(subdir))
Ultimately I suppose, it's another means to a similar end. It's just a
bit more discrete as to what's going on in the operation... if this is
a good idea, I'll modify functions.mk to match this. $(abspath) //
$(realpath) junk is already a PITA, so it might be a good idea just to
remove it in favor of keeping things as-is.
Thanks,
-Garrett
------------------------------------------------------------------------------
Join us December 9, 2009 for the Red Hat Virtual Experience,
a free event focused on virtualization and cloud computing.
Attend in-depth sessions from your desk. Your couch. Anywhere.
http://p.sf.net/sfu/redhat-sfdev2dev
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [LTP] [PATCH] Support for parallel compilation
2009-12-05 8:01 ` Garrett Cooper
@ 2009-12-19 2:05 ` Jiří Paleček
2009-12-19 4:30 ` Garrett Cooper
0 siblings, 1 reply; 6+ messages in thread
From: Jiří Paleček @ 2009-12-19 2:05 UTC (permalink / raw)
To: Garrett Cooper; +Cc: ltp-list, Mike Frysinger
Hello,
On Sat, 05 Dec 2009 09:01:52 +0100, Garrett Cooper <yanegomi@gmail.com>
wrote:
> On Fri, Dec 4, 2009 at 4:03 PM, Jiří Paleček <jpalecek@web.de> wrote:
>> On Fri, 04 Dec 2009 20:13:11 +0100, Mike Frysinger <vapier@gentoo.org>
>> wrote:
>>
>>> On Friday 04 December 2009 12:04:06 Jiri Palecek wrote:
>>>> -.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
>>>> +.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
>>>> $(foreach tgt,$(RECURSIVE_TARGETS),$(addprefix
>>>> trunk-subdir-$(tgt)--,$(SUBDIRS)))
>>>
>>> these foreach look like an overly complicated form of $(patsubst):
>>> $(patsubst %,moo_%,$(DIRS))
>>
>> It may look like it, but they aren't. What you've written is a simple
>> addprefix, while the foreach does some for of cartesian product
>> ($(TARGETS)x$(DIRS)).
>
> 1. I checked in the fixes for the autotools targets after verifying
> them (thank you btw!).
> 2. Inline mkdirs are bad, because mkdir's can fail or cause failures
> over NFS due to excessive I/O, which is why I quickly removed the test
> -d || mkdir -p logic in the 2nd major draft of the makefile
> infrastructure.
OK.
> 3. Although your idea for fixing the lack of recursive make'ing is
> good, what about this instead?
>
> #
> # A canned set of operations for leaf makes. This creates a series of
> targets, like:
> #
> # $(subdir)-all $(subdir)-clean $(subdir)-install:
> #
> # 1 - Recursive targets to act upon.
> # 2 - the subdirectory to do the leaf_make on.
> #
> define generate_dir_dep
> ifndef __$(1)_dir_dep__
> __$(1)_dir_dep__ = 1
> $(1):
> mkdir -p "$$@"
> endif
> endif
>
> define leaf_make
> $$(foreach recursive_target,$(1),$$(addprefix
> $(2)-$$(recursive_target))): $(1)-%: $(2)
> $(MAKE) -C $* -f ""$(abs_srcdir)/$$*/Makefile" $$(patsubst
> $(1)-%,%,$$@)
> endef
> $(foreach subdir $(SUBDIRS),$(eval $(call generate_dir_dep,$(subdir))))
> $(foreach subdir,$(SUBDIRS),$(eval $(call leaf_make
> $(RECURSIVE_TARGETS),$(subdir))
>
> Ultimately I suppose, it's another means to a similar end. It's just a
> bit more discrete as to what's going on in the operation... if this is
> a good idea, I'll modify functions.mk to match this. $(abspath) //
> $(realpath) junk is already a PITA, so it might be a good idea just to
> remove it in favor of keeping things as-is.
Well, I can't say I got used to your generated rules and double dollars
(you've got $* instead of $$* in the above IMHO). But if you deem it
better, fine by me.
Regards
Jiri Palecek
------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [LTP] [PATCH] Support for parallel compilation
2009-12-19 2:05 ` Jiří Paleček
@ 2009-12-19 4:30 ` Garrett Cooper
0 siblings, 0 replies; 6+ messages in thread
From: Garrett Cooper @ 2009-12-19 4:30 UTC (permalink / raw)
To: Jiří Paleček; +Cc: ltp-list, Mike Frysinger
2009/12/18 Jiří Paleček <jpalecek@web.de>:
> Hello,
>
> On Sat, 05 Dec 2009 09:01:52 +0100, Garrett Cooper <yanegomi@gmail.com>
> wrote:
>
>> On Fri, Dec 4, 2009 at 4:03 PM, Jiří Paleček <jpalecek@web.de> wrote:
>>>
>>> On Fri, 04 Dec 2009 20:13:11 +0100, Mike Frysinger <vapier@gentoo.org>
>>> wrote:
>>>
>>>> On Friday 04 December 2009 12:04:06 Jiri Palecek wrote:
>>>>>
>>>>> -.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
>>>>> +.PHONY: $(RECURSIVE_TARGETS) $(addprefix trunk-,$(RECURSIVE_TARGETS))
>>>>> $(foreach tgt,$(RECURSIVE_TARGETS),$(addprefix
>>>>> trunk-subdir-$(tgt)--,$(SUBDIRS)))
>>>>
>>>> these foreach look like an overly complicated form of $(patsubst):
>>>> $(patsubst %,moo_%,$(DIRS))
>>>
>>> It may look like it, but they aren't. What you've written is a simple
>>> addprefix, while the foreach does some for of cartesian product
>>> ($(TARGETS)x$(DIRS)).
>>
>> 1. I checked in the fixes for the autotools targets after verifying
>> them (thank you btw!).
>> 2. Inline mkdirs are bad, because mkdir's can fail or cause failures
>> over NFS due to excessive I/O, which is why I quickly removed the test
>> -d || mkdir -p logic in the 2nd major draft of the makefile
>> infrastructure.
>
> OK.
>
>> 3. Although your idea for fixing the lack of recursive make'ing is
>> good, what about this instead?
>>
>> #
>> # A canned set of operations for leaf makes. This creates a series of
>> targets, like:
>> #
>> # $(subdir)-all $(subdir)-clean $(subdir)-install:
>> #
>> # 1 - Recursive targets to act upon.
>> # 2 - the subdirectory to do the leaf_make on.
>> #
>> define generate_dir_dep
>> ifndef __$(1)_dir_dep__
>> __$(1)_dir_dep__ = 1
>> $(1):
>> mkdir -p "$$@"
>> endif
>> endif
>>
>> define leaf_make
>> $$(foreach recursive_target,$(1),$$(addprefix
>> $(2)-$$(recursive_target))): $(1)-%: $(2)
>> $(MAKE) -C $* -f ""$(abs_srcdir)/$$*/Makefile" $$(patsubst
>> $(1)-%,%,$$@)
>> endef
>> $(foreach subdir $(SUBDIRS),$(eval $(call generate_dir_dep,$(subdir))))
>> $(foreach subdir,$(SUBDIRS),$(eval $(call leaf_make
>> $(RECURSIVE_TARGETS),$(subdir))
>>
>> Ultimately I suppose, it's another means to a similar end. It's just a
>> bit more discrete as to what's going on in the operation... if this is
>> a good idea, I'll modify functions.mk to match this. $(abspath) //
>> $(realpath) junk is already a PITA, so it might be a good idea just to
>> remove it in favor of keeping things as-is.
>
> Well, I can't say I got used to your generated rules and double dollars
> (you've got $* instead of $$* in the above IMHO). But if you deem it better,
> fine by me.
Yes, you're right on the s/\Q$*\E/\Q$$*\E/. The $$ is required, as
per http://www.gnu.org/software/make/manual/make.html#Eval-Function,
unless I use $(value) (I just tested out this function and it
functions as expected on 3.80).
My primary concern when we're doing these changes is that the
changes need to be as straightforward as possible. If you disagree,
please let me know. My offer was as much of a proposal as your's was.
I see a great deal of value in what you're proposing, so please feel
free to let me know what you feel :).
Thanks!
-Garrett
------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2009-12-19 4:30 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <4b1944a0.8713f30a.3b2f.4a0fSMTPIN_ADDED@mx.google.com>
2009-12-04 19:13 ` [LTP] [PATCH] Support for parallel compilation Mike Frysinger
2009-12-05 0:03 ` Jiří Paleček
2009-12-05 8:01 ` Garrett Cooper
2009-12-19 2:05 ` Jiří Paleček
2009-12-19 4:30 ` Garrett Cooper
2009-12-04 17:04 Jiri Palecek
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox