[Buildroot] [PATCH v10 0/8] Add support for top-level parallel make

[Buildroot] [PATCH v10 0/8] Add support for top-level parallel make

[Fabio Porcedda]

Hi all,
this is a patch set for adding support for top-level parallel make in
buildroot, the common problem scattered in buildroot's top-level
makefiles is that in the rules it relies on the order of evaluation of
the prerequisites, to be able to use top-level parallel make instead
of reling on the left to right ordering of evaluation of the
prerequisites we must add an explicit rule to describe the
dependencies.

With this patch set the top-level parallel make seems to works fine,
example:
	make clean
	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))

Before to try it remeber to remove the ".NOTPARALLEL" line.

On my quad core system the building time for qemu_x86_defconfig
is 14m versus 11m and with ccache enabled 7m vs 5m.

I've tested the qemu_x86_defconfig, uclibc/eglibc/glibc,
ccache off/on, sstrip off/on.

I've finished to work on this patch set, Arnout has acked some patches so
i hope that this patch set is going to be merged soon.

Thank you for all reviews and in particular Arnout.

Best regards
Fabio Porcedda

v10:
 - Added Acked-by: Arnout Vandecappelle on the first two patches
 - Improved descption and text of the last patch
 - Rebased over master
v9:
 - Added Acked-by: Arnout Vandecappelle on some patches.
 - Changed the patches contents and descriptions following advices of Arnout
   - 1th patch: fixed to support ccache & sstrip
   - 2th patch: added pkg-cmake.mk
   - 2th patch: splitted ifeq
   - 3th patch: improved description
   - 3th patch: does not use %_BUILD_DEPENDENCIES anymore
   - 3th patch: added a comment about "|"
   - 4th patch: added a comment
   - 5th patch: added a comment
   - 6th patch: 
 - 2th patch: fixed a comment
 - 8th patch: now it change only the comment
v8:
 - rebased over master
 - added patche for for base dependency
 - added patche for for glibc package
 - added patche for for uclibc package
 - removed patch already merged
 - changed some descriptions
 - modified the patch for toolchain dependency to prevent circular dependency
v7:
 - add the latest patch
 - add to the first patch the <pkgname>_TOOLCHAIN variable
 - improve the fifth patch
v6:
 - added the fifth patch
 - updated the fourth patch adding the install targets
 - updated the second patch to remove TARGETS_ALL
v5:
 - added the fourth patch
 - fixed some typos
 - rewrited the second patch to use only $$($(2)_TARGET_*) in the rules
 - add support for top-level parallel make for the glibc package
v4:
 - rebased over master
 - add Acked-by: Thomas Petazzoni on the third patch
 - changed the orderd of the patches
v3:
 - add back the patch "package: add toolchain dependency to
    inner-generic-package" because now is working fine.
 - add Acked-by: Arnout Vandecappelle to the third patch.
 - reworked the second patch following Arnout suggestions.
v2:
 - remove patch "package: add toolchain dependency to inner-generic-package"
   because was not working fine against recent toolchain changes.

Fabio Porcedda (8):
  package: add base dependency to every package
  package: add toolchain dependency to every target package
  package: add support for top-level parallel make
  uclibc: add support for top-level parallel make
  glibc: add support for top-level parallel make
  Makefile: add support for top-level parallel make
  package: enable jobserver for recursive make
  Makefile: update comment about top-level parallel Makefile

 Makefile                                           | 34 ++++++-----
 fs/common.mk                                       |  4 +-
 package/glibc/glibc.mk                             | 12 +++-
 package/linux-headers/linux-headers.mk             |  3 +
 package/pkg-autotools.mk                           |  3 +-
 package/pkg-cmake.mk                               |  2 +-
 package/pkg-generic.mk                             | 67 ++++++++++++++--------
 package/uclibc/uclibc.mk                           | 12 +++-
 .../toolchain-buildroot/toolchain-buildroot.mk     |  2 +
 toolchain/toolchain-external/toolchain-external.mk |  2 +
 toolchain/toolchain/toolchain.mk                   |  5 +-
 11 files changed, 97 insertions(+), 49 deletions(-)

-- 
1.8.5

[Buildroot] [PATCH v10 1/8] package: add base dependency to every package

[Fabio Porcedda]

Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to
every package.
This way we can build correctly every package right after the clean
stage.
As example with this commit we can build successfully the glibc right
after the clean stage:
	make clean glibc

This is also a step forward supporting top-level parallel make.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 package/pkg-generic.mk           | 5 +++++
 toolchain/toolchain/toolchain.mk | 3 +--
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 45b808a..668c7da 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -426,6 +426,11 @@ $(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
 $(1)-build:		$(1)-configure \
 			$$($(2)_TARGET_BUILD)
 
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
+ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
+endif
+
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 # In the normal case (no package override), the sequence of steps is
 #  source, by downloading
diff --git a/toolchain/toolchain/toolchain.mk b/toolchain/toolchain/toolchain.mk
index 44ed629..8559ac9 100644
--- a/toolchain/toolchain/toolchain.mk
+++ b/toolchain/toolchain/toolchain.mk
@@ -14,5 +14,4 @@ endif
 
 $(eval $(generic-package))
 
-toolchain-source: prepare dirs dependencies $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
-
+toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
-- 
1.8.5

[Buildroot] [PATCH v10 2/8] package: add toolchain dependency to every target package

[Fabio Porcedda]

This commit makes the dependency from the target toolchain explicit.
This way we can buid from command line a package that use
innger-generic-package right after the configuration phase, example:

	make clean <package-name>

Also remove TARGETS_ALL because the only purpose was to add toolchain
dependency so it's superseded by this commit.

To prevent circular dependency add the new variable
<pkgname>_ADD_TOOLCHAIN_DEPENDENCY to avoid adding the toolchain
dependency for toolchain packages.

This is also a step forward supporting top-level parallel make.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 Makefile                                             |  8 ++------
 package/glibc/glibc.mk                               |  3 +++
 package/linux-headers/linux-headers.mk               |  3 +++
 package/pkg-autotools.mk                             |  3 ++-
 package/pkg-cmake.mk                                 |  2 +-
 package/pkg-generic.mk                               | 14 ++++++++++++--
 package/uclibc/uclibc.mk                             |  3 +++
 toolchain/toolchain-buildroot/toolchain-buildroot.mk |  2 ++
 toolchain/toolchain-external/toolchain-external.mk   |  2 ++
 toolchain/toolchain/toolchain.mk                     |  2 ++
 10 files changed, 32 insertions(+), 10 deletions(-)

diff --git a/Makefile b/Makefile
index 925c1d2..1866d0a 100644
--- a/Makefile
+++ b/Makefile
@@ -388,7 +388,6 @@ TARGETS+=target-post-image
 
 TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
 TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
-TARGETS_ALL:=$(patsubst %,__real_tgt_%,$(TARGETS))
 
 # host-* dependencies have to be handled specially, as those aren't
 # visible in Kconfig and hence not added to a variable like TARGETS.
@@ -411,9 +410,6 @@ HOST_SOURCE += $(addsuffix -source,$(sort $(TARGETS_HOST_DEPS) $(HOST_DEPS)))
 TARGETS_LEGAL_INFO:=$(patsubst %,%-legal-info,\
 		$(TARGETS) $(BASE_TARGETS) $(TARGETS_HOST_DEPS) $(HOST_DEPS))))
 
-# all targets depend on the crosscompiler and it's prerequisites
-$(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) %
-
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
 
@@ -422,11 +418,11 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG)
 
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: $(BASE_TARGETS) $(TARGETS_ALL)
+world: $(TARGETS)
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars \
-	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ALL) \
+	$(BASE_TARGETS) $(TARGETS) \
 	$(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \
 	$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
index 96de02a..0968f67 100644
--- a/package/glibc/glibc.mk
+++ b/package/glibc/glibc.mk
@@ -31,6 +31,9 @@ endif
 GLIBC_LICENSE = GPLv2+ (programs), LGPLv2.1+, BSD-3c, MIT (library)
 GLIBC_LICENSE_FILES = $(addprefix $(GLIBC_SRC_SUBDIR)/,COPYING COPYING.LIB LICENSES)
 
+# glibc is part of the toolchain so disable the toolchain dependency
+GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 # Before (e)glibc is configured, we must have the first stage
 # cross-compiler and the kernel headers
 GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
diff --git a/package/linux-headers/linux-headers.mk b/package/linux-headers/linux-headers.mk
index 30d3076..6dac9e3 100644
--- a/package/linux-headers/linux-headers.mk
+++ b/package/linux-headers/linux-headers.mk
@@ -17,6 +17,9 @@ LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
 
 LINUX_HEADERS_INSTALL_STAGING = YES
 
+# linux-headers is part of the toolchain so disable the toolchain dependency
+LINUX_HEADERS_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 define LINUX_HEADERS_INSTALL_STAGING_CMDS
 	(cd $(@D); \
 		$(TARGET_MAKE_ENV) $(MAKE) \
diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
index a66ecd4..8961248 100644
--- a/package/pkg-autotools.mk
+++ b/package/pkg-autotools.mk
@@ -206,7 +206,8 @@ endef
 # This must be repeated from inner-generic-package, otherwise we get an empty
 # _DEPENDENCIES if _AUTORECONF is YES.  Also filter the result of _AUTORECONF
 # away from the non-host rule
-$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\
+$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \
+				host-toolchain $(1),\
     $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
 
diff --git a/package/pkg-cmake.mk b/package/pkg-cmake.mk
index 463cd63..fe42253 100644
--- a/package/pkg-cmake.mk
+++ b/package/pkg-cmake.mk
@@ -88,7 +88,7 @@ endif
 
 # This must be repeated from inner-generic-package, otherwise we only get
 # host-cmake in _DEPENDENCIES because of the following line
-$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+$(2)_DEPENDENCIES ?= $(filter-out host-toolchain $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
 $(2)_DEPENDENCIES += host-cmake
 
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 668c7da..15fe303 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -349,8 +349,18 @@ endif
 
 $(2)_REDISTRIBUTE		?= YES
 
-
-$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+# When a target package is a toolchain dependency set this variable to
+# 'NO' so the 'toolchain' dependency is not added to prevent a circular
+# dependency
+$(2)_ADD_TOOLCHAIN_DEPENDENCY	?= YES
+
+$(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
+	$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+ifeq ($(5),target)
+ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
+$(2)_DEPENDENCIES += toolchain
+endif
+endif
 
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
index 9e04771..b1585af 100644
--- a/package/uclibc/uclibc.mk
+++ b/package/uclibc/uclibc.mk
@@ -19,6 +19,9 @@ endif
 
 UCLIBC_INSTALL_STAGING = YES
 
+# uclibc is part of the toolchain so disable the toolchain dependency
+UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 # Before uClibc is configured, we must have the first stage
 # cross-compiler and the kernel headers
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
diff --git a/toolchain/toolchain-buildroot/toolchain-buildroot.mk b/toolchain/toolchain-buildroot/toolchain-buildroot.mk
index f1f07a9..d941854 100644
--- a/toolchain/toolchain-buildroot/toolchain-buildroot.mk
+++ b/toolchain/toolchain-buildroot/toolchain-buildroot.mk
@@ -14,4 +14,6 @@ BUILDROOT_LIBC = $(call qstrip,$(BR2_TOOLCHAIN_BUILDROOT_LIBC))
 
 TOOLCHAIN_BUILDROOT_DEPENDENCIES = host-gcc-final
 
+TOOLCHAIN_BUILDROOT_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 $(eval $(generic-package))
diff --git a/toolchain/toolchain-external/toolchain-external.mk b/toolchain/toolchain-external/toolchain-external.mk
index bd4f375..b85a45c 100644
--- a/toolchain/toolchain-external/toolchain-external.mk
+++ b/toolchain/toolchain-external/toolchain-external.mk
@@ -362,6 +362,8 @@ TOOLCHAIN_EXTERNAL_SITE =
 TOOLCHAIN_EXTERNAL_SOURCE =
 endif
 
+TOOLCHAIN_EXTERNAL_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 TOOLCHAIN_EXTERNAL_INSTALL_STAGING = YES
 
 ifeq ($(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2012R1)$(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2012R2),y)
diff --git a/toolchain/toolchain/toolchain.mk b/toolchain/toolchain/toolchain.mk
index 8559ac9..7241fe7 100644
--- a/toolchain/toolchain/toolchain.mk
+++ b/toolchain/toolchain/toolchain.mk
@@ -12,6 +12,8 @@ else ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y)
 TOOLCHAIN_DEPENDENCIES += toolchain-external
 endif
 
+TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 $(eval $(generic-package))
 
 toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
-- 
1.8.5

[Buildroot] [PATCH v10 3/8] package: add support for top-level parallel make

[Fabio Porcedda]

To be able to use top-level parallel make we must not depend in a rule
on the order of evaluation of the prerequisites, so instead of relying
on the left to right ordering of evaluation of the prerequisites add
an explicit rule to describe the dependencies.

We cannot use the pattern rules because they must have the same
dependency for every package, but we need to change the dependencies
depending on $(2)_OVERRIDE_SRCDIR variable value, so we must use a
more flexible way like $(2)_TARGET_% variables.

So add explicit dependencies for the following stamp files:
  $(2)_TARGET_EXTRACT
  $(2)_TARGET_PATCH
  $(2)_TARGET_CONFIGURE
  $(2)_TARGET_BUILD
  $(2)_TARGET_INSTALL_STAGING
  $(2)_TARGET_INSTALL_TARGET
  $(2)_TARGET_INSTALL_IMAGES
  $(2)_TARGET_INSTALL_HOST

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/glibc/glibc.mk   |  6 +++---
 package/pkg-generic.mk   | 40 +++++++++++++++++++++++-----------------
 package/uclibc/uclibc.mk |  6 +++---
 3 files changed, 29 insertions(+), 23 deletions(-)

diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
index 0968f67..3013df3 100644
--- a/package/glibc/glibc.mk
+++ b/package/glibc/glibc.mk
@@ -38,9 +38,6 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
 # cross-compiler and the kernel headers
 GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
 
-# Before (e)glibc is built, we must have the second stage cross-compiler
-glibc-build: host-gcc-intermediate
-
 GLIBC_SUBDIR = build
 
 GLIBC_INSTALL_STAGING = YES
@@ -142,3 +139,6 @@ define GLIBC_INSTALL_TARGET_CMDS
 endef
 
 $(eval $(autotools-package))
+
+# Before (e)glibc is built, we must have the second stage cross-compiler
+$(GLIBC_TARGET_BUILD): | host-gcc-intermediate
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 15fe303..fc2e04e 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -411,30 +411,37 @@ $(1)-install:		$(1)-install-staging $(1)-install-target $(1)-install-images
 endif
 
 ifeq ($$($(2)_INSTALL_TARGET),YES)
-$(1)-install-target:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_TARGET)
+$(1)-install-target:	$$($(2)_TARGET_INSTALL_TARGET)
 else
 $(1)-install-target:
 endif
 
 ifeq ($$($(2)_INSTALL_STAGING),YES)
-$(1)-install-staging:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_STAGING)
+$(1)-install-staging:	$$($(2)_TARGET_INSTALL_STAGING)
 else
 $(1)-install-staging:
 endif
 
 ifeq ($$($(2)_INSTALL_IMAGES),YES)
-$(1)-install-images:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_IMAGES)
+$(1)-install-images:	$$($(2)_TARGET_INSTALL_IMAGES)
 else
 $(1)-install-images:
 endif
 
-$(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
+$(1)-install-host:      $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-build:		$(1)-configure \
-			$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \
+$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \
+	$$($(2)_TARGET_BUILD)
+
+$(1)-build:		$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_BUILD):	$$($(2)_TARGET_CONFIGURE)
+
+# The symbol "|" specify a order-only-prerequisite, this is needed for
+# phony requisites to avoid rebuilding every time the target.
+
+$(1)-configure:			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	| $$($(2)_DEPENDENCIES)
 
 $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
 ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
@@ -448,13 +455,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 #  extract
 #  patch
 #  configure
-$(1)-configure:		$(1)-patch $(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_PATCH)
 
-$(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
+$(1)-patch:		$$($(2)_TARGET_PATCH)
+$$($(2)_TARGET_PATCH):	$$($(2)_TARGET_EXTRACT)
 
-$(1)-extract:		$(1)-source \
-			$$($(2)_TARGET_EXTRACT)
+$(1)-extract:			$$($(2)_TARGET_EXTRACT)
+$$($(2)_TARGET_EXTRACT):	$$($(2)_TARGET_SOURCE)
 
 $(1)-depends:		$$($(2)_DEPENDENCIES)
 
@@ -464,10 +471,9 @@ else
 #  source, by rsyncing
 #  depends
 #  configure
-$(1)-configure:		$(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_RSYNC)
 
-$(1)-depends:		$(1)-rsync $$($(2)_DEPENDENCIES)
+$(1)-depends:		$$($(2)_DEPENDENCIES)
 
 $(1)-patch:		$(1)-rsync
 $(1)-extract:		$(1)-rsync
diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
index b1585af..56761f6 100644
--- a/package/uclibc/uclibc.mk
+++ b/package/uclibc/uclibc.mk
@@ -26,9 +26,6 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
 # cross-compiler and the kernel headers
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
 
-# Before uClibc is built, we must have the second stage cross-compiler
-uclibc-build: host-gcc-intermediate
-
 # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config
 # setting.
 ifndef UCLIBC_CONFIG_FILE
@@ -551,3 +548,6 @@ uclibc-menuconfig: dirs uclibc-patch
 	rm -f $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed}
 
 $(eval $(generic-package))
+
+# Before uClibc is built, we must have the second stage cross-compiler
+$(UCLIBC_TARGET_BUILD): | host-gcc-intermediate
-- 
1.8.5

[Buildroot] [PATCH v10 4/8] uclibc: add support for top-level parallel make

[Fabio Porcedda]

To be able to use top-level parallel make we must not depend in a rule
on the order of evaluation of the prerequisites, so instead of relyng
on the left to right ordering of evaluation of the prerequisites add
an explicit rule to describe the dependencies.

The uclibc-install-target depends on uclibc-install-staging so add a
rule for it.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 package/uclibc/uclibc.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
index 56761f6..e74ec64 100644
--- a/package/uclibc/uclibc.mk
+++ b/package/uclibc/uclibc.mk
@@ -551,3 +551,6 @@ $(eval $(generic-package))
 
 # Before uClibc is built, we must have the second stage cross-compiler
 $(UCLIBC_TARGET_BUILD): | host-gcc-intermediate
+
+# The uclibc-install-target use files from uclibc-install-staging
+$(UCLIBC_TARGET_INSTALL_TARGET): $(UCLIBC_TARGET_INSTALL_STAGING)
-- 
1.8.5

[Buildroot] [PATCH v10 5/8] glibc: add support for top-level parallel make

[Fabio Porcedda]

To be able to use top-level parallel make we must not depend in a rule
on the order of evaluation of the prerequisites, so instead of relyng
on the left to right ordering of evaluation of the prerequisites add
an explicit rule to describe the dependencies.

The glibc-install-target depends on glibc-install-staging so add a
rule for it.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 package/glibc/glibc.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
index 3013df3..621ce5f 100644
--- a/package/glibc/glibc.mk
+++ b/package/glibc/glibc.mk
@@ -142,3 +142,6 @@ $(eval $(autotools-package))
 
 # Before (e)glibc is built, we must have the second stage cross-compiler
 $(GLIBC_TARGET_BUILD): | host-gcc-intermediate
+
+# The glic-install-target use files from glibc-install-staging
+$(GLIBC_TARGET_INSTALL_TARGET): $(GLIBC_TARGET_INSTALL_STAGING)
-- 
1.8.5

[Buildroot] [PATCH v10 6/8] Makefile: add support for top-level parallel make

[Fabio Porcedda]

To be able to use top-level parallel make we must not depend in a rule
on the order of evaluation of the prerequisites, so instead of relyng on
the left to right ordering of evaluation of the prerequisites add an
explicit rule to describe the dependencies.

Add explicit rules to describe the following dependency chain:
$(TARGETS) -> target-finalize -> rootfs-* -> target-post-image

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 Makefile     | 15 ++++++---------
 fs/common.mk |  4 ++--
 2 files changed, 8 insertions(+), 11 deletions(-)

diff --git a/Makefile b/Makefile
index 1866d0a..ef2582e 100644
--- a/Makefile
+++ b/Makefile
@@ -366,8 +366,6 @@ include system/system.mk
 
 include $(BR2_EXTERNAL)/external.mk
 
-TARGETS+=target-finalize
-
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
 TARGETS+=target-purgelocales
 endif
@@ -384,8 +382,6 @@ endif
 
 include fs/common.mk
 
-TARGETS+=target-post-image
-
 TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
 TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
 
@@ -418,11 +414,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG)
 
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: $(TARGETS)
+world: target-post-image
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars \
-	$(BASE_TARGETS) $(TARGETS) \
+	target-finalize target-post-image \
+	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \
 	$(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \
 	$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
@@ -482,7 +479,7 @@ endif
 STRIP_FIND_CMD += -type f -perm /111
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print
 
-target-finalize:
+target-finalize: $(TARGETS)
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
 		$(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmake
@@ -588,7 +585,7 @@ target-generatelocales: host-localedef
 	done
 endif
 
-target-post-image:
+target-post-image: target-finalize $(TARGETS_ROOTFS)
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \
 		$(USER_HOOKS_EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
@@ -625,7 +622,7 @@ legal-info: dirs legal-info-clean legal-info-prepare $(TARGETS_LEGAL_INFO) \
 	@rm -f $(LEGAL_WARNINGS)
 
 show-targets:
-	@echo $(TARGETS)
+	@echo $(TARGETS) $(TARGETS_ROOTFS)
 
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
diff --git a/fs/common.mk b/fs/common.mk
index aa9d961..0d054b5 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -63,7 +63,7 @@ ROOTFS_$(2)_COMPRESS_EXT = .xz
 ROOTFS_$(2)_COMPRESS_CMD = $$(XZ) -9 -C crc32 -c
 endif
 
-$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
+$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
 	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
 	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
 	rm -f $$(FAKEROOT_SCRIPT)
@@ -94,7 +94,7 @@ rootfs-$(1)-show-depends:
 rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
-TARGETS += rootfs-$(1)
+TARGETS_ROOTFS += rootfs-$(1)
 endif
 endef
 
-- 
1.8.5

[Buildroot] [PATCH v10 7/8] package: enable jobserver for recursive make

[Fabio Porcedda]

Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS)
commands to enable jobserver for the sub-make.

Without the '+' prefix GNU make does not detect the sub-make so it
disable the jobserver for the sub-make.

From GNU make documentation:
Using the MAKE variable has the same effect as using a ?+? character
at the beginning of the recipe line.  This special feature is only
enabled if the MAKE variable appears directly in the recipe: it does
not apply if the MAKE variable is referenced through expansion of
another variable. In the latter case you must use the ?+? token to get
these special effects.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-generic.mk | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index fc2e04e..95e56bc 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -170,7 +170,7 @@ $(BUILD_DIR)/%/.stamp_configured:
 $(BUILD_DIR)/%/.stamp_built::
 	@$(call step_start,build)
 	@$(call MESSAGE,"Building")
-	$($(PKG)_BUILD_CMDS)
+	+$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 	@$(call step_end,build)
@@ -179,7 +179,7 @@ $(BUILD_DIR)/%/.stamp_built::
 $(BUILD_DIR)/%/.stamp_host_installed:
 	@$(call step_start,install-host)
 	@$(call MESSAGE,"Installing to host directory")
-	$($(PKG)_INSTALL_CMDS)
+	+$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 	@$(call step_end,install-host)
@@ -188,7 +188,7 @@ $(BUILD_DIR)/%/.stamp_host_installed:
 $(BUILD_DIR)/%/.stamp_staging_installed:
 	@$(call step_start,install-staging)
 	@$(call MESSAGE,"Installing to staging directory")
-	$($(PKG)_INSTALL_STAGING_CMDS)
+	+$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
 	$(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \
 		$(call MESSAGE,"Fixing package configuration files") ;\
@@ -204,7 +204,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 $(BUILD_DIR)/%/.stamp_images_installed:
 	@$(call step_start,install-image)
 	@$(call MESSAGE,"Installing to images directory")
-	$($(PKG)_INSTALL_IMAGES_CMDS)
+	+$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 	@$(call step_end,install-image)
@@ -217,7 +217,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
 	$(if $(BR2_INIT_SYSV)$(BR2_INIT_BUSYBOX),\
 		$($(PKG)_INSTALL_INIT_SYSV))
-	$($(PKG)_INSTALL_TARGET_CMDS)
+	+$($(PKG)_INSTALL_TARGET_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep))
 	$(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \
 		$(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
-- 
1.8.5

[Buildroot] [PATCH v10 8/8] Makefile: update comment about top-level parallel Makefile

[Fabio Porcedda]

After the latest patches top-level parallel Makefile is working but
there is still an issue when a package has an unspecified optional
dependency so change the comment to explain that.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 Makefile | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index ef2582e..27a6a65 100644
--- a/Makefile
+++ b/Makefile
@@ -42,7 +42,20 @@ export HOSTARCH := $(shell uname -m | \
 	    -e s/macppc/powerpc/\
 	    -e s/sh.*/sh/)
 
-# This top-level Makefile can *not* be executed in parallel
+# Parallel execution of this Makefile is disabled because it could
+# change the build result if a package has an unspecified optional
+# dependency.
+# Parallel execution changes the packages building order, that can be
+# a problem when a package has an unspecified optional dependency,
+# because if that dependency is present when the package is built, it
+# is used, otherwise it isn't (but compilation happily proceeds). This
+# means that the packages building order is relevant in that case, and
+# the end result will differ if the order is swapped due to parallel
+# building.
+# Taking in account the above warning, if you still want to execute
+# this top-level Makefile in parrallel comment the following line and
+# execute:
+#	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))
 .NOTPARALLEL:
 
 # absolute path
-- 
1.8.5

[Buildroot] [PATCH v10 7/8] package: enable jobserver for recursive make

[Arnout Vandecappelle]

On 18/12/13 11:36, Fabio Porcedda wrote:
> Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS)
> commands to enable jobserver for the sub-make.
>
> Without the '+' prefix GNU make does not detect the sub-make so it
> disable the jobserver for the sub-make.

  $(PKG)_BUILD_CMDS is a multiline define, and the recursive make call 
may not be the first command. Can you check if this still works if the 
build commands are something like:

define FOO_BUILD_CMDS
	echo "Hello world!"
	$(MAKE) -C $(@D)
endef


  Regards,
  Arnout

>
> From GNU make documentation:
> Using the MAKE variable has the same effect as using a ?+? character
> at the beginning of the recipe line.  This special feature is only
> enabled if the MAKE variable appears directly in the recipe: it does
> not apply if the MAKE variable is referenced through expansion of
> another variable. In the latter case you must use the ?+? token to get
> these special effects.
>
> Signed-off-by: Fabio Porcedda<fabio.porcedda@gmail.com>
[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v10 0/8] Add support for top-level parallel make

[Arnout Vandecappelle]

On 18/12/13 11:36, Fabio Porcedda wrote:
> this is a patch set for adding support for top-level parallel make in
> buildroot, the common problem scattered in buildroot's top-level
> makefiles is that in the rules it relies on the order of evaluation of
> the prerequisites, to be able to use top-level parallel make instead
> of reling on the left to right ordering of evaluation of the
> prerequisites we must add an explicit rule to describe the
> dependencies.
>
> With this patch set the top-level parallel make seems to works fine,
> example:
> 	make clean
> 	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))

  Would it be possible to disable BR2_JLEVEL automatically when -j 
appears on the command line?

>
> Before to try it remeber to remove the ".NOTPARALLEL" line.
>
> On my quad core system the building time for qemu_x86_defconfig
> is 14m versus 11m and with ccache enabled 7m vs 5m.

  These are not very relevant benchmarks because there's not a lot that 
can be done in parallel there. It would be much more interesting with an 
allpackageyesconfig (which you'll have to modify by hand BTW to exclude 
some packages that don't build...).

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v10 8/8] Makefile: update comment about top-level parallel Makefile

[Arnout Vandecappelle]

On 18/12/13 11:36, Fabio Porcedda wrote:
> After the latest patches top-level parallel Makefile is working but
> there is still an issue when a package has an unspecified optional
> dependency so change the comment to explain that.
>
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
> ---
>   Makefile | 15 ++++++++++++++-
>   1 file changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/Makefile b/Makefile
> index ef2582e..27a6a65 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -42,7 +42,20 @@ export HOSTARCH := $(shell uname -m | \
>   	    -e s/macppc/powerpc/\
>   	    -e s/sh.*/sh/)
>
> -# This top-level Makefile can *not* be executed in parallel
> +# Parallel execution of this Makefile is disabled because it could
> +# change the build result if a package has an unspecified optional
> +# dependency.

  ... or if two packages manipulate the same file in the target directory.

> +# Parallel execution changes the packages building order, that can be
> +# a problem when a package has an unspecified optional dependency,
> +# because if that dependency is present when the package is built, it
> +# is used, otherwise it isn't (but compilation happily proceeds). This

  This is a repeat of the first sentence.

> +# means that the packages building order is relevant in that case, and
> +# the end result will differ if the order is swapped due to parallel
> +# building.
> +# Taking in account the above warning, if you still want to execute

  Taking into acount

> +# this top-level Makefile in parrallel comment the following line and

  parallel


  Regards,
  Arnout

> +# execute:
> +#	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))
>   .NOTPARALLEL:
>
>   # absolute path
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v10 7/8] package: enable jobserver for recursive make

[Peter Korsgaard]

>>>>> "Fabio" == Fabio Porcedda <fabio.porcedda@gmail.com> writes:

 > Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS)
 > commands to enable jobserver for the sub-make.

 > Without the '+' prefix GNU make does not detect the sub-make so it
 > disable the jobserver for the sub-make.

 > From GNU make documentation:
 > Using the MAKE variable has the same effect as using a ?+? character
 > at the beginning of the recipe line.  This special feature is only
 > enabled if the MAKE variable appears directly in the recipe: it does
 > not apply if the MAKE variable is referenced through expansion of
 > another variable. In the latter case you must use the ?+? token to get
 > these special effects.

Wouldn't it make more sense to add the '+' to MAKE / MAKE1?

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v10 7/8] package: enable jobserver for recursive make

[Fabio Porcedda]

Hi Peter,
thanks for reviewing,

On Thu, Dec 19, 2013 at 9:32 PM, Peter Korsgaard <jacmet@uclibc.org> wrote:
>>>>>> "Fabio" == Fabio Porcedda <fabio.porcedda@gmail.com> writes:
>
>  > Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS)
>  > commands to enable jobserver for the sub-make.
>
>  > Without the '+' prefix GNU make does not detect the sub-make so it
>  > disable the jobserver for the sub-make.
>
>  > From GNU make documentation:
>  > Using the MAKE variable has the same effect as using a ?+? character
>  > at the beginning of the recipe line.  This special feature is only
>  > enabled if the MAKE variable appears directly in the recipe: it does
>  > not apply if the MAKE variable is referenced through expansion of
>  > another variable. In the latter case you must use the ?+? token to get
>  > these special effects.
>
> Wouldn't it make more sense to add the '+' to MAKE / MAKE1?

Unfortunately that does not work, the '+' must be the first character
of the command line, e.g.:

 PATH=/home/fabiopo/buildroot/output/host/bin:/home/fabiopo/buildroot/output/host/usr/bin:/home/fabiopo/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
LD_LIBRARY_PATH="/home/fabiopo/buildroot/output/host/usr/lib:"
PKG_CONFIG="/home/fabiopo/buildroot/output/host/usr/bin/pkg-config"
PKG_CONFIG_SYSROOT_DIR="/"
PKG_CONFIG_LIBDIR="/home/fabiopo/buildroot/output/host/usr/lib/pkgconfig"
PERLLIB="/home/fabiopo/buildroot/output/host/usr/lib/perl"
+/usr/bin/make -j5  -C
/home/fabiopo/buildroot/output/build/host-binutils-2.22/
/bin/sh: +/usr/bin/make: No such file or directory
make: *** [/home/fabiopo/buildroot/output/build/host-binutils-2.22/.stamp_built]
Error 127

Best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v10 7/8] package: enable jobserver for recursive make

[Fabio Porcedda]

On Thu, Dec 19, 2013 at 6:27 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/12/13 11:36, Fabio Porcedda wrote:
>>
>> Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS)
>> commands to enable jobserver for the sub-make.
>>
>> Without the '+' prefix GNU make does not detect the sub-make so it
>> disable the jobserver for the sub-make.
>
>
>  $(PKG)_BUILD_CMDS is a multiline define, and the recursive make call may
> not be the first command. Can you check if this still works if the build
> commands are something like:
>
> define FOO_BUILD_CMDS
>         echo "Hello world!"
>         $(MAKE) -C $(@D)
> endef

The following test seems to work fine:

define test
    echo test
    $($(PKG)_BUILD_CMDS)
endef

# Build
$(BUILD_DIR)/%/.stamp_built::
    @$(call step_start,build)
    @$(call MESSAGE,"Building")
    +$(test)
    ...

Test output:
-----
$ make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1)) -s wget-build
>>> wget 1.14 Building
test
Making all in lib
Making all in src
Making all in doc
Making all in po
Making all in tests
Making all in util
----

If i remove the '+' charter:
----
$ make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1)) -s wget-build
>>> wget 1.14 Building
test
make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent
make rule.
Making all in lib
Making all in src
Making all in doc
Making all in po
Making all in tests
Making all in util
----

Best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v10 8/8] Makefile: update comment about top-level parallel Makefile

[Fabio Porcedda]

Hi Arnout,
thanks for reviewing.

On Thu, Dec 19, 2013 at 6:37 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/12/13 11:36, Fabio Porcedda wrote:
>>
>> After the latest patches top-level parallel Makefile is working but
>> there is still an issue when a package has an unspecified optional
>> dependency so change the comment to explain that.
>>
>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>> ---
>>   Makefile | 15 ++++++++++++++-
>>   1 file changed, 14 insertions(+), 1 deletion(-)
>>
>> diff --git a/Makefile b/Makefile
>> index ef2582e..27a6a65 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -42,7 +42,20 @@ export HOSTARCH := $(shell uname -m | \
>>             -e s/macppc/powerpc/\
>>             -e s/sh.*/sh/)
>>
>> -# This top-level Makefile can *not* be executed in parallel
>> +# Parallel execution of this Makefile is disabled because it could
>> +# change the build result if a package has an unspecified optional
>> +# dependency.
>
>
>  ... or if two packages manipulate the same file in the target directory.

Ok, added.

>
>> +# Parallel execution changes the packages building order, that can be
>> +# a problem when a package has an unspecified optional dependency,
>> +# because if that dependency is present when the package is built, it
>> +# is used, otherwise it isn't (but compilation happily proceeds). This
>
>
>  This is a repeat of the first sentence.

So i just remove it?

>
>> +# means that the packages building order is relevant in that case, and
>> +# the end result will differ if the order is swapped due to parallel
>> +# building.
>> +# Taking in account the above warning, if you still want to execute
>
>
>  Taking into acount

Ok fixed.

>
>> +# this top-level Makefile in parrallel comment the following line and
>
>
>  parallel

Ok fixed.

<snip>

Best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v10 0/8] Add support for top-level parallel make

[Fabio Porcedda]

On Thu, Dec 19, 2013 at 6:33 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/12/13 11:36, Fabio Porcedda wrote:
>>
>> this is a patch set for adding support for top-level parallel make in
>> buildroot, the common problem scattered in buildroot's top-level
>> makefiles is that in the rules it relies on the order of evaluation of
>> the prerequisites, to be able to use top-level parallel make instead
>> of reling on the left to right ordering of evaluation of the
>> prerequisites we must add an explicit rule to describe the
>> dependencies.
>>
>> With this patch set the top-level parallel make seems to works fine,
>> example:
>>         make clean
>>         make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))
>
>
>  Would it be possible to disable BR2_JLEVEL automatically when -j appears on
> the command line?

It would be great but i don't know i way to check if the -j options is
being used.

>
>>
>> Before to try it remeber to remove the ".NOTPARALLEL" line.
>>
>> On my quad core system the building time for qemu_x86_defconfig
>> is 14m versus 11m and with ccache enabled 7m vs 5m.
>
>
>  These are not very relevant benchmarks because there's not a lot that can
> be done in parallel there. It would be much more interesting with an
> allpackageyesconfig (which you'll have to modify by hand BTW to exclude some
> packages that don't build...).

You are right, i've already tried to do that but disabling all the
falling packages takes time,
i will try do that again.

It would be best if at least in the default configuration all the
packages build fine.
The autobuild does not try to test allpackageyesconfig, does it?

Best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v10 7/8] package: enable jobserver for recursive make

[Peter Korsgaard]

>>>>> "Fabio" == Fabio Porcedda <fabio.porcedda@gmail.com> writes:

Hi,

 >> Wouldn't it make more sense to add the '+' to MAKE / MAKE1?

 > Unfortunately that does not work, the '+' must be the first character
 > of the command line, e.g.:

Ahh, ok :/

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v10 0/8] Add support for top-level parallel make

[Thomas Petazzoni]

Dear Fabio Porcedda,

On Fri, 20 Dec 2013 15:36:56 +0100, Fabio Porcedda wrote:

> The autobuild does not try to test allpackageyesconfig, does it?

But, the autobuild is not designed for that. However, an
allyespackageconfig configuration is something we could add to the
Jenkins if you want.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v10 0/8] Add support for top-level parallel make

[Arnout Vandecappelle]

On 20/12/13 16:34, Thomas Petazzoni wrote:
> Dear Fabio Porcedda,
>
> On Fri, 20 Dec 2013 15:36:56 +0100, Fabio Porcedda wrote:
>
>> The autobuild does not try to test allpackageyesconfig, does it?
>
> But, the autobuild is not designed for that. However, an
> allyespackageconfig configuration is something we could add to the
> Jenkins if you want.

  If you do that, expect it to fail :-)

  It will certainly fail if you include a kernel in that config: some 
packages (Xenomai, rtnet, some modules) only work with certain kernel 
versions and AFAIK there is no single kernel that works for all these 
packages.

  Not building a kernel excludes many packages so may give you more luck.

  Regards,
  Arnout
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v10 7/8] package: enable jobserver for recursive make

[Arnout Vandecappelle]

On 20/12/13 14:24, Fabio Porcedda wrote:
> On Thu, Dec 19, 2013 at 6:27 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>> On 18/12/13 11:36, Fabio Porcedda wrote:
>>>
>>> Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS)
>>> commands to enable jobserver for the sub-make.
>>>
>>> Without the '+' prefix GNU make does not detect the sub-make so it
>>> disable the jobserver for the sub-make.
>>
>>
>>   $(PKG)_BUILD_CMDS is a multiline define, and the recursive make call may
>> not be the first command. Can you check if this still works if the build
>> commands are something like:
>>
>> define FOO_BUILD_CMDS
>>          echo "Hello world!"
>>          $(MAKE) -C $(@D)
>> endef
>
> The following test seems to work fine:

  OK, good!

  Regards,
  Arnout

>
> define test
>      echo test
>      $($(PKG)_BUILD_CMDS)
> endef
>
> # Build
> $(BUILD_DIR)/%/.stamp_built::
>      @$(call step_start,build)
>      @$(call MESSAGE,"Building")
>      +$(test)
>      ...
>
> Test output:
> -----
> $ make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1)) -s wget-build
>>>> wget 1.14 Building
> test
> Making all in lib
> Making all in src
> Making all in doc
> Making all in po
> Making all in tests
> Making all in util
> ----
>
> If i remove the '+' charter:
> ----
> $ make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1)) -s wget-build
>>>> wget 1.14 Building
> test
> make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent
> make rule.
> Making all in lib
> Making all in src
> Making all in doc
> Making all in po
> Making all in tests
> Making all in util
> ----
>
> Best regards
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v10 8/8] Makefile: update comment about top-level parallel Makefile

[Arnout Vandecappelle]

On 20/12/13 14:56, Fabio Porcedda wrote:
> On Thu, Dec 19, 2013 at 6:37 PM, Arnout Vandecappelle<arnout@mind.be>  wrote:
>> >On 18/12/13 11:36, Fabio Porcedda wrote:
>>> >>
>>> >>After the latest patches top-level parallel Makefile is working but
>>> >>there is still an issue when a package has an unspecified optional
>>> >>dependency so change the comment to explain that.
>>> >>
>>> >>Signed-off-by: Fabio Porcedda<fabio.porcedda@gmail.com>
>>> >>---
>>> >>   Makefile | 15 ++++++++++++++-
>>> >>   1 file changed, 14 insertions(+), 1 deletion(-)
>>> >>
>>> >>diff --git a/Makefile b/Makefile
>>> >>index ef2582e..27a6a65 100644
>>> >>--- a/Makefile
>>> >>+++ b/Makefile
>>> >>@@ -42,7 +42,20 @@ export HOSTARCH := $(shell uname -m | \
>>> >>             -e s/macppc/powerpc/\
>>> >>             -e s/sh.*/sh/)
>>> >>
>>> >>-# This top-level Makefile can*not*  be executed in parallel
>>> >>+# Parallel execution of this Makefile is disabled because it could
>>> >>+# change the build result if a package has an unspecified optional
>>> >>+# dependency.
>> >
>> >
>> >  ... or if two packages manipulate the same file in the target directory.
> Ok, added.
>
>> >
>>> >>+# Parallel execution changes the packages building order, that can be
>>> >>+# a problem when a package has an unspecified optional dependency,
>>> >>+# because if that dependency is present when the package is built, it
>>> >>+# is used, otherwise it isn't (but compilation happily proceeds). This
>> >
>> >
>> >  This is a repeat of the first sentence.
> So i just remove it?
>

  Probably better to remove the first sentence, the second one is more 
descriptive.

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v10 0/8] Add support for top-level parallel make

[Fabio Porcedda]

On Fri, Dec 20, 2013 at 4:46 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 20/12/13 16:34, Thomas Petazzoni wrote:
>>
>> Dear Fabio Porcedda,
>>
>> On Fri, 20 Dec 2013 15:36:56 +0100, Fabio Porcedda wrote:
>>
>>> The autobuild does not try to test allpackageyesconfig, does it?
>>
>>
>> But, the autobuild is not designed for that. However, an
>> allyespackageconfig configuration is something we could add to the
>> Jenkins if you want.

I think It would be great.
IMHO it's a useful test because this way we know if packages build
fine at least with the default config.
Building again the default config is the first test that a package
must pass, doesn't it?

>
>  If you do that, expect it to fail :-)
>
>  It will certainly fail if you include a kernel in that config: some
> packages (Xenomai, rtnet, some modules) only work with certain kernel
> versions and AFAIK there is no single kernel that works for all these
> packages.
>
>  Not building a kernel excludes many packages so may give you more luck.

Thanks and best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v10 8/8] Makefile: update comment about top-level parallel Makefile

[Fabio Porcedda]

On Fri, Dec 20, 2013 at 4:48 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 20/12/13 14:56, Fabio Porcedda wrote:
>>
>> On Thu, Dec 19, 2013 at 6:37 PM, Arnout Vandecappelle<arnout@mind.be>
>> wrote:
>>>
>>> >On 18/12/13 11:36, Fabio Porcedda wrote:
>>>>
>>>> >>
>>>> >>After the latest patches top-level parallel Makefile is working but
>>>> >>there is still an issue when a package has an unspecified optional
>>>> >>dependency so change the comment to explain that.
>>>> >>
>>>> >>Signed-off-by: Fabio Porcedda<fabio.porcedda@gmail.com>
>>>> >>---
>>>> >>   Makefile | 15 ++++++++++++++-
>>>> >>   1 file changed, 14 insertions(+), 1 deletion(-)
>>>> >>
>>>> >>diff --git a/Makefile b/Makefile
>>>> >>index ef2582e..27a6a65 100644
>>>> >>--- a/Makefile
>>>> >>+++ b/Makefile
>>>> >>@@ -42,7 +42,20 @@ export HOSTARCH := $(shell uname -m | \
>>>> >>             -e s/macppc/powerpc/\
>>>> >>             -e s/sh.*/sh/)
>>>> >>
>>>> >>-# This top-level Makefile can*not*  be executed in parallel
>>>>
>>>> >>+# Parallel execution of this Makefile is disabled because it could
>>>> >>+# change the build result if a package has an unspecified optional
>>>> >>+# dependency.
>>>
>>> >
>>> >
>>> >  ... or if two packages manipulate the same file in the target
>>> > directory.
>>
>> Ok, added.
>>
>>> >
>>>>
>>>> >>+# Parallel execution changes the packages building order, that can be
>>>> >>+# a problem when a package has an unspecified optional dependency,
>>>> >>+# because if that dependency is present when the package is built, it
>>>> >>+# is used, otherwise it isn't (but compilation happily proceeds).
>>>> >> This
>>>
>>> >
>>> >
>>> >  This is a repeat of the first sentence.
>>
>> So i just remove it?
>>
>
>  Probably better to remove the first sentence, the second one is more
> descriptive.

This is the fixed comment:

# Parallel execution of this Makefile is disabled because it changes
# the packages building order, that can be a problem when a package
# has an unspecified optional dependency, because if that dependency
# is present when the package is built, it is used, otherwise it isn't
# (but compilation happily proceeds). This means that the packages
# building order is relevant in that case, and the end result will
# differ if the order is swapped due to parallel building.  Also
# changing the building order can be a problem if two packages
# manipulate the same file in the target directory.
# Taking into account the above warnings, if you still want to execute
# this top-level Makefile in parallel comment the following line and
# execute:
#    make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))

Regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v10 8/8] Makefile: update comment about top-level parallel Makefile

[Mike Zick]

On Tue, 7 Jan 2014 11:05:52 +0100
Fabio Porcedda <fabio.porcedda@gmail.com> wrote:

> This is the fixed comment:
> 
> # Parallel execution of this Makefile is disabled because it changes
> # the packages building order, that can be a problem when a package
> # has an unspecified optional dependency, because if that dependency
> # is present when the package is built, it is used, otherwise it isn't
> # (but compilation happily proceeds). This means that the packages
> # building order is relevant in that case, and the end result will
> # differ if the order is swapped due to parallel building.  Also
> # changing the building order can be a problem if two packages
> # manipulate the same file in the target directory.
> # Taking into account the above warnings, if you still want to execute
> # this top-level Makefile in parallel comment the following line and
> # execute:
> #    make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))
> 
> Regards
>

Been a silent reader for years here now, but I just must make a
suggestion:

# Parallel execution of this top-level Makefile is disabled.
#
# Single execution is forced here for two reasons:
#
# Maintaining reproducible builds:
# Packages with an unspecified, optional, dependency could
# otherwise change the build order among multiple builds.
#
# Prevent file modification collisions:
# This can happen if two packages manipulate the same file.
#
# After taking into account the above considerations;
# If you still want to execute this top-level Makefile in 
# parallel, comment out the following line and then execute:
#    make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))

Just clarify which line is "the following line" in the context
of this comment. 

Add styling to first line of each paragraph to match the context.

- - - -

I will return now to my read-only mode.
Mike

[Buildroot] [PATCH v10 0/8] Add support for top-level parallel make

[Fabio Porcedda]

On Thu, Dec 19, 2013 at 6:33 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/12/13 11:36, Fabio Porcedda wrote:
<snip>
>> On my quad core system the building time for qemu_x86_defconfig
>> is 14m versus 11m and with ccache enabled 7m vs 5m.
>
>
>  These are not very relevant benchmarks because there's not a lot that can
> be done in parallel there. It would be much more interesting with an
> allpackageyesconfig (which you'll have to modify by hand BTW to exclude some
> packages that don't build...).

I've build  the first 220 packages of the "allpackageyesconfig", now
the numbers look better: 43m vs 18m.

Best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v10 8/8] Makefile: update comment about top-level parallel Makefile

[Fabio Porcedda]

On Tue, Jan 7, 2014 at 2:17 PM, Mike Zick <minimod@morethan.org> wrote:
> On Tue, 7 Jan 2014 11:05:52 +0100
> Fabio Porcedda <fabio.porcedda@gmail.com> wrote:
>
>> This is the fixed comment:
>>
>> # Parallel execution of this Makefile is disabled because it changes
>> # the packages building order, that can be a problem when a package
>> # has an unspecified optional dependency, because if that dependency
>> # is present when the package is built, it is used, otherwise it isn't
>> # (but compilation happily proceeds). This means that the packages
>> # building order is relevant in that case, and the end result will
>> # differ if the order is swapped due to parallel building.  Also
>> # changing the building order can be a problem if two packages
>> # manipulate the same file in the target directory.
>> # Taking into account the above warnings, if you still want to execute
>> # this top-level Makefile in parallel comment the following line and
>> # execute:
>> #    make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))
>>
>> Regards
>>
>
> Been a silent reader for years here now, but I just must make a
> suggestion:

Hi Mike,
welcome as a writer :)


> # Parallel execution of this top-level Makefile is disabled.
> #
> # Single execution is forced here for two reasons:
> #
> # Maintaining reproducible builds:
> # Packages with an unspecified, optional, dependency could
> # otherwise change the build order among multiple builds.
> #
> # Prevent file modification collisions:
> # This can happen if two packages manipulate the same file.
> #
> # After taking into account the above considerations;
> # If you still want to execute this top-level Makefile in
> # parallel, comment out the following line and then execute:
> #    make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))
>
> Just clarify which line is "the following line" in the context
> of this comment.
>
> Add styling to first line of each paragraph to match the context.

Thanks for the suggestion, maybe a mix of the two:

# Parallel execution of this Makefile is disabled because it changes
# the packages building order, that can be a problem for two reasons:
# - If a package has an unspecified optional dependency and that
#   dependency is present when the package is built, it is used,
#   otherwise it isn't (but compilation happily proceeds) so the end
#   result will differ if the order is swapped due to parallel
#   building.
# - Also changing the building order can be a problem if two packages
#   manipulate the same file in the target directory.
#
# Taking into account the above considerations, if you still want to execute
# this top-level Makefile in parallel comment the ".NOTPARALLEL" line and
# build using the following command:
#    make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))

Regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v10 1/8] package: add base dependency to every package

[Arnout Vandecappelle]

On 18/12/13 11:36, Fabio Porcedda wrote:
> Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to
> every package.
> This way we can build correctly every package right after the clean
> stage.
> As example with this commit we can build successfully the glibc right
> after the clean stage:
> 	make clean glibc
>
> This is also a step forward supporting top-level parallel make.
>
> Signed-off-by: Fabio Porcedda<fabio.porcedda@gmail.com>
> Acked-by: Arnout Vandecappelle (Essensium/Mind)<arnout@mind.be>

  Peter,

  This one is fixing a problem reported by a user and has already gone 
through 10 iterations. Can it be committed?

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v10 1/8] package: add base dependency to every package

[Fabio Porcedda]

On Thu, Jan 9, 2014 at 6:30 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/12/13 11:36, Fabio Porcedda wrote:
>>
>> Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to
>> every package.
>> This way we can build correctly every package right after the clean
>> stage.
>> As example with this commit we can build successfully the glibc right
>> after the clean stage:
>>         make clean glibc
>>
>> This is also a step forward supporting top-level parallel make.
>>
>> Signed-off-by: Fabio Porcedda<fabio.porcedda@gmail.com>
>> Acked-by: Arnout Vandecappelle (Essensium/Mind)<arnout@mind.be>
>
>
>  Peter,
>
>  This one is fixing a problem reported by a user and has already gone
> through 10 iterations. Can it be committed?

I've sent a rebased version, so now are 11 iterations...

Best regards
-- 
Fabio Porcedda