[Buildroot] [PATCH v3 0/3] Fix for top-level parallel make part 1

[Buildroot] [PATCH v3 0/3] Fix for top-level parallel make part 1

[Fabio Porcedda]

Hi all,
this is the first patch set for fixing top-level parallel make in buildroot,
the common problem scattered in buildroot's top-level makefile is that in the
rules it relies on the order of evaluation of the prerequisites,
to be able to use top-level parallel make instead of reling on the left to
right ordering of evaluation of the prerequisites we must add an explicit
rule to describe the dependencies.

With this patch set the package building seems to work fine with top-level
parallel make, example:

	make clean toolchain
	make BR2_JLEVEL= -j<n> <package-name list> 

Before to try that you have to remove the ".NOTPARALLEL" line in the top-level Makefile.

The part that remain to fix for top-level parallel make is the toolchain
building and targets after building all the packages.

Best regards
Fabio Porcedda

v3:
 - add back the patch "package: add toolchain dependency to
    inner-generic-package" because now is working fine.
 - add Acked-by: Arnout Vandecappelle to the first patch.
 - reworked the second patch following Arnout suggestions.
v2:
 - remove patch "package: add toolchain dependency to inner-generic-package"
   because was not working fine against recent toolchain changes.

Fabio Porcedda (3):
  package/Makefile.in: add a way to don't force jobs in sub-make
  pkg-generic: fix rules for top-level parallel make
  package: add toolchain dependency to inner-generic-package

 Makefile                 |  2 +-
 package/Makefile.in      |  2 +-
 package/pkg-autotools.mk |  3 ++-
 package/pkg-generic.mk   | 51 ++++++++++++++++++++++++------------------------
 package/uclibc/uclibc.mk |  2 +-
 5 files changed, 31 insertions(+), 29 deletions(-)

-- 
1.8.4

[Buildroot] [PATCH v3 1/3] package/Makefile.in: add a way to don't force jobs in sub-make

[Fabio Porcedda]

When the "BR2_JLEVEL" variable is empty use "make"
without the "-j" option.
To be able to use top-level parallel make we must don't force
the number of jobs in sub-make.

Example:
	make BR2_JLEVEL= -j8

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 package/Makefile.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/Makefile.in b/package/Makefile.in
index 91ae1b9..6702df4 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -18,7 +18,7 @@ PARALLEL_JOBS:=$(BR2_JLEVEL)
 endif
 
 MAKE1:=$(HOSTMAKE) -j1
-MAKE:=$(HOSTMAKE) -j$(PARALLEL_JOBS)
+MAKE:=$(HOSTMAKE) $(if $(PARALLEL_JOBS),-j$(PARALLEL_JOBS))
 
 # Compute GNU_TARGET_NAME
 GNU_TARGET_NAME=$(ARCH)-buildroot-$(TARGET_OS)-$(LIBC)$(ABI)
-- 
1.8.4

[Buildroot] [PATCH v3 2/3] pkg-generic: fix rules for top-level parallel make

[Fabio Porcedda]

To be able to use top-level parallel make we must don't depend in a rule
on the order of evaluation of the prerequisites, so instead of reling on
the left to right ordering of evaluation of the prerequisites add
an explicit rule to describe the dependencies.

So add explicit dependencies for the following stamp files:
   %/.stamp_extracted
   %/.stamp_patched
   %/.stamp_configured
   %/.stamp_built
   %/.stamp_host_installed
   %/.stamp_staging_installed
   %/.stamp_images_installed
   %/.stamp_target_installed

Because the %-build target is not anymore part of the dependcy chain,
add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead.
This new variable is used only by the uclibc package for building
the toolchain.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-generic.mk   | 45 +++++++++++++++++++++------------------------
 package/uclibc/uclibc.mk |  2 +-
 2 files changed, 22 insertions(+), 25 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 4dd395b..5ea41ff 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -53,7 +53,7 @@ ifeq ($(DL_MODE),DOWNLOAD)
 endif
 
 # Unpack the archive
-$(BUILD_DIR)/%/.stamp_extracted:
+$(BUILD_DIR)/%/.stamp_extracted: $(BUILD_DIR)/%/.stamp_downloaded
 	@$(call MESSAGE,"Extracting")
 	$(Q)mkdir -p $(@D)
 	$($(PKG)_EXTRACT_CMDS)
@@ -88,7 +88,7 @@ endif
 # prefix of the patches
 $(BUILD_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
 $(BUILD_DIR)/%/.stamp_patched: PATCH_BASE_DIRS = $($(PKG)_DIR_PREFIX)/$(RAWNAME) $(call qstrip,$(BR2_GLOBAL_PATCH_DIR))/$(RAWNAME)
-$(BUILD_DIR)/%/.stamp_patched:
+$(BUILD_DIR)/%/.stamp_patched: $(BUILD_DIR)/%/.stamp_extracted
 	@$(call MESSAGE,"Patching $($(PKG)_DIR_PREFIX)/$(RAWNAME)")
 	$(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(foreach p,$($(PKG)_PATCH),support/scripts/apply-patches.sh $(@D) $(DL_DIR) $(notdir $(p))$(sep))
@@ -115,21 +115,21 @@ $(BUILD_DIR)/%/.stamp_configured:
 	$(Q)touch $@
 
 # Build
-$(BUILD_DIR)/%/.stamp_built::
+$(BUILD_DIR)/%/.stamp_built:	$(BUILD_DIR)/%/.stamp_configured
 	@$(call MESSAGE,"Building")
 	$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Install to host dir
-$(BUILD_DIR)/%/.stamp_host_installed:
+$(BUILD_DIR)/%/.stamp_host_installed:		$(BUILD_DIR)/%/.stamp_built
 	@$(call MESSAGE,"Installing to host directory")
 	$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Install to staging dir
-$(BUILD_DIR)/%/.stamp_staging_installed:
+$(BUILD_DIR)/%/.stamp_staging_installed:	$(BUILD_DIR)/%/.stamp_built
 	@$(call MESSAGE,"Installing to staging directory")
 	$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
@@ -143,14 +143,14 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 	$(Q)touch $@
 
 # Install to images dir
-$(BUILD_DIR)/%/.stamp_images_installed:
+$(BUILD_DIR)/%/.stamp_images_installed:		$(BUILD_DIR)/%/.stamp_built
 	@$(call MESSAGE,"Installing to images directory")
 	$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Install to target dir
-$(BUILD_DIR)/%/.stamp_target_installed:
+$(BUILD_DIR)/%/.stamp_target_installed:		$(BUILD_DIR)/%/.stamp_built
 	@$(call MESSAGE,"Installing to target")
 	$(if $(BR2_INIT_SYSTEMD),\
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
@@ -308,6 +308,7 @@ $(2)_REDISTRIBUTE		?= YES
 
 
 $(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+$(2)_BUILD_DEPENDENCIES		?=
 
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
@@ -359,30 +360,29 @@ $(1)-install:		$(1)-install-staging $(1)-install-target $(1)-install-images
 endif
 
 ifeq ($$($(2)_INSTALL_TARGET),YES)
-$(1)-install-target:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_TARGET)
+$(1)-install-target:	$$($(2)_TARGET_INSTALL_TARGET)
 else
 $(1)-install-target:
 endif
 
 ifeq ($$($(2)_INSTALL_STAGING),YES)
-$(1)-install-staging:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_STAGING)
+$(1)-install-staging:	$$($(2)_TARGET_INSTALL_STAGING)
 else
 $(1)-install-staging:
 endif
 
 ifeq ($$($(2)_INSTALL_IMAGES),YES)
-$(1)-install-images:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_IMAGES)
+$(1)-install-images:	$$($(2)_TARGET_INSTALL_IMAGES)
 else
 $(1)-install-images:
 endif
 
-$(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
+$(1)-install-host:      $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-build:		$(1)-configure \
-			$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_BUILD):	| $$($(2)_BUILD_DEPENDENCIES)
+$(1)-build:		$$($(2)_TARGET_BUILD)
+
+$(1)-configure:		$$($(2)_TARGET_CONFIGURE)
 
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 # In the normal case (no package override), the sequence of steps is
@@ -391,13 +391,11 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 #  extract
 #  patch
 #  configure
-$(1)-configure:		$(1)-patch $(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	| $$($(2)_DEPENDENCIES) $$($(2)_TARGET_PATCH)
 
-$(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
+$(1)-patch:		$$($(2)_TARGET_PATCH)
 
-$(1)-extract:		$(1)-source \
-			$$($(2)_TARGET_EXTRACT)
+$(1)-extract:		$$($(2)_TARGET_EXTRACT)
 
 $(1)-depends:		$$($(2)_DEPENDENCIES)
 
@@ -407,10 +405,9 @@ else
 #  source, by rsyncing
 #  depends
 #  configure
-$(1)-configure:		$(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	| $$($(2)_DEPENDENCIES) $$($(2)_TARGET_RSYNC)
 
-$(1)-depends:		$(1)-rsync $$($(2)_DEPENDENCIES)
+$(1)-depends:		$$($(2)_DEPENDENCIES)
 
 $(1)-patch:		$(1)-rsync
 $(1)-extract:		$(1)-rsync
diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
index 39877bb..b960f59 100644
--- a/package/uclibc/uclibc.mk
+++ b/package/uclibc/uclibc.mk
@@ -23,7 +23,7 @@ UCLIBC_INSTALL_STAGING = YES
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
 
 # Before uClibc is built, we must have the second stage cross-compiler
-uclibc-build: host-gcc-intermediate
+UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
 
 # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config
 # setting.
-- 
1.8.4

[Buildroot] [PATCH v3 3/3] package: add toolchain dependency to inner-generic-package

[Fabio Porcedda]

This commit makes the dependency from the target toolchain explicit.
This way we can buid from command line a package that use
innger-generic-package right after the configuration phase, example:

	make clean <package-name>

This is a step forward top-level parallel make.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 Makefile                 | 2 +-
 package/pkg-autotools.mk | 3 ++-
 package/pkg-generic.mk   | 6 +++++-
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 485d0a4..5ed3d60 100644
--- a/Makefile
+++ b/Makefile
@@ -397,7 +397,7 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
 toolchain: prepare dirs dependencies $(BASE_TARGETS)
 
-world: toolchain $(TARGETS_ALL)
+world: $(TARGETS_ALL)
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars \
diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
index b0eddde..aafa396 100644
--- a/package/pkg-autotools.mk
+++ b/package/pkg-autotools.mk
@@ -202,7 +202,8 @@ endef
 # This must be repeated from inner-generic-package, otherwise we get an empty
 # _DEPENDENCIES if _AUTORECONF is YES.  Also filter the result of _AUTORECONF
 # away from the non-host rule
-$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\
+$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \
+				host-toolchain $(1),\
     $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
 
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 5ea41ff..e511833 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -307,7 +307,11 @@ endif
 $(2)_REDISTRIBUTE		?= YES
 
 
-$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+$(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
+	$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+ifeq ($$($(2)_TYPE),target)
+$(2)_DEPENDENCIES += toolchain
+endif
 $(2)_BUILD_DEPENDENCIES		?=
 
 $(2)_INSTALL_STAGING		?= NO
-- 
1.8.4

[Buildroot] [PATCH v3 1/3] package/Makefile.in: add a way to don't force jobs in sub-make

[Thomas Petazzoni]

Dear Fabio Porcedda,

On Tue, 10 Sep 2013 09:46:02 +0200, Fabio Porcedda wrote:
> When the "BR2_JLEVEL" variable is empty use "make"
> without the "-j" option.
> To be able to use top-level parallel make we must don't force
> the number of jobs in sub-make.
> 
> Example:
> 	make BR2_JLEVEL= -j8
> 
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Acked-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com