[Buildroot] [PATCH v4 0/2] Top-level parallel makefile improvements

[Buildroot] [PATCH v4 0/2] Top-level parallel makefile improvements

[Fabio Porcedda]

Hi all,
this patch set improve top-level parallel makefile support.

v4:
 - add a patch for luarocks packages
v3:
 - add Acked-by (Arnout)
v2:
 - use MAKEFLAGS to check if the "-j" option is used
 - don't empty the PARALLEL_JOBS variable (Thomas P.)

Fabio Porcedda (2):
  packages: fix and improve support for top-level parallel makefile
  pkg-luarocks: fix top-level parallel makefile support

 Makefile                     | 4 ++--
 package/Makefile.in          | 3 ++-
 package/luarocks/luarocks.mk | 6 ++++--
 package/pkg-luarocks.mk      | 4 ++--
 4 files changed, 10 insertions(+), 7 deletions(-)

-- 
2.4.3

[Buildroot] [PATCH v4 1/2] packages: fix and improve support for top-level parallel makefile

[Fabio Porcedda]

To fix packages that fail to build when PARALLEL_JOBS is empty instead
of using an empty PARALLEL_JOBS just avoid to using it in
the MAKE variable.

Check the MAKEFLAGS variable to know automatically if the -j option is
being used, but use the "=" operator instead of the ":=" operator
because it can be checked only in a "recursively expanded variable".
Use "override" because otherwise it's impossible to change the
automatic variable "MAKE".

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 Makefile            | 4 ++--
 package/Makefile.in | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 8eb21d4..b1bcf78 100644
--- a/Makefile
+++ b/Makefile
@@ -58,8 +58,8 @@ export HOSTARCH := $(shell uname -m | \
 #
 # Taking into account the above considerations, if you still want to execute
 # this top-level Makefile in parallel comment the ".NOTPARALLEL" line and
-# build using the following command:
-#	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))
+# use the -j<jobs> option when building, e.g:
+#      make -j$((`getconf _NPROCESSORS_ONLN`+1))
 .NOTPARALLEL:
 
 # absolute path
diff --git a/package/Makefile.in b/package/Makefile.in
index c02d31f..2ed7cf7 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -18,7 +18,8 @@ PARALLEL_JOBS := $(BR2_JLEVEL)
 endif
 
 MAKE1 := $(HOSTMAKE) -j1
-MAKE := $(HOSTMAKE) $(if $(PARALLEL_JOBS),-j$(PARALLEL_JOBS))
+override MAKE = $(HOSTMAKE) \
+	$(if $(findstring j,$(filter-out --%,$(MAKEFLAGS))),,-j$(PARALLEL_JOBS))
 
 ifeq ($(BR2_TOOLCHAIN_BUILDROOT),y)
 TARGET_VENDOR = $(call qstrip,$(BR2_TOOLCHAIN_BUILDROOT_VENDOR))
-- 
2.4.3

[Buildroot] [PATCH v4 2/2] pkg-luarocks: fix top-level parallel makefile support

[Fabio Porcedda]

In the *-install-target phase the manifest file is being updated, if multiply packages try to update it they fail.

To avoid multiple access to the manifest file use flock to sync
multiple luarocks packages.

e.g. installing three luarocks packages:
make lua-cjson-build lua-coat-build lua-coatpersistent-build
make lua-cjson lua-coat lua-coatpersistent -j

Fix error:
Updating manifest for /home/tetsuya/buildroot/br2/output/target/usr/lib/luarocks/rocks
No existing manifest. Attempting to rebuild...

Error: rock_manifest file not found for lua-coat 0.9.1-1 - not a LuaRocks 2 tree?

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/luarocks/luarocks.mk | 6 ++++--
 package/pkg-luarocks.mk      | 4 ++--
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/package/luarocks/luarocks.mk b/package/luarocks/luarocks.mk
index 2b6c975..413e23d 100644
--- a/package/luarocks/luarocks.mk
+++ b/package/luarocks/luarocks.mk
@@ -53,8 +53,10 @@ endef
 
 $(eval $(host-generic-package))
 
-LUAROCKS_RUN = LUA_PATH="$(HOST_DIR)/usr/share/lua/$(LUAINTERPRETER_ABIVER)/?.lua" \
-	$(LUA_RUN) $(HOST_DIR)/usr/bin/luarocks
+LUAROCKS_RUN_ENV = LUA_PATH="$(HOST_DIR)/usr/share/lua/$(LUAINTERPRETER_ABIVER)/?.lua"
+LUAROCKS_RUN_CMD = $(LUA_RUN) $(HOST_DIR)/usr/bin/luarocks
+
+LUAROCKS_RUN = $(LUAROCKS_RUN_ENV) $(LUAROCKS_RUN_CMD)
 
 define LUAROCKS_FINALIZE_TARGET
 	rm -rf $(TARGET_DIR)/usr/lib/luarocks
diff --git a/package/pkg-luarocks.mk b/package/pkg-luarocks.mk
index 83f338e..0a7ba47 100644
--- a/package/pkg-luarocks.mk
+++ b/package/pkg-luarocks.mk
@@ -58,8 +58,8 @@ endif
 #
 ifndef $(2)_INSTALL_TARGET_CMDS
 define $(2)_INSTALL_TARGET_CMDS
-	cd $$($(2)_SRCDIR) && \
-		$$(LUAROCKS_RUN) make --keep $$($(2)_ROCKSPEC) $$($(2)_BUILD_OPTS)
+	cd $$($(2)_SRCDIR) && $$(LUAROCKS_RUN_ENV) flock $$(TARGET_DIR) \
+		$$(LUAROCKS_RUN_CMD) make --keep $$($(2)_ROCKSPEC) $$($(2)_BUILD_OPTS)
 endef
 endif
 
-- 
2.4.3

[Buildroot] [PATCH v4 1/2] packages: fix and improve support for top-level parallel makefile

[Thomas Petazzoni]

Dear Fabio Porcedda,

On Fri, 26 Jun 2015 05:58:53 +0200, Fabio Porcedda wrote:
> To fix packages that fail to build when PARALLEL_JOBS is empty instead

Which packages specifically?

> of using an empty PARALLEL_JOBS just avoid to using it in

s/avoid to/avoid/

>  MAKE1 := $(HOSTMAKE) -j1
> -MAKE := $(HOSTMAKE) $(if $(PARALLEL_JOBS),-j$(PARALLEL_JOBS))
> +override MAKE = $(HOSTMAKE) \
> +	$(if $(findstring j,$(filter-out --%,$(MAKEFLAGS))),,-j$(PARALLEL_JOBS))

Why should we force to not use make -j for each individual package when
top-level parallel build is used ? Maybe for some scenarios it is better
to have 2 top-level jobs, and then 2 jobs per package ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v4 1/2] packages: fix and improve support for top-level parallel makefile

[Fabio Porcedda]

On Fri, Jun 26, 2015 at 8:09 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Fabio Porcedda,
>
> On Fri, 26 Jun 2015 05:58:53 +0200, Fabio Porcedda wrote:
>> To fix packages that fail to build when PARALLEL_JOBS is empty instead
>
> Which packages specifically?

At least the boost and jack2 packages fail.

>> of using an empty PARALLEL_JOBS just avoid to using it in
>
> s/avoid to/avoid/
>
>>  MAKE1 := $(HOSTMAKE) -j1
>> -MAKE := $(HOSTMAKE) $(if $(PARALLEL_JOBS),-j$(PARALLEL_JOBS))
>> +override MAKE = $(HOSTMAKE) \
>> +     $(if $(findstring j,$(filter-out --%,$(MAKEFLAGS))),,-j$(PARALLEL_JOBS))
>
> Why should we force to not use make -j for each individual package when
> top-level parallel build is used ? Maybe for some scenarios it is better
> to have 2 top-level jobs, and then 2 jobs per package ?

The good thing about GNU Make jobs is that the number of jobs
available are shared between each instance of make so if you want to
increase the number of jobs you just need to increase the <jobs> value
in  the top make -j<jobs> command.

e.g. 2 in top-level and 2 in each package, the actual cpu utilization
start from 1 to 4.
Using 4 in top-level and let make share the jobs for each packages the
cpu utilization start from 1 to 4 but you have a higher chance to use
4 processor because in this case you can have:

- 4 packages that use make1
or
- 1 package that use make but it actually use 4 processor himself
or
- 3 extract operation and 1 package that use make1


Another example, you have 8 cores, you use top-make -j8 and sub-make -j8.
This time the number of processes goes up to 64 that it's not good,
too much process take more time to be completed.
In my tests on a cpu with 8 cores -j16 is slower than -j9.

So I think it's better to let gnu make handle the jobs distribution.

BR
-- 
Fabio Porcedda

[Buildroot] [PATCH v4 1/2] packages: fix and improve support for top-level parallel makefile

[Thomas Petazzoni]

Hello Fabio,

On Fri, 26 Jun 2015 21:39:43 +0200, Fabio Porcedda wrote:

> > On Fri, 26 Jun 2015 05:58:53 +0200, Fabio Porcedda wrote:
> >> To fix packages that fail to build when PARALLEL_JOBS is empty instead
> >
> > Which packages specifically?
> 
> At least the boost and jack2 packages fail.

Ok, this should probably been indicated in the commit log, in order to
make it more specific than "To fix 'packages'".


> The good thing about GNU Make jobs is that the number of jobs
> available are shared between each instance of make so if you want to
> increase the number of jobs you just need to increase the <jobs> value
> in  the top make -j<jobs> command.
> 
> e.g. 2 in top-level and 2 in each package, the actual cpu utilization
> start from 1 to 4.
> Using 4 in top-level and let make share the jobs for each packages the
> cpu utilization start from 1 to 4 but you have a higher chance to use
> 4 processor because in this case you can have:
> 
> - 4 packages that use make1
> or
> - 1 package that use make but it actually use 4 processor himself
> or
> - 3 extract operation and 1 package that use make1
> 
> 
> Another example, you have 8 cores, you use top-make -j8 and sub-make -j8.
> This time the number of processes goes up to 64 that it's not good,
> too much process take more time to be completed.
> In my tests on a cpu with 8 cores -j16 is slower than -j9.
> 
> So I think it's better to let gnu make handle the jobs distribution.

Ah, I didn't know this. This is a very good explanation! Why isn't it
part of the commit log? It very clearly explains why we want to not
pass any -j option to the sub-makes used to build the packages.

However, I don't see how it properly fixes the packages that are
directly using PARALLEL_JOBS. Yes, now, PARALLEL_JOBS is no longer
empty. But it's going to be number of CPU + 1, which is not good when
we're doing top-level parallel job, as you explained very well. So why
would all make-based packages be built with "make", and packages using
special build systems (midori, qt, qt5, jack2, cmake, boost) would
still use a PARALLEL_JOBS number higher than 1. What is your plan to
solve this problem?

Thanks a lot!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v4 1/2] packages: fix and improve support for top-level parallel makefile

[Fabio Porcedda]

On Sat, Jun 27, 2015 at 4:10 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello Fabio,
>
> On Fri, 26 Jun 2015 21:39:43 +0200, Fabio Porcedda wrote:
>
>> > On Fri, 26 Jun 2015 05:58:53 +0200, Fabio Porcedda wrote:
>> >> To fix packages that fail to build when PARALLEL_JOBS is empty instead
>> >
>> > Which packages specifically?
>>
>> At least the boost and jack2 packages fail.
>
> Ok, this should probably been indicated in the commit log, in order to
> make it more specific than "To fix 'packages'".

Ok

>> The good thing about GNU Make jobs is that the number of jobs
>> available are shared between each instance of make so if you want to
>> increase the number of jobs you just need to increase the <jobs> value
>> in  the top make -j<jobs> command.
>>
>> e.g. 2 in top-level and 2 in each package, the actual cpu utilization
>> start from 1 to 4.
>> Using 4 in top-level and let make share the jobs for each packages the
>> cpu utilization start from 1 to 4 but you have a higher chance to use
>> 4 processor because in this case you can have:
>>
>> - 4 packages that use make1
>> or
>> - 1 package that use make but it actually use 4 processor himself
>> or
>> - 3 extract operation and 1 package that use make1
>>
>>
>> Another example, you have 8 cores, you use top-make -j8 and sub-make -j8.
>> This time the number of processes goes up to 64 that it's not good,
>> too much process take more time to be completed.
>> In my tests on a cpu with 8 cores -j16 is slower than -j9.
>>
>> So I think it's better to let gnu make handle the jobs distribution.
>
> Ah, I didn't know this. This is a very good explanation! Why isn't it
> part of the commit log? It very clearly explains why we want to not
> pass any -j option to the sub-makes used to build the packages.

Ok i will add it.

> However, I don't see how it properly fixes the packages that are
> directly using PARALLEL_JOBS. Yes, now, PARALLEL_JOBS is no longer
> empty. But it's going to be number of CPU + 1, which is not good when
> we're doing top-level parallel job, as you explained very well. So why
> would all make-based packages be built with "make", and packages using
> special build systems (midori, qt, qt5, jack2, cmake, boost) would
> still use a PARALLEL_JOBS number higher than 1. What is your plan to
> solve this problem?

The ideal solution is that those special build systems are capable to
use the jobs server of gnu make, but that is not an easy feat.

IMHO the pragmatical solution is to avoid the worst case, even if
using CPU+1 is not the optimal solution, it's still better than using
an inferior number and risking to leave some cpu idling.
In other words the excessive cpu utilization is less harmful than
under utilizing cpus.

BR
-- 
Fabio Porcedda