[Buildroot] [PATCH v2] uclibc: enable parallel building of libraries

[Buildroot] [PATCH v2] uclibc: enable parallel building of libraries

[Alexey Brodkin]

The use of MAKE1 for uClibc dates back 10 years:

commit 8e5fb3fb4ab09b4dc04fe7cb3f7becce6514117b
Author: Eric Andersen <andersen@codepoet.org>
Date:   Sat Dec 11 13:01:10 2004 +0000

    Add initial BR2_JLEVEL support, with some exceptions for apps that
    have broken 'make -j' support

Since that time there were lots of improvements in uClibc that seem to allow
parallel building of libs finally.

Unfortunately uClibc tests have dependences on previously built files,
that's why tests left with MAKE1.

Signed-off-by: Alexey Brodkin <abrodkin@synopsys.com>

Cc: Anton Kolesov <akolesov@synopsys.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
In v2 uClibc tests are left with MAKE1 because there're strict
dependences.
---
 package/uclibc/uclibc.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
index b42fe55..28a8a08 100644
--- a/package/uclibc/uclibc.mk
+++ b/package/uclibc/uclibc.mk
@@ -494,13 +494,13 @@ endef
 endif
 
 define UCLIBC_BUILD_CMDS
-	$(MAKE1) -C $(@D) \
+	$(MAKE) -C $(@D) \
 		$(UCLIBC_MAKE_FLAGS) \
 		PREFIX= \
 		DEVEL_PREFIX=/ \
 		RUNTIME_PREFIX=/ \
 		all
-	$(MAKE1) -C $(@D)/utils \
+	$(MAKE) -C $(@D)/utils \
 		PREFIX=$(HOST_DIR) \
 		HOSTCC="$(HOSTCC)" hostutils
 	$(UCLIBC_BUILD_TEST_SUITE)
-- 
1.9.3

[Buildroot] [PATCH v2] uclibc: enable parallel building of libraries

[Thomas De Schampheleire]

Hi Alexey,

On Fri, Jun 27, 2014 at 6:26 PM, Alexey Brodkin
<Alexey.Brodkin@synopsys.com> wrote:
> The use of MAKE1 for uClibc dates back 10 years:
>
> commit 8e5fb3fb4ab09b4dc04fe7cb3f7becce6514117b
> Author: Eric Andersen <andersen@codepoet.org>
> Date:   Sat Dec 11 13:01:10 2004 +0000
>
>     Add initial BR2_JLEVEL support, with some exceptions for apps that
>     have broken 'make -j' support
>
> Since that time there were lots of improvements in uClibc that seem to allow
> parallel building of libs finally.
>
> Unfortunately uClibc tests have dependences on previously built files,
> that's why tests left with MAKE1.
>
> Signed-off-by: Alexey Brodkin <abrodkin@synopsys.com>
>
> Cc: Anton Kolesov <akolesov@synopsys.com>
> Cc: Peter Korsgaard <peter@korsgaard.com>
> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> In v2 uClibc tests are left with MAKE1 because there're strict
> dependences.
> ---
>  package/uclibc/uclibc.mk | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
> index b42fe55..28a8a08 100644
> --- a/package/uclibc/uclibc.mk
> +++ b/package/uclibc/uclibc.mk
> @@ -494,13 +494,13 @@ endef
>  endif
>
>  define UCLIBC_BUILD_CMDS
> -       $(MAKE1) -C $(@D) \
> +       $(MAKE) -C $(@D) \
>                 $(UCLIBC_MAKE_FLAGS) \
>                 PREFIX= \
>                 DEVEL_PREFIX=/ \
>                 RUNTIME_PREFIX=/ \
>                 all
> -       $(MAKE1) -C $(@D)/utils \
> +       $(MAKE) -C $(@D)/utils \
>                 PREFIX=$(HOST_DIR) \
>                 HOSTCC="$(HOSTCC)" hostutils
>         $(UCLIBC_BUILD_TEST_SUITE)

There are other occurrences of MAKE1 in uclibc.mak: for the oldconfig
step (in SETUP_DOT_CONFIG), configure commands, installation commands,
and menuconfig step.
I think these should be replaced by MAKE too (and tested, of course).

Best regards,
Thomas

[Buildroot] [PATCH v2] uclibc: enable parallel building of libraries

[Alexey Brodkin]

Hi Thomas,

> There are other occurrences of MAKE1 in uclibc.mak: for the oldconfig
> step (in SETUP_DOT_CONFIG), configure commands, installation commands,
> and menuconfig step.
> I think these should be replaced by MAKE too (and tested, of course).

I did notice there're many more MAKE1 in use.

But knowing unfortunate fate of uClibc in terms of parallel building I wanted to do minimal changes that still make significant difference in Buildroot usage (i.e. make building faster).
The most time consuming process is uClibc building itself and I made it parallel - that shortens uClibc significantly.

As for other invocations of MAKE1 IMHO there's not much sense in use of parallel make for configuration like "oldconfig" and friends (not much things here are compiled) or "install" (where we only have file copy operations and that's why I don't expect any gain in speed if we run them in parallel).

So I'd prefer to proceed with incremental changes. Once parallel build is proven for all arches by autobuilder (which may disclose issues on arches I haven't try the change yet - I did tests for ARC and x86) we may consider more changes (note I already found that built-in tests won't build in parallel).

What do you think?

-Alexey

[Buildroot] [PATCH v2] uclibc: enable parallel building of libraries

[Thomas De Schampheleire]

Hi Alexey,

On Sun, Jun 29, 2014 at 11:59 AM, Alexey Brodkin
<Alexey.Brodkin@synopsys.com> wrote:
> Hi Thomas,
>
>> There are other occurrences of MAKE1 in uclibc.mak: for the oldconfig
>> step (in SETUP_DOT_CONFIG), configure commands, installation commands,
>> and menuconfig step.
>> I think these should be replaced by MAKE too (and tested, of course).
>
> I did notice there're many more MAKE1 in use.
>
> But knowing unfortunate fate of uClibc in terms of parallel building I wanted to do minimal changes that still make significant difference in Buildroot usage (i.e. make building faster).
> The most time consuming process is uClibc building itself and I made it parallel - that shortens uClibc significantly.
>
> As for other invocations of MAKE1 IMHO there's not much sense in use of parallel make for configuration like "oldconfig" and friends (not much things here are compiled) or "install" (where we only have file copy operations and that's why I don't expect any gain in speed if we run them in parallel).
>
> So I'd prefer to proceed with incremental changes. Once parallel build is proven for all arches by autobuilder (which may disclose issues on arches I haven't try the change yet - I did tests for ARC and x86) we may consider more changes (note I already found that built-in tests won't build in parallel).
>
> What do you think?

Sure, we can wait a short while before enabling $(MAKE) for the other
targets. But I think we should certainly do it, because the buildroot
default is $(MAKE) not $(MAKE1). So using $(MAKE1) should only be done
if there are good reason.

One particular reason that I want to go for $(MAKE) is for the
menuconfig target, as I'm working on a kconfig-package infrastructure
that will line up the kconfig handling in busybox, uclibc, linux, ...
Since uclibc uses MAKE1 here, it deviates from the other packages
using kconfig, posing a problem for commonalization in
kconfig-package.

Best regards,
Thomas

[Buildroot] [PATCH v2] uclibc: enable parallel building of libraries

[Alexey Brodkin]

Hi Thomas,

On Sun, 2014-06-29 at 14:43 +0200, Thomas De Schampheleire wrote:
> 
> Sure, we can wait a short while before enabling $(MAKE) for the other
> targets. But I think we should certainly do it, because the buildroot
> default is $(MAKE) not $(MAKE1). So using $(MAKE1) should only be done
> if there are good reason.
> 
> One particular reason that I want to go for $(MAKE) is for the
> menuconfig target, as I'm working on a kconfig-package infrastructure
> that will line up the kconfig handling in busybox, uclibc, linux, ...
> Since uclibc uses MAKE1 here, it deviates from the other packages
> using kconfig, posing a problem for commonalization in
> kconfig-package.

Makes perfect sense!
If this topic is important for you may I ask you to remind me about it
in a while if I forget to do/test/post mentioned changes in coming
weeks?

It's not a rocket science but mostly a matter of testing how changes
impact building. Even with this patch (which is already applied - kudos
to Thomas P) I faced troubles trying to build uClibc tests: for x86 due
to pretty old uClibc I had to find and apply this patch:
====
test: tls: fix build with newer binutils
http://git.uclibc.org/uClibc/commit/test?id=931e8391565323ed2e589c83b83a7345813a5514
====

-Alexey

[Buildroot] [PATCH v2] uclibc: enable parallel building of libraries

[Thomas Petazzoni]

Dear Alexey Brodkin,

On Fri, 27 Jun 2014 20:26:00 +0400, Alexey Brodkin wrote:
> The use of MAKE1 for uClibc dates back 10 years:
> 
> commit 8e5fb3fb4ab09b4dc04fe7cb3f7becce6514117b
> Author: Eric Andersen <andersen@codepoet.org>
> Date:   Sat Dec 11 13:01:10 2004 +0000
> 
>     Add initial BR2_JLEVEL support, with some exceptions for apps that
>     have broken 'make -j' support
> 
> Since that time there were lots of improvements in uClibc that seem to allow
> parallel building of libs finally.
> 
> Unfortunately uClibc tests have dependences on previously built files,
> that's why tests left with MAKE1.
> 
> Signed-off-by: Alexey Brodkin <abrodkin@synopsys.com>

Applied, thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2] uclibc: enable parallel building of libraries

[Thomas Petazzoni]

Dear Alexey Brodkin,

On Fri, 27 Jun 2014 20:26:00 +0400, Alexey Brodkin wrote:
> The use of MAKE1 for uClibc dates back 10 years:
> 
> commit 8e5fb3fb4ab09b4dc04fe7cb3f7becce6514117b
> Author: Eric Andersen <andersen@codepoet.org>
> Date:   Sat Dec 11 13:01:10 2004 +0000
> 
>     Add initial BR2_JLEVEL support, with some exceptions for apps that
>     have broken 'make -j' support
> 
> Since that time there were lots of improvements in uClibc that seem to allow
> parallel building of libs finally.
> 
> Unfortunately uClibc tests have dependences on previously built files,
> that's why tests left with MAKE1.
> 
> Signed-off-by: Alexey Brodkin <abrodkin@synopsys.com>

This patch apparently breaks the build on AVR32, which uses an old
(0.9.31) version of uClibc. So maybe we should use $(MAKE) instead of
$(MAKE1) only for uClibc >= 0.9.33 ?

See
http://jenkins.free-electrons.com/job/buildroot/config=atstk100x_defconfig/298/consoleText
for the build log. Both AVR32 defconfigs failed to build tonight, and
the only patch I can think of that was applied yesterday in relation to
uClibc is the parallel build one.

Thanks,

Thomas Petazzoni
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2] uclibc: enable parallel building of libraries

[Alexey Brodkin]

Hi Thomas,

On Mon, 2014-06-30 at 09:24 +0200, Thomas Petazzoni wrote:
> This patch apparently breaks the build on AVR32, which uses an old
> (0.9.31) version of uClibc. So maybe we should use $(MAKE) instead of
> $(MAKE1) only for uClibc >= 0.9.33 ?
> 
> See
> http://jenkins.free-electrons.com/job/buildroot/config=atstk100x_defconfig/298/consoleText
> for the build log. Both AVR32 defconfigs failed to build tonight, and
> the only patch I can think of that was applied yesterday in relation to
> uClibc is the parallel build one.

Thanks for this.
Let me look at this issue on AVR32.

Frankly I'd prefer to escape conditions withing "uclibc.mk" - this
condition will remain for many years until somebody tries to do another
round of clean-up.

I'll try to find an upstream patch that fixes disclosed issue and put it
in "package/uclibc/0.9.31.1".

What do you think of this approach?

-Alexey

[Buildroot] [PATCH v2] uclibc: enable parallel building of libraries

[Thomas Petazzoni]

Dear Alexey Brodkin,

On Mon, 30 Jun 2014 07:29:50 +0000, Alexey Brodkin wrote:

> Thanks for this.
> Let me look at this issue on AVR32.
> 
> Frankly I'd prefer to escape conditions withing "uclibc.mk" - this
> condition will remain for many years until somebody tries to do another
> round of clean-up.
> 
> I'll try to find an upstream patch that fixes disclosed issue and put it
> in "package/uclibc/0.9.31.1".
> 
> What do you think of this approach?

I personally think it's OK to use $(MAKE) for uClibc >= 0.9.33 and
$(MAKE1) for older versions. We only have 0.9.31 to support AVR32, and
since it's an obsolete architecture, I believe we might propose to
deprecate it during the 2014.08 cycle, so I don't think it's worth
spending time on issues relevant only to 0.9.31. If that's done, then
we'll be able to get rid of 0.9.31 and 0.9.32. I don't think it makes
sense to support such old uClibc releases.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com