[Buildroot] Buildroot and top-level parallel make

[Buildroot] Buildroot and top-level parallel make

[Alex Bradbury]

As I understand it, buildroot will allow parallelism while building
individual packages yet does not support "top-level parallel make".
i.e. it is not possible to build multiple packages in parallel. I
found a reference to this problem at http://lwn.net/Articles/416916/
and it was also listed as a WIP in the "cross build systems: present
and future" slides from 2010.

I was wondering what the outcome of experiments to allow top-level
parallelism were? What are the major barriers to allowing this? Is it
still something seen as feasible to add to buildroot, or is it looking
like it would add too much complexity?

Thanks,

Alex

[Buildroot] Buildroot and top-level parallel make

[Arnout Vandecappelle]

On 06/28/12 14:34, Alex Bradbury wrote:
> As I understand it, buildroot will allow parallelism while building
> individual packages yet does not support "top-level parallel make".
> i.e. it is not possible to build multiple packages in parallel. I
> found a reference to this problem at http://lwn.net/Articles/416916/
> and it was also listed as a WIP in the "cross build systems: present
> and future" slides from 2010.
>
> I was wondering what the outcome of experiments to allow top-level
> parallelism were? What are the major barriers to allowing this? Is it
> still something seen as feasible to add to buildroot, or is it looking
> like it would add too much complexity?

  AFAIK nobody is actually working on this.  Myself I have had a little
thought about it before, and the main problem seems that there are a few
targets in the top-level Makefile that depend on the left-to-right
evaluation order of dependencies.

  It would of course be difficult to verify that a top-level parallel make
works, but now we have the autobuild system it can be done...

  Regards,
  Arnout
-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Buildroot and top-level parallel make

[Alex Bradbury]

On 30 June 2012 13:48, Arnout Vandecappelle <arnout@mind.be> wrote:
> ?AFAIK nobody is actually working on this. ?Myself I have had a little
> thought about it before, and the main problem seems that there are a few
> targets in the top-level Makefile that depend on the left-to-right
> evaluation order of dependencies.
>
> ?It would of course be difficult to verify that a top-level parallel make
> works, but now we have the autobuild system it can be done...

redo has the rather nice --shuffle option to help shake out issues
with parallel builds:
https://github.com/apenwarr/redo/#dependency-problems-that-only-show-up-during-parallel-builds

Admittedly I only had a quick look, but it doesn't look like GNU make
has something similar which is a shame. OpenBSD make does as one of
its debug options though:
http://www.openbsd.org/cgi-bin/man.cgi?query=make

Alex

[Buildroot] Buildroot and top-level parallel make

[Alex Bradbury]

On 28 June 2012 13:34, Alex Bradbury <asb@asbradbury.org> wrote:
> I was wondering what the outcome of experiments to allow top-level
> parallelism were? What are the major barriers to allowing this? Is it
> still something seen as feasible to add to buildroot, or is it looking
> like it would add too much complexity?

Thomas, I was wondering if you'd been able to recall what issues your
initial experiments with top-level parallel make faced?

Thanks,

Alex

[Buildroot] Buildroot and top-level parallel make

[Alex Bradbury]

On 4 July 2012 13:20, Alex Bradbury <asb@asbradbury.org> wrote:
> On 28 June 2012 13:34, Alex Bradbury <asb@asbradbury.org> wrote:
>> I was wondering what the outcome of experiments to allow top-level
>> parallelism were? What are the major barriers to allowing this? Is it
>> still something seen as feasible to add to buildroot, or is it looking
>> like it would add too much complexity?
>
> Thomas, I was wondering if you'd been able to recall what issues your
> initial experiments with top-level parallel make faced?

I hate to be a pain...but ping?

Alex

[Buildroot] Buildroot and top-level parallel make

[Arnout Vandecappelle]

On 07/13/12 12:34, Alex Bradbury wrote:
> On 4 July 2012 13:20, Alex Bradbury<asb@asbradbury.org>  wrote:
>> >  On 28 June 2012 13:34, Alex Bradbury<asb@asbradbury.org>  wrote:
>>> >>  I was wondering what the outcome of experiments to allow top-level
>>> >>  parallelism were? What are the major barriers to allowing this? Is it
>>> >>  still something seen as feasible to add to buildroot, or is it looking
>>> >>  like it would add too much complexity?
>> >
>> >  Thomas, I was wondering if you'd been able to recall what issues your
>> >  initial experiments with top-level parallel make faced?
> I hate to be a pain...but ping?

  I guess Thomas doesn't remember :-)

  I believe the main problem is in the non-package targets, e.g.
target-generic-hostname.  They are just added to TARGETS without
further dependencies.  Same is true for the toolchain: it is just
prepended to TARGETS.

  Changing that requires careful thinking about how the TARGETS work,
and is not easy to verify because there are a lot of corner cases involved.

  Regards,
  Arnout
-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Buildroot and top-level parallel make

[Thomas Petazzoni]

Le Fri, 13 Jul 2012 11:34:58 +0100,
Alex Bradbury <asb@asbradbury.org> a ?crit :

> On 4 July 2012 13:20, Alex Bradbury <asb@asbradbury.org> wrote:
> > On 28 June 2012 13:34, Alex Bradbury <asb@asbradbury.org> wrote:
> >> I was wondering what the outcome of experiments to allow top-level
> >> parallelism were? What are the major barriers to allowing this? Is it
> >> still something seen as feasible to add to buildroot, or is it looking
> >> like it would add too much complexity?
> >
> > Thomas, I was wondering if you'd been able to recall what issues your
> > initial experiments with top-level parallel make faced?
> 
> I hate to be a pain...but ping?

Sorry, I've been way too busy last week. As Arnout said, I don't
remember the details, but here are some elements:

 * Adding top-level parallel build would introduce a lot of
   "randomness" in builds, making them harder to reproduce. If one
   package lacks a dependency on another package, but can optionally
   rely on it, then with top-level parallel build, sometimes the
   package will use this optional library, sometimes not. Of course,
   it's a bug in the package, that should have expressed this optional
   dependency in its .mk file, but still it's a problem. I think this
   non-reproducibility was the major issue we were having, because it
   would cause problems to users that may be hard to reproduce.

 * Adding top-level parallel build requires changing how the logging is
   done. At the moment, all the build details are logged on stdout.
   With top-level parallel build, we would need to redirect the build
   log of each package to a separate log file.

 * The current package infrastructure uses phony targets for
   sequencing, and we use phony targets in many other places, and phony
   targets don't play well with parallel build, they would have to be
   replaced with real targets (based on stamp files for example).

Of course, none of these make it impossible, but when we had the
discussion, I think we said that there were tasks with higher
priorities than this one. Don't hesitate to start experimenting and
prototyping around that if you're interested. However, we don't
guarantee that it will be integrated, it will depends on how much
complexity this adds.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com