[Buildroot] Parallel top-level make and multiple rootfs types

[Buildroot] Parallel top-level make and multiple rootfs types

[Jan Kundrát]

Hi,
I've enabled that unsupported feature for parallel top-level make by 
removing ".NOTPARALLEL". I'm doing this because it cuts my build time by a 
factor of four (roughly 50min -> 12min; yes, these serialized configure 
steps are extremely expensive on a 24-core build node).

Everything works well for me up to and including the `target-finalize` 
step. I have multiple CONFIG_TARGET_ROOTFS_*, and their rules clobber each 
other's auxiliary files, either directly (_fakeroot.fs), or via mkusers 
which creates duplicate rows in target/etc/group.

I tried to fix this by something like:

  .NOTPARALLEL: $$(BINARIES_DIR)/rootfs.$(1)

...but it just caused the entire build to once again become serial, 
package-after-package, which is not what I want.

Is there a better fix than running `make -j32 target-finalize && make` as a 
workaround?

With kind regards,
Jan

[Buildroot] Parallel top-level make and multiple rootfs types

[Jan Kundrát]

On ?tvrtek 26. ??jna 2017 1:10:05 CEST, Jan Kundr?t wrote:
> Hi,
> I've enabled that unsupported feature for parallel top-level 
> make by removing ".NOTPARALLEL". I'm doing this because it cuts 
> my build time by a factor of four (roughly 50min -> 12min; yes, 
> these serialized configure steps are extremely expensive on a 
> 24-core build node).
>
> Everything works well for me up to and including the 
> `target-finalize` step. I have multiple CONFIG_TARGET_ROOTFS_*, 
> and their rules clobber each other's auxiliary files, either 
> directly (_fakeroot.fs), or via mkusers which creates duplicate 
> rows in target/etc/group.
>
> I tried to fix this by something like:
>
>  .NOTPARALLEL: $$(BINARIES_DIR)/rootfs.$(1)
>
> ...but it just caused the entire build to once again become 
> serial, package-after-package, which is not what I want.
>
> Is there a better fix than running `make -j32 target-finalize 
> && make` as a workaround?

Hi,
can I disable parallel execution of the rootfs-* build steps while still 
building everything up to target-finalize in parallel?

Cheers,
Jan

[Buildroot] Parallel top-level make and multiple rootfs types

[Arnout Vandecappelle]

On 02-11-17 19:27, Jan Kundr?t wrote:
> On ?tvrtek 26. ??jna 2017 1:10:05 CEST, Jan Kundr?t wrote:
>> Hi,
>> I've enabled that unsupported feature for parallel top-level make by removing
>> ".NOTPARALLEL". I'm doing this because it cuts my build time by a factor of
>> four (roughly 50min -> 12min; yes, these serialized configure steps are
>> extremely expensive on a 24-core build node).
>>
>> Everything works well for me up to and including the `target-finalize` step. I
>> have multiple CONFIG_TARGET_ROOTFS_*, and their rules clobber each other's
>> auxiliary files, either directly (_fakeroot.fs), or via mkusers which creates
>> duplicate rows in target/etc/group.
>>
>> I tried to fix this by something like:
>>
>> ?.NOTPARALLEL: $$(BINARIES_DIR)/rootfs.$(1)
>>
>> ...but it just caused the entire build to once again become serial,
>> package-after-package, which is not what I want.
>>
>> Is there a better fix than running `make -j32 target-finalize && make` as a
>> workaround?
> 
> Hi,
> can I disable parallel execution of the rootfs-* build steps while still
> building everything up to target-finalize in parallel?

 No, that's not possible with make. I don't know if Thomas has already thought
about this aspect of top-level parallel...

 I think the solution is to change the way we build the rootfs: instead of doing
the fakeroot steps over and over again, do it once, save with the -s option, and
then do the individual filesystems in a separate (readonly) fakeroot with -i to
read the previously created fakeroot file.

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Parallel top-level make and multiple rootfs types

[Thomas Petazzoni]

Hello,

On Sat, 4 Nov 2017 21:56:58 +0100, Arnout Vandecappelle wrote:

> > can I disable parallel execution of the rootfs-* build steps while still
> > building everything up to target-finalize in parallel?  
> 
>  No, that's not possible with make. I don't know if Thomas has already thought
> about this aspect of top-level parallel...

No, I haven't thought about this aspect of top-level parallel build.

>  I think the solution is to change the way we build the rootfs: instead of doing
> the fakeroot steps over and over again, do it once, save with the -s option, and
> then do the individual filesystems in a separate (readonly) fakeroot with -i to
> read the previously created fakeroot file.

I think Yann did some work in terms of parallel generation of
filesystem images, but I'm not sure exactly what he did :)

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] Parallel top-level make and multiple rootfs types

[Yann E. MORIN]

Thomas, Arnout, All,

On 2017-11-04 23:05 +0100, Thomas Petazzoni spake thusly:
> On Sat, 4 Nov 2017 21:56:58 +0100, Arnout Vandecappelle wrote:
> > > can I disable parallel execution of the rootfs-* build steps while still
> > > building everything up to target-finalize in parallel?  
> > 
> >  No, that's not possible with make. I don't know if Thomas has already thought
> > about this aspect of top-level parallel...
> 
> No, I haven't thought about this aspect of top-level parallel build.
> 
> >  I think the solution is to change the way we build the rootfs: instead of doing
> > the fakeroot steps over and over again, do it once, save with the -s option, and
> > then do the individual filesystems in a separate (readonly) fakeroot with -i to
> > read the previously created fakeroot file.
> 
> I think Yann did some work in terms of parallel generation of
> filesystem images, but I'm not sure exactly what he did :)

Quite a lot, in fact... See:
    https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/fs-3

I wanted to send it earlier, but I did not yet find time to completely
test it...

The runtime-tests infra needed a few tweaks to make it work correctly,
and it also caught a few regressions that I already fixed, but it really
needs a bit more testing I think...

Hopefully I get some time to respin it at the bginning iof next cycle...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'