[Buildroot] Speeding up builds

[Buildroot] Speeding up builds

[Dallas Clement]

Hi All,

I'm just looking for some tips on how to speed up my build times.  My
rootfs contains many packages with lots of dependencies.  Even with 8 fast
Xeon cores, 64GB of RAM, SSDs, and SAS drives, my builds are taking 1 hour
and 50 minutes.

When I observe CPU utilization with htop, I rarely see more than one CPU
core getting worked hard.  Occasionally, I will see all 8 cores maxed at
100%.

make -j 16 seems to make little difference.

What can I do to get all my CPU cores working all the time?

Thanks,

Dallas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131104/1e04a278/attachment.html>

[Buildroot] Speeding up builds

[Thomas De Schampheleire]

Hi Dallas,

Dallas Clement <dallas.a.clement@gmail.com> wrote:
>Hi All,
>
>I'm just looking for some tips on how to speed up my build times.  My
>rootfs contains many packages with lots of dependencies.  Even with 8 fast
>Xeon cores, 64GB of RAM, SSDs, and SAS drives, my builds are taking 1 hour
>and 50 minutes.
>
>When I observe CPU utilization with htop, I rarely see more than one CPU
>core getting worked hard.  Occasionally, I will see all 8 cores maxed at
>100%.
>
>make -j 16 seems to make little difference.
>
>What can I do to get all my CPU cores working all the time?

What is the value of BR2_JLEVEL in your .config?
And have you tried enabling ccache?

What takes a significant amount of time are the configure steps of all packages, which are not parallelized.

Best regards,
Thomas

[Buildroot] Speeding up builds

[Dallas Clement]

Hi Thomas,

It's BR2_JLEVEL=0, and I have confirmed that when make is invoked it is -j
16  (8 cores X 2) on my build system.

I have tried ccache, ran into problems, and ended up aborting.  I may pick
up that effort again.

My focus right now is reducing the build time for a fresh clean build with
no caching or exisiting object files.

That's a bummer about the configure steps being serialized.  I have
observed this as well.

Thanks,

Dallas


On Mon, Nov 4, 2013 at 1:06 PM, Thomas De Schampheleire <
patrickdepinguin@gmail.com> wrote:

> Hi Dallas,
>
> Dallas Clement <dallas.a.clement@gmail.com> wrote:
> >Hi All,
> >
> >I'm just looking for some tips on how to speed up my build times.  My
> >rootfs contains many packages with lots of dependencies.  Even with 8 fast
> >Xeon cores, 64GB of RAM, SSDs, and SAS drives, my builds are taking 1 hour
> >and 50 minutes.
> >
> >When I observe CPU utilization with htop, I rarely see more than one CPU
> >core getting worked hard.  Occasionally, I will see all 8 cores maxed at
> >100%.
> >
> >make -j 16 seems to make little difference.
> >
> >What can I do to get all my CPU cores working all the time?
>
> What is the value of BR2_JLEVEL in your .config?
> And have you tried enabling ccache?
>
> What takes a significant amount of time are the configure steps of all
> packages, which are not parallelized.
>
> Best regards,
> Thomas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131104/4a235859/attachment.html>

[Buildroot] Speeding up builds

[Thomas De Schampheleire]

Hi Dallas,

On Mon, Nov 4, 2013 at 8:41 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> Hi Thomas,
>
> It's BR2_JLEVEL=0, and I have confirmed that when make is invoked it is -j
> 16  (8 cores X 2) on my build system.
>
> I have tried ccache, ran into problems, and ended up aborting.  I may pick
> up that effort again.
>
> My focus right now is reducing the build time for a fresh clean build with
> no caching or exisiting object files.
>
> That's a bummer about the configure steps being serialized.  I have observed
> this as well.

Are you using an external toolchain, or are you rebuilding that one
from scratch as well?
One common optimization is building the toolchain only once, tarring
the result, and using that as external toolchain for the real build.

Best regards,
Thomas

[Buildroot] Speeding up builds

[Dallas Clement]

Hello Thomas,

> Are you using an external toolchain, or are you rebuilding that one
> from scratch as well?

Unfortunately, no gains to be had there.  I am using an external toolchain.

Thanks,

Dallas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131104/bc5b46e7/attachment.html>

[Buildroot] Speeding up builds

[Dallas Clement]

Does anyone know if there is still a shared configuration cache option?  I
might get some mileage out of that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131104/9731463b/attachment.html>

[Buildroot] Speeding up builds

[Thomas Petazzoni]

Dear Dallas Clement,

On Mon, 4 Nov 2013 17:15:24 -0600, Dallas Clement wrote:

> Does anyone know if there is still a shared configuration cache
> option?  I might get some mileage out of that.

No, we removed it years ago. Different configure.ac scripts use
identical variable names in the cache to store the result of tests that
are not exactly the same. It was causing too much breakage, so we
removed it.

commit 5f71fc1f19c21676f029b63499e0341a89cf2049
Author: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Date:   Thu Nov 4 20:31:25 2010 +0100

    Remove support for shared configuration cache
    
    The configuration cache shared between packages, while being in
    principle a nice idea to speed-up the configuration of packages by
    avoiding repetitive identical checks, turned out to be unreliable due
    to the subtle differences between similar but not identical checks in
    different packages. After spending some time trying to fix those, we
    concluded that supporting the shared configuration cache is definitely
    too hard and too unreliable, and that we'd better get rid of it
    altogether.
    
    This patch therefore removes the shared configuration cache
    infrastructure and usage.
    
    Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
    Signed-off-by: Peter Korsgaard <jacmet@sunsite.dk>

-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Speeding up builds

[Thomas Petazzoni]

Dear Dallas Clement,

On Mon, 4 Nov 2013 13:00:54 -0600, Dallas Clement wrote:

> I'm just looking for some tips on how to speed up my build times.  My
> rootfs contains many packages with lots of dependencies.  Even with 8
> fast Xeon cores, 64GB of RAM, SSDs, and SAS drives, my builds are
> taking 1 hour and 50 minutes.

Just checking: are you running these builds directly on the machine, or
are you using some kind of virtual machine?

Do you think you could provide us with your Buildroot configuration, so
we can maybe have a look?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Speeding up builds

[Dallas Clement]

Hi Thomas,

> Just checking: are you running these builds directly on the machine, or
> are you using some kind of virtual machine?

No, I'm not.  Running directly on RHEL 6.4 x86_64

> Do you think you could provide us with your Buildroot configuration, so
we can maybe have a look?

You bet, please see attached.

Thanks much,

Dallas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131104/e0752986/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config
Type: application/octet-stream
Size: 40187 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131104/e0752986/attachment-0001.obj>

[Buildroot] Speeding up builds

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Dear Dallas Clement,
 > On Mon, 4 Nov 2013 13:00:54 -0600, Dallas Clement wrote:

 >> I'm just looking for some tips on how to speed up my build times.  My
 >> rootfs contains many packages with lots of dependencies.  Even with 8
 >> fast Xeon cores, 64GB of RAM, SSDs, and SAS drives, my builds are
 >> taking 1 hour and 50 minutes.

~2h sounds quite long indeed. I don't use as many packages as you, but a
fairly big config (123 packages, including Qt and Gstreamer1) takes <
20mins on a pretty normal quad haswell (E3-1240) / 32G RAM / SSDs.

-- 
Bye, Peter Korsgaard

[Buildroot] Speeding up builds

[Dallas Clement]

Apart from makefile -j during C file compilation, it seems that building is
very serialized.  A lot of time is spent in ./configure script execution,
and quite a bit in installation of files.  All of this appears to be very
serial in nature.  I realize there is not much that can be done about
autoconf / automake to avoid this serialization.

It would be great if buildroot could build multiple packages in parallel.
That way, multiple cores could at least be performing activities like
./configure which tend to be serial.   Has there ever been any discussion
about this?


On Mon, Nov 4, 2013 at 5:48 PM, Peter Korsgaard <jacmet@uclibc.org> wrote:

> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> writes:
>
>  > Dear Dallas Clement,
>  > On Mon, 4 Nov 2013 13:00:54 -0600, Dallas Clement wrote:
>
>  >> I'm just looking for some tips on how to speed up my build times.  My
>  >> rootfs contains many packages with lots of dependencies.  Even with 8
>  >> fast Xeon cores, 64GB of RAM, SSDs, and SAS drives, my builds are
>  >> taking 1 hour and 50 minutes.
>
> ~2h sounds quite long indeed. I don't use as many packages as you, but a
> fairly big config (123 packages, including Qt and Gstreamer1) takes <
> 20mins on a pretty normal quad haswell (E3-1240) / 32G RAM / SSDs.
>
> --
> Bye, Peter Korsgaard
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131105/f96ec5c0/attachment.html>

[Buildroot] Speeding up builds

[Thomas De Schampheleire]

Hi Dallas,

Please don't top-post...

Dallas Clement <dallas.a.clement@gmail.com> wrote:
>Apart from makefile -j during C file compilation, it seems that building is
>very serialized.  A lot of time is spent in ./configure script execution,
>and quite a bit in installation of files.  All of this appears to be very
>serial in nature.  I realize there is not much that can be done about
>autoconf / automake to avoid this serialization.
>
>It would be great if buildroot could build multiple packages in parallel.
>That way, multiple cores could at least be performing activities like
>./configure which tend to be serial.   Has there ever been any discussion
>about this?

Yes, definitely. There is at least one unsolved problem though. Some packages have optional dependencies: if the dependency is present when the package is built, it is used, otherwise it isn't (but compilation happily proceeds). This means that the package order is relevant in this case, and the end result will differ if the order is swapped due to parallel building. Expressing all these potential dependencies is unrealistic.

Have a look in the recent buildroot mailing list archives for the patches from Fabio Porcedda and the discussion around it. He has been working on this topic.

Best regards,
Thomas