[Buildroot] 2019.02 performance

[Buildroot] 2019.02 performance

[Trent Piepho]

With 2018.02, I had to disable the package size and related hooks as
they were too slow: over half the build time.  As I'm sure some
remember, there was a thread about it and performance was
improved.  Here's the results with 2019.02.  I'm using a real config
for a product we have and testing on a reasonably modern native Linux
workstation with a decent NVMe SSD and a four core processor.  Using
ccache, which had already built the software.

Total build time: real 22:53, CPU 39:04.

Breakdown by build step (host/stage/target/images mean install same),
across all packages:

configure                666.223
build                    381.480
host                     122.191
extract                   57.489
target                    57.039
stage                     28.356
patch                      9.711
postimage                  9.008
finalize                   6.396
images                     0.063
download                   0.022


The single threaded configure and cmake scripts remain the killer,
through total of all install steps is about 2/3 of the build step time.

Breakdown for the global instrumentation hooks, across all packages and
build steps:

check_host_rpath          64.345
step_pkg_size             32.992
check_bin_arch            26.286
step_check_build_dir       8.365
step_time                  7.895

pkg_size is much more reasonable now.  In total, these are about 10.2%
of the total build time.

Next are benchmarks from a cloud VM, which is considerably slower.

Total build time: Real 53:58, CPU 55:22

configure               1021.933
build                    936.449
host                     206.171
extract                  106.642
target                    98.028
stage                     53.747
finalize                  40.840
download                  25.091
postimage                 22.876
patch                     19.153
images                     0.119

The GIH breakdown:

check_host_rpath         117.981
step_pkg_size             49.881
check_bin_arch            46.035
step_time                 12.573
step_check_build_dir       9.307

About 7.2% of total build time.

[Buildroot] 2019.02 performance

[Thomas Petazzoni]

Hello,

First of all, thanks a *lot* for sharing this kind of analysis. This is
very useful.

On Mon, 11 Mar 2019 21:59:03 +0000
Trent Piepho <tpiepho@impinj.com> wrote:

> With 2018.02, I had to disable the package size and related hooks as
> they were too slow: over half the build time.  As I'm sure some
> remember, there was a thread about it and performance was
> improved.  Here's the results with 2019.02.  I'm using a real config
> for a product we have and testing on a reasonably modern native Linux
> workstation with a decent NVMe SSD and a four core processor.  Using
> ccache, which had already built the software.

A hot ccache will drastically decrease the time of the "build" step of
each package, and make the "configure" and "install" steps look
comparatively very long. A build without ccache support or with a cold
cache will show a much larger "build" time I believe.

> Breakdown by build step (host/stage/target/images mean install same),
> across all packages:
> 
> configure                666.223
> build                    381.480
> host                     122.191
> extract                   57.489
> target                    57.039
> stage                     28.356
> patch                      9.711
> postimage                  9.008
> finalize                   6.396
> images                     0.063
> download                   0.022
> 
> 
> The single threaded configure and cmake scripts remain the killer,
> through total of all install steps is about 2/3 of the build step time.
> 
> Breakdown for the global instrumentation hooks, across all packages and
> build steps:
> 
> check_host_rpath          64.345
> step_pkg_size             32.992
> check_bin_arch            26.286
> step_check_build_dir       8.365
> step_time                  7.895
> 
> pkg_size is much more reasonable now.  In total, these are about 10.2%
> of the total build time.

10% just for the instrumentation is still a lot. Our of 22 minutes of
your build time, it's still ~2 minutes. The clear dominating factor
here is check_host_rpath. If I'm reading the check-host-rpath
correctly, we are checking all files in $(HOST_DIR) at the end of the
build of every package. Couldn't this be optimized by using the list of
files installed by a package, like we do for check-bin-arch ?

I think that back when check-host-rpath we introduced, we did not had
the list of files installed by host packages. But now that we do,
perhaps we should use it ?

For the other ones, I don't immediately see any easy optimization
route, except perhaps writing some dedicated C programs instead of
using the slower shell scripts forking like crazy.

Best regards,

Thomas Petazzoni
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] 2019.02 performance

[Arnout Vandecappelle]

On 12/03/2019 09:05, Thomas Petazzoni wrote:
> For the other ones, I don't immediately see any easy optimization
> route, except perhaps writing some dedicated C programs instead of
> using the slower shell scripts forking like crazy.

 Assuming check-bin-arch becomes the dominant one when we use the pkg-file-list
for check_host_rpath, I've taking a look at how crazy this script is forking.
The script itself is executed once per package, which doesn't sound excessive.
It then forks a single sed to get the files for the package, and for each
non-ignored file it does a readelf, a sed and a head. This doesn't sound
excessive either...

 So the only thing I can think of, is to create a small C program that reads the
first 4 bytes of a file and checks if it is .ELF, to avoid the readelf+sed+head.
Doesn't sound like it would make that much of a difference...

 We could also use the .files-list*.txt which contains only the files belonging
to the package itself, which saves the outer sed. But that would only make a
difference if there's a huge number of files in your output.

 Regards,
 Arnout

[Buildroot] 2019.02 performance

[Trent Piepho]

On Tue, 2019-03-12 at 09:05 +0100, Thomas Petazzoni wrote:
> 
> A hot ccache will drastically decrease the time of the "build" step of
> each package, and make the "configure" and "install" steps look
> comparatively very long. A build without ccache support or with a cold
> cache will show a much larger "build" time I believe.

Yes, it will be larger.  I don't have a benchmark handy with that info.
 Perhaps I'll do one overnight.  But what I see here, is that nearly
every build we do, both manually and as part of CI, has a very high
ccache hit rate.  I was keeping track of the stats from CI for a while
on that.  

> > 
> > pkg_size is much more reasonable now.  In total, these are about 10.2%
> > of the total build time.
> 
> 10% just for the instrumentation is still a lot. Our of 22 minutes of
> your build time, it's still ~2 minutes. The clear dominating factor

This is just the build image time (from scratch + ccache).  There is
also a documentation build, and steps like legal info, package build
time graphs, and dependency graphs are quite long.  The dependency
graphs are clearly the biggest low hanging fruit once this step is
considered.

Though check host rpath could be sped up!  One thing that might skew
these results is our host packages to target packages ratio might be
higher than typical.  If I took out the host based python software and
host based image crypto signing stuff there'd be a lot fewer host ELF
files.