[Buildroot] [PATCH 2/2] core/instrumentation: shave minutes off the build time

[Trent Piepho <tpiepho@impinj.com>]

On Thu, 2018-03-22 at 18:11 +0100, Thomas De Schampheleire wrote:
> 
> > > The current timestamp-based approach not guaranteeing an accurate list is
> > > problematic for many such uses. And as you already mentioned, since we don't have
> > > full control over the build steps done in any given package, we don't know which
> > > timestamps they will use. There may be very good reasons to install certain
> > > files with their original timestamp and not the one from the build.
> > 
> > These are all valid concerns, but what do you suggest ?
> > 
> > The current approach of hashing all files clearly doesn't scale, as a
> > significant amount of build time is now spent on hashing files.
> > 

The hash approach also fails to handle symlinks well.

I think the answer is to install each package into a clean per-package
install tree, then combine those trees.  It will fix all the these
problems, allow more parallelism, and quite possible be outright more
efficient since it will scale with the number of files, while all
currently presented approaches scale with the number of files
multiplied by the number of packages.

> I can only observe that previously, when we still only listed the target files,
> the impact did not seem to be that bad, and the concerns about impact on build
> time arose with the creation of staging and host lists.

That was the big one, but here's a breakdown:

step_pkg_size-stage      143.50
step_pkg_size-target     267.14
step_pkg_size-host       419.21

Target alone is still over four minutes.  This was for a 12 minutes
building + 14 minutes instrumentation build.