[Buildroot] [PATCH 2/2] core/instrumentation: shave minutes off the build time

[Yann E. MORIN <yann.morin.1998@free.fr>]

Peter, All,

On 2018-03-18 15:14 +0100, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
>  > As part of the build, we run some instrumentation hooks to gather
>  > statistics about the usage of the target/, staging/ and host/
>  > directories, so that we can generate reports for the user, that
>  > shows:
>  >   - for each file, what package installed it,
>  >   - for each package,the size that it installed.
[--SNIP--]
>  > So, we switch to using mtime, because in the end that's still good-enough
>  > for our use-case: generating some graphs. It is not mission-critical, and
>  > if a graph is slightly off, that's not biggy. It can anyway be attributed
>  > to a broken package's buildsystem, which should get fixed.
>  >   - /foo/bar/ exists
>  >   - a package installs /foo/bar/buz
>  >   - mtime of /foo/bar/ is changed to account for the nex file in it.
> 
> Playing around with this, I noticed two other issues:
> 
> - It doesn't work for packages using rsync to install,
>   E.G. skeleton-init-common as rsync also sets the mtime to match the
>   source files

We could maybe tell rsycn not to do that, then?

> - It breaks for <pkg>-reinstall

Well, we can't guarantee anything except with a clean build from scratch
anyway.

> I don't think either of those are really big issues compared to the huge
> slowdown, but it is worth noticing.

Well, the -reinstall was already not working correctly, because the list
pf files before/after would be alsmost the same, and the md5-diff would
miss all the laready-installed files for the package.

The rsync issue is new, but we can "fix" it in a later patch, then, for
those packages like the skeletons, by using the --no-times option for
example.

However, if a third-party package internally uses rsync as its install
method, we're screwed. But who would be insane enough to do that? ;-]

Alternatively, we could use ctime instead of mtime, maybe? Or check
both?

>  > +define step_pkg_size_inner
>  > +	cd $(2); \
>  > +	find . \( -type f -o -type L \) \
>  > +		-newer $($(PKG)_DIR)/.stamp_built \
>  > +		-exec printf '$(1),%s\n' {} + \
>  > +		>> $(BUILD_DIR)/packages-file-list$(3).txt
> 
> What find version are you using? My fileutils find (and the busybox
> applet) use 'l' for symlinks, so I've changed it to that.

Doh, the s/L/l/ is still uncomitted here. Dang...

> Committed with that fixed (and a few tweaks to the commit message),
> thanks.

Thanks! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'