RE: [bitbake-devel] Cache timings with various changes

[Peter Kjellerstedt <peter.kjellerstedt@axis.com>]

> -----Original Message-----
> From: bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 23 november 2022 01:07
> To: bitbake-devel <bitbake-devel@lists.openembedded.org>
> Cc: Joshua Watt <JPEWhacker@gmail.com>
> Subject: [bitbake-devel] Cache timings with various changes
> 
> I've talked about changing the data we cache in various forums. These
> are some actual numbers from a local system (88 core):
> 
> Patch                   Full reparse time    Cache load time    Cache size (compressed)     Cache size (uncompressed)
> -----                   -----------------    ---------------    -----------------------     -------------------------
> master baseline         5.732s               2.261s             N/A                         24M
> frozenset changes       4.326s               2.289s             N/A                         24M
> pickle in one           4.509s               2.177s             N/A                         24M
> compress with zstd      6.040s               2.204s             3.4M                        24M
> add hash data to cache  9.693s               4.327s             17M                         83M
> 
> It seems the frozenset improvements I have speed things up nicely and
> look like a good change. The compression does slow things down, as does
> pickling the cache in one go.
> 
> The patches are in poky-contrib rpurdie/t222. I'll post the frozenset
> one, the others look like they need more thought.
> 
> Cheers,
> 
> Richard

Not sure how you measured the times, so I just looked at the progress bars 
for "Loading cache" and "Parsing recipes". With our layers (4717 recipes) 
and my computer with 8 cores (16 threads), this is what I got:

* Without rpurdie/t222 applied, it takes 0-2 s to load the cache, it takes 
  43-46 s to parse the recipes and the cache is 185 MB.
* After applying up to "cache: Switch to compressing the cache file with 
  zstd", the cache load and recipe parse times remains the same, but the 
  size of the cache decreases to 17 MB
* With all of rpurdie/t222 applied it takes 2-3 to load the cache, 51-53 s 
  to parse the recipes and the cache is 55 MB.

//Peter