Re: [OE-core] sstate cache management

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2023-02-22 at 17:56 +0000, Alex Kiernan wrote:
> I needed to do something about our shared sstate store and waded into
> the sstate cache management problem as the existing script takes hours
> to run over NFS (which for better or worse is where ours is). I've set
> myself the problem of replacing the existing script with something
> more extensible, understandable and performant.
> 
> I've got something which I believed was roughly right, but I'm ending
> up with questions I can't answer when comparing the two outputs...
> 
> If I run the existing shell script against a tiny sstate-cache (on my
> laptop) I get 420 duplicate files eligible for removal, if I run my
> script I get 491, looking into the delta, I pick out things like
> these:
> 
> $ find sstate-cache/ -name
> sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:*_populate_sysroot.tar.zst*
> -ls
>     49067     16 -rw-r--r--   1 alexk    alexk       14435 Feb 18
> 15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst
>     49129     16 -rw-r--r--   1 alexk    alexk       15205 Feb 18
> 15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst.siginfo
>   2490392     16 -rw-r--r--   1 alexk    alexk       15204 Feb 20
> 13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst.siginfo
>    339439     16 -rw-r--r--   1 alexk    alexk       14423 Feb 20
> 13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst
> 
> Which look to me like I should be able to delete the older ones, or am
> I missing something? Trying to follow what the existing script is
> supposed to do is challenging!

I'd say delete the older one but it does depend a lot on what you're
building against the cache (e.g. multiple releases). The system is
meant to touch files it uses and the autobuilder just ages out things
not touched for X time where X has varied depending on the pressure on
our NAS.

Cheers,

Richard