sstate cache management

sstate cache management

[Alex Kiernan]

I needed to do something about our shared sstate store and waded into
the sstate cache management problem as the existing script takes hours
to run over NFS (which for better or worse is where ours is). I've set
myself the problem of replacing the existing script with something
more extensible, understandable and performant.

I've got something which I believed was roughly right, but I'm ending
up with questions I can't answer when comparing the two outputs...

If I run the existing shell script against a tiny sstate-cache (on my
laptop) I get 420 duplicate files eligible for removal, if I run my
script I get 491, looking into the delta, I pick out things like
these:

$ find sstate-cache/ -name
sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:*_populate_sysroot.tar.zst*
-ls
    49067     16 -rw-r--r--   1 alexk    alexk       14435 Feb 18
15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst
    49129     16 -rw-r--r--   1 alexk    alexk       15205 Feb 18
15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst.siginfo
  2490392     16 -rw-r--r--   1 alexk    alexk       15204 Feb 20
13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst.siginfo
   339439     16 -rw-r--r--   1 alexk    alexk       14423 Feb 20
13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst

Which look to me like I should be able to delete the older ones, or am
I missing something? Trying to follow what the existing script is
supposed to do is challenging!

-- 
Alex Kiernan

Re: [OE-core] sstate cache management

[Alexandre Belloni]

Hello,

This doesn't answer your question but did you look into
https://lore.kernel.org/all/20221121111102.5556-1-tomasz.dziendzielski@gmail.com/?

On 22/02/2023 17:56:35+0000, Alex Kiernan wrote:
> I needed to do something about our shared sstate store and waded into
> the sstate cache management problem as the existing script takes hours
> to run over NFS (which for better or worse is where ours is). I've set
> myself the problem of replacing the existing script with something
> more extensible, understandable and performant.
> 
> I've got something which I believed was roughly right, but I'm ending
> up with questions I can't answer when comparing the two outputs...
> 
> If I run the existing shell script against a tiny sstate-cache (on my
> laptop) I get 420 duplicate files eligible for removal, if I run my
> script I get 491, looking into the delta, I pick out things like
> these:
> 
> $ find sstate-cache/ -name
> sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:*_populate_sysroot.tar.zst*
> -ls
>     49067     16 -rw-r--r--   1 alexk    alexk       14435 Feb 18
> 15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst
>     49129     16 -rw-r--r--   1 alexk    alexk       15205 Feb 18
> 15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst.siginfo
>   2490392     16 -rw-r--r--   1 alexk    alexk       15204 Feb 20
> 13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst.siginfo
>    339439     16 -rw-r--r--   1 alexk    alexk       14423 Feb 20
> 13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst
> 
> Which look to me like I should be able to delete the older ones, or am
> I missing something? Trying to follow what the existing script is
> supposed to do is challenging!
> 
> -- 
> Alex Kiernan

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#177579): https://lists.openembedded.org/g/openembedded-core/message/177579
> Mute This Topic: https://lists.openembedded.org/mt/97165650/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] sstate cache management

[Alex Kiernan]

On Wed, Feb 22, 2023 at 7:36 PM Alexandre Belloni
<alexandre.belloni@bootlin.com> wrote:
>
> Hello,
>
> This doesn't answer your question but did you look into
> https://lore.kernel.org/all/20221121111102.5556-1-tomasz.dziendzielski@gmail.com/?
>

Thanks, yeah, I did... it really didn't scratch the particular itch I
had, mostly because I want to stuff other things in there. It feels a
bit like I'm rejecting it based on "not invented here", likely I just
need to get to a place where I can post what I have!

> On 22/02/2023 17:56:35+0000, Alex Kiernan wrote:
> > I needed to do something about our shared sstate store and waded into
> > the sstate cache management problem as the existing script takes hours
> > to run over NFS (which for better or worse is where ours is). I've set
> > myself the problem of replacing the existing script with something
> > more extensible, understandable and performant.
> >
> > I've got something which I believed was roughly right, but I'm ending
> > up with questions I can't answer when comparing the two outputs...
> >
> > If I run the existing shell script against a tiny sstate-cache (on my
> > laptop) I get 420 duplicate files eligible for removal, if I run my
> > script I get 491, looking into the delta, I pick out things like
> > these:
> >
> > $ find sstate-cache/ -name
> > sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:*_populate_sysroot.tar.zst*
> > -ls
> >     49067     16 -rw-r--r--   1 alexk    alexk       14435 Feb 18
> > 15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst
> >     49129     16 -rw-r--r--   1 alexk    alexk       15205 Feb 18
> > 15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst.siginfo
> >   2490392     16 -rw-r--r--   1 alexk    alexk       15204 Feb 20
> > 13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst.siginfo
> >    339439     16 -rw-r--r--   1 alexk    alexk       14423 Feb 20
> > 13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst
> >
> > Which look to me like I should be able to delete the older ones, or am
> > I missing something? Trying to follow what the existing script is
> > supposed to do is challenging!
> >
> > --
> > Alex Kiernan
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#177579): https://lists.openembedded.org/g/openembedded-core/message/177579
> > Mute This Topic: https://lists.openembedded.org/mt/97165650/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com



-- 
Alex Kiernan

Re: [OE-core] sstate cache management

[Richard Purdie]

On Wed, 2023-02-22 at 17:56 +0000, Alex Kiernan wrote:
> I needed to do something about our shared sstate store and waded into
> the sstate cache management problem as the existing script takes hours
> to run over NFS (which for better or worse is where ours is). I've set
> myself the problem of replacing the existing script with something
> more extensible, understandable and performant.
> 
> I've got something which I believed was roughly right, but I'm ending
> up with questions I can't answer when comparing the two outputs...
> 
> If I run the existing shell script against a tiny sstate-cache (on my
> laptop) I get 420 duplicate files eligible for removal, if I run my
> script I get 491, looking into the delta, I pick out things like
> these:
> 
> $ find sstate-cache/ -name
> sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:*_populate_sysroot.tar.zst*
> -ls
>     49067     16 -rw-r--r--   1 alexk    alexk       14435 Feb 18
> 15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst
>     49129     16 -rw-r--r--   1 alexk    alexk       15205 Feb 18
> 15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst.siginfo
>   2490392     16 -rw-r--r--   1 alexk    alexk       15204 Feb 20
> 13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst.siginfo
>    339439     16 -rw-r--r--   1 alexk    alexk       14423 Feb 20
> 13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst
> 
> Which look to me like I should be able to delete the older ones, or am
> I missing something? Trying to follow what the existing script is
> supposed to do is challenging!

I'd say delete the older one but it does depend a lot on what you're
building against the cache (e.g. multiple releases). The system is
meant to touch files it uses and the autobuilder just ages out things
not touched for X time where X has varied depending on the pressure on
our NAS.

Cheers,

Richard

Re: [OE-core] sstate cache management

[Alex Kiernan]

On Wed, Feb 22, 2023 at 8:33 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2023-02-22 at 17:56 +0000, Alex Kiernan wrote:
> > I needed to do something about our shared sstate store and waded into
> > the sstate cache management problem as the existing script takes hours
> > to run over NFS (which for better or worse is where ours is). I've set
> > myself the problem of replacing the existing script with something
> > more extensible, understandable and performant.
> >
> > I've got something which I believed was roughly right, but I'm ending
> > up with questions I can't answer when comparing the two outputs...
> >
> > If I run the existing shell script against a tiny sstate-cache (on my
> > laptop) I get 420 duplicate files eligible for removal, if I run my
> > script I get 491, looking into the delta, I pick out things like
> > these:
> >
> > $ find sstate-cache/ -name
> > sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:*_populate_sysroot.tar.zst*
> > -ls
> >     49067     16 -rw-r--r--   1 alexk    alexk       14435 Feb 18
> > 15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst
> >     49129     16 -rw-r--r--   1 alexk    alexk       15205 Feb 18
> > 15:29 sstate-cache/25/59/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:2559429e2a553085bc657f3d2a21a111818061448e5fa2aa16398afb5dad8b90_populate_sysroot.tar.zst.siginfo
> >   2490392     16 -rw-r--r--   1 alexk    alexk       15204 Feb 20
> > 13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst.siginfo
> >    339439     16 -rw-r--r--   1 alexk    alexk       14423 Feb 20
> > 13:24 sstate-cache/bf/08/sstate:libcap-ng:core2-64-poky-linux:0.8.3:r0:core2-64:10:bf08f26e6bc5ab56ed128441fb05edeef41aa881330d04eea127a93ede51713d_populate_sysroot.tar.zst
> >
> > Which look to me like I should be able to delete the older ones, or am
> > I missing something? Trying to follow what the existing script is
> > supposed to do is challenging!
>
> I'd say delete the older one but it does depend a lot on what you're
> building against the cache (e.g. multiple releases). The system is
> meant to touch files it uses and the autobuilder just ages out things
> not touched for X time where X has varied depending on the pressure on
> our NAS.
>

Thanks, so seems like the existing script not spotting that those
could be removed (using `-d`) is just a bug?

-- 
Alex Kiernan