[PATCH] dev-manual: disk-space: mention faster "find" command to trim sstate cache

[PATCH] dev-manual: disk-space: mention faster "find" command to trim sstate cache

[michael.opdenacker]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

[YOCTO #15182]

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Yoann CONGAL <yoann.congal@smile.fr>
Reported-by: Randy MacLeod <randy.macleod@windriver.com>
Reported-by: Josef Holzmayr <jester@theyoctojester.info>
---
 documentation/dev-manual/disk-space.rst | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/documentation/dev-manual/disk-space.rst b/documentation/dev-manual/disk-space.rst
index c63591cc7a..670f3d2792 100644
--- a/documentation/dev-manual/disk-space.rst
+++ b/documentation/dev-manual/disk-space.rst
@@ -27,19 +27,25 @@ Purging Duplicate Shared State Cache Files
 ==========================================
 
 After multiple build iterations, the Shared State (sstate) cache can contain
-duplicate cache files for a given package, while only the most recent one
-is likely to be reusable. The following command purges all but the
-newest sstate cache file for each package::
+duplicate cache files for a given package, consuming a substantial amount of
+disk space. However, only the most recent cache files are likeky to be reusable.
 
-   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+The following command is a quick way to purge all the cache files which
+are older than a specified number of days::
 
-This command will ask you to confirm the deletions it identifies.
+   find build/sstate-cache -type f -atime +$DAYS -delete
 
-.. note::
+OpenEmbedded-Core also offers a command which can be used to look for
+cache files only for enabled architectures, and purge all but the newest
+ones on each architecture::
 
-   The duplicated sstate cache files of one package must have the same
-   architecture, which means that sstate cache files with multiple
-   architectures are not considered as duplicate.
+   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
 
+This command will ask you to confirm the deletions it identifies.
 Run ``sstate-cache-management.sh`` for more details about this script.
 
+.. note::
+
+   As this command is much more cautious and selective, removing only cache files,
+   it will execute much slower than the simple ``find`` command described above.
+   Therefore, it may not be your best option to trim huge cache directories.
-- 
2.34.1

Re: [docs] [PATCH] dev-manual: disk-space: mention faster "find" command to trim sstate cache

[Richard Purdie]

On Fri, 2023-08-11 at 11:09 +0200, Michael Opdenacker via
lists.yoctoproject.org wrote:
> From: Michael Opdenacker <michael.opdenacker@bootlin.com>
> 
> [YOCTO #15182]
> 
> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> Reported-by: Yoann CONGAL <yoann.congal@smile.fr>
> Reported-by: Randy MacLeod <randy.macleod@windriver.com>
> Reported-by: Josef Holzmayr <jester@theyoctojester.info>
> ---
>  documentation/dev-manual/disk-space.rst | 24 +++++++++++++++---------
>  1 file changed, 15 insertions(+), 9 deletions(-)
> 
> diff --git a/documentation/dev-manual/disk-space.rst b/documentation/dev-manual/disk-space.rst
> index c63591cc7a..670f3d2792 100644
> --- a/documentation/dev-manual/disk-space.rst
> +++ b/documentation/dev-manual/disk-space.rst
> @@ -27,19 +27,25 @@ Purging Duplicate Shared State Cache Files
>  ==========================================
>  
>  After multiple build iterations, the Shared State (sstate) cache can contain
> -duplicate cache files for a given package, while only the most recent one
> -is likely to be reusable. The following command purges all but the
> -newest sstate cache file for each package::
> +duplicate cache files for a given package, consuming a substantial amount of
> +disk space. However, only the most recent cache files are likeky to be reusable.
>  
> -   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
> +The following command is a quick way to purge all the cache files which
> +are older than a specified number of days::
>  
> -This command will ask you to confirm the deletions it identifies.
> +   find build/sstate-cache -type f -atime +$DAYS -delete
>  
> -.. note::
> +OpenEmbedded-Core also offers a command which can be used to look for
> +cache files only for enabled architectures, and purge all but the newest
> +ones on each architecture::
>  
> -   The duplicated sstate cache files of one package must have the same
> -   architecture, which means that sstate cache files with multiple
> -   architectures are not considered as duplicate.
> +   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
>  
> +This command will ask you to confirm the deletions it identifies.
>  Run ``sstate-cache-management.sh`` for more details about this script.

As long as bitbake builds have writable access to the sstate cache,
files are touched as they are accessed so it is easy to know which ones
are being used and which ones are not. As such you can use mtime in the
above example even for a partition mounted as "noatime" for performance
reasons.

>  
> +.. note::
> +
> +   As this command is much more cautious and selective, removing only cache files,
> +   it will execute much slower than the simple ``find`` command described above.
> +   Therefore, it may not be your best option to trim huge cache directories.

It doesn't remove only cache files, it removes files that it considers
to be unreachable by exploring a set of build configurations. It
requires full build environment to be available and doesn't work well
covering multiple releases. It also doesn't work in a limited
environment like a BSD based NAS where the above find command would
still likely work easily.

Can we get rid of sstate-cache-management.sh? :)

Cheers,

Richard

Re: [docs] [PATCH] dev-manual: disk-space: mention faster "find" command to trim sstate cache

[Michael Opdenacker]

Hi Richard,

On 11.08.23 at 17:06, Richard Purdie wrote:
> On Fri, 2023-08-11 at 11:09 +0200, Michael Opdenacker via
> lists.yoctoproject.org wrote:
>> From: Michael Opdenacker <michael.opdenacker@bootlin.com>
>>
>> [YOCTO #15182]
>>
>> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
>> Reported-by: Yoann CONGAL <yoann.congal@smile.fr>
>> Reported-by: Randy MacLeod <randy.macleod@windriver.com>
>> Reported-by: Josef Holzmayr <jester@theyoctojester.info>
>> ---
>>   documentation/dev-manual/disk-space.rst | 24 +++++++++++++++---------
>>   1 file changed, 15 insertions(+), 9 deletions(-)
>>
>> diff --git a/documentation/dev-manual/disk-space.rst b/documentation/dev-manual/disk-space.rst
>> index c63591cc7a..670f3d2792 100644
>> --- a/documentation/dev-manual/disk-space.rst
>> +++ b/documentation/dev-manual/disk-space.rst
>> @@ -27,19 +27,25 @@ Purging Duplicate Shared State Cache Files
>>   ==========================================
>>   
>>   After multiple build iterations, the Shared State (sstate) cache can contain
>> -duplicate cache files for a given package, while only the most recent one
>> -is likely to be reusable. The following command purges all but the
>> -newest sstate cache file for each package::
>> +duplicate cache files for a given package, consuming a substantial amount of
>> +disk space. However, only the most recent cache files are likeky to be reusable.
>>   
>> -   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
>> +The following command is a quick way to purge all the cache files which
>> +are older than a specified number of days::
>>   
>> -This command will ask you to confirm the deletions it identifies.
>> +   find build/sstate-cache -type f -atime +$DAYS -delete
>>   
>> -.. note::
>> +OpenEmbedded-Core also offers a command which can be used to look for
>> +cache files only for enabled architectures, and purge all but the newest
>> +ones on each architecture::
>>   
>> -   The duplicated sstate cache files of one package must have the same
>> -   architecture, which means that sstate cache files with multiple
>> -   architectures are not considered as duplicate.
>> +   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
>>   
>> +This command will ask you to confirm the deletions it identifies.
>>   Run ``sstate-cache-management.sh`` for more details about this script.
> As long as bitbake builds have writable access to the sstate cache,
> files are touched as they are accessed so it is easy to know which ones
> are being used and which ones are not. As such you can use mtime in the
> above example even for a partition mounted as "noatime" for performance
> reasons.


Good idea, this way the solution is a little broader as "noatime" is 
frequently used.

>
>>   
>> +.. note::
>> +
>> +   As this command is much more cautious and selective, removing only cache files,
>> +   it will execute much slower than the simple ``find`` command described above.
>> +   Therefore, it may not be your best option to trim huge cache directories.
> It doesn't remove only cache files, it removes files that it considers
> to be unreachable by exploring a set of build configurations. It
> requires full build environment to be available and doesn't work well
> covering multiple releases. It also doesn't work in a limited
> environment like a BSD based NAS where the above find command would
> still likely work easily.
>
> Can we get rid of sstate-cache-management.sh? :)

Thanks for these useful details about this script!

Are you suggesting we could at least remove sstate-cache-management.sh 
from the docs?
Thanks again,
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com