[PATCH v2] dev-manual: disk-space: mention faster "find" command to trim sstate cache

[PATCH v2] dev-manual: disk-space: mention faster "find" command to trim sstate cache

[michael.opdenacker]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

[YOCTO #15182]

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Yoann CONGAL <yoann.congal@smile.fr>
Reported-by: Randy MacLeod <randy.macleod@windriver.com>
Reported-by: Josef Holzmayr <jester@theyoctojester.info>

---

- Changes in V2:
  - Using a more universal "-mtime" option to find files to remove
    as suggested by Richard Purdie. V1 was using "-atime" which
    won't work with systems mounted with the "noatime" option used
    to increase disk performance.
  - Add more details about sstate-cache-management.sh provided by
    Richard Purdie, perhaps not recommended but useful to document
    as long as it is shipped in OE-core.
---
 documentation/dev-manual/disk-space.rst | 31 ++++++++++++++++++-------
 1 file changed, 22 insertions(+), 9 deletions(-)

diff --git a/documentation/dev-manual/disk-space.rst b/documentation/dev-manual/disk-space.rst
index c63591cc7a..01cdf911c4 100644
--- a/documentation/dev-manual/disk-space.rst
+++ b/documentation/dev-manual/disk-space.rst
@@ -27,19 +27,32 @@ Purging Duplicate Shared State Cache Files
 ==========================================
 
 After multiple build iterations, the Shared State (sstate) cache can contain
-duplicate cache files for a given package, while only the most recent one
-is likely to be reusable. The following command purges all but the
-newest sstate cache file for each package::
+duplicate cache files for a given package, consuming a substantial amount of
+disk space. However, only the most recent cache files are likeky to be reusable.
 
-   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+The following command is a quick way to purge all the cache files which
+haven't been used for a least a specified number of days::
 
-This command will ask you to confirm the deletions it identifies.
+   find build/sstate-cache -type f -mtime +$DAYS -delete
 
-.. note::
+The above command relies on the fact that BitBake touches the sstate cache
+files as it accesses them, when it has write access to the cache.
+
+For more advanced needs, OpenEmbedded-Core also offers a more elaborate
+command. It has the ability to purge all but the newest cache files on each
+architecture, and also to remove files that it considers unreachable by
+exploring a set of build configurations. However, this command
+requires a full build environment to be available and doesn't work well
+covering multiple releases. It won't work either on limited environments
+such as BSD based NAS::
 
-   The duplicated sstate cache files of one package must have the same
-   architecture, which means that sstate cache files with multiple
-   architectures are not considered as duplicate.
+   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
 
+This command will ask you to confirm the deletions it identifies.
 Run ``sstate-cache-management.sh`` for more details about this script.
 
+.. note::
+
+   As this command is much more cautious and selective, removing only cache files,
+   it will execute much slower than the simple ``find`` command described above.
+   Therefore, it may not be your best option to trim huge cache directories.
-- 
2.34.1

Re: [docs] [PATCH v2] dev-manual: disk-space: mention faster "find" command to trim sstate cache

[Richard Purdie]

On Thu, 2023-08-17 at 16:15 +0200, Michael Opdenacker via
lists.yoctoproject.org wrote:
> From: Michael Opdenacker <michael.opdenacker@bootlin.com>
> 
> [YOCTO #15182]
> 
> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> Reported-by: Yoann CONGAL <yoann.congal@smile.fr>
> Reported-by: Randy MacLeod <randy.macleod@windriver.com>
> Reported-by: Josef Holzmayr <jester@theyoctojester.info>
> 
> ---
> 
> - Changes in V2:
>   - Using a more universal "-mtime" option to find files to remove
>     as suggested by Richard Purdie. V1 was using "-atime" which
>     won't work with systems mounted with the "noatime" option used
>     to increase disk performance.
>   - Add more details about sstate-cache-management.sh provided by
>     Richard Purdie, perhaps not recommended but useful to document
>     as long as it is shipped in OE-core.
> ---
>  documentation/dev-manual/disk-space.rst | 31 ++++++++++++++++++-------
>  1 file changed, 22 insertions(+), 9 deletions(-)
> 
> diff --git a/documentation/dev-manual/disk-space.rst b/documentation/dev-manual/disk-space.rst
> index c63591cc7a..01cdf911c4 100644
> --- a/documentation/dev-manual/disk-space.rst
> +++ b/documentation/dev-manual/disk-space.rst
> @@ -27,19 +27,32 @@ Purging Duplicate Shared State Cache Files
>  ==========================================
>  
>  After multiple build iterations, the Shared State (sstate) cache can contain
> -duplicate cache files for a given package, while only the most recent one
> -is likely to be reusable. The following command purges all but the
> -newest sstate cache file for each package::
> +duplicate cache files for a given package, consuming a substantial amount of
> +disk space. However, only the most recent cache files are likeky to be reusable.
>  
> -   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
> +The following command is a quick way to purge all the cache files which
> +haven't been used for a least a specified number of days::
>  
> -This command will ask you to confirm the deletions it identifies.
> +   find build/sstate-cache -type f -mtime +$DAYS -delete
>  
> -.. note::
> +The above command relies on the fact that BitBake touches the sstate cache
> +files as it accesses them, when it has write access to the cache.

Add "You could use -atime instead if the partition isn't mounted with
the noatime option for a read only cache."

?

Otherwise looks good thanks.

Cheers,

Richard

[PATCH v3] dev-manual: disk-space: mention faster "find" command to trim sstate cache

[michael.opdenacker]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

[YOCTO #15182]

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Yoann CONGAL <yoann.congal@smile.fr>
Reported-by: Randy MacLeod <randy.macleod@windriver.com>
Reported-by: Josef Holzmayr <jester@theyoctojester.info>

---

- Changes in V3:
  - Mention the use of "-atime" if we don't have the cache on storage
    mounted with the "-noatime" option, as suggested by Richard Purdie.

- Changes in V2:
  - Using a more universal "-mtime" option to find files to remove
    as suggested by Richard Purdie. V1 was using "-atime" which
    won't work with systems mounted with the "noatime" option used
    to increase disk performance.
  - Add more details about sstate-cache-management.sh provided by
    Richard Purdie, perhaps not recommended but useful to document
    as long as it is shipped in OE-core.
---
 documentation/dev-manual/disk-space.rst | 34 ++++++++++++++++++-------
 1 file changed, 25 insertions(+), 9 deletions(-)

diff --git a/documentation/dev-manual/disk-space.rst b/documentation/dev-manual/disk-space.rst
index c63591cc7a..a84bef4511 100644
--- a/documentation/dev-manual/disk-space.rst
+++ b/documentation/dev-manual/disk-space.rst
@@ -27,19 +27,35 @@ Purging Duplicate Shared State Cache Files
 ==========================================
 
 After multiple build iterations, the Shared State (sstate) cache can contain
-duplicate cache files for a given package, while only the most recent one
-is likely to be reusable. The following command purges all but the
-newest sstate cache file for each package::
+duplicate cache files for a given package, consuming a substantial amount of
+disk space. However, only the most recent cache files are likeky to be reusable.
 
-   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+The following command is a quick way to purge all the cache files which
+haven't been used for a least a specified number of days::
 
-This command will ask you to confirm the deletions it identifies.
+   find build/sstate-cache -type f -mtime +$DAYS -delete
 
-.. note::
+The above command relies on the fact that BitBake touches the sstate cache
+files as it accesses them, when it has write access to the cache.
 
-   The duplicated sstate cache files of one package must have the same
-   architecture, which means that sstate cache files with multiple
-   architectures are not considered as duplicate.
+You could use ``-atime`` instead of ``-mtime`` if the partition isn't mounted
+with the ``noatime`` option for a read only cache.
 
+For more advanced needs, OpenEmbedded-Core also offers a more elaborate
+command. It has the ability to purge all but the newest cache files on each
+architecture, and also to remove files that it considers unreachable by
+exploring a set of build configurations. However, this command
+requires a full build environment to be available and doesn't work well
+covering multiple releases. It won't work either on limited environments
+such as BSD based NAS::
+
+   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+
+This command will ask you to confirm the deletions it identifies.
 Run ``sstate-cache-management.sh`` for more details about this script.
 
+.. note::
+
+   As this command is much more cautious and selective, removing only cache files,
+   it will execute much slower than the simple ``find`` command described above.
+   Therefore, it may not be your best option to trim huge cache directories.
-- 
2.34.1