Re: [PATCH] gitlab-ci: Decrease the size of the compiler cache

["Daniel P. Berrangé" <berrange@redhat.com>]

On Mon, Oct 20, 2025 at 10:54:31AM +0200, Thomas Huth wrote:
> From: Thomas Huth <thuth@redhat.com>
> 
> Uploading the cache from the runner takes a long time in the MSYS2
> job, mostly due to the size of the compiler cache.
> However, looking at runs with a non-initialized cache, and by doing
> a "du -sh ." in the build directory, it seems like a build only
> takes about 236 MiB of data, so the compiler cache with 500 MiB
> certainly contains a lot of stale files. Thus decrease the size of
> the ccache to a more reasonable value to speed up the MSYS2 job in
> our CI (and add a "du -sh" at the end to have a reference for the
> required cache size in the future).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  Looking at the latest runs in the CI, our recent attempt to decrease
>  the cache size by cleaning the pacman cache did not help much:
>  https://gitlab.com/qemu-project/qemu/-/jobs/11747329283
>  ... that run contains the "pacman -Sc" command, but the "Saving cache
>  for successful job" step at the end still takes close to 20 minutes.
>  So we likely have to shrink the compiler cache, too. In this run here:
>  https://gitlab.com/thuth/qemu/-/jobs/11770708859#L1769
>  I added a "du -sh" and you can see that the build directory only
>  takes 236 MB there. So a ccache with the size of 250M should be
>  sufficient for the MSYS2 job.

FWIW, in my fork I see

Cacheable calls:   638 / 647 (98.61%)
  Hits:            629 / 638 (98.59%)
    Direct:        629 / 629 (100.0%)
    Preprocessed:    0 / 629 ( 0.00%)
  Misses:            9 / 638 ( 1.41%)
Uncacheable calls:   9 / 647 ( 1.39%)
Local storage:
  Cache size (GB): 0.1 / 0.5 (29.54%)
  Hits:            629 / 638 (98.59%)
  Misses:            9 / 638 ( 1.41%)

IOW, even ~160 MB is sufficient for 99% cache hit, so 250 MB is
about 2/3rds spare headroom.

If I look at the saving cache part in my fork job I see:

  msys64/var/cache: found 284 matching artifact files and directories 
  ccache: found 12825 matching artifact files and directories 

while in QEMU job I see

  msys64/var/cache: found 342 matching artifact files and directories 
  ccache: found 46881 matching artifact files and directories 

so the ccache usage is almost x4 bigger in terms of # of files, so
that does likely account for a good portion of the time.

I'm surprised the msys64/var/cache is still bigger than my fork though,
as I would have expected them to be basically the same.

I wonder if we shouldn't recursively list the msys64 cache as a debug
aid ?

>  .gitlab-ci.d/windows.yml | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 6e1135d8b86..e2fef543899 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -94,7 +94,7 @@ msys2-64bit:
>    - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
>    - $env:CCACHE_BASEDIR = "$env:CI_PROJECT_DIR"
>    - $env:CCACHE_DIR = "$env:CCACHE_BASEDIR/ccache"
> -  - $env:CCACHE_MAXSIZE = "500M"
> +  - $env:CCACHE_MAXSIZE = "250M"
>    - $env:CCACHE_DEPEND = 1 # cache misses are too expensive with preprocessor mode
>    - $env:CC = "ccache gcc"
>    - mkdir build
> @@ -103,5 +103,6 @@ msys2-64bit:
>    - ..\msys64\usr\bin\bash -lc "../configure $CONFIGURE_ARGS"
>    - ..\msys64\usr\bin\bash -lc "make -j$env:JOBS"
>    - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
> +  - ..\msys64\usr\bin\bash -lc "du -sh ."

Do we want to keep this in the final commit ?

We have ccache size printed, and we could do with msys64/var/cache
size being printed at least.

>    - ..\msys64\usr\bin\bash -lc "ccache --show-stats"
>    - Write-Output "Finished build at $(Get-Date -Format u)"

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|