Re: [PATCH] gitlab-ci: Decrease the size of the compiler cache

["Daniel P. Berrangé" <berrange@redhat.com>]

On Mon, Oct 20, 2025 at 03:18:33PM +0200, Thomas Huth wrote:
> On 20/10/2025 11.20, Daniel P. Berrangé wrote:
> > On Mon, Oct 20, 2025 at 10:54:31AM +0200, Thomas Huth wrote:
> > > From: Thomas Huth <thuth@redhat.com>
> > > 
> > > Uploading the cache from the runner takes a long time in the MSYS2
> > > job, mostly due to the size of the compiler cache.
> > > However, looking at runs with a non-initialized cache, and by doing
> > > a "du -sh ." in the build directory, it seems like a build only
> > > takes about 236 MiB of data, so the compiler cache with 500 MiB
> > > certainly contains a lot of stale files. Thus decrease the size of
> > > the ccache to a more reasonable value to speed up the MSYS2 job in
> > > our CI (and add a "du -sh" at the end to have a reference for the
> > > required cache size in the future).
> > > 
> > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > ---
> > >   Looking at the latest runs in the CI, our recent attempt to decrease
> > >   the cache size by cleaning the pacman cache did not help much:
> > >   https://gitlab.com/qemu-project/qemu/-/jobs/11747329283
> > >   ... that run contains the "pacman -Sc" command, but the "Saving cache
> > >   for successful job" step at the end still takes close to 20 minutes.
> > >   So we likely have to shrink the compiler cache, too. In this run here:
> > >   https://gitlab.com/thuth/qemu/-/jobs/11770708859#L1769
> > >   I added a "du -sh" and you can see that the build directory only
> > >   takes 236 MB there. So a ccache with the size of 250M should be
> > >   sufficient for the MSYS2 job.
> > 
> > FWIW, in my fork I see
> > 
> > Cacheable calls:   638 / 647 (98.61%)
> >    Hits:            629 / 638 (98.59%)
> >      Direct:        629 / 629 (100.0%)
> >      Preprocessed:    0 / 629 ( 0.00%)
> >    Misses:            9 / 638 ( 1.41%)
> > Uncacheable calls:   9 / 647 ( 1.39%)
> > Local storage:
> >    Cache size (GB): 0.1 / 0.5 (29.54%)
> >    Hits:            629 / 638 (98.59%)
> >    Misses:            9 / 638 ( 1.41%)
> > 
> > IOW, even ~160 MB is sufficient for 99% cache hit, so 250 MB is
> > about 2/3rds spare headroom.
> 
> Good point. So 250 MB will likely still waste time with unnecessary uploads.
> 
> Some few MBs headroom likely still make sense in case we compile more code
> in the future, so maybe we should use something like 180 MB for the cache
> size?

Yeah, 180 MB, and we can look at cache hit rate every now & then
to see if we're exceeding it.

> > I wonder if we shouldn't recursively list the msys64 cache as a debug
> > aid ?
> 
> I agree, we likely should at least temporarily add that to debug the issue.
> Do you want to send a patch, or want me to do it?

I'll let you.

> > >   .gitlab-ci.d/windows.yml | 3 ++-
> > >   1 file changed, 2 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> > > index 6e1135d8b86..e2fef543899 100644
> > > --- a/.gitlab-ci.d/windows.yml
> > > +++ b/.gitlab-ci.d/windows.yml
> > > @@ -94,7 +94,7 @@ msys2-64bit:
> > >     - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
> > >     - $env:CCACHE_BASEDIR = "$env:CI_PROJECT_DIR"
> > >     - $env:CCACHE_DIR = "$env:CCACHE_BASEDIR/ccache"
> > > -  - $env:CCACHE_MAXSIZE = "500M"
> > > +  - $env:CCACHE_MAXSIZE = "250M"
> > >     - $env:CCACHE_DEPEND = 1 # cache misses are too expensive with preprocessor mode
> > >     - $env:CC = "ccache gcc"
> > >     - mkdir build
> > > @@ -103,5 +103,6 @@ msys2-64bit:
> > >     - ..\msys64\usr\bin\bash -lc "../configure $CONFIGURE_ARGS"
> > >     - ..\msys64\usr\bin\bash -lc "make -j$env:JOBS"
> > >     - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
> > > +  - ..\msys64\usr\bin\bash -lc "du -sh ."
> > 
> > Do we want to keep this in the final commit ?
> > > We have ccache size printed, and we could do with msys64/var/cache
> > size being printed at least.
> 
> Well, the cache can collect stale garbage, as you can currently see in the
> CI jobs of the qemu-project, but the build folder should give you a very
> clear picture of how many MiBs of object code are really necessary, so I'd
> prefer to keep it (at least for a while).

Ok

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|