[PATCH] gitlab-ci: Decrease the size of the compiler cache

[PATCH] gitlab-ci: Decrease the size of the compiler cache

[Thomas Huth]

From: Thomas Huth <thuth@redhat.com>

Uploading the cache from the runner takes a long time in the MSYS2
job, mostly due to the size of the compiler cache.
However, looking at runs with a non-initialized cache, and by doing
a "du -sh ." in the build directory, it seems like a build only
takes about 236 MiB of data, so the compiler cache with 500 MiB
certainly contains a lot of stale files. Thus decrease the size of
the ccache to a more reasonable value to speed up the MSYS2 job in
our CI (and add a "du -sh" at the end to have a reference for the
required cache size in the future).

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 Looking at the latest runs in the CI, our recent attempt to decrease
 the cache size by cleaning the pacman cache did not help much:
 https://gitlab.com/qemu-project/qemu/-/jobs/11747329283
 ... that run contains the "pacman -Sc" command, but the "Saving cache
 for successful job" step at the end still takes close to 20 minutes.
 So we likely have to shrink the compiler cache, too. In this run here:
 https://gitlab.com/thuth/qemu/-/jobs/11770708859#L1769
 I added a "du -sh" and you can see that the build directory only
 takes 236 MB there. So a ccache with the size of 250M should be
 sufficient for the MSYS2 job.

 .gitlab-ci.d/windows.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index 6e1135d8b86..e2fef543899 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -94,7 +94,7 @@ msys2-64bit:
   - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
   - $env:CCACHE_BASEDIR = "$env:CI_PROJECT_DIR"
   - $env:CCACHE_DIR = "$env:CCACHE_BASEDIR/ccache"
-  - $env:CCACHE_MAXSIZE = "500M"
+  - $env:CCACHE_MAXSIZE = "250M"
   - $env:CCACHE_DEPEND = 1 # cache misses are too expensive with preprocessor mode
   - $env:CC = "ccache gcc"
   - mkdir build
@@ -103,5 +103,6 @@ msys2-64bit:
   - ..\msys64\usr\bin\bash -lc "../configure $CONFIGURE_ARGS"
   - ..\msys64\usr\bin\bash -lc "make -j$env:JOBS"
   - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
+  - ..\msys64\usr\bin\bash -lc "du -sh ."
   - ..\msys64\usr\bin\bash -lc "ccache --show-stats"
   - Write-Output "Finished build at $(Get-Date -Format u)"
-- 
2.51.0

Re: [PATCH] gitlab-ci: Decrease the size of the compiler cache

[Daniel P. Berrangé]

On Mon, Oct 20, 2025 at 10:54:31AM +0200, Thomas Huth wrote:
> From: Thomas Huth <thuth@redhat.com>
> 
> Uploading the cache from the runner takes a long time in the MSYS2
> job, mostly due to the size of the compiler cache.
> However, looking at runs with a non-initialized cache, and by doing
> a "du -sh ." in the build directory, it seems like a build only
> takes about 236 MiB of data, so the compiler cache with 500 MiB
> certainly contains a lot of stale files. Thus decrease the size of
> the ccache to a more reasonable value to speed up the MSYS2 job in
> our CI (and add a "du -sh" at the end to have a reference for the
> required cache size in the future).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  Looking at the latest runs in the CI, our recent attempt to decrease
>  the cache size by cleaning the pacman cache did not help much:
>  https://gitlab.com/qemu-project/qemu/-/jobs/11747329283
>  ... that run contains the "pacman -Sc" command, but the "Saving cache
>  for successful job" step at the end still takes close to 20 minutes.
>  So we likely have to shrink the compiler cache, too. In this run here:
>  https://gitlab.com/thuth/qemu/-/jobs/11770708859#L1769
>  I added a "du -sh" and you can see that the build directory only
>  takes 236 MB there. So a ccache with the size of 250M should be
>  sufficient for the MSYS2 job.

FWIW, in my fork I see

Cacheable calls:   638 / 647 (98.61%)
  Hits:            629 / 638 (98.59%)
    Direct:        629 / 629 (100.0%)
    Preprocessed:    0 / 629 ( 0.00%)
  Misses:            9 / 638 ( 1.41%)
Uncacheable calls:   9 / 647 ( 1.39%)
Local storage:
  Cache size (GB): 0.1 / 0.5 (29.54%)
  Hits:            629 / 638 (98.59%)
  Misses:            9 / 638 ( 1.41%)

IOW, even ~160 MB is sufficient for 99% cache hit, so 250 MB is
about 2/3rds spare headroom.

If I look at the saving cache part in my fork job I see:

  msys64/var/cache: found 284 matching artifact files and directories 
  ccache: found 12825 matching artifact files and directories 

while in QEMU job I see

  msys64/var/cache: found 342 matching artifact files and directories 
  ccache: found 46881 matching artifact files and directories 

so the ccache usage is almost x4 bigger in terms of # of files, so
that does likely account for a good portion of the time.

I'm surprised the msys64/var/cache is still bigger than my fork though,
as I would have expected them to be basically the same.

I wonder if we shouldn't recursively list the msys64 cache as a debug
aid ?

>  .gitlab-ci.d/windows.yml | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 6e1135d8b86..e2fef543899 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -94,7 +94,7 @@ msys2-64bit:
>    - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
>    - $env:CCACHE_BASEDIR = "$env:CI_PROJECT_DIR"
>    - $env:CCACHE_DIR = "$env:CCACHE_BASEDIR/ccache"
> -  - $env:CCACHE_MAXSIZE = "500M"
> +  - $env:CCACHE_MAXSIZE = "250M"
>    - $env:CCACHE_DEPEND = 1 # cache misses are too expensive with preprocessor mode
>    - $env:CC = "ccache gcc"
>    - mkdir build
> @@ -103,5 +103,6 @@ msys2-64bit:
>    - ..\msys64\usr\bin\bash -lc "../configure $CONFIGURE_ARGS"
>    - ..\msys64\usr\bin\bash -lc "make -j$env:JOBS"
>    - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
> +  - ..\msys64\usr\bin\bash -lc "du -sh ."

Do we want to keep this in the final commit ?

We have ccache size printed, and we could do with msys64/var/cache
size being printed at least.

>    - ..\msys64\usr\bin\bash -lc "ccache --show-stats"
>    - Write-Output "Finished build at $(Get-Date -Format u)"

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] gitlab-ci: Decrease the size of the compiler cache

[Philippe Mathieu-Daudé]

On 20/10/25 10:54, Thomas Huth wrote:
> From: Thomas Huth <thuth@redhat.com>
> 
> Uploading the cache from the runner takes a long time in the MSYS2
> job, mostly due to the size of the compiler cache.
> However, looking at runs with a non-initialized cache, and by doing
> a "du -sh ." in the build directory, it seems like a build only
> takes about 236 MiB of data, so the compiler cache with 500 MiB
> certainly contains a lot of stale files. Thus decrease the size of
> the ccache to a more reasonable value to speed up the MSYS2 job in
> our CI (and add a "du -sh" at the end to have a reference for the
> required cache size in the future).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   Looking at the latest runs in the CI, our recent attempt to decrease
>   the cache size by cleaning the pacman cache did not help much:
>   https://gitlab.com/qemu-project/qemu/-/jobs/11747329283
>   ... that run contains the "pacman -Sc" command, but the "Saving cache
>   for successful job" step at the end still takes close to 20 minutes.
>   So we likely have to shrink the compiler cache, too. In this run here:
>   https://gitlab.com/thuth/qemu/-/jobs/11770708859#L1769
>   I added a "du -sh" and you can see that the build directory only
>   takes 236 MB there. So a ccache with the size of 250M should be
>   sufficient for the MSYS2 job.
> 
>   .gitlab-ci.d/windows.yml | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH] gitlab-ci: Decrease the size of the compiler cache

[Thomas Huth]

On 20/10/2025 11.20, Daniel P. Berrangé wrote:
> On Mon, Oct 20, 2025 at 10:54:31AM +0200, Thomas Huth wrote:
>> From: Thomas Huth <thuth@redhat.com>
>>
>> Uploading the cache from the runner takes a long time in the MSYS2
>> job, mostly due to the size of the compiler cache.
>> However, looking at runs with a non-initialized cache, and by doing
>> a "du -sh ." in the build directory, it seems like a build only
>> takes about 236 MiB of data, so the compiler cache with 500 MiB
>> certainly contains a lot of stale files. Thus decrease the size of
>> the ccache to a more reasonable value to speed up the MSYS2 job in
>> our CI (and add a "du -sh" at the end to have a reference for the
>> required cache size in the future).
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   Looking at the latest runs in the CI, our recent attempt to decrease
>>   the cache size by cleaning the pacman cache did not help much:
>>   https://gitlab.com/qemu-project/qemu/-/jobs/11747329283
>>   ... that run contains the "pacman -Sc" command, but the "Saving cache
>>   for successful job" step at the end still takes close to 20 minutes.
>>   So we likely have to shrink the compiler cache, too. In this run here:
>>   https://gitlab.com/thuth/qemu/-/jobs/11770708859#L1769
>>   I added a "du -sh" and you can see that the build directory only
>>   takes 236 MB there. So a ccache with the size of 250M should be
>>   sufficient for the MSYS2 job.
> 
> FWIW, in my fork I see
> 
> Cacheable calls:   638 / 647 (98.61%)
>    Hits:            629 / 638 (98.59%)
>      Direct:        629 / 629 (100.0%)
>      Preprocessed:    0 / 629 ( 0.00%)
>    Misses:            9 / 638 ( 1.41%)
> Uncacheable calls:   9 / 647 ( 1.39%)
> Local storage:
>    Cache size (GB): 0.1 / 0.5 (29.54%)
>    Hits:            629 / 638 (98.59%)
>    Misses:            9 / 638 ( 1.41%)
> 
> IOW, even ~160 MB is sufficient for 99% cache hit, so 250 MB is
> about 2/3rds spare headroom.

Good point. So 250 MB will likely still waste time with unnecessary uploads.

Some few MBs headroom likely still make sense in case we compile more code 
in the future, so maybe we should use something like 180 MB for the cache size?

> If I look at the saving cache part in my fork job I see:
> 
>    msys64/var/cache: found 284 matching artifact files and directories
>    ccache: found 12825 matching artifact files and directories
> 
> while in QEMU job I see
> 
>    msys64/var/cache: found 342 matching artifact files and directories
>    ccache: found 46881 matching artifact files and directories
> 
> so the ccache usage is almost x4 bigger in terms of # of files, so
> that does likely account for a good portion of the time.
> 
> I'm surprised the msys64/var/cache is still bigger than my fork though,
> as I would have expected them to be basically the same.

That's weird, indeed. Maybe some other stale files from the past?

> I wonder if we shouldn't recursively list the msys64 cache as a debug
> aid ?

I agree, we likely should at least temporarily add that to debug the issue. 
Do you want to send a patch, or want me to do it?

>>   .gitlab-ci.d/windows.yml | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
>> index 6e1135d8b86..e2fef543899 100644
>> --- a/.gitlab-ci.d/windows.yml
>> +++ b/.gitlab-ci.d/windows.yml
>> @@ -94,7 +94,7 @@ msys2-64bit:
>>     - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
>>     - $env:CCACHE_BASEDIR = "$env:CI_PROJECT_DIR"
>>     - $env:CCACHE_DIR = "$env:CCACHE_BASEDIR/ccache"
>> -  - $env:CCACHE_MAXSIZE = "500M"
>> +  - $env:CCACHE_MAXSIZE = "250M"
>>     - $env:CCACHE_DEPEND = 1 # cache misses are too expensive with preprocessor mode
>>     - $env:CC = "ccache gcc"
>>     - mkdir build
>> @@ -103,5 +103,6 @@ msys2-64bit:
>>     - ..\msys64\usr\bin\bash -lc "../configure $CONFIGURE_ARGS"
>>     - ..\msys64\usr\bin\bash -lc "make -j$env:JOBS"
>>     - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
>> +  - ..\msys64\usr\bin\bash -lc "du -sh ."
> 
> Do we want to keep this in the final commit ?
>> We have ccache size printed, and we could do with msys64/var/cache
> size being printed at least.

Well, the cache can collect stale garbage, as you can currently see in the 
CI jobs of the qemu-project, but the build folder should give you a very 
clear picture of how many MiBs of object code are really necessary, so I'd 
prefer to keep it (at least for a while).

  Thomas

Re: [PATCH] gitlab-ci: Decrease the size of the compiler cache

[Daniel P. Berrangé]

On Mon, Oct 20, 2025 at 03:18:33PM +0200, Thomas Huth wrote:
> On 20/10/2025 11.20, Daniel P. Berrangé wrote:
> > On Mon, Oct 20, 2025 at 10:54:31AM +0200, Thomas Huth wrote:
> > > From: Thomas Huth <thuth@redhat.com>
> > > 
> > > Uploading the cache from the runner takes a long time in the MSYS2
> > > job, mostly due to the size of the compiler cache.
> > > However, looking at runs with a non-initialized cache, and by doing
> > > a "du -sh ." in the build directory, it seems like a build only
> > > takes about 236 MiB of data, so the compiler cache with 500 MiB
> > > certainly contains a lot of stale files. Thus decrease the size of
> > > the ccache to a more reasonable value to speed up the MSYS2 job in
> > > our CI (and add a "du -sh" at the end to have a reference for the
> > > required cache size in the future).
> > > 
> > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > ---
> > >   Looking at the latest runs in the CI, our recent attempt to decrease
> > >   the cache size by cleaning the pacman cache did not help much:
> > >   https://gitlab.com/qemu-project/qemu/-/jobs/11747329283
> > >   ... that run contains the "pacman -Sc" command, but the "Saving cache
> > >   for successful job" step at the end still takes close to 20 minutes.
> > >   So we likely have to shrink the compiler cache, too. In this run here:
> > >   https://gitlab.com/thuth/qemu/-/jobs/11770708859#L1769
> > >   I added a "du -sh" and you can see that the build directory only
> > >   takes 236 MB there. So a ccache with the size of 250M should be
> > >   sufficient for the MSYS2 job.
> > 
> > FWIW, in my fork I see
> > 
> > Cacheable calls:   638 / 647 (98.61%)
> >    Hits:            629 / 638 (98.59%)
> >      Direct:        629 / 629 (100.0%)
> >      Preprocessed:    0 / 629 ( 0.00%)
> >    Misses:            9 / 638 ( 1.41%)
> > Uncacheable calls:   9 / 647 ( 1.39%)
> > Local storage:
> >    Cache size (GB): 0.1 / 0.5 (29.54%)
> >    Hits:            629 / 638 (98.59%)
> >    Misses:            9 / 638 ( 1.41%)
> > 
> > IOW, even ~160 MB is sufficient for 99% cache hit, so 250 MB is
> > about 2/3rds spare headroom.
> 
> Good point. So 250 MB will likely still waste time with unnecessary uploads.
> 
> Some few MBs headroom likely still make sense in case we compile more code
> in the future, so maybe we should use something like 180 MB for the cache
> size?

Yeah, 180 MB, and we can look at cache hit rate every now & then
to see if we're exceeding it.

> > I wonder if we shouldn't recursively list the msys64 cache as a debug
> > aid ?
> 
> I agree, we likely should at least temporarily add that to debug the issue.
> Do you want to send a patch, or want me to do it?

I'll let you.

> > >   .gitlab-ci.d/windows.yml | 3 ++-
> > >   1 file changed, 2 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> > > index 6e1135d8b86..e2fef543899 100644
> > > --- a/.gitlab-ci.d/windows.yml
> > > +++ b/.gitlab-ci.d/windows.yml
> > > @@ -94,7 +94,7 @@ msys2-64bit:
> > >     - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
> > >     - $env:CCACHE_BASEDIR = "$env:CI_PROJECT_DIR"
> > >     - $env:CCACHE_DIR = "$env:CCACHE_BASEDIR/ccache"
> > > -  - $env:CCACHE_MAXSIZE = "500M"
> > > +  - $env:CCACHE_MAXSIZE = "250M"
> > >     - $env:CCACHE_DEPEND = 1 # cache misses are too expensive with preprocessor mode
> > >     - $env:CC = "ccache gcc"
> > >     - mkdir build
> > > @@ -103,5 +103,6 @@ msys2-64bit:
> > >     - ..\msys64\usr\bin\bash -lc "../configure $CONFIGURE_ARGS"
> > >     - ..\msys64\usr\bin\bash -lc "make -j$env:JOBS"
> > >     - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
> > > +  - ..\msys64\usr\bin\bash -lc "du -sh ."
> > 
> > Do we want to keep this in the final commit ?
> > > We have ccache size printed, and we could do with msys64/var/cache
> > size being printed at least.
> 
> Well, the cache can collect stale garbage, as you can currently see in the
> CI jobs of the qemu-project, but the build folder should give you a very
> clear picture of how many MiBs of object code are really necessary, so I'd
> prefer to keep it (at least for a while).

Ok

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|