Gitlab CI caching is not working

Gitlab CI caching is not working

[Thomas Huth]

  Hi!

While looking at some recent CI jobs, I noticed that the caching of the 
Gitlab-CI jobs is not working at all anymore. In the build jobs, the ccache 
saving is not working and causing a complete cache miss of each compile:

  https://gitlab.com/qemu-project/qemu/-/jobs/7802183187#L5328

And, maybe more important, in the avocado/functional jobs we don't cache the 
assets anymore, causing a re-download of multiple gigabytes each time:

  https://gitlab.com/qemu-project/qemu/-/jobs/7802183251#L29

(the du -chs in line 35 is not executed, thus the cache is nonexistent)

The problem is not new, it's been there for some weeks already, e.g. here's 
a run from the last freeze period (when the job was only running avocado tests):

  https://gitlab.com/qemu-project/qemu/-/jobs/7753544153#L86

There is a suspicious message at the beginning of the logs:

  "No URL provided, cache will not be downloaded from shared cache server. 
Instead a local version of cache will be extracted."

... but since we use throw-away containers for building, I guess there is no 
local version of the cache?

Anyway, the problem only exists for the k8s runners, in my private clone of 
the repository that uses shared runners from gitlab, the caching is working 
right.

Could somebody please have a look into this? Fixing the caching might speed 
up our build time quite a bit, I think.

  Thomas

Re: Gitlab CI caching is not working

[Daniel P. Berrangé]

On Thu, Sep 12, 2024 at 07:38:37AM +0200, Thomas Huth wrote:
> 
>  Hi!
> 
> While looking at some recent CI jobs, I noticed that the caching of the
> Gitlab-CI jobs is not working at all anymore. In the build jobs, the ccache
> saving is not working and causing a complete cache miss of each compile:
> 
>  https://gitlab.com/qemu-project/qemu/-/jobs/7802183187#L5328
> 
> And, maybe more important, in the avocado/functional jobs we don't cache the
> assets anymore, causing a re-download of multiple gigabytes each time:
> 
>  https://gitlab.com/qemu-project/qemu/-/jobs/7802183251#L29
> 
> (the du -chs in line 35 is not executed, thus the cache is nonexistent)
> 
> The problem is not new, it's been there for some weeks already, e.g. here's
> a run from the last freeze period (when the job was only running avocado
> tests):
> 
>  https://gitlab.com/qemu-project/qemu/-/jobs/7753544153#L86
> 
> There is a suspicious message at the beginning of the logs:
> 
>  "No URL provided, cache will not be downloaded from shared cache server.
> Instead a local version of cache will be extracted."
> 
> ... but since we use throw-away containers for building, I guess there is no
> local version of the cache?
> 
> Anyway, the problem only exists for the k8s runners, in my private clone of
> the repository that uses shared runners from gitlab, the caching is working
> right.

Right, by default caches are local to the runner's execution environment.
If you want a distributed cache, that has to be turned on in the runner
configuration:

  https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|