Re: [PATCH] gitlab-ci: Replace Docker with Kaniko

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, May 16, 2024 at 07:24:04PM +0100, Daniel P. Berrangé wrote:
> On Thu, May 16, 2024 at 05:52:43PM +0100, Camilla Conte wrote:
> > Enables caching from the qemu-project repository.
> > 
> > Uses a dedicated "$NAME-cache" tag for caching, to address limitations.
> > See issue "when using --cache=true, kaniko fail to push cache layer [...]":
> > https://github.com/GoogleContainerTools/kaniko/issues/1459
> 
> After investigating, this is a result of a different design approach
> for caching in kaniko.
> 
> In docker, it can leverage any existing image as a cache source,
> reusing individual layers that were present. IOW, there's no
> difference between a cache and a final image, they're one and the
> same thing
> 
> In kaniko, the cache is a distinct object type. IIUC, it is not
> populated with the individual layers, instead it has a custom
> format for storing the cached content. Therefore the concept of
> storing the cache at the same location as the final image, is
> completely inappropriate - you can't store two completely different
> kinds of content at the same place.
> 
> That is also why you can't just "git pull" the fetch the cache
> image(s) beforehand, and also why it doesn't look like you can
> use multiple cache sources with kaniko.
> 
> None of this is inherantly a bad thing..... except when it comes
> to data storage. By using Kaniko we would, at minimum, doubling
> the amount of data storage we consume in the gitlab registry.

Double is actually just the initial case. The cache is storing layers
using docker tags, whose name appears based on a hash of the "RUN"
command.

IOW, the first time we build a container we have double the usage.
When a dockerfile is updated changing a 'RUN' command, we now have
triple the storage usage for cache. Update the RUN command again,
and we now have quadruple the storage. etc.

Kaniko does not appear to purge cache entries itself, and will rely
on something else to do the cache purging.

GitLab has support for purging old docker tags, but I'm not an
admin on the QEMU project namespace, so can't tell if it can be
enabled or not ? Many older projects have this permanently disabled
due to historical compat issues in gitlab after they introduced the
feature.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|