[RFC PATCH v2 0/3] ci: Speed up container stage

[Fabiano Rosas <farosas@suse.de>]

Turns out we were already pulling the image, but for forks we were
pulling from a different registry and for the main project we had
caching disabled due to a bug with docker/gitlab.

I've now replaced the usage of docker.py with a plain docker command
and reintroduced the caching.

I also removed the COMMON_TAG which was unused.

v1:
https://lore.kernel.org/r/20230223142154.31975-1-farosas@suse.de

I'm not sure if this was discussed previously, but I noticed we're not
pulling the images we push to the registry at every pipeline run.

I would expect we don't actually need to rebuild container images at
_every_ pipeline run, so I propose we add a "docker pull" to the
container templates. We already have that for the docker-edk2|opensbi
images.

Some containers can take a long time to build (14 mins) and pulling
the image first without building can cut the time to about 3
mins. With this we can save almost 2h of cumulative CI time per
pipeline run:

| master   | pull-only |  diff    | container
| 0:02:34  | 0:02:09   | 00:00:25 |  alpha-debian-cross-container
| 0:04:45  | 0:02:40   | 00:02:05 |  amd64-alpine-container
| 0:09:51  | 0:02:56   | 00:06:55 |  amd64-centos8-container
| 0:07:21  | 0:02:49   | 00:04:32 |  amd64-debian-container
| 0:06:00  | 0:02:37   | 00:03:23 |  amd64-debian-cross-container
| 0:14:22  | 0:03:41   | 00:10:41 |  amd64-debian-user-cross-container
| 0:10:14  | 0:03:24   | 00:06:50 |  amd64-fedora-container
| 0:12:09  | 0:02:49   | 00:09:20 |  amd64-opensuse-leap-container
| 0:07:33  | 0:02:45   | 00:04:48 |  amd64-ubuntu2004-container
| 0:08:28  | 0:03:07   | 00:05:21 |  arm64-debian-cross-container
| 0:04:27  | 0:02:58   | 00:01:29 |  armel-debian-cross-container
| 0:08:01  | 0:02:55   | 00:05:06 |  armhf-debian-cross-container
| 0:03:33  | 0:02:18   | 00:01:15 |  cris-fedora-cross-container
| 0:00:28  | 0:00:28   | 00:00:00 |  docker-edk2
| 0:00:25  | 0:00:28   |-00:00:03 |  docker-opensbi
| 0:08:34  | 0:03:10   | 00:05:24 |  hexagon-cross-container
| 0:02:34  | 0:02:08   | 00:00:26 |  hppa-debian-cross-container
| 0:04:50  | 0:02:28   | 00:02:22 |  i386-fedora-cross-container
| 0:02:36  | 0:02:12   | 00:00:24 |  m68k-debian-cross-container
| 0:02:40  | 0:02:09   | 00:00:31 |  mips-debian-cross-container
| 0:02:38  | 0:02:09   | 00:00:29 |  mips64-debian-cross-container
| 0:04:28  | 0:02:48   | 00:01:40 |  mips64el-debian-cross-container
| 0:07:07  | 0:02:46   | 00:04:21 |  mipsel-debian-cross-container
| 0:03:51  | 0:02:21   | 00:01:30 |  powerpc-test-cross-container
| 0:08:52  | 0:03:00   | 00:05:52 |  ppc64el-debian-cross-container
| 0:06:07  | 0:02:49   | 00:03:18 |  python-container
| 0:04:37  | 0:02:26   | 00:02:11 |  riscv64-debian-cross-container
| 0:02:39  | 0:02:08   | 00:00:31 |  riscv64-debian-test-cross-container
| 0:08:03  | 0:03:00   | 00:05:03 |  s390x-debian-cross-container
| 0:02:34  | 0:02:08   | 00:00:26 |  sh4-debian-cross-container
| 0:02:37  | 0:02:09   | 00:00:28 |  sparc64-debian-cross-container
| 0:04:25  | 0:02:17   | 00:02:08 |  tricore-debian-cross-container
| 0:12:51  | 0:03:27   | 00:09:24 |  win32-fedora-cross-container
| 0:11:16  | 0:03:29   | 00:07:47 |  win64-fedora-cross-container
| 0:03:28  | 0:02:20   | 00:01:08 |  xtensa-debian-cross-container
                       | 01:57:30 |

We would need to devise a mechanism (not included here) to force the
re-build of the container images when needed, perhaps an environment
variable or even a whole new "container build" stage before the
"container" stage.

What do you think?

Fabiano Rosas (3):
  gitlab: Use plain docker in container-template.yml
  gitlab: Cache container images
  gitlab: Remove COMMON_TAG from container-template.yml

 .gitlab-ci.d/container-template.yml | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

-- 
2.35.3