* [RFC PATCH v3 0/1] ci: Speed up container stage
@ 2023-02-27 15:11 Fabiano Rosas
2023-02-27 15:11 ` [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml Fabiano Rosas
2023-02-28 10:29 ` [RFC PATCH v3 0/1] ci: Speed up container stage Alex Bennée
0 siblings, 2 replies; 10+ messages in thread
From: Fabiano Rosas @ 2023-02-27 15:11 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Alex Bennée, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Beraldo Leal,
Daniel P . Berrangé
- squashed the two patches (Daniel)
- dropped the "docker pull" command. The --cache-from option already
fetches the cached layers. Per docker's documentation:
"Upon importing the cache, the builder will only pull the JSON
metadata from the registry and determine possible cache hits based
on that information. If there is a cache hit, the matched layers
are pulled into the local environment."
https://docs.docker.com/engine/reference/commandline/build/#cache-from
Thanks
v2:
https://lore.kernel.org/r/20230224125207.19616-1-farosas@suse.de
Turns out we were already pulling the image, but for forks we were
pulling from a different registry and for the main project we had
caching disabled due to a bug with docker/gitlab.
I've now replaced the usage of docker.py with a plain docker command
and reintroduced the caching.
I also removed the COMMON_TAG which was unused.
v1:
https://lore.kernel.org/r/20230223142154.31975-1-farosas@suse.de
I'm not sure if this was discussed previously, but I noticed we're not
pulling the images we push to the registry at every pipeline run.
I would expect we don't actually need to rebuild container images at
_every_ pipeline run, so I propose we add a "docker pull" to the
container templates. We already have that for the docker-edk2|opensbi
images.
Some containers can take a long time to build (14 mins) and pulling
the image first without building can cut the time to about 3
mins. With this we can save almost 2h of cumulative CI time per
pipeline run:
| master | pull-only | diff | container
| 0:02:34 | 0:02:09 | 00:00:25 | alpha-debian-cross-container
| 0:04:45 | 0:02:40 | 00:02:05 | amd64-alpine-container
| 0:09:51 | 0:02:56 | 00:06:55 | amd64-centos8-container
| 0:07:21 | 0:02:49 | 00:04:32 | amd64-debian-container
| 0:06:00 | 0:02:37 | 00:03:23 | amd64-debian-cross-container
| 0:14:22 | 0:03:41 | 00:10:41 | amd64-debian-user-cross-container
| 0:10:14 | 0:03:24 | 00:06:50 | amd64-fedora-container
| 0:12:09 | 0:02:49 | 00:09:20 | amd64-opensuse-leap-container
| 0:07:33 | 0:02:45 | 00:04:48 | amd64-ubuntu2004-container
| 0:08:28 | 0:03:07 | 00:05:21 | arm64-debian-cross-container
| 0:04:27 | 0:02:58 | 00:01:29 | armel-debian-cross-container
| 0:08:01 | 0:02:55 | 00:05:06 | armhf-debian-cross-container
| 0:03:33 | 0:02:18 | 00:01:15 | cris-fedora-cross-container
| 0:00:28 | 0:00:28 | 00:00:00 | docker-edk2
| 0:00:25 | 0:00:28 |-00:00:03 | docker-opensbi
| 0:08:34 | 0:03:10 | 00:05:24 | hexagon-cross-container
| 0:02:34 | 0:02:08 | 00:00:26 | hppa-debian-cross-container
| 0:04:50 | 0:02:28 | 00:02:22 | i386-fedora-cross-container
| 0:02:36 | 0:02:12 | 00:00:24 | m68k-debian-cross-container
| 0:02:40 | 0:02:09 | 00:00:31 | mips-debian-cross-container
| 0:02:38 | 0:02:09 | 00:00:29 | mips64-debian-cross-container
| 0:04:28 | 0:02:48 | 00:01:40 | mips64el-debian-cross-container
| 0:07:07 | 0:02:46 | 00:04:21 | mipsel-debian-cross-container
| 0:03:51 | 0:02:21 | 00:01:30 | powerpc-test-cross-container
| 0:08:52 | 0:03:00 | 00:05:52 | ppc64el-debian-cross-container
| 0:06:07 | 0:02:49 | 00:03:18 | python-container
| 0:04:37 | 0:02:26 | 00:02:11 | riscv64-debian-cross-container
| 0:02:39 | 0:02:08 | 00:00:31 | riscv64-debian-test-cross-container
| 0:08:03 | 0:03:00 | 00:05:03 | s390x-debian-cross-container
| 0:02:34 | 0:02:08 | 00:00:26 | sh4-debian-cross-container
| 0:02:37 | 0:02:09 | 00:00:28 | sparc64-debian-cross-container
| 0:04:25 | 0:02:17 | 00:02:08 | tricore-debian-cross-container
| 0:12:51 | 0:03:27 | 00:09:24 | win32-fedora-cross-container
| 0:11:16 | 0:03:29 | 00:07:47 | win64-fedora-cross-container
| 0:03:28 | 0:02:20 | 00:01:08 | xtensa-debian-cross-container
| 01:57:30 |
We would need to devise a mechanism (not included here) to force the
re-build of the container images when needed, perhaps an environment
variable or even a whole new "container build" stage before the
"container" stage.
What do you think?
Fabiano Rosas (1):
gitlab: Use plain docker in container-template.yml
.gitlab-ci.d/container-template.yml | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
--
2.35.3
^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml
2023-02-27 15:11 [RFC PATCH v3 0/1] ci: Speed up container stage Fabiano Rosas
@ 2023-02-27 15:11 ` Fabiano Rosas
2023-02-27 19:00 ` Alex Bennée
2023-02-28 10:14 ` Daniel P. Berrangé
2023-02-28 10:29 ` [RFC PATCH v3 0/1] ci: Speed up container stage Alex Bennée
1 sibling, 2 replies; 10+ messages in thread
From: Fabiano Rosas @ 2023-02-27 15:11 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Alex Bennée, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Beraldo Leal,
Daniel P . Berrangé
Our dockerfiles no longer reference layers from other qemu images so
we can now use 'docker build' on them.
Also reinstate the caching that was disabled due to bad interactions
with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use
BUILDKIT in GitLab either"). We now believe those issues to be fixed.
The COMMON_TAG needed to be fixed for the caching to work. The
docker.py script was not using the variable, but constructing the
correct URL directly.
Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
.gitlab-ci.d/container-template.yml | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index c434b9c8f3..519b8a9482 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -6,17 +6,16 @@
- docker:dind
before_script:
- export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
- - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
+ - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
- apk add python3
- docker info
- docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
script:
- echo "TAG:$TAG"
- echo "COMMON_TAG:$COMMON_TAG"
- - ./tests/docker/docker.py --engine docker build
- -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
- -r $CI_REGISTRY/qemu-project/qemu
- - docker tag "qemu/$NAME" "$TAG"
+ - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
+ --build-arg BUILDKIT_INLINE_CACHE=1
+ -f "tests/docker/dockerfiles/$NAME.docker" "."
- docker push "$TAG"
after_script:
- docker logout
--
2.35.3
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml
2023-02-27 15:11 ` [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml Fabiano Rosas
@ 2023-02-27 19:00 ` Alex Bennée
2023-02-27 19:39 ` Alex Bennée
2023-02-27 21:12 ` Fabiano Rosas
2023-02-28 10:14 ` Daniel P. Berrangé
1 sibling, 2 replies; 10+ messages in thread
From: Alex Bennée @ 2023-02-27 19:00 UTC (permalink / raw)
To: Fabiano Rosas
Cc: qemu-devel, Thomas Huth, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Beraldo Leal,
Daniel P . Berrangé
Fabiano Rosas <farosas@suse.de> writes:
> Our dockerfiles no longer reference layers from other qemu images so
> we can now use 'docker build' on them.
>
> Also reinstate the caching that was disabled due to bad interactions
> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use
> BUILDKIT in GitLab either"). We now believe those issues to be fixed.
>
> The COMMON_TAG needed to be fixed for the caching to work. The
> docker.py script was not using the variable, but constructing the
> correct URL directly.
>
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
> .gitlab-ci.d/container-template.yml | 9 ++++-----
> 1 file changed, 4 insertions(+), 5 deletions(-)
>
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index c434b9c8f3..519b8a9482 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -6,17 +6,16 @@
> - docker:dind
> before_script:
> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
> - apk add python3
> - docker info
> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> script:
> - echo "TAG:$TAG"
> - echo "COMMON_TAG:$COMMON_TAG"
> - - ./tests/docker/docker.py --engine docker build
> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
> - -r $CI_REGISTRY/qemu-project/qemu
> - - docker tag "qemu/$NAME" "$TAG"
> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
> + --build-arg BUILDKIT_INLINE_CACHE=1
> + -f "tests/docker/dockerfiles/$NAME.docker" "."
I wonder why this doesn't injest a bunch of context. If I run:
docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t qemu
/debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" .
it attempts to bring my entire build directory in as build context. This
is why we use the - < docker form in the Makefile.
> - docker push "$TAG"
> after_script:
> - docker logout
So what I don't understand is if I do:
docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross
docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg
BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - <
/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker
I still see pretty much a full rebuild of the image.
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml
2023-02-27 19:00 ` Alex Bennée
@ 2023-02-27 19:39 ` Alex Bennée
2023-02-27 21:12 ` Fabiano Rosas
1 sibling, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2023-02-27 19:39 UTC (permalink / raw)
To: Alex Bennée
Cc: Fabiano Rosas, qemu-devel, Thomas Huth,
Philippe Mathieu-Daudé, Wainer dos Santos Moschetta,
Beraldo Leal, Daniel P . Berrangé
Alex Bennée <alex.bennee@linaro.org> writes:
> Fabiano Rosas <farosas@suse.de> writes:
>
>> Our dockerfiles no longer reference layers from other qemu images so
>> we can now use 'docker build' on them.
>>
>> Also reinstate the caching that was disabled due to bad interactions
>> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use
>> BUILDKIT in GitLab either"). We now believe those issues to be fixed.
>>
>> The COMMON_TAG needed to be fixed for the caching to work. The
>> docker.py script was not using the variable, but constructing the
>> correct URL directly.
>>
>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>> ---
>> .gitlab-ci.d/container-template.yml | 9 ++++-----
>> 1 file changed, 4 insertions(+), 5 deletions(-)
>>
>> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
>> index c434b9c8f3..519b8a9482 100644
>> --- a/.gitlab-ci.d/container-template.yml
>> +++ b/.gitlab-ci.d/container-template.yml
>> @@ -6,17 +6,16 @@
>> - docker:dind
>> before_script:
>> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
>> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
>> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
>> - apk add python3
>> - docker info
>> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>> script:
>> - echo "TAG:$TAG"
>> - echo "COMMON_TAG:$COMMON_TAG"
>> - - ./tests/docker/docker.py --engine docker build
>> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
>> - -r $CI_REGISTRY/qemu-project/qemu
>> - - docker tag "qemu/$NAME" "$TAG"
>> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
>> + --build-arg BUILDKIT_INLINE_CACHE=1
>> + -f "tests/docker/dockerfiles/$NAME.docker" "."
>
> I wonder why this doesn't injest a bunch of context. If I run:
>
> docker build --cache-from
> registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg
> BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t
> qemu
> /debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" .
>
> it attempts to bring my entire build directory in as build context. This
> is why we use the - < docker form in the Makefile.
>
>> - docker push "$TAG"
>> after_script:
>> - docker logout
>
> So what I don't understand is if I do:
>
> docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross
> docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg
> BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - <
> /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker
>
> I still see pretty much a full rebuild of the image.
For reference at the end:
🕙19:37:36 alex@zen:container-diff.git on master took 19s
➜ ./out/container-diff diff daemon://registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross daemon://qemu/debian-alpha-cross --type=file
-----File-----
These entries have been added to qemu/debian-alpha-cross:
FILE SIZE
/usr/share/doc/base-files/FAQ 6B
/usr/share/info/dir 781B
These entries have been deleted from qemu/debian-alpha-cross: None
These entries have been changed between registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross and qemu/debian-alpha-cross:
FILE SIZE1 SIZE2
/var/cache/debconf/templates.dat 762.2K 764.1K
/var/cache/debconf/templates.dat-old 762.2K 764.1K
/var/lib/dpkg/available 70.7K 70.7K
/var/log/dpkg.log 12.9K 20.2K
/var/log/apt/term.log 7.4K 15.5K
/var/cache/ldconfig/aux-cache 5.5K 5.5K
/var/cache/debconf/config.dat-old 4.5K 4.7K
/var/log/apt/history.log 1.5K 2K
/etc/shadow 501B 501B
/etc/shadow- 501B 501B
/etc/apt/sources.list 430B 430B
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml
2023-02-27 19:00 ` Alex Bennée
2023-02-27 19:39 ` Alex Bennée
@ 2023-02-27 21:12 ` Fabiano Rosas
2023-02-27 21:18 ` Fabiano Rosas
` (2 more replies)
1 sibling, 3 replies; 10+ messages in thread
From: Fabiano Rosas @ 2023-02-27 21:12 UTC (permalink / raw)
To: Alex Bennée
Cc: qemu-devel, Thomas Huth, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Beraldo Leal,
Daniel P . Berrangé
Alex Bennée <alex.bennee@linaro.org> writes:
> Fabiano Rosas <farosas@suse.de> writes:
>
>> Our dockerfiles no longer reference layers from other qemu images so
>> we can now use 'docker build' on them.
>>
>> Also reinstate the caching that was disabled due to bad interactions
>> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use
>> BUILDKIT in GitLab either"). We now believe those issues to be fixed.
>>
>> The COMMON_TAG needed to be fixed for the caching to work. The
>> docker.py script was not using the variable, but constructing the
>> correct URL directly.
>>
>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>> ---
>> .gitlab-ci.d/container-template.yml | 9 ++++-----
>> 1 file changed, 4 insertions(+), 5 deletions(-)
>>
>> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
>> index c434b9c8f3..519b8a9482 100644
>> --- a/.gitlab-ci.d/container-template.yml
>> +++ b/.gitlab-ci.d/container-template.yml
>> @@ -6,17 +6,16 @@
>> - docker:dind
>> before_script:
>> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
>> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
>> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
>> - apk add python3
>> - docker info
>> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>> script:
>> - echo "TAG:$TAG"
>> - echo "COMMON_TAG:$COMMON_TAG"
>> - - ./tests/docker/docker.py --engine docker build
>> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
>> - -r $CI_REGISTRY/qemu-project/qemu
>> - - docker tag "qemu/$NAME" "$TAG"
>> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
>> + --build-arg BUILDKIT_INLINE_CACHE=1
>> + -f "tests/docker/dockerfiles/$NAME.docker" "."
>
> I wonder why this doesn't injest a bunch of context. If I run:
>
> docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t qemu
> /debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" .
>
> it attempts to bring my entire build directory in as build context. This
> is why we use the - < docker form in the Makefile.
>
I only see that without DOCKER_BUILDKIT=1. With the variable set it does
like in the CI jobs. I presume it is being set automatically by gitlab,
but we could add it to the script to be explicit.
>> - docker push "$TAG"
>> after_script:
>> - docker logout
>
> So what I don't understand is if I do:
>
> docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross
> docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg
> BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - <
> /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker
>
> I still see pretty much a full rebuild of the image.
I don't use docker and podman does not support caching. I have
fresh-installed docker today and indeed it seems to not use the cache at
every build. We're missing something.
Sometimes it works:
===
$ docker system prune -a -f
Deleted build cache objects:
xzbxmzaib3a8s0ufetop5ikhi
0ce9qln4ipd2vgf9xw9to0fdb
se4hq3rce3lad20t9sqqnubob
Total reclaimed space: 5.845kB
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
$ DOCKER_BUILDKIT=1 docker build --tag
registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --cache-from
registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --build-arg
BUILDKIT_INLINE_CACHE=1 -f debian-amd64.docker .
[+] Building 57.8s (12/12) FINISHED
=> [internal] load build definition from debian-amd64.docker 0.2s
=> => transferring dockerfile: 5.90kB 0.0s
=> [internal] load .dockerignore 0.3s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/debian:11-slim 1.8s
=> importing cache manifest from registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest 0.7s
=> [1/6] FROM docker.io/library/debian:11-slim@sha256:8eaee63a5ea83744e62d5bf88e7d472d7f19b5feda3bfc6a2304cc074f269269 0.0s
=> CACHED [2/6] RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install -y eatmydata && eatmydata apt-get dist-upgrade -y && eatmydata apt-get i 0.0s
=> CACHED [3/6] RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -y --no-install-recommends cscope global linux-headers-amd64 0.0s
=> CACHED [4/6] RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap 0.0s
=> CACHED [5/6] RUN cd /usr/src/netmap && git checkout v11.3 0.0s
=> CACHED [6/6] RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install 54.9s
=> => pulling sha256:bb263680fed18eecdc67f885094df6f589bafc19004839d7fdf141df236a61aa 1.2s
=> => pulling sha256:f63092d28e024c119d6a4e257d0c62bb2117be93605f1f2363b1d9cb8fc9dc6c 44.6s
=> => pulling sha256:30fcedea9de9f22da43e20a79b40516c6e4a55e70eea401acacca18912e86769 4.5s
=> => pulling sha256:6f498daa6d20055a31e4681ec878e86129c3c1eaf5cf40f7ed87631accef1906 3.6s
=> => pulling sha256:79b1b0d7bce9a16b4a01e2055cd402d93e04efded50e2ffcf6be0d7c67e4895a 0.6s
=> => pulling sha256:250b927c5ce2d6d3e9d24bf085f649ce951966c9ec9c9e00ab8292dfb02c8b17 0.7s
=> exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:3663223c0a31c2be678c2fe110ba284b80b8d04d2d3290c14be7a4d7320f8a67 0.0s
=> => naming to registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export
===
There are issues pointing to some general unreliability of the cache:
https://github.com/moby/buildkit/issues/2274
https://github.com/moby/buildkit/issues/1981
What is your docker version? Mine is 20.10.17-ce and Gitlab's is
23.0.1. The issues above are on 20.10.11.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml
2023-02-27 21:12 ` Fabiano Rosas
@ 2023-02-27 21:18 ` Fabiano Rosas
2023-02-28 10:03 ` Daniel P. Berrangé
2023-02-28 10:23 ` Alex Bennée
2 siblings, 0 replies; 10+ messages in thread
From: Fabiano Rosas @ 2023-02-27 21:18 UTC (permalink / raw)
To: Alex Bennée
Cc: qemu-devel, Thomas Huth, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Beraldo Leal,
Daniel P . Berrangé
Fabiano Rosas <farosas@suse.de> writes:
> Alex Bennée <alex.bennee@linaro.org> writes:
>
>> Fabiano Rosas <farosas@suse.de> writes:
>>
>>> Our dockerfiles no longer reference layers from other qemu images so
>>> we can now use 'docker build' on them.
>>>
>>> Also reinstate the caching that was disabled due to bad interactions
>>> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use
>>> BUILDKIT in GitLab either"). We now believe those issues to be fixed.
>>>
>>> The COMMON_TAG needed to be fixed for the caching to work. The
>>> docker.py script was not using the variable, but constructing the
>>> correct URL directly.
>>>
>>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>>> ---
>>> .gitlab-ci.d/container-template.yml | 9 ++++-----
>>> 1 file changed, 4 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
>>> index c434b9c8f3..519b8a9482 100644
>>> --- a/.gitlab-ci.d/container-template.yml
>>> +++ b/.gitlab-ci.d/container-template.yml
>>> @@ -6,17 +6,16 @@
>>> - docker:dind
>>> before_script:
>>> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
>>> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
>>> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
>>> - apk add python3
>>> - docker info
>>> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>>> script:
>>> - echo "TAG:$TAG"
>>> - echo "COMMON_TAG:$COMMON_TAG"
>>> - - ./tests/docker/docker.py --engine docker build
>>> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
>>> - -r $CI_REGISTRY/qemu-project/qemu
>>> - - docker tag "qemu/$NAME" "$TAG"
>>> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
>>> + --build-arg BUILDKIT_INLINE_CACHE=1
>>> + -f "tests/docker/dockerfiles/$NAME.docker" "."
>>
>> I wonder why this doesn't injest a bunch of context. If I run:
>>
>> docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t qemu
>> /debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" .
>>
>> it attempts to bring my entire build directory in as build context. This
>> is why we use the - < docker form in the Makefile.
>>
>
> I only see that without DOCKER_BUILDKIT=1. With the variable set it does
> like in the CI jobs. I presume it is being set automatically by gitlab,
> but we could add it to the script to be explicit.
>
>>> - docker push "$TAG"
>>> after_script:
>>> - docker logout
>>
>> So what I don't understand is if I do:
>>
>> docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross
>> docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg
>> BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - <
>> /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker
>>
>> I still see pretty much a full rebuild of the image.
>
> I don't use docker and podman does not support caching. I have
> fresh-installed docker today and indeed it seems to not use the cache at
> every build. We're missing something.
>
> Sometimes it works:
>
Oops, sorry about the long lines, here it is again:
===
$ docker system prune -a -f
Deleted build cache objects:
xzbxmzaib3a8s0ufetop5ikhi
0ce9qln4ipd2vgf9xw9to0fdb
se4hq3rce3lad20t9sqqnubob
Total reclaimed space: 5.845kB
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
$ DOCKER_BUILDKIT=1 docker build --tag
registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --cache-from
registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --build-arg
BUILDKIT_INLINE_CACHE=1 -f debian-amd64.docker .
[+] Building 57.8s (12/12) FINISHED
=> [internal] load build definition from debian-amd64.docker
=> => transferring dockerfile: 5.90kB
=> [internal] load .dockerignore
=> => transferring context: 2B
=> [internal] load metadata for docker.io/library/debian:11-slim
=> importing cache manifest from registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest
=> [1/6] FROM
docker.io/library/debian:11-slim@sha256:8eaee63a5ea83744e62d5bf88e7d472d7f19b5feda3bfc6a2304cc074f269269
=> CACHED [2/6] RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install -y eatmydata
&& eatmydata apt-get dist-upgrade -y && eatmydata apt-get i
=> CACHED [3/6] RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -y --no-install-recommends cscope
global linux-headers-amd64
=> CACHED [4/6] RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap
=> CACHED [5/6] RUN cd /usr/src/netmap && git checkout v11.3
=> CACHED [6/6] RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d
/usr/src/linux-headers-*-amd64) && make install
=> => pulling sha256:bb263680fed18eecdc67f885094df6f589bafc19004839d7fdf141df236a61aa
=> => pulling sha256:f63092d28e024c119d6a4e257d0c62bb2117be93605f1f2363b1d9cb8fc9dc6c
=> => pulling sha256:30fcedea9de9f22da43e20a79b40516c6e4a55e70eea401acacca18912e86769
=> => pulling sha256:6f498daa6d20055a31e4681ec878e86129c3c1eaf5cf40f7ed87631accef1906
=> => pulling sha256:79b1b0d7bce9a16b4a01e2055cd402d93e04efded50e2ffcf6be0d7c67e4895a
=> => pulling sha256:250b927c5ce2d6d3e9d24bf085f649ce951966c9ec9c9e00ab8292dfb02c8b17
=> exporting to image
=> => exporting layers
=> => writing image sha256:3663223c0a31c2be678c2fe110ba284b80b8d04d2d3290c14be7a4d7320f8a67
=> => naming to registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest
=> exporting cache
=> => preparing build cache for export
===
>
> There are issues pointing to some general unreliability of the cache:
>
> https://github.com/moby/buildkit/issues/2274
> https://github.com/moby/buildkit/issues/1981
>
> What is your docker version? Mine is 20.10.17-ce and Gitlab's is
> 23.0.1. The issues above are on 20.10.11.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml
2023-02-27 21:12 ` Fabiano Rosas
2023-02-27 21:18 ` Fabiano Rosas
@ 2023-02-28 10:03 ` Daniel P. Berrangé
2023-02-28 10:23 ` Alex Bennée
2 siblings, 0 replies; 10+ messages in thread
From: Daniel P. Berrangé @ 2023-02-28 10:03 UTC (permalink / raw)
To: Fabiano Rosas
Cc: Alex Bennée, qemu-devel, Thomas Huth,
Philippe Mathieu-Daudé, Wainer dos Santos Moschetta,
Beraldo Leal
On Mon, Feb 27, 2023 at 06:12:30PM -0300, Fabiano Rosas wrote:
> Alex Bennée <alex.bennee@linaro.org> writes:
>
> > Fabiano Rosas <farosas@suse.de> writes:
> >
> >> Our dockerfiles no longer reference layers from other qemu images so
> >> we can now use 'docker build' on them.
> >>
> >> Also reinstate the caching that was disabled due to bad interactions
> >> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use
> >> BUILDKIT in GitLab either"). We now believe those issues to be fixed.
> >>
> >> The COMMON_TAG needed to be fixed for the caching to work. The
> >> docker.py script was not using the variable, but constructing the
> >> correct URL directly.
> >>
> >> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> >> ---
> >> .gitlab-ci.d/container-template.yml | 9 ++++-----
> >> 1 file changed, 4 insertions(+), 5 deletions(-)
> >>
> >> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> >> index c434b9c8f3..519b8a9482 100644
> >> --- a/.gitlab-ci.d/container-template.yml
> >> +++ b/.gitlab-ci.d/container-template.yml
> >> @@ -6,17 +6,16 @@
> >> - docker:dind
> >> before_script:
> >> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> >> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
> >> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
> >> - apk add python3
> >> - docker info
> >> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> >> script:
> >> - echo "TAG:$TAG"
> >> - echo "COMMON_TAG:$COMMON_TAG"
> >> - - ./tests/docker/docker.py --engine docker build
> >> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
> >> - -r $CI_REGISTRY/qemu-project/qemu
> >> - - docker tag "qemu/$NAME" "$TAG"
> >> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
> >> + --build-arg BUILDKIT_INLINE_CACHE=1
> >> + -f "tests/docker/dockerfiles/$NAME.docker" "."
> >
> > I wonder why this doesn't injest a bunch of context. If I run:
> >
> > docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t qemu
> > /debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" .
> >
> > it attempts to bring my entire build directory in as build context. This
> > is why we use the - < docker form in the Makefile.
> >
>
> I only see that without DOCKER_BUILDKIT=1. With the variable set it does
> like in the CI jobs. I presume it is being set automatically by gitlab,
> but we could add it to the script to be explicit.
>
> >> - docker push "$TAG"
> >> after_script:
> >> - docker logout
> >
> > So what I don't understand is if I do:
> >
> > docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross
> > docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg
> > BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - <
> > /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker
> >
> > I still see pretty much a full rebuild of the image.
>
> I don't use docker and podman does not support caching. I have
> fresh-installed docker today and indeed it seems to not use the cache at
> every build. We're missing something.
Empirically in gitlab I see it working as desired after applying this
patch. My first job building alpine container does all commands:
https://gitlab.com/berrange/qemu/-/jobs/3845066410
My second job building alpine container uses the cache:
https://gitlab.com/berrange/qemu/-/jobs/3845340414
IME of using this in libvirt the caching almost always did its job as
expected in gitlab.
IOW I wouldn't worry too much about local behaviour possibly differing.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml
2023-02-27 15:11 ` [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml Fabiano Rosas
2023-02-27 19:00 ` Alex Bennée
@ 2023-02-28 10:14 ` Daniel P. Berrangé
1 sibling, 0 replies; 10+ messages in thread
From: Daniel P. Berrangé @ 2023-02-28 10:14 UTC (permalink / raw)
To: Fabiano Rosas
Cc: qemu-devel, Thomas Huth, Alex Bennée,
Philippe Mathieu-Daudé, Wainer dos Santos Moschetta,
Beraldo Leal
On Mon, Feb 27, 2023 at 12:11:10PM -0300, Fabiano Rosas wrote:
> Our dockerfiles no longer reference layers from other qemu images so
> we can now use 'docker build' on them.
>
> Also reinstate the caching that was disabled due to bad interactions
> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use
> BUILDKIT in GitLab either"). We now believe those issues to be fixed.
>
> The COMMON_TAG needed to be fixed for the caching to work. The
> docker.py script was not using the variable, but constructing the
> correct URL directly.
>
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
> .gitlab-ci.d/container-template.yml | 9 ++++-----
> 1 file changed, 4 insertions(+), 5 deletions(-)
Tested-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
>
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index c434b9c8f3..519b8a9482 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -6,17 +6,16 @@
> - docker:dind
> before_script:
> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
> - apk add python3
> - docker info
> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> script:
> - echo "TAG:$TAG"
> - echo "COMMON_TAG:$COMMON_TAG"
> - - ./tests/docker/docker.py --engine docker build
> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
> - -r $CI_REGISTRY/qemu-project/qemu
> - - docker tag "qemu/$NAME" "$TAG"
> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
> + --build-arg BUILDKIT_INLINE_CACHE=1
> + -f "tests/docker/dockerfiles/$NAME.docker" "."
> - docker push "$TAG"
> after_script:
> - docker logout
> --
> 2.35.3
>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml
2023-02-27 21:12 ` Fabiano Rosas
2023-02-27 21:18 ` Fabiano Rosas
2023-02-28 10:03 ` Daniel P. Berrangé
@ 2023-02-28 10:23 ` Alex Bennée
2 siblings, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2023-02-28 10:23 UTC (permalink / raw)
To: Fabiano Rosas
Cc: qemu-devel, Thomas Huth, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Beraldo Leal,
Daniel P . Berrangé
Fabiano Rosas <farosas@suse.de> writes:
> Alex Bennée <alex.bennee@linaro.org> writes:
>
>> Fabiano Rosas <farosas@suse.de> writes:
>>
<snip>
>
> There are issues pointing to some general unreliability of the cache:
>
> https://github.com/moby/buildkit/issues/2274
> https://github.com/moby/buildkit/issues/1981
>
> What is your docker version? Mine is 20.10.17-ce and Gitlab's is
> 23.0.1. The issues above are on 20.10.11.
Ahh Debian stable strikes again:
19:34:42 [root@zen:~] # docker --version
Docker version 20.10.5+dfsg1, build 55c4c88
Bookworm is only on 20.10.23+dfsg1-1 and others
Maybe it's time to enable the docker repos again? Debian does offer
podman but I had issues with mapping user ids between container and
volume with it. All the user files appeared own by root.
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH v3 0/1] ci: Speed up container stage
2023-02-27 15:11 [RFC PATCH v3 0/1] ci: Speed up container stage Fabiano Rosas
2023-02-27 15:11 ` [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml Fabiano Rosas
@ 2023-02-28 10:29 ` Alex Bennée
1 sibling, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2023-02-28 10:29 UTC (permalink / raw)
To: Fabiano Rosas
Cc: qemu-devel, Thomas Huth, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Beraldo Leal,
Daniel P . Berrangé
Fabiano Rosas <farosas@suse.de> writes:
<snip>
>
> What do you think?
>
> Fabiano Rosas (1):
> gitlab: Use plain docker in container-template.yml
>
> .gitlab-ci.d/container-template.yml | 9 ++++-----
> 1 file changed, 4 insertions(+), 5 deletions(-)
Queued to testing/next, thanks.
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2023-02-28 10:30 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-02-27 15:11 [RFC PATCH v3 0/1] ci: Speed up container stage Fabiano Rosas
2023-02-27 15:11 ` [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml Fabiano Rosas
2023-02-27 19:00 ` Alex Bennée
2023-02-27 19:39 ` Alex Bennée
2023-02-27 21:12 ` Fabiano Rosas
2023-02-27 21:18 ` Fabiano Rosas
2023-02-28 10:03 ` Daniel P. Berrangé
2023-02-28 10:23 ` Alex Bennée
2023-02-28 10:14 ` Daniel P. Berrangé
2023-02-28 10:29 ` [RFC PATCH v3 0/1] ci: Speed up container stage Alex Bennée
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).