From: Fabiano Rosas <farosas@suse.de>
To: "Alex Bennée" <alex.bennee@linaro.org>
Cc: qemu-devel@nongnu.org, "Thomas Huth" <thuth@redhat.com>,
"Philippe Mathieu-Daudé" <philmd@linaro.org>,
"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
"Beraldo Leal" <bleal@redhat.com>,
"Daniel P . Berrangé" <berrange@redhat.com>
Subject: Re: [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml
Date: Mon, 27 Feb 2023 18:12:30 -0300 [thread overview]
Message-ID: <87lekiyfr5.fsf@suse.de> (raw)
In-Reply-To: <87fsargc9w.fsf@linaro.org>
Alex Bennée <alex.bennee@linaro.org> writes:
> Fabiano Rosas <farosas@suse.de> writes:
>
>> Our dockerfiles no longer reference layers from other qemu images so
>> we can now use 'docker build' on them.
>>
>> Also reinstate the caching that was disabled due to bad interactions
>> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use
>> BUILDKIT in GitLab either"). We now believe those issues to be fixed.
>>
>> The COMMON_TAG needed to be fixed for the caching to work. The
>> docker.py script was not using the variable, but constructing the
>> correct URL directly.
>>
>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>> ---
>> .gitlab-ci.d/container-template.yml | 9 ++++-----
>> 1 file changed, 4 insertions(+), 5 deletions(-)
>>
>> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
>> index c434b9c8f3..519b8a9482 100644
>> --- a/.gitlab-ci.d/container-template.yml
>> +++ b/.gitlab-ci.d/container-template.yml
>> @@ -6,17 +6,16 @@
>> - docker:dind
>> before_script:
>> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
>> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
>> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
>> - apk add python3
>> - docker info
>> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>> script:
>> - echo "TAG:$TAG"
>> - echo "COMMON_TAG:$COMMON_TAG"
>> - - ./tests/docker/docker.py --engine docker build
>> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
>> - -r $CI_REGISTRY/qemu-project/qemu
>> - - docker tag "qemu/$NAME" "$TAG"
>> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
>> + --build-arg BUILDKIT_INLINE_CACHE=1
>> + -f "tests/docker/dockerfiles/$NAME.docker" "."
>
> I wonder why this doesn't injest a bunch of context. If I run:
>
> docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t qemu
> /debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" .
>
> it attempts to bring my entire build directory in as build context. This
> is why we use the - < docker form in the Makefile.
>
I only see that without DOCKER_BUILDKIT=1. With the variable set it does
like in the CI jobs. I presume it is being set automatically by gitlab,
but we could add it to the script to be explicit.
>> - docker push "$TAG"
>> after_script:
>> - docker logout
>
> So what I don't understand is if I do:
>
> docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross
> docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg
> BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - <
> /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker
>
> I still see pretty much a full rebuild of the image.
I don't use docker and podman does not support caching. I have
fresh-installed docker today and indeed it seems to not use the cache at
every build. We're missing something.
Sometimes it works:
===
$ docker system prune -a -f
Deleted build cache objects:
xzbxmzaib3a8s0ufetop5ikhi
0ce9qln4ipd2vgf9xw9to0fdb
se4hq3rce3lad20t9sqqnubob
Total reclaimed space: 5.845kB
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
$ DOCKER_BUILDKIT=1 docker build --tag
registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --cache-from
registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --build-arg
BUILDKIT_INLINE_CACHE=1 -f debian-amd64.docker .
[+] Building 57.8s (12/12) FINISHED
=> [internal] load build definition from debian-amd64.docker 0.2s
=> => transferring dockerfile: 5.90kB 0.0s
=> [internal] load .dockerignore 0.3s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/debian:11-slim 1.8s
=> importing cache manifest from registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest 0.7s
=> [1/6] FROM docker.io/library/debian:11-slim@sha256:8eaee63a5ea83744e62d5bf88e7d472d7f19b5feda3bfc6a2304cc074f269269 0.0s
=> CACHED [2/6] RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install -y eatmydata && eatmydata apt-get dist-upgrade -y && eatmydata apt-get i 0.0s
=> CACHED [3/6] RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -y --no-install-recommends cscope global linux-headers-amd64 0.0s
=> CACHED [4/6] RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap 0.0s
=> CACHED [5/6] RUN cd /usr/src/netmap && git checkout v11.3 0.0s
=> CACHED [6/6] RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install 54.9s
=> => pulling sha256:bb263680fed18eecdc67f885094df6f589bafc19004839d7fdf141df236a61aa 1.2s
=> => pulling sha256:f63092d28e024c119d6a4e257d0c62bb2117be93605f1f2363b1d9cb8fc9dc6c 44.6s
=> => pulling sha256:30fcedea9de9f22da43e20a79b40516c6e4a55e70eea401acacca18912e86769 4.5s
=> => pulling sha256:6f498daa6d20055a31e4681ec878e86129c3c1eaf5cf40f7ed87631accef1906 3.6s
=> => pulling sha256:79b1b0d7bce9a16b4a01e2055cd402d93e04efded50e2ffcf6be0d7c67e4895a 0.6s
=> => pulling sha256:250b927c5ce2d6d3e9d24bf085f649ce951966c9ec9c9e00ab8292dfb02c8b17 0.7s
=> exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:3663223c0a31c2be678c2fe110ba284b80b8d04d2d3290c14be7a4d7320f8a67 0.0s
=> => naming to registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export
===
There are issues pointing to some general unreliability of the cache:
https://github.com/moby/buildkit/issues/2274
https://github.com/moby/buildkit/issues/1981
What is your docker version? Mine is 20.10.17-ce and Gitlab's is
23.0.1. The issues above are on 20.10.11.
next prev parent reply other threads:[~2023-02-27 21:13 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-27 15:11 [RFC PATCH v3 0/1] ci: Speed up container stage Fabiano Rosas
2023-02-27 15:11 ` [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml Fabiano Rosas
2023-02-27 19:00 ` Alex Bennée
2023-02-27 19:39 ` Alex Bennée
2023-02-27 21:12 ` Fabiano Rosas [this message]
2023-02-27 21:18 ` Fabiano Rosas
2023-02-28 10:03 ` Daniel P. Berrangé
2023-02-28 10:23 ` Alex Bennée
2023-02-28 10:14 ` Daniel P. Berrangé
2023-02-28 10:29 ` [RFC PATCH v3 0/1] ci: Speed up container stage Alex Bennée
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87lekiyfr5.fsf@suse.de \
--to=farosas@suse.de \
--cc=alex.bennee@linaro.org \
--cc=berrange@redhat.com \
--cc=bleal@redhat.com \
--cc=philmd@linaro.org \
--cc=qemu-devel@nongnu.org \
--cc=thuth@redhat.com \
--cc=wainersm@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).