Improvements to Gitlab CI container builds

Improvements to Gitlab CI container builds

[Camilla Conte]

Given my recent deep-dive in this part of the Gitlab CI, I
figured I could propose some improvements that came to mind
along the way.

The last patch removes the need for Docker in Docker.

[PATCH 1/5] gitlab-ci: Remove unused Python package

[Camilla Conte]

Python should have been removed in this commit:
https://gitlab.com/qemu-project/qemu/-/commit/94b8b146df84ba472f461398d93fb9cdf0db8f94

Signed-off-by: Camilla Conte <cconte@redhat.com>
---
 .gitlab-ci.d/container-template.yml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index 9ac4a0ee25..fd7deb1c29 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -7,7 +7,6 @@
   before_script:
     - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
     - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
-    - apk add python3
     - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
     - until docker info; do sleep 1; done
   script:
-- 
2.40.1

Re: [PATCH 1/5] gitlab-ci: Remove unused Python package

[Daniel P. Berrangé]

On Wed, May 31, 2023 at 04:08:20PM +0100, Camilla Conte wrote:
> Python should have been removed in this commit:
> https://gitlab.com/qemu-project/qemu/-/commit/94b8b146df84ba472f461398d93fb9cdf0db8f94
> 
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .gitlab-ci.d/container-template.yml | 1 -
>  1 file changed, 1 deletion(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH 2/5] gitlab-ci: Reference rules instead of extends

[Camilla Conte]

This allows for the jobs to extend different templates while
having a shared template for rules.

Docs:
https://docs.gitlab.com/ee/ci/jobs/job_control.html#reuse-rules-in-different-jobs
Signed-off-by: Camilla Conte <cconte@redhat.com>
---
 .gitlab-ci.d/opensbi.yml | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
index 2e9d51764e..d0221632f6 100644
--- a/.gitlab-ci.d/opensbi.yml
+++ b/.gitlab-ci.d/opensbi.yml
@@ -40,7 +40,8 @@
       when: on_success
 
 docker-opensbi:
-  extends: .opensbi_job_rules
+  rules:
+    - !reference [.opensbi_job_rules, rules]
   stage: containers
   image: docker:stable
   services:
@@ -59,7 +60,8 @@ docker-opensbi:
     - docker push $IMAGE_TAG
 
 build-opensbi:
-  extends: .opensbi_job_rules
+  rules:
+    - !reference [.opensbi_job_rules, rules]
   stage: build
   needs: ['docker-opensbi']
   artifacts:
-- 
2.40.1

Re: [PATCH 2/5] gitlab-ci: Reference rules instead of extends

[Daniel P. Berrangé]

On Wed, May 31, 2023 at 04:08:21PM +0100, Camilla Conte wrote:
> This allows for the jobs to extend different templates while
> having a shared template for rules.
> 
> Docs:
> https://docs.gitlab.com/ee/ci/jobs/job_control.html#reuse-rules-in-different-jobs
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .gitlab-ci.d/opensbi.yml | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH 3/5] gitlab-ci: Move variables outside script

[Camilla Conte]

Also split long line for clarity.

Docs:
https://docs.gitlab.com/ee/ci/variables/#use-cicd-variables-in-other-variables
Signed-off-by: Camilla Conte <cconte@redhat.com>
---
 .gitlab-ci.d/container-template.yml | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index fd7deb1c29..d146d1d197 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -1,20 +1,27 @@
 .container_job_template:
   extends: .base_job_template
-  image: docker:stable
   stage: containers
+  image: docker:stable
   services:
     - docker:stable-dind
+  variables:
+    TAG: "$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
+    COMMON_TAG: "$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
+    DOCKERFILE: "$CI_PROJECT_DIR/tests/docker/dockerfiles/$NAME.docker"
+    BUILD_CONTEXT: "$CI_PROJECT_DIR"
   before_script:
-    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
-    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
     - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
     - until docker info; do sleep 1; done
   script:
     - echo "TAG:$TAG"
     - echo "COMMON_TAG:$COMMON_TAG"
-    - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
+    - >
+      docker build
+      --tag "$TAG"
+      --cache-from "$TAG"
+      --cache-from "$COMMON_TAG"
       --build-arg BUILDKIT_INLINE_CACHE=1
-      -f "tests/docker/dockerfiles/$NAME.docker" "."
+      -f "$DOCKERFILE" "$BUILD_CONTEXT"
     - docker push "$TAG"
   after_script:
     - docker logout
-- 
2.40.1

Re: [PATCH 3/5] gitlab-ci: Move variables outside script

[Daniel P. Berrangé]

On Wed, May 31, 2023 at 04:08:22PM +0100, Camilla Conte wrote:
> Also split long line for clarity.


> 
> Docs:
> https://docs.gitlab.com/ee/ci/variables/#use-cicd-variables-in-other-variables
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .gitlab-ci.d/container-template.yml | 17 ++++++++++++-----
>  1 file changed, 12 insertions(+), 5 deletions(-)
> 
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index fd7deb1c29..d146d1d197 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -1,20 +1,27 @@
>  .container_job_template:
>    extends: .base_job_template
> -  image: docker:stable
>    stage: containers
> +  image: docker:stable
>    services:
>      - docker:stable-dind
> +  variables:
> +    TAG: "$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> +    COMMON_TAG: "$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
> +    DOCKERFILE: "$CI_PROJECT_DIR/tests/docker/dockerfiles/$NAME.docker"
> +    BUILD_CONTEXT: "$CI_PROJECT_DIR"
>    before_script:
> -    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> -    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
>      - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>      - until docker info; do sleep 1; done
>    script:
>      - echo "TAG:$TAG"
>      - echo "COMMON_TAG:$COMMON_TAG"
> -    - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
> +    - >

This leading '>' seems redundant

> +      docker build
> +      --tag "$TAG"
> +      --cache-from "$TAG"
> +      --cache-from "$COMMON_TAG"
>        --build-arg BUILDKIT_INLINE_CACHE=1
> -      -f "tests/docker/dockerfiles/$NAME.docker" "."
> +      -f "$DOCKERFILE" "$BUILD_CONTEXT"

This is a functional change as $BUILD_CONTEXT expands
to something different that '.'.

IOW, this should be a separate commit

>      - docker push "$TAG"
>    after_script:
>      - docker logout
> -- 
> 2.40.1
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH 4/5] gitlab-ci: Use container template for opensbi

[Camilla Conte]

Use the same template for all the jobs in the "container" stage.

Changes the URL of the "opensbi-cross-build" images by
using the same URL pattern as the other images.

Removes pushing the image to the additional CI_COMMIT_SHA tag.
It seems unnecessary.

Example of the old URLs:
- registry.gitlab.com/qemu-project/qemu:00a0bdc...
- registry.gitlab.com/qemu-project/qemu:opensbi-cross-build

Example of the new URL:
- registry.gitlab.com/qemu-project/qemu/qemu/opensbi-cross-build

Signed-off-by: Camilla Conte <cconte@redhat.com>
---
 .gitlab-ci.d/opensbi.yml | 23 ++++++++---------------
 1 file changed, 8 insertions(+), 15 deletions(-)

diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
index d0221632f6..bb90c56bcf 100644
--- a/.gitlab-ci.d/opensbi.yml
+++ b/.gitlab-ci.d/opensbi.yml
@@ -1,3 +1,6 @@
+include:
+  - local: '/.gitlab-ci.d/container-template.yml'
+
 # All jobs needing docker-opensbi must use the same rules it uses.
 .opensbi_job_rules:
   rules:
@@ -40,24 +43,14 @@
       when: on_success
 
 docker-opensbi:
+  extends: .container_job_template
   rules:
     - !reference [.opensbi_job_rules, rules]
-  stage: containers
-  image: docker:stable
-  services:
-    - docker:stable-dind
   variables:
     GIT_DEPTH: 3
-    IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build
-  before_script:
-    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
-    - until docker info; do sleep 1; done
-  script:
-    - docker pull $IMAGE_TAG || true
-    - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
-                                           --tag $IMAGE_TAG .gitlab-ci.d/opensbi
-    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
-    - docker push $IMAGE_TAG
+    NAME: opensbi-cross-build
+    DOCKERFILE: "$CI_PROJECT_DIR/.gitlab-ci.d/opensbi/Dockerfile"
+    BUILD_CONTEXT: "$CI_PROJECT_DIR/.gitlab-ci.d/opensbi"
 
 build-opensbi:
   rules:
@@ -72,7 +65,7 @@ build-opensbi:
       - opensbi32-generic-stderr.log
       - opensbi64-generic-stdout.log
       - opensbi64-generic-stderr.log
-  image: $CI_REGISTRY_IMAGE:opensbi-cross-build
+  image: $CI_REGISTRY_IMAGE/qemu/opensbi-cross-build:latest
   variables:
     GIT_DEPTH: 3
   script: # Clone the required submodules and build OpenSBI
-- 
2.40.1

Re: [PATCH 4/5] gitlab-ci: Use container template for opensbi

[Daniel P. Berrangé]

On Wed, May 31, 2023 at 04:08:23PM +0100, Camilla Conte wrote:
> Use the same template for all the jobs in the "container" stage.
> 
> Changes the URL of the "opensbi-cross-build" images by
> using the same URL pattern as the other images.
> 
> Removes pushing the image to the additional CI_COMMIT_SHA tag.
> It seems unnecessary.
> 
> Example of the old URLs:
> - registry.gitlab.com/qemu-project/qemu:00a0bdc...
> - registry.gitlab.com/qemu-project/qemu:opensbi-cross-build
> 
> Example of the new URL:
> - registry.gitlab.com/qemu-project/qemu/qemu/opensbi-cross-build
> 
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .gitlab-ci.d/opensbi.yml | 23 ++++++++---------------
>  1 file changed, 8 insertions(+), 15 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH 5/5] gitlab-ci: Replace Docker with Kaniko

[Camilla Conte]

Previous attempt by Alex Bennée, for the records:
https://lore.kernel.org/qemu-devel/20230330101141.30199-12-alex.bennee@linaro.org/

Does not enable caching, as suggested in a previous review:
https://lore.kernel.org/qemu-devel/ZCVpMDLPJcdnkAJz@redhat.com/

Does not specify a context since no Dockerfile is using COPY or ADD instructions.

Does not enable reproducible builds as
that results in builds failing with an out of memory error.
See issue "Using --reproducible loads entire image into memory":
https://github.com/GoogleContainerTools/kaniko/issues/862

Signed-off-by: Camilla Conte <cconte@redhat.com>
---
 .gitlab-ci.d/container-template.yml           | 25 +++++--------------
 .gitlab-ci.d/opensbi.yml                      |  1 -
 .../ci/gitlab-kubernetes-runners/values.yaml  | 12 ---------
 3 files changed, 6 insertions(+), 32 deletions(-)

diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index d146d1d197..0e93f5f13e 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -1,27 +1,14 @@
 .container_job_template:
   extends: .base_job_template
   stage: containers
-  image: docker:stable
-  services:
-    - docker:stable-dind
+  image:
+    name: gcr.io/kaniko-project/executor:debug
+    entrypoint: [""]
   variables:
     TAG: "$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
-    COMMON_TAG: "$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
     DOCKERFILE: "$CI_PROJECT_DIR/tests/docker/dockerfiles/$NAME.docker"
-    BUILD_CONTEXT: "$CI_PROJECT_DIR"
-  before_script:
-    - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
-    - until docker info; do sleep 1; done
   script:
     - echo "TAG:$TAG"
-    - echo "COMMON_TAG:$COMMON_TAG"
-    - >
-      docker build
-      --tag "$TAG"
-      --cache-from "$TAG"
-      --cache-from "$COMMON_TAG"
-      --build-arg BUILDKIT_INLINE_CACHE=1
-      -f "$DOCKERFILE" "$BUILD_CONTEXT"
-    - docker push "$TAG"
-  after_script:
-    - docker logout
+    - /kaniko/executor
+      --dockerfile "$DOCKERFILE"
+      --destination "$TAG"
diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
index bb90c56bcf..e531ce2728 100644
--- a/.gitlab-ci.d/opensbi.yml
+++ b/.gitlab-ci.d/opensbi.yml
@@ -50,7 +50,6 @@ docker-opensbi:
     GIT_DEPTH: 3
     NAME: opensbi-cross-build
     DOCKERFILE: "$CI_PROJECT_DIR/.gitlab-ci.d/opensbi/Dockerfile"
-    BUILD_CONTEXT: "$CI_PROJECT_DIR/.gitlab-ci.d/opensbi"
 
 build-opensbi:
   rules:
diff --git a/scripts/ci/gitlab-kubernetes-runners/values.yaml b/scripts/ci/gitlab-kubernetes-runners/values.yaml
index 204a96a842..ed6c5894bc 100644
--- a/scripts/ci/gitlab-kubernetes-runners/values.yaml
+++ b/scripts/ci/gitlab-kubernetes-runners/values.yaml
@@ -4,27 +4,15 @@ rbac:
   create: true
 concurrent: 200
 runners:
-  privileged: true
   config: |
     [[runners]]
       limit = 100
-      environment = [
-        "DOCKER_HOST=tcp://docker:2376",
-        "DOCKER_TLS_CERTDIR=/certs",
-        "DOCKER_TLS_VERIFY=1",
-        "DOCKER_CERT_PATH=/certs/client"
-      ]
       [runners.kubernetes]
         poll_timeout = 1200
-        image = "ubuntu:20.04"
         cpu_request = "0.5"
         service_cpu_request = "0.5"
         helper_cpu_request = "0.25"
         cpu_request_overwrite_max_allowed = "7"
         memory_request_overwrite_max_allowed = "30Gi"
-      [[runners.kubernetes.volumes.empty_dir]]
-        name = "docker-certs"
-        mount_path = "/certs/client"
-        medium = "Memory"
       [runners.kubernetes.node_selector]
         agentpool = "jobs"
-- 
2.40.1

Re: [PATCH 5/5] gitlab-ci: Replace Docker with Kaniko

[Daniel P. Berrangé]

On Wed, May 31, 2023 at 04:08:24PM +0100, Camilla Conte wrote:
> Previous attempt by Alex Bennée, for the records:
> https://lore.kernel.org/qemu-devel/20230330101141.30199-12-alex.bennee@linaro.org/
> 
> Does not enable caching, as suggested in a previous review:
> https://lore.kernel.org/qemu-devel/ZCVpMDLPJcdnkAJz@redhat.com/

In this mail I suggested we entirely skip running kaniko if we
determine the dockerfile content is unchanged since last build...

> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index d146d1d197..0e93f5f13e 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml

> +    - /kaniko/executor
> +      --dockerfile "$DOCKERFILE"
> +      --destination "$TAG"

....this is running kaniko unconditionally on every pipeline.

I fear this is going to increase the CI credits we are burning on every
pipeline. While QEMU has added its private k8s runner, our contributors
forks are still burning limited CI credits.

Do you have pipelines showing the CI credit usage of a pipeline before
and after this change ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|