Re: [PATCH 1/2] Add CI configuration for Kubernetes

["Daniel P. Berrangé" <berrange@redhat.com>]

On Fri, Apr 07, 2023 at 03:52:51PM +0100, Camilla Conte wrote:
> Configure Gitlab CI to run on Kubernetes
> according to the official documentation.
> https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#docker-in-docker-with-tls-enabled-in-kubernetes
> 
> These changes are needed because of the CI jobs
> using Docker-in-Docker (dind).
> As soon as Docker-in-Docker is replaced with Kaniko,
> these changes can be reverted.
> 
> I documented what I did to set up the Kubernetes runner on the wiki:
> https://wiki.qemu.org/Testing/CI/KubernetesRunners
> 
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .gitlab-ci.d/container-template.yml |  6 +++---
>  .gitlab-ci.d/default.yml            |  3 +++
>  .gitlab-ci.d/opensbi.yml            |  8 +++-----
>  .gitlab-ci.d/qemu-project.yml       | 17 +++++++++++++++++
>  4 files changed, 26 insertions(+), 8 deletions(-)
>  create mode 100644 .gitlab-ci.d/default.yml
> 
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index 519b8a9482..f55a954741 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -1,14 +1,14 @@
>  .container_job_template:
>    extends: .base_job_template
> -  image: docker:stable
> +  image: docker:20.10.16
>    stage: containers
>    services:
> -    - docker:dind
> +    - docker:20.10.16-dind
>    before_script:
>      - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
>      - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
>      - apk add python3
> -    - docker info
> +    - until docker info; do sleep 1; done
>      - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>    script:
>      - echo "TAG:$TAG"
> diff --git a/.gitlab-ci.d/default.yml b/.gitlab-ci.d/default.yml
> new file mode 100644
> index 0000000000..292be8b91c
> --- /dev/null
> +++ b/.gitlab-ci.d/default.yml
> @@ -0,0 +1,3 @@
> +default:
> +  tags:
> +    - $RUNNER_TAG

Can we just put this in base.yml instead of creating a new file.

> diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
> index 9a651465d8..5b0b47b57b 100644
> --- a/.gitlab-ci.d/opensbi.yml
> +++ b/.gitlab-ci.d/opensbi.yml
> @@ -42,17 +42,15 @@
>  docker-opensbi:
>    extends: .opensbi_job_rules
>    stage: containers
> -  image: docker:stable
> +  image: docker:20.10.16
>    services:
> -    - docker:stable-dind
> +    - docker:20.10.16-dind

Can you elaborate on this ?  I know the docs about use that particular
version tag, but they don't appear to explain why. If this is not
actually a hard requirements, we should keep using the stable tag.

>    variables:
>      GIT_DEPTH: 3
>      IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build
> -    # We don't use TLS
> -    DOCKER_HOST: tcp://docker:2375
> -    DOCKER_TLS_CERTDIR: ""

So IIUC, this was always redundant when using gitlab CI. We should just
remove these in a standalone commit.

>    before_script:
>      - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
> +    - until docker info; do sleep 1; done

Was this really needed ?  The docs don't show that, and docker login is
synchronous, so I wouldn't expect us to them poll on 'docker info'.

In container-template.yml we in fact do the reverse

    - docker info
    - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"

imho best make this opensbi.yml file match contanier-template.yml, and
could be part of the same cleanup commit that removes thhose two docker
env vars.

>    script:
>      - docker pull $IMAGE_TAG || true
>      - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
> diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
> index a7ed447fe4..57b175f5c2 100644
> --- a/.gitlab-ci.d/qemu-project.yml
> +++ b/.gitlab-ci.d/qemu-project.yml
> @@ -1,7 +1,24 @@
>  # This file contains the set of jobs run by the QEMU project:
>  # https://gitlab.com/qemu-project/qemu/-/pipelines
>  
> +variables:
> +  RUNNER_TAG: ""
> +
> +workflow:
> +  rules:
> +    # Set additional variables when running on Kubernetes.
> +    # https://wiki.qemu.org/Testing/CI/KubernetesRunners
> +    - if: $RUNNER_TAG == "k8s"
> +      variables:
> +        DOCKER_HOST: tcp://docker:2376
> +        DOCKER_TLS_CERTDIR: "/certs"
> +        DOCKER_TLS_VERIFY: 1
> +        DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"

Is there anyway we can get the runner itself to set these
correctly by default ?

IMHO the ideal would be that the k8s runners are registerd with the
qemu project to run *any* jobs without requiring tags. That way the
runners will "just work" when share runners are unavailable/exhausted,
like we have with Eldon's runner

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|