Add CI configuration for Kubernetes

Add CI configuration for Kubernetes

[Camilla Conte]

Here's a second version (v2) of patches to support the Kubernetes runner for Gitlab CI.
You can find the v1 thread here: https://lore.kernel.org/qemu-devel/20230407145252.32955-1-cconte@redhat.com/.

[PATCH v2 1/5] Remove redundant CI variables

[Camilla Conte]

These are not needed when using gitlab.com shared runners.

Signed-off-by: Camilla Conte <cconte@redhat.com>
---
 .gitlab-ci.d/opensbi.yml | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
index 9a651465d8..13070575b6 100644
--- a/.gitlab-ci.d/opensbi.yml
+++ b/.gitlab-ci.d/opensbi.yml
@@ -48,9 +48,6 @@ docker-opensbi:
   variables:
     GIT_DEPTH: 3
     IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build
-    # We don't use TLS
-    DOCKER_HOST: tcp://docker:2375
-    DOCKER_TLS_CERTDIR: ""
   before_script:
     - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
   script:
-- 
2.40.1

Re: [PATCH v2 1/5] Remove redundant CI variables

[Daniel P. Berrangé]

On Mon, May 22, 2023 at 06:41:50PM +0100, Camilla Conte wrote:
> These are not needed when using gitlab.com shared runners.
> 
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .gitlab-ci.d/opensbi.yml | 3 ---
>  1 file changed, 3 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v2 2/5] Use docker "stable" tag

[Camilla Conte]

Use the same tag in all jobs.

Signed-off-by: Camilla Conte <cconte@redhat.com>
---
 .gitlab-ci.d/container-template.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index 519b8a9482..11569dd900 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -3,7 +3,7 @@
   image: docker:stable
   stage: containers
   services:
-    - docker:dind
+    - docker:stable-dind
   before_script:
     - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
     - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
-- 
2.40.1

Re: [PATCH v2 2/5] Use docker "stable" tag

[Daniel P. Berrangé]

On Mon, May 22, 2023 at 06:41:51PM +0100, Camilla Conte wrote:
> Use the same tag in all jobs.
> 
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .gitlab-ci.d/container-template.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 2/5] Use docker "stable" tag

[Daniel P. Berrangé]

On Mon, May 22, 2023 at 06:41:51PM +0100, Camilla Conte wrote:
> Use the same tag in all jobs.
> 
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .gitlab-ci.d/container-template.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index 519b8a9482..11569dd900 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -3,7 +3,7 @@
>    image: docker:stable
>    stage: containers
>    services:
> -    - docker:dind
> +    - docker:stable-dind

For reasons I don't understand, this change is responsible for all our
failed CI jobs involving Fedora being unable to resolve DNS

Fedora 37 - x86_64                              0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fedora':
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-37&arch=x86_64&countme=1 [getaddrinfo() thread failed to start]
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-37&arch=x86_64 [getaddrinfo() thread failed to start]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-37&arch=x86_64 [getaddrinfo() thread failed to start]


Switching back to ':dind' instead of ':stable-dind' makes it work again.


AFAICT, the ':stable-dind' and ':stable' tags are no longer even supported.

The last published content under these tags dates from 2020, and these
tag names are not mentioned on

    https://hub.docker.com/_/docker


While reverting this change would fix our Fedora problems, it leaves usage
of the 'stable-dind' tag in opensbi.yml, as well as 'stable' tag usage in
other places.

Thus I'm going to test a patch to move everything to 'latest' / 'dind'


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 2/5] Use docker "stable" tag

[Daniel P. Berrangé]

On Tue, May 30, 2023 at 06:39:26PM +0100, Daniel P. Berrangé wrote:
> On Mon, May 22, 2023 at 06:41:51PM +0100, Camilla Conte wrote:
> > Use the same tag in all jobs.
> > 
> > Signed-off-by: Camilla Conte <cconte@redhat.com>
> > ---
> >  .gitlab-ci.d/container-template.yml | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> > index 519b8a9482..11569dd900 100644
> > --- a/.gitlab-ci.d/container-template.yml
> > +++ b/.gitlab-ci.d/container-template.yml
> > @@ -3,7 +3,7 @@
> >    image: docker:stable
> >    stage: containers
> >    services:
> > -    - docker:dind
> > +    - docker:stable-dind
> 
> For reasons I don't understand, this change is responsible for all our
> failed CI jobs involving Fedora being unable to resolve DNS

> AFAICT, the ':stable-dind' and ':stable' tags are no longer even supported.

And I found confirmation of this:

  https://github.com/docker-library/docker/commit/606c63960a4845af7077721eb3900c706f5d0c5e

"This will remove the "test" and "stable" aliases for whatever
 major release comes after 19.03 (so once the next release comes
 out, "stable" will become a stale alias).

 I'm not 100% sure this is the right answer, but the "stable"
 alias has definitely been a huge source of confusion very
 recently (and it doesn't have much meaning anymore anyways
 given the death of the "edge" channel)."

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 2/5] Use docker "stable" tag

[Richard Henderson]

On 5/30/23 10:49, Daniel P. Berrangé wrote:
> On Tue, May 30, 2023 at 06:39:26PM +0100, Daniel P. Berrangé wrote:
>> On Mon, May 22, 2023 at 06:41:51PM +0100, Camilla Conte wrote:
>>> Use the same tag in all jobs.
>>>
>>> Signed-off-by: Camilla Conte <cconte@redhat.com>
>>> ---
>>>   .gitlab-ci.d/container-template.yml | 2 +-
>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
>>> index 519b8a9482..11569dd900 100644
>>> --- a/.gitlab-ci.d/container-template.yml
>>> +++ b/.gitlab-ci.d/container-template.yml
>>> @@ -3,7 +3,7 @@
>>>     image: docker:stable
>>>     stage: containers
>>>     services:
>>> -    - docker:dind
>>> +    - docker:stable-dind
>>
>> For reasons I don't understand, this change is responsible for all our
>> failed CI jobs involving Fedora being unable to resolve DNS
> 
>> AFAICT, the ':stable-dind' and ':stable' tags are no longer even supported.
> 
> And I found confirmation of this:
> 
>    https://github.com/docker-library/docker/commit/606c63960a4845af7077721eb3900c706f5d0c5e
> 
> "This will remove the "test" and "stable" aliases for whatever
>   major release comes after 19.03 (so once the next release comes
>   out, "stable" will become a stale alias).
> 
>   I'm not 100% sure this is the right answer, but the "stable"
>   alias has definitely been a huge source of confusion very
>   recently (and it doesn't have much meaning anymore anyways
>   given the death of the "edge" channel)."

Excellent, thanks for the sleuthing.


r~

[PATCH v2 3/5] Add loop over docker info

[Camilla Conte]

Wait for docker info to return successfuly to ensure that
the docker server (daemon) started.
This is needed for jobs running on Kubernetes.
See https://wiki.qemu.org/Testing/CI/KubernetesRunners.

Signed-off-by: Camilla Conte <cconte@redhat.com>
---
 .gitlab-ci.d/container-template.yml | 2 +-
 .gitlab-ci.d/opensbi.yml            | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index 11569dd900..9ac4a0ee25 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -8,8 +8,8 @@
     - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
     - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
     - apk add python3
-    - docker info
     - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+    - until docker info; do sleep 1; done
   script:
     - echo "TAG:$TAG"
     - echo "COMMON_TAG:$COMMON_TAG"
diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
index 13070575b6..2e9d51764e 100644
--- a/.gitlab-ci.d/opensbi.yml
+++ b/.gitlab-ci.d/opensbi.yml
@@ -50,6 +50,7 @@ docker-opensbi:
     IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build
   before_script:
     - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
+    - until docker info; do sleep 1; done
   script:
     - docker pull $IMAGE_TAG || true
     - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
-- 
2.40.1

Re: [PATCH v2 3/5] Add loop over docker info

[Daniel P. Berrangé]

On Mon, May 22, 2023 at 06:41:52PM +0100, Camilla Conte wrote:
> Wait for docker info to return successfuly to ensure that
> the docker server (daemon) started.
> This is needed for jobs running on Kubernetes.
> See https://wiki.qemu.org/Testing/CI/KubernetesRunners.
> 
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .gitlab-ci.d/container-template.yml | 2 +-
>  .gitlab-ci.d/opensbi.yml            | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 3/5] Add loop over docker info

[Richard Henderson]

On 5/22/23 10:41, Camilla Conte wrote:
> +    - until docker info; do sleep 1; done

How long are you expecting this to have to wait?
Less than 5 seconds?


r~

Re: [PATCH v2 3/5] Add loop over docker info

[Camilla Conte]

On Tue, May 23, 2023 at 2:06 PM Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> On 5/22/23 10:41, Camilla Conte wrote:
> > +    - until docker info; do sleep 1; done
>
> How long are you expecting this to have to wait?
> Less than 5 seconds?

I remember seeing jobs where it took longer than five seconds, but I
can't find them at the moment.
I guess you wish the "until" loop to fail sooner, rather than having
to wait for the whole job timing out.
What about setting a five seconds timeout for the loop and raise it
later if we see
jobs failing?

>
>
> r~
>

[PATCH v2 4/5] Add CI variable RUNNER_TAG

[Camilla Conte]

This allows to set a job tag dinamically.
We need this to be able to select the Kubernetes runner.
See https://wiki.qemu.org/Testing/CI/KubernetesRunners.

Signed-off-by: Camilla Conte <cconte@redhat.com>
---
 .gitlab-ci.d/qemu-project.yml | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index a7ed447fe4..4d914c4897 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -1,6 +1,13 @@
 # This file contains the set of jobs run by the QEMU project:
 # https://gitlab.com/qemu-project/qemu/-/pipelines
 
+variables:
+  RUNNER_TAG: ""
+
+default:
+  tags:
+    - $RUNNER_TAG
+
 include:
   - local: '/.gitlab-ci.d/base.yml'
   - local: '/.gitlab-ci.d/stages.yml'
-- 
2.40.1

Re: [PATCH v2 4/5] Add CI variable RUNNER_TAG

[Daniel P. Berrangé]

On Mon, May 22, 2023 at 06:41:53PM +0100, Camilla Conte wrote:
> This allows to set a job tag dinamically.

typo -  s/dinamically/dynamically/

can be fixed when a maintainer queues this, no need to repost
just for this typo.

> We need this to be able to select the Kubernetes runner.
> See https://wiki.qemu.org/Testing/CI/KubernetesRunners.
> 
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .gitlab-ci.d/qemu-project.yml | 7 +++++++
>  1 file changed, 7 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v2 5/5] Add Kubernetes runner configuration

[Camilla Conte]

Custom values for the gitlab-runner Helm chart.
See https://wiki.qemu.org/Testing/CI/KubernetesRunners.

Signed-off-by: Camilla Conte <cconte@redhat.com>
---
 .../ci/gitlab-kubernetes-runners/values.yaml  | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 scripts/ci/gitlab-kubernetes-runners/values.yaml

diff --git a/scripts/ci/gitlab-kubernetes-runners/values.yaml b/scripts/ci/gitlab-kubernetes-runners/values.yaml
new file mode 100644
index 0000000000..204a96a842
--- /dev/null
+++ b/scripts/ci/gitlab-kubernetes-runners/values.yaml
@@ -0,0 +1,30 @@
+gitlabUrl: "https://gitlab.com/"
+runnerRegistrationToken: ""
+rbac:
+  create: true
+concurrent: 200
+runners:
+  privileged: true
+  config: |
+    [[runners]]
+      limit = 100
+      environment = [
+        "DOCKER_HOST=tcp://docker:2376",
+        "DOCKER_TLS_CERTDIR=/certs",
+        "DOCKER_TLS_VERIFY=1",
+        "DOCKER_CERT_PATH=/certs/client"
+      ]
+      [runners.kubernetes]
+        poll_timeout = 1200
+        image = "ubuntu:20.04"
+        cpu_request = "0.5"
+        service_cpu_request = "0.5"
+        helper_cpu_request = "0.25"
+        cpu_request_overwrite_max_allowed = "7"
+        memory_request_overwrite_max_allowed = "30Gi"
+      [[runners.kubernetes.volumes.empty_dir]]
+        name = "docker-certs"
+        mount_path = "/certs/client"
+        medium = "Memory"
+      [runners.kubernetes.node_selector]
+        agentpool = "jobs"
-- 
2.40.1

Re: [PATCH v2 5/5] Add Kubernetes runner configuration

[Daniel P. Berrangé]

On Mon, May 22, 2023 at 06:41:54PM +0100, Camilla Conte wrote:
> Custom values for the gitlab-runner Helm chart.
> See https://wiki.qemu.org/Testing/CI/KubernetesRunners.
> 
> Signed-off-by: Camilla Conte <cconte@redhat.com>
> ---
>  .../ci/gitlab-kubernetes-runners/values.yaml  | 30 +++++++++++++++++++
>  1 file changed, 30 insertions(+)
>  create mode 100644 scripts/ci/gitlab-kubernetes-runners/values.yaml

Can't say I'm massively familiar with this, but there's nothing
obviously terrible, so on the basis that you've tested it works

  Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


> 
> diff --git a/scripts/ci/gitlab-kubernetes-runners/values.yaml b/scripts/ci/gitlab-kubernetes-runners/values.yaml
> new file mode 100644
> index 0000000000..204a96a842
> --- /dev/null
> +++ b/scripts/ci/gitlab-kubernetes-runners/values.yaml
> @@ -0,0 +1,30 @@
> +gitlabUrl: "https://gitlab.com/"
> +runnerRegistrationToken: ""
> +rbac:
> +  create: true
> +concurrent: 200
> +runners:
> +  privileged: true
> +  config: |
> +    [[runners]]
> +      limit = 100
> +      environment = [
> +        "DOCKER_HOST=tcp://docker:2376",
> +        "DOCKER_TLS_CERTDIR=/certs",
> +        "DOCKER_TLS_VERIFY=1",
> +        "DOCKER_CERT_PATH=/certs/client"
> +      ]
> +      [runners.kubernetes]
> +        poll_timeout = 1200
> +        image = "ubuntu:20.04"

Is it viable to update to 22.04 whic is the current newest LTS release
stream ?

> +        cpu_request = "0.5"
> +        service_cpu_request = "0.5"
> +        helper_cpu_request = "0.25"
> +        cpu_request_overwrite_max_allowed = "7"
> +        memory_request_overwrite_max_allowed = "30Gi"
> +      [[runners.kubernetes.volumes.empty_dir]]
> +        name = "docker-certs"
> +        mount_path = "/certs/client"
> +        medium = "Memory"
> +      [runners.kubernetes.node_selector]
> +        agentpool = "jobs"

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Add CI configuration for Kubernetes

[Richard Henderson]

On 5/22/23 10:41, Camilla Conte wrote:
> Here's a second version (v2) of patches to support the Kubernetes runner for Gitlab CI.
> You can find the v1 thread here: https://lore.kernel.org/qemu-devel/20230407145252.32955-1-cconte@redhat.com/.


This does not work:

https://gitlab.com/qemu-project/qemu/-/pipelines/875254290

In particular, most jobs spent 30 minutes (until I cancelled them), e.g.

https://gitlab.com/qemu-project/qemu/-/jobs/4329346855#L7561

Client:
  Debug Mode: false
Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?


r~

Re: Add CI configuration for Kubernetes

[Daniel P. Berrangé]

On Mon, May 22, 2023 at 03:52:30PM -0700, Richard Henderson wrote:
> On 5/22/23 10:41, Camilla Conte wrote:
> > Here's a second version (v2) of patches to support the Kubernetes runner for Gitlab CI.
> > You can find the v1 thread here: https://lore.kernel.org/qemu-devel/20230407145252.32955-1-cconte@redhat.com/.
> 
> 
> This does not work:
> 
> https://gitlab.com/qemu-project/qemu/-/pipelines/875254290
> 
> In particular, most jobs spent 30 minutes (until I cancelled them), e.g.
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/4329346855#L7561
> 
> Client:
>  Debug Mode: false
> Server:
> ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
> Is the docker daemon running?

Seems the problem is with the launcher for the k8s runner which throws
a mass of warnings & errors earlier in the logs about pods not being
ready

  https://gitlab.com/qemu-project/qemu/-/jobs/4329346855#L12

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Add CI configuration for Kubernetes

[Camilla Conte]

On Mon, May 22, 2023 at 11:52 PM Richard Henderson
<richard.henderson@linaro.org> wrote:
> This does not work:
>
> https://gitlab.com/qemu-project/qemu/-/pipelines/875254290

My bad, sorry. I didn't update the runner with the new values.yaml.
The Docker environment variables were missing so it's looking for the
Docker server at the wrong place.

Now I updated the runner. You have to add the "k8s" tag to the runner
again in the settings:
https://gitlab.com/qemu-project/qemu/-/settings/ci_cd#Runners.
As soon as the "k8s" tag is there the pipeline should be able to run.

Re: Add CI configuration for Kubernetes

[Richard Henderson]

On 5/23/23 01:24, Camilla Conte wrote:
> On Mon, May 22, 2023 at 11:52 PM Richard Henderson
> <richard.henderson@linaro.org> wrote:
>> This does not work:
>>
>> https://gitlab.com/qemu-project/qemu/-/pipelines/875254290
> 
> My bad, sorry. I didn't update the runner with the new values.yaml.
> The Docker environment variables were missing so it's looking for the
> Docker server at the wrong place.
> 
> Now I updated the runner. You have to add the "k8s" tag to the runner
> again in the settings:
> https://gitlab.com/qemu-project/qemu/-/settings/ci_cd#Runners.
> As soon as the "k8s" tag is there the pipeline should be able to run.
> 

The k8s tag has been there on the runner all along.  Did you mean something else?


r~

Re: Add CI configuration for Kubernetes

[Camilla Conte]

On Tue, May 23, 2023 at 2:08 PM Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> On 5/23/23 01:24, Camilla Conte wrote:
> > On Mon, May 22, 2023 at 11:52 PM Richard Henderson
> > <richard.henderson@linaro.org> wrote:
> >> This does not work:
> >>
> >> https://gitlab.com/qemu-project/qemu/-/pipelines/875254290
> >
> > My bad, sorry. I didn't update the runner with the new values.yaml.
> > The Docker environment variables were missing so it's looking for the
> > Docker server at the wrong place.
> >
> > Now I updated the runner. You have to add the "k8s" tag to the runner
> > again in the settings:
> > https://gitlab.com/qemu-project/qemu/-/settings/ci_cd#Runners.
> > As soon as the "k8s" tag is there the pipeline should be able to run.
> >
>
> The k8s tag has been there on the runner all along.  Did you mean something else?

I had Paolo adding it a few hours ago.

>
>
> r~
>

[PATCH v3 5/5] Add Kubernetes runner configuration

[Camilla Conte]

Custom values for the gitlab-runner Helm chart.
See https://wiki.qemu.org/Testing/CI/KubernetesRunners.

Signed-off-by: Camilla Conte <cconte@redhat.com>
---
Changes since previous version (PATCH v2 5/5):
- Update ubuntu tag to 22.04. I tested it and it works.
- Remove concurrent jobs limits (use defaults).

 .../ci/gitlab-kubernetes-runners/values.yaml  | 28 +++++++++++++++++++
 1 file changed, 28 insertions(+)
 create mode 100644 scripts/ci/gitlab-kubernetes-runners/values.yaml

diff --git a/scripts/ci/gitlab-kubernetes-runners/values.yaml b/scripts/ci/gitlab-kubernetes-runners/values.yaml
new file mode 100644
index 0000000000..34c9befdce
--- /dev/null
+++ b/scripts/ci/gitlab-kubernetes-runners/values.yaml
@@ -0,0 +1,28 @@
+gitlabUrl: "https://gitlab.com/"
+runnerRegistrationToken: ""
+rbac:
+  create: true
+runners:
+  privileged: true
+  config: |
+    [[runners]]
+      environment = [
+        "DOCKER_HOST=tcp://docker:2376",
+        "DOCKER_TLS_CERTDIR=/certs",
+        "DOCKER_TLS_VERIFY=1",
+        "DOCKER_CERT_PATH=/certs/client"
+      ]
+      [runners.kubernetes]
+        poll_timeout = 1200
+        image = "ubuntu:22.04"
+        cpu_request = "0.5"
+        service_cpu_request = "0.5"
+        helper_cpu_request = "0.25"
+        cpu_request_overwrite_max_allowed = "7"
+        memory_request_overwrite_max_allowed = "30Gi"
+      [[runners.kubernetes.volumes.empty_dir]]
+        name = "docker-certs"
+        mount_path = "/certs/client"
+        medium = "Memory"
+      [runners.kubernetes.node_selector]
+        agentpool = "jobs"
-- 
2.40.1

Re: Add CI configuration for Kubernetes

[Richard Henderson]

On 5/22/23 10:41, Camilla Conte wrote:
> Here's a second version (v2) of patches to support the Kubernetes runner for Gitlab CI.
> You can find the v1 thread here: https://lore.kernel.org/qemu-devel/20230407145252.32955-1-cconte@redhat.com/.

Applied to master.  With the k8s tag on the azure runner, it seems to work.


r~