KernelCI, gitlab testing notes

KernelCI, gitlab testing notes

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2696 bytes --]

Hi!

* 4.4 kernelci warnings

https://linux.kernelci.org/build/cip/branch/linux-4.4.y-cip/kernel/v4.4.302-cip70-98-g7f7838c92740f/

Thanks for pointer. This looks good. Most of warnings are
"net/ipv4/inet_hashtables.c:608:68: warning: suggest parentheses
around ‘+’ in operand of ‘&’ [-Wparentheses]" which is my fault and on
my TODO list.

* SMC QEMU x86-64

https://storage.kernelci.org/cip/linux-4.4.y-cip/v4.4.302-cip70-98-g7f7838c92740f/x86_64/x86_64_defconfig/gcc-10/lab-collabora/smc-qemu_x86_64.html

Ok, I'll need to know more about the config. Is it possible that qemu
runs paravirtualized -- KVM?

If yes, we are basically testing whatever hardware it happens to run
on. Not good.

If no... the "soft" cpu it runs on does not have those bugs.

* Understanding gitlab results

+https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3184699360

Ok, so what should I be looking at?

-----------------------------------
374All submitted tests were successful
375-----------------------------------
376------------------------------
377Job Summary
378------------------------------
379Job #763143 Finished. Job health: Complete. URL: https://lava.ciplatform.org/scheduler/job/763143
380Job #763147 Finished. Job health: Complete. URL: https://lava.ciplatform.org/scheduler/job/763147
382

I have see. "Job health: incomplete" and that indicated problems. I
see "All submitted tests were successful". I guess that's ok.

https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3141003858

Now that's pretty evil:

* 0_spectre-meltdown-checker-test.CVE-2018-12126 [fail]
266* 0_spectre-meltdown-checker-test.CVE-2018-3646 [pass]
...
288-----------------------------------
289All submitted tests were successful
290-----------------------------------
291

First, make it stand out visually. [pass] => [ok] and [fail] =>
[FAILURE] or something like that.

Second, saying all tests were successful (line 289) when there's
failure is ... confusing.

* LTP failures

I'm not sure where we run this or how. Anyway.

https://lava.ciplatform.org/scheduler/job/763461

I picked up one failure randomly, and that's config failure, not
kernel failure: utimensat01 1 TBROK: can't read /etc/sudoers

Not sure what is going there: quotactl01. Do we have quotas enabled?
syslog01 and friends is also failing. Is syslog configured correctly?

I guess best way would be to run ltp on 4.4-mainline, 4.4.302, 4.4-cip
and compare the results.

Best regards,
								Pavel
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [cip-dev] KernelCI, gitlab testing notes

[Bezdeka, Florian]

Hi all,

On Thu, 2022-10-20 at 18:08 +0200, Pavel Machek via lists.cip-
project.org wrote:
> Hi!
> 
> * 4.4 kernelci warnings
> 
> https://linux.kernelci.org/build/cip/branch/linux-4.4.y-cip/kernel/v4.4.302-cip70-98-g7f7838c92740f/
> 
> Thanks for pointer. This looks good. Most of warnings are
> "net/ipv4/inet_hashtables.c:608:68: warning: suggest parentheses
> around ‘+’ in operand of ‘&’ [-Wparentheses]" which is my fault and on
> my TODO list.

Great, so the filtering seems to work as expected.

> 
> * SMC QEMU x86-64
> 
> https://storage.kernelci.org/cip/linux-4.4.y-cip/v4.4.302-cip70-98-g7f7838c92740f/x86_64/x86_64_defconfig/gcc-10/lab-collabora/smc-qemu_x86_64.html
> 
> Ok, I'll need to know more about the config. Is it possible that qemu
> runs paravirtualized -- KVM?

According to line 174 of the referenced job log above /dev/kvm is
mounted into the container running what they seem to call "qemu
emulator" and -enable-kvm is given.

Depending on the concrete hardware this test is scheduled on, it might
deliver different results. That seems to be a general issue. I can try
to address that.

Is the 4.4 series missing the backport of the mitigation for this CVE
or is that really a "can be fixed by microcode only" thing? I don't
want to silence a real issue. My question is: If we would run a newer
kernel, would we still be affected on the same (virtual) hardware?

> 
> If yes, we are basically testing whatever hardware it happens to run
> on. Not good.
> 
> If no... the "soft" cpu it runs on does not have those bugs.
> 
> * Understanding gitlab results
> 

[snip, I don't know the gitlab infrastructure]

RE: KernelCI, gitlab testing notes

[Chris Paterson]

Hello Pavel,

> From: Pavel Machek <pavel@denx.de>
> Sent: 20 October 2022 17:09
> 
> Hi!
> 

[...]

> 
> * Understanding gitlab results
> 
> +https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3184699360
> 
> Ok, so what should I be looking at?
> 
> -----------------------------------
> 374All submitted tests were successful
> 375-----------------------------------
> 376------------------------------
> 377Job Summary
> 378------------------------------
> 379Job #763143 Finished. Job health: Complete. URL:
> https://lava.ciplatform.org/scheduler/job/763143
> 380Job #763147 Finished. Job health: Complete. URL:
> https://lava.ciplatform.org/scheduler/job/763147
> 382
> 
> I have see. "Job health: incomplete" and that indicated problems. I
> see "All submitted tests were successful". I guess that's ok.
> 
> https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3141003858
> 
> Now that's pretty evil:
> 
> * 0_spectre-meltdown-checker-test.CVE-2018-12126 [fail]
> 266* 0_spectre-meltdown-checker-test.CVE-2018-3646 [pass]
> ...
> 288-----------------------------------
> 289All submitted tests were successful
> 290-----------------------------------
> 291

Yes, this isn't clear.
What it should say is "All submitted test jobs completed".
Test job = LAVA test job that can contain many test cases

> 
> First, make it stand out visually. [pass] => [ok] and [fail] =>
> [FAILURE] or something like that.
> 
> Second, saying all tests were successful (line 289) when there's
> failure is ... confusing.

Agreed. I've updated the wording in this commit [0], part of a dev branch [1] I'm working on.

[0] https://gitlab.com/cip-project/cip-testing/linux-cip-ci/-/commit/f273d55b1c78d80f2e44f10ca22334e64eeb4613
[1] https://gitlab.com/cip-project/cip-testing/linux-cip-ci/-/commits/patersonc/check-test-results

Can you look at the new output in jobs in this pipeline [2]?
At the moment I've set the whole GitLab CI job to fail if there are any test cases that fail.
However I really don't like this approach as it means that merge requests in GitLab can never be merged if there are test cases that always fail, as all jobs need to pass.
E.g. SMC tests where there isn't a mitigation for a particular CVE [3]

[2] https://gitlab.com/cip-project/cip-testing/linux-cip-ci/-/pipelines/676545968
[3] https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3141003858#L262

To avoid this issue we could create a separate job that runs at the end of all testing, that goes through and checks for failures.
You'd then only have to look at one GitLab CI job to get the full report.
I'd still need this job to return a green tick though if the actual job runs successfully, even if there are failed test cases.

Does this approach sound okay?

Of course, perhaps we should just switch over to KernelCI fully and make use of the features they already support such as test regression detections.

Kind regards, Chris

> 
> * LTP failures
> 
> I'm not sure where we run this or how. Anyway.
> 
> https://lava.ciplatform.org/scheduler/job/763461
> 
> I picked up one failure randomly, and that's config failure, not
> kernel failure: utimensat01 1 TBROK: can't read /etc/sudoers
> 
> Not sure what is going there: quotactl01. Do we have quotas enabled?
> syslog01 and friends is also failing. Is syslog configured correctly?
> 
> I guess best way would be to run ltp on 4.4-mainline, 4.4.302, 4.4-cip
> and compare the results.
> 
> Best regards,
> 								Pavel
> --
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Still random failures at gitlab

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 6004 bytes --]

Hi!

I'm still getting random failures at gitlab. Is the info below enough
/ can I retry the tests now to get real results?

Thanks and best regards,
									Pavel

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/pipelines/679070235

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/jobs/3238998916

Running with gitlab-runner 15.4.0 (43b2dc3d)
2  on 1665142213-cip-project-small-v9-project-gitlab-runner-85dctt8xr Uf_MsYPS
3  feature flags: FF_KUBERNETES_HONOR_ENTRYPOINT:true
4
Resolving secrets
00:00
6
Preparing the "kubernetes" executor
00:00
7Using Kubernetes namespace: gitlab
8Using Kubernetes executor with image registry.gitlab.com/cip-project/cip-testing/linux-cip-ci:test-v3 ...
9Using attach strategy to execute scripts...
11
Preparing environment
03:07
12Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
13	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
14Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
15	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
16Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
17	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
18Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
19	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
20Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
21	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
22Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
23	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
24Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
25	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
26Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
27	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
28Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
29	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
30Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
31	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
32Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
33	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
34Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
35	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
36Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
37	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
38Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
39	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
40Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
41	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
42Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-0nzck6 to be running, status is Pending
43	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling."
44Waiting for pod gitlab/runner-ufmsyps-project



-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Still random failures at gitlab

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 3382 bytes --]

Hi!

> I'm still getting random failures at gitlab. Is the info below enough
> / can I retry the tests now to get real results?
> 
> Thanks and best regards,
> 									Pavel

4.9 testing has different failure mode:

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/pipelines/679070249

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/jobs/3238999016

Running with gitlab-runner 15.4.0 (43b2dc3d)
2  on 1665142213-cip-project-small-v9-project-gitlab-runner-85dctt8xr Uf_MsYPS
3  feature flags: FF_KUBERNETES_HONOR_ENTRYPOINT:true
4
Resolving secrets
00:00
6
Preparing the "kubernetes" executor
00:00
7Using Kubernetes namespace: gitlab
8Using Kubernetes executor with image registry.gitlab.com/cip-project/cip-testing/linux-cip-ci:test-v3 ...
9Using attach strategy to execute scripts...
11
Preparing environment
00:06
12Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-8tdl4w to be running, status is Pending
13	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
14	ContainersNotReady: "containers with unready status: [build helper]"
15	ContainersNotReady: "containers with unready status: [build helper]"
16Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-8tdl4w to be running, status is Pending
17	ContainersNotReady: "containers with unready status: [build helper]"
18	ContainersNotReady: "containers with unready status: [build helper]"
19Running on runner-ufmsyps-project-14394223-concurrent-8tdl4w via 1665142213-cip-project-small-v9-project-gitlab-runner-85dctt8xr...
21
Getting source from Git repository
00:01
22Skipping Git repository setup
23Skipping Git checkout
24Skipping Git submodules setup
26
Downloading artifacts
01:32
27Downloading artifacts for build:arm_cip_bbb_defconfig (3238998979)...
28ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://gitlab.com/api/v4/jobs/3238998979/artifacts?direct_download=true: Get "https://gitlab.com/api/v4/jobs/3238998979/artifacts?direct_download=true": dial tcp: i/o timeout  id=3238998979 token=BszELFyL
29WARNING: Retrying...                                error=invalid argument
30ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://gitlab.com/api/v4/jobs/3238998979/artifacts?: Get "https://gitlab.com/api/v4/jobs/3238998979/artifacts?": dial tcp: i/o timeout  id=3238998979 token=BszELFyL
31WARNING: Retrying...                                error=invalid argument
32ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://gitlab.com/api/v4/jobs/3238998979/artifacts?: Get "https://gitlab.com/api/v4/jobs/3238998979/artifacts?": dial tcp: i/o timeout  id=3238998979 token=BszELFyL
33FATAL: invalid argument                            
35
Uploading artifacts for failed job
00:00
36Uploading artifacts...
37WARNING: output: no matching files. Ensure that the artifact path is relative to the working directory 
38ERROR: No files to upload                          
40
Cleaning up project directory and file based variables
00:01
42ERROR: Job failed: command terminated with exit code 1





-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

RE: Still random failures at gitlab

[Chris Paterson]

Hi Pavel,

> From: Pavel Machek <pavel@denx.de>
> Sent: 27 October 2022 20:25
> 
> Hi!
> 
> I'm still getting random failures at gitlab. Is the info below enough
> / can I retry the tests now to get real results?
> 
> Thanks and best regards,
> 									Pavel
> 
> https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-
> /pipelines/679070235
> 
> https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-
> /jobs/3238998916

Thank you for the report and links.
I noticed similar issues yesterday as well.

I'll let you know when we've found the issue behind it.

Kind regards, Chris

> 
> Running with gitlab-runner 15.4.0 (43b2dc3d)
> 2  on 1665142213-cip-project-small-v9-project-gitlab-runner-85dctt8xr
> Uf_MsYPS
> 3  feature flags: FF_KUBERNETES_HONOR_ENTRYPOINT:true
> 4
> Resolving secrets
> 00:00
> 6
> Preparing the "kubernetes" executor
> 00:00
> 7Using Kubernetes namespace: gitlab
> 8Using Kubernetes executor with image registry.gitlab.com/cip-project/cip-
> testing/linux-cip-ci:test-v3 ...
> 9Using attach strategy to execute scripts...
> 11
> Preparing environment
> 03:07
> 12Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 13	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 14Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 15	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 16Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 17	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 18Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 19	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 20Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 21	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 22Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 23	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 24Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 25	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 26Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 27	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 28Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 29	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 30Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 31	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 32Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 33	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 34Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 35	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 36Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 37	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 38Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 39	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 40Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 41	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 42Waiting for pod gitlab/runner-ufmsyps-project-14394223-concurrent-
> 0nzck6 to be running, status is Pending
> 43	Unschedulable: "0/1 nodes are available: 1 node(s) had untolerated
> taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are
> available: 1 Preemption is not helpful for scheduling."
> 44Waiting for pod gitlab/runner-ufmsyps-project
> 
> 
> 
> --
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany