Re: AWS CI Oddities

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

On Tue, Oct 28, 2025 at 10:41:32AM +0100, Camilla Conte wrote:
> Hi Stefan,
> 
> This can happen because the worker node is out of resources (likely
> memory) for the running pods.
> We can set a memory minimum for all pods, or we can set it per-job.
> 
> If there are some specific jobs that are memory-hungry, please set
> KUBERNETES_MEMORY_REQUEST in the job variables.
> https://docs.gitlab.com/runner/executors/kubernetes/#overwrite-container-resources
> 
> Else, I can set a global default.

Hi Camilla,
Sizing each CI job requires memory metrics that I don't have. Does AWS give
you any insight into how much free RAM is available on k8s worker nodes
over time?

Gemini suggests that Kubernetes Metric Server can be enabled and then
`kubectl top` can be used to monitor Pod resource usage. Alternatively,
CloudWatch Container Insights can be enabled in AWS to get Pod memory
usage.

Once we have data we can either raise the limit (assuming there are no
huge outliers) or label the outlier jobs (to avoid overprovisioning the
normal jobs).

By the way, I haven't figured out how to access the AWS resources from
my own AWS account. I only see the total cost from our accounts, but
don't have visibility or control over what is running. So I'm unable to
investigate the EKS cluster myself at the moment.

Thanks,
Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]