From: Ihor Solodrai <ihor.solodrai@linux.dev>
To: Ilya Leoshkevich <iii@linux.ibm.com>
Cc: bpf <bpf@vger.kernel.org>, Alexei Starovoitov <ast@kernel.org>,
Andrii Nakryiko <andrii@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Martin KaFai Lau <martin.lau@linux.dev>
Subject: Re: [BPF CI] OOMs on s390x runners
Date: Thu, 5 Feb 2026 15:50:40 -0800 [thread overview]
Message-ID: <019b0f02-d177-457f-89e2-96727ee48cff@linux.dev> (raw)
In-Reply-To: <62cab7f1-06ea-434b-99e5-ba8b1a7d7eb5@linux.ibm.com>
On 1/21/26 2:56 PM, Ilya Leoshkevich wrote:
>
> On 1/20/26 18:59, Ihor Solodrai wrote:
>> Hi Ilya,
>>
>> The BPF selftests regularly fail on s390x runners with OOMs.
>>
>> The s390x hosts that I've been maintaining have 16G of memory, hosting
>> 2 runners each. VMs (qemu instances) running the tests currently get
>> 5G of memory each.
>>
>> I noticed that the s390x runners didn't have swap set up, so I added
>> an ansible config to set up a swapfile. This seems to have helped with
>> OOM failures.
>>
>> https://github.com/libbpf/ci/commit/8767dc05ab84c88da198af3c651511e731ddbac7
>>
>> I spot checked a couple of recent failures, and all of them happened
>> on the "ebpf1-worker-*" runners that you've been maintaining IIRC.
>> Here is a couple of examples:
>> - https://github.com/kernel-patches/bpf/actions/runs/21178791628/job/60916006746
>> - https://github.com/kernel-patches/bpf/actions/runs/21163448380/job/60863171062
>> - https://github.com/kernel-patches/bpf/actions/runs/21157785072/job/60846570064
>>
>> Could you please re-provision the hosts with the swap setup?
>> Let's see if that helps.
>>
>> Thanks!
>
> Hi Ihor,
>
>
> Thanks for the scripts!
>
> I reprovisioned all 5 builders, everything looks healthy.
>
> I will check on them again tomorrow morning.
Hi Ilya.
We ended up disabling KASAN for s390x and that helped with OOMs.
I noticed a couple "no space left on device" errors on your runners
today, could you please take a look? Example:
https://github.com/kernel-patches/bpf/actions/runs/21732338737/job/62690585484
Thanks!
>
>
> Best regards,
>
> Ilya
>
next prev parent reply other threads:[~2026-02-05 23:50 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-01-20 17:59 [BPF CI] OOMs on s390x runners Ihor Solodrai
2026-01-21 22:56 ` Ilya Leoshkevich
2026-02-05 23:50 ` Ihor Solodrai [this message]
2026-02-10 21:23 ` Ilya Leoshkevich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=019b0f02-d177-457f-89e2-96727ee48cff@linux.dev \
--to=ihor.solodrai@linux.dev \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=iii@linux.ibm.com \
--cc=martin.lau@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox