Re: [REGRESSION] osnoise: "eventpoll: Replace rwlock with spinlock" causes ~50us noise spikes on isolated PREEMPT_RT cores

Ionut Nechita (Wind River)
2026-04-01T17:01:51Z

From: Ionut Nechita <ionut.nechita@windriver.com>

Crystal, Jan, Florian, thanks for the detailed feedback. I've redone
all testing addressing each point raised. All tests below use HT
disabled (sibling cores offlined), as Jan requested.

Setup:
- Hardware: Intel Xeon Gold 6338N (Ice Lake, single socket,
32 cores, HT disabled via sibling cores offlined)
- Boot: nohz_full=1-16 isolcpus=nohz,domain,managed_irq,1-16
rcu_nocbs=1-31 kthread_cpus=0 irqaffinity=17-31
iommu=pt nmi_watchdog=0 intel_pstate=none skew_tick=1
- eosnoise run with: ./osnoise -c 1-15
- Duration: 120s per test

Tested kernels (all vanilla, built from upstream sources):
- 6.18.20-vanilla (non-RT, PREEMPT_DYNAMIC)
- 6.18.20-vanilla (PREEMPT_RT, with and without rwlock revert)
- 7.0.0-rc6-next-20260331 (PREEMPT_RT, with and without rwlock revert)

I tested 6 configurations to isolate the exact failure mode:

# Kernel Config Tool Revert Result
-- --------------- -------- --------------- ------- ----------------
1 6.18.20 non-RT eosnoise no clean (100%)
2 6.18.20 RT eosnoise no D state (hung)
3 6.18.20 RT eosnoise yes clean (100%)
4 6.18.20 RT kernel osnoise no clean (99.999%)
5 7.0-rc6-next RT eosnoise no 93% avail, 57us
6 7.0-rc6-next RT eosnoise yes clean (99.99%)

Key findings:

1. On 6.18.20-rt with spinlock, eosnoise hangs permanently in D state.

The process blocks in do_epoll_ctl() during perf_buffer__new() setup
(libbpf's perf_event_open + epoll_ctl loop). strace shows progressive
degradation as fds are added to the epoll instance:

CPU 0-13: epoll_ctl ~8 us (normal)
CPU 14: epoll_ctl 16 ms (2000x slower)
CPU 15: epoll_ctl 80 ms (10000x slower)
CPU 16: epoll_ctl 80 ms
CPU 17: epoll_ctl 20 ms
CPU 18: epoll_ctl -- hung, never returns --

Kernel stack of the hung process (3+ minutes in D state):

[<0>] do_epoll_ctl+0xa57/0xf20
[<0>] __x64_sys_epoll_ctl+0x5d/0xa0
[<0>] do_syscall_64+0x7c/0xe30
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

2. On 7.0-rc6-next-rt with spinlock, eosnoise runs but with severe
noise. The difference from 6.18 is likely additional fixes in
linux-next that prevent the complete deadlock but not the contention.

3. Kernel osnoise tracer (test #4) shows zero noise on the same
6.18.20-rt+spinlock kernel where eosnoise hangs. This confirms the
issue is specifically in the epoll rt_mutex path, not in osnoise
measurement methodology.

Kernel osnoise output (6.18.20-rt, spinlock, no revert):
99.999% availability, 1-4 ns max noise, RES=6 total in 120s

4. Non-RT kernel (test #1) with the same spinlock change shows zero
noise. This confirms the issue is the spinlock-to-rt_mutex conversion
on PREEMPT_RT, not the spinlock change itself.

IRQ deltas on isolated CPU1 (120s):

6.18.20-rt 6.18.20-rt 6.18.20 6.18.20-rt
spinlock rwlock(rev) non-RT kernel osnoise
RES (IPI): (D state) 3 1 6
LOC (timer): (D state) 3,325 1,185 245
IWI (irq work): (D state) 565,988 1,433 121

7.0-rc6-rt 7.0-rc6-rt
spinlock rwlock(rev)
RES (IPI): 330,000+ 2
LOC (timer): 120,585 120,585
IWI (irq work): 585,785 585,785

The mechanism, refined:

Crystal was right that this is specific to the BPF perf_event_output +
epoll pattern, not any arbitrary epoll user. I verified this: a plain
perf_event_open + epoll_ctl program without BPF does not trigger the
issue.

What triggers it is libbpf's perf_buffer__new(), which creates one
PERF_COUNT_SW_BPF_OUTPUT perf_event per CPU, mmaps the ring buffer,
and adds all fds to a single epoll instance. When BPF programs are
attached to high-frequency tracepoints (irq_handler_entry/exit,
softirq_entry/exit, sched_switch), every interrupt on every CPU calls
bpf_perf_event_output() which invokes ep_poll_callback() under
ep->lock.

On PREEMPT_RT, ep->lock is an rt_mutex. With 15+ CPUs generating
callbacks simultaneously into the same epoll instance, the rt_mutex
PI mechanism creates unbounded contention. On 6.18 this results in
a permanent D state hang. On 7.0 it results in ~330,000 reschedule
IPIs hitting isolated cores over 120 seconds (~2,750/s per core).

With rwlock, ep_poll_callback() uses read_lock which allows concurrent
readers without cross-CPU contention — the callbacks execute in
parallel without generating IPIs.

This pattern (BPF tracepoint programs + perf ring buffer + epoll) is
the standard architecture used by BCC tools (opensnoop, execsnoop,
biolatency, tcpconnect, etc.), bpftrace, and any libbpf-based
observability tool. A permanent D state hang when running such tools
on PREEMPT_RT is a significant regression.

I'm not proposing a specific fix -- the previous suggestions
(raw_spinlock trylock, lockless path) were rightly rejected. But the
regression exists and needs to be addressed. The ep->lock contention
under high-frequency BPF callbacks on PREEMPT_RT is a new problem
that the rwlock->spinlock conversion introduced.

Separate question: could eosnoise itself be improved to avoid this
contention? For example, using one epoll instance per CPU instead of
a single shared one, or using BPF ring buffer (BPF_MAP_TYPE_RINGBUF)
instead of the per-cpu perf buffer which requires epoll. If the
consensus is that the kernel side is working as intended and the tool
should adapt, I'd like to understand what the recommended pattern is
for BPF observability tools on PREEMPT_RT.

Ionut