Re: [REGRESSION] osnoise: "eventpoll: Replace rwlock with spinlock" causes ~50us noise spikes on isolated PREEMPT_RT cores

Ionut Nechita (Wind River)
2026-03-27T18:37:20Z

From: Ionut Nechita <ionut.nechita@windriver.com>

On Thu, 2026-03-27 at 08:44 +0100, Florian Bezdeka wrote:
> A revert alone is not an option as it would bring back [1] and [2]
> for all LTS releases that did not receive [3].

Florian, Crystal, thanks for the feedback.

I understand the revert concern regarding the CFS throttle deadlock.
However, I want to clarify that the noise regression on isolated cores
is a separate issue from the deadlock fixed by [3], and it remains
unfixed even on linux-next which has [3] merged or not.

I've done extensive testing across multiple kernels to identify the
exact mechanism. Here are the results.

Tool: eBPF-based osnoise tracer (https://gitlab.com/rt-linux-tools/eosnoise)
which uses perf_event_open() + epoll on each monitored CPU, combined
with /proc/interrupts delta measurement.

Setup:
- Hardware: x86_64, SMT/HT enabled (CPUs 0-63)
- Boot: nohz_full=1-16,33-48 isolcpus=nohz,domain,managed_irq,1-16,33-48
rcu_nocbs=1-31,33-63 kthread_cpus=0,32 irqaffinity=17-31,49-63
- Duration: 120s per test

IRQ delta on isolated CPUs (representative CPU1, 120s sample):

6.12.79-rt 6.18.20-rt 7.0-rc5-next-rt 6.18.19-rt 7.0-rc5-next-rt
spinlock spinlock spinlock rwlock(rev) rwlock(rev)
RES (IPI): 324,279 323,864 321,594 0 1
LOC (timer): 50,827 53,995 59,793 125,791 125,791
IWI (irq work): 359,590 357,289 357,798 588,245 588,245

osnoise on isolated CPUs (per 950ms sample):

6.12.79-rt 6.18.20-rt 7.0-rc5-next-rt 6.18.19-rt 7.0-rc5-next-rt
spinlock spinlock spinlock rwlock(rev) rwlock(rev)
MAX noise (ns): ~57,000 ~57,000 ~57,000 ~9 ~140
IRQ/sample: ~7,280 ~7,030 ~7,020 ~1 ~961
Thread/sample: ~6,330 ~6,090 ~6,090 ~1 ~1
Availability: ~93.5% ~93.5% ~93.5% ~100% ~99.99%

The smoking gun is RES (reschedule IPI): ~322,000 on every isolated CPU
in 120 seconds with the spinlock, essentially zero with rwlock. That is
~2,680 reschedule IPIs per second hitting each isolated core.

The mechanism: on PREEMPT_RT, spinlock_t becomes rt_mutex. When the
eBPF osnoise tool (or any BPF/perf tool using epoll) calls
epoll_ctl(EPOLL_CTL_ADD) for perf events on each CPU, ep_poll_callback()
runs under ep->lock (now rt_mutex) in IRQ context. The rt_mutex PI
mechanism sends reschedule IPIs to wake waiters, which hit isolated
cores. With rwlock, read_lock() in ep_poll_callback() does not generate
cross-CPU IPIs.

Note on the tool: the eBPF osnoise tracer itself creates epoll activity
on all CPUs via perf_event_open() + epoll_ctl(). This is representative
of real-world scenarios where any BPF/perf monitoring tool, or system
services like systemd/journald using epoll, would trigger the same
regression on isolated cores.

When using the kernel's built-in osnoise tracer (which does not use
epoll), isolated cores show 1ns noise / 1 IRQ per sample on all kernels
regardless of spinlock vs rwlock — confirming the noise source is
specifically the epoll spinlock contention path.

Key finding: the task-based CFS throttle series [3] (Aaron Lu, merged
in 6.18/linux-next) does NOT fix this issue. The regression is identical
on 6.12, 6.18, and linux-next 7.0-rc5 with the spinlock. Only reverting
to rwlock eliminates it.

To answer Crystal's question "when would you ever reach that path on an
isolated CPU?" — the answer is: any tool or service that uses
perf_event_open() + epoll across all CPUs (BPF tools, perf, monitoring
agents) will trigger ep_poll_callback() on isolated CPUs. On RT with the
spinlock, this generates ~2,680 reschedule IPIs/s per isolated core.

The eventpoll spinlock noise regression needs its own fix — perhaps
a lockless path in ep_poll_callback() for the RT case, or
converting ep->lock to a raw_spinlock with trylock semantics to avoid
the rt_mutex IPI overhead.

Ionut