* [PATCH] eventpoll: check the event mask before taking ep->lock
@ 2026-11-06 4:46 Jiayuan Chen
2026-11-20 8:53 ` Christian Brauner
0 siblings, 1 reply; 3+ messages in thread
From: Jiayuan Chen @ 2026-11-06 4:46 UTC (permalink / raw)
To: linux-fsdevel, brauner
Cc: Jiayuan Chen, Alexander Viro, Jan Kara, linux-kernel
ep_poll_callback() takes ep->lock and only then checks whether the
event matches epi->event.events. But epi->event.events is not protected
by ep->lock, ep_modify() and the EPOLLONESHOT disarm update it under
ep->mtx.
So a wakeup for an event nobody asked for takes and releases ep->lock
for nothing. E.g. a UDP socket registered with EPOLLIN only gets an
EPOLLOUT wakeup for every freed tx skb.
Check the mask first and return early if the event is not wanted. The
comment in ep_modify() counts on ep_poll_callback() taking ep->lock, so
do smp_mb() and read the mask again before dropping the event. Wanted
events take the lock as before.
Benchmark, based on stress-epoll.c [1]: N threads share one epoll, each
thread registers its own eventfd with EPOLLPRI, which eventfd never
raises, and loops on write() + read(). Both call ep_poll_callback(),
with EPOLLIN and EPOLLOUT. 12 vCPU VM, ops/ms:
threads before after
1 396 402
4 1514 1604
12 1264 4817
The original stress-epoll.c hits this path too: the poller read()s the
eventfds, and eventfd_read() wakes with EPOLLOUT while they are
registered with EPOLLIN. 32 vCPU VM, events/ms, average of 6 runs:
threads before after
8 1293 1315
16 1407 1422
32 1304 1431
Real case, udp rr over veth with 1000 connected sockets registered with
EPOLLIN: _raw_spin_lock_irqsave drops by ~1%.
[1] https://github.com/rouming/test-tools/blob/master/stress-epoll.c
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
---
fs/eventpoll.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index e0c4bf88a838..29e034e3a295 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -1495,8 +1495,22 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v
struct eventpoll *ep = epi->ep;
__poll_t pollflags = key_to_poll(key);
unsigned long flags;
+ __poll_t events;
int ewake = 0;
+ /*
+ * epi->event.events is updated under ep->mtx, not ep->lock, so an
+ * event nobody asked for can be dropped without taking ep->lock.
+ */
+ events = READ_ONCE(epi->event.events);
+ if (pollflags && !(pollflags & (events | POLLFREE))) {
+ /* Pairs with smp_mb() in ep_modify(). */
+ smp_mb();
+ events = READ_ONCE(epi->event.events);
+ if (!(pollflags & events))
+ return !(events & EPOLLEXCLUSIVE);
+ }
+
spin_lock_irqsave(&ep->lock, flags);
ep_set_busy_poll_napi_id(epi);
--
2.43.0
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] eventpoll: check the event mask before taking ep->lock
2026-11-06 4:46 [PATCH] eventpoll: check the event mask before taking ep->lock Jiayuan Chen
@ 2026-11-20 8:53 ` Christian Brauner
2026-11-20 9:28 ` Jiayuan Chen
0 siblings, 1 reply; 3+ messages in thread
From: Christian Brauner @ 2026-11-20 8:53 UTC (permalink / raw)
To: Jiayuan Chen; +Cc: linux-fsdevel, Alexander Viro, Jan Kara, linux-kernel
On Fri, Nov 06, 2026 at 12:46:23PM +0800, Jiayuan Chen wrote:
> Check the mask first and return early if the event is not wanted. The
> comment in ep_modify() counts on ep_poll_callback() taking ep->lock, so
> do smp_mb() and read the mask again before dropping the event. Wanted
> events take the lock as before.
Then this should update the comment too:
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index 133a764..8a564d1 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -2029,9 +2029,8 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi,
* 1) Flush epi changes above to other CPUs. This ensures
* we do not miss events from ep_poll_callback if an
* event occurs immediately after we call f_op->poll().
- * We need this because we did not take ep->lock while
- * changing epi above (but ep_poll_callback does take
- * ep->lock).
+ * Pairs with the smp_mb() in ep_poll_callback() before it
+ * drops an event that isn't in epi->event.events.
*
* 2) We also need to ensure we do not miss _past_ events
* when calling f_op->poll(). This barrier also
> + if (!(pollflags & events))
> + return !(events & EPOLLEXCLUSIVE);
> + }
> +
> spin_lock_irqsave(&ep->lock, flags);
>
> ep_set_busy_poll_napi_id(epi);
This also skips ep_set_busy_poll_napi_id() for unwanted events now.
So with busy polling an EPOLLOUT wakeup from sock_wfree() for a socket
that only asked for EPOLLIN won't update ep->napi_id anymore, e.g.,
after ep_busy_loop() reset it on timeout.
Is that intended?
--
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] eventpoll: check the event mask before taking ep->lock
2026-11-20 8:53 ` Christian Brauner
@ 2026-11-20 9:28 ` Jiayuan Chen
0 siblings, 0 replies; 3+ messages in thread
From: Jiayuan Chen @ 2026-11-20 9:28 UTC (permalink / raw)
To: Christian Brauner; +Cc: linux-fsdevel, Alexander Viro, Jan Kara, linux-kernel
On 11/20/26 4:53 PM, Christian Brauner wrote:
> On Fri, Nov 06, 2026 at 12:46:23PM +0800, Jiayuan Chen wrote:
>> Check the mask first and return early if the event is not wanted. The
>> comment in ep_modify() counts on ep_poll_callback() taking ep->lock, so
>> do smp_mb() and read the mask again before dropping the event. Wanted
>> events take the lock as before.
>
> Then this should update the comment too:
>
Thanks Christian. You are right.
> diff --git a/fs/eventpoll.c b/fs/eventpoll.c
> index 133a764..8a564d1 100644
> --- a/fs/eventpoll.c
> +++ b/fs/eventpoll.c
> @@ -2029,9 +2029,8 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi,
> * 1) Flush epi changes above to other CPUs. This ensures
> * we do not miss events from ep_poll_callback if an
> * event occurs immediately after we call f_op->poll().
> - * We need this because we did not take ep->lock while
> - * changing epi above (but ep_poll_callback does take
> - * ep->lock).
> + * Pairs with the smp_mb() in ep_poll_callback() before it
> + * drops an event that isn't in epi->event.events.
> *
> * 2) We also need to ensure we do not miss _past_ events
> * when calling f_op->poll(). This barrier also
>
>> + if (!(pollflags & events))
>> + return !(events & EPOLLEXCLUSIVE);
>> + }
>> +
>> spin_lock_irqsave(&ep->lock, flags);
>>
>> ep_set_busy_poll_napi_id(epi);
>
> This also skips ep_set_busy_poll_napi_id() for unwanted events now.
> So with busy polling an EPOLLOUT wakeup from sock_wfree() for a socket
> that only asked for EPOLLIN won't update ep->napi_id anymore, e.g.,
> after ep_busy_loop() reset it on timeout.
>
> Is that intended?
>
I missed that. I have no data showing that skipping it is fine, We should keep
old behaviour and calls ep_set_busy_poll_napi_id() before the early return.
It then runs without ep->lock, so the read and the write of ep->napi_id
in it become READ_ONCE()/WRITE_ONCE(). The other readers already use
READ_ONCE() outside the lock.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-20 9:28 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-06 4:46 [PATCH] eventpoll: check the event mask before taking ep->lock Jiayuan Chen
2026-11-20 8:53 ` Christian Brauner
2026-11-20 9:28 ` Jiayuan Chen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.