* [PATCH v2] eventpoll: check the event mask before taking ep->lock
@ 2026-11-20 23:30 Jiayuan Chen
0 siblings, 0 replies; only message in thread
From: Jiayuan Chen @ 2026-11-20 23:30 UTC (permalink / raw)
To: linux-fsdevel, brauner
Cc: Jiayuan Chen, Alexander Viro, Jan Kara, linux-kernel
ep_poll_callback() takes ep->lock and only then checks whether the
event matches epi->event.events. But epi->event.events is not protected
by ep->lock, ep_modify() and the EPOLLONESHOT disarm update it under
ep->mtx.
So a wakeup for an event nobody asked for takes and releases ep->lock
for nothing. E.g. a UDP socket registered with only EPOLLIN still gets
EPOLLOUT wakeups when its tx skbs are freed.
Check the mask first and return early if the event is not wanted. The
comment in ep_modify() counts on ep_poll_callback() taking ep->lock, so
do smp_mb() and read the mask again before dropping the event, and
update that comment. Wanted events take the lock as before.
ep_set_busy_poll_napi_id() is still called for a dropped event, now
without ep->lock, so use READ_ONCE()/WRITE_ONCE() for the remaining
plain accesses to ep->napi_id.
Benchmark, based on stress-epoll.c [1]: N threads share one epoll, each
thread registers its own eventfd with EPOLLPRI, which eventfd never
raises, and loops on write() + read(). Both call ep_poll_callback(),
with EPOLLIN and EPOLLOUT. 12 vCPU VM, ops/ms:
threads before after
1 396 402
4 1514 1604
12 1264 4817
The original stress-epoll.c hits this path too: the poller read()s the
eventfds, and eventfd_read() wakes with EPOLLOUT while they are
registered with EPOLLIN. 32 vCPU VM, events/ms, average of 6 runs:
threads before after
8 1293 1315
16 1407 1422
32 1304 1431
Real case, udp rr over veth with 1000 connected sockets registered with
EPOLLIN: _raw_spin_lock_irqsave drops by ~1%.
[1] https://github.com/rouming/test-tools/blob/master/stress-epoll.c
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
---
v2:
- update the comment in ep_modify()
- keep calling ep_set_busy_poll_napi_id() for dropped events
v1:https://lore.kernel.org/linux-fsdevel/20261106044624.278356-1-jiayuan.chen@linux.dev/
---
fs/eventpoll.c | 27 ++++++++++++++++++++++-----
1 file changed, 22 insertions(+), 5 deletions(-)
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index b959af04772c..ba61f362eb08 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -687,7 +687,7 @@ static bool ep_busy_loop(struct eventpoll *ep)
*/
if (prefer_busy_poll)
napi_resume_irqs(napi_id);
- ep->napi_id = 0;
+ WRITE_ONCE(ep->napi_id, 0);
return false;
}
return false;
@@ -720,11 +720,11 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi)
* or
* Nothing to do if we already have this ID
*/
- if (!napi_id_valid(napi_id) || napi_id == ep->napi_id)
+ if (!napi_id_valid(napi_id) || napi_id == READ_ONCE(ep->napi_id))
return;
/* record NAPI ID for use in next busy poll */
- ep->napi_id = napi_id;
+ WRITE_ONCE(ep->napi_id, napi_id);
}
static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd,
@@ -1495,8 +1495,24 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v
struct eventpoll *ep = epi->ep;
__poll_t pollflags = key_to_poll(key);
unsigned long flags;
+ __poll_t events;
int ewake = 0;
+ /*
+ * epi->event.events is updated under ep->mtx, not ep->lock, so an
+ * event nobody asked for can be dropped without taking ep->lock.
+ */
+ events = READ_ONCE(epi->event.events);
+ if (pollflags && !(pollflags & (events | POLLFREE))) {
+ /* Pairs with smp_mb() in ep_modify(). */
+ smp_mb();
+ events = READ_ONCE(epi->event.events);
+ if (!(pollflags & events)) {
+ ep_set_busy_poll_napi_id(epi);
+ return !(events & EPOLLEXCLUSIVE);
+ }
+ }
+
spin_lock_irqsave(&ep->lock, flags);
ep_set_busy_poll_napi_id(epi);
@@ -2016,8 +2032,9 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi,
* we do not miss events from ep_poll_callback if an
* event occurs immediately after we call f_op->poll().
* We need this because we did not take ep->lock while
- * changing epi above (but ep_poll_callback does take
- * ep->lock).
+ * changing epi above. ep_poll_callback either takes
+ * ep->lock before it checks epi->event.events, or does
+ * smp_mb() before it drops an event without the lock.
*
* 2) We also need to ensure we do not miss _past_ events
* when calling f_op->poll(). This barrier also
--
2.43.0
^ permalink raw reply related [flat|nested] only message in thread
only message in thread, other threads:[~2026-11-20 23:30 UTC | newest]
Thread overview: (only message) (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 23:30 [PATCH v2] eventpoll: check the event mask before taking ep->lock Jiayuan Chen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.