All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jiayuan Chen <jiayuan.chen@linux.dev>
To: linux-fsdevel@vger.kernel.org, brauner@kernel.org
Cc: Jiayuan Chen <jiayuan.chen@linux.dev>,
	Alexander Viro <viro@zeniv.linux.org.uk>, Jan Kara <jack@suse.cz>,
	linux-kernel@vger.kernel.org
Subject: [PATCH v2] eventpoll: check the event mask before taking ep->lock
Date: Sat, 21 Nov 2026 07:30:36 +0800	[thread overview]
Message-ID: <20261120233037.124956-1-jiayuan.chen@linux.dev> (raw)

ep_poll_callback() takes ep->lock and only then checks whether the
event matches epi->event.events. But epi->event.events is not protected
by ep->lock, ep_modify() and the EPOLLONESHOT disarm update it under
ep->mtx.

So a wakeup for an event nobody asked for takes and releases ep->lock
for nothing. E.g. a UDP socket registered with only EPOLLIN still gets
EPOLLOUT wakeups when its tx skbs are freed.

Check the mask first and return early if the event is not wanted. The
comment in ep_modify() counts on ep_poll_callback() taking ep->lock, so
do smp_mb() and read the mask again before dropping the event, and
update that comment. Wanted events take the lock as before.

ep_set_busy_poll_napi_id() is still called for a dropped event, now
without ep->lock, so use READ_ONCE()/WRITE_ONCE() for the remaining
plain accesses to ep->napi_id.

Benchmark, based on stress-epoll.c [1]: N threads share one epoll, each
thread registers its own eventfd with EPOLLPRI, which eventfd never
raises, and loops on write() + read(). Both call ep_poll_callback(),
with EPOLLIN and EPOLLOUT. 12 vCPU VM, ops/ms:

  threads   before    after
        1      396      402
        4     1514     1604
       12     1264     4817

The original stress-epoll.c hits this path too: the poller read()s the
eventfds, and eventfd_read() wakes with EPOLLOUT while they are
registered with EPOLLIN. 32 vCPU VM, events/ms, average of 6 runs:

  threads   before    after
        8     1293     1315
       16     1407     1422
       32     1304     1431

Real case, udp rr over veth with 1000 connected sockets registered with
EPOLLIN: _raw_spin_lock_irqsave drops by ~1%.

[1] https://github.com/rouming/test-tools/blob/master/stress-epoll.c

Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
---

v2:
 - update the comment in ep_modify()
 - keep calling ep_set_busy_poll_napi_id() for dropped events
v1:https://lore.kernel.org/linux-fsdevel/20261106044624.278356-1-jiayuan.chen@linux.dev/
---
 fs/eventpoll.c | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)

diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index b959af04772c..ba61f362eb08 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -687,7 +687,7 @@ static bool ep_busy_loop(struct eventpoll *ep)
 		 */
 		if (prefer_busy_poll)
 			napi_resume_irqs(napi_id);
-		ep->napi_id = 0;
+		WRITE_ONCE(ep->napi_id, 0);
 		return false;
 	}
 	return false;
@@ -720,11 +720,11 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi)
 	 *	or
 	 * Nothing to do if we already have this ID
 	 */
-	if (!napi_id_valid(napi_id) || napi_id == ep->napi_id)
+	if (!napi_id_valid(napi_id) || napi_id == READ_ONCE(ep->napi_id))
 		return;
 
 	/* record NAPI ID for use in next busy poll */
-	ep->napi_id = napi_id;
+	WRITE_ONCE(ep->napi_id, napi_id);
 }
 
 static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd,
@@ -1495,8 +1495,24 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v
 	struct eventpoll *ep = epi->ep;
 	__poll_t pollflags = key_to_poll(key);
 	unsigned long flags;
+	__poll_t events;
 	int ewake = 0;
 
+	/*
+	 * epi->event.events is updated under ep->mtx, not ep->lock, so an
+	 * event nobody asked for can be dropped without taking ep->lock.
+	 */
+	events = READ_ONCE(epi->event.events);
+	if (pollflags && !(pollflags & (events | POLLFREE))) {
+		/* Pairs with smp_mb() in ep_modify(). */
+		smp_mb();
+		events = READ_ONCE(epi->event.events);
+		if (!(pollflags & events)) {
+			ep_set_busy_poll_napi_id(epi);
+			return !(events & EPOLLEXCLUSIVE);
+		}
+	}
+
 	spin_lock_irqsave(&ep->lock, flags);
 
 	ep_set_busy_poll_napi_id(epi);
@@ -2016,8 +2032,9 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi,
 	 *    we do not miss events from ep_poll_callback if an
 	 *    event occurs immediately after we call f_op->poll().
 	 *    We need this because we did not take ep->lock while
-	 *    changing epi above (but ep_poll_callback does take
-	 *    ep->lock).
+	 *    changing epi above.  ep_poll_callback either takes
+	 *    ep->lock before it checks epi->event.events, or does
+	 *    smp_mb() before it drops an event without the lock.
 	 *
 	 * 2) We also need to ensure we do not miss _past_ events
 	 *    when calling f_op->poll().  This barrier also
-- 
2.43.0


                 reply	other threads:[~2026-11-20 23:30 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120233037.124956-1-jiayuan.chen@linux.dev \
    --to=jiayuan.chen@linux.dev \
    --cc=brauner@kernel.org \
    --cc=jack@suse.cz \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=viro@zeniv.linux.org.uk \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.