* [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
@ 2026-11-11 21:02 Mani Tofigh via B4 Relay
2026-11-11 23:34 ` Oleg Nesterov
` (2 more replies)
0 siblings, 3 replies; 12+ messages in thread
From: Mani Tofigh via B4 Relay @ 2026-11-11 21:02 UTC (permalink / raw)
To: Christian Brauner, Alexander Viro
Cc: Jan Kara, Oleg Nesterov, linux-fsdevel, linux-kernel, tz2294,
stable, Mani Tofigh
From: Mani Tofigh <mt4022@columbia.edu>
A task blocked in read() on a pipe can sleep while the pipe holds data,
when an epoll(EPOLLEXCLUSIVE) watcher of the read end was added before
it. The watcher's entry on ->rd_wait is exclusive and sits ahead of the
reader, so a write into an empty pipe wakes only the watcher. The
watcher reads without sleeping in anon_pipe_read() and doesn't pass the
wakeup on. Since commit f1cebdec9739 ("pipe: only enable the extra
wake_up(rd_wait) for EPOLLET consumers") the next writes into the
non-empty pipe don't wake anyone anymore:
Task E Task A Task C
epoll_wait(ep, -1)
/* EPOLLEXCLUSIVE */
read(p[0])
write(p[1], "1")
/* wakes E only */
write(p[1], "2")
/* no wakeup */
read(p[0], 1 byte)
exit()
/* still asleep, "2" is in the pipe */
A then sleeps until the next write into an empty pipe, if ever. Before
that commit any pipe_poll() enabled the per-write wakeups. Those
reached A because an EPOLLEXCLUSIVE entry only takes a wakeup while a
task sleeps in epoll_wait().
So EPOLLEXCLUSIVE consumers depend on the per-write wakeups too, for a
different reason than EPOLLET. Enable them for EPOLLEXCLUSIVE as well
and say why in the comment. With the fix, A reads the second byte.
This bug was identified and confirmed with the help of a deterministic
hypervisor currently under development. The C reproducer below however
hits it within about 2s.
Fixes: f1cebdec9739 ("pipe: only enable the extra wake_up(rd_wait) for EPOLLET consumers")
Cc: stable@vger.kernel.org
Signed-off-by: Mani Tofigh <mt4022@columbia.edu>
---
Reproducer: https://github.com/manitofigh/files/blob/main/linux-repros/pipeexcl.c
There are a couple of older gaps like this (a single write, and the
write side with EPOLLOUT|EPOLLEXCLUSIVE). I left both alone here though.
---
fs/pipe.c | 6 +++++-
include/linux/pipe_fs_i.h | 2 +-
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/fs/pipe.c b/fs/pipe.c
index 4c1122597a3c..8ad792d300c9 100644
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -700,9 +700,13 @@ pipe_poll(struct file *filp, poll_table *wait)
* and the ->pseudo_edgetrigger check in anon_pipe_write().
* Currently io_uring sets EPOLLET for multishot polls, so it gets the
* same behaviour.
+ *
+ * EPOLLEXCLUSIVE needs them too. Its entry is an exclusive waiter on
+ * ->rd_wait that can take the wakeup meant for a blocked reader, and
+ * unlike a woken reader in anon_pipe_read() it doesn't pass it on.
*/
if ((filp->f_mode & FMODE_READ) &&
- wait && (wait->_key & EPOLLET) &&
+ wait && (wait->_key & (EPOLLET | EPOLLEXCLUSIVE)) &&
unlikely(!READ_ONCE(pipe->pseudo_edgetrigger)))
WRITE_ONCE(pipe->pseudo_edgetrigger, true);
diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h
index 95f6236c5c5b..6b2e14ec6ef8 100644
--- a/include/linux/pipe_fs_i.h
+++ b/include/linux/pipe_fs_i.h
@@ -90,7 +90,7 @@ struct anon_pipe_prealloc {
* @files: number of struct file referring this pipe (protected by ->i_lock)
* @r_counter: reader counter
* @w_counter: writer counter
- * @pseudo_edgetrigger: has an EPOLLET consumer, enable per-write wakeups
+ * @pseudo_edgetrigger: enable per-write wakeups for EPOLLET/EPOLLEXCLUSIVE
* @fasync_readers: reader side fasync
* @fasync_writers: writer side fasync
* @bufs: the circular array of pipe buffers
---
base-commit: b0993210e420f2f1d61e6719fbcd3966b96ff9ed
change-id: 20261111-pipe-epollexclusive-wakeup-981528041b4d
Best regards,
--
Mani Tofigh <mt4022@columbia.edu>
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-11 21:02 [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers Mani Tofigh via B4 Relay
@ 2026-11-11 23:34 ` Oleg Nesterov
2026-11-12 10:26 ` Oleg Nesterov
2026-11-12 12:56 ` Oleg Nesterov
2026-11-17 13:27 ` Oleg Nesterov
2 siblings, 1 reply; 12+ messages in thread
From: Oleg Nesterov @ 2026-11-11 23:34 UTC (permalink / raw)
To: Mani Tofigh via B4 Relay
Cc: Christian Brauner, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
On 11/11, Mani Tofigh via B4 Relay wrote:
>
> From: Mani Tofigh <mt4022@columbia.edu>
>
> A task blocked in read() on a pipe can sleep while the pipe holds data,
> when an epoll(EPOLLEXCLUSIVE) watcher of the read end was added before
> it. The watcher's entry on ->rd_wait is exclusive and sits ahead of the
> reader, so a write into an empty pipe wakes only the watcher.
I am already sleeping, but at first glance this is already wrong?
I mean, a WQ_FLAG_EXCLUSIVE waiter should not sit ahead of non-exclusive
waiter, no?
Oleg.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-11 23:34 ` Oleg Nesterov
@ 2026-11-12 10:26 ` Oleg Nesterov
0 siblings, 0 replies; 12+ messages in thread
From: Oleg Nesterov @ 2026-11-12 10:26 UTC (permalink / raw)
To: Mani Tofigh via B4 Relay
Cc: Christian Brauner, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
On 11/12, Oleg Nesterov wrote:
>
> On 11/11, Mani Tofigh via B4 Relay wrote:
> >
> > From: Mani Tofigh <mt4022@columbia.edu>
> >
> > A task blocked in read() on a pipe can sleep while the pipe holds data,
> > when an epoll(EPOLLEXCLUSIVE) watcher of the read end was added before
> > it. The watcher's entry on ->rd_wait is exclusive and sits ahead of the
> > reader, so a write into an empty pipe wakes only the watcher.
>
> I am already sleeping, but at first glance this is already wrong?
>
> I mean, a WQ_FLAG_EXCLUSIVE waiter should not sit ahead of non-exclusive
> waiter, no?
Ah, I forgot that anon_pipe_read() uses wait_event_exclusive(), and it
assumes that this wait-event is the only reason to set wake_next_reader...
I seem to understand, let me think about it.
Oleg.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-11 21:02 [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers Mani Tofigh via B4 Relay
2026-11-11 23:34 ` Oleg Nesterov
@ 2026-11-12 12:56 ` Oleg Nesterov
2026-11-17 13:27 ` Oleg Nesterov
2 siblings, 0 replies; 12+ messages in thread
From: Oleg Nesterov @ 2026-11-12 12:56 UTC (permalink / raw)
To: Mani Tofigh via B4 Relay
Cc: Christian Brauner, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
This probably needs more discussion... I will write another email.
But so far I think we don't really care.
On 11/11, Mani Tofigh via B4 Relay wrote:
>
> A task blocked in read() on a pipe can sleep while the pipe holds data,
> when an epoll(EPOLLEXCLUSIVE) watcher of the read end was added before
> it. The watcher's entry on ->rd_wait is exclusive and sits ahead of the
> reader, so a write into an empty pipe wakes only the watcher. The
> watcher reads without sleeping in anon_pipe_read() and doesn't pass the
> wakeup on. Since commit f1cebdec9739 ("pipe: only enable the extra
> wake_up(rd_wait) for EPOLLET consumers") the next writes into the
> non-empty pipe don't wake anyone anymore:
>
> Task E Task A Task C
> epoll_wait(ep, -1)
> /* EPOLLEXCLUSIVE */
> read(p[0])
> write(p[1], "1")
> /* wakes E only */
> write(p[1], "2")
> /* no wakeup */
> read(p[0], 1 byte)
> exit()
> /* still asleep, "2" is in the pipe */
Yes, in this case f1cebdec9739 ("pipe: only enable the extra wake_up(rd_wait)
for EPOLLET consumers") makes a difference.
However, this trivial test-case
#include <unistd.h>
#include <sys/epoll.h>
#include <sys/wait.h>
#include <assert.h>
#include <stdio.h>
int main(void)
{
int pfd[2], efd;
struct epoll_event evt = { .events = EPOLLIN | EPOLLEXCLUSIVE };
char c;
pipe(pfd);
efd = epoll_create1(0);
epoll_ctl(efd, EPOLL_CTL_ADD, pfd[0], &evt);
if (!fork()) {
epoll_wait(efd, &evt, 1, -1);
read(pfd[0], &c, 1);
printf("1st: %c\n", c);
return 0;
}
sleep(1);
if (!fork()) {
read(pfd[0], &c, 1);
printf("2nd: %c\n", c);
return 0;
}
sleep(1);
write(pfd[1], "12", 2);
assert(wait(NULL) > 0);
assert(wait(NULL) > 0);
return 0;
}
hangs even with this patch
+++ b/fs/pipe.c
@@ -749,6 +749,7 @@ pipe_poll(struct file *filp, poll_table *wait)
unlikely(!READ_ONCE(pipe->pseudo_edgetrigger)))
WRITE_ONCE(pipe->pseudo_edgetrigger, true);
+WRITE_ONCE(pipe->pseudo_edgetrigger, true);
/*
* Reading pipe state only -- no need for acquiring the semaphore.
*
which reverts f1cebdec9739.
The 2nd child sleeps in read(pfd[0]) forever even if the pipe is not empty.
So I am not sure we want to enforce another oddity in pipe_poll/read ...
Oleg.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-11 21:02 [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers Mani Tofigh via B4 Relay
2026-11-11 23:34 ` Oleg Nesterov
2026-11-12 12:56 ` Oleg Nesterov
@ 2026-11-17 13:27 ` Oleg Nesterov
2026-11-17 16:22 ` Linus Torvalds
2 siblings, 1 reply; 12+ messages in thread
From: Oleg Nesterov @ 2026-11-17 13:27 UTC (permalink / raw)
To: Mani Tofigh via B4 Relay, Breno Leitao, Christian Brauner,
Linus Torvalds, Mateusz Guzik
Cc: Alexander Viro, Jan Kara, linux-fsdevel, linux-kernel, tz2294,
stable, Mani Tofigh
Mani reports the case when f1cebdec9739 ("pipe: only enable the extra
wake_up(rd_wait) for EPOLLET consumers") makes a difference, see his
email below.
But lets forget about that commit for the moment. With or without that
commit this trivial test-case hangs,
#include <unistd.h>
#include <sys/epoll.h>
#include <sys/wait.h>
#include <assert.h>
#include <stdio.h>
int main(void)
{
int pfd[2], efd;
struct epoll_event evt = { .events = EPOLLIN | EPOLLEXCLUSIVE };
char c;
pipe(pfd);
efd = epoll_create1(0);
epoll_ctl(efd, EPOLL_CTL_ADD, pfd[0], &evt);
if (!fork()) {
epoll_wait(efd, &evt, 1, -1);
read(pfd[0], &c, 1);
printf("1st: %c\n", c);
return 0;
}
sleep(1);
if (!fork()) {
read(pfd[0], &c, 1);
printf("2nd: %c\n", c);
return 0;
}
sleep(1);
write(pfd[1], "12", 2);
assert(wait(NULL) > 0);
assert(wait(NULL) > 0);
return 0;
}
the 2nd non-epoll child sleeps in read(pfd[0]) even if pipe is not empty.
anon_pipe_read() assumes that all exclusive waiters on pipe->rd_wait come
from anon_pipe_read()->wait_event_interruptible_exclusive(), and the woken
reader will set wake_next_reader = true.
However, EPOLLEXCLUSIVE breaks this logic. ep_ptable_queue_proc() does
add_wait_queue_exclusive() if EPOLLEXCLUSIVE is set. So write(pfd[1]) above
wakes up the 1st "epoll" child, which then reads one byte from the pipe without
sleeping, so it doesn't set wake_next_reader and thus it doesn't wakeup the
2nd waiter.
Again, this happens with or without f1cebdec9739. But that commit makes a
difference: the 2nd child will continue to sleep even after another write
to this pipe. So Mani suggests the trivial change which should restore the
previous behaviour
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -700,9 +700,13 @@ pipe_poll(struct file *filp, poll_table *wait)
* and the ->pseudo_edgetrigger check in anon_pipe_write().
* Currently io_uring sets EPOLLET for multishot polls, so it gets the
* same behaviour.
+ *
+ * EPOLLEXCLUSIVE needs them too. Its entry is an exclusive waiter on
+ * ->rd_wait that can take the wakeup meant for a blocked reader, and
+ * unlike a woken reader in anon_pipe_read() it doesn't pass it on.
*/
if ((filp->f_mode & FMODE_READ) &&
- wait && (wait->_key & EPOLLET) &&
+ wait && (wait->_key & (EPOLLET | EPOLLEXCLUSIVE)) &&
unlikely(!READ_ONCE(pipe->pseudo_edgetrigger)))
WRITE_ONCE(pipe->pseudo_edgetrigger, true);
But I can't decide if we actually want to bless another historical
"nonsensical" wakeup until have a user application that actually cares?
Again, the test-case above will hang with or without the patch above, with
or without f1cebdec9739. So do we really care? If yes, then perhaps we should
also change anon_pipe_read(), something like
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -468,6 +468,8 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
}
if (pipe_is_empty(pipe))
wake_next_reader = false;
+ else if (pipe->pseudo_edgetrigger) // for EPOLLEXCLUSIVE
+ wake_next_reader = true;
/* Consumed buffers may have refilled the pool; trim it and unlock. */
anon_pipe_trim_and_unlock(pipe);
?
(of course, this won't help if the 1st child doesn't do read() after epoll_wait()).
What do you think?
Oleg.
On 11/11, Mani Tofigh via B4 Relay wrote:
>
> From: Mani Tofigh <mt4022@columbia.edu>
>
> A task blocked in read() on a pipe can sleep while the pipe holds data,
> when an epoll(EPOLLEXCLUSIVE) watcher of the read end was added before
> it. The watcher's entry on ->rd_wait is exclusive and sits ahead of the
> reader, so a write into an empty pipe wakes only the watcher. The
> watcher reads without sleeping in anon_pipe_read() and doesn't pass the
> wakeup on. Since commit f1cebdec9739 ("pipe: only enable the extra
> wake_up(rd_wait) for EPOLLET consumers") the next writes into the
> non-empty pipe don't wake anyone anymore:
>
> Task E Task A Task C
> epoll_wait(ep, -1)
> /* EPOLLEXCLUSIVE */
> read(p[0])
> write(p[1], "1")
> /* wakes E only */
> write(p[1], "2")
> /* no wakeup */
> read(p[0], 1 byte)
> exit()
> /* still asleep, "2" is in the pipe */
>
> A then sleeps until the next write into an empty pipe, if ever. Before
> that commit any pipe_poll() enabled the per-write wakeups. Those
> reached A because an EPOLLEXCLUSIVE entry only takes a wakeup while a
> task sleeps in epoll_wait().
>
> So EPOLLEXCLUSIVE consumers depend on the per-write wakeups too, for a
> different reason than EPOLLET. Enable them for EPOLLEXCLUSIVE as well
> and say why in the comment. With the fix, A reads the second byte.
>
> This bug was identified and confirmed with the help of a deterministic
> hypervisor currently under development. The C reproducer below however
> hits it within about 2s.
>
> Fixes: f1cebdec9739 ("pipe: only enable the extra wake_up(rd_wait) for EPOLLET consumers")
> Cc: stable@vger.kernel.org
> Signed-off-by: Mani Tofigh <mt4022@columbia.edu>
> ---
> Reproducer: https://github.com/manitofigh/files/blob/main/linux-repros/pipeexcl.c
>
> There are a couple of older gaps like this (a single write, and the
> write side with EPOLLOUT|EPOLLEXCLUSIVE). I left both alone here though.
> ---
> fs/pipe.c | 6 +++++-
> include/linux/pipe_fs_i.h | 2 +-
> 2 files changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/fs/pipe.c b/fs/pipe.c
> index 4c1122597a3c..8ad792d300c9 100644
> --- a/fs/pipe.c
> +++ b/fs/pipe.c
> @@ -700,9 +700,13 @@ pipe_poll(struct file *filp, poll_table *wait)
> * and the ->pseudo_edgetrigger check in anon_pipe_write().
> * Currently io_uring sets EPOLLET for multishot polls, so it gets the
> * same behaviour.
> + *
> + * EPOLLEXCLUSIVE needs them too. Its entry is an exclusive waiter on
> + * ->rd_wait that can take the wakeup meant for a blocked reader, and
> + * unlike a woken reader in anon_pipe_read() it doesn't pass it on.
> */
> if ((filp->f_mode & FMODE_READ) &&
> - wait && (wait->_key & EPOLLET) &&
> + wait && (wait->_key & (EPOLLET | EPOLLEXCLUSIVE)) &&
> unlikely(!READ_ONCE(pipe->pseudo_edgetrigger)))
> WRITE_ONCE(pipe->pseudo_edgetrigger, true);
>
> diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h
> index 95f6236c5c5b..6b2e14ec6ef8 100644
> --- a/include/linux/pipe_fs_i.h
> +++ b/include/linux/pipe_fs_i.h
> @@ -90,7 +90,7 @@ struct anon_pipe_prealloc {
> * @files: number of struct file referring this pipe (protected by ->i_lock)
> * @r_counter: reader counter
> * @w_counter: writer counter
> - * @pseudo_edgetrigger: has an EPOLLET consumer, enable per-write wakeups
> + * @pseudo_edgetrigger: enable per-write wakeups for EPOLLET/EPOLLEXCLUSIVE
> * @fasync_readers: reader side fasync
> * @fasync_writers: writer side fasync
> * @bufs: the circular array of pipe buffers
>
> ---
> base-commit: b0993210e420f2f1d61e6719fbcd3966b96ff9ed
> change-id: 20261111-pipe-epollexclusive-wakeup-981528041b4d
>
> Best regards,
> --
> Mani Tofigh <mt4022@columbia.edu>
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-17 13:27 ` Oleg Nesterov
@ 2026-11-17 16:22 ` Linus Torvalds
2026-11-17 17:25 ` Oleg Nesterov
0 siblings, 1 reply; 12+ messages in thread
From: Linus Torvalds @ 2026-11-17 16:22 UTC (permalink / raw)
To: Oleg Nesterov
Cc: Mani Tofigh via B4 Relay, Breno Leitao, Christian Brauner,
Mateusz Guzik, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
On Tue, 17 Nov 2026 at 05:27, Oleg Nesterov <oleg@redhat.com> wrote:
>
> However, EPOLLEXCLUSIVE breaks this logic. ep_ptable_queue_proc() does
> add_wait_queue_exclusive() if EPOLLEXCLUSIVE is set.
Christ, how I hate epoll. It's a pile of stinking shit.
This literally violates THE WHOLE AND ONLY POINT of exclusive waits.
The rule for an exclusive wait is that it *will* act on the wakeup -
either by consuming the data, or - if it can't consume it all - by
waking up the next waiter itself.
So it's not that "EPOLLEXCLUSIVE breaks this logic".
The logic is fine. EPOLLEXCLUSIVE is broken beyond words.
I really think we need to just actively de-emphasize epoll. Starting
with saying "you can't add EPOLLEXCLUSIVE, because you don't honor
it".
Linus
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-17 16:22 ` Linus Torvalds
@ 2026-11-17 17:25 ` Oleg Nesterov
2026-11-17 17:35 ` Linus Torvalds
0 siblings, 1 reply; 12+ messages in thread
From: Oleg Nesterov @ 2026-11-17 17:25 UTC (permalink / raw)
To: Linus Torvalds
Cc: Mani Tofigh via B4 Relay, Breno Leitao, Christian Brauner,
Mateusz Guzik, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
On 11/17, Linus Torvalds wrote:
>
> On Tue, 17 Nov 2026 at 05:27, Oleg Nesterov <oleg@redhat.com> wrote:
> >
> > However, EPOLLEXCLUSIVE breaks this logic. ep_ptable_queue_proc() does
> > add_wait_queue_exclusive() if EPOLLEXCLUSIVE is set.
>
> Christ, how I hate epoll. It's a pile of stinking shit.
>
> This literally violates THE WHOLE AND ONLY POINT of exclusive waits.
>
> The rule for an exclusive wait is that it *will* act on the wakeup -
> either by consuming the data, or - if it can't consume it all - by
> waking up the next waiter itself.
>
> So it's not that "EPOLLEXCLUSIVE breaks this logic".
>
> The logic is fine. EPOLLEXCLUSIVE is broken beyond words.
Well, I thought so too. That is why initially I replied "we don't care"
to Mani. But then I decided to summarize and add CC's.
So. Do you agree that we can "ignore" this report from Mani for now?
Until we find a user application that already depends on the behaviour
before f1cebdec9739 (the next write() wakes up a reader which sleeps on
->rd_wait for no reason).
Just in case... io_uring/poll.c:__io_queue_proc does the same:
if (poll->events & EPOLLEXCLUSIVE) {
add_wait_queue_exclusive(head, &poll->wait);
} else {
add_wait_queue(head, &poll->wait);
}
Oleg.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-17 17:25 ` Oleg Nesterov
@ 2026-11-17 17:35 ` Linus Torvalds
2026-11-17 17:58 ` Oleg Nesterov
0 siblings, 1 reply; 12+ messages in thread
From: Linus Torvalds @ 2026-11-17 17:35 UTC (permalink / raw)
To: Oleg Nesterov
Cc: Mani Tofigh via B4 Relay, Breno Leitao, Christian Brauner,
Mateusz Guzik, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
On Tue, 17 Nov 2026 at 09:25, Oleg Nesterov <oleg@redhat.com> wrote:
>
> So. Do you agree that we can "ignore" this report from Mani for now?
I think we can ignore it, but I also have a cunning plan.
Normally, exclusive wait events always go at the tail of the wait
queue. We do that mainly because we want to wake all non-exclusive
ones first, and because it's fair.
But pipes don't *have* non-exclusive waiters (ignoring the splice
case, which waits for space, and if there are readers or writes that
would take the space we don't care).
So what we could do is to just make the pipe code open-code its own
wait event logic, and add its waiters at the head.
Voilà - we never even care about epoll braindamage.
Yes, that makes pipes unfair in that new readers get preference over
old readers, and maybe that's not great. It might affect things like
the make jobserver. I'd expect it would probably improve throughput -
new waiters are hot waiters - but it's most definitely worth
mentioning.
It won't fix the "EPOLL edge triggered braindamage", where old
applications asked for edge triggered things and actually wanted level
triggers. But it should solve *this* issue.
Hmm?
Let me stew on this a bit.
Linus
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-17 17:35 ` Linus Torvalds
@ 2026-11-17 17:58 ` Oleg Nesterov
2026-11-17 19:12 ` Linus Torvalds
0 siblings, 1 reply; 12+ messages in thread
From: Oleg Nesterov @ 2026-11-17 17:58 UTC (permalink / raw)
To: Linus Torvalds
Cc: Mani Tofigh via B4 Relay, Breno Leitao, Christian Brauner,
Mateusz Guzik, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
On 11/17, Linus Torvalds wrote:
>
> On Tue, 17 Nov 2026 at 09:25, Oleg Nesterov <oleg@redhat.com> wrote:
> >
> > So. Do you agree that we can "ignore" this report from Mani for now?
>
> I think we can ignore it, but I also have a cunning plan.
Good. this was my main (the only) question.
> So what we could do is to just make the pipe code open-code its own
> wait event logic, and add its waiters at the head.
perhaps... and we already have WQ_FLAG_PRIORITY, but we can't re-use it
without more changes...
Oleg.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-17 17:58 ` Oleg Nesterov
@ 2026-11-17 19:12 ` Linus Torvalds
2026-11-17 19:48 ` Linus Torvalds
0 siblings, 1 reply; 12+ messages in thread
From: Linus Torvalds @ 2026-11-17 19:12 UTC (permalink / raw)
To: Oleg Nesterov
Cc: Mani Tofigh via B4 Relay, Breno Leitao, Christian Brauner,
Mateusz Guzik, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
On Tue, 17 Nov 2026 at 09:58, Oleg Nesterov <oleg@redhat.com> wrote:
>
> On 11/17, Linus Torvalds wrote:
> >
> > So what we could do is to just make the pipe code open-code its own
> > wait event logic, and add its waiters at the head.
>
> perhaps... and we already have WQ_FLAG_PRIORITY, but we can't re-use it
> without more changes...
Ok, I looked at this, and it gets ugly, and I'd worry about the
unfairness. Yes, the unfairness probably helps throughput, but every
time we've doen that, it ends up being a starvation issue for some
case.
I think "ignore it - bad epoll actors get to use both broken pieces"
issue unless somebody comes up with something nicer.
Linus
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-17 19:12 ` Linus Torvalds
@ 2026-11-17 19:48 ` Linus Torvalds
2026-11-17 20:41 ` Oleg Nesterov
0 siblings, 1 reply; 12+ messages in thread
From: Linus Torvalds @ 2026-11-17 19:48 UTC (permalink / raw)
To: Oleg Nesterov
Cc: Mani Tofigh via B4 Relay, Breno Leitao, Christian Brauner,
Mateusz Guzik, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
[-- Attachment #1: Type: text/plain, Size: 799 bytes --]
On Tue, 17 Nov 2026 at 11:12, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> Ok, I looked at this, and it gets ugly
Side note: as part of looking at it, I did look at what it would
involve in our wait.h infrastructure.
That header isn't _horrible_, but some of it was uglier than needed,
and adding WQ_FLAG_PRIORITY would have been a huge pain because the
code unnecessarily special-cases WQ_FLAG_EXCLUSIVE.
So while I threw the rest of my trial patch away, this patch remained
as a cleanup. It *looks* sane to me, and I think the end result looks
better and would be more flexible.
For example, it would have allowed just adding a new user of
___wait_event() that used WQ_FLAG_PRIORITY to put itself at the head
of the queue.
NOTE! Entirely untested. Comments?
Linus
[-- Attachment #2: 0001-wait-event-clean-up-the-internal-helper-arguments.patch --]
[-- Type: text/x-patch, Size: 5969 bytes --]
From c8f60d670fa1215ce5cd38f9c7e7cb2758bc5a58 Mon Sep 17 00:00:00 2001
From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue, 17 Nov 2026 11:42:03 -0800
Subject: [PATCH] wait-event: clean up the internal helper arguments
The ___wait_event() helper for historical reasons took a "0/1" argument
for whether the wait-queue entry should be marked WQ_FLAG_EXCLUSIVE or
not. That didn't really help readability, but it also limited the
wq_flags for that helper to only that one flag.
Just make that macro argument be the actual 'wq_flags' value, which
simplifies the helper, and makes it more generic. It also happens to
make the users of this macro more obvious when an unexplained '1'
argument becomes a more self-explanatory 'WQ_FLAG_EXCLUSIVE'.
Also change the existing users that use that raw ___wait_event()
interface with just a plain schedule() to use the pre-existing helpers
macros: __wait_event_state() and __wait_event_state_exclusive().
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---
include/linux/wait.h | 36 +++++++++++++++---------------------
1 file changed, 15 insertions(+), 21 deletions(-)
diff --git a/include/linux/wait.h b/include/linux/wait.h
index c2af98b0074d..b3e94c71cd3e 100644
--- a/include/linux/wait.h
+++ b/include/linux/wait.h
@@ -299,13 +299,13 @@ extern void init_wait_entry(struct wait_queue_entry *wq_entry, int flags);
* otherwise.
*/
-#define ___wait_event(wq_head, condition, state, exclusive, ret, cmd) \
+#define ___wait_event(wq_head, condition, state, wq_flag, ret, cmd) \
({ \
__label__ __out; \
struct wait_queue_entry __wq_entry; \
long __ret = ret; /* explicit shadow */ \
\
- init_wait_entry(&__wq_entry, exclusive ? WQ_FLAG_EXCLUSIVE : 0); \
+ init_wait_entry(&__wq_entry, wq_flag); \
for (;;) { \
long __int = prepare_to_wait_event(&wq_head, &__wq_entry, state);\
\
@@ -327,8 +327,7 @@ __out: __ret; \
})
#define __wait_event(wq_head, condition) \
- (void)___wait_event(wq_head, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
- schedule())
+ (void)__wait_event_state(wq_head, condition, TASK_UNINTERRUPTIBLE)
/**
* wait_event - sleep until a condition gets true
@@ -366,8 +365,7 @@ do { \
} while (0)
#define __wait_event_freezable(wq_head, condition) \
- ___wait_event(wq_head, condition, (TASK_INTERRUPTIBLE|TASK_FREEZABLE), \
- 0, 0, schedule())
+ __wait_event_state(wq_head, condition, (TASK_INTERRUPTIBLE|TASK_FREEZABLE))
/**
* wait_event_freezable - sleep (or freeze) until a condition gets true
@@ -442,8 +440,8 @@ do { \
})
#define __wait_event_exclusive_cmd(wq_head, condition, cmd1, cmd2) \
- (void)___wait_event(wq_head, condition, TASK_UNINTERRUPTIBLE, 1, 0, \
- cmd1; schedule(); cmd2)
+ (void)___wait_event(wq_head, condition, TASK_UNINTERRUPTIBLE, \
+ WQ_FLAG_EXCLUSIVE, 0, cmd1; schedule(); cmd2)
/*
* Just like wait_event_cmd(), except it sets exclusive flag
*/
@@ -480,8 +478,7 @@ do { \
} while (0)
#define __wait_event_interruptible(wq_head, condition) \
- ___wait_event(wq_head, condition, TASK_INTERRUPTIBLE, 0, 0, \
- schedule())
+ __wait_event_state(wq_head, condition, TASK_INTERRUPTIBLE)
/**
* wait_event_interruptible - sleep until a condition gets true
@@ -620,8 +617,7 @@ do { \
})
#define __wait_event_interruptible_exclusive(wq, condition) \
- ___wait_event(wq, condition, TASK_INTERRUPTIBLE, 1, 0, \
- schedule())
+ __wait_event_state_exclusive(wq, condition, TASK_INTERRUPTIBLE)
#define wait_event_interruptible_exclusive(wq, condition) \
({ \
@@ -633,8 +629,7 @@ do { \
})
#define __wait_event_killable_exclusive(wq, condition) \
- ___wait_event(wq, condition, TASK_KILLABLE, 1, 0, \
- schedule())
+ __wait_event_state_exclusive(wq, condition, TASK_KILLABLE)
#define wait_event_killable_exclusive(wq, condition) \
({ \
@@ -647,8 +642,7 @@ do { \
#define __wait_event_freezable_exclusive(wq, condition) \
- ___wait_event(wq, condition, (TASK_INTERRUPTIBLE|TASK_FREEZABLE), 1, 0,\
- schedule())
+ __wait_event_state_exclusive(wq, condition, (TASK_INTERRUPTIBLE|TASK_FREEZABLE))
#define wait_event_freezable_exclusive(wq, condition) \
({ \
@@ -676,7 +670,7 @@ do { \
do { \
might_sleep(); \
if (!(condition)) \
- ___wait_event(wq_head, condition, TASK_IDLE, 0, 0, schedule()); \
+ __wait_event_state(wq_head, condition, TASK_IDLE); \
} while (0)
/**
@@ -700,7 +694,7 @@ do { \
do { \
might_sleep(); \
if (!(condition)) \
- ___wait_event(wq_head, condition, TASK_IDLE, 1, 0, schedule()); \
+ __wait_event_state_exclusive(wq_head, condition, TASK_IDLE); \
} while (0)
#define __wait_event_idle_timeout(wq_head, condition, timeout) \
@@ -738,7 +732,7 @@ do { \
#define __wait_event_idle_exclusive_timeout(wq_head, condition, timeout) \
___wait_event(wq_head, ___wait_cond_timeout(condition), \
- TASK_IDLE, 1, timeout, \
+ TASK_IDLE, WQ_FLAG_EXCLUSIVE, timeout, \
__ret = schedule_timeout(__ret))
/**
@@ -911,7 +905,7 @@ extern int do_wait_intr_irq(wait_queue_head_t *, wait_queue_entry_t *);
#define __wait_event_killable(wq, condition) \
- ___wait_event(wq, condition, TASK_KILLABLE, 0, 0, schedule())
+ __wait_event_state(wq, condition, TASK_KILLABLE)
/**
* wait_event_killable - sleep until a condition gets true
@@ -981,7 +975,7 @@ extern int do_wait_intr_irq(wait_queue_head_t *, wait_queue_entry_t *);
})
#define __wait_event_state_exclusive(wq, condition, state) \
- ___wait_event(wq, condition, state, 1, 0, schedule())
+ ___wait_event(wq, condition, state, WQ_FLAG_EXCLUSIVE, 0, schedule())
#define wait_event_state_exclusive(wq, condition, state) \
({ \
--
2.56.0.313.g975a75ca90
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-17 19:48 ` Linus Torvalds
@ 2026-11-17 20:41 ` Oleg Nesterov
0 siblings, 0 replies; 12+ messages in thread
From: Oleg Nesterov @ 2026-11-17 20:41 UTC (permalink / raw)
To: Linus Torvalds
Cc: Mani Tofigh via B4 Relay, Breno Leitao, Christian Brauner,
Mateusz Guzik, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
On 11/17, Linus Torvalds wrote:
>
> Side note: as part of looking at it, I did look at what it would
> involve in our wait.h infrastructure.
>
> That header isn't _horrible_, but some of it was uglier than needed,
> and adding WQ_FLAG_PRIORITY would have been a huge pain because the
> code unnecessarily special-cases WQ_FLAG_EXCLUSIVE.
>
> So while I threw the rest of my trial patch away, this patch remained
> as a cleanup. It *looks* sane to me, and I think the end result looks
> better and would be more flexible.
>
> For example, it would have allowed just adding a new user of
> ___wait_event() that used WQ_FLAG_PRIORITY to put itself at the head
> of the queue.
...
> -#define ___wait_event(wq_head, condition, state, exclusive, ret, cmd) \
> +#define ___wait_event(wq_head, condition, state, wq_flag, ret, cmd) \
> ({ \
> __label__ __out; \
> struct wait_queue_entry __wq_entry; \
> long __ret = ret; /* explicit shadow */ \
> \
> - init_wait_entry(&__wq_entry, exclusive ? WQ_FLAG_EXCLUSIVE : 0); \
> + init_wait_entry(&__wq_entry, wq_flag); \
> for (;;) { \
> long __int = prepare_to_wait_event(&wq_head, &__wq_entry, state);\
Damn ;) I simply can't resist.
I swear, I was thinking of exactly that same change during our discussion.
So I think it is obviously good and makes sense, but I will try to re-check
it after sleep.
Oleg.
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2026-11-17 20:41 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-11 21:02 [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers Mani Tofigh via B4 Relay
2026-11-11 23:34 ` Oleg Nesterov
2026-11-12 10:26 ` Oleg Nesterov
2026-11-12 12:56 ` Oleg Nesterov
2026-11-17 13:27 ` Oleg Nesterov
2026-11-17 16:22 ` Linus Torvalds
2026-11-17 17:25 ` Oleg Nesterov
2026-11-17 17:35 ` Linus Torvalds
2026-11-17 17:58 ` Oleg Nesterov
2026-11-17 19:12 ` Linus Torvalds
2026-11-17 19:48 ` Linus Torvalds
2026-11-17 20:41 ` Oleg Nesterov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox