io_uring: SENDMSG/WRITEV return -EAGAIN on O_NONBLOCK socket after peer half-close

io_uring: SENDMSG/WRITEV return -EAGAIN on O_NONBLOCK socket after peer half-close

[Lukáš Lalinský]

Hi,

IORING_OP_SENDMSG can complete with -EAGAIN even though MSG_DONTWAIT
was not used. It happens when the send is waiting for buffer space, the
peer calls shutdown(SHUT_WR), and the socket has O_NONBLOCK set.

IORING_OP_SEND in the same situation keeps waiting and completes
normally once the peer reads. So does SENDMSG on a socket without
O_NONBLOCK. IORING_OP_WRITEV behaves the same way as SENDMSG.

I consider this a bug. Based on previous discussions [1][2], io_uring
should ignore O_NONBLOCK for operations that take a per-call nonblock
flag. SENDMSG mostly does: on the full socket the request doesn't fail
immediately, it waits. It only fails after the peer half-closes. At the
very least it's inconsistent, because SEND and SENDMSG/WRITEV behave
differently on the same socket.

There is no good option for handling the error. Resubmitting fails
again the same way. Waiting for writability with
IORING_OP_POLL_ADD(POLLOUT) doesn't work after a half-close either, it
completes immediately with POLLRDHUP, even though POLLRDHUP wasn't
requested. The only workarounds are dropping the O_NONBLOCK flag, or
using a separate epoll to wait for writability.

If I'm reading the code correctly, after the half-close EPOLLRDHUP
keeps waking the request's internal poll. The request runs out of
APOLL_MAX_RETRY and is punted to io-wq, where the blocking attempt goes
through __sys_sendmsg_sock() -> ____sys_sendmsg(). That function turns
O_NONBLOCK into MSG_DONTWAIT, so the attempt returns -EAGAIN instead of
blocking. io_send() calls sock_sendmsg() directly, which is why SEND
blocks in the worker instead. WRITEV seems to hit the same thing in
sock_write_iter(), and IORING_OP_RECVMSG through ____sys_recvmsg(),
after enough wakeups that find no data.

We hit this in zio, an async runtime for Zig, via an HTTP/2 server's
half-close tests [3]. ringline, a Rust io_uring runtime, hit it
independently on 6.12 and 7.1 [4], and libosmocore seems to have hit
the same -EAGAIN and worked around it by clearing O_NONBLOCK [5]. We'll
work around it in zio for now.

Here is a minimal reproducer:

/*
 * IORING_OP_SENDMSG or IORING_OP_WRITEV with flags 0 is parked on a
 * full AF_UNIX stream socket. If the socket has O_NONBLOCK and the
 * peer then calls shutdown(SHUT_WR), the request completes with
 * -EAGAIN. IORING_OP_SEND, or any of them without O_NONBLOCK, keeps
 * waiting and completes once the peer reads.
 *
 * Build: cc -O2 -Wall -o sendmsg_eagain sendmsg_eagain.c -luring
 */
#include <errno.h>
#include <fcntl.h>
#include <liburing.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/utsname.h>
#include <unistd.h>

#define CHECK(x) do { if (!(x)) { perror(#x); exit(2); } } while (0)

enum { OP_SEND, OP_SENDMSG, OP_WRITEV };
static const char *const op_names[] = {
	"SEND   ", "SENDMSG", "WRITEV ",
};

static char buf[65536];

static int run(int op, int nonblock)
{
	struct __kernel_timespec ts = { .tv_sec = 1 };
	struct iovec iov = { .iov_base = buf, .iov_len = sizeof(buf) };
	struct msghdr msg = { .msg_iov = &iov, .msg_iovlen = 1 };
	struct io_uring_sqe *sqe;
	struct io_uring_cqe *cqe;
	struct io_uring ring;
	char sink[65536];
	int sv[2], ret, res;

	CHECK(socketpair(AF_UNIX, SOCK_STREAM, 0, sv) == 0);
	if (nonblock)
		CHECK(fcntl(sv[0], F_SETFL,
			    fcntl(sv[0], F_GETFL) | O_NONBLOCK) == 0);

	/* Fill the send buffer. */
	while (send(sv[0], buf, sizeof(buf), MSG_DONTWAIT) > 0)
		;
	CHECK(errno == EAGAIN);

	ret = io_uring_queue_init(4, &ring, 0);
	if (ret) {
		fprintf(stderr, "queue_init: %s\n", strerror(-ret));
		exit(2);
	}
	sqe = io_uring_get_sqe(&ring);
	if (op == OP_SEND)
		io_uring_prep_send(sqe, sv[0], buf, sizeof(buf), 0);
	else if (op == OP_SENDMSG)
		io_uring_prep_sendmsg(sqe, sv[0], &msg, 0);
	else
		io_uring_prep_writev(sqe, sv[0], &iov, 1, 0);
	CHECK(io_uring_submit(&ring) == 1);

	/* The request must be waiting for space, not completed. */
	CHECK(io_uring_peek_cqe(&ring, &cqe) == -EAGAIN);

	CHECK(shutdown(sv[1], SHUT_WR) == 0);

	ret = io_uring_wait_cqe_timeout(&ring, &cqe, &ts);
	if (ret == 0) {
		res = cqe->res;
		printf("%s O_NONBLOCK=%d: res=%d after peer shutdown\n",
		       op_names[op], nonblock, res);
	} else {
		CHECK(ret == -ETIME);
		/* Still waiting after 1s: drain the peer until done. */
		do {
			while (recv(sv[1], sink, sizeof(sink),
				    MSG_DONTWAIT) > 0)
				;
			ts.tv_sec = 0;
			ts.tv_nsec = 10 * 1000 * 1000;
			ret = io_uring_wait_cqe_timeout(&ring, &cqe,
							&ts);
		} while (ret == -ETIME);
		CHECK(ret == 0);
		res = cqe->res;
		printf("%s O_NONBLOCK=%d: res=%d after peer read\n",
		       op_names[op], nonblock, res);
	}
	io_uring_cqe_seen(&ring, cqe);
	io_uring_queue_exit(&ring);
	close(sv[0]);
	close(sv[1]);
	return res == sizeof(buf) ? 0 : 1;
}

int main(void)
{
	struct utsname u;
	int op, fail = 0;

	CHECK(uname(&u) == 0);
	printf("kernel %s, liburing %d.%d\n", u.release,
	       io_uring_major_version(), io_uring_minor_version());
	for (op = OP_SEND; op <= OP_WRITEV; op++) {
		fail |= run(op, 0);
		fail |= run(op, 1);
	}
	return fail;
}

Output:

kernel 7.0.0-38-generic, liburing 2.5
SEND    O_NONBLOCK=0: res=65536 after peer read
SEND    O_NONBLOCK=1: res=65536 after peer read
SENDMSG O_NONBLOCK=0: res=65536 after peer read
SENDMSG O_NONBLOCK=1: res=-11 after peer shutdown
WRITEV  O_NONBLOCK=0: res=65536 after peer read
WRITEV  O_NONBLOCK=1: res=-11 after peer shutdown

The relevant code is unchanged in current master.

[1] https://lore.kernel.org/io-uring/e83c9ef0-8016-2c4a-9dd0-b2a1318238ef@kernel.dk/
[2] https://github.com/axboe/liburing/issues/1270
[3] https://github.com/lalinsky/zio/issues/864
[4] https://github.com/ringline-rs/ringline/issues/603
[5] https://gerrit.osmocom.org/c/libosmocore/+/41671

Lukas

Re: io_uring: SENDMSG/WRITEV return -EAGAIN on O_NONBLOCK socket after peer half-close

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 2312 bytes --]

On 10/29/26 12:54 PM, Lukáš Lalinský wrote:
> Hi,
> 
> IORING_OP_SENDMSG can complete with -EAGAIN even though MSG_DONTWAIT
> was not used. It happens when the send is waiting for buffer space, the
> peer calls shutdown(SHUT_WR), and the socket has O_NONBLOCK set.
> 
> IORING_OP_SEND in the same situation keeps waiting and completes
> normally once the peer reads. So does SENDMSG on a socket without
> O_NONBLOCK. IORING_OP_WRITEV behaves the same way as SENDMSG.
> 
> I consider this a bug. Based on previous discussions [1][2], io_uring
> should ignore O_NONBLOCK for operations that take a per-call nonblock
> flag. SENDMSG mostly does: on the full socket the request doesn't fail
> immediately, it waits. It only fails after the peer half-closes. At the
> very least it's inconsistent, because SEND and SENDMSG/WRITEV behave
> differently on the same socket.
> 
> There is no good option for handling the error. Resubmitting fails
> again the same way. Waiting for writability with
> IORING_OP_POLL_ADD(POLLOUT) doesn't work after a half-close either, it
> completes immediately with POLLRDHUP, even though POLLRDHUP wasn't
> requested. The only workarounds are dropping the O_NONBLOCK flag, or
> using a separate epoll to wait for writability.
> 
> If I'm reading the code correctly, after the half-close EPOLLRDHUP
> keeps waking the request's internal poll. The request runs out of
> APOLL_MAX_RETRY and is punted to io-wq, where the blocking attempt goes
> through __sys_sendmsg_sock() -> ____sys_sendmsg(). That function turns
> O_NONBLOCK into MSG_DONTWAIT, so the attempt returns -EAGAIN instead of
> blocking. io_send() calls sock_sendmsg() directly, which is why SEND
> blocks in the worker instead. WRITEV seems to hit the same thing in
> sock_write_iter(), and IORING_OP_RECVMSG through ____sys_recvmsg(),
> after enough wakeups that find no data.
> 
> We hit this in zio, an async runtime for Zig, via an HTTP/2 server's
> half-close tests [3]. ringline, a Rust io_uring runtime, hit it
> independently on 6.12 and 7.1 [4], and libosmocore seems to have hit
> the same -EAGAIN and worked around it by clearing O_NONBLOCK [5]. We'll
> work around it in zio for now.

Can you try with these two patches? I think the problem is two-fold,
each patch tries to reason about that.

-- 
Jens Axboe

[-- Attachment #2: 0002-net-leave-O_NONBLOCK-to-the-sendmsg-and-recvmsg-sysc.patch --]
[-- Type: text/x-patch, Size: 2680 bytes --]

From beba6d231302329700c3a45777d9c2da32b06f0b Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Thu, 29 Oct 2026 13:38:18 -0600
Subject: [PATCH 2/2] net: leave O_NONBLOCK to the sendmsg and recvmsg syscalls
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

____sys_sendmsg() and ____sys_recvmsg() turn O_NONBLOCK on the socket
into MSG_DONTWAIT. That is right for the system calls, but io_uring uses
__sys_sendmsg_sock() and __sys_recvmsg_sock() for a blocking attempt
from io-wq as well, after the non-blocking attempts with MSG_DONTWAIT
have failed. On an O_NONBLOCK socket that attempt can't block either and
the request completes with -EAGAIN, unlike IORING_OP_SEND and RECV,
which call sock_sendmsg() and sock_recvmsg() directly and don't look at
O_NONBLOCK at all.

Move the O_NONBLOCK handling to ___sys_sendmsg() and ___sys_recvmsg(),
which the sendmsg, sendmmsg, recvmsg and recvmmsg system calls go
through. The io_uring helpers then pass the flags the caller asked for.

Reported-by: Lukáš Lalinský <lukas@lalinsky.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 net/socket.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/net/socket.c b/net/socket.c
index 2e3b5737c481..08658a9b8065 100644
--- a/net/socket.c
+++ b/net/socket.c
@@ -2697,8 +2697,6 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
 	flags &= ~MSG_INTERNAL_SENDMSG_FLAGS;
 	msg_sys->msg_flags = flags;
 
-	if (sock->file->f_flags & O_NONBLOCK)
-		msg_sys->msg_flags |= MSG_DONTWAIT;
 	/*
 	 * If this is sendmmsg() and current destination address is same as
 	 * previously succeeded address, omit asking LSM's decision.
@@ -2762,6 +2760,9 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
 
 	msg_sys->msg_name = &address;
 
+	if (sock->file->f_flags & O_NONBLOCK)
+		flags |= MSG_DONTWAIT;
+
 	err = sendmsg_copy_msghdr(msg_sys, msg, flags, &iov);
 	if (err < 0)
 		return err;
@@ -2926,9 +2927,6 @@ static int ____sys_recvmsg(struct socket *sock, struct msghdr *msg_sys,
 	/* We assume all kernel code knows the size of sockaddr_storage */
 	msg_sys->msg_namelen = 0;
 
-	if (sock->file->f_flags & O_NONBLOCK)
-		flags |= MSG_DONTWAIT;
-
 	if (unlikely(nosec))
 		err = sock_recvmsg_nosec(sock, msg_sys, flags);
 	else
@@ -2970,6 +2968,9 @@ static int ___sys_recvmsg(struct socket *sock, struct user_msghdr __user *msg,
 	struct sockaddr __user *uaddr;
 	ssize_t err;
 
+	if (sock->file->f_flags & O_NONBLOCK)
+		flags |= MSG_DONTWAIT;
+
 	err = recvmsg_copy_msghdr(msg_sys, msg, flags, &uaddr, &iov);
 	if (err < 0)
 		return err;
-- 
2.56.0


[-- Attachment #3: 0001-io_uring-poll-don-t-poll-a-writer-for-a-read-side-ha.patch --]
[-- Type: text/x-patch, Size: 2821 bytes --]

From 38459806e68413c84521a05d7eef3e89a2b29507 Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Thu, 29 Oct 2026 13:38:18 -0600
Subject: [PATCH 1/2] io_uring/poll: don't poll a writer for a read side hangup
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

io_init_poll_iocb() adds EPOLLRDHUP to every poll, including the async
poll of a send that waits for buffer space. Once the peer has shut down
its sending side, the socket keeps reporting EPOLLRDHUP, so each re-arm
in __io_arm_poll_handler() finds a ready mask and the request is issued
again. The send still can't make progress, the request burns through
APOLL_MAX_RETRY in a few milliseconds and ends up in io-wq.

A send in io-wq blocks until the peer reads, but sendmsg and writev go
through ____sys_sendmsg() and sock_write_iter(), which turn O_NONBLOCK
into MSG_DONTWAIT, and the application gets -EAGAIN for a request that
never asked for a non-blocking send.

Only add EPOLLRDHUP for polls that want to read. A peer that stopped
sending is no event for a writer, and a poll that asks for it explicitly
still gets it.

Fixes: 464dca612bc6 ("io_uring: mask in error/nval/hangup consistently for poll")
Reported-by: Lukáš Lalinský <lukas@lalinsky.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 io_uring/poll.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/io_uring/poll.c b/io_uring/poll.c
index 4f04bda42f7f..ac2979586b4f 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -128,12 +128,24 @@ static void io_poll_req_insert(struct io_kiocb *req)
 	hlist_add_head(&req->hash_node, &table->hbs[index].list);
 }
 
+#define IO_POLL_UNMASK	(EPOLLERR|EPOLLHUP|EPOLLNVAL)
+
+static __poll_t io_poll_unmask(__poll_t events)
+{
+	/*
+	 * A peer that stopped sending is an event for a reader, but a write
+	 * only poll must stay armed until the file is writable again.
+	 */
+	if (events & (EPOLLIN | EPOLLRDNORM | EPOLLRDBAND))
+		return IO_POLL_UNMASK | EPOLLRDHUP;
+	return IO_POLL_UNMASK;
+}
+
 static void io_init_poll_iocb(struct io_poll *poll, __poll_t events)
 {
 	poll->head = NULL;
-#define IO_POLL_UNMASK	(EPOLLERR|EPOLLHUP|EPOLLNVAL|EPOLLRDHUP)
 	/* mask in events that we always want/need */
-	poll->events = events | IO_POLL_UNMASK;
+	poll->events = events | io_poll_unmask(events);
 	INIT_LIST_HEAD(&poll->wait.entry);
 	init_waitqueue_func_entry(&poll->wait, io_poll_wake);
 }
@@ -971,7 +983,7 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags)
 
 			poll->events &= ~0xffff;
 			poll->events |= poll_update->events & 0xffff;
-			poll->events |= IO_POLL_UNMASK;
+			poll->events |= io_poll_unmask(poll->events);
 		}
 		if (poll_update->update_user_data)
 			preq->cqe.user_data = poll_update->new_user_data;
-- 
2.56.0

Re: io_uring: SENDMSG/WRITEV return -EAGAIN on O_NONBLOCK socket after peer half-close

[Lukáš Lalinský]

On Thu, Oct 29, 2026, at 8:48 PM, Jens Axboe wrote:
> Can you try with these two patches? I think the problem is two-fold,
> each patch tries to reason about that.

Thanks for the patches! I didn't expect a fix so quickly. I tested them
on current master in a VM, each patch separately and both together.

Patch 1 alone fixes the problem I reported. After the peer's
half-close, SEND, SENDMSG and WRITEV all keep waiting and complete
normally once the peer reads, and the waiting requests no longer end
up in io-wq workers (100 waiting sends, 0 workers, vs 100 workers
before).

There are other ways to hit the same -EAGAIN though. For example, if
the peer sends TCP urgent data (MSG_OOB) that the application doesn't
read, a SENDMSG on an O_NONBLOCK socket that waits for buffer space
completes with -EAGAIN, even with patch 1. A waiting RECVMSG can also
complete with -EAGAIN, when data keeps arriving but is taken by
another reader of the same socket each time. Patch 2 fixes both cases.

There is one more inconsistency even with both patches applied:
in the urgent data case, WRITEV on an O_NONBLOCK socket still completes
with -EAGAIN, while SEND and SENDMSG on the same socket keep waiting.
So whether WRITEV waits or fails depends on whether the peer sent
urgent data, which the application can't really predict. It's not a
problem for me, since I always use SEND/SENDMSG on sockets, but it is
an inconsistent behavior.

Lukas

Re: io_uring: SENDMSG/WRITEV return -EAGAIN on O_NONBLOCK socket after peer half-close

[Jens Axboe]

On 11/2/26 7:00 AM, Luk?? Lalinsk? wrote:
> On Thu, Oct 29, 2026, at 9:42 PM, Luk?? Lalinsk? wrote:
>> There is one more inconsistency even with both patches applied:
>> in the urgent data case, WRITEV on an O_NONBLOCK socket still completes
>> with -EAGAIN, while SEND and SENDMSG on the same socket keep waiting.
>> So whether WRITEV waits or fails depends on whether the peer sent
>> urgent data, which the application can't really predict. It's not a
>> problem for me, since I always use SEND/SENDMSG on sockets, but it is
>> an inconsistent behavior.
> 
> Hi Jens,
> 
> I ran into one more issue while testing this. The SEND or SENDMSG
> operations become uncancelable for a brief time after submission,
> either in the half-closed state before your patches, or with the urgent
> data flag after your patches. If ASYNC_CANCEL is submitted right
> after the send on such socket, it completes with -ENOENT, but the
> send stays pending. A second cancel issued later finds it. Without the
> urgent data, the first cancel finds it and the send completes with
> -ECANCELED. Your patches made it work properly for the half-closed
> case.

Can you try with this on top?

diff --git a/io_uring/poll.c b/io_uring/poll.c
index ac2979586b4f..82485e1350dd 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -739,7 +739,7 @@ int io_arm_apoll(struct io_kiocb *req, unsigned issue_flags, __poll_t mask)
 int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags)
 {
 	const struct io_issue_def *def = &io_issue_defs[req->opcode];
-	__poll_t mask = POLLPRI | POLLERR;
+	__poll_t mask = EPOLLERR;
 
 	if (!def->pollin && !def->pollout)
 		return IO_APOLL_ABORTED;
@@ -747,7 +747,8 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags)
 		return IO_APOLL_ABORTED;
 
 	if (def->pollin) {
-		mask |= EPOLLIN | EPOLLRDNORM;
+		/* urgent data is an event for a reader only */
+		mask |= EPOLLIN | EPOLLRDNORM | EPOLLPRI;
 
 		/* If reading from MSG_ERRQUEUE using recvmsg, ignore POLLIN */
 		if (req->flags & REQ_F_CLEAR_POLLIN)

-- 
Jens Axboe

Re: io_uring: SENDMSG/WRITEV return -EAGAIN on O_NONBLOCK socket after peer half-close

[Lukáš Lalinský]

On Tue, Nov 3, 2026, at 5:00 PM, Jens Axboe wrote:
> Can you try with this on top?

Hi Jens,

I tested the three patches together on current master. Everything I
know of is fixed:

- After the peer's half-close, a waiting SEND, SENDMSG or WRITEV keeps
  waiting and completes normally once the peer reads, with or without
  O_NONBLOCK.
- The same now holds when the peer has sent urgent data that the
  application doesn't read.
- A RECVMSG that is woken repeatedly without data no longer completes
  with -EAGAIN.
- The cancel now finds the send, both after a half-close and with
  unread urgent data, and the send completes with -ECANCELED.

Our runtime's test suite also passes on the patched kernel.

Thanks!

Lukas

Re: io_uring: SENDMSG/WRITEV return -EAGAIN on O_NONBLOCK socket after peer half-close

[Jens Axboe]

On 11/6/26 5:04 AM, Lukáš Lalinský wrote:
> On Tue, Nov 3, 2026, at 5:00 PM, Jens Axboe wrote:
>> Can you try with this on top?
> 
> Hi Jens,
> 
> I tested the three patches together on current master. Everything I
> know of is fixed:
> 
> - After the peer's half-close, a waiting SEND, SENDMSG or WRITEV keeps
>   waiting and completes normally once the peer reads, with or without
>   O_NONBLOCK.
> - The same now holds when the peer has sent urgent data that the
>   application doesn't read.
> - A RECVMSG that is woken repeatedly without data no longer completes
>   with -EAGAIN.
> - The cancel now finds the send, both after a half-close and with
>   unread urgent data, and the send completes with -ECANCELED.
> 
> Our runtime's test suite also passes on the patched kernel.

Great! Patch 1+3 are in Linus's tree now, and I sent out the net
patch as well to netdev just now. Hopefully all will land in
7.4, and I will ensure they all go to stable as well.

-- 
Jens Axboe