Linux io-uring development
 help / color / mirror / Atom feed
* io_uring: SENDMSG/WRITEV return -EAGAIN on O_NONBLOCK socket after peer half-close
@ 2026-10-29 18:54 Lukáš Lalinský
  2026-10-29 19:48 ` Jens Axboe
  0 siblings, 1 reply; 6+ messages in thread
From: Lukáš Lalinský @ 2026-10-29 18:54 UTC (permalink / raw)
  To: Jens Axboe; +Cc: io-uring

Hi,

IORING_OP_SENDMSG can complete with -EAGAIN even though MSG_DONTWAIT
was not used. It happens when the send is waiting for buffer space, the
peer calls shutdown(SHUT_WR), and the socket has O_NONBLOCK set.

IORING_OP_SEND in the same situation keeps waiting and completes
normally once the peer reads. So does SENDMSG on a socket without
O_NONBLOCK. IORING_OP_WRITEV behaves the same way as SENDMSG.

I consider this a bug. Based on previous discussions [1][2], io_uring
should ignore O_NONBLOCK for operations that take a per-call nonblock
flag. SENDMSG mostly does: on the full socket the request doesn't fail
immediately, it waits. It only fails after the peer half-closes. At the
very least it's inconsistent, because SEND and SENDMSG/WRITEV behave
differently on the same socket.

There is no good option for handling the error. Resubmitting fails
again the same way. Waiting for writability with
IORING_OP_POLL_ADD(POLLOUT) doesn't work after a half-close either, it
completes immediately with POLLRDHUP, even though POLLRDHUP wasn't
requested. The only workarounds are dropping the O_NONBLOCK flag, or
using a separate epoll to wait for writability.

If I'm reading the code correctly, after the half-close EPOLLRDHUP
keeps waking the request's internal poll. The request runs out of
APOLL_MAX_RETRY and is punted to io-wq, where the blocking attempt goes
through __sys_sendmsg_sock() -> ____sys_sendmsg(). That function turns
O_NONBLOCK into MSG_DONTWAIT, so the attempt returns -EAGAIN instead of
blocking. io_send() calls sock_sendmsg() directly, which is why SEND
blocks in the worker instead. WRITEV seems to hit the same thing in
sock_write_iter(), and IORING_OP_RECVMSG through ____sys_recvmsg(),
after enough wakeups that find no data.

We hit this in zio, an async runtime for Zig, via an HTTP/2 server's
half-close tests [3]. ringline, a Rust io_uring runtime, hit it
independently on 6.12 and 7.1 [4], and libosmocore seems to have hit
the same -EAGAIN and worked around it by clearing O_NONBLOCK [5]. We'll
work around it in zio for now.

Here is a minimal reproducer:

/*
 * IORING_OP_SENDMSG or IORING_OP_WRITEV with flags 0 is parked on a
 * full AF_UNIX stream socket. If the socket has O_NONBLOCK and the
 * peer then calls shutdown(SHUT_WR), the request completes with
 * -EAGAIN. IORING_OP_SEND, or any of them without O_NONBLOCK, keeps
 * waiting and completes once the peer reads.
 *
 * Build: cc -O2 -Wall -o sendmsg_eagain sendmsg_eagain.c -luring
 */
#include <errno.h>
#include <fcntl.h>
#include <liburing.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/utsname.h>
#include <unistd.h>

#define CHECK(x) do { if (!(x)) { perror(#x); exit(2); } } while (0)

enum { OP_SEND, OP_SENDMSG, OP_WRITEV };
static const char *const op_names[] = {
	"SEND   ", "SENDMSG", "WRITEV ",
};

static char buf[65536];

static int run(int op, int nonblock)
{
	struct __kernel_timespec ts = { .tv_sec = 1 };
	struct iovec iov = { .iov_base = buf, .iov_len = sizeof(buf) };
	struct msghdr msg = { .msg_iov = &iov, .msg_iovlen = 1 };
	struct io_uring_sqe *sqe;
	struct io_uring_cqe *cqe;
	struct io_uring ring;
	char sink[65536];
	int sv[2], ret, res;

	CHECK(socketpair(AF_UNIX, SOCK_STREAM, 0, sv) == 0);
	if (nonblock)
		CHECK(fcntl(sv[0], F_SETFL,
			    fcntl(sv[0], F_GETFL) | O_NONBLOCK) == 0);

	/* Fill the send buffer. */
	while (send(sv[0], buf, sizeof(buf), MSG_DONTWAIT) > 0)
		;
	CHECK(errno == EAGAIN);

	ret = io_uring_queue_init(4, &ring, 0);
	if (ret) {
		fprintf(stderr, "queue_init: %s\n", strerror(-ret));
		exit(2);
	}
	sqe = io_uring_get_sqe(&ring);
	if (op == OP_SEND)
		io_uring_prep_send(sqe, sv[0], buf, sizeof(buf), 0);
	else if (op == OP_SENDMSG)
		io_uring_prep_sendmsg(sqe, sv[0], &msg, 0);
	else
		io_uring_prep_writev(sqe, sv[0], &iov, 1, 0);
	CHECK(io_uring_submit(&ring) == 1);

	/* The request must be waiting for space, not completed. */
	CHECK(io_uring_peek_cqe(&ring, &cqe) == -EAGAIN);

	CHECK(shutdown(sv[1], SHUT_WR) == 0);

	ret = io_uring_wait_cqe_timeout(&ring, &cqe, &ts);
	if (ret == 0) {
		res = cqe->res;
		printf("%s O_NONBLOCK=%d: res=%d after peer shutdown\n",
		       op_names[op], nonblock, res);
	} else {
		CHECK(ret == -ETIME);
		/* Still waiting after 1s: drain the peer until done. */
		do {
			while (recv(sv[1], sink, sizeof(sink),
				    MSG_DONTWAIT) > 0)
				;
			ts.tv_sec = 0;
			ts.tv_nsec = 10 * 1000 * 1000;
			ret = io_uring_wait_cqe_timeout(&ring, &cqe,
							&ts);
		} while (ret == -ETIME);
		CHECK(ret == 0);
		res = cqe->res;
		printf("%s O_NONBLOCK=%d: res=%d after peer read\n",
		       op_names[op], nonblock, res);
	}
	io_uring_cqe_seen(&ring, cqe);
	io_uring_queue_exit(&ring);
	close(sv[0]);
	close(sv[1]);
	return res == sizeof(buf) ? 0 : 1;
}

int main(void)
{
	struct utsname u;
	int op, fail = 0;

	CHECK(uname(&u) == 0);
	printf("kernel %s, liburing %d.%d\n", u.release,
	       io_uring_major_version(), io_uring_minor_version());
	for (op = OP_SEND; op <= OP_WRITEV; op++) {
		fail |= run(op, 0);
		fail |= run(op, 1);
	}
	return fail;
}

Output:

kernel 7.0.0-38-generic, liburing 2.5
SEND    O_NONBLOCK=0: res=65536 after peer read
SEND    O_NONBLOCK=1: res=65536 after peer read
SENDMSG O_NONBLOCK=0: res=65536 after peer read
SENDMSG O_NONBLOCK=1: res=-11 after peer shutdown
WRITEV  O_NONBLOCK=0: res=65536 after peer read
WRITEV  O_NONBLOCK=1: res=-11 after peer shutdown

The relevant code is unchanged in current master.

[1] https://lore.kernel.org/io-uring/e83c9ef0-8016-2c4a-9dd0-b2a1318238ef@kernel.dk/
[2] https://github.com/axboe/liburing/issues/1270
[3] https://github.com/lalinsky/zio/issues/864
[4] https://github.com/ringline-rs/ringline/issues/603
[5] https://gerrit.osmocom.org/c/libosmocore/+/41671

Lukas

^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-11-06 23:54 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-29 18:54 io_uring: SENDMSG/WRITEV return -EAGAIN on O_NONBLOCK socket after peer half-close Lukáš Lalinský
2026-10-29 19:48 ` Jens Axboe
2026-10-29 20:42   ` Lukáš Lalinský
     [not found]     ` <5475728c-6cd9-4d61-bcee-5c8a42f1951d@app.fastmail.com>
2026-11-03 16:00       ` Jens Axboe
2026-11-06 12:04         ` Lukáš Lalinský
2026-11-06 23:54           ` Jens Axboe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox