af-unix: ECONNRESET with fully consumed out-of-band data

af-unix: ECONNRESET with fully consumed out-of-band data

[Christian Brauner]

Hey,

I've played with out-of-band data on unix sockets and I'm observing strange
behavior. Below is a minimal reproducer.

This is sending exactly one byte of out-of-band data from the client to the
server. The client shuts down the write side aftewards and issues a blocking
read waiting for the server to sever the connection.

The server consumes the single byte of out-of-band data sent by the client and
closes the connection.

The client should see a zero read as all data has been consumed but instead it
sees ECONNRESET indicating an unclean shutdown.

But it's even stranger. If the server issues a regular data read() after
consuming the single out-of-band byte it will get a zero read indicating EOF as
the child shutdown the write side. The fun part is that this zero read in the
parent also makes the child itself see a zero read/EOF after the client severs
the connection indicating a clean shutdown. Which makes no sense to me
whatsoever.

In contrast, when sending exactly one byte of regular data the client sees a
zero read aka EOF correctly indicating a clean shutdown.

It seems a bug to me that a single byte of out-of-band data leads to an unclean
shutdown even though it has been correctly consumed and there's no more data
left in the socket.

Maybe that's expected and there's a reasonable explanation but that's very
unexpected behavior.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <errno.h>
#include <sys/wait.h>

int main(void) {
	int sv[2];
	pid_t pid;
	char buf[16];
	ssize_t n;

	if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0)
		_exit(EXIT_FAILURE);

	pid = fork();
	if (pid < 0)
		_exit(EXIT_FAILURE);

	if (pid == 0) {
		close(sv[0]);

		/* Send OOB data to the server. */
		printf("child: %zd\n", send(sv[1], "1", 1, MSG_OOB));

		/* We're done sending data so shutdown the write side. */
		shutdown(sv[1], SHUT_WR);

		/* We expect to see EOF here, but we see ECONNRESET instead. */
		if (read(sv[1], buf, 1) != 0) {
			fprintf(stderr, "%d => %m - Child read did not return EOF\n", errno);
			_exit(EXIT_FAILURE);
		}

		_exit(EXIT_SUCCESS);
	}

	/* The parent acts as a client here. */
	close(sv[1]);

	/* Hack: MSG_OOB doesn't block, so we need to make sure the OOB data has arrived. */
	sleep(2);
	
	/* Read the OOB data. */
	printf("%zd\n", recv(sv[0], buf, sizeof(buf), MSG_OOB));

	/* If you uncomment the following code you can make the child see a zero read/EOF: */
	// printf("%zd\n", read(sv[0], buf, sizeof(buf)));

	/*
	 * Close the connection. The child should see EOF but sees ECONNRESET instead...
	 * Try removing MSG_OOB and see how the child sees EOF instead.
	 */
	close(sv[0]);

	waitpid(pid, NULL, 0);
	_exit(EXIT_SUCCESS);
}

Re: af-unix: ECONNRESET with fully consumed out-of-band data

[Kuniyuki Iwashima]

From: Christian Brauner <brauner@kernel.org>
Date: Thu, 29 May 2025 12:37:54 +0200
> Hey,
> 
> I've played with out-of-band data on unix sockets and I'm observing strange
> behavior. Below is a minimal reproducer.
> 
> This is sending exactly one byte of out-of-band data from the client to the
> server. The client shuts down the write side aftewards and issues a blocking
> read waiting for the server to sever the connection.
> 
> The server consumes the single byte of out-of-band data sent by the client and
> closes the connection.
> 
> The client should see a zero read as all data has been consumed but instead it
> sees ECONNRESET indicating an unclean shutdown.
> 
> But it's even stranger. If the server issues a regular data read() after
> consuming the single out-of-band byte it will get a zero read indicating EOF as
> the child shutdown the write side. The fun part is that this zero read in the
> parent also makes the child itself see a zero read/EOF after the client severs
> the connection indicating a clean shutdown. Which makes no sense to me
> whatsoever.
> 
> In contrast, when sending exactly one byte of regular data the client sees a
> zero read aka EOF correctly indicating a clean shutdown.
> 
> It seems a bug to me that a single byte of out-of-band data leads to an unclean
> shutdown even though it has been correctly consumed and there's no more data
> left in the socket.

Thanks for the report!

This is definitely a bug.  Even after reading the OOB data, skb holding
the 1 byte must stay in the recv queue to mark the OOB boundary.

So, we need to consider that when close()ing a socket like:

---8<---
diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index bd507f74e35e..13d5d53c0e53 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -654,6 +654,11 @@ static void unix_sock_destructor(struct sock *sk)
 #endif
 }
 
+static unsigned int unix_skb_len(const struct sk_buff *skb)
+{
+	return skb->len - UNIXCB(skb).consumed;
+}
+
 static void unix_release_sock(struct sock *sk, int embrion)
 {
 	struct unix_sock *u = unix_sk(sk);
@@ -681,6 +686,11 @@ static void unix_release_sock(struct sock *sk, int embrion)
 	unix_state_unlock(sk);
 
 #if IS_ENABLED(CONFIG_AF_UNIX_OOB)
+	skb = skb_peek(&sk->sk_receive_queue);
+	if (skb && !unix_skb_len(skb)) {
+		__skb_unlink(skb, &sk->sk_receive_queue);
+		consume_skb(skb);
+	}
 	u->oob_skb = NULL;
 #endif
 
@@ -2569,11 +2579,6 @@ static long unix_stream_data_wait(struct sock *sk, long timeo,
 	return timeo;
 }
 
-static unsigned int unix_skb_len(const struct sk_buff *skb)
-{
-	return skb->len - UNIXCB(skb).consumed;
-}
-
 struct unix_stream_read_state {
 	int (*recv_actor)(struct sk_buff *, int, int,
 			  struct unix_stream_read_state *);
---8<---

I'll post an official patch later.


> 
> Maybe that's expected and there's a reasonable explanation but that's very
> unexpected behavior.
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <unistd.h>
> #include <sys/socket.h>
> #include <errno.h>
> #include <sys/wait.h>
> 
> int main(void) {
> 	int sv[2];
> 	pid_t pid;
> 	char buf[16];
> 	ssize_t n;
> 
> 	if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0)
> 		_exit(EXIT_FAILURE);
> 
> 	pid = fork();
> 	if (pid < 0)
> 		_exit(EXIT_FAILURE);
> 
> 	if (pid == 0) {
> 		close(sv[0]);
> 
> 		/* Send OOB data to the server. */
> 		printf("child: %zd\n", send(sv[1], "1", 1, MSG_OOB));
> 
> 		/* We're done sending data so shutdown the write side. */
> 		shutdown(sv[1], SHUT_WR);
> 
> 		/* We expect to see EOF here, but we see ECONNRESET instead. */
> 		if (read(sv[1], buf, 1) != 0) {
> 			fprintf(stderr, "%d => %m - Child read did not return EOF\n", errno);
> 			_exit(EXIT_FAILURE);
> 		}
> 
> 		_exit(EXIT_SUCCESS);
> 	}
> 
> 	/* The parent acts as a client here. */
> 	close(sv[1]);
> 
> 	/* Hack: MSG_OOB doesn't block, so we need to make sure the OOB data has arrived. */
> 	sleep(2);
> 	
> 	/* Read the OOB data. */
> 	printf("%zd\n", recv(sv[0], buf, sizeof(buf), MSG_OOB));
> 
> 	/* If you uncomment the following code you can make the child see a zero read/EOF: */
> 	// printf("%zd\n", read(sv[0], buf, sizeof(buf)));
> 
> 	/*
> 	 * Close the connection. The child should see EOF but sees ECONNRESET instead...
> 	 * Try removing MSG_OOB and see how the child sees EOF instead.
> 	 */
> 	close(sv[0]);
> 
> 	waitpid(pid, NULL, 0);
> 	_exit(EXIT_SUCCESS);
> }
>