af-unix: ECONNRESET with fully consumed out-of-band data

[Christian Brauner <brauner@kernel.org>]

Hey,

I've played with out-of-band data on unix sockets and I'm observing strange
behavior. Below is a minimal reproducer.

This is sending exactly one byte of out-of-band data from the client to the
server. The client shuts down the write side aftewards and issues a blocking
read waiting for the server to sever the connection.

The server consumes the single byte of out-of-band data sent by the client and
closes the connection.

The client should see a zero read as all data has been consumed but instead it
sees ECONNRESET indicating an unclean shutdown.

But it's even stranger. If the server issues a regular data read() after
consuming the single out-of-band byte it will get a zero read indicating EOF as
the child shutdown the write side. The fun part is that this zero read in the
parent also makes the child itself see a zero read/EOF after the client severs
the connection indicating a clean shutdown. Which makes no sense to me
whatsoever.

In contrast, when sending exactly one byte of regular data the client sees a
zero read aka EOF correctly indicating a clean shutdown.

It seems a bug to me that a single byte of out-of-band data leads to an unclean
shutdown even though it has been correctly consumed and there's no more data
left in the socket.

Maybe that's expected and there's a reasonable explanation but that's very
unexpected behavior.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <errno.h>
#include <sys/wait.h>

int main(void) {
	int sv[2];
	pid_t pid;
	char buf[16];
	ssize_t n;

	if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0)
		_exit(EXIT_FAILURE);

	pid = fork();
	if (pid < 0)
		_exit(EXIT_FAILURE);

	if (pid == 0) {
		close(sv[0]);

		/* Send OOB data to the server. */
		printf("child: %zd\n", send(sv[1], "1", 1, MSG_OOB));

		/* We're done sending data so shutdown the write side. */
		shutdown(sv[1], SHUT_WR);

		/* We expect to see EOF here, but we see ECONNRESET instead. */
		if (read(sv[1], buf, 1) != 0) {
			fprintf(stderr, "%d => %m - Child read did not return EOF\n", errno);
			_exit(EXIT_FAILURE);
		}

		_exit(EXIT_SUCCESS);
	}

	/* The parent acts as a client here. */
	close(sv[1]);

	/* Hack: MSG_OOB doesn't block, so we need to make sure the OOB data has arrived. */
	sleep(2);
	
	/* Read the OOB data. */
	printf("%zd\n", recv(sv[0], buf, sizeof(buf), MSG_OOB));

	/* If you uncomment the following code you can make the child see a zero read/EOF: */
	// printf("%zd\n", read(sv[0], buf, sizeof(buf)));

	/*
	 * Close the connection. The child should see EOF but sees ECONNRESET instead...
	 * Try removing MSG_OOB and see how the child sees EOF instead.
	 */
	close(sv[0]);

	waitpid(pid, NULL, 0);
	_exit(EXIT_SUCCESS);
}