[possible regression in 6.5-rc1] sendmsg()/splice() returns EBADMSG

[possible regression in 6.5-rc1] sendmsg()/splice() returns EBADMSG

[Tetsuo Handa]

(Branched from https://lkml.kernel.org/r/63006262-f808-50ab-97b8-c2193c7a9ba1@I-love.SAKURA.ne.jp .)

I found that the following program started returning EBADMSG. Bisection for sendmsg() reached
commit c5c37af6ecad ("tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES") and bisection
for splice() reached commit 2dc334f1a63a ("splice, net: Use sendmsg(MSG_SPLICE_PAGES) rather
than ->sendpage()"). Is this program doing something wrong?

  6.4.0-rc5-00892-g2dc334f1a63a-dirty    argc==1 ? splice()=EBADMSG, sendmsg()=EBADMSG : sendmsg()=success, splice()=EBADMSG
  6.4.0-rc5-00891-g81840b3b91aa-dirty    argc==1 ? splice()=success, sendmsg()=EBADMSG : sendmsg()=success, splice()=success

  6.4.0-rc2-00520-gc5c37af6ecad-dirty    argc==1 ? splice()=success, sendmsg()=EBADMSG : sendmsg()=success, splice()=success
  6.4.0-rc2-00519-g270a1c3de47e-dirty    argc==1 ? splice()=success, sendmsg()=success : sendmsg()=success, splice()=success

----------------------------------------
#define _GNU_SOURCE
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#define SOL_TCP 6
#define TCP_REPAIR 19
#define TCP_ULP 31
#define TLS_TX 1

int main(int argc, char *argv[])
{
	struct iovec iov = {
		.iov_base = "@@@@@@@@@@@@@@@@",
		.iov_len = 16
	};
	struct msghdr hdr = {
		.msg_iov = &iov,
		.msg_iovlen = 1,
		.msg_flags = MSG_FASTOPEN
	};
	const struct sockaddr_in6 addr = { .sin6_family = AF_INET6, .sin6_addr = in6addr_loopback };
	const int one = 1;
	int ret_ignored = 0;
	const int fd = socket(PF_INET6, SOCK_STREAM, IPPROTO_IP);
	int pipe_fds[2] = { -1, -1 };
	static char buf[32768] = { };

	ret_ignored += pipe(pipe_fds);
	setsockopt(fd, SOL_TCP, TCP_REPAIR, &one, sizeof(one));
	connect(fd, (struct sockaddr *) &addr, sizeof(addr));
	setsockopt(fd, SOL_TCP, TCP_ULP, "tls", 4);
	setsockopt(fd, SOL_TLS, TLS_TX,"\3\0035\0%T\244\205\333\f0\362B\221\243\234\206\216\220\243u\347\342P|1\24}Q@\377\227\353\222B\354\264u[\346", 40);
	ret_ignored += write(pipe_fds[1], buf, 2432);
	ret_ignored += write(pipe_fds[1], buf, 10676);
	ret_ignored += write(pipe_fds[1], buf, 17996);
	if (argc == 1) {
		ret_ignored += splice(pipe_fds[0], NULL, fd, NULL, 1048576, SPLICE_F_MORE);
		ret_ignored += sendmsg(fd, &hdr, MSG_DONTWAIT | MSG_MORE);
	} else {
		ret_ignored += sendmsg(fd, &hdr, MSG_DONTWAIT | MSG_MORE);
		ret_ignored += splice(pipe_fds[0], NULL, fd, NULL, 1048576, SPLICE_F_MORE);
	}
	return ret_ignored * 0;
}
----------------------------------------

Re: [possible regression in 6.5-rc1] sendmsg()/splice() returns EBADMSG

[Jakub Kicinski]

On Sat, 8 Jul 2023 08:45:50 +0900 Tetsuo Handa wrote:
> (Branched from https://lkml.kernel.org/r/63006262-f808-50ab-97b8-c2193c7a9ba1@I-love.SAKURA.ne.jp .)
> 
> I found that the following program started returning EBADMSG. Bisection for sendmsg() reached
> commit c5c37af6ecad ("tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES") and bisection
> for splice() reached commit 2dc334f1a63a ("splice, net: Use sendmsg(MSG_SPLICE_PAGES) rather
> than ->sendpage()"). Is this program doing something wrong?
> 
>   6.4.0-rc5-00892-g2dc334f1a63a-dirty    argc==1 ? splice()=EBADMSG, sendmsg()=EBADMSG : sendmsg()=success, splice()=EBADMSG
>   6.4.0-rc5-00891-g81840b3b91aa-dirty    argc==1 ? splice()=success, sendmsg()=EBADMSG : sendmsg()=success, splice()=success
> 
>   6.4.0-rc2-00520-gc5c37af6ecad-dirty    argc==1 ? splice()=success, sendmsg()=EBADMSG : sendmsg()=success, splice()=success
>   6.4.0-rc2-00519-g270a1c3de47e-dirty    argc==1 ? splice()=success, sendmsg()=success : sendmsg()=success, splice()=success

> 	setsockopt(fd, SOL_TCP, TCP_REPAIR, &one, sizeof(one));

I think it's just because the repro puts the socket in repair mode, 
and the current code doesn't want to play with repair mode as nicely.

I added:
	// needs #include <linux/tcp.h>
        int val = TCP_SEND_QUEUE;                                                   
        setsockopt(fd, SOL_TCP, TCP_REPAIR_QUEUE, &val, sizeof(val)); 

here (i.e. after putting the socket in repair mode), and I don't get 
the EBADMSG any more. Both sendmsg and splice succeed.

Can you check if we're back to the KMSAN problem with those lines added?


FWIW you can also try to repro with real tls sockets (not in repair
mode) by adding cases to tools/testing/selftests/net/tls.c for example:

TEST_F(tls, bla_bla)
{
	struct iovec iov = {
		.iov_base = "@@@@@@@@@@@@@@@@",
		.iov_len = 16
	};
	struct msghdr hdr = {
		.msg_iov = &iov,
		.msg_iovlen = 1,
		.msg_flags = MSG_FASTOPEN
	};
	int pipe_fds[2] = { -1, -1 };
	static char buf[32768] = { };
	int ret;

	ret = pipe(pipe_fds);
	ASSERT_EQ(ret, 0);

	EXPECT_EQ(write(pipe_fds[1], buf, 2432), 2432);
	EXPECT_EQ(write(pipe_fds[1], buf, 10676), 10676);
	EXPECT_EQ(write(pipe_fds[1], buf, 17996), 17996);

	EXPECT_EQ(splice(pipe_fds[0], NULL, self->fd, NULL, 1048576,
			 SPLICE_F_MORE), 17996 + 10676 + 2432);
	EXPECT_EQ(sendmsg(self->fd, &hdr, MSG_DONTWAIT | MSG_MORE), 16);
}

Then compiling:

make -C tools/testing/selftests/net/

And running:

tools/testing/selftests/net/tls -r tls.13_aes_gcm_256.bla_bla