[RFC net] tls: TLS_SW sendfile() stalls at large MSS

[RFC net] tls: TLS_SW sendfile() stalls at large MSS

[Mike Fara]

Hi,

Software-kTLS (TLS_SW) TX over sendfile()/splice() drops to the TCP
persist-timer cadence (tens of KB/s, with individual sendfile() calls blocking
for tens of seconds) when the path MSS is large -- e.g. loopback (MSS 65483) or
jumbo frames. At a typical 1448-byte MSS it does not occur. Plain TCP
sendfile() on the same path is unaffected, and kTLS write() (no splice) is
unaffected, so it is specific to TLS_SW + the splice/sendfile path.

It triggers only on large-MSS paths with software kTLS (no NIC TLS offload), so
it is a niche path -- but it is a clean, reproducible multi-order-of-magnitude
cliff, so it seems worth a look. Reproduces on current mainline. CCing David
Howells as the author of the 2023 sendpage->MSG_SPLICE_PAGES splice_to_socket()
rework referenced below, and Eric/Paolo as this is as much a TCP-corking
interaction as a TLS one.

Environment
-----------
- net/tls TLS_SW (no NIC offload; ethtool tls-hw-tx-offload: off [fixed]).
- AES-GCM; gcm(aes) resolves to generic-gcm-vaes-avx512.

Reproducer (no OpenSSL/handshake; TLS_TX programmed with a fixed key, the
receiver discards ciphertext, like tools/testing/selftests/net/tls.c):

    cc -O2 -Wall -o ktls_sendfile_stall ktls_sendfile_stall.c
    ./ktls_sendfile_stall          # default loopback MSS (65483)
    ./ktls_sendfile_stall 1448     # clamp sender MSS via TCP_MAXSEG

Observed (loopback, single box):

    MSS=default    sent=     4.0 MiB in  52.08s  =>  0.0001 GiB/s   (stalled)
    MSS=1448       sent=  2048.0 MiB in   1.65s  =>  1.2106 GiB/s

i.e. ~four orders of magnitude; at the default MSS a single sendfile() blocks
for tens of seconds. For contrast, on the same loopback path:

    plain TCP sendfile() (no TLS ULP):           7.87  GiB/s
    kTLS write() (TLS_SW, no splice, 2 GiB):      1.99  GiB/s

Analysis
--------
During the stall the sending thread is parked here:

    [<0>] sk_stream_wait_memory+0x256/0x380
    [<0>] tls_sw_sendmsg+0x1f1/0xc40       /* tls_sw_sendmsg_locked, inlined */
    [<0>] inet_sendmsg+0x7f/0x90
    [<0>] sock_sendmsg+0x183/0x1a0
    [<0>] splice_to_socket+0x3e0/0x5b0
    [<0>] splice_direct_to_actor+0xf7/0x2c0
    [<0>] do_splice_direct+0x71/0xd0
    [<0>] do_sendfile+0x390/0x440

and ss(8) shows exactly one completed TLS record held, behind a persist timer,
with the peer window wide open:

    ESTAB ... timer:(persist,028ms,0) ... notsent:16406 snd_wnd:1114112
            ... mss:65483 ... tcp-ulp-tls version: 1.3 ... txconf: sw

notsent:16406 is one TLS1.3 record (TLS_MAX_PAYLOAD_SIZE 16384 + 22). It is a
*completed* record, yet TCP is corking it.

The chain appears to be:

  1. For sendfile, splice_direct_to_actor() sets SPLICE_F_MORE on every
     iteration and clears it only on the final chunk that fulfils the request,
     so splice_to_socket() passes MSG_MORE on every send but the last. (This is
     the intended coalescing behaviour from the 2023 MSG_SPLICE_PAGES rework;
     naming it as the origin is a hypothesis, not a confirmed bisect.)
  2. tls_sw_sendmsg_locked() forwards msg->msg_flags (incl. MSG_MORE) into
     bpf_exec_tx_verdict()/tls_push_record() even for a *full* record, so the
     completed record reaches tcp_sendmsg_locked() (via tls_push_sg(), which
     builds msghdr.msg_flags = MSG_SPLICE_PAGES | flags) with MSG_MORE set.
  3. With a large MSS the 16 KB record is far below MSS, so TCP corks it
     waiting to fill a segment.
  4. tls_sw then can't build the next record -- it blocks in
     sk_stream_wait_memory() for memory the corked record is holding.
  5. tcp_write_xmit() leaves the corked record unsent with packets_out == 0, so
     tcp_check_probe_timer() arms the persist (probe-0) timer even though the
     window is open; each expiry pushes ~one record -> the observed rate. At a
     1448-byte MSS each 16 KB record already exceeds MSS and is sent, so the
     cork never engages.

Candidate fix (illustrative sketch -- NOT a submittable patch, no S-o-b; the
right shape is your call given the coalescing intent). A full TLS record is a
natural transmit boundary, so arguably MSG_MORE should not be honoured for it,
only for a trailing partial record. In tls_sw_sendmsg_locked():

    if (full_record || eor) {
+           unsigned int send_flags = msg->msg_flags;
+
+           if (full_record)
+                   send_flags &= ~MSG_MORE;
            ret = bpf_exec_tx_verdict(msg_pl, sk, full_record,
-                                     record_type, &copied, msg->msg_flags);
+                                     record_type, &copied, send_flags);

Caveats I'm aware of: (a) there are two bpf_exec_tx_verdict() call sites in the
function passing msg->msg_flags -- the splice path reaches the one at `copied:`
label (via `goto copied`), so this one-site change covers the repro, but a
complete fix would clear MSG_MORE at both; (b) this stops coalescing each
record's trailing partial into the next TSO segment, partly undoing the 2023
optimisation, so a narrower fix that only flushes when about to block in
sk_stream_wait_memory() may be preferable. Happy to spin whichever you prefer
and run it through the tls selftests.

Thanks,
Mike Fara

--- ktls_sendfile_stall.c ---
// ktls_sendfile_stall.c
//
// Minimal, dependency-free reproducer for a software-kTLS (TLS_SW) TX stall:
// sendfile()/splice() over a kTLS socket collapses to the TCP persist-timer
// cadence when the path MSS is large (loopback's 65483, or jumbo frames),
// because splice sets MSG_MORE on every chunk but the last and tls_sw forwards
// it to TCP even for *completed* records, so TCP corks the sub-MSS record while
// tls_sw blocks in sk_stream_wait_memory().
//
// No handshake / no OpenSSL: TLS_TX is programmed with a fixed key (the receiver
// discards ciphertext; we only measure the TX path), exactly like the in-tree
// tls selftest. Clamping the sender MSS to a realistic value makes the stall
// vanish, isolating the large-MSS amplifier.
//
//   cc -O2 -Wall -o ktls_sendfile_stall ktls_sendfile_stall.c
//   ./ktls_sendfile_stall          # default loopback MSS (65483) -> stalls
//   ./ktls_sendfile_stall 1448     # clamp sender MSS via TCP_MAXSEG -> normal
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <signal.h>
#include <time.h>
#include <sys/socket.h>
#include <sys/sendfile.h>
#include <sys/wait.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <linux/tls.h>

#ifndef SOL_TLS
#define SOL_TLS 282
#endif
#ifndef TCP_ULP
#define TCP_ULP 31
#endif

#define PORT     18999
#define FILESZ   (4 * 1024 * 1024)            /* 4 MiB backing file, looped     */
#define TOTAL    (2ULL * 1024 * 1024 * 1024)  /* try to send 2 GiB              */
#define STALL_S  20.0                         /* give up after this many seconds */

static void die(const char *what) { perror(what); exit(1); }

static double now(void)
{
	struct timespec t;
	clock_gettime(CLOCK_MONOTONIC, &t);
	return t.tv_sec + t.tv_nsec / 1e9;
}

static void set_ktls_tx(int fd)
{
	struct tls12_crypto_info_aes_gcm_128 ci;

	if (setsockopt(fd, IPPROTO_TCP, TCP_ULP, "tls", sizeof("tls")))
		die("setsockopt(TCP_ULP, tls)");
	memset(&ci, 0, sizeof(ci));
	ci.info.version     = TLS_1_3_VERSION;          /* matches the 16406 ss line */
	ci.info.cipher_type = TLS_CIPHER_AES_GCM_128;
	memset(ci.iv,      1, sizeof(ci.iv));
	memset(ci.key,     2, sizeof(ci.key));
	memset(ci.salt,    3, sizeof(ci.salt));
	memset(ci.rec_seq, 0, sizeof(ci.rec_seq));
	if (setsockopt(fd, SOL_TLS, TLS_TX, &ci, sizeof(ci)))
		die("setsockopt(SOL_TLS, TLS_TX)");
}

int main(int argc, char **argv)
{
	int mss = (argc > 1) ? atoi(argv[1]) : 0;   /* 0 == leave default; >0 clamps */
	char path[] = "/tmp/ktls_repro_dataXXXXXX";
	struct sockaddr_in a;
	char *buf;
	int ffd, lfd, s, one = 1;
	pid_t pid;
	double t0, el;
	unsigned long long sent = 0;
	off_t off = 0;

	/* A dead/RST'd receiver must surface as EPIPE from sendfile(), not a
	 * silent SIGPIPE kill that would make the reproducer look broken. */
	signal(SIGPIPE, SIG_IGN);

	/* backing file */
	ffd = mkstemp(path);
	if (ffd < 0)
		die("mkstemp");
	buf = malloc(FILESZ);
	if (!buf)
		die("malloc");
	memset(buf, 'x', FILESZ);
	if (write(ffd, buf, FILESZ) != FILESZ)
		die("write");
	free(buf);

	memset(&a, 0, sizeof(a));
	a.sin_family      = AF_INET;
	a.sin_port        = htons(PORT);
	a.sin_addr.s_addr = htonl(INADDR_LOOPBACK);

	lfd = socket(AF_INET, SOCK_STREAM, 0);
	if (lfd < 0)
		die("socket");
	setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
	if (mss > 0)   /* announce a small MSS in the SYN-ACK; inherited by accept() */
		setsockopt(lfd, IPPROTO_TCP, TCP_MAXSEG, &mss, sizeof(mss));
	if (bind(lfd, (void *)&a, sizeof(a)))
		die("bind");
	if (listen(lfd, 1))
		die("listen");

	pid = fork();
	if (pid < 0)
		die("fork");
	if (pid == 0) {                 /* receiver: connect, drain ciphertext, discard */
		int c = socket(AF_INET, SOCK_STREAM, 0);
		char *r = malloc(1 << 20);
		ssize_t n;

		if (c < 0 || !r)
			_exit(1);
		while (connect(c, (void *)&a, sizeof(a)))
			usleep(1000);
		while ((n = read(c, r, 1 << 20)) > 0)
			;
		_exit(0);
	}

	s = accept(lfd, 0, 0);
	if (s < 0)
		die("accept");
	set_ktls_tx(s);

	t0 = now();
	while (sent < TOTAL) {
		ssize_t n;
		size_t want;

		if (off >= FILESZ)
			off = 0;
		want = (size_t)(FILESZ - off);
		if (want > TOTAL - sent)
			want = TOTAL - sent;
		n = sendfile(s, ffd, &off, want);
		if (n < 0) {
			if (errno == EINTR)
				continue;
			perror("sendfile");
			break;
		}
		if (n == 0)
			break;
		sent += n;
		if (now() - t0 > STALL_S) {
			printf("[gave up after %.0fs]\n", STALL_S);
			break;
		}
	}
	el = now() - t0;
	printf("MSS=%-9s  sent=%8.1f MiB in %6.2fs  =>  %.4f GiB/s\n",
	       mss ? argv[1] : "default", sent / 1048576.0, el,
	       sent / el / (1024.0 * 1024 * 1024));

	close(s);
	kill(pid, SIGKILL);
	waitpid(pid, NULL, 0);
	unlink(path);
	return 0;
}

Re: [RFC net] tls: TLS_SW sendfile() stalls at large MSS

[Jiayuan Chen]

On 6/4/26 1:19 AM, Mike Fara wrote:
> Hi,
>
> Software-kTLS (TLS_SW) TX over sendfile()/splice() drops to the TCP
> persist-timer cadence (tens of KB/s, with individual sendfile() calls blocking
> for tens of seconds) when the path MSS is large -- e.g. loopback (MSS 65483) or
> jumbo frames. At a typical 1448-byte MSS it does not occur. Plain TCP
> sendfile() on the same path is unaffected, and kTLS write() (no splice) is
> unaffected, so it is specific to TLS_SW + the splice/sendfile path.
>
> It triggers only on large-MSS paths with software kTLS (no NIC TLS offload), so
> it is a niche path -- but it is a clean, reproducible multi-order-of-magnitude
> cliff, so it seems worth a look. Reproduces on current mainline. CCing David
> Howells as the author of the 2023 sendpage->MSG_SPLICE_PAGES splice_to_socket()
> rework referenced below, and Eric/Paolo as this is as much a TCP-corking
> interaction as a TLS one.
>
> Environment
> -----------
> - net/tls TLS_SW (no NIC offload; ethtool tls-hw-tx-offload: off [fixed]).
> - AES-GCM; gcm(aes) resolves to generic-gcm-vaes-avx512.
>
> Reproducer (no OpenSSL/handshake; TLS_TX programmed with a fixed key, the
> receiver discards ciphertext, like tools/testing/selftests/net/tls.c):
>
>      cc -O2 -Wall -o ktls_sendfile_stall ktls_sendfile_stall.c
>      ./ktls_sendfile_stall          # default loopback MSS (65483)
>      ./ktls_sendfile_stall 1448     # clamp sender MSS via TCP_MAXSEG
>
> Observed (loopback, single box):
>
>      MSS=default    sent=     4.0 MiB in  52.08s  =>  0.0001 GiB/s   (stalled)
>      MSS=1448       sent=  2048.0 MiB in   1.65s  =>  1.2106 GiB/s
>
> i.e. ~four orders of magnitude; at the default MSS a single sendfile() blocks
> for tens of seconds. For contrast, on the same loopback path:
>
>      plain TCP sendfile() (no TLS ULP):           7.87  GiB/s
>      kTLS write() (TLS_SW, no splice, 2 GiB):      1.99  GiB/s


I tested your ktls_sendfile_stall.c under stable 6.6 and upstream, but 
both of them works correctly.

~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448 sent= 2048.0 MiB in 
2.32s => 0.8603 GiB/s ~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448 
sent= 2048.0 MiB in 2.37s => 0.8439 GiB/s ~/code/tmp$ 
./ktls_sendfile_stall MSS=default sent= 2048.0 MiB in 1.64s => 1.2204 
GiB/s :~/code/tmp$ ./ktls_sendfile_stall MSS=default sent= 2048.0 MiB in 
1.70s => 1.1737 GiB/s ~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448 
sent= 2048.0 MiB in 2.33s => 0.8570 GiB/s

~/code/tmp$ ethtool -k lo | grep "tls-hw-tx-offload" tls-hw-tx-offload: 
off [fixed]

Re: [RFC net] tls: TLS_SW sendfile() stalls at large MSS

[Jiayuan Chen]

On 6/4/26 2:53 PM, WindowsForum.com wrote:
> Thanks for testing. The non-reproduction is maybe now the key data 
> point. My reproducer omitted a precondition my hosts happened to meet: 
> a low net.ipv4.tcp_notsent_lowat. To reproduce, add before running:
>
> sysctl -w net.ipv4.tcp_notsent_lowat=16384


I see.

>
> Root cause
> ----------
> The stalling hosts have tcp_notsent_lowat=16384 (local web tuning); 
> the stock default is effectively disabled. A TLS 1.3 record is 16406 
> bytes (TLS_MAX_PAYLOAD_SIZE 16384 + 22), just above that watermark -- 
> so once tls_sw queues a single completed record, notsent (16406) 
> exceeds the lowat, tcp_stream_memory_free() returns false, and tls_sw 
> parks in sk_stream_wait_memory() holding exactly one corked record 
> (the notsent:16406 + persist state from the original dump). With the 
> default lowat, tls_sw keeps queuing, the MSG_MORE cork flushes at each 
> sendfile() boundary, packets_out stays non-zero, and the persist timer 
> never arms -- which is why stock kernels don't show it.
>
> Three conditions must coincide:
>   (a) MSG_MORE forwarded on a completed record -> the sub-MSS record 
> is corked [the bug];
>   (b) tcp_notsent_lowat < one TLS record (16406) -> tls_sw blocks 
> after that one record instead of streaming past it [the trigger I'd 
> omitted];
>   (c) large MSS -> the record is sub-MSS, so the cork engages [the 
> amplifier].
>
> Confirmed by flipping only that knob: on a stalling host, restoring 
> the default lowat -> 2.89 GiB/s; on a healthy host, setting 
> lowat=16384 -> stalls (~0.0001 GiB/s). Everything that merely 
> correlated (kernel build, congestion control/qdisc, wmem/rmem, 
> tcp_mem, tcp_limit_output_bytes, CPU count, AES-GCM impl) was 
> flip-tested and ruled out.
>
> This doesn't change the proposed fix: clearing MSG_MORE for a full 
> record sends it immediately, so the deadlock can't form regardless of 
> tcp_notsent_lowat.


IMO, force-clearing the MSG_MORE flag for each record is not a good idea,

since we want multiple "APPLICATION DATA" frames in one TCP payload.

>
> If you had not submitted your reply I don't think I would have kept 
> testing it - hope this information is useful to the group.
>
>
Maybe we can skip the sk_stream_memory_free check if MSG_MORE is 
present. The lower tcp_sendmsg_locked will check it again.

Re: [RFC net] tls: TLS_SW sendfile() stalls at large MSS

[David Laight]

On Wed, 03 Jun 2026 17:19:22 +0000 (UTC)
Mike Fara <admin@windowsforum.com> wrote:

> Hi,
> 
> Software-kTLS (TLS_SW) TX over sendfile()/splice() drops to the TCP
> persist-timer cadence (tens of KB/s, with individual sendfile() calls blocking
> for tens of seconds) when the path MSS is large -- e.g. loopback (MSS 65483) or
> jumbo frames. At a typical 1448-byte MSS it does not occur. Plain TCP
> sendfile() on the same path is unaffected, and kTLS write() (no splice) is
> unaffected, so it is specific to TLS_SW + the splice/sendfile path.
> 
> It triggers only on large-MSS paths with software kTLS (no NIC TLS offload), so
> it is a niche path -- but it is a clean, reproducible multi-order-of-magnitude
> cliff, so it seems worth a look. Reproduces on current mainline. CCing David
> Howells as the author of the 2023 sendpage->MSG_SPLICE_PAGES splice_to_socket()
> rework referenced below, and Eric/Paolo as this is as much a TCP-corking
> interaction as a TLS one.
> 
> Environment
> -----------
> - net/tls TLS_SW (no NIC offload; ethtool tls-hw-tx-offload: off [fixed]).
> - AES-GCM; gcm(aes) resolves to generic-gcm-vaes-avx512.
> 
> Reproducer (no OpenSSL/handshake; TLS_TX programmed with a fixed key, the
> receiver discards ciphertext, like tools/testing/selftests/net/tls.c):
> 
>     cc -O2 -Wall -o ktls_sendfile_stall ktls_sendfile_stall.c
>     ./ktls_sendfile_stall          # default loopback MSS (65483)
>     ./ktls_sendfile_stall 1448     # clamp sender MSS via TCP_MAXSEG
> 
> Observed (loopback, single box):
> 
>     MSS=default    sent=     4.0 MiB in  52.08s  =>  0.0001 GiB/s   (stalled)
>     MSS=1448       sent=  2048.0 MiB in   1.65s  =>  1.2106 GiB/s
> 
> i.e. ~four orders of magnitude; at the default MSS a single sendfile() blocks
> for tens of seconds. For contrast, on the same loopback path:
> 
>     plain TCP sendfile() (no TLS ULP):           7.87  GiB/s
>     kTLS write() (TLS_SW, no splice, 2 GiB):      1.99  GiB/s
> 
> Analysis
> --------
> During the stall the sending thread is parked here:
> 
>     [<0>] sk_stream_wait_memory+0x256/0x380
>     [<0>] tls_sw_sendmsg+0x1f1/0xc40       /* tls_sw_sendmsg_locked, inlined */
>     [<0>] inet_sendmsg+0x7f/0x90
>     [<0>] sock_sendmsg+0x183/0x1a0
>     [<0>] splice_to_socket+0x3e0/0x5b0
>     [<0>] splice_direct_to_actor+0xf7/0x2c0
>     [<0>] do_splice_direct+0x71/0xd0
>     [<0>] do_sendfile+0x390/0x440
> 
> and ss(8) shows exactly one completed TLS record held, behind a persist timer,
> with the peer window wide open:
> 
>     ESTAB ... timer:(persist,028ms,0) ... notsent:16406 snd_wnd:1114112
>             ... mss:65483 ... tcp-ulp-tls version: 1.3 ... txconf: sw
> 
> notsent:16406 is one TLS1.3 record (TLS_MAX_PAYLOAD_SIZE 16384 + 22). It is a
> *completed* record, yet TCP is corking it.
> 
> The chain appears to be:
> 
>   1. For sendfile, splice_direct_to_actor() sets SPLICE_F_MORE on every
>      iteration and clears it only on the final chunk that fulfils the request,
>      so splice_to_socket() passes MSG_MORE on every send but the last. (This is
>      the intended coalescing behaviour from the 2023 MSG_SPLICE_PAGES rework;
>      naming it as the origin is a hypothesis, not a confirmed bisect.)
>   2. tls_sw_sendmsg_locked() forwards msg->msg_flags (incl. MSG_MORE) into
>      bpf_exec_tx_verdict()/tls_push_record() even for a *full* record, so the
>      completed record reaches tcp_sendmsg_locked() (via tls_push_sg(), which
>      builds msghdr.msg_flags = MSG_SPLICE_PAGES | flags) with MSG_MORE set.
>   3. With a large MSS the 16 KB record is far below MSS, so TCP corks it
>      waiting to fill a segment.

Shouldn't it be using the MSS set by TCP_MAXSEG?
I guess there are also interactions with SO_SNDBUF.
If the MSS is larger than the SO_SNDBUF value don't you need to send the packet
even if it is 'corked' for any reason?
IIRC all the 'cork' options are just a hint for TCP and can be ignored. 

>   4. tls_sw then can't build the next record -- it blocks in
>      sk_stream_wait_memory() for memory the corked record is holding.

Shouldn't it be using SO_SNDBUF limit there?
I'd have thought the memory wouldn't be freed until the ack is received.
I don't see why you shouldn't have lots of short messages in flight.

-- David

>   5. tcp_write_xmit() leaves the corked record unsent with packets_out == 0, so
>      tcp_check_probe_timer() arms the persist (probe-0) timer even though the
>      window is open; each expiry pushes ~one record -> the observed rate. At a
>      1448-byte MSS each 16 KB record already exceeds MSS and is sent, so the
>      cork never engages.
> 
> Candidate fix (illustrative sketch -- NOT a submittable patch, no S-o-b; the
> right shape is your call given the coalescing intent). A full TLS record is a
> natural transmit boundary, so arguably MSG_MORE should not be honoured for it,
> only for a trailing partial record. In tls_sw_sendmsg_locked():
> 
>     if (full_record || eor) {
> +           unsigned int send_flags = msg->msg_flags;
> +
> +           if (full_record)
> +                   send_flags &= ~MSG_MORE;
>             ret = bpf_exec_tx_verdict(msg_pl, sk, full_record,
> -                                     record_type, &copied, msg->msg_flags);
> +                                     record_type, &copied, send_flags);
> 
> Caveats I'm aware of: (a) there are two bpf_exec_tx_verdict() call sites in the
> function passing msg->msg_flags -- the splice path reaches the one at `copied:`
> label (via `goto copied`), so this one-site change covers the repro, but a
> complete fix would clear MSG_MORE at both; (b) this stops coalescing each
> record's trailing partial into the next TSO segment, partly undoing the 2023
> optimisation, so a narrower fix that only flushes when about to block in
> sk_stream_wait_memory() may be preferable. Happy to spin whichever you prefer
> and run it through the tls selftests.
> 
> Thanks,
> Mike Fara
> 
> --- ktls_sendfile_stall.c ---
> // ktls_sendfile_stall.c
> //
> // Minimal, dependency-free reproducer for a software-kTLS (TLS_SW) TX stall:
> // sendfile()/splice() over a kTLS socket collapses to the TCP persist-timer
> // cadence when the path MSS is large (loopback's 65483, or jumbo frames),
> // because splice sets MSG_MORE on every chunk but the last and tls_sw forwards
> // it to TCP even for *completed* records, so TCP corks the sub-MSS record while
> // tls_sw blocks in sk_stream_wait_memory().
> //
> // No handshake / no OpenSSL: TLS_TX is programmed with a fixed key (the receiver
> // discards ciphertext; we only measure the TX path), exactly like the in-tree
> // tls selftest. Clamping the sender MSS to a realistic value makes the stall
> // vanish, isolating the large-MSS amplifier.
> //
> //   cc -O2 -Wall -o ktls_sendfile_stall ktls_sendfile_stall.c
> //   ./ktls_sendfile_stall          # default loopback MSS (65483) -> stalls
> //   ./ktls_sendfile_stall 1448     # clamp sender MSS via TCP_MAXSEG -> normal
> #define _GNU_SOURCE
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <unistd.h>
> #include <errno.h>
> #include <signal.h>
> #include <time.h>
> #include <sys/socket.h>
> #include <sys/sendfile.h>
> #include <sys/wait.h>
> #include <netinet/in.h>
> #include <netinet/tcp.h>
> #include <arpa/inet.h>
> #include <linux/tls.h>
> 
> #ifndef SOL_TLS
> #define SOL_TLS 282
> #endif
> #ifndef TCP_ULP
> #define TCP_ULP 31
> #endif
> 
> #define PORT     18999
> #define FILESZ   (4 * 1024 * 1024)            /* 4 MiB backing file, looped     */
> #define TOTAL    (2ULL * 1024 * 1024 * 1024)  /* try to send 2 GiB              */
> #define STALL_S  20.0                         /* give up after this many seconds */
> 
> static void die(const char *what) { perror(what); exit(1); }
> 
> static double now(void)
> {
> 	struct timespec t;
> 	clock_gettime(CLOCK_MONOTONIC, &t);
> 	return t.tv_sec + t.tv_nsec / 1e9;
> }
> 
> static void set_ktls_tx(int fd)
> {
> 	struct tls12_crypto_info_aes_gcm_128 ci;
> 
> 	if (setsockopt(fd, IPPROTO_TCP, TCP_ULP, "tls", sizeof("tls")))
> 		die("setsockopt(TCP_ULP, tls)");
> 	memset(&ci, 0, sizeof(ci));
> 	ci.info.version     = TLS_1_3_VERSION;          /* matches the 16406 ss line */
> 	ci.info.cipher_type = TLS_CIPHER_AES_GCM_128;
> 	memset(ci.iv,      1, sizeof(ci.iv));
> 	memset(ci.key,     2, sizeof(ci.key));
> 	memset(ci.salt,    3, sizeof(ci.salt));
> 	memset(ci.rec_seq, 0, sizeof(ci.rec_seq));
> 	if (setsockopt(fd, SOL_TLS, TLS_TX, &ci, sizeof(ci)))
> 		die("setsockopt(SOL_TLS, TLS_TX)");
> }
> 
> int main(int argc, char **argv)
> {
> 	int mss = (argc > 1) ? atoi(argv[1]) : 0;   /* 0 == leave default; >0 clamps */
> 	char path[] = "/tmp/ktls_repro_dataXXXXXX";
> 	struct sockaddr_in a;
> 	char *buf;
> 	int ffd, lfd, s, one = 1;
> 	pid_t pid;
> 	double t0, el;
> 	unsigned long long sent = 0;
> 	off_t off = 0;
> 
> 	/* A dead/RST'd receiver must surface as EPIPE from sendfile(), not a
> 	 * silent SIGPIPE kill that would make the reproducer look broken. */
> 	signal(SIGPIPE, SIG_IGN);
> 
> 	/* backing file */
> 	ffd = mkstemp(path);
> 	if (ffd < 0)
> 		die("mkstemp");
> 	buf = malloc(FILESZ);
> 	if (!buf)
> 		die("malloc");
> 	memset(buf, 'x', FILESZ);
> 	if (write(ffd, buf, FILESZ) != FILESZ)
> 		die("write");
> 	free(buf);
> 
> 	memset(&a, 0, sizeof(a));
> 	a.sin_family      = AF_INET;
> 	a.sin_port        = htons(PORT);
> 	a.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
> 
> 	lfd = socket(AF_INET, SOCK_STREAM, 0);
> 	if (lfd < 0)
> 		die("socket");
> 	setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
> 	if (mss > 0)   /* announce a small MSS in the SYN-ACK; inherited by accept() */
> 		setsockopt(lfd, IPPROTO_TCP, TCP_MAXSEG, &mss, sizeof(mss));
> 	if (bind(lfd, (void *)&a, sizeof(a)))
> 		die("bind");
> 	if (listen(lfd, 1))
> 		die("listen");
> 
> 	pid = fork();
> 	if (pid < 0)
> 		die("fork");
> 	if (pid == 0) {                 /* receiver: connect, drain ciphertext, discard */
> 		int c = socket(AF_INET, SOCK_STREAM, 0);
> 		char *r = malloc(1 << 20);
> 		ssize_t n;
> 
> 		if (c < 0 || !r)
> 			_exit(1);
> 		while (connect(c, (void *)&a, sizeof(a)))
> 			usleep(1000);
> 		while ((n = read(c, r, 1 << 20)) > 0)
> 			;
> 		_exit(0);
> 	}
> 
> 	s = accept(lfd, 0, 0);
> 	if (s < 0)
> 		die("accept");
> 	set_ktls_tx(s);
> 
> 	t0 = now();
> 	while (sent < TOTAL) {
> 		ssize_t n;
> 		size_t want;
> 
> 		if (off >= FILESZ)
> 			off = 0;
> 		want = (size_t)(FILESZ - off);
> 		if (want > TOTAL - sent)
> 			want = TOTAL - sent;
> 		n = sendfile(s, ffd, &off, want);
> 		if (n < 0) {
> 			if (errno == EINTR)
> 				continue;
> 			perror("sendfile");
> 			break;
> 		}
> 		if (n == 0)
> 			break;
> 		sent += n;
> 		if (now() - t0 > STALL_S) {
> 			printf("[gave up after %.0fs]\n", STALL_S);
> 			break;
> 		}
> 	}
> 	el = now() - t0;
> 	printf("MSS=%-9s  sent=%8.1f MiB in %6.2fs  =>  %.4f GiB/s\n",
> 	       mss ? argv[1] : "default", sent / 1048576.0, el,
> 	       sent / el / (1024.0 * 1024 * 1024));
> 
> 	close(s);
> 	kill(pid, SIGKILL);
> 	waitpid(pid, NULL, 0);
> 	unlink(path);
> 	return 0;
> }
> 

Re: [RFC net] tls: TLS_SW sendfile() stalls at large MSS

[Eric Dumazet]

On Wed, Jun 3, 2026 at 11:53 PM WindowsForum.com <admin@windowsforum.com> wrote:
>
> Thanks for testing. The non-reproduction is maybe now the key data point. My reproducer omitted a precondition my hosts happened to meet: a low net.ipv4.tcp_notsent_lowat. To reproduce, add before running:
>
> sysctl -w net.ipv4.tcp_notsent_lowat=16384
>
> Root cause
> ----------
> The stalling hosts have tcp_notsent_lowat=16384 (local web tuning); the stock default is effectively disabled. A TLS 1.3 record is 16406 bytes (TLS_MAX_PAYLOAD_SIZE 16384 + 22), just above that watermark -- so once tls_sw queues a single completed record, notsent (16406) exceeds the lowat, tcp_stream_memory_free() returns false, and tls_sw parks in sk_stream_wait_memory() holding exactly one corked record (the notsent:16406 + persist state from the original dump). With the default lowat, tls_sw keeps queuing, the MSG_MORE cork flushes at each sendfile() boundary, packets_out stays non-zero, and the persist timer never arms -- which is why stock kernels don't show it.
>
> Three conditions must coincide:
>   (a) MSG_MORE forwarded on a completed record -> the sub-MSS record is corked [the bug];
>   (b) tcp_notsent_lowat < one TLS record (16406) -> tls_sw blocks after that one record instead of streaming past it [the trigger I'd omitted];
>   (c) large MSS -> the record is sub-MSS, so the cork engages [the amplifier].
>
> Confirmed by flipping only that knob: on a stalling host, restoring the default lowat -> 2.89 GiB/s; on a healthy host, setting lowat=16384 -> stalls (~0.0001 GiB/s). Everything that merely correlated (kernel build, congestion control/qdisc, wmem/rmem, tcp_mem, tcp_limit_output_bytes, CPU count, AES-GCM impl) was flip-tested and ruled out.
>
> This doesn't change the proposed fix: clearing MSG_MORE for a full record sends it immediately, so the deadlock can't form regardless of tcp_notsent_lowat.
>
> If you had not submitted your reply I don't think I would have kept testing it - hope this information is useful to the group.


I see no reason using such a small net.ipv4.tcp_notsent_lowat

Recommended/practical value for this sysctl is 2MB to reach line rate
on modern NICS.

tcp sendmsg() has an skb granularity; an skb packs around 64KB of
payload (this might be bigger if BIG TCP is enabled),
so 16KB is clearely asking for troubles.

>
>
> On Wed, Jun 3, 2026 at 11:12 PM Jiayuan Chen <jiayuan.chen@linux.dev> wrote:
> >
> >
> > On 6/4/26 1:19 AM, Mike Fara wrote:
> > > Hi,
> > >
> > > Software-kTLS (TLS_SW) TX over sendfile()/splice() drops to the TCP
> > > persist-timer cadence (tens of KB/s, with individual sendfile() calls blocking
> > > for tens of seconds) when the path MSS is large -- e.g. loopback (MSS 65483) or
> > > jumbo frames. At a typical 1448-byte MSS it does not occur. Plain TCP
> > > sendfile() on the same path is unaffected, and kTLS write() (no splice) is
> > > unaffected, so it is specific to TLS_SW + the splice/sendfile path.
> > >
> > > It triggers only on large-MSS paths with software kTLS (no NIC TLS offload), so
> > > it is a niche path -- but it is a clean, reproducible multi-order-of-magnitude
> > > cliff, so it seems worth a look. Reproduces on current mainline. CCing David
> > > Howells as the author of the 2023 sendpage->MSG_SPLICE_PAGES splice_to_socket()
> > > rework referenced below, and Eric/Paolo as this is as much a TCP-corking
> > > interaction as a TLS one.
> > >
> > > Environment
> > > -----------
> > > - net/tls TLS_SW (no NIC offload; ethtool tls-hw-tx-offload: off [fixed]).
> > > - AES-GCM; gcm(aes) resolves to generic-gcm-vaes-avx512.
> > >
> > > Reproducer (no OpenSSL/handshake; TLS_TX programmed with a fixed key, the
> > > receiver discards ciphertext, like tools/testing/selftests/net/tls.c):
> > >
> > >      cc -O2 -Wall -o ktls_sendfile_stall ktls_sendfile_stall.c
> > >      ./ktls_sendfile_stall          # default loopback MSS (65483)
> > >      ./ktls_sendfile_stall 1448     # clamp sender MSS via TCP_MAXSEG
> > >
> > > Observed (loopback, single box):
> > >
> > >      MSS=default    sent=     4.0 MiB in  52.08s  =>  0.0001 GiB/s   (stalled)
> > >      MSS=1448       sent=  2048.0 MiB in   1.65s  =>  1.2106 GiB/s
> > >
> > > i.e. ~four orders of magnitude; at the default MSS a single sendfile() blocks
> > > for tens of seconds. For contrast, on the same loopback path:
> > >
> > >      plain TCP sendfile() (no TLS ULP):           7.87  GiB/s
> > >      kTLS write() (TLS_SW, no splice, 2 GiB):      1.99  GiB/s
> >
> >
> > I tested your ktls_sendfile_stall.c under stable 6.6 and upstream, but
> > both of them works correctly.
> >
> > ~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448 sent= 2048.0 MiB in
> > 2.32s => 0.8603 GiB/s ~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448
> > sent= 2048.0 MiB in 2.37s => 0.8439 GiB/s ~/code/tmp$
> > ./ktls_sendfile_stall MSS=default sent= 2048.0 MiB in 1.64s => 1.2204
> > GiB/s :~/code/tmp$ ./ktls_sendfile_stall MSS=default sent= 2048.0 MiB in
> > 1.70s => 1.1737 GiB/s ~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448
> > sent= 2048.0 MiB in 2.33s => 0.8570 GiB/s
> >
> > ~/code/tmp$ ethtool -k lo | grep "tls-hw-tx-offload" tls-hw-tx-offload:
> > off [fixed]
> >