[Patch mptcp-net 0/3] Fix the transmission stall due to backlog

[Patch mptcp-net 0/3] Fix the transmission stall due to backlog

[Gang Yan]

From: Gang Yan <yangang@kylinos.cn>


Hi Matt, Paolo:

During my testing, I discovered two issues in MPTCP related to the 
backlog_list, both of which can lead to transmission stalls. In this
patchset, I have added a test program to reproduce the issues, along with
two workarounds. Additional details and more specific workarounds are also
included in the notes to help clarify the exact scenarios where the
problems occur.

I would appreciate your feedback and hope we can find a better fix for
these issues.

Gang Yan (3):
  mptcp: add backlog_list bug reproducer test
  mptcp: fix receive stalls when 'ack_seq' in backlog_list
  mptcp: fix stall because of data_ready

 net/mptcp/protocol.c                          |   9 +-
 tools/testing/selftests/net/mptcp/Makefile    |   1 +
 .../testing/selftests/net/mptcp/multi_chunk.c | 148 ++++++++++++++++++
 .../selftests/net/mptcp/multi_chunk.sh        |  37 +++++
 4 files changed, 192 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/net/mptcp/multi_chunk.c
 create mode 100755 tools/testing/selftests/net/mptcp/multi_chunk.sh

-- 
2.43.0

[Patch mptcp-net 1/3] mptcp: add backlog_list bug reproducer test

[Gang Yan]

From: Gang Yan <yangang@kylinos.cn>

This patch introduces a test program to reproduce bugs related to the
backlog_list in MPTCP. The test is derived from tls.c in the selftests
suite, but adapted to work without TLS configuration specifically for
MPTCP testing.

The program performs chunked sendfile operations with various payload
sizes to exercise different code paths and trigger backlog_list-related
issues.

It can be run in .virtme-exec-run:
        run_loop run_selftest_one ./multi_chunk.sh

'''
selftest Test: ./multi_chunk.sh
TAP version 13
1..1
[stalls for a while]
^C

'''

Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
Signed-off-by: Gang Yan <yangang@kylinos.cn>
---
 tools/testing/selftests/net/mptcp/Makefile    |   1 +
 .../testing/selftests/net/mptcp/multi_chunk.c | 148 ++++++++++++++++++
 .../selftests/net/mptcp/multi_chunk.sh        |  37 +++++
 3 files changed, 186 insertions(+)
 create mode 100644 tools/testing/selftests/net/mptcp/multi_chunk.c
 create mode 100755 tools/testing/selftests/net/mptcp/multi_chunk.sh

diff --git a/tools/testing/selftests/net/mptcp/Makefile b/tools/testing/selftests/net/mptcp/Makefile
index 22ba0da2adb8..087e6ae6f0b8 100644
--- a/tools/testing/selftests/net/mptcp/Makefile
+++ b/tools/testing/selftests/net/mptcp/Makefile
@@ -24,6 +24,7 @@ TEST_GEN_FILES := \
 	mptcp_diag \
 	mptcp_inq \
 	mptcp_sockopt \
+	multi_chunk \
 	pm_nl_ctl \
 # end of TEST_GEN_FILES
 
diff --git a/tools/testing/selftests/net/mptcp/multi_chunk.c b/tools/testing/selftests/net/mptcp/multi_chunk.c
new file mode 100644
index 000000000000..8c97db58a6db
--- /dev/null
+++ b/tools/testing/selftests/net/mptcp/multi_chunk.c
@@ -0,0 +1,148 @@
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <sys/sendfile.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <errno.h>
+#include <arpa/inet.h>
+#include <sys/wait.h>
+
+#ifndef IPPROTO_MPTCP
+#define IPPROTO_MPTCP 262
+#endif
+
+#define TLS_PAYLOAD_MAX_LEN 16384
+#define TEST_PORT 12345
+
+static void chunked_sendfile(int cfd, int sfd,
+			     size_t chunk_size,
+			     size_t extra_payload_size)
+{
+	char buf[TLS_PAYLOAD_MAX_LEN];
+	uint16_t test_payload_size;
+	size_t recved = 0;
+	size_t sent = 0;
+	int size = 0;
+	int ret;
+	char filename[] = "/tmp/mytemp.XXXXXX";
+	int fd = mkstemp(filename);
+	off_t offset = 0;
+
+	unlink(filename);
+	if (fd <= 0) {
+		perror("tempfile");
+		exit(1);
+	}
+	if (chunk_size < 1) {
+		perror("chunksize");
+		exit(1);
+	}
+
+	test_payload_size = chunk_size + extra_payload_size;
+	if (test_payload_size > TLS_PAYLOAD_MAX_LEN) {
+		perror("payload_size");
+		exit(1);
+	}
+	memset(buf, 1, test_payload_size);
+	size = write(fd, buf, test_payload_size);
+	if (size != test_payload_size) {
+		perror("file write");
+		exit(1);
+	}
+	fsync(fd);
+
+	while (size > 0) {
+		ret = sendfile(sfd, fd, &offset, chunk_size);
+		if (ret <= 0)
+			exit(1);
+		size -= ret;
+		sent += ret;
+	}
+	printf("[client] sent %zu bytes\n", sent);
+
+	recved = recv(cfd, buf, test_payload_size, MSG_WAITALL);
+	printf("[server] receieved %zu bytes\n", recved);
+
+	if (recved != test_payload_size)
+		exit(1);
+
+	close(fd);
+}
+
+int main()
+{
+	int sfd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
+	int cfd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
+	struct sockaddr_in addr = {0};
+	socklen_t addrlen = sizeof(addr);
+
+	printf("==== multi_chunk_sendfile MPTCP test ====\n");
+
+	if (sfd < 0 || cfd < 0) {
+		perror("socket");
+		exit(1);
+	}
+
+	addr.sin_family = AF_INET;
+	addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
+	addr.sin_port = htons(TEST_PORT);
+
+	if (bind(sfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
+		perror("bind");
+		exit(1);
+	}
+
+	if (listen(sfd, 1) < 0) {
+		perror("listen");
+		exit(1);
+	}
+
+	if (getsockname(sfd, (struct sockaddr *)&addr, &addrlen) < 0) {
+		perror("getsockname");
+		exit(1);
+	}
+
+	if (connect(cfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
+		perror("connect");
+		exit(1);
+	}
+
+	int nfd = accept(sfd, NULL, NULL);
+	if (nfd < 0) {
+		perror("accept");
+		exit(1);
+	}
+
+	chunked_sendfile(cfd, nfd, 4096, 4096);
+	chunked_sendfile(cfd, nfd, 4096, 0);
+	chunked_sendfile(cfd, nfd, 4096, 1);
+	chunked_sendfile(cfd, nfd, 4096, 2048);
+	chunked_sendfile(cfd, nfd, 8192, 2048);
+	chunked_sendfile(cfd, nfd, 4096, 8192);
+	chunked_sendfile(cfd, nfd, 8192, 4096);
+	chunked_sendfile(cfd, nfd, 12288, 1024);
+	chunked_sendfile(cfd, nfd, 12288, 2000);
+	chunked_sendfile(cfd, nfd, 15360, 100);
+	chunked_sendfile(cfd, nfd, 15360, 300);
+	chunked_sendfile(cfd, nfd, 1, 4096);
+	chunked_sendfile(cfd, nfd, 2048, 4096);
+	chunked_sendfile(cfd, nfd, 2048, 8192);
+	chunked_sendfile(cfd, nfd, 4096, 8192);
+	chunked_sendfile(cfd, nfd, 1024, 12288);
+	chunked_sendfile(cfd, nfd, 2000, 12288);
+	chunked_sendfile(cfd, nfd, 100, 15360);
+	chunked_sendfile(cfd, nfd, 300, 15360);
+
+	close(cfd);
+	close(nfd);
+	close(sfd);
+
+	printf("==== test ends ====\n");
+	return 0;
+}
diff --git a/tools/testing/selftests/net/mptcp/multi_chunk.sh b/tools/testing/selftests/net/mptcp/multi_chunk.sh
new file mode 100755
index 000000000000..c0352c89087f
--- /dev/null
+++ b/tools/testing/selftests/net/mptcp/multi_chunk.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+. "$(dirname "${0}")/mptcp_lib.sh"
+
+cleanup()
+{
+	if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then
+		kill "$pid" 2>/dev/null
+		wait "$pid" 2>/dev/null
+	fi
+
+	mptcp_lib_ns_exit "${ns1}"
+}
+
+init()
+{
+	mptcp_lib_ns_init ns1
+
+	local i
+	for i in $(seq 1 4); do
+		mptcp_lib_pm_nl_add_endpoint "${ns1}" "127.0.0.1" flags signal port 1000$i
+	done
+
+	mptcp_lib_pm_nl_set_limits "${ns1}" 8 8
+
+	ip netns exec ${ns1} ip mptcp endpoint show
+	ip netns exec ${ns1} ip mptcp limits
+}
+
+init
+trap cleanup EXIT
+
+ip netns exec $ns1 ./multi_chunk &
+
+pid=$!
+wait $pid
-- 
2.43.0

Re: [Patch mptcp-net 1/3] mptcp: add backlog_list bug reproducer test

[Paolo Abeni]

On 2/5/26 7:41 AM, Gang Yan wrote:
> From: Gang Yan <yangang@kylinos.cn>
> 
> This patch introduces a test program to reproduce bugs related to the
> backlog_list in MPTCP. The test is derived from tls.c in the selftests
> suite, but adapted to work without TLS configuration specifically for
> MPTCP testing.
> 
> The program performs chunked sendfile operations with various payload
> sizes to exercise different code paths and trigger backlog_list-related
> issues.
> 
> It can be run in .virtme-exec-run:
>         run_loop run_selftest_one ./multi_chunk.sh
> 
> '''
> selftest Test: ./multi_chunk.sh
> TAP version 13
> 1..1
> [stalls for a while]
> ^C
> 
> '''
> 
> Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
> Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
> Signed-off-by: Gang Yan <yangang@kylinos.cn>
> ---
>  tools/testing/selftests/net/mptcp/Makefile    |   1 +
>  .../testing/selftests/net/mptcp/multi_chunk.c | 148 ++++++++++++++++++
>  .../selftests/net/mptcp/multi_chunk.sh        |  37 +++++
>  3 files changed, 186 insertions(+)
>  create mode 100644 tools/testing/selftests/net/mptcp/multi_chunk.c
>  create mode 100755 tools/testing/selftests/net/mptcp/multi_chunk.sh
> 
> diff --git a/tools/testing/selftests/net/mptcp/Makefile b/tools/testing/selftests/net/mptcp/Makefile
> index 22ba0da2adb8..087e6ae6f0b8 100644
> --- a/tools/testing/selftests/net/mptcp/Makefile
> +++ b/tools/testing/selftests/net/mptcp/Makefile
> @@ -24,6 +24,7 @@ TEST_GEN_FILES := \
>  	mptcp_diag \
>  	mptcp_inq \
>  	mptcp_sockopt \
> +	multi_chunk \
>  	pm_nl_ctl \
>  # end of TEST_GEN_FILES
>  
> diff --git a/tools/testing/selftests/net/mptcp/multi_chunk.c b/tools/testing/selftests/net/mptcp/multi_chunk.c
> new file mode 100644
> index 000000000000..8c97db58a6db
> --- /dev/null
> +++ b/tools/testing/selftests/net/mptcp/multi_chunk.c
> @@ -0,0 +1,148 @@
> +#define _GNU_SOURCE
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <unistd.h>
> +#include <fcntl.h>
> +#include <sys/socket.h>
> +#include <netinet/in.h>
> +#include <sys/sendfile.h>
> +#include <sys/types.h>
> +#include <sys/stat.h>
> +#include <errno.h>
> +#include <arpa/inet.h>
> +#include <sys/wait.h>
> +
> +#ifndef IPPROTO_MPTCP
> +#define IPPROTO_MPTCP 262
> +#endif

Minor nit: I think we can avoid the compiler guards here; older test
program have them for historical reasons.

> +
> +#define TLS_PAYLOAD_MAX_LEN 16384

Possibly this macro should be renamed, i.e. to PAYLOAD_MAX_LEN and
likely the value increased to 64K


> +#define TEST_PORT 12345
> +
> +static void chunked_sendfile(int cfd, int sfd,
> +			     size_t chunk_size,
> +			     size_t extra_payload_size)
> +{
> +	char buf[TLS_PAYLOAD_MAX_LEN];
> +	uint16_t test_payload_size;
> +	size_t recved = 0;
> +	size_t sent = 0;
> +	int size = 0;
> +	int ret;
> +	char filename[] = "/tmp/mytemp.XXXXXX";
> +	int fd = mkstemp(filename);
> +	off_t offset = 0;
> +
> +	unlink(filename);
> +	if (fd <= 0) {
> +		perror("tempfile");
> +		exit(1);

Here and below you could use the error() libcall instead of the
perror/exit combo.

> +	}
> +	if (chunk_size < 1) {
> +		perror("chunksize");
> +		exit(1);
> +	}
> +
> +	test_payload_size = chunk_size + extra_payload_size;
> +	if (test_payload_size > TLS_PAYLOAD_MAX_LEN) {
> +		perror("payload_size");
> +		exit(1);
> +	}
> +	memset(buf, 1, test_payload_size);
> +	size = write(fd, buf, test_payload_size);
> +	if (size != test_payload_size) {
> +		perror("file write");
> +		exit(1);
> +	}
> +	fsync(fd);
> +
> +	while (size > 0) {
> +		ret = sendfile(sfd, fd, &offset, chunk_size);
> +		if (ret <= 0)
> +			exit(1);
> +		size -= ret;
> +		sent += ret;
> +	}
> +	printf("[client] sent %zu bytes\n", sent);
> +
> +	recved = recv(cfd, buf, test_payload_size, MSG_WAITALL);
> +	printf("[server] receieved %zu bytes\n", recved);

Possibly the above 2 print should be off by default and enabled via
command line switch.

> +
> +	if (recved != test_payload_size)
> +		exit(1);
> +
> +	close(fd);
> +}
> +
> +int main()
> +{
> +	int sfd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
> +	int cfd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
> +	struct sockaddr_in addr = {0};
> +	socklen_t addrlen = sizeof(addr);
> +
> +	printf("==== multi_chunk_sendfile MPTCP test ====\n");
> +
> +	if (sfd < 0 || cfd < 0) {
> +		perror("socket");
> +		exit(1);
> +	}
> +
> +	addr.sin_family = AF_INET;
> +	addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
> +	addr.sin_port = htons(TEST_PORT);
> +
> +	if (bind(sfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
> +		perror("bind");
> +		exit(1);
> +	}
> +
> +	if (listen(sfd, 1) < 0) {
> +		perror("listen");
> +		exit(1);
> +	}
> +
> +	if (getsockname(sfd, (struct sockaddr *)&addr, &addrlen) < 0) {
> +		perror("getsockname");
> +		exit(1);
> +	}
> +
> +	if (connect(cfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
> +		perror("connect");
> +		exit(1);
> +	}
> +
> +	int nfd = accept(sfd, NULL, NULL);
> +	if (nfd < 0) {
> +		perror("accept");
> +		exit(1);
> +	}
> +
> +	chunked_sendfile(cfd, nfd, 4096, 4096);
> +	chunked_sendfile(cfd, nfd, 4096, 0);
> +	chunked_sendfile(cfd, nfd, 4096, 1);
> +	chunked_sendfile(cfd, nfd, 4096, 2048);
> +	chunked_sendfile(cfd, nfd, 8192, 2048);
> +	chunked_sendfile(cfd, nfd, 4096, 8192);
> +	chunked_sendfile(cfd, nfd, 8192, 4096);
> +	chunked_sendfile(cfd, nfd, 12288, 1024);
> +	chunked_sendfile(cfd, nfd, 12288, 2000);
> +	chunked_sendfile(cfd, nfd, 15360, 100);
> +	chunked_sendfile(cfd, nfd, 15360, 300);
> +	chunked_sendfile(cfd, nfd, 1, 4096);
> +	chunked_sendfile(cfd, nfd, 2048, 4096);
> +	chunked_sendfile(cfd, nfd, 2048, 8192);
> +	chunked_sendfile(cfd, nfd, 4096, 8192);
> +	chunked_sendfile(cfd, nfd, 1024, 12288);
> +	chunked_sendfile(cfd, nfd, 2000, 12288);
> +	chunked_sendfile(cfd, nfd, 100, 15360);
> +	chunked_sendfile(cfd, nfd, 300, 15360);
> +
> +	close(cfd);
> +	close(nfd);
> +	close(sfd);
> +
> +	printf("==== test ends ====\n");
> +	return 0;
> +}
> diff --git a/tools/testing/selftests/net/mptcp/multi_chunk.sh b/tools/testing/selftests/net/mptcp/multi_chunk.sh
> new file mode 100755
> index 000000000000..c0352c89087f
> --- /dev/null
> +++ b/tools/testing/selftests/net/mptcp/multi_chunk.sh
> @@ -0,0 +1,37 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +
> +. "$(dirname "${0}")/mptcp_lib.sh"
> +
> +cleanup()
> +{
> +	if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then
> +		kill "$pid" 2>/dev/null
> +		wait "$pid" 2>/dev/null
> +	fi
> +
> +	mptcp_lib_ns_exit "${ns1}"
> +}
> +
> +init()
> +{
> +	mptcp_lib_ns_init ns1
> +
> +	local i
> +	for i in $(seq 1 4); do
> +		mptcp_lib_pm_nl_add_endpoint "${ns1}" "127.0.0.1" flags signal port 1000$i
> +	done
> +
> +	mptcp_lib_pm_nl_set_limits "${ns1}" 8 8
> +
> +	ip netns exec ${ns1} ip mptcp endpoint show
> +	ip netns exec ${ns1} ip mptcp limits
> +}
> +
> +init
> +trap cleanup EXIT
> +
> +ip netns exec $ns1 ./multi_chunk &
> +
> +pid=$!
> +wait $pid

I think it would be nice to emit an ok/fail message as other tests

/P

Re: [Patch mptcp-net 1/3] mptcp: add backlog_list bug reproducer test

[gang.yan]

>February 5, 2026 at 5:20 PM, "Paolo Abeni" <pabeni@redhat.com mailto:pabeni@redhat.com?to=%22Paolo%20Abeni%22%20%3Cpabeni%40redhat.com%3E > wrote:
Hi, Paolo:

Thanks for your detailed review comments.

This patch is for temporary testing purposes only to reproduce the
backlog_list-related issues, and it's not meant for merging. In the upcoming
v2 version, I'll incorporate all your suggestions and add the appropriate
"do-not-merge" tag.

Best regards,
Gang

> 
> On 2/5/26 7:41 AM, Gang Yan wrote:
> 
> > 
> > From: Gang Yan <yangang@kylinos.cn>
> >  
> >  This patch introduces a test program to reproduce bugs related to the
> >  backlog_list in MPTCP. The test is derived from tls.c in the selftests
> >  suite, but adapted to work without TLS configuration specifically for
> >  MPTCP testing.
> >  
> >  The program performs chunked sendfile operations with various payload
> >  sizes to exercise different code paths and trigger backlog_list-related
> >  issues.
> >  
> >  It can be run in .virtme-exec-run:
> >  run_loop run_selftest_one ./multi_chunk.sh
> >  
> >  '''
> >  selftest Test: ./multi_chunk.sh
> >  TAP version 13
> >  1..1
> >  [stalls for a while]
> >  ^C
> >  
> >  '''
> >  
> >  Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
> >  Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
> >  Signed-off-by: Gang Yan <yangang@kylinos.cn>
> >  ---
> >  tools/testing/selftests/net/mptcp/Makefile | 1 +
> >  .../testing/selftests/net/mptcp/multi_chunk.c | 148 ++++++++++++++++++
> >  .../selftests/net/mptcp/multi_chunk.sh | 37 +++++
> >  3 files changed, 186 insertions(+)
> >  create mode 100644 tools/testing/selftests/net/mptcp/multi_chunk.c
> >  create mode 100755 tools/testing/selftests/net/mptcp/multi_chunk.sh
> >  
> >  diff --git a/tools/testing/selftests/net/mptcp/Makefile b/tools/testing/selftests/net/mptcp/Makefile
> >  index 22ba0da2adb8..087e6ae6f0b8 100644
> >  --- a/tools/testing/selftests/net/mptcp/Makefile
> >  +++ b/tools/testing/selftests/net/mptcp/Makefile
> >  @@ -24,6 +24,7 @@ TEST_GEN_FILES := \
> >  mptcp_diag \
> >  mptcp_inq \
> >  mptcp_sockopt \
> >  + multi_chunk \
> >  pm_nl_ctl \
> >  # end of TEST_GEN_FILES
> >  
> >  diff --git a/tools/testing/selftests/net/mptcp/multi_chunk.c b/tools/testing/selftests/net/mptcp/multi_chunk.c
> >  new file mode 100644
> >  index 000000000000..8c97db58a6db
> >  --- /dev/null
> >  +++ b/tools/testing/selftests/net/mptcp/multi_chunk.c
> >  @@ -0,0 +1,148 @@
> >  +#define _GNU_SOURCE
> >  +#include <stdio.h>
> >  +#include <stdlib.h>
> >  +#include <string.h>
> >  +#include <unistd.h>
> >  +#include <fcntl.h>
> >  +#include <sys/socket.h>
> >  +#include <netinet/in.h>
> >  +#include <sys/sendfile.h>
> >  +#include <sys/types.h>
> >  +#include <sys/stat.h>
> >  +#include <errno.h>
> >  +#include <arpa/inet.h>
> >  +#include <sys/wait.h>
> >  +
> >  +#ifndef IPPROTO_MPTCP
> >  +#define IPPROTO_MPTCP 262
> >  +#endif
> > 
> Minor nit: I think we can avoid the compiler guards here; older test
> program have them for historical reasons.
> 
> > 
> > +
> >  +#define TLS_PAYLOAD_MAX_LEN 16384
> > 
> Possibly this macro should be renamed, i.e. to PAYLOAD_MAX_LEN and
> likely the value increased to 64K
> 
> > 
> > +#define TEST_PORT 12345
> >  +
> >  +static void chunked_sendfile(int cfd, int sfd,
> >  + size_t chunk_size,
> >  + size_t extra_payload_size)
> >  +{
> >  + char buf[TLS_PAYLOAD_MAX_LEN];
> >  + uint16_t test_payload_size;
> >  + size_t recved = 0;
> >  + size_t sent = 0;
> >  + int size = 0;
> >  + int ret;
> >  + char filename[] = "/tmp/mytemp.XXXXXX";
> >  + int fd = mkstemp(filename);
> >  + off_t offset = 0;
> >  +
> >  + unlink(filename);
> >  + if (fd <= 0) {
> >  + perror("tempfile");
> >  + exit(1);
> > 
> Here and below you could use the error() libcall instead of the
> perror/exit combo.
> 
> > 
> > + }
> >  + if (chunk_size < 1) {
> >  + perror("chunksize");
> >  + exit(1);
> >  + }
> >  +
> >  + test_payload_size = chunk_size + extra_payload_size;
> >  + if (test_payload_size > TLS_PAYLOAD_MAX_LEN) {
> >  + perror("payload_size");
> >  + exit(1);
> >  + }
> >  + memset(buf, 1, test_payload_size);
> >  + size = write(fd, buf, test_payload_size);
> >  + if (size != test_payload_size) {
> >  + perror("file write");
> >  + exit(1);
> >  + }
> >  + fsync(fd);
> >  +
> >  + while (size > 0) {
> >  + ret = sendfile(sfd, fd, &offset, chunk_size);
> >  + if (ret <= 0)
> >  + exit(1);
> >  + size -= ret;
> >  + sent += ret;
> >  + }
> >  + printf("[client] sent %zu bytes\n", sent);
> >  +
> >  + recved = recv(cfd, buf, test_payload_size, MSG_WAITALL);
> >  + printf("[server] receieved %zu bytes\n", recved);
> > 
> Possibly the above 2 print should be off by default and enabled via
> command line switch.
> 
> > 
> > +
> >  + if (recved != test_payload_size)
> >  + exit(1);
> >  +
> >  + close(fd);
> >  +}
> >  +
> >  +int main()
> >  +{
> >  + int sfd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
> >  + int cfd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
> >  + struct sockaddr_in addr = {0};
> >  + socklen_t addrlen = sizeof(addr);
> >  +
> >  + printf("==== multi_chunk_sendfile MPTCP test ====\n");
> >  +
> >  + if (sfd < 0 || cfd < 0) {
> >  + perror("socket");
> >  + exit(1);
> >  + }
> >  +
> >  + addr.sin_family = AF_INET;
> >  + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
> >  + addr.sin_port = htons(TEST_PORT);
> >  +
> >  + if (bind(sfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
> >  + perror("bind");
> >  + exit(1);
> >  + }
> >  +
> >  + if (listen(sfd, 1) < 0) {
> >  + perror("listen");
> >  + exit(1);
> >  + }
> >  +
> >  + if (getsockname(sfd, (struct sockaddr *)&addr, &addrlen) < 0) {
> >  + perror("getsockname");
> >  + exit(1);
> >  + }
> >  +
> >  + if (connect(cfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
> >  + perror("connect");
> >  + exit(1);
> >  + }
> >  +
> >  + int nfd = accept(sfd, NULL, NULL);
> >  + if (nfd < 0) {
> >  + perror("accept");
> >  + exit(1);
> >  + }
> >  +
> >  + chunked_sendfile(cfd, nfd, 4096, 4096);
> >  + chunked_sendfile(cfd, nfd, 4096, 0);
> >  + chunked_sendfile(cfd, nfd, 4096, 1);
> >  + chunked_sendfile(cfd, nfd, 4096, 2048);
> >  + chunked_sendfile(cfd, nfd, 8192, 2048);
> >  + chunked_sendfile(cfd, nfd, 4096, 8192);
> >  + chunked_sendfile(cfd, nfd, 8192, 4096);
> >  + chunked_sendfile(cfd, nfd, 12288, 1024);
> >  + chunked_sendfile(cfd, nfd, 12288, 2000);
> >  + chunked_sendfile(cfd, nfd, 15360, 100);
> >  + chunked_sendfile(cfd, nfd, 15360, 300);
> >  + chunked_sendfile(cfd, nfd, 1, 4096);
> >  + chunked_sendfile(cfd, nfd, 2048, 4096);
> >  + chunked_sendfile(cfd, nfd, 2048, 8192);
> >  + chunked_sendfile(cfd, nfd, 4096, 8192);
> >  + chunked_sendfile(cfd, nfd, 1024, 12288);
> >  + chunked_sendfile(cfd, nfd, 2000, 12288);
> >  + chunked_sendfile(cfd, nfd, 100, 15360);
> >  + chunked_sendfile(cfd, nfd, 300, 15360);
> >  +
> >  + close(cfd);
> >  + close(nfd);
> >  + close(sfd);
> >  +
> >  + printf("==== test ends ====\n");
> >  + return 0;
> >  +}
> >  diff --git a/tools/testing/selftests/net/mptcp/multi_chunk.sh b/tools/testing/selftests/net/mptcp/multi_chunk.sh
> >  new file mode 100755
> >  index 000000000000..c0352c89087f
> >  --- /dev/null
> >  +++ b/tools/testing/selftests/net/mptcp/multi_chunk.sh
> >  @@ -0,0 +1,37 @@
> >  +#!/bin/bash
> >  +# SPDX-License-Identifier: GPL-2.0
> >  +
> >  +. "$(dirname "${0}")/mptcp_lib.sh"
> >  +
> >  +cleanup()
> >  +{
> >  + if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then
> >  + kill "$pid" 2>/dev/null
> >  + wait "$pid" 2>/dev/null
> >  + fi
> >  +
> >  + mptcp_lib_ns_exit "${ns1}"
> >  +}
> >  +
> >  +init()
> >  +{
> >  + mptcp_lib_ns_init ns1
> >  +
> >  + local i
> >  + for i in $(seq 1 4); do
> >  + mptcp_lib_pm_nl_add_endpoint "${ns1}" "127.0.0.1" flags signal port 1000$i
> >  + done
> >  +
> >  + mptcp_lib_pm_nl_set_limits "${ns1}" 8 8
> >  +
> >  + ip netns exec ${ns1} ip mptcp endpoint show
> >  + ip netns exec ${ns1} ip mptcp limits
> >  +}
> >  +
> >  +init
> >  +trap cleanup EXIT
> >  +
> >  +ip netns exec $ns1 ./multi_chunk &
> >  +
> >  +pid=$!
> >  +wait $pid
> > 
> I think it would be nice to emit an ok/fail message as other tests
> 
> /P
>

Re: [Patch mptcp-net 1/3] mptcp: add backlog_list bug reproducer test

[Matthieu Baerts]

Hi Gang,

Thank you for looking at this.

Small detail: I'm not sure what you are using to generate the patches
(b4 prep/send is great BTW), but it is strange to have 'Patch' instead
of PATCH in the subjects.

On 05/02/2026 07:41, Gang Yan wrote:
> From: Gang Yan <yangang@kylinos.cn>
> 
> This patch introduces a test program to reproduce bugs related to the
> backlog_list in MPTCP. The test is derived from tls.c in the selftests
> suite, but adapted to work without TLS configuration specifically for
> MPTCP testing.
> 
> The program performs chunked sendfile operations with various payload
> sizes to exercise different code paths and trigger backlog_list-related
> issues.

I understood it was just a tmp patch just to show how to reproduce the
issue, but I think it is good to make sure such regression is not
reintroduced. So if this test is quick to run, it should be kept. No?

Except if the error could be reproduced with packetdrill? Or eventually
if mptcp_connect.(c|sh) could be easily adapted to reproduce the issue
instead?

> 
> It can be run in .virtme-exec-run:
>         run_loop run_selftest_one ./multi_chunk.sh

(That's specific to the MPTCP docker image, no need to specify this here
to avoid confusions)

> 
> '''
> selftest Test: ./multi_chunk.sh
> TAP version 13
> 1..1
> [stalls for a while]
> ^C
> 
> '''
> 
> Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
> Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
> Signed-off-by: Gang Yan <yangang@kylinos.cn>
> ---
>  tools/testing/selftests/net/mptcp/Makefile    |   1 +
>  .../testing/selftests/net/mptcp/multi_chunk.c | 148 ++++++++++++++++++
>  .../selftests/net/mptcp/multi_chunk.sh        |  37 +++++
>  3 files changed, 186 insertions(+)
>  create mode 100644 tools/testing/selftests/net/mptcp/multi_chunk.c
>  create mode 100755 tools/testing/selftests/net/mptcp/multi_chunk.sh
> 
> diff --git a/tools/testing/selftests/net/mptcp/Makefile b/tools/testing/selftests/net/mptcp/Makefile
> index 22ba0da2adb8..087e6ae6f0b8 100644
> --- a/tools/testing/selftests/net/mptcp/Makefile
> +++ b/tools/testing/selftests/net/mptcp/Makefile
> @@ -24,6 +24,7 @@ TEST_GEN_FILES := \
>  	mptcp_diag \
>  	mptcp_inq \
>  	mptcp_sockopt \
> +	multi_chunk \
>  	pm_nl_ctl \
>  # end of TEST_GEN_FILES
>  
> diff --git a/tools/testing/selftests/net/mptcp/multi_chunk.c b/tools/testing/selftests/net/mptcp/multi_chunk.c
> new file mode 100644
> index 000000000000..8c97db58a6db
> --- /dev/null
> +++ b/tools/testing/selftests/net/mptcp/multi_chunk.c
> @@ -0,0 +1,148 @@
> +#define _GNU_SOURCE
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <unistd.h>
> +#include <fcntl.h>
> +#include <sys/socket.h>
> +#include <netinet/in.h>
> +#include <sys/sendfile.h>
> +#include <sys/types.h>
> +#include <sys/stat.h>
> +#include <errno.h>
> +#include <arpa/inet.h>
> +#include <sys/wait.h>

Please sort the headers, see:

https://lore.kernel.org/mptcp/20260130-mptcp-sft-sort-includes-v1-1-ab022fad9741@kernel.org/

> +
> +#ifndef IPPROTO_MPTCP
> +#define IPPROTO_MPTCP 262
> +#endif
> +
> +#define TLS_PAYLOAD_MAX_LEN 16384
> +#define TEST_PORT 12345
> +
> +static void chunked_sendfile(int cfd, int sfd,
> +			     size_t chunk_size,
> +			     size_t extra_payload_size)
> +{
> +	char buf[TLS_PAYLOAD_MAX_LEN];
> +	uint16_t test_payload_size;
> +	size_t recved = 0;
> +	size_t sent = 0;
> +	int size = 0;
> +	int ret;
> +	char filename[] = "/tmp/mytemp.XXXXXX";
> +	int fd = mkstemp(filename);
> +	off_t offset = 0;

Ideally keep the reverse XMAS tree for the order.

> +
> +	unlink(filename);
> +	if (fd <= 0) {
> +		perror("tempfile");
> +		exit(1);
> +	}
> +	if (chunk_size < 1) {
> +		perror("chunksize");
> +		exit(1);
> +	}
> +
> +	test_payload_size = chunk_size + extra_payload_size;
> +	if (test_payload_size > TLS_PAYLOAD_MAX_LEN) {
> +		perror("payload_size");
> +		exit(1);
> +	}
> +	memset(buf, 1, test_payload_size);
> +	size = write(fd, buf, test_payload_size);
> +	if (size != test_payload_size) {
> +		perror("file write");
> +		exit(1);
> +	}
> +	fsync(fd);
> +
> +	while (size > 0) {
> +		ret = sendfile(sfd, fd, &offset, chunk_size);
> +		if (ret <= 0)

It would be good to print an error to stderr.

Also, maybe you will need to at least close(fd) to be sure the tmp file
will be removed, no?


> +			exit(1);
> +		size -= ret;
> +		sent += ret;
> +	}
> +	printf("[client] sent %zu bytes\n", sent);
> +
> +	recved = recv(cfd, buf, test_payload_size, MSG_WAITALL);
> +	printf("[server] receieved %zu bytes\n", recved);
> +
> +	if (recved != test_payload_size)

Same here.

> +		exit(1);
> +
> +	close(fd);
> +}
> +
> +int main()
> +{
> +	int sfd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
> +	int cfd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
> +	struct sockaddr_in addr = {0};
> +	socklen_t addrlen = sizeof(addr);
> +
> +	printf("==== multi_chunk_sendfile MPTCP test ====\n");

Maybe these printf() here and at the bottom are not really needed?

> +
> +	if (sfd < 0 || cfd < 0) {
> +		perror("socket");
> +		exit(1);
> +	}
> +
> +	addr.sin_family = AF_INET;
> +	addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
> +	addr.sin_port = htons(TEST_PORT);
> +
> +	if (bind(sfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
> +		perror("bind");
> +		exit(1);
> +	}
> +
> +	if (listen(sfd, 1) < 0) {
> +		perror("listen");
> +		exit(1);
> +	}
> +
> +	if (getsockname(sfd, (struct sockaddr *)&addr, &addrlen) < 0) {
> +		perror("getsockname");
> +		exit(1);
> +	}
> +
> +	if (connect(cfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
> +		perror("connect");
> +		exit(1);
> +	}
> +
> +	int nfd = accept(sfd, NULL, NULL);
> +	if (nfd < 0) {
> +		perror("accept");
> +		exit(1);
> +	}
> +
> +	chunked_sendfile(cfd, nfd, 4096, 4096);
> +	chunked_sendfile(cfd, nfd, 4096, 0);
> +	chunked_sendfile(cfd, nfd, 4096, 1);
> +	chunked_sendfile(cfd, nfd, 4096, 2048);
> +	chunked_sendfile(cfd, nfd, 8192, 2048);
> +	chunked_sendfile(cfd, nfd, 4096, 8192);
> +	chunked_sendfile(cfd, nfd, 8192, 4096);
> +	chunked_sendfile(cfd, nfd, 12288, 1024);
> +	chunked_sendfile(cfd, nfd, 12288, 2000);
> +	chunked_sendfile(cfd, nfd, 15360, 100);
> +	chunked_sendfile(cfd, nfd, 15360, 300);
> +	chunked_sendfile(cfd, nfd, 1, 4096);
> +	chunked_sendfile(cfd, nfd, 2048, 4096);
> +	chunked_sendfile(cfd, nfd, 2048, 8192);
> +	chunked_sendfile(cfd, nfd, 4096, 8192);
> +	chunked_sendfile(cfd, nfd, 1024, 12288);
> +	chunked_sendfile(cfd, nfd, 2000, 12288);
> +	chunked_sendfile(cfd, nfd, 100, 15360);
> +	chunked_sendfile(cfd, nfd, 300, 15360);
> +
> +	close(cfd);
> +	close(nfd);
> +	close(sfd);
> +
> +	printf("==== test ends ====\n");
> +	return 0;
> +}
> diff --git a/tools/testing/selftests/net/mptcp/multi_chunk.sh b/tools/testing/selftests/net/mptcp/multi_chunk.sh
> new file mode 100755
> index 000000000000..c0352c89087f
> --- /dev/null
> +++ b/tools/testing/selftests/net/mptcp/multi_chunk.sh
> @@ -0,0 +1,37 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +
> +. "$(dirname "${0}")/mptcp_lib.sh"
> +
> +cleanup()
> +{
> +	if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then
> +		kill "$pid" 2>/dev/null
> +		wait "$pid" 2>/dev/null
> +	fi
> +
> +	mptcp_lib_ns_exit "${ns1}"
> +}
> +
> +init()
> +{
> +	mptcp_lib_ns_init ns1
> +
> +	local i
> +	for i in $(seq 1 4); do
> +		mptcp_lib_pm_nl_add_endpoint "${ns1}" "127.0.0.1" flags signal port 1000$i
> +	done
> +
> +	mptcp_lib_pm_nl_set_limits "${ns1}" 8 8
> +
> +	ip netns exec ${ns1} ip mptcp endpoint show
> +	ip netns exec ${ns1} ip mptcp limits

These two lines are not needed.

> +}
> +
> +init
> +trap cleanup EXIT
> +
> +ip netns exec $ns1 ./multi_chunk &
> +
> +pid=$!
> +wait $pid

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

[Patch mptcp-net 2/3] mptcp: fix receive stalls when 'ack_seq' in backlog_list

[Gang Yan]

From: Gang Yan <yangang@kylinos.cn>

This patch fixes the first backlog_list issue, which can be 100%
reproduced by the multi_chunk.sh test case.

The problem occurs when out-of-order (OFO) queue data accumulates,
causing sk_rmem_alloc to exceed sk_rcvbuf. In this condition, skbs
in backlog_list fail to move to sk->receive_queue even when the
receive queue is empty, leading to transmission stalls.

This patch adds an empty check for sk->receieve_queue, when it is empty,
the skb should moved too.

Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
Signed-off-by: Gang Yan <yangang@kylinos.cn>
---
 net/mptcp/protocol.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
index ff06bbee7781..b6bafc37eea4 100644
--- a/net/mptcp/protocol.c
+++ b/net/mptcp/protocol.c
@@ -2176,7 +2176,8 @@ static bool __mptcp_move_skbs(struct sock *sk, struct list_head *skbs, u32 *delt
 	*delta = 0;
 	while (1) {
 		/* If the msk recvbuf is full stop, don't drop */
-		if (sk_rmem_alloc_get(sk) > sk->sk_rcvbuf)
+		if (sk_rmem_alloc_get(sk) > sk->sk_rcvbuf &&
+		    !skb_queue_empty(&sk->sk_receive_queue))
 			break;
 
 		prefetch(skb->next);
@@ -2208,7 +2209,8 @@ static bool mptcp_can_spool_backlog(struct sock *sk, struct list_head *skbs)
 
 	/* Don't spool the backlog if the rcvbuf is full. */
 	if (list_empty(&msk->backlog_list) ||
-	    sk_rmem_alloc_get(sk) > sk->sk_rcvbuf)
+	    (sk_rmem_alloc_get(sk) > sk->sk_rcvbuf &&
+	     !skb_queue_empty(&sk->sk_receive_queue)))
 		return false;
 
 	INIT_LIST_HEAD(skbs);
-- 
2.43.0

Re: [Patch mptcp-net 2/3] mptcp: fix receive stalls when 'ack_seq' in backlog_list

[Paolo Abeni]

On 2/5/26 7:41 AM, Gang Yan wrote:
> From: Gang Yan <yangang@kylinos.cn>
> 
> This patch fixes the first backlog_list issue, which can be 100%
> reproduced by the multi_chunk.sh test case.
> 
> The problem occurs when out-of-order (OFO) queue data accumulates,
> causing sk_rmem_alloc to exceed sk_rcvbuf. In this condition, skbs
> in backlog_list fail to move to sk->receive_queue even when the
> receive queue is empty, leading to transmission stalls.

Very good catch! How did you hit the issue?

> 
> This patch adds an empty check for sk->receieve_queue, when it is empty,
> the skb should moved too.
> 
> Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
> Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
> Signed-off-by: Gang Yan <yangang@kylinos.cn>
> ---
>  net/mptcp/protocol.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
> index ff06bbee7781..b6bafc37eea4 100644
> --- a/net/mptcp/protocol.c
> +++ b/net/mptcp/protocol.c
> @@ -2176,7 +2176,8 @@ static bool __mptcp_move_skbs(struct sock *sk, struct list_head *skbs, u32 *delt
>  	*delta = 0;
>  	while (1) {
>  		/* If the msk recvbuf is full stop, don't drop */
> -		if (sk_rmem_alloc_get(sk) > sk->sk_rcvbuf)
> +		if (sk_rmem_alloc_get(sk) > sk->sk_rcvbuf &&
> +		    !skb_queue_empty(&sk->sk_receive_queue))

I think we need to be more conservative, otherwise this could be
exploited to allow unbound increase of the rcvbuf.

Unfortunately it does not look easy, but I think it would be doable:

- convert the backlog to a rbtree, sorted by MPTCP-level sequence
number; that would likely require:
  - factoring out a slightly more generic helper out of mptcp_data_queue_ofo
  - use it for both mptcp_data_queue_ofo() and enqueue to backlog
- implement an helper to check if we can dequeue from the backlog; it
should check that we are below memory bounds or that the first skb in
the backlog is in-sequence.
- use such helper here and in mptcp_can_spool_backlog()

I see it's not easy, but I think is necessary. Simpler alternatives more
than welcome!

(side note, on top of my head I think that even plain TCP stumbled upon
a similar situation, i.e. could overrun some limits for in-sequence skbs).

/P

Re: [Patch mptcp-net 2/3] mptcp: fix receive stalls when 'ack_seq' in backlog_list

[gang.yan]

February 5, 2026 at 9:14 PM, gang.yan@linux.dev mailto:gang.yan@linux.dev  wrote:


> 
> > 
> > February 5, 2026 at 5:36 PM, "Paolo Abeni" <pabeni@redhat.com mailto:pabeni@redhat.com?to=%22Paolo%20Abeni%22%20%3Cpabeni%40redhat.com%3E > wrote:
> > 
> Hi, Paolo:
> 
> > 
> > On 2/5/26 7:41 AM, Gang Yan wrote:
> >  
> >  
> >  From: Gang Yan <yangang@kylinos.cn>
> >  
> >  This patch fixes the first backlog_list issue, which can be 100%
> >  reproduced by the multi_chunk.sh test case.
> >  
> >  The problem occurs when out-of-order (OFO) queue data accumulates,
> >  causing sk_rmem_alloc to exceed sk_rcvbuf. In this condition, skbs
> >  in backlog_list fail to move to sk->receive_queue even when the
> >  receive queue is empty, leading to transmission stalls.
> >  
> >  Very good catch! How did you hit the issue?
> > 
> This issue was exactly exposed during the development of NVMe over MPTCP
> and MPTCP KTLS support that Geliang and I have been working on recently.
> So, we created the 'multi_chunk.sh' to reproduce it. 
> 
> > 
> > This patch adds an empty check for sk->receieve_queue, when it is empty,
> >  the skb should moved too.
> >  
> >  Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
> >  Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
> >  Signed-off-by: Gang Yan <yangang@kylinos.cn>
> >  ---
> >  net/mptcp/protocol.c | 6 ++++--
> >  1 file changed, 4 insertions(+), 2 deletions(-)
> >  
> >  diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
> >  index ff06bbee7781..b6bafc37eea4 100644
> >  --- a/net/mptcp/protocol.c
> >  +++ b/net/mptcp/protocol.c
> >  @@ -2176,7 +2176,8 @@ static bool __mptcp_move_skbs(struct sock *sk, struct list_head *skbs, u32 *delt
> >  *delta = 0;
> >  while (1) {
> >  /* If the msk recvbuf is full stop, don't drop */
> >  - if (sk_rmem_alloc_get(sk) > sk->sk_rcvbuf)
> >  + if (sk_rmem_alloc_get(sk) > sk->sk_rcvbuf &&
> >  + !skb_queue_empty(&sk->sk_receive_queue))
> >  
> >  I think we need to be more conservative, otherwise this could be
> >  exploited to allow unbound increase of the rcvbuf.
> >  
> >  Unfortunately it does not look easy, but I think it would be doable:
> >  
> >  - convert the backlog to a rbtree, sorted by MPTCP-level sequence
> >  number; that would likely require:
> >  - factoring out a slightly more generic helper out of mptcp_data_queue_ofo
> >  - use it for both mptcp_data_queue_ofo() and enqueue to backlog
> >  - implement an helper to check if we can dequeue from the backlog; it
> >  should check that we are below memory bounds or that the first skb in
> >  the backlog is in-sequence.
> >  - use such helper here and in mptcp_can_spool_backlog()
> >  
> >  I see it's not easy, but I think is necessary. Simpler alternatives more
> >  than welcome!
> > 
> I fully agree with your concern. And your proposed solution is much more robust
> and aligns with the long-term stability of the MPTCP stack.
> 
> I will follow your guidance to implement these changes step by step and submit
> a v2 version of the patch as soon as possible.
> 
> Thanks again for your insightful feedback!
> 
> Best regards,
> Gang
> 
Hi,

I’ve been working on this gradually, but progress is slower than expected.
Switching backlog_list to an rbtree appears to introduce more race conditions,
and I’m also dealing with KASAN errors and warnings around ssk->rmem_alloc.

I remain committed to this development and will continue working on it. Once
meaningful progress is made, I will release v2 as soon as possible.

Thanks
Gang
> > 
> > (side note, on top of my head I think that even plain TCP stumbled upon
> >  a similar situation, i.e. could overrun some limits for in-sequence skbs).
> >  
> >  /P
> >
>

[Patch mptcp-net 3/3] mptcp: fix stall because of data_ready

[Gang Yan]

From: Gang Yan <yangang@kylinos.cn>

This patch fixes the second type of backlog_list stall issue that
occurs when the data_ready callback attempts to trigger data transfer.
The issue reproduces at approximately a 20% rate (once every five runs)
when running multi_chunk.sh tests.

The stall occurs under the following conditions:
1. A large amount of out-of-order (OFO) data causes sk_rmem_alloc to
   exceed sk_rcvbuf
2. The skb matching the current ack_seq is not present in backlog_list
3. Data reception relies on data_ready callback notification

In this scenario, the data_ready callback (via mptcp_data_ready() ->
sk->sk_data_ready()) attempts to trigger data movement, but
__mptcp_move_skbs_from_subflow() repeatedly moves the ack_seq skb into
backlog_list and returns false:

'''
[  144.961282][    C0] MPTCP: msk->ack_seq:3442119990924456661, map_seq:3442119990924456661, offset:0, fin:0
[  144.961293][    C0] MPTCP: [MPTCP_BACKLOG] #0 map_seq=3442119990924655746 end_seq=3442119990924660542 len=4796
[  144.962310][    C0] MPTCP: [MPTCP_BACKLOG] #1 map_seq=3442119990924491850 end_seq=3442119990924500364 len=8514
[  144.962783][    C0] MPTCP: [MPTCP_BACKLOG] #2 map_seq=3442119990924660542 end_seq=3442119990924726001 len=65459
[  144.963260][    C0] MPTCP: [MPTCP_BACKLOG] #3 map_seq=3442119990924508114 end_seq=3442119990924514971 len=6857
[  144.963729][    C0] MPTCP: [MPTCP_BACKLOG] #4 map_seq=3442119990924726001 end_seq=3442119990924731093 len=5092
[  144.964193][    C0] MPTCP: [MPTCP_BACKLOG] #5 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
[  144.964651][    C0] MPTCP: [MPTCP_BACKLOG] #6 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
[  144.965164][    C0] MPTCP: [MPTCP_BACKLOG] #7 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
[  144.965711][    C0] MPTCP: [MPTCP_BACKLOG] #8 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
[  144.966260][    C0] MPTCP: [MPTCP_BACKLOG] #9 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
[  144.966804][    C0] MPTCP: [MPTCP_BACKLOG] #10 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
[  144.967308][    C0] MPTCP: [MPTCP_BACKLOG] #11 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
[  144.967793][    C0] MPTCP: [MPTCP_BACKLOG] #12 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
...
'''

The fix adds a check for an empty receive queue in addition to the
rcvbuf comparison. When the receive queue is empty, skbs should still
be moved to prevent the stall. With this patch, all mptcp_tls tests
pass successfully.

Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
Signed-off-by: Gang Yan <yangang@kylinos.cn>
---
 net/mptcp/protocol.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
index b6bafc37eea4..054aa72c9aa6 100644
--- a/net/mptcp/protocol.c
+++ b/net/mptcp/protocol.c
@@ -739,7 +739,8 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
 
 			mptcp_init_skb(ssk, skb, offset, len);
 
-			if (own_msk && sk_rmem_alloc_get(sk) < sk->sk_rcvbuf) {
+			if (own_msk && (sk_rmem_alloc_get(sk) < sk->sk_rcvbuf ||
+					skb_queue_empty(&sk->sk_receive_queue))) {
 				mptcp_subflow_lend_fwdmem(subflow, skb);
 				ret |= __mptcp_move_skb(sk, skb);
 			} else {
-- 
2.43.0

Re: [Patch mptcp-net 3/3] mptcp: fix stall because of data_ready

[Paolo Abeni]

On 2/5/26 7:41 AM, Gang Yan wrote:
> From: Gang Yan <yangang@kylinos.cn>
> 
> This patch fixes the second type of backlog_list stall issue that
> occurs when the data_ready callback attempts to trigger data transfer.
> The issue reproduces at approximately a 20% rate (once every five runs)
> when running multi_chunk.sh tests.
> 
> The stall occurs under the following conditions:
> 1. A large amount of out-of-order (OFO) data causes sk_rmem_alloc to
>    exceed sk_rcvbuf

Thinking again about this scenario, the above condition is quite
unexpected to me. Could you please share a pcap capture of this stall?
In theory the sender should not fill the rcv window with OoO skb only.

> 2. The skb matching the current ack_seq is not present in backlog_list
> 3. Data reception relies on data_ready callback notification
> 
> In this scenario, the data_ready callback (via mptcp_data_ready() ->
> sk->sk_data_ready()) attempts to trigger data movement, but
> __mptcp_move_skbs_from_subflow() repeatedly moves the ack_seq skb into
> backlog_list and returns false:
> 
> '''
> [  144.961282][    C0] MPTCP: msk->ack_seq:3442119990924456661, map_seq:3442119990924456661, offset:0, fin:0
> [  144.961293][    C0] MPTCP: [MPTCP_BACKLOG] #0 map_seq=3442119990924655746 end_seq=3442119990924660542 len=4796
> [  144.962310][    C0] MPTCP: [MPTCP_BACKLOG] #1 map_seq=3442119990924491850 end_seq=3442119990924500364 len=8514
> [  144.962783][    C0] MPTCP: [MPTCP_BACKLOG] #2 map_seq=3442119990924660542 end_seq=3442119990924726001 len=65459
> [  144.963260][    C0] MPTCP: [MPTCP_BACKLOG] #3 map_seq=3442119990924508114 end_seq=3442119990924514971 len=6857
> [  144.963729][    C0] MPTCP: [MPTCP_BACKLOG] #4 map_seq=3442119990924726001 end_seq=3442119990924731093 len=5092
> [  144.964193][    C0] MPTCP: [MPTCP_BACKLOG] #5 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> [  144.964651][    C0] MPTCP: [MPTCP_BACKLOG] #6 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> [  144.965164][    C0] MPTCP: [MPTCP_BACKLOG] #7 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> [  144.965711][    C0] MPTCP: [MPTCP_BACKLOG] #8 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> [  144.966260][    C0] MPTCP: [MPTCP_BACKLOG] #9 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> [  144.966804][    C0] MPTCP: [MPTCP_BACKLOG] #10 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> [  144.967308][    C0] MPTCP: [MPTCP_BACKLOG] #11 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> [  144.967793][    C0] MPTCP: [MPTCP_BACKLOG] #12 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> ...
> '''
> 
> The fix adds a check for an empty receive queue in addition to the
> rcvbuf comparison. When the receive queue is empty, skbs should still
> be moved to prevent the stall. With this patch, all mptcp_tls tests
> pass successfully.
> 
> Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
> Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
> Signed-off-by: Gang Yan <yangang@kylinos.cn>
> ---
>  net/mptcp/protocol.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
> index b6bafc37eea4..054aa72c9aa6 100644
> --- a/net/mptcp/protocol.c
> +++ b/net/mptcp/protocol.c
> @@ -739,7 +739,8 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
>  
>  			mptcp_init_skb(ssk, skb, offset, len);
>  
> -			if (own_msk && sk_rmem_alloc_get(sk) < sk->sk_rcvbuf) {
> +			if (own_msk && (sk_rmem_alloc_get(sk) < sk->sk_rcvbuf ||
> +					skb_queue_empty(&sk->sk_receive_queue))) {

Similar consideration WRT the previous patch, we need protect against
possible attacks. I think checking and allowing in-sequence skbs should
be enough:

			if (own_msk && (sk_rmem_alloc_get(sk) < sk->sk_rcvbuf ||
			    MPTCP_SKB_CB(skb)->map_seq == msk->ack_seq)) {

/P

Re: [Patch mptcp-net 3/3] mptcp: fix stall because of data_ready

[gang.yan]

[-- Attachment #1: Type: text/plain, Size: 4695 bytes --]

February 5, 2026 at 6:07 PM, "Paolo Abeni" <pabeni@redhat.com mailto:pabeni@redhat.com?to=%22Paolo%20Abeni%22%20%3Cpabeni%40redhat.com%3E > wrote:
Hi, Paolo:

> 
> On 2/5/26 7:41 AM, Gang Yan wrote:
> 
> > 
> > From: Gang Yan <yangang@kylinos.cn>
> >  
> >  This patch fixes the second type of backlog_list stall issue that
> >  occurs when the data_ready callback attempts to trigger data transfer.
> >  The issue reproduces at approximately a 20% rate (once every five runs)
> >  when running multi_chunk.sh tests.
> >  
> >  The stall occurs under the following conditions:
> >  1. A large amount of out-of-order (OFO) data causes sk_rmem_alloc to
> >  exceed sk_rcvbuf
> > 
> Thinking again about this scenario, the above condition is quite
> unexpected to me. Could you please share a pcap capture of this stall?
> In theory the sender should not fill the rcv window with OoO skb only.

The captured packets (ns1.pcap) are attached to this email.

> 
> > 
> > 2. The skb matching the current ack_seq is not present in backlog_list
> >  3. Data reception relies on data_ready callback notification
> >  
> >  In this scenario, the data_ready callback (via mptcp_data_ready() ->
> >  sk->sk_data_ready()) attempts to trigger data movement, but
> >  __mptcp_move_skbs_from_subflow() repeatedly moves the ack_seq skb into
> >  backlog_list and returns false:
> >  
> >  '''
> >  [ 144.961282][ C0] MPTCP: msk->ack_seq:3442119990924456661, map_seq:3442119990924456661, offset:0, fin:0
> >  [ 144.961293][ C0] MPTCP: [MPTCP_BACKLOG] #0 map_seq=3442119990924655746 end_seq=3442119990924660542 len=4796
> >  [ 144.962310][ C0] MPTCP: [MPTCP_BACKLOG] #1 map_seq=3442119990924491850 end_seq=3442119990924500364 len=8514
> >  [ 144.962783][ C0] MPTCP: [MPTCP_BACKLOG] #2 map_seq=3442119990924660542 end_seq=3442119990924726001 len=65459
> >  [ 144.963260][ C0] MPTCP: [MPTCP_BACKLOG] #3 map_seq=3442119990924508114 end_seq=3442119990924514971 len=6857
> >  [ 144.963729][ C0] MPTCP: [MPTCP_BACKLOG] #4 map_seq=3442119990924726001 end_seq=3442119990924731093 len=5092
> >  [ 144.964193][ C0] MPTCP: [MPTCP_BACKLOG] #5 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.964651][ C0] MPTCP: [MPTCP_BACKLOG] #6 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.965164][ C0] MPTCP: [MPTCP_BACKLOG] #7 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.965711][ C0] MPTCP: [MPTCP_BACKLOG] #8 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.966260][ C0] MPTCP: [MPTCP_BACKLOG] #9 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.966804][ C0] MPTCP: [MPTCP_BACKLOG] #10 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.967308][ C0] MPTCP: [MPTCP_BACKLOG] #11 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.967793][ C0] MPTCP: [MPTCP_BACKLOG] #12 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  ...
> >  '''
> >  
> >  The fix adds a check for an empty receive queue in addition to the
> >  rcvbuf comparison. When the receive queue is empty, skbs should still
> >  be moved to prevent the stall. With this patch, all mptcp_tls tests
> >  pass successfully.
> >  
> >  Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
> >  Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
> >  Signed-off-by: Gang Yan <yangang@kylinos.cn>
> >  ---
> >  net/mptcp/protocol.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >  
> >  diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
> >  index b6bafc37eea4..054aa72c9aa6 100644
> >  --- a/net/mptcp/protocol.c
> >  +++ b/net/mptcp/protocol.c
> >  @@ -739,7 +739,8 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
> >  
> >  mptcp_init_skb(ssk, skb, offset, len);
> >  
> >  - if (own_msk && sk_rmem_alloc_get(sk) < sk->sk_rcvbuf) {
> >  + if (own_msk && (sk_rmem_alloc_get(sk) < sk->sk_rcvbuf ||
> >  + skb_queue_empty(&sk->sk_receive_queue))) {
> > 
> Similar consideration WRT the previous patch, we need protect against
> possible attacks. I think checking and allowing in-sequence skbs should
> be enough:
> 
>  if (own_msk && (sk_rmem_alloc_get(sk) < sk->sk_rcvbuf ||
>  MPTCP_SKB_CB(skb)->map_seq == msk->ack_seq)) {
> 
This patch addresses the blocking issue that affects the NVMe-over-MPTCP testing.
Could this fix be considered for merging separately into the export branch as a
standalone fix? If that’s acceptable, I’ll prepare and send a v2 patch focused
only on this fix shortly.

Best regards,
Gang


> /P
>

[-- Attachment #2: ns1.pcap --]
[-- Type: application/vnd.tcpdump.pcap, Size: 672053 bytes --]

Re: [Patch mptcp-net 3/3] mptcp: fix stall because of data_ready

[gang.yan]

February 5, 2026 at 9:27 PM, gang.yan@linux.dev mailto:gang.yan@linux.dev  wrote:


> 
> February 5, 2026 at 6:07 PM, "Paolo Abeni" <pabeni@redhat.com mailto:pabeni@redhat.com?to=%22Paolo%20Abeni%22%20%3Cpabeni%40redhat.com%3E > wrote:
> Hi, Paolo:
> 
> > 
> > On 2/5/26 7:41 AM, Gang Yan wrote:
> >  
> >  
> >  From: Gang Yan <yangang@kylinos.cn>
> >  
> >  This patch fixes the second type of backlog_list stall issue that
> >  occurs when the data_ready callback attempts to trigger data transfer.
> >  The issue reproduces at approximately a 20% rate (once every five runs)
> >  when running multi_chunk.sh tests.
> >  
> >  The stall occurs under the following conditions:
> >  1. A large amount of out-of-order (OFO) data causes sk_rmem_alloc to
> >  exceed sk_rcvbuf
> >  
> >  Thinking again about this scenario, the above condition is quite
> >  unexpected to me. Could you please share a pcap capture of this stall?
> >  In theory the sender should not fill the rcv window with OoO skb only.
> > 
> The captured packets (ns1.pcap) are attached to this email.
> 
> > 
> > 2. The skb matching the current ack_seq is not present in backlog_list
> >  3. Data reception relies on data_ready callback notification
> >  
> >  In this scenario, the data_ready callback (via mptcp_data_ready() ->
> >  sk->sk_data_ready()) attempts to trigger data movement, but
> >  __mptcp_move_skbs_from_subflow() repeatedly moves the ack_seq skb into
> >  backlog_list and returns false:
> >  
> >  '''
> >  [ 144.961282][ C0] MPTCP: msk->ack_seq:3442119990924456661, map_seq:3442119990924456661, offset:0, fin:0
> >  [ 144.961293][ C0] MPTCP: [MPTCP_BACKLOG] #0 map_seq=3442119990924655746 end_seq=3442119990924660542 len=4796
> >  [ 144.962310][ C0] MPTCP: [MPTCP_BACKLOG] #1 map_seq=3442119990924491850 end_seq=3442119990924500364 len=8514
> >  [ 144.962783][ C0] MPTCP: [MPTCP_BACKLOG] #2 map_seq=3442119990924660542 end_seq=3442119990924726001 len=65459
> >  [ 144.963260][ C0] MPTCP: [MPTCP_BACKLOG] #3 map_seq=3442119990924508114 end_seq=3442119990924514971 len=6857
> >  [ 144.963729][ C0] MPTCP: [MPTCP_BACKLOG] #4 map_seq=3442119990924726001 end_seq=3442119990924731093 len=5092
> >  [ 144.964193][ C0] MPTCP: [MPTCP_BACKLOG] #5 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.964651][ C0] MPTCP: [MPTCP_BACKLOG] #6 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.965164][ C0] MPTCP: [MPTCP_BACKLOG] #7 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.965711][ C0] MPTCP: [MPTCP_BACKLOG] #8 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.966260][ C0] MPTCP: [MPTCP_BACKLOG] #9 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.966804][ C0] MPTCP: [MPTCP_BACKLOG] #10 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.967308][ C0] MPTCP: [MPTCP_BACKLOG] #11 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  [ 144.967793][ C0] MPTCP: [MPTCP_BACKLOG] #12 map_seq=3442119990924456661 end_seq=3442119990924464310 len=7649
> >  ...
> >  '''
> >  
> >  The fix adds a check for an empty receive queue in addition to the
> >  rcvbuf comparison. When the receive queue is empty, skbs should still
> >  be moved to prevent the stall. With this patch, all mptcp_tls tests
> >  pass successfully.
> >  
> >  Co-developed-by: Geliang Tang <tanggeliang@kylinos.cn>
> >  Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
> >  Signed-off-by: Gang Yan <yangang@kylinos.cn>
> >  ---
> >  net/mptcp/protocol.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >  
> >  diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
> >  index b6bafc37eea4..054aa72c9aa6 100644
> >  --- a/net/mptcp/protocol.c
> >  +++ b/net/mptcp/protocol.c
> >  @@ -739,7 +739,8 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
> >  
> >  mptcp_init_skb(ssk, skb, offset, len);
> >  
> >  - if (own_msk && sk_rmem_alloc_get(sk) < sk->sk_rcvbuf) {
> >  + if (own_msk && (sk_rmem_alloc_get(sk) < sk->sk_rcvbuf ||
> >  + skb_queue_empty(&sk->sk_receive_queue))) {
> >  
> >  Similar consideration WRT the previous patch, we need protect against
> >  possible attacks. I think checking and allowing in-sequence skbs should
> >  be enough:
> >  
> >  if (own_msk && (sk_rmem_alloc_get(sk) < sk->sk_rcvbuf ||
> >  MPTCP_SKB_CB(skb)->map_seq == msk->ack_seq)) {
> > 
> This patch addresses the blocking issue that affects the NVMe-over-MPTCP testing.
> Could this fix be considered for merging separately into the export branch as a
> standalone fix? If that’s acceptable, I’ll prepare and send a v2 patch focused
> only on this fix shortly.
> 
> Best regards,
> Gang
> 
Hi Matt, Paolo:

Please note that v2 of this patch has already been sent, as it is a dependency for
the upcoming NVMe over MPTCP support.

Thanks
Gang

> > 
> > /P
> >
>

Re: [Patch mptcp-net 0/3] Fix the transmission stall due to backlog

[MPTCP CI]

Hi Gang,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join): Success! ✅
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Success! ✅
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/21701956586

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/189e81c8d4e0
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1051012


If there are some issues, you can reproduce them using the same environment as
the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-normal

For more details:

    https://github.com/multipath-tcp/mptcp-upstream-virtme-docker


Please note that despite all the efforts that have been already done to have a
stable tests suite when executed on a public CI like here, it is possible some
reported issues are not due to your modifications. Still, do not hesitate to
help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)