[PATCH mptcp-net 0/3] selftests: mptcp: avoid spurious errors on TCP disconnect

[PATCH mptcp-net 0/3] selftests: mptcp: avoid spurious errors on TCP disconnect

[Matthieu Baerts (NGI0)]

This should fix the recent instabilities seen by MPTCP and NIPA CIs
where the 'mptcp_connect.sh' tests fail regularly when running the
'disconnect' subtests with "plain" TCP sockets, e.g.

  # INFO: disconnect
  # 63 ns1 MPTCP -> ns1 (10.0.1.1:20001      ) MPTCP     (duration   996ms) [ OK ]
  # 64 ns1 MPTCP -> ns1 (10.0.1.1:20002      ) TCP       (duration   851ms) [ OK ]
  # 65 ns1 TCP   -> ns1 (10.0.1.1:20003      ) MPTCP     Unexpected revents: POLLERR/POLLNVAL(19)
  # (duration   896ms) [FAIL] file received by server does not match (in, out):
  # -rw-r--r-- 1 root root 11112852 Aug 19 09:16 /tmp/tmp.hlJe5DoMoq.disconnect
  # Trailing bytes are:
  # /{ga 6@=#.8:-rw------- 1 root root 10085368 Aug 19 09:16 /tmp/tmp.blClunilxx
  # Trailing bytes are:
  # /{ga 6@=#.8:66 ns1 MPTCP -> ns1 (dead:beef:1::1:20004) MPTCP     (duration   987ms) [ OK ]
  # 67 ns1 MPTCP -> ns1 (dead:beef:1::1:20005) TCP       (duration   911ms) [ OK ]
  # 68 ns1 TCP   -> ns1 (dead:beef:1::1:20006) MPTCP     (duration   980ms) [ OK ]
  # [FAIL] Tests of the full disconnection have failed

Patch 2 fixes this issue, while Patch 1 and 3 improves the errors
reported by the selftest.

Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
---
Matthieu Baerts (NGI0) (3):
      selftests: mptcp: connect: catch IO errors on listen side
      selftests: mptcp: avoid spurious errors on TCP disconnect
      selftests: mptcp: print trailing bytes with hexdump

 tools/testing/selftests/net/mptcp/mptcp_connect.c | 10 +++++++---
 tools/testing/selftests/net/mptcp/mptcp_lib.sh    | 12 +++++++++++-
 2 files changed, 18 insertions(+), 4 deletions(-)
---
base-commit: 27427f653031e9aca497b147b2d8eb86bf12fb9e
change-id: 20250806-sft-mptcp-disc-err-3357b769bcdb

Best regards,
-- 
Matthieu Baerts (NGI0) <matttbe@kernel.org>

[PATCH mptcp-net 1/3] selftests: mptcp: connect: catch IO errors on listen side

[Matthieu Baerts (NGI0)]

IO errors were correctly printed to stderr, and propagated up to the
main loop for the server side, but the returned value was ignored. As a
consequence, the program for the listener side was no longer exiting
with an error code in case of IO issues.

Because of that, some issues might not have been seen. But very likely,
most issues either had an effect on the client side, or the file
transfer was not the expected one, e.g. the connection got reset before
the end. Still, it is better to fix this.

The main consequence of this issue is the error that was reported by the
selftests: the received and sent files were different, and the MIB
counters were not printed. Also, when such errors happened during the
'disconnect' tests, the program tried to continue until the timeout.

Now when an IO error is detected, the program exits directly with an
error.

Fixes: 05be5e273c84 ("selftests: mptcp: add disconnect tests")
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
---
 tools/testing/selftests/net/mptcp/mptcp_connect.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/net/mptcp/mptcp_connect.c b/tools/testing/selftests/net/mptcp/mptcp_connect.c
index 4f07ac9fa207cb08a934582b98d688d0b9512f97..c1586a7286123a509495ac319fd624b98f0bfd18 100644
--- a/tools/testing/selftests/net/mptcp/mptcp_connect.c
+++ b/tools/testing/selftests/net/mptcp/mptcp_connect.c
@@ -1112,6 +1112,8 @@ int main_loop_s(int listensock)
 	salen = sizeof(ss);
 	remotesock = accept(listensock, (struct sockaddr *)&ss, &salen);
 	if (remotesock >= 0) {
+		int err;
+
 		maybe_close(listensock);
 		check_sockaddr(pf, &ss, salen);
 		check_getpeername(remotesock, &ss, salen);
@@ -1125,7 +1127,9 @@ int main_loop_s(int listensock)
 		SOCK_TEST_TCPULP(remotesock, 0);
 
 		memset(&winfo, 0, sizeof(winfo));
-		copyfd_io(fd, remotesock, 1, true, &winfo);
+		err = copyfd_io(fd, remotesock, 1, true, &winfo);
+		if (err)
+			return err;
 	} else {
 		perror("accept");
 		return 1;

-- 
2.50.0

[PATCH mptcp-net 2/3] selftests: mptcp: avoid spurious errors on TCP disconnect

[Matthieu Baerts (NGI0)]

The disconnect test-case, with 'plain' TCP sockets generates spurious
errors, e.g.

  07 ns1 TCP   -> ns1 (dead:beef:1::1:10006) MPTCP
  read: Connection reset by peer
  read: Connection reset by peer
  (duration   155ms) [FAIL] client exit code 3, server 3

  netns ns1-FloSdv (listener) socket stat for 10006:
  TcpActiveOpens                  2                  0.0
  TcpPassiveOpens                 2                  0.0
  TcpEstabResets                  2                  0.0
  TcpInSegs                       274                0.0
  TcpOutSegs                      276                0.0
  TcpOutRsts                      3                  0.0
  TcpExtPruneCalled               2                  0.0
  TcpExtRcvPruned                 1                  0.0
  TcpExtTCPPureAcks               104                0.0
  TcpExtTCPRcvCollapsed           2                  0.0
  TcpExtTCPBacklogCoalesce        42                 0.0
  TcpExtTCPRcvCoalesce            43                 0.0
  TcpExtTCPChallengeACK           1                  0.0
  TcpExtTCPFromZeroWindowAdv      42                 0.0
  TcpExtTCPToZeroWindowAdv        41                 0.0
  TcpExtTCPWantZeroWindowAdv      13                 0.0
  TcpExtTCPOrigDataSent           164                0.0
  TcpExtTCPDelivered              165                0.0
  TcpExtTCPRcvQDrop               1                  0.0

In the failing scenarios (TCP -> MPTCP), the involved sockets are
actually plain TCP ones, as fallbacks for passive sockets at 2WHS time
cause the MPTCP listeners to actually create 'plain' TCP sockets.

Similar to commit 218cc166321f ("selftests: mptcp: avoid spurious errors
on disconnect"), the root cause is in the user-space bits: the test
program tries to disconnect as soon as all the pending data has been
spooled, generating an RST. If such option reaches the peer before the
latter has reached the closed status, the TCP socket will report an
error to the user-space, as per protocol specification, causing the
above failure. Note that it looks like this issue got more visible since
the "tcp: receiver changes" series from commit 06baf9bfa6ca ("Merge
branch 'tcp-receiver-changes'").

Address the issue by explicitly waiting for all the relevant TCP sockets
to reach a closed status before performing the disconnect. More
precisely, the test program now waits if there are still some sockets in
ESTABLISHED, FIN_WAIT_1, LAST_ACK, or CLOSING. So there is a reconnect
only in case of FIN_WAIT_2, CLOSE_WAIT, TIME_WAIT, and CLOSED sockets.

While at it, use 'ss' with '-n' to avoid resolving service names which
is not needed here.

Fixes: 218cc166321f ("selftests: mptcp: avoid spurious errors on disconnect")
Suggested-by: Paolo Abeni <pabeni@redhat.com>
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
---
 tools/testing/selftests/net/mptcp/mptcp_connect.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/net/mptcp/mptcp_connect.c b/tools/testing/selftests/net/mptcp/mptcp_connect.c
index c1586a7286123a509495ac319fd624b98f0bfd18..00b0307b7da2236fd3ba4ec4b6cec91b6b4b2538 100644
--- a/tools/testing/selftests/net/mptcp/mptcp_connect.c
+++ b/tools/testing/selftests/net/mptcp/mptcp_connect.c
@@ -1251,7 +1251,7 @@ void xdisconnect(int fd)
 	else
 		xerror("bad family");
 
-	strcpy(cmd, "ss -M | grep -q ");
+	strcpy(cmd, "ss -Mnt state established state fin-wait-1 state last-ack state closing | grep -q ");
 	cmdlen = strlen(cmd);
 	if (!inet_ntop(addr.ss_family, raw_addr, &cmd[cmdlen],
 		       sizeof(cmd) - cmdlen))
@@ -1261,7 +1261,7 @@ void xdisconnect(int fd)
 
 	/*
 	 * wait until the pending data is completely flushed and all
-	 * the MPTCP sockets reached the closed status.
+	 * the sockets reached the closed status.
 	 * disconnect will bypass/ignore/drop any pending data.
 	 */
 	for (i = 0; ; i += msec_sleep) {

-- 
2.50.0

Re: [PATCH mptcp-net 2/3] selftests: mptcp: avoid spurious errors on TCP disconnect

[Paolo Abeni]

On 8/19/25 8:05 PM, Matthieu Baerts (NGI0) wrote:

> diff --git a/tools/testing/selftests/net/mptcp/mptcp_connect.c b/tools/testing/selftests/net/mptcp/mptcp_connect.c
> index c1586a7286123a509495ac319fd624b98f0bfd18..00b0307b7da2236fd3ba4ec4b6cec91b6b4b2538 100644
> --- a/tools/testing/selftests/net/mptcp/mptcp_connect.c
> +++ b/tools/testing/selftests/net/mptcp/mptcp_connect.c
> @@ -1251,7 +1251,7 @@ void xdisconnect(int fd)
>  	else
>  		xerror("bad family");
>  
> -	strcpy(cmd, "ss -M | grep -q ");
> +	strcpy(cmd, "ss -Mnt state established state fin-wait-1 state last-ack state closing | grep -q ");

Just to be on the safe side, also add 'close-wait'?

/P

[PATCH mptcp-net 3/3] selftests: mptcp: print trailing bytes with hexdump

[Matthieu Baerts (NGI0)]

If hexdump or busybox hexdump is available.

This is better than printing random bytes in the terminal.

Suggested-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
---
 tools/testing/selftests/net/mptcp/mptcp_lib.sh | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/net/mptcp/mptcp_lib.sh b/tools/testing/selftests/net/mptcp/mptcp_lib.sh
index 09cd24b2ae466205dacbdf8289eb86c08534c475..20ce95c9d35538a03c1c3cac843657d258fc9715 100644
--- a/tools/testing/selftests/net/mptcp/mptcp_lib.sh
+++ b/tools/testing/selftests/net/mptcp/mptcp_lib.sh
@@ -382,9 +382,19 @@ mptcp_lib_make_file() {
 
 # $1: file
 mptcp_lib_print_file_err() {
+	local end
+
 	ls -l "${1}" 1>&2
 	echo "Trailing bytes are: "
-	tail -c 27 "${1}"
+	end=$(tail -c 32 "${1}")
+	if command -v hexdump >/dev/null; then
+		echo "${end}" | hexdump -C | head -n2
+	elif busybox hexdump --help 2>/dev/null; then
+		echo "${end}" | busybox hexdump -C | head -n2
+	else
+		echo "${end}"
+		echo
+	fi
 }
 
 # $1: input file ; $2: output file ; $3: what kind of file

-- 
2.50.0

Re: [PATCH mptcp-net 3/3] selftests: mptcp: print trailing bytes with hexdump

[Mat Martineau]

On Tue, 19 Aug 2025, Matthieu Baerts (NGI0) wrote:

> If hexdump or busybox hexdump is available.
>
> This is better than printing random bytes in the terminal.
>
> Suggested-by: Jakub Kicinski <kuba@kernel.org>
> Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
> ---
> tools/testing/selftests/net/mptcp/mptcp_lib.sh | 12 +++++++++++-
> 1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/tools/testing/selftests/net/mptcp/mptcp_lib.sh b/tools/testing/selftests/net/mptcp/mptcp_lib.sh
> index 09cd24b2ae466205dacbdf8289eb86c08534c475..20ce95c9d35538a03c1c3cac843657d258fc9715 100644
> --- a/tools/testing/selftests/net/mptcp/mptcp_lib.sh
> +++ b/tools/testing/selftests/net/mptcp/mptcp_lib.sh
> @@ -382,9 +382,19 @@ mptcp_lib_make_file() {
>
> # $1: file
> mptcp_lib_print_file_err() {
> +	local end
> +
> 	ls -l "${1}" 1>&2
> 	echo "Trailing bytes are: "
> -	tail -c 27 "${1}"
> +	end=$(tail -c 32 "${1}")
> +	if command -v hexdump >/dev/null; then

Hi Matthieu -

How about:

tail -c 32 "${1}" | od -t x1

'od' is in the posix spec so it should be on more systems than hexdump 
(it's in coreutils along with 'tail', for example). Also, today I learned 
that 'od' can format hexadecimal...


> +		echo "${end}" | hexdump -C | head -n2
> +	elif busybox hexdump --help 2>/dev/null; then
> +		echo "${end}" | busybox hexdump -C | head -n2
> +	else
> +		echo "${end}"

Would be best to avoid fallback to raw binary (hence the 'od' suggestion 
above).

- Mat


> +		echo
> +	fi
> }
>
> # $1: input file ; $2: output file ; $3: what kind of file
>
> -- 
> 2.50.0
>
>
>

Re: [PATCH mptcp-net 0/3] selftests: mptcp: avoid spurious errors on TCP disconnect

[MPTCP CI]

Hi Matthieu,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal: Unstable: 1 failed test(s): packetdrill_fastopen 🔴
- KVM Validation: debug: Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/17078272848

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/3c178d4c6234
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=993167


If there are some issues, you can reproduce them using the same environment as
the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-normal

For more details:

    https://github.com/multipath-tcp/mptcp-upstream-virtme-docker


Please note that despite all the efforts that have been already done to have a
stable tests suite when executed on a public CI like here, it is possible some
reported issues are not due to your modifications. Still, do not hesitate to
help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)

Re: [PATCH mptcp-net 0/3] selftests: mptcp: avoid spurious errors on TCP disconnect

[Mat Martineau]

On Tue, 19 Aug 2025, Matthieu Baerts (NGI0) wrote:

> This should fix the recent instabilities seen by MPTCP and NIPA CIs
> where the 'mptcp_connect.sh' tests fail regularly when running the
> 'disconnect' subtests with "plain" TCP sockets, e.g.
>
>  # INFO: disconnect
>  # 63 ns1 MPTCP -> ns1 (10.0.1.1:20001      ) MPTCP     (duration   996ms) [ OK ]
>  # 64 ns1 MPTCP -> ns1 (10.0.1.1:20002      ) TCP       (duration   851ms) [ OK ]
>  # 65 ns1 TCP   -> ns1 (10.0.1.1:20003      ) MPTCP     Unexpected revents: POLLERR/POLLNVAL(19)
>  # (duration   896ms) [FAIL] file received by server does not match (in, out):
>  # -rw-r--r-- 1 root root 11112852 Aug 19 09:16 /tmp/tmp.hlJe5DoMoq.disconnect
>  # Trailing bytes are:
>  # /{ga 6@=#.8:-rw------- 1 root root 10085368 Aug 19 09:16 /tmp/tmp.blClunilxx
>  # Trailing bytes are:
>  # /{ga 6@=#.8:66 ns1 MPTCP -> ns1 (dead:beef:1::1:20004) MPTCP     (duration   987ms) [ OK ]
>  # 67 ns1 MPTCP -> ns1 (dead:beef:1::1:20005) TCP       (duration   911ms) [ OK ]
>  # 68 ns1 TCP   -> ns1 (dead:beef:1::1:20006) MPTCP     (duration   980ms) [ OK ]
>  # [FAIL] Tests of the full disconnection have failed
>
> Patch 2 fixes this issue, while Patch 1 and 3 improves the errors
> reported by the selftest.
>
> Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
> ---

With Paolo's suggestion for patch 2 (include close-wait state) and mine 
for patch 3 ('od') included, this LGTM:

Reviewed-by: Mat Martineau <martineau@kernel.org>


> Matthieu Baerts (NGI0) (3):
>      selftests: mptcp: connect: catch IO errors on listen side
>      selftests: mptcp: avoid spurious errors on TCP disconnect
>      selftests: mptcp: print trailing bytes with hexdump
>
> tools/testing/selftests/net/mptcp/mptcp_connect.c | 10 +++++++---
> tools/testing/selftests/net/mptcp/mptcp_lib.sh    | 12 +++++++++++-
> 2 files changed, 18 insertions(+), 4 deletions(-)
> ---
> base-commit: 27427f653031e9aca497b147b2d8eb86bf12fb9e
> change-id: 20250806-sft-mptcp-disc-err-3357b769bcdb
>
> Best regards,
> -- 
> Matthieu Baerts (NGI0) <matttbe@kernel.org>
>
>
>