[REGRESSION] 6.12.111: "net: advertise TCP MSS from the configured MTU, not the learned PMTU" breaks TCP through relays when the peer's network drops ICMP

[REGRESSION] 6.12.111: "net: advertise TCP MSS from the configured MTU, not the learned PMTU" breaks TCP through relays when the peer's network drops ICMP

[Echoo Wall]

Hi,

After a routine stable update from 6.12.107 to 6.12.111 (Debian trixie-security,
6.12.111-1), TCP connections to our servers started timing out for a subset of
clients. Booting back to 6.12.107 / 6.12.101 fixed it immediately. I believe the
cause is:

  2640e64195948a601430d230c9864f5426574cde
  net: advertise TCP MSS from the configured MTU, not the learned PMTU
  (in stable 6.12.111 and 6.18.51)

#regzbot introduced: 2640e64195948a601430d230c9864f5426574cde

Topology
--------

  client ---(internet)---> relay (Linux, nftables DNAT + masquerade) ---> server
                                                               ^
                                                 path MTU 1456 on this hop

- Servers have a 1500-byte interface MTU, but the path between the relay and the
  server has a PMTU of 1456 (encapsulation upstream). The server learns this
  correctly: `ip route get <relay>` shows `cache mtu 1456`.
- Clients reach the server only through the relay (port-based DNAT, SNAT to the
  relay address).
- Some client networks (typical corporate firewalls) drop ICMP
  "fragmentation needed".

Behaviour
---------

Before (<= 6.12.110): the server's SYN-ACK advertised MSS 1416, derived from the
learned PMTU. Clients never sent segments that were too large. Everything worked,
including from networks that filter ICMP.

After (6.12.111): the SYN-ACK advertises MSS 1460 from the configured MTU. The
client sends 1500-byte packets. They are dropped at the 1456 hop, and the ICMP
error never reaches the client because its network filters it. The handshake
completes, then the connection stalls: a classic PMTU black hole.

Direct connections to the same servers, without the relay hop, keep working, and
so do servers still on <= 6.12.110 behind the same relays. That made the problem
very hard to attribute: it only appears from certain client networks, only on
relayed paths, and only after a kernel update that nothing points to.

What I verified
---------------

- Reproducible per host by switching only the kernel. Rebooting without changing
  the kernel does not help. Same sysctls on both kernels (309 TCP parameters
  compared), same userspace.
- On 6.12.107 the server has `cache mtu 1456` towards the relay, so the
  SYN-ACK MSS is derived from it (1416). With this commit the advertised
  MSS comes from the 1500-byte device MTU (1460) instead. That matches the
  symptom exactly, but I have not yet captured a 6.12.111 SYN-ACK on the
  affected path.
- As a workaround we now clamp MSS on the relay
  (`tcp flags syn tcp option maxseg size > 1416 tcp option maxseg size set 1416`
  in the forward chain). I have not yet re-tested 6.12.111 behind it.

Why I think this matters
------------------------

The commit message argues that on symmetric paths nothing is lost because "the
peer usually already knows the real path MTU". That assumption does not hold when:

  1. the peer sits behind a firewall that drops ICMP fragmentation-needed, which
     is common in enterprise networks; and/or
  2. the narrow hop is only on the server side (relay, tunnel or overlay egress),
     so the peer has no way to learn it other than via ICMP.

In both cases the receiver-side PMTU knowledge was the only thing that kept the
connection working. The old behaviour was effectively a built-in safeguard for
these black holes, and it has been in place for a very long time.

This is now shipping as a security update in LTS/stable kernels and
distributions. It silently breaks previously working deployments, and only for
some users, so most operators will not connect the symptom to the kernel. The
6.6 / 6.1 / 5.15 backports appear to be still pending, so there may still be
time to reconsider before it spreads further.

Possible ways forward (just suggestions):

  - revert from stable and keep it in mainline until there is a safer variant;
  - make it opt-in (sysctl or per-route flag) for the asymmetric / DSR case it
    targets;
  - or only ignore the learned PMTU when there is evidence of asymmetry, instead
    of unconditionally.

I am happy to test patches on the affected setup.

Thanks.
Esko Mobius

Re: [REGRESSION] 6.12.111: "net: advertise TCP MSS from the configured MTU, not the learned PMTU" breaks TCP through relays when the peer's network drops ICMP

[Thorsten Leemhuis]

On 10/9/26 18:27, Echoo Wall wrote:
> 
> After a routine stable update from 6.12.107 to 6.12.111 (Debian trixie-security,
> 6.12.111-1), TCP connections to our servers started timing out for a subset of
> clients. Booting back to 6.12.107 / 6.12.101 fixed it immediately. I believe the
> cause is:
> 
>   2640e64195948a601430d230c9864f5426574cde
>   net: advertise TCP MSS from the configured MTU, not the learned PMTU
>   (in stable 6.12.111 and 6.18.51)
> 
> #regzbot introduced: 2640e64195948a601430d230c9864f5426574cde

Thx for the report. That is a mainline commit, but this might be a
stable specific thing due to a broken backport. These cases in the end
might be handled by different people, so somebody needs to check which
if the two it is. Could you thus check if 7.3-rc7 or later is still
affected? If it is, please try to check if reverting the commit fixes
things; if it is not, please check a revert with the stable series you use.

Ciao, Thorsten
> Topology
> --------
> 
>   client ---(internet)---> relay (Linux, nftables DNAT + masquerade) ---> server
>                                                                ^
>                                                  path MTU 1456 on this hop
> 
> - Servers have a 1500-byte interface MTU, but the path between the relay and the
>   server has a PMTU of 1456 (encapsulation upstream). The server learns this
>   correctly: `ip route get <relay>` shows `cache mtu 1456`.
> - Clients reach the server only through the relay (port-based DNAT, SNAT to the
>   relay address).
> - Some client networks (typical corporate firewalls) drop ICMP
>   "fragmentation needed".
> 
> Behaviour
> ---------
> 
> Before (<= 6.12.110): the server's SYN-ACK advertised MSS 1416, derived from the
> learned PMTU. Clients never sent segments that were too large. Everything worked,
> including from networks that filter ICMP.
> 
> After (6.12.111): the SYN-ACK advertises MSS 1460 from the configured MTU. The
> client sends 1500-byte packets. They are dropped at the 1456 hop, and the ICMP
> error never reaches the client because its network filters it. The handshake
> completes, then the connection stalls: a classic PMTU black hole.
> 
> Direct connections to the same servers, without the relay hop, keep working, and
> so do servers still on <= 6.12.110 behind the same relays. That made the problem
> very hard to attribute: it only appears from certain client networks, only on
> relayed paths, and only after a kernel update that nothing points to.
> 
> What I verified
> ---------------
> 
> - Reproducible per host by switching only the kernel. Rebooting without changing
>   the kernel does not help. Same sysctls on both kernels (309 TCP parameters
>   compared), same userspace.
> - On 6.12.107 the server has `cache mtu 1456` towards the relay, so the
>   SYN-ACK MSS is derived from it (1416). With this commit the advertised
>   MSS comes from the 1500-byte device MTU (1460) instead. That matches the
>   symptom exactly, but I have not yet captured a 6.12.111 SYN-ACK on the
>   affected path.
> - As a workaround we now clamp MSS on the relay
>   (`tcp flags syn tcp option maxseg size > 1416 tcp option maxseg size set 1416`
>   in the forward chain). I have not yet re-tested 6.12.111 behind it.
> 
> Why I think this matters
> ------------------------
> 
> The commit message argues that on symmetric paths nothing is lost because "the
> peer usually already knows the real path MTU". That assumption does not hold when:
> 
>   1. the peer sits behind a firewall that drops ICMP fragmentation-needed, which
>      is common in enterprise networks; and/or
>   2. the narrow hop is only on the server side (relay, tunnel or overlay egress),
>      so the peer has no way to learn it other than via ICMP.
> 
> In both cases the receiver-side PMTU knowledge was the only thing that kept the
> connection working. The old behaviour was effectively a built-in safeguard for
> these black holes, and it has been in place for a very long time.
> 
> This is now shipping as a security update in LTS/stable kernels and
> distributions. It silently breaks previously working deployments, and only for
> some users, so most operators will not connect the symptom to the kernel. The
> 6.6 / 6.1 / 5.15 backports appear to be still pending, so there may still be
> time to reconsider before it spreads further.
> 
> Possible ways forward (just suggestions):
> 
>   - revert from stable and keep it in mainline until there is a safer variant;
>   - make it opt-in (sysctl or per-route flag) for the asymmetric / DSR case it
>     targets;
>   - or only ignore the learned PMTU when there is evidence of asymmetry, instead
>     of unconditionally.
> 
> I am happy to test patches on the affected setup.
> 
> Thanks.
> Esko Mobius

Re: [REGRESSION] 6.12.111: "net: advertise TCP MSS from the configured MTU, not the learned PMTU" breaks TCP through relays when the peer's network drops ICMP

[Echoo Wall]

On 10/15/26, Thorsten Leemhuis wrote:
> Thx for the report. That is a mainline commit, but this might be a
> stable specific thing due to a broken backport. These cases in the end
> might be handled by different people, so somebody needs to check which
> if the two it is. Could you thus check if 7.3-rc7 or later is still
> affected? If it is, please try to check if reverting the commit fixes
> things; if it is not, please check a revert with the stable series you use.

7.3-rc7 is affected, and reverting 2640e64195948a601430d230c9864f5426574cde
on top of 7.3-rc7 fixes it. So this is not a broken backport; mainline
behaves the same as 6.12.111.

To make this reproducible without our production setup, I wrote a small
reproducer that runs on a single host with network namespaces (script
below):

  cli --(1500)-- r1 --(1456)-- r2 --(1500)-- srv

srv first learns PMTU 1456 towards cli (r2 sends it ICMP frag-needed),
then cli connects to srv and uploads 1 MB. r1 drops the ICMP
frag-needed that would go back to cli, like a firewall in front of the
client. All kernels run in the same KVM guest (Debian 13 userspace).

Results (3 runs each, identical every time):

  kernel                        srv PMTU   SYN-ACK MSS   1 MB upload
  6.12.107 (Debian)             1456       1416          OK (0.03s)
  6.12.111 (Debian)             1456       1460          stalls, 10s timeout
  7.3-rc7                       1456       1460          stalls, 10s timeout
  7.3-rc7 + revert of 2640e641  1456       1416          OK (0.03s)

The two 7.3-rc7 kernels were built from the same tree and the same
.config (Debian's 6.12 config + localmodconfig); the only difference is
the revert, which applies cleanly (4 files changed, 3 insertions(+),
65 deletions(-)).

Controls:

- Forcing srv to advertise 1460 (ip route ... advmss 1460) stalls on
  every kernel, including 6.12.107 and the revert, and r1 counts 8
  dropped frag-needed per run. So the black hole is real and is
  triggered exactly by the advertised MSS.
- Clamping MSS to 1416 on r1 (nft: tcp option maxseg size set 1416)
  makes 6.12.111 work again (3/3). That is the workaround we have
  deployed on our relays.

This also confirms the two points I had marked as not yet verified in
my original report.

Reproducer (run as root; needs iproute2, nft, tcpdump, python3, bc;
optional argument "force1460" or "clamp" for the controls above):

---8<---
#!/bin/bash
# cli --(1500)-- r1 --(1456)-- r2 --(1500)-- srv
set -u
MODE="${1:-}"
NS="cli r1 r2 srv"
cleanup(){ for n in $NS; do ip netns del $n 2>/dev/null; done; }
cleanup
for n in $NS; do ip netns add $n; ip -n $n link set lo up; done
ip link add c0 netns cli type veth peer name r1a netns r1
ip link add r1b netns r1 type veth peer name r2a netns r2
ip link add r2b netns r2 type veth peer name s0 netns srv
ip -n cli addr add 10.0.1.2/24 dev c0;  ip -n r1 addr add 10.0.1.1/24 dev r1a
ip -n r1 addr add 10.0.2.1/24 dev r1b;  ip -n r2 addr add 10.0.2.2/24 dev r2a
ip -n r2 addr add 10.0.3.1/24 dev r2b;  ip -n srv addr add 10.0.3.2/24 dev s0
ip -n r1 link set r1b mtu 1456; ip -n r2 link set r2a mtu 1456
for d in "cli c0" "r1 r1a" "r1 r1b" "r2 r2a" "r2 r2b" "srv s0"; do set -- $d; ip -n $1 link set $2 up; done
ip -n cli route add default via 10.0.1.1
ip -n srv route add default via 10.0.3.1
ip -n r1 route add 10.0.3.0/24 via 10.0.2.2
ip -n r2 route add 10.0.1.0/24 via 10.0.2.1
for n in r1 r2; do ip netns exec $n sysctl -qw net.ipv4.ip_forward=1; done
for n in cli srv; do ip netns exec $n sysctl -qw net.ipv4.tcp_mtu_probing=0; done
ip netns exec r1 nft -f - <<NFT
table inet fw {
  chain f1 {
    type filter hook forward priority 0;
    ip daddr 10.0.1.0/24 icmp type destination-unreachable icmp code frag-needed counter drop
  }
  chain o1 {
    type filter hook output priority 0;
    ip daddr 10.0.1.0/24 icmp type destination-unreachable icmp code frag-needed counter drop
  }
}
NFT
[ "$MODE" = "force1460" ] && ip -n srv route add 10.0.1.0/24 via 10.0.3.1 advmss 1460
[ "$MODE" = "clamp" ] && ip netns exec r1 nft add rule inet fw f1 tcp flags syn tcp option maxseg size gt 1416 tcp option maxseg size set 1416
ip netns exec srv ping -c 2 -W 1 -M do -s 1472 10.0.1.2 >/dev/null 2>&1
ip netns exec srv ping -c 1 -W 1 -M do -s 1472 10.0.1.2 >/dev/null 2>&1
PMTU=$(ip netns exec srv ip route get 10.0.1.2 | grep -o 'mtu [0-9]*' || echo "none")
ip netns exec srv python3 -c '
import socket
s=socket.socket(); s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1); s.bind(("0.0.0.0",9000)); s.listen()
c,_=s.accept(); n=0
while True:
    b=c.recv(65536)
    if not b: break
    n+=len(b)
c.sendall(str(n).encode()); c.close()' &
SP=$!
rm -f /tmp/synack.pcap
ip netns exec srv tcpdump --immediate-mode -nni any -U -w /tmp/synack.pcap 'tcp src port 9000 and tcp[13] & 18 == 18' 2>/dev/null &
TP=$!
sleep 3
START=$(date +%s.%N)
RES=$(ip netns exec cli timeout 10 python3 -c '
import socket
s=socket.create_connection(("10.0.3.2",9000),timeout=10)
s.sendall(b"x"*1048576); s.shutdown(socket.SHUT_WR); print(s.recv(100).decode())' 2>&1 | tail -1)
END=$(date +%s.%N)
sleep 2; kill $TP $SP 2>/dev/null; wait 2>/dev/null
MSS=$(tcpdump -nr /tmp/synack.pcap -v 2>/dev/null | grep -o 'mss [0-9]*' | head -1)
DROPS=$(ip netns exec r1 nft list ruleset | grep -o "packets [0-9]*" | awk "{s+=\$2} END{print s}")
echo "kernel=$(uname -r)${MODE:+ [$MODE]} | srv PMTU: ${PMTU} | SYN-ACK: ${MSS:-none} | 1MB upload: $( [ "$RES" = "1048576" ] && printf 'OK in %.2fs' $(echo "$END - $START" | bc) || echo "FAILED (${RES:-timeout})") | frag-needed dropped by r1: ${DROPS:-0}"
cleanup
---8<---

Note that in "clamp" mode the capture on srv still shows 1460, since the
rewrite happens on r1; the client sees 1416.

Thanks.
Esko Mobius


> 2026年10月15日 17:04，Thorsten Leemhuis <regressions@leemhuis.info> 写道：
> 
> On 10/9/26 18:27, Echoo Wall wrote:
>> 
>> After a routine stable update from 6.12.107 to 6.12.111 (Debian trixie-security,
>> 6.12.111-1), TCP connections to our servers started timing out for a subset of
>> clients. Booting back to 6.12.107 / 6.12.101 fixed it immediately. I believe the
>> cause is:
>> 
>>  2640e64195948a601430d230c9864f5426574cde
>>  net: advertise TCP MSS from the configured MTU, not the learned PMTU
>>  (in stable 6.12.111 and 6.18.51)
>> 
>> #regzbot introduced: 2640e64195948a601430d230c9864f5426574cde
> 
> Thx for the report. That is a mainline commit, but this might be a
> stable specific thing due to a broken backport. These cases in the end
> might be handled by different people, so somebody needs to check which
> if the two it is. Could you thus check if 7.3-rc7 or later is still
> affected? If it is, please try to check if reverting the commit fixes
> things; if it is not, please check a revert with the stable series you use.
> 
> Ciao, Thorsten
>> Topology
>> --------
>> 
>>  client ---(internet)---> relay (Linux, nftables DNAT + masquerade) ---> server
>>                                                               ^
>>                                                 path MTU 1456 on this hop
>> 
>> - Servers have a 1500-byte interface MTU, but the path between the relay and the
>>  server has a PMTU of 1456 (encapsulation upstream). The server learns this
>>  correctly: `ip route get <relay>` shows `cache mtu 1456`.
>> - Clients reach the server only through the relay (port-based DNAT, SNAT to the
>>  relay address).
>> - Some client networks (typical corporate firewalls) drop ICMP
>>  "fragmentation needed".
>> 
>> Behaviour
>> ---------
>> 
>> Before (<= 6.12.110): the server's SYN-ACK advertised MSS 1416, derived from the
>> learned PMTU. Clients never sent segments that were too large. Everything worked,
>> including from networks that filter ICMP.
>> 
>> After (6.12.111): the SYN-ACK advertises MSS 1460 from the configured MTU. The
>> client sends 1500-byte packets. They are dropped at the 1456 hop, and the ICMP
>> error never reaches the client because its network filters it. The handshake
>> completes, then the connection stalls: a classic PMTU black hole.
>> 
>> Direct connections to the same servers, without the relay hop, keep working, and
>> so do servers still on <= 6.12.110 behind the same relays. That made the problem
>> very hard to attribute: it only appears from certain client networks, only on
>> relayed paths, and only after a kernel update that nothing points to.
>> 
>> What I verified
>> ---------------
>> 
>> - Reproducible per host by switching only the kernel. Rebooting without changing
>>  the kernel does not help. Same sysctls on both kernels (309 TCP parameters
>>  compared), same userspace.
>> - On 6.12.107 the server has `cache mtu 1456` towards the relay, so the
>>  SYN-ACK MSS is derived from it (1416). With this commit the advertised
>>  MSS comes from the 1500-byte device MTU (1460) instead. That matches the
>>  symptom exactly, but I have not yet captured a 6.12.111 SYN-ACK on the
>>  affected path.
>> - As a workaround we now clamp MSS on the relay
>>  (`tcp flags syn tcp option maxseg size > 1416 tcp option maxseg size set 1416`
>>  in the forward chain). I have not yet re-tested 6.12.111 behind it.
>> 
>> Why I think this matters
>> ------------------------
>> 
>> The commit message argues that on symmetric paths nothing is lost because "the
>> peer usually already knows the real path MTU". That assumption does not hold when:
>> 
>>  1. the peer sits behind a firewall that drops ICMP fragmentation-needed, which
>>     is common in enterprise networks; and/or
>>  2. the narrow hop is only on the server side (relay, tunnel or overlay egress),
>>     so the peer has no way to learn it other than via ICMP.
>> 
>> In both cases the receiver-side PMTU knowledge was the only thing that kept the
>> connection working. The old behaviour was effectively a built-in safeguard for
>> these black holes, and it has been in place for a very long time.
>> 
>> This is now shipping as a security update in LTS/stable kernels and
>> distributions. It silently breaks previously working deployments, and only for
>> some users, so most operators will not connect the symptom to the kernel. The
>> 6.6 / 6.1 / 5.15 backports appear to be still pending, so there may still be
>> time to reconsider before it spreads further.
>> 
>> Possible ways forward (just suggestions):
>> 
>>  - revert from stable and keep it in mainline until there is a safer variant;
>>  - make it opt-in (sysctl or per-route flag) for the asymmetric / DSR case it
>>    targets;
>>  - or only ignore the learned PMTU when there is evidence of asymmetry, instead
>>    of unconditionally.
>> 
>> I am happy to test patches on the affected setup.
>> 
>> Thanks.
>> Esko Mobius
>