[PATCH net v3] ip_tunnel: drop stale dst from generated PMTU ICMP replies

[PATCH net v3] ip_tunnel: drop stale dst from generated PMTU ICMP replies

[Laika Price via B4 Relay]

From: Laika Price <laikabcprice@gmail.com>

iptunnel_pmtud_build_icmp(...) and iptunnel_pmtud_build_icmpv6(...) take
in an sk_buff, modify it to create a PMTU ICMP error reply, and return it.
As part of these modifications, the source/destination ethernet and IP
addresses are swapped around which makes the sk_buff's current dst invalid.

If the stale dst is left, the packet can skip input routing and be
forwarded using the original output device. This was observed when sending
packets to a VXLAN over a WireGuard tunnel - the ICMP reply was generated
but it was sent over the VXLAN instead of to the WireGuard tunnel.

This patch drops the stale dst after building the PMTU reply so that the
packet is routed using its new headers when it is reinjected.

The pmtu_ipv4_br_vxlan4_exception test generates PMTU exceptions by
pinging an IP on the other side of a tunnel. This was incorrect as it
would return upon the first ICMP Fragmentation Needed due to the -w flag
being used in conjunction with || return 1.

This patch updates pmtu_ipv4_br_vxlan4_exception to be in line with how
PMTU exceptions are generated in other tests such as in test_pmtu_ipvX

    run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1800 ${dst1}
    run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1800 ${dst2}

Signed-off-by: Laika Price <laikabcprice@gmail.com>
---
Changes in v3:
- Squashed the selftest update into the ip_tunnel fix so the patch remains
  bisectable.
- Link to v2: https://patch.msgid.link/20260613-master-v2-0-061b70fd45dd@gmail.com

Changes in v2:
- Fixed incorrect PMTU exception generation in the selftest.
- Link to v1: https://patch.msgid.link/20260613-master-v1-1-df796e8e2d74@gmail.com
---
 net/ipv4/ip_tunnel_core.c           | 2 ++
 tools/testing/selftests/net/pmtu.sh | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/net/ipv4/ip_tunnel_core.c b/net/ipv4/ip_tunnel_core.c
index d3c677e9b..949150e43 100644
--- a/net/ipv4/ip_tunnel_core.c
+++ b/net/ipv4/ip_tunnel_core.c
@@ -267,6 +267,7 @@ static int iptunnel_pmtud_build_icmp(struct sk_buff *skb, int mtu)
 
 	eth_header(skb, skb->dev, ntohs(eh.h_proto), eh.h_source, eh.h_dest, 0);
 	skb_reset_mac_header(skb);
+	skb_dst_drop(skb);
 
 	return skb->len;
 }
@@ -370,6 +371,7 @@ static int iptunnel_pmtud_build_icmpv6(struct sk_buff *skb, int mtu)
 
 	eth_header(skb, skb->dev, ntohs(eh.h_proto), eh.h_source, eh.h_dest, 0);
 	skb_reset_mac_header(skb);
+	skb_dst_drop(skb);
 
 	return skb->len;
 }
diff --git a/tools/testing/selftests/net/pmtu.sh b/tools/testing/selftests/net/pmtu.sh
index a3323c21f..9498d9f53 100755
--- a/tools/testing/selftests/net/pmtu.sh
+++ b/tools/testing/selftests/net/pmtu.sh
@@ -1456,8 +1456,8 @@ test_pmtu_ipvX_over_bridged_vxlanY_or_geneveY_exception() {
 	mtu "${ns_a}" ${type}_a $((${ll_mtu} + 1000))
 	mtu "${ns_b}" ${type}_b $((${ll_mtu} + 1000))
 
-	run_cmd ${ns_c} ${ping} -q -M want -i 0.1 -c 10 -s $((${ll_mtu} + 500)) ${dst} || return 1
-	run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1  -s $((${ll_mtu} + 500)) ${dst} || return 1
+	run_cmd ${ns_c} ${ping} -q -M want -i 0.1 -w 1 -s $((${ll_mtu} + 500)) ${dst}
+	run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s $((${ll_mtu} + 500)) ${dst}
 
 	# Check that exceptions were created
 	pmtu="$(route_get_dst_pmtu_from_exception "${ns_c}" ${dst})"

---
base-commit: 2a2974b5145cdf2f4db134be1a2157e9ca4a1cf0
change-id: 20260613-master-b749dfae5ecc

Best regards,
--  
Laika Price <laikabcprice@gmail.com>

Re: [PATCH net v3] ip_tunnel: drop stale dst from generated PMTU ICMP replies

[Jakub Kicinski]

On Sun, 14 Jun 2026 00:13:57 +0100 Laika Price via B4 Relay wrote:
> Changes in v3:
> - Squashed the selftest update into the ip_tunnel fix so the patch remains
>   bisectable.
> - Link to v2: https://patch.msgid.link/20260613-master-v2-0-061b70fd45dd@gmail.com
> 
> Changes in v2:
> - Fixed incorrect PMTU exception generation in the selftest.
> - Link to v1: https://patch.msgid.link/20260613-master-v1-1-df796e8e2d74@gmail.com

It's still failing the tests.
Please do not repost it again until someone guides you towards 
a correct fix.

Re: [PATCH net v3] ip_tunnel: drop stale dst from generated PMTU ICMP replies

[Ido Schimmel]

On Sun, Jun 14, 2026 at 12:13:57AM +0100, Laika Price via B4 Relay wrote:
> From: Laika Price <laikabcprice@gmail.com>
> 
> iptunnel_pmtud_build_icmp(...) and iptunnel_pmtud_build_icmpv6(...) take
> in an sk_buff, modify it to create a PMTU ICMP error reply, and return it.
> As part of these modifications, the source/destination ethernet and IP
> addresses are swapped around which makes the sk_buff's current dst invalid.
> 
> If the stale dst is left, the packet can skip input routing and be
> forwarded using the original output device. This was observed when sending
> packets to a VXLAN over a WireGuard tunnel - the ICMP reply was generated
> but it was sent over the VXLAN instead of to the WireGuard tunnel.
> 
> This patch drops the stale dst after building the PMTU reply so that the
> packet is routed using its new headers when it is reinjected.
> 
> The pmtu_ipv4_br_vxlan4_exception test generates PMTU exceptions by
> pinging an IP on the other side of a tunnel. This was incorrect as it
> would return upon the first ICMP Fragmentation Needed due to the -w flag
> being used in conjunction with || return 1.
> 
> This patch updates pmtu_ipv4_br_vxlan4_exception to be in line with how
> PMTU exceptions are generated in other tests such as in test_pmtu_ipvX
> 
>     run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1800 ${dst1}
>     run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1800 ${dst2}

1. Please split the selftest fix to a separate patch (patch #1), explain
why the test is currently passing and why it's going to break with the
subsequent code change.

2. Use the appropriate Fixes tag for each patch.

3. Go over this doc:

https://docs.kernel.org/process/maintainer-netdev.html

4. Use ingest_mdir.py to test your patches:

https://github.com/linux-netdev/nipa#running-locally

> 
> Signed-off-by: Laika Price <laikabcprice@gmail.com>
> ---
> Changes in v3:
> - Squashed the selftest update into the ip_tunnel fix so the patch remains
>   bisectable.
> - Link to v2: https://patch.msgid.link/20260613-master-v2-0-061b70fd45dd@gmail.com
> 
> Changes in v2:
> - Fixed incorrect PMTU exception generation in the selftest.
> - Link to v1: https://patch.msgid.link/20260613-master-v1-1-df796e8e2d74@gmail.com
> ---
>  net/ipv4/ip_tunnel_core.c           | 2 ++
>  tools/testing/selftests/net/pmtu.sh | 4 ++--
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/net/ipv4/ip_tunnel_core.c b/net/ipv4/ip_tunnel_core.c
> index d3c677e9b..949150e43 100644
> --- a/net/ipv4/ip_tunnel_core.c
> +++ b/net/ipv4/ip_tunnel_core.c
> @@ -267,6 +267,7 @@ static int iptunnel_pmtud_build_icmp(struct sk_buff *skb, int mtu)
>  
>  	eth_header(skb, skb->dev, ntohs(eh.h_proto), eh.h_source, eh.h_dest, 0);
>  	skb_reset_mac_header(skb);
> +	skb_dst_drop(skb);

This probably needs to be:

if (skb_valid_dst(skb))
	skb_dst_drop(skb);

Both VXLAN and GENEVE use the dst after skb_tunnel_check_pmtu() when in
external mode, so you can't drop it unconditionally. This shouldn't be a
problem because both IPv4 and IPv6 will resolve a new dst if the
current one isn't valid (i.e., it's a dst metadata one).

>  
>  	return skb->len;
>  }
> @@ -370,6 +371,7 @@ static int iptunnel_pmtud_build_icmpv6(struct sk_buff *skb, int mtu)
>  
>  	eth_header(skb, skb->dev, ntohs(eh.h_proto), eh.h_source, eh.h_dest, 0);
>  	skb_reset_mac_header(skb);
> +	skb_dst_drop(skb);
>  
>  	return skb->len;
>  }
> diff --git a/tools/testing/selftests/net/pmtu.sh b/tools/testing/selftests/net/pmtu.sh
> index a3323c21f..9498d9f53 100755
> --- a/tools/testing/selftests/net/pmtu.sh
> +++ b/tools/testing/selftests/net/pmtu.sh
> @@ -1456,8 +1456,8 @@ test_pmtu_ipvX_over_bridged_vxlanY_or_geneveY_exception() {
>  	mtu "${ns_a}" ${type}_a $((${ll_mtu} + 1000))
>  	mtu "${ns_b}" ${type}_b $((${ll_mtu} + 1000))
>  
> -	run_cmd ${ns_c} ${ping} -q -M want -i 0.1 -c 10 -s $((${ll_mtu} + 500)) ${dst} || return 1
> -	run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1  -s $((${ll_mtu} + 500)) ${dst} || return 1
> +	run_cmd ${ns_c} ${ping} -q -M want -i 0.1 -w 1 -s $((${ll_mtu} + 500)) ${dst}
> +	run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s $((${ll_mtu} + 500)) ${dst}
>  
>  	# Check that exceptions were created
>  	pmtu="$(route_get_dst_pmtu_from_exception "${ns_c}" ${dst})"
> 
> ---
> base-commit: 2a2974b5145cdf2f4db134be1a2157e9ca4a1cf0
> change-id: 20260613-master-b749dfae5ecc
> 
> Best regards,
> --  
> Laika Price <laikabcprice@gmail.com>
> 
> 

Re: [PATCH net v3] ip_tunnel: drop stale dst from generated PMTU ICMP replies

[Yaroslav Dudkov]

Hi

Since this series hasn't moved since June, and I've been debugging the
same code path independently, sharing what I found in the hope it
unblocks things. The kernel fix should of course remain Laika's;
consider this supporting data.

> 1. Please split the selftest fix to a separate patch (patch #1), explain
> why the test is currently passing and why it's going to break with the
> subsequent code change.

Below is that explanation (1-3), plus history (4) and a reproducer for
a July 2025 field report of what looks like the same regression.

In short: there are two independent defects. The stale dst in the
reply builder (what this series fixes), and a selftest stimulus that
can only pass while that reply is *not* delivered -- so any correct
kernel fix turns the bridged subtests red until the test is fixed
first. Everything below was measured on net at v7.2-7323-gf967455fb2a5,
under virtme-ng.

1. Selftest problem

In test_pmtu_ipvX_over_bridged_vxlanY_or_geneveY_exception():

    run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 ... || return 1

With a deadline (-w) set, iputils exits on the first socket error
(main_loop(): "if (rts->deadline && rts->nerrors) break;"), before any
echo reply can arrive, and finish() then returns non-zero because
nreceived is still 0. So this line fails exactly when the ICMP error
is delivered, and passes when the error dies in the kernel. The ns_c
line just above already uses -c 10 and survives the delivered error:
it reports "+1 errors" in its statistics yet exits 0.

2. Kernel side

When iptunnel_pmtud_build_icmp{,v6}() gets the skb it doesn't touch
the attached dst, so on reinjection skb_valid_dst() is true, input
routing is skipped, and the reply is dispatched through the stale
dst. What happens next depends on the dst:

- Locally generated traffic (a socket on the bridge host itself):
  the dst is an output route, whose ->input is still the
  dst_discard() stub from dst_init(). The reinjected PTB is freed
  silently. Verified with bpftrace on kfree_skb().

- Forwarded traffic (the WireGuard setup from the original report):
  same mechanism up to the ->input call, but here the dst is the
  forward route, so ->input is ip_forward and the reply is forwarded
  along the stale route -- out the original egress, into the VXLAN
  instead of back to the sender. That is exactly the reported
  symptom. Reproducer script for this case is attached at the end of
  the mail.

  In the reproducer every oversized DF packet is lost with zero
  errors on the sender's socket -- all feedback channels are closed.
  The router's own Frag Needed never fires because the forwarding
  path deliberately ignores learned path MTUs
  (net.ipv4.ip_forward_use_pmtu=0 by default), and the route
  exception written by skb_dst_update_pmtu_no_confirm() is readable
  only by local sockets for the same reason. Setting
  ip_forward_use_pmtu=1 takes {10 tx, 0 rx, 0 errors} to
  {10 tx, 7 rx, +1 error}, which pins the mechanism -- but it is
  not a fix: under default policy the generated PTB is the only
  feedback channel this sender class has, and the stale dst kills
  exactly that channel.

- IPv6 differs in shape, not outcome: the unicast branch of
  ip6_rt_init_dst() sets ->input = ip6_forward, for output and
  forward routes alike -- and ip6_rcv_core() takes IP6CB(skb)->iif
  from the stale dst's device, so the redirect precondition
  iif == oif holds by construction. Measured with an on-link stale
  route: forwarding=0 drops the PTB at the forwarding check;
  forwarding=1 emits a spurious ICMPv6 Redirect (target equal to
  the node's own address, PTB quoted inside) and then dies
  resolving that same address as a neighbour. With a gatewayed
  stale route it should misroute out the original egress like IPv4.
  Never delivered to the sender either way.

3. Why the tests are green today

- skb_dst_update_pmtu_no_confirm() runs before building the reply
  in skb_tunnel_check_pmtu(), so for a local sender the route
  exception is created whether or not the ICMP ever arrives -- and
  the test only asserts the exception's existence.

- With br_netfilter loaded, br_nf_pre_routing_finish() replaces the
  stale dst on the reinjected PTB (it re-enters through the vxlan
  port) with the bridge's fake rtable, which is dropped again before
  ip_rcv -- so the PTB is routed from scratch and delivered, and
  then (1) turns delivery into FAIL. Reproduced with a couple of
  commands:

    modprobe br_netfilter
    ./pmtu.sh

  All 16 bridged subtests fail on a vanilla kernel. A fixed kernel
  fails this recipe identically -- which is the point: the test has
  to be fixed in any scenario. The same happens in a plain
  sequential selftest run, because fcnal-test.sh loads br_netfilter
  and does not unload it.

4. History

The root cause is 8930424777e4 ("tunnels: Accept PACKET_HOST in
skb_tunnel_check_pmtu().") combined with 4cb47a8644cc ("tunnels:
PMTU discovery support for directly bridged IP packets").

4cb47a8644cc shipped the builder with

    if (!reply || skb->pkt_type == PACKET_HOST)
        return 0;

so replies were only generated for bridged-through L2 frames. Those
reach the builder with no dst -- they never visit the host's L3 --
so the missing dst invalidation was unreachable, and the selftest
(df40e39c0df0, same series) was written against that semantic: its
-w 1 line could not receive an error by design.

8930424777e4 (6.15-rc1, since backported to stable, at least
6.1.135 per the Debian report) removed the pkt_type check. From
that point on local and forwarded senders -- the first callers with
a live dst, attached by the host's own routing -- reach the
builder, the stale dst problem became reachable, and the selftest
assumption was silently invalidated.

This matches Debian bug #1108860 [1] (July 2025): the same
VXLAN-over-WireGuard breakage in production, bisected to
8930424777e4, revert confirmed to fix it. That thread ended with a
request for more details and, ideally, a self-contained reproducer,
and stalled -- the script below is meant to be that reproducer. The
two reports look like the same regression, so I'm Cc'ing the people
from that thread.

5. What I suggest

- Selftest fix first, as its own patch -- otherwise any correct
  kernel fix turns the bridged subtests red: switch the -w 1 line
  to -c 10, keeping "|| return 1". Measured on both a masked and a
  delivering kernel: without a deadline the first error does not
  terminate ping (it consumes -c budget instead), exceptions still
  get created, and the line exits 0 in both worlds. Unlike dropping
  the "|| return 1" guards (the v3 approach), this fixes the
  stimulus while the guards keep catching real breakage. Neither v1
  nor v3 carried Fixes tags; for this patch the lines come from
  df40e39c0df0 and their assumption was invalidated by
  8930424777e4, so dual Fixes tags would route the backport to
  every stable tree that has the latter. I have this patch ready
  and can send it right away -- it is independent of the kernel fix
  and passes on both kernels -- unless Laika prefers to fold it
  into the series. If I don't hear back either way in a week or so,
  I'll send it on its own with a Link: to this thread.

> This probably needs to be:
>
> if (skb_valid_dst(skb))
>	skb_dst_drop(skb);
>
> Both VXLAN and GENEVE use the dst after skb_tunnel_check_pmtu() when in
> external mode, so you can't drop it unconditionally.

- Can confirm this guarded variant works on both sides: the 16 OVS
  subtest failures from v1 were exactly this metadata issue, and
  with the guard all 16 pass on a guarded kernel; on the bridged
  side the reproducer below goes from 10/0 (no notification at all)
  to 10 tx / 8 rx / +1 error, the error being the tunnel's PTB
  finally reaching the sender. Happy to give Tested-by on a v4 with
  the guard. Fixes-wise: 4cb47a8644cc introduced the missing
  invalidation and 8930424777e4 made it reachable (and is what
  Debian bisected to) -- dual tags again, since prose in the commit
  message won't route stable backports. Given #1108860,
  Reported-by:/Closes: tags for the Debian report may be
  appropriate.

- The in-tree test cannot catch this bug class at all: it only
  asserts the exception, which the silent update always creates.
  The reproducer below is nearly a pmtu.sh subtest for the
  forwarded case (assert that the PTB actually reaches the sender);
  I can follow up with that as a separate patch.

[1] https://bugs.debian.org/1108860

------reproducer: forwarded sender, persistent PMTU blackhole------

for n in D A B; do ip netns del $n 2>/dev/null; done
ip netns add D; ip netns add A; ip netns add B

ip link add veth_D-A netns D type veth peer name veth_A-D netns A
ip -n D addr add 192.168.3.10/24 dev veth_D-A
ip -n A addr add 192.168.3.1/24  dev veth_A-D
ip -n D link set veth_D-A mtu 5000 up
ip -n A link set veth_A-D mtu 5000 up
ip -n D link set lo up; ip -n A link set lo up; ip -n B link set lo up

ip link add veth_A-B netns A type veth peer name veth_B-A netns B
ip -n A addr add 10.0.1.1/24 dev veth_A-B
ip -n B addr add 10.0.1.2/24 dev veth_B-A
ip -n A link set veth_A-B mtu 4000 up
ip -n B link set veth_B-A mtu 4000 up

ip -n A link add br0 type bridge
ip -n A addr add 192.168.2.1/24 dev br0
ip -n A link set br0 mtu 5000 up
ip -n A link add vxlan_a type vxlan id 1 local 10.0.1.1 \
	remote 10.0.1.2 dstport 4789 df set ttl 64
ip -n A link set vxlan_a mtu 5000 master br0 up
ip -n B link add vxlan_b type vxlan id 1 local 10.0.1.2 \
	remote 10.0.1.1 dstport 4789 df set ttl 64
ip -n B addr add 192.168.2.2/24 dev vxlan_b
ip -n B link set vxlan_b mtu 5000 up

ip -n B addr add 192.168.4.1/32 dev lo
ip -n A route add 192.168.4.1/32 via 192.168.2.2
ip -n D route add 192.168.2.0/24 via 192.168.3.1
ip -n D route add 192.168.4.1/32 via 192.168.3.1
ip -n B route add 192.168.3.0/24 via 192.168.2.1
ip netns exec A sysctl -qw net.ipv4.ip_forward=1

ip netns exec D ping -c 2 -s 56 192.168.4.1 >/dev/null
for n in D A B; do ip netns exec $n ip route flush cached; done
ip netns exec D ping -q -M want -i 0.1 -c 10 -s 4500 192.168.4.1 || true

  Result on an unfixed kernel, default sysctls: 10 transmitted, 0 received,
  no ICMP errors on the socket, and 10 misrouted PTBs inside the tunnel
  (ns_b IpInAddrErrors +10). Persistent: no packet ever heals the path.
  Mechanism check: sysctl -w net.ipv4.ip_forward_use_pmtu=1 in ns_a, flush
  caches, re-run -> 10/7/+1 error (the router's own Frag Needed takes over).
  On a fixed (guarded) kernel, default sysctls: 10/8/+1, the error being
  "From 192.168.4.1 icmp_seq=1 Frag needed and DF set (mtu = 3950)" --
  the tunnel's PTB finally reaching the sender.
-------------------------------------------------------------------

Thanks,
Yaroslav Dudkov

Re: [PATCH net v3] ip_tunnel: drop stale dst from generated PMTU ICMP replies

[Ido Schimmel]

On Tue, Sep 01, 2026 at 08:17:09AM +0000, Yaroslav Dudkov wrote:
> Since this series hasn't moved since June, and I've been debugging the
> same code path independently, sharing what I found in the hope it
> unblocks things. The kernel fix should of course remain Laika's;
> consider this supporting data.

I will send the kernel fix that I suggested and the selftest adjustment
as a single patch later today. Will add a Reported-by tag with your
name. Feel free to add a Tested-by tag later on.

Thanks