[PATCH net v3] vxlan: reset the remote's dst cache when snooping moves a MAC

[PATCH net v3] vxlan: reset the remote's dst cache when snooping moves a MAC

[Zhefu Peng]

vxlan_snoop() handles a known MAC that now arrives from another remote
by rewriting the remote's IP address in place. It leaves the remote's
dst_cache alone, so the next packet to that MAC is routed with the dst
cached for the previous remote. When the two remotes have different
next hops, the packet leaves the wrong way. For example, if the
previous remote was on-link and the new one is behind a router, the
device resolves the new remote's address on the local link, the
neighbour stays incomplete, and traffic to the MAC is blackholed until
the cached route is invalidated, such as by "ip route flush cache".

The same branch also leaves remote_ifindex unchanged, although the
early return compares it and vxlan_xmit_one() uses it as the output
interface. When a MAC moves between an IPv6 link-local and another
address, the remote keeps the previous interface, and every later
packet from that remote enters the migration branch again.

vxlan_fdb_replace(), which makes the same change through netlink,
already resets the cache, updates remote_ifindex and clears offloaded.
Do the same when snooping.

Reproduced with network namespaces: a learning VXLAN device has one
peer VTEP on-link and another behind a router, and an inner MAC moves
from the first peer to the second. Without this patch, pings to the MAC
fail after the move until the route cache is flushed: in 10 of 10 runs
with one CPU, and in 5 of 6 runs with two CPUs, where it depends on
whether the sending CPU holds the stale entry. With the patch, they
succeed in all 16 runs. In a second setup over IPv6, the MAC moves from
a peer with a global address to a peer with a link-local one. Without
this patch, the move and every later packet from the new peer each send
an RTM_NEWNEIGH notification, 33 in each run. With the patch, only the
move sends one.

This was found by our AI-driven automated live migration tests. The fix
and this changelog were written with the help of an AI coding assistant.

Fixes: 0c1d70af924b ("net: use dst_cache for vxlan device")
Fixes: 87613de95041 ("vxlan: fix snooping for link-local IPv6 addresses")
Signed-off-by: Zhefu Peng <ilskdw@gmail.com>
---
v3:
- use my real name in From: and Signed-off-by:

v2: https://lore.kernel.org/netdev/20261110153020.9635-1-ilskdw@gmail.com/
- also update remote_ifindex and clear offloaded, as vxlan_fdb_replace() does
- add the IPv6 link-local case to the changelog and a second Fixes tag

v1: https://lore.kernel.org/netdev/20261106144406.66035-1-ilskdw@gmail.com/

 drivers/net/vxlan/vxlan_core.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c
index d11028b37..6bc54642e 100644
--- a/drivers/net/vxlan/vxlan_core.c
+++ b/drivers/net/vxlan/vxlan_core.c
@@ -1482,7 +1482,10 @@ static enum skb_drop_reason vxlan_snoop(struct net_device *dev,
 				    "%pM migrated from %pIS to %pIS\n",
 				    src_mac, &rdst->remote_ip.sa, &src_ip->sa);
 
+		dst_cache_reset(&rdst->dst_cache);
 		rdst->remote_ip = *src_ip;
+		rdst->remote_ifindex = ifindex;
+		rdst->offloaded = false;
 		vxlan_fdb_notify(vxlan, f, rdst, RTM_NEWNEIGH, true, NULL);
 	} else {
 		/* learned new entry */
-- 
2.47.3

Re: [PATCH net v3] vxlan: reset the remote's dst cache when snooping moves a MAC

[Nicolas Dichtel]

Le 11/11/2026 à 19:00, Zhefu Peng a écrit :
> vxlan_snoop() handles a known MAC that now arrives from another remote
> by rewriting the remote's IP address in place. It leaves the remote's
> dst_cache alone, so the next packet to that MAC is routed with the dst
> cached for the previous remote. When the two remotes have different
> next hops, the packet leaves the wrong way. For example, if the
> previous remote was on-link and the new one is behind a router, the
> device resolves the new remote's address on the local link, the
> neighbour stays incomplete, and traffic to the MAC is blackholed until
> the cached route is invalidated, such as by "ip route flush cache".
> 
> The same branch also leaves remote_ifindex unchanged, although the
> early return compares it and vxlan_xmit_one() uses it as the output
> interface. When a MAC moves between an IPv6 link-local and another
> address, the remote keeps the previous interface, and every later
> packet from that remote enters the migration branch again.
> 
> vxlan_fdb_replace(), which makes the same change through netlink,
> already resets the cache, updates remote_ifindex and clears offloaded.
> Do the same when snooping.
> 
> Reproduced with network namespaces: a learning VXLAN device has one
> peer VTEP on-link and another behind a router, and an inner MAC moves
> from the first peer to the second. Without this patch, pings to the MAC
> fail after the move until the route cache is flushed: in 10 of 10 runs
> with one CPU, and in 5 of 6 runs with two CPUs, where it depends on
> whether the sending CPU holds the stale entry. With the patch, they
> succeed in all 16 runs. In a second setup over IPv6, the MAC moves from
> a peer with a global address to a peer with a link-local one. Without
> this patch, the move and every later packet from the new peer each send
> an RTM_NEWNEIGH notification, 33 in each run. With the patch, only the
> move sends one.
Maybe a selftest could be added to check this scenario?

Re: [PATCH net v3] vxlan: reset the remote's dst cache when snooping moves a MAC

[Zhefu Peng]

On 13/11/2026 16:05, Nicolas Dichtel wrote:
> Maybe a selftest could be added to check this scenario?

Sure. v4 will add one under tools/testing/selftests/net that covers
both cases: the IPv4 move behind a router and the IPv6 link-local move.

pw-bot: cr

Re: [PATCH net v3] vxlan: reset the remote's dst cache when snooping moves a MAC

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 4 potential
issue(s) to consider.

Critical: 0 · High: 0 · Medium: 3 · Low: 1

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [Medium] In vxlan_snoop() (drivers/net/vxlan/vxlan_core.c), the patch
  writes the snoop-derived `ifindex` into rdst->remote_ifindex (line…
- [Medium] The stale-dst_cache bug the patch claims to fix can still
  happen through a race with a concurrent transmitter.
- [Medium] vxlan_snoop() now clears rdst->offloaded
  (drivers/net/vxlan/vxlan_core.c line 1488) right before…
- [Low] vxlan_snoop() (drivers/net/vxlan/vxlan_core.c, lines 1485-1488)
  now writes rdst->remote_ifindex and rdst->offloaded from RX softirq…

--

vxlan: reset the remote's dst cache when snooping moves a MAC

When vxlan_snoop() migrates a learned FDB entry to a new remote, this patch
resets the remote's dst_cache, sets remote_ifindex to the snooped ifindex
and clears the offloaded flag, the same way vxlan_fdb_replace() does. The
goal is to stop traffic being routed through a stale cached dst after a MAC
move, and to stop repeated migrations for IPv6 link-local remotes.

> diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c
> index d11028b374721..6bc54642e7f45 100644
> --- a/drivers/net/vxlan/vxlan_core.c
> +++ b/drivers/net/vxlan/vxlan_core.c
> @@ -1482,7 +1482,10 @@ static enum skb_drop_reason vxlan_snoop(struct net_device *dev,
>  				    "%pM migrated from %pIS to %pIS\n",
>  				    src_mac, &rdst->remote_ip.sa, &src_ip->sa);
>  
> +		dst_cache_reset(&rdst->dst_cache);
>  		rdst->remote_ip = *src_ip;

[Severity: Medium]
Can a concurrent transmitter still put the old remote's dst back into the
cache after this reset?

vxlan_xmit_one() reads the remote without any synchronization:

vxlan_xmit_one() {
    ...
	ifindex = rdst->remote_ifindex;

	if (addr_family == AF_INET) {
		key.u.ipv4.src = cfg->saddr.sin.sin_addr.s_addr;
		key.u.ipv4.dst = rdst->remote_ip.sin.sin_addr.s_addr;
    ...
	dst_cache = &rdst->dst_cache;
    ...
}

On a miss, dst_cache_per_cpu_get() sets refresh_ts when the lookup
happens, not when the new dst is stored:

fail:
	idst->refresh_ts = jiffies;
	return NULL;

The entry then counts as valid as long as refresh_ts is after reset_ts.
So this sequence seems possible:

CPU1 (TX)                          CPU0 (vxlan_snoop)
reads old remote_ip/ifindex
                                   dst_cache_reset()   /* reset_ts = J1 */
                                   rdst->remote_ip = *src_ip;
                                   rdst->remote_ifindex = ifindex;
<jiffies ticks>
udp_tunnel_dst_lookup()
  dst_cache_get_ip4() misses, refresh_ts = J2 > J1
  routes to the old remote
  dst_cache_set_ip4() stores the old remote's dst

After this, packets sent on CPU1 carry the new remote_ip but use the old
remote's dst. That is the same blackhole the changelog describes, and it
lasts until the dst goes obsolete or the cache is flushed.

The plain stores also let the TX side read a remote_ip from one remote and
a remote_ifindex from the other.

vxlan_fdb_replace() has the same window. Does this patch fully close the
case it targets?

> +		rdst->remote_ifindex = ifindex;

[Severity: Medium]
Does this clear an egress interface that user space configured for a
remote that is not link-local?

In vxlan_snoop(), ifindex is non-zero only for IPv6 link-local sources:

	if (src_ip->sa.sa_family == AF_INET6 &&
	    (ipv6_addr_type(&src_ip->sin6.sin6_addr) & IPV6_ADDR_LINKLOCAL))
		ifindex = src_ifindex;

However, vxlan_fdb_parse() accepts NDA_IFINDEX for any remote address
family, and vxlan_fdb_add() accepts dynamic (NUD_REACHABLE) entries.

Take a dynamic entry added with "via <dev>" for an IPv4 or global IPv6
remote A, on a device with learning enabled. The first packet from A then
fails the early return:

	if (likely(vxlan_addr_equal(&rdst->remote_ip, src_ip) &&
		   rdst->remote_ifindex == ifindex))
		return SKB_NOT_DROPPED_YET;

It fails because remote_ifindex holds the user's value while ifindex is 0.
The entry is not static, so the code logs "migrated from A to A" and this
new line sets remote_ifindex to 0.

>From then on, vxlan_xmit_one() sees ifindex == 0 and falls back to the
socket's bound device for the route lookup:

	if (!ifindex)
		ifindex = sock4->sk->sk_bound_dev_if;

RTM_NEWNEIGH also stops reporting NDA_IFINDEX for this entry.

Before this patch, this case sent a notification on every packet but kept
the configured interface. Could remote_ifindex be compared and overwritten
only when the old or the new remote is IPv6 link-local?

> +		rdst->offloaded = false;

[Severity: Medium]
Can clearing offloaded here leave a stale entry in the mlxsw hardware
FDB?

At this point the ASIC may still have a tunnel FDB entry for this MAC that
points at the old remote. vxlan_fdb_switchdev_notifier_info() copies the
flag into every switchdev notification:

	fdb_info->offloaded = rd->offloaded;

mlxsw_sp_switchdev_vxlan_fdb_del() skips the hardware removal when that
copy is false:

	vxlan_fdb_info = &switchdev_work->vxlan_fdb_info;
	if (!vxlan_fdb_info->offloaded)
		return;

The new remote is programmed later, from deferred work in
mlxsw_sp_switchdev_vxlan_fdb_add(). That function can return without
setting offloaded back to true, either when br_fdb_find_port() does not
return the vxlan device or when mlxsw_sp_port_fdb_tunnel_uc_op() fails.

There is a second case. If the vxlan FDB entry is deleted after the snoop
but before the ADD work runs, the DEL captures offloaded == false. The ADD
then programs the hardware, and vxlan_fdb_offloaded_set() finds no entry
to mark.

In both cases a later DEL skips the hardware removal. The ASIC then keeps a
static MAC to VTEP entry that software no longer tracks, until FID/NVE
teardown. Before this patch the snoop path left offloaded set, so a later
DEL removed the hardware entry by MAC/FID.

vxlan_fdb_replace() and new entry creation follow the same pattern. This
patch adds a trigger driven by received packets, and mlxsw supports
VXLAN_F_LEARN for an IPv4 underlay.

>  		vxlan_fdb_notify(vxlan, f, rdst, RTM_NEWNEIGH, true, NULL);

[Severity: Low]
A related question about locking: this branch runs in the RX softirq
without vxlan->hash_lock. The other code that writes remote_ifindex and
offloaded holds that lock: vxlan_fdb_offloaded_set(), vxlan_fdb_replace()
through vxlan_fdb_update_existing(), and vxlan_fdb_clear_offload().

Can this race with vxlan_fdb_offloaded_set()? Under hash_lock it looks up
the rdst using the old (remote_ip, port, vni, remote_ifindex) tuple:

	rdst = vxlan_fdb_find_rdst(f, &fdb_info->remote_ip,
				   fdb_info->remote_port,
				   fdb_info->remote_vni,
				   fdb_info->remote_ifindex);
	if (!rdst)
		goto out;

	rdst->offloaded = fdb_info->offloaded;

If the snoop rewrites the tuple and clears offloaded between that lookup
and the store, the clear is lost. The new remote then shows as offloaded.

A concurrent "bridge fdb replace" is similar. If vxlan_fdb_replace() runs
after the snoop has passed the f->state check, the rdst can end up with
remote_ip from one writer and remote_ifindex from the other. If the
replace made the entry static, the snoop path never touches it again and
the mixed state stays.

The unlocked remote_ip store was there before this patch. This patch adds
two more fields to it, while the vxlan_fdb_replace() code it copies runs
under hash_lock.

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261111180001.2475-1-ilskdw%40gmail.com