Re: [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops

[Jack Ma <jack4it@gmail.com>]

On Tue, Jul 14, 2026 at 08:42:01PM +0300, Ido Schimmel wrote:
> It's unclear to me why you need the UDP port based demux. On a single
> node, do you have multiple pods that belong to the same tenant and
> therefore share the same VNI? From the above, my understanding is that
> the answer is "no".

Good question, and I think my earlier framing sent you down the wrong
path, sorry about that. The honest answer is actually "yes". By design
a node can hold several VTEPs on the same VNI, as long as they belong
to different HA sets. Anti-affinity only keeps the two replicas within
one HA set off the same node. Two pods from different HA sets on the
same VNI are free to co-locate, and that co-location is the case that
needs a per-pod port.

> Assuming the answer is "no", instead of having multiple VXLAN sockets
> in the host netns (each bound to a different port), why not have a
> single VXLAN socket bound to the standard port (4789) and let the
> VXLAN driver perform the demux based on the VNI? [...] creating
> multiple VXLAN devices in the host netns that only differ in their
> VNI and then move each to the relevant pod netns.

For different-VNI pods that is a clean fit, and I want to try it out
regardless. The outer packet still carries the VNI, so a shared socket
can steer by VNI without decapsulating, and the host never touches the
tenant frame.

Where it stops working for me is two same-VNI pods on one node. They
share the underlay IP, the port (4789), and the VNI, so the outer
header is the same for both:

  outer: NodeIP:4789, VNI=X  ->  pod A (HA set 1)
  outer: NodeIP:4789, VNI=X  ->  pod B (HA set 2)

There is nothing left in the outer header to steer on, so the host
would have to look past it and terminate the tunnel itself, which is
the VTEP role I would rather keep inside the pod. A distinct port per
pod puts the distinguishing bit back in the outer header, so the host
can stay a stateless outer-UDP forwarder.

> Given the anti-affinity constraint, you will only need to load balance
> between pods on different nodes that use the standard port and that is
> already supported by the FDB nexthop groups.

Agreed that the across-node load balancing already works. What the
VNI-demux version costs is packing density. With one shared underlay
IP per node, VNI demux allows at most one VTEP per (VNI, node), so N
same-VNI HA sets of two replicas need 2N nodes. With a per-pod port
the same sets fit on two nodes (replica 1 on node A, replica 2 on node
B, each pod on its own port), and anti-affinity still holds. For 5
sets on one VNI that is 10 nodes versus 2.

So the port is not strictly required, and your VNI-demux model is a
real option I will prototype. The tradeoff I am weighing is density
plus keeping the host out of the tenant datapath, against the uAPI
cost, which is small: NHA_FDB_PORT is the nexthop analog of the
per-entry NDA_PORT that already exists, it is control-plane only, and
a leg with no port behaves exactly as today.

Happy to keep digging if it is useful. Thanks again for thinking it
through with me.