Re: [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops

[Ido Schimmel <idosch@nvidia.com>]

On Tue, Jul 14, 2026 at 03:43:09AM +0000, Jack Ma wrote:
> On Mon, Jul 13, 2026 at 03:23:39PM +0300, Ido Schimmel wrote:
> > > Some deployments pack several receivers behind a single underlay IP and tell
> > > them apart by UDP destination port.  To spread flows across such receivers they
> >
> > I don't understand the "tell them apart" phrasing. Aren't all of these
> > receivers interchangeable given you are load balancing between them?
> 
> Sorry, that wording conflated two separate things. The load-balancing
> targets are interchangeable; the UDP port is not what makes them
> distinct services, it is how the underlay addresses each one:
> 
>   - Addressing: several VTEPs are packed behind one underlay IP, each
>     reachable at a distinct UDP destination port. The port is the
>     underlay demux that delivers an encapsulated packet to the right
>     VTEP -- it selects *where*, not *which service*.
>   - Load-balancing: when those VTEPs form an HA set (shared inner
>     MAC/IP), a sender spreads flows across them with an fdb nexthop
>     group. The legs are interchangeable, but each one lives at a
>     different (underlay IP, UDP port).
> 
> So the legs are interchangeable endpoints that happen to sit at
> different (IP, port) addresses. A group can already carry a distinct IP
> per leg but not a distinct port, which is the gap here. I'll reword the
> cover and patch 1 to say this explicitly.
> 
> > I think you will need to provide more details about the use case [...]
> > In your use case, why can't the load balancing happen at the target
> > host (e.g., using XDP / TC / SO_REUSEPORT / flow classification)?
> 
> I think the key difference is that there isn't really a shared
> host-side datapath to load-balance in, and the UDP port is doing demux
> rather than load-balancing. The shape is a bit unusual, so maybe a
> couple of pictures help.
> 
> First, how forwarders sit on a receiver node. A forwarder here is
> just the pod that terminates the overlay tunnel (it hosts the VXLAN
> VTEP) and relays traffic to and from one tenant's workload. Many such
> pods share one mesh-routable underlay IP, each demuxed by UDP port:
> 
>   receiver node -- one mesh-routable underlay IP (NodeIP_A)
>   +----------------------------------------------------+
>   |  host netns: stateless outer-UDP demux by dst port |
>   |              (host does NOT terminate the tunnel)  |
>   |                                                    |
>   |     dst :40000        dst :40001        dst :40002 |
>   |         |                 |                 |      |
>   |   +-----v----+      +-----v----+      +-----v----+ |
>   |   | pod0 ns  |      | pod1 ns  |      | pod2 ns  | |
>   |   | vxlan    |      | vxlan    |      | vxlan    | |
>   |   | VTEP     |      | VTEP     |      | VTEP     | |
>   |   | decap    |      | decap    |      | decap    | |
>   |   +----------+      +----------+      +----------+ |
>   +----------------------------------------------------+
>   (up to ~10 forwarder pods packed per node)
> 
>   These packed pods are unrelated -- different tenants on different
>   VNIs (below) -- so the per-pod UDP port is node-level demux, not an
>   HA construct. A single forwarder's HA replicas are the orthogonal
>   axis: anti-affinity spreads them across nodes (never co-resident),
>   so one member's nexthop group has its legs on distinct node IPs.
>   But each leg is still reachable only at (node IP, that pod's UDP
>   port), so within one group the legs differ in IP *and* port. A
>   group can already carry a distinct IP per leg, but it takes the UDP
>   port from the device (a single value), so it can't send each leg to
>   its own port. That last part is really the gap we keep running into.
> 
> The pods sharing a node need not be related: each one belongs to a
> separate tenant on its own VXLAN VNI, and a node can pack forwarders
> for many of them, each on its own UDP port. That per-pod port is what
> lets different tenants co-locate without colliding, and the host --
> which only demuxes outer UDP -- never has to reason about tenancy.

It's unclear to me why you need the UDP port based demux. On a single
node, do you have multiple pods that belong to the same tenant and
therefore share the same VNI? From the above, my understanding is that
the answer is "no".

Assuming the answer is "no", instead of having multiple VXLAN sockets in
the host netns (each bound to a different port), why not have a single
VXLAN socket bound to the standard port (4789) and let the VXLAN driver
perform the demux based on the VNI? You can achieve that by creating
multiple VXLAN devices in the host netns that only differ in their VNI
and then move each to the relevant pod netns.

Given the anti-affinity constraint, you will only need to load balance
between pods on different nodes that use the standard port and that is
already supported by the FDB nexthop groups.