Re: [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops

[Jack Ma <jack4it@gmail.com>]

On Mon, Jul 13, 2026 at 03:23:39PM +0300, Ido Schimmel wrote:
> > Some deployments pack several receivers behind a single underlay IP and tell
> > them apart by UDP destination port.  To spread flows across such receivers they
>
> I don't understand the "tell them apart" phrasing. Aren't all of these
> receivers interchangeable given you are load balancing between them?

Sorry, that wording conflated two separate things. The load-balancing
targets are interchangeable; the UDP port is not what makes them
distinct services, it is how the underlay addresses each one:

  - Addressing: several VTEPs are packed behind one underlay IP, each
    reachable at a distinct UDP destination port. The port is the
    underlay demux that delivers an encapsulated packet to the right
    VTEP -- it selects *where*, not *which service*.
  - Load-balancing: when those VTEPs form an HA set (shared inner
    MAC/IP), a sender spreads flows across them with an fdb nexthop
    group. The legs are interchangeable, but each one lives at a
    different (underlay IP, UDP port).

So the legs are interchangeable endpoints that happen to sit at
different (IP, port) addresses. A group can already carry a distinct IP
per leg but not a distinct port, which is the gap here. I'll reword the
cover and patch 1 to say this explicitly.

> I think you will need to provide more details about the use case [...]
> In your use case, why can't the load balancing happen at the target
> host (e.g., using XDP / TC / SO_REUSEPORT / flow classification)?

I think the key difference is that there isn't really a shared
host-side datapath to load-balance in, and the UDP port is doing demux
rather than load-balancing. The shape is a bit unusual, so maybe a
couple of pictures help.

First, how forwarders sit on a receiver node. A forwarder here is
just the pod that terminates the overlay tunnel (it hosts the VXLAN
VTEP) and relays traffic to and from one tenant's workload. Many such
pods share one mesh-routable underlay IP, each demuxed by UDP port:

  receiver node -- one mesh-routable underlay IP (NodeIP_A)
  +----------------------------------------------------+
  |  host netns: stateless outer-UDP demux by dst port |
  |              (host does NOT terminate the tunnel)  |
  |                                                    |
  |     dst :40000        dst :40001        dst :40002 |
  |         |                 |                 |      |
  |   +-----v----+      +-----v----+      +-----v----+ |
  |   | pod0 ns  |      | pod1 ns  |      | pod2 ns  | |
  |   | vxlan    |      | vxlan    |      | vxlan    | |
  |   | VTEP     |      | VTEP     |      | VTEP     | |
  |   | decap    |      | decap    |      | decap    | |
  |   +----------+      +----------+      +----------+ |
  +----------------------------------------------------+
  (up to ~10 forwarder pods packed per node)

  These packed pods are unrelated -- different tenants on different
  VNIs (below) -- so the per-pod UDP port is node-level demux, not an
  HA construct. A single forwarder's HA replicas are the orthogonal
  axis: anti-affinity spreads them across nodes (never co-resident),
  so one member's nexthop group has its legs on distinct node IPs.
  But each leg is still reachable only at (node IP, that pod's UDP
  port), so within one group the legs differ in IP *and* port. A
  group can already carry a distinct IP per leg, but it takes the UDP
  port from the device (a single value), so it can't send each leg to
  its own port. That last part is really the gap we keep running into.

The pods sharing a node need not be related: each one belongs to a
separate tenant on its own VXLAN VNI, and a node can pack forwarders
for many of them, each on its own UDP port. That per-pod port is what
lets different tenants co-locate without colliding, and the host --
which only demuxes outer UDP -- never has to reason about tenancy.

Zooming into one forwarder pod, there isn't really anything to
load-balance on the host: the tunnel terminates on a vxlan device
inside the pod's own netns, and the pod reaches the customer through a
separate NIC:

  one forwarder pod -- its own netns, tenant VNI X
  +-------------------------------------------------+
  |                                                 |
  |   on/off-ramp NIC   <--- customer data plane    |
  |   |   on-ramp (ingress) / off-ramp (egress)     |
  |   |   inner packet                              |
  |   vxlan (VTEP)   encap / decap for VNI X,       |
  |   |              listens on this pod's UDP port |
  |   |   outer VXLAN UDP                           |
  |   eth0 (underlay)   NodeIP:port                 |
  |   |   to peer VTEPs over the                    |
  |   v   mesh underlay                             |
  |                                                 |
  +-------------------------------------------------+

The load-balancing already happens at the sender -- the fdb nexthop
group hashes the inner flow and picks a leg. The only missing piece is
that the selected leg can't carry its own UDP port, so the sender
reaches the right pod's IP but always the device's port.

The host-side alternatives seem to assume a shared datapath that isn't
really there in this design:

  - SO_REUSEPORT balances sockets within a single netns; here the
    receivers are in different netns -- in fact different tenants --
    so there isn't a shared host socket to reuse.
  - An XDP/TC host fan-out would mean the host has to become a VTEP --
    terminate the tunnel and re-dispatch inner traffic across netns
    and tenant/VNI boundaries -- which puts the host into the tenant
    data path (today it only does a stateless outer-UDP demux) and
    largely duplicates what an fdb nexthop group does: hashing an
    inner flow across a set of endpoints. The group can already vary
    the leg's IP; the port is the part it can't.

On the uAPI cost: I don't think this adds a new datapath concept. A
single fdb entry already carries a per-destination UDP port (NDA_PORT),
and vxlan_xmit_one() already prefers rdst->remote_port when it's set.
NHA_FDB_PORT is meant to be the nexthop analog of that existing
attribute: control-plane only, no datapath change, and backward
compatible (a leg with no port falls back to the device port as today).

Happy to go into the deployment/addressing constraints in more detail if
that would help.

Thanks for the review.