Re: VXLAN FDB nexthop groups: per-nexthop UDP dst port / VNI feasibility

[Ido Schimmel <idosch@nvidia.com>]

Please use plain text instead of HTML. Otherwise your emails won't make
it to the list.

On Wed, Jul 08, 2026 at 05:52:05AM +0000, Jack Ma wrote:
>    Hi David, Ido, Roopa,
>    I am using VXLAN FDB nexthop groups (ip nexthop add ... fdb, then
>    bridge fdb add ... nhid) to do L2 ECMP across a set of HA endpoints.
>    In my design each leg is a distinct (VTEP IP, UDP destination port)
>    pair: several endpoints live behind the same node IP and are
>    disambiguated by the VXLAN UDP destination port.
>    Reading the current code (v6.12), it looks like an FDB nexthop can
>    only carry a gateway IP, so a per-leg port cannot be expressed:
>      * vxlan_fdb_nh_path_select() copies only nhc_gw into the synthetic
>        rdst; remote_port and remote_vni stay zero
>        (include/net/vxlan.h).
>      * vxlan_xmit_one() then falls back to the device-wide port:
>        dst_port = rdst->remote_port ? rdst->remote_port
>                                     : vxlan->cfg.dst_port
>        (drivers/net/vxlan/vxlan_core.c).
>      * rtm_to_nh_config() rejects NHA_ENCAP and NHA_OIF on fdb nexthops
>        ("Fdb attribute can not be used with encap, oif or blackhole",
>        net/ipv4/nexthop.c).
>      * vxlan_fdb_parse() rejects NDA_PORT / NDA_VNI / NDA_IFINDEX when
>        NDA_NH_ID is present ("DST, VNI, ifindex and port are mutually
>        exclusive with NH_ID").
>    The original commits (38428d68719c "nexthop: support for fdb ecmp
>    nexthops" and 1274e1cc4226 "vxlan: ecmp support for mac fdb entries")
>    say "these nexthops only have ip", which matches the EVPN multihoming
>    use case where every VTEP in the segment shares one UDP destination
>    port.
>    My questions:
>      1. Was IP-only a deliberate scoping decision for the EVPN-MH use
>         case, or is there a deeper reason a per-nexthop UDP destination
>         port (and VNI) could not be carried on an fdb nexthop, for
>         example via NHA_ENCAP / lwtunnel, which route nexthops already
>         accept?

IP-only was specific for EVPN-MH and it simply reuses the gateway IP
that is already present in the nexthop structure.

>      2. If legs genuinely need distinct UDP destination ports, is the
>         per-remote rdst path (bridge fdb ... dst ... port ...) the
>         intended mechanism, with the understanding that it gives
>         head-end replication semantics rather than nexthop-object ECMP?

Yes, but you will get packet replication instead of load balancing,
which I believe is not what you want.

>      3. Would you be open in principle to a patch that lets an fdb
>         nexthop carry a per-nexthop dst port (and VNI)? If so I am happy
>         to prototype it as an RFC.

I have some questions for you before I answer yours.

Can you explain why different VTEPs need to share a single IP? With
different IPs you can simply reuse the existing FDB nexthops without any
changes.

In addition, why disambiguate by the VXLAN UDP destination port and not
VNI? Even if we allow attaching tunnel encapsulation to FDB nexthops via
NHA_ENCAP, there does not seem to be uAPI in place to encode the
destination port (see ip_tun_policy).

Assuming you can live with a fixed port, I think you can achieve what
you want with BPF and no kernel changes. You can call
bpf_skb_set_tunnel_key [1] from a TC bpf program attached to the egress
of the VXLAN device (configured in "external" mode) and have it encode
the relevant VTEP based on the calculated hash.

[1] https://docs.ebpf.io/linux/helper-function/bpf_skb_set_tunnel_key/