VXLAN FDB nexthop groups: per-nexthop UDP dst port / VNI feasibility

VXLAN FDB nexthop groups: per-nexthop UDP dst port / VNI feasibility

[Jack Ma]

Hi David, Ido, Roopa,

I am using VXLAN FDB nexthop groups ( ip nexthop add ... fdb ,
followed by  bridge fdb add ... nhid ) to implement L2 ECMP across
a set of HA endpoints.

In my design, each leg is a distinct (VTEP IP, UDP destination port)
pair. Several endpoints live behind the same node IP and are
disambiguated by the VXLAN UDP destination port.

Reading the current code (v6.12), it looks like an FDB nexthop can
carry only a gateway IP, so a per-leg port cannot be expressed:

•  vxlan_fdb_nh_path_select()  copies only  nhc_gw  into the synthetic
 rdst ;  remote_port  and  remote_vni  remain zero
( include/net/vxlan.h ).
•  vxlan_xmit_one()  then falls back to the device-wide port:
 dst_port = rdst->remote_port ? rdst->remote_port : vxlan->cfg.dst_port 
( drivers/net/vxlan/vxlan_core.c ).
•  rtm_to_nh_config()  rejects  NHA_ENCAP  and  NHA_OIF  on FDB
nexthops:
 Fdb attribute can not be used with encap, oif or blackhole 
( net/ipv4/nexthop.c ).
•  vxlan_fdb_parse()  rejects  NDA_PORT ,  NDA_VNI , and  NDA_IFINDEX 
when  NDA_NH_ID  is present:
 DST, VNI, ifindex and port are mutually exclusive with NH_ID 

The original commits, 38428d68719c ("nexthop: support for fdb ecmp
nexthops") and 1274e1cc4226 ("vxlan: ecmp support for mac fdb
entries"), say "these nexthops only have ip." This matches the EVPN
multihoming use case, where every VTEP in the segment shares one UDP
destination port.

My questions:

1. Was IP-only a deliberate scoping decision for the EVPN-MH use
case, or is there a deeper reason why a per-nexthop UDP destination
port and VNI could not be carried on an FDB nexthop, for example
through  NHA_ENCAP  or lwtunnel, which route nexthops already
accept?
2. If legs genuinely need distinct UDP destination ports, is the
per-remote  rdst  path ( bridge fdb ... dst ... port ... ) the
intended mechanism, with the understanding that it provides
head-end replication semantics rather than nexthop-object ECMP?
3. Would you be open in principle to a patch that lets an FDB
nexthop carry a per-nexthop destination port and VNI? If so, I
would be happy to prototype it as an RFC.

Thanks very much for any pointers.

Best regards,
-Jack

Re: VXLAN FDB nexthop groups: per-nexthop UDP dst port / VNI feasibility

[David Ahern]

On 7/9/26 11:30 PM, Jack Ma wrote:
> Hi David, Ido, Roopa,
> 
> I am using VXLAN FDB nexthop groups ( ip nexthop add ... fdb ,
> followed by  bridge fdb add ... nhid ) to implement L2 ECMP across
> a set of HA endpoints.
> 
> In my design, each leg is a distinct (VTEP IP, UDP destination port)
> pair. Several endpoints live behind the same node IP and are
> disambiguated by the VXLAN UDP destination port.
> 
> Reading the current code (v6.12), it looks like an FDB nexthop can
> carry only a gateway IP, so a per-leg port cannot be expressed:
> 
> •  vxlan_fdb_nh_path_select()  copies only  nhc_gw  into the synthetic
>  rdst ;  remote_port  and  remote_vni  remain zero
> ( include/net/vxlan.h ).
> •  vxlan_xmit_one()  then falls back to the device-wide port:
>  dst_port = rdst->remote_port ? rdst->remote_port : vxlan->cfg.dst_port 
> ( drivers/net/vxlan/vxlan_core.c ).
> •  rtm_to_nh_config()  rejects  NHA_ENCAP  and  NHA_OIF  on FDB
> nexthops:
>  Fdb attribute can not be used with encap, oif or blackhole 
> ( net/ipv4/nexthop.c ).
> •  vxlan_fdb_parse()  rejects  NDA_PORT ,  NDA_VNI , and  NDA_IFINDEX 
> when  NDA_NH_ID  is present:
>  DST, VNI, ifindex and port are mutually exclusive with NH_ID 
> 
> The original commits, 38428d68719c ("nexthop: support for fdb ecmp
> nexthops") and 1274e1cc4226 ("vxlan: ecmp support for mac fdb
> entries"), say "these nexthops only have ip." This matches the EVPN
> multihoming use case, where every VTEP in the segment shares one UDP
> destination port.

The first commit was 6 years ago; those brain cells have been recycled.
:-) I suspect there was no reason to consider different UDP port per
leg. Roopa: does that sound right?


> 
> My questions:
> 
> 1. Was IP-only a deliberate scoping decision for the EVPN-MH use
> case, or is there a deeper reason why a per-nexthop UDP destination
> port and VNI could not be carried on an FDB nexthop, for example
> through  NHA_ENCAP  or lwtunnel, which route nexthops already
> accept?
> 2. If legs genuinely need distinct UDP destination ports, is the
> per-remote  rdst  path ( bridge fdb ... dst ... port ... ) the
> intended mechanism, with the understanding that it provides
> head-end replication semantics rather than nexthop-object ECMP?
> 3. Would you be open in principle to a patch that lets an FDB
> nexthop carry a per-nexthop destination port and VNI? If so, I
> would be happy to prototype it as an RFC.
> 

It seems reasonable to me.

Re: VXLAN FDB nexthop groups: per-nexthop UDP dst port / VNI feasibility

[Ido Schimmel]

On Fri, Jul 10, 2026 at 05:30:40AM +0000, Jack Ma wrote:
> Thanks very much for any pointers.

I already commented here:

https://lore.kernel.org/netdev/20260709102247.GA1536940@shredder/

Re: VXLAN FDB nexthop groups: per-nexthop UDP dst port / VNI feasibility

[Jack Ma]

Thank you, Ido, for the comments. I'm new to this side of the work, so please excuse my plain-text email handling!

I have submitted the patch here: https://lore.kernel.org/netdev/20260712191218.236-1-jack4it@gmail.com/. I also switched to my Gmail address for easier git and email workflow. Thanks to David as well!

To give some context on my approach: I am implementing a multi-tenant VXLAN overlay where we pack multiple VTEPs onto a shared pool of machines. Because we can't assign a directly routable IP to each individual VTEP, I am trying to route traffic from the shared host IP to a specific VTEP using unique port mappings. This is running in a Kubernetes/pod-based environment, if that helps clarify the use case.

Thank you so much, and I hope the patch can be reviewed.

Best,
-Jack
________________________________________
From: Ido Schimmel <idosch@nvidia.com>
Sent: Sunday, July 12, 2026 12:05 AM
To: Jack Ma <jack4it@hotmail.com>
Cc: David Ahern <dsahern@kernel.org>; Roopa Prabhu <roopa@nvidia.com>; netdev@vger.kernel.org <netdev@vger.kernel.org>; Nikolay Aleksandrov <razor@blackwall.org>
Subject: Re: VXLAN FDB nexthop groups: per-nexthop UDP dst port / VNI feasibility

On Fri, Jul 10, 2026 at 05:30:40AM +0000, Jack Ma wrote:
> Thanks very much for any pointers.

I already commented here:

https://lore.kernel.org/netdev/20260709102247.GA1536940@shredder/

Re: VXLAN FDB nexthop groups: per-nexthop UDP dst port / VNI feasibility

[Ido Schimmel]

Please use plain text instead of HTML. Otherwise your emails won't make
it to the list.

On Wed, Jul 08, 2026 at 05:52:05AM +0000, Jack Ma wrote:
>    Hi David, Ido, Roopa,
>    I am using VXLAN FDB nexthop groups (ip nexthop add ... fdb, then
>    bridge fdb add ... nhid) to do L2 ECMP across a set of HA endpoints.
>    In my design each leg is a distinct (VTEP IP, UDP destination port)
>    pair: several endpoints live behind the same node IP and are
>    disambiguated by the VXLAN UDP destination port.
>    Reading the current code (v6.12), it looks like an FDB nexthop can
>    only carry a gateway IP, so a per-leg port cannot be expressed:
>      * vxlan_fdb_nh_path_select() copies only nhc_gw into the synthetic
>        rdst; remote_port and remote_vni stay zero
>        (include/net/vxlan.h).
>      * vxlan_xmit_one() then falls back to the device-wide port:
>        dst_port = rdst->remote_port ? rdst->remote_port
>                                     : vxlan->cfg.dst_port
>        (drivers/net/vxlan/vxlan_core.c).
>      * rtm_to_nh_config() rejects NHA_ENCAP and NHA_OIF on fdb nexthops
>        ("Fdb attribute can not be used with encap, oif or blackhole",
>        net/ipv4/nexthop.c).
>      * vxlan_fdb_parse() rejects NDA_PORT / NDA_VNI / NDA_IFINDEX when
>        NDA_NH_ID is present ("DST, VNI, ifindex and port are mutually
>        exclusive with NH_ID").
>    The original commits (38428d68719c "nexthop: support for fdb ecmp
>    nexthops" and 1274e1cc4226 "vxlan: ecmp support for mac fdb entries")
>    say "these nexthops only have ip", which matches the EVPN multihoming
>    use case where every VTEP in the segment shares one UDP destination
>    port.
>    My questions:
>      1. Was IP-only a deliberate scoping decision for the EVPN-MH use
>         case, or is there a deeper reason a per-nexthop UDP destination
>         port (and VNI) could not be carried on an fdb nexthop, for
>         example via NHA_ENCAP / lwtunnel, which route nexthops already
>         accept?

IP-only was specific for EVPN-MH and it simply reuses the gateway IP
that is already present in the nexthop structure.

>      2. If legs genuinely need distinct UDP destination ports, is the
>         per-remote rdst path (bridge fdb ... dst ... port ...) the
>         intended mechanism, with the understanding that it gives
>         head-end replication semantics rather than nexthop-object ECMP?

Yes, but you will get packet replication instead of load balancing,
which I believe is not what you want.

>      3. Would you be open in principle to a patch that lets an fdb
>         nexthop carry a per-nexthop dst port (and VNI)? If so I am happy
>         to prototype it as an RFC.

I have some questions for you before I answer yours.

Can you explain why different VTEPs need to share a single IP? With
different IPs you can simply reuse the existing FDB nexthops without any
changes.

In addition, why disambiguate by the VXLAN UDP destination port and not
VNI? Even if we allow attaching tunnel encapsulation to FDB nexthops via
NHA_ENCAP, there does not seem to be uAPI in place to encode the
destination port (see ip_tun_policy).

Assuming you can live with a fixed port, I think you can achieve what
you want with BPF and no kernel changes. You can call
bpf_skb_set_tunnel_key [1] from a TC bpf program attached to the egress
of the VXLAN device (configured in "external" mode) and have it encode
the relevant VTEP based on the calculated hash.

[1] https://docs.ebpf.io/linux/helper-function/bpf_skb_set_tunnel_key/