[PATCH net-next v4 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops

[PATCH net-next v4 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops

[Jack Ma]

FDB nexthops let a VXLAN fdb entry point at a group of remote VTEPs, with the
kernel flow-hashing across the group (commit 1274e1cc4226 ("vxlan: ecmp support
for mac fdb entries")).  Each leg carries its own remote IP, but the UDP
destination port is always taken from the VXLAN device (vxlan->cfg.dst_port)
and cannot be set per leg.

This series adds an optional per-nexthop UDP destination port for fdb nexthops,
so a group's legs can share a remote IP and differ only in UDP port.

Motivation

The deployment runs an overlay in which each tenant's traffic is terminated by
a "forwarder": a pod that hosts the VXLAN VTEP, decapsulates the tenant's
overlay, and relays it to and from that tenant's workload.  Forwarders for many
different tenants are packed onto the same receiver node behind one
mesh-routable underlay IP, and are demultiplexed purely by UDP destination
port.  The host does a stateless outer-UDP demux by port; it never terminates
the tunnel:

  receiver node -- one mesh-routable underlay IP (NodeIP_A)
  +----------------------------------------------------+
  |  host netns: stateless outer-UDP demux by dst port |
  |              (host does NOT terminate the tunnel)  |
  |                                                    |
  |     dst :40000        dst :40001        dst :40002 |
  |         |                 |                 |      |
  |   +-----v----+      +-----v----+      +-----v----+ |
  |   | pod0 ns  |      | pod1 ns  |      | pod2 ns  | |
  |   | vxlan    |      | vxlan    |      | vxlan    | |
  |   | VTEP     |      | VTEP     |      | VTEP     | |
  |   | decap    |      | decap    |      | decap    | |
  |   +----------+      +----------+      +----------+ |
  +----------------------------------------------------+
  (up to ~10 forwarder pods packed per node)

The packed pods are unrelated: each belongs to a different tenant on its own
VXLAN VNI, so the per-pod UDP port is node-level demux, not an HA construct.
The host, which only demuxes outer UDP, never has to reason about tenancy.

A single forwarder is made highly available by running replicas.  The replicas
of one forwarder share a single anycast overlay identity: one inner MAC and IP.
Clients address that one identity, and a sender spreads flows across the live
replicas with an fdb nexthop group.  Failover is transparent: a dead replica is
just dropped from the group, with no client re-resolution or route change.  The
single identity is deliberate; the endpoint is consumed one layer up as a
single stable address, so giving each replica its own address would push
multi-address handling and health-checking up into that consumer.

Anti-affinity keeps the two replicas of one HA set on different nodes, so a
group's legs land on distinct node IPs.  But each leg is still reachable only
at (node IP, that pod's UDP port), so within one group the legs differ in IP
*and* port.  A group can already carry a distinct IP per leg, but it takes the
UDP port from the device (a single value), so it cannot send each leg to its
own port.  That is the gap this series closes.

Zooming into one forwarder pod, there is nothing for the host to load-balance:
the tunnel terminates on a vxlan device inside the pod's own netns, and the pod
reaches its tenant through a separate NIC:

  one forwarder pod -- its own netns, tenant VNI X
  +-------------------------------------------------+
  |                                                 |
  |   on/off-ramp NIC   <--- customer data plane    |
  |   |   on-ramp (ingress) / off-ramp (egress)     |
  |   |   inner packet                              |
  |   vxlan (VTEP)   encap / decap for VNI X,       |
  |   |              listens on this pod's UDP port |
  |   |   outer VXLAN UDP                           |
  |   eth0 (underlay)   NodeIP:port                 |
  |   |   to peer VTEPs over the                    |
  |   v   mesh underlay                             |
  |                                                 |
  +-------------------------------------------------+

Existing mechanisms do not fit this shape:

  - L3 multipath in the overlay needs each leg to be a distinct routable
    nexthop with its own address.  Since an HA set is a single anycast address
    by design, there are no distinct per-leg addresses to route over; the fdb
    nexthop group bridging to that shared MAC is what load-balances.
  - Host-side fan-out (XDP / TC / SO_REUSEPORT) assumes a shared host datapath
    that is not there.  SO_REUSEPORT balances sockets within one netns, but the
    receivers are in different netns (in fact different tenants).  An XDP/TC
    fan-out would require the host to terminate the tunnel and re-dispatch
    inner traffic across netns and VNI boundaries, i.e. become a VTEP, which
    puts the host into the tenant datapath and largely duplicates what an fdb
    nexthop group already does.
  - Demuxing on VNI instead of port (one shared 4789 socket, multiple vxlan
    devices differing only in VNI, moved into each pod's netns) works when the
    co-located pods have different VNIs.  It does not help two same-VNI HA sets
    on one node: their outer headers are identical, so the host would again
    have to terminate the tunnel to tell them apart.  It also costs packing
    density: with one shared underlay IP, VNI demux allows at most one VTEP per
    (VNI, node), so N same-VNI HA sets of two replicas need 2N nodes, whereas a
    per-pod port fits them on two nodes with anti-affinity preserved.

This series adds the attribute:

  - Patch 1 adds a netlink attribute NHA_DST_PORT (__be16, mirroring NDA_PORT),
    stored in struct nh_info and echoed back on dump.  It is only accepted
    together with NHA_FDB and NHA_GATEWAY.  Control-plane only; datapath
    behaviour is unchanged.
  - Patch 2 wires it into the VXLAN datapath: vxlan_fdb_nh_path_select() sets
    rdst->remote_port to the selected leg's port.  vxlan_xmit_one() already
    prefers rdst->remote_port when non-zero and otherwise falls back to the
    device port, so nexthops without a port are unaffected (backward
    compatible).
  - Patch 3 extends the fdb nexthop selftests.

On the uAPI: this does not add a new datapath concept.  A single fdb entry
already carries a per-destination UDP port (NDA_PORT), and vxlan_xmit_one()
already prefers rdst->remote_port when set.  NHA_DST_PORT is the nexthop analog
of that existing attribute: control-plane only, no datapath change, and
backward compatible (a leg with no port falls back to the device port as
today).  It sits at the nexthop level rather than under NHA_ENCAP because fdb
nexthops do not use the NHA_ENCAP / LWT infrastructure.

Example:

  ip nexthop add id 1  via 192.0.2.10 fdb dst_port 4789
  ip nexthop add id 2  via 192.0.2.10 fdb dst_port 5789
  ip nexthop add id 10 group 1/2 fdb
  bridge fdb add 00:11:22:33:44:55 dev vxlan0 nhid 10

Both legs share gateway 192.0.2.10 and differ only in UDP port; the kernel
hashes flows across them.

Testing: kernel and iproute2 built on net-next.  The fdb nexthop cases in
fib_nexthops.sh pass, and the datapath test in test_vxlan_nh.sh confirms a
per-nexthop port is used on the wire in place of the device default.  The
series is bisectable: patches 1 and 2 each build individually.

A matching iproute2 change adds the `dst_port` keyword to `ip nexthop`; it
will be posted to the iproute2 list alongside this series.

Changes in v4:
- Rename NHA_FDB_PORT to the generic NHA_DST_PORT (struct field nh_dst_port,
  nh_info.dst_port, iproute2 keyword dst_port).  Review noted "fdb port" is
  too specific and can be confused with a bridge port; the attribute is now
  named generically while validation stays fdb-only (Ido Schimmel, David
  Ahern).
- selftests: split the dst_port cases out of ipv4_fdb_grp_fcnal() /
  ipv6_fdb_grp_fcnal() into dedicated ipv4_fdb_port_fcnal() /
  ipv6_fdb_port_fcnal() tests, and require iproute2 dst_port support in
  test_vxlan_nh.sh (Ido Schimmel).

Changes in v3 (all addressing Ido Schimmel's review):
- patch 1: type NHA_FDB_PORT with NLA_POLICY_MIN(NLA_BE16, 1) so the
  attribute is big-endian and a zero port is rejected by the policy.
- patch 1: reserve the NHA_FDB_PORT dump size based on nh_info->fdb_port
  instead of fdb_nh, so only nexthops that carry a port account for it.
- patch 2: fix the nexthop_path_fdb_result() continuation-line alignment
  and drop the redundant fdb_port NULL check; its only caller always passes
  a valid pointer.
- patch 3: fold the control-plane checks into ipv4_fdb_grp_fcnal() and
  ipv6_fdb_grp_fcnal() in fib_nexthops.sh rather than a standalone script,
  and add datapath coverage to test_vxlan_nh.sh.

Changes in v2:
- Expand the cover letter with the deployment model and the
  addressing-vs-load-balancing distinction that motivates the attribute, as
  requested (Ido Schimmel).
- Reword the problem statement: the UDP port is underlay demux (addressing),
  not what distinguishes otherwise-interchangeable load-balancing targets.
- Drop the speculative NHA_FDB_VNI follow-up note; there is no use case for a
  per-leg VNI in an fdb nexthop group.
- No functional change to the patches.

v3: https://lore.kernel.org/netdev/20260721-b4-vxlan-fdb-port-v3-0-9aa0a543a78a@gmail.com/
v2: https://lore.kernel.org/netdev/20260717-b4-vxlan-fdb-port-v2-0-f4862e8fe867@gmail.com/
v1: https://lore.kernel.org/netdev/20260712191218.236-1-jack4it@gmail.com/

Signed-off-by: Jack Ma <jack4it@gmail.com>
---
Jack Ma (3):
      net: nexthop: add NHA_DST_PORT for fdb nexthops
      vxlan: honor per-nexthop fdb destination port
      selftests: net: add coverage for fdb nexthop dst_port

 include/net/nexthop.h                        |  6 ++-
 include/net/vxlan.h                          |  5 +-
 include/uapi/linux/nexthop.h                 |  3 ++
 net/ipv4/nexthop.c                           | 20 ++++++-
 tools/testing/selftests/net/fib_nexthops.sh  | 81 ++++++++++++++++++++++++++++
 tools/testing/selftests/net/test_vxlan_nh.sh | 34 +++++++++++-
 6 files changed, 144 insertions(+), 5 deletions(-)
---
base-commit: f6f3b36c15ed44de1fbb44e645e4fae8c4a4453e
change-id: 20260712-b4-vxlan-fdb-port-486bb215eab6

Best regards,
--  
Jack Ma <jack4it@gmail.com>

[PATCH net-next v4 1/3] net: nexthop: add NHA_DST_PORT for fdb nexthops

[Jack Ma]

Commit 1274e1cc4226 ("vxlan: ecmp support for mac fdb entries") lets a
single inner MAC be reached through a group of remote VTEPs, with the
kernel flow-hashing across the group members.  Each member carries its
own remote IP, but the UDP destination port is always taken from the
VXLAN device (vxlan->cfg.dst_port) and cannot be set per member.

Some deployments pack several receivers behind one underlay IP and tell
them apart by UDP port, so they need a per-nexthop destination port to
spread flows across (IP, port) tuples rather than IP alone.

Add a netlink attribute NHA_DST_PORT (__be16, mirroring NDA_PORT) that
carries an optional UDP destination port on an fdb nexthop.  It is only
accepted together with NHA_FDB and NHA_GATEWAY; it is stored in struct
nh_info and echoed back on dump.  The attribute is named generically
rather than fdb-specific so it can be reused should another nexthop type
ever need a destination port.  This patch is control-plane plumbing
only; the VXLAN datapath is wired up in a follow-up patch, so behaviour
is unchanged for now.

Signed-off-by: Jack Ma <jack4it@gmail.com>
---
 include/net/nexthop.h        |  2 ++
 include/uapi/linux/nexthop.h |  3 +++
 net/ipv4/nexthop.c           | 20 +++++++++++++++++++-
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/include/net/nexthop.h b/include/net/nexthop.h
index 572e69cda4766..7673aaeff3e28 100644
--- a/include/net/nexthop.h
+++ b/include/net/nexthop.h
@@ -28,6 +28,7 @@ struct nh_config {
 	u8		nh_protocol;
 	u8		nh_blackhole;
 	u8		nh_fdb;
+	__be16		nh_dst_port;
 	u32		nh_flags;
 
 	int		nh_ifindex;
@@ -63,6 +64,7 @@ struct nh_info {
 	u8			family;
 	bool			reject_nh;
 	bool			fdb_nh;
+	__be16			dst_port;
 
 	union {
 		struct fib_nh_common	fib_nhc;
diff --git a/include/uapi/linux/nexthop.h b/include/uapi/linux/nexthop.h
index bc49baf4a267f..59cf1cee93bd7 100644
--- a/include/uapi/linux/nexthop.h
+++ b/include/uapi/linux/nexthop.h
@@ -83,6 +83,9 @@ enum {
 	/* u32; read-only; whether any driver collects HW stats */
 	NHA_HW_STATS_USED,
 
+	/* be16; UDP destination port for an fdb nexthop (e.g. VXLAN) */
+	NHA_DST_PORT,
+
 	__NHA_MAX,
 };
 
diff --git a/net/ipv4/nexthop.c b/net/ipv4/nexthop.c
index 6205bd57aa852..0021380b8c1de 100644
--- a/net/ipv4/nexthop.c
+++ b/net/ipv4/nexthop.c
@@ -39,6 +39,7 @@ static const struct nla_policy rtm_nh_policy_new[] = {
 	[NHA_ENCAP_TYPE]	= { .type = NLA_U16 },
 	[NHA_ENCAP]		= { .type = NLA_NESTED },
 	[NHA_FDB]		= { .type = NLA_FLAG },
+	[NHA_DST_PORT]		= NLA_POLICY_MIN(NLA_BE16, 1),
 	[NHA_RES_GROUP]		= { .type = NLA_NESTED },
 	[NHA_HW_STATS_ENABLE]	= NLA_POLICY_MAX(NLA_U32, true),
 };
@@ -956,6 +957,9 @@ static int nh_fill_node(struct sk_buff *skb, struct nexthop *nh,
 	} else if (nhi->fdb_nh) {
 		if (nla_put_flag(skb, NHA_FDB))
 			goto nla_put_failure;
+		if (nhi->dst_port &&
+		    nla_put_be16(skb, NHA_DST_PORT, nhi->dst_port))
+			goto nla_put_failure;
 	} else {
 		const struct net_device *dev;
 
@@ -1055,6 +1059,9 @@ static size_t nh_nlmsg_size_single(struct nexthop *nh)
 		break;
 	}
 
+	if (nhi->dst_port)
+		sz += nla_total_size(2);	/* NHA_DST_PORT */
+
 	if (nhi->fib_nhc.nhc_lwtstate) {
 		sz += lwtunnel_get_encap_size(nhi->fib_nhc.nhc_lwtstate);
 		sz += nla_total_size(2);  /* NHA_ENCAP_TYPE */
@@ -2956,8 +2963,10 @@ static struct nexthop *nexthop_create(struct net *net, struct nh_config *cfg,
 	nhi->family = cfg->nh_family;
 	nhi->fib_nhc.nhc_scope = RT_SCOPE_LINK;
 
-	if (cfg->nh_fdb)
+	if (cfg->nh_fdb) {
 		nhi->fdb_nh = 1;
+		nhi->dst_port = cfg->nh_dst_port;
+	}
 
 	if (cfg->nh_blackhole) {
 		nhi->reject_nh = 1;
@@ -3147,6 +3156,15 @@ static int rtm_to_nh_config(struct net *net, struct sk_buff *skb,
 		cfg->nh_fdb = nla_get_flag(tb[NHA_FDB]);
 	}
 
+	if (tb[NHA_DST_PORT]) {
+		if (!tb[NHA_FDB] || !tb[NHA_GATEWAY]) {
+			NL_SET_ERR_MSG(extack,
+				       "Destination port can only be set on fdb nexthops that have a gateway");
+			goto out;
+		}
+		cfg->nh_dst_port = nla_get_be16(tb[NHA_DST_PORT]);
+	}
+
 	if (tb[NHA_GROUP]) {
 		if (nhm->nh_family != AF_UNSPEC) {
 			NL_SET_ERR_MSG(extack, "Invalid family for group");

-- 
2.43.0

[PATCH net-next v4 2/3] vxlan: honor per-nexthop fdb destination port

[Jack Ma]

When an fdb entry points at a nexthop group, vxlan_fdb_nh_path_select()
resolves the selected leg's remote IP but leaves the UDP destination port
at the device default (vxlan->cfg.dst_port).

Extend nexthop_path_fdb_result() to also return the selected nexthop's
NHA_DST_PORT (0 when unset) and have vxlan_fdb_nh_path_select() store it
in rdst->remote_port.  vxlan_xmit_one() already prefers rdst->remote_port
when non-zero and falls back to the device port otherwise, so nexthops
without a port are unaffected.

This lets one fdb nexthop group load-balance a flow across legs that
share an underlay IP but differ in UDP destination port.

Signed-off-by: Jack Ma <jack4it@gmail.com>
---
 include/net/nexthop.h | 4 +++-
 include/net/vxlan.h   | 5 ++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/include/net/nexthop.h b/include/net/nexthop.h
index 7673aaeff3e28..f86c115074d7a 100644
--- a/include/net/nexthop.h
+++ b/include/net/nexthop.h
@@ -576,7 +576,8 @@ struct fib_nh_common *nexthop_fdb_nhc(struct nexthop *nh)
 }
 
 static inline struct fib_nh_common *nexthop_path_fdb_result(struct nexthop *nh,
-							    int hash)
+							    int hash,
+							    __be16 *dst_port)
 {
 	struct nh_info *nhi;
 	struct nexthop *nhp;
@@ -585,6 +586,7 @@ static inline struct fib_nh_common *nexthop_path_fdb_result(struct nexthop *nh,
 	if (unlikely(!nhp))
 		return NULL;
 	nhi = rcu_dereference(nhp->nh_info);
+	*dst_port = nhi->dst_port;
 	return &nhi->fib_nhc;
 }
 #endif
diff --git a/include/net/vxlan.h b/include/net/vxlan.h
index dfba89695efcf..6e64757151b88 100644
--- a/include/net/vxlan.h
+++ b/include/net/vxlan.h
@@ -567,8 +567,9 @@ static inline bool vxlan_fdb_nh_path_select(struct nexthop *nh,
 					    struct vxlan_rdst *rdst)
 {
 	struct fib_nh_common *nhc;
+	__be16 dst_port = 0;
 
-	nhc = nexthop_path_fdb_result(nh, hash >> 1);
+	nhc = nexthop_path_fdb_result(nh, hash >> 1, &dst_port);
 	if (unlikely(!nhc))
 		return false;
 
@@ -583,6 +584,8 @@ static inline bool vxlan_fdb_nh_path_select(struct nexthop *nh,
 		break;
 	}
 
+	rdst->remote_port = dst_port;
+
 	return true;
 }
 

-- 
2.43.0

[PATCH net-next v4 3/3] selftests: net: add coverage for fdb nexthop dst_port

[Jack Ma]

Add coverage for the new per-nexthop VXLAN destination port
(NHA_DST_PORT).

In fib_nexthops.sh, new ipv4_fdb_port_fcnal() and ipv6_fdb_port_fcnal()
tests check that a dst_port is accepted on an fdb nexthop that has a
gateway and echoed back on dump, that it is rejected without a gateway
and rejected when zero, that a group may hold legs that differ only in
UDP port, and that a portless fdb nexthop omits the attribute.  The
tests SKIP when iproute2 lacks the "dst_port" keyword.

In test_vxlan_nh.sh, basic_tx_common() gains a second fdb nexthop group
whose nexthop carries a destination port that differs from the VXLAN
device default, plus a flower filter keyed on that port, to confirm the
per-nexthop port is used on the wire.  The test now requires an iproute2
with dst_port support.

Signed-off-by: Jack Ma <jack4it@gmail.com>
---
 tools/testing/selftests/net/fib_nexthops.sh  | 81 ++++++++++++++++++++++++++++
 tools/testing/selftests/net/test_vxlan_nh.sh | 34 +++++++++++-
 2 files changed, 113 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/net/fib_nexthops.sh b/tools/testing/selftests/net/fib_nexthops.sh
index ac868a7316946..834ba1c0676c8 100755
--- a/tools/testing/selftests/net/fib_nexthops.sh
+++ b/tools/testing/selftests/net/fib_nexthops.sh
@@ -30,6 +30,7 @@ IPV4_TESTS="
 	ipv4_large_res_grp
 	ipv4_compat_mode
 	ipv4_fdb_grp_fcnal
+	ipv4_fdb_port_fcnal
 	ipv4_mpath_select
 	ipv4_torture
 	ipv4_res_torture
@@ -44,6 +45,7 @@ IPV6_TESTS="
 	ipv6_large_res_grp
 	ipv6_compat_mode
 	ipv6_fdb_grp_fcnal
+	ipv6_fdb_port_fcnal
 	ipv6_mpath_select
 	ipv6_torture
 	ipv6_res_torture
@@ -432,6 +434,15 @@ check_nexthop_fdb_support()
 	fi
 }
 
+check_nexthop_fdb_port_support()
+{
+	$IP nexthop help 2>&1 | grep -q "dst_port"
+	if [ $? -ne 0 ]; then
+		echo "SKIP: iproute2 too old, missing nexthop dst_port support"
+		return $ksft_skip
+	fi
+}
+
 check_nexthop_res_support()
 {
 	$IP nexthop help 2>&1 | grep -q resilient
@@ -541,6 +552,41 @@ ipv6_fdb_grp_fcnal()
 	$IP link del dev vx10
 }
 
+ipv6_fdb_port_fcnal()
+{
+	echo
+	echo "IPv6 fdb nexthop dst_port functional"
+	echo "------------------------------------"
+
+	check_nexthop_fdb_port_support
+	if [ $? -eq $ksft_skip ]; then
+		return $ksft_skip
+	fi
+
+	# NHA_DST_PORT: optional per-nexthop VXLAN destination UDP port,
+	# letting an fdb nexthop group balance a flow across legs that share
+	# an underlay IP but listen on different UDP ports.
+	run_cmd "$IP nexthop add id 80 via 2001:db8:91::2 fdb dst_port 4790"
+	check_nexthop "id 80" "id 80 via 2001:db8:91::2 scope link fdb dst_port 4790"
+	log_test $? 0 "Fdb nexthop with dst_port"
+
+	run_cmd "$IP nexthop add id 81 fdb dst_port 4790"
+	log_test $? 2 "Fdb nexthop with dst_port but no gateway"
+
+	run_cmd "$IP nexthop add id 81 via 2001:db8:91::2 fdb dst_port 0"
+	log_test $? 2 "Fdb nexthop with dst_port 0"
+
+	run_cmd "$IP nexthop add id 82 via 2001:db8:91::2 fdb dst_port 4789"
+	run_cmd "$IP nexthop add id 83 via 2001:db8:91::3 fdb dst_port 5789"
+	run_cmd "$IP nexthop add id 106 group 82/83 fdb"
+	check_nexthop "id 106" "id 106 group 82/83 fdb"
+	log_test $? 0 "Fdb nexthop group with legs differing in dst_port"
+
+	run_cmd "$IP nexthop add id 84 via 2001:db8:91::2 fdb"
+	check_nexthop "id 84" "id 84 via 2001:db8:91::2 scope link fdb"
+	log_test $? 0 "Fdb nexthop without dst_port omits dst_port"
+}
+
 ipv4_fdb_grp_fcnal()
 {
 	local rc
@@ -641,6 +687,41 @@ ipv4_fdb_grp_fcnal()
 	$IP link del dev vx10
 }
 
+ipv4_fdb_port_fcnal()
+{
+	echo
+	echo "IPv4 fdb nexthop dst_port functional"
+	echo "------------------------------------"
+
+	check_nexthop_fdb_port_support
+	if [ $? -eq $ksft_skip ]; then
+		return $ksft_skip
+	fi
+
+	# NHA_DST_PORT: optional per-nexthop VXLAN destination UDP port,
+	# letting an fdb nexthop group balance a flow across legs that share
+	# an underlay IP but listen on different UDP ports.
+	run_cmd "$IP nexthop add id 30 via 172.16.1.2 fdb dst_port 4790"
+	check_nexthop "id 30" "id 30 via 172.16.1.2 scope link fdb dst_port 4790"
+	log_test $? 0 "Fdb nexthop with dst_port"
+
+	run_cmd "$IP nexthop add id 31 fdb dst_port 4790"
+	log_test $? 2 "Fdb nexthop with dst_port but no gateway"
+
+	run_cmd "$IP nexthop add id 31 via 172.16.1.2 fdb dst_port 0"
+	log_test $? 2 "Fdb nexthop with dst_port 0"
+
+	run_cmd "$IP nexthop add id 32 via 172.16.1.2 fdb dst_port 4789"
+	run_cmd "$IP nexthop add id 33 via 172.16.1.3 fdb dst_port 5789"
+	run_cmd "$IP nexthop add id 105 group 32/33 fdb"
+	check_nexthop "id 105" "id 105 group 32/33 fdb"
+	log_test $? 0 "Fdb nexthop group with legs differing in dst_port"
+
+	run_cmd "$IP nexthop add id 34 via 172.16.1.2 fdb"
+	check_nexthop "id 34" "id 34 via 172.16.1.2 scope link fdb"
+	log_test $? 0 "Fdb nexthop without dst_port omits dst_port"
+}
+
 ipv4_mpath_select()
 {
 	local rc dev match h addr
diff --git a/tools/testing/selftests/net/test_vxlan_nh.sh b/tools/testing/selftests/net/test_vxlan_nh.sh
index 20f3369f776b1..7b5dc1d37fa02 100755
--- a/tools/testing/selftests/net/test_vxlan_nh.sh
+++ b/tools/testing/selftests/net/test_vxlan_nh.sh
@@ -56,6 +56,16 @@ tc_stats_get()
 	tc_rule_handle_stats_get "dev dummy1 egress" 101 ".packets" "-n $ns1"
 }
 
+nh_stats_get_port()
+{
+	ip -n "$ns1" -s -j nexthop show id 20 | jq ".[][\"group_stats\"][][\"packets\"]"
+}
+
+tc_stats_get_port()
+{
+	tc_rule_handle_stats_get "dev dummy1 egress" 102 ".packets" "-n $ns1"
+}
+
 basic_tx_common()
 {
 	local af_str=$1; shift
@@ -90,6 +100,26 @@ basic_tx_common()
 	busywait "$BUSYWAIT_TIMEOUT" until_counter_is "== 1" tc_stats_get > /dev/null
 	check_err $? "tc filter stats did not increase"
 
+	# Add a second FDB nexthop group whose nexthop carries a per-nexthop
+	# destination port (NHA_DST_PORT) that differs from the VXLAN device
+	# default. Matching outer traffic must egress with that port, so a
+	# separate flower filter keyed on the new port catches it.
+	run_cmd "tc -n $ns1 filter add dev dummy1 egress proto $proto pref 1 handle 102 \
+		flower ip_proto udp dst_ip $remote_addr dst_port 4790 action pass"
+
+	run_cmd "ip -n $ns1 nexthop add id 2 via $remote_addr fdb dst_port 4790"
+	run_cmd "ip -n $ns1 nexthop add id 20 group 2 fdb"
+
+	run_cmd "bridge -n $ns1 fdb add 00:11:22:33:44:66 dev vx0 self static nhid 20"
+
+	run_cmd "ip netns exec $ns1 mausezahn vx0 -a own -b 00:11:22:33:44:66 -c 1 -q"
+
+	busywait "$BUSYWAIT_TIMEOUT" until_counter_is "== 1" nh_stats_get_port > /dev/null
+	check_err $? "FDB nexthop group stats did not increase (with port)"
+
+	busywait "$BUSYWAIT_TIMEOUT" until_counter_is "== 1" tc_stats_get_port > /dev/null
+	check_err $? "tc filter stats did not increase (with port)"
+
 	log_test "VXLAN FDB nexthop: $af_str basic Tx"
 }
 
@@ -210,8 +240,8 @@ require_command arping
 require_command ndisc6
 require_command jq
 
-if ! ip nexthop help 2>&1 | grep -q "stats"; then
-	echo "SKIP: iproute2 ip too old, missing nexthop stats support"
+if ! ip nexthop help 2>&1 | grep -q "dst_port"; then
+	echo "SKIP: iproute2 ip too old, missing nexthop dst_port support"
 	exit "$ksft_skip"
 fi
 

-- 
2.43.0

Re: [PATCH net-next v4 1/3] net: nexthop: add NHA_DST_PORT for fdb nexthops

[Ido Schimmel]

On Thu, Jul 23, 2026 at 01:17:53AM +0000, Jack Ma wrote:
> Commit 1274e1cc4226 ("vxlan: ecmp support for mac fdb entries") lets a
> single inner MAC be reached through a group of remote VTEPs, with the
> kernel flow-hashing across the group members.  Each member carries its
> own remote IP, but the UDP destination port is always taken from the
> VXLAN device (vxlan->cfg.dst_port) and cannot be set per member.
> 
> Some deployments pack several receivers behind one underlay IP and tell
> them apart by UDP port, so they need a per-nexthop destination port to
> spread flows across (IP, port) tuples rather than IP alone.
> 
> Add a netlink attribute NHA_DST_PORT (__be16, mirroring NDA_PORT) that
> carries an optional UDP destination port on an fdb nexthop.  It is only
> accepted together with NHA_FDB and NHA_GATEWAY; it is stored in struct
> nh_info and echoed back on dump.  The attribute is named generically
> rather than fdb-specific so it can be reused should another nexthop type
> ever need a destination port.  This patch is control-plane plumbing
> only; the VXLAN datapath is wired up in a follow-up patch, so behaviour
> is unchanged for now.
> 
> Signed-off-by: Jack Ma <jack4it@gmail.com>

Reviewed-by: Ido Schimmel <idosch@nvidia.com>

Re: [PATCH net-next v4 2/3] vxlan: honor per-nexthop fdb destination port

[Ido Schimmel]

On Thu, Jul 23, 2026 at 01:17:54AM +0000, Jack Ma wrote:
> When an fdb entry points at a nexthop group, vxlan_fdb_nh_path_select()
> resolves the selected leg's remote IP but leaves the UDP destination port
> at the device default (vxlan->cfg.dst_port).
> 
> Extend nexthop_path_fdb_result() to also return the selected nexthop's
> NHA_DST_PORT (0 when unset) and have vxlan_fdb_nh_path_select() store it
> in rdst->remote_port.  vxlan_xmit_one() already prefers rdst->remote_port
> when non-zero and falls back to the device port otherwise, so nexthops
> without a port are unaffected.
> 
> This lets one fdb nexthop group load-balance a flow across legs that
> share an underlay IP but differ in UDP destination port.
> 
> Signed-off-by: Jack Ma <jack4it@gmail.com>

Reviewed-by: Ido Schimmel <idosch@nvidia.com>

Re: [PATCH net-next v4 3/3] selftests: net: add coverage for fdb nexthop dst_port

[Ido Schimmel]

On Thu, Jul 23, 2026 at 01:17:55AM +0000, Jack Ma wrote:
> Add coverage for the new per-nexthop VXLAN destination port
> (NHA_DST_PORT).
> 
> In fib_nexthops.sh, new ipv4_fdb_port_fcnal() and ipv6_fdb_port_fcnal()
> tests check that a dst_port is accepted on an fdb nexthop that has a
> gateway and echoed back on dump, that it is rejected without a gateway
> and rejected when zero, that a group may hold legs that differ only in
> UDP port, and that a portless fdb nexthop omits the attribute.  The
> tests SKIP when iproute2 lacks the "dst_port" keyword.
> 
> In test_vxlan_nh.sh, basic_tx_common() gains a second fdb nexthop group
> whose nexthop carries a destination port that differs from the VXLAN
> device default, plus a flower filter keyed on that port, to confirm the
> per-nexthop port is used on the wire.  The test now requires an iproute2
> with dst_port support.
> 
> Signed-off-by: Jack Ma <jack4it@gmail.com>

Reviewed-by: Ido Schimmel <idosch@nvidia.com>

Re: [PATCH net-next v4 1/3] net: nexthop: add NHA_DST_PORT for fdb nexthops

[David Ahern]

On 7/22/26 7:17 PM, Jack Ma wrote:
> Commit 1274e1cc4226 ("vxlan: ecmp support for mac fdb entries") lets a
> single inner MAC be reached through a group of remote VTEPs, with the
> kernel flow-hashing across the group members.  Each member carries its
> own remote IP, but the UDP destination port is always taken from the
> VXLAN device (vxlan->cfg.dst_port) and cannot be set per member.
> 
> Some deployments pack several receivers behind one underlay IP and tell
> them apart by UDP port, so they need a per-nexthop destination port to
> spread flows across (IP, port) tuples rather than IP alone.
> 
> Add a netlink attribute NHA_DST_PORT (__be16, mirroring NDA_PORT) that
> carries an optional UDP destination port on an fdb nexthop.  It is only
> accepted together with NHA_FDB and NHA_GATEWAY; it is stored in struct
> nh_info and echoed back on dump.  The attribute is named generically
> rather than fdb-specific so it can be reused should another nexthop type
> ever need a destination port.  This patch is control-plane plumbing
> only; the VXLAN datapath is wired up in a follow-up patch, so behaviour
> is unchanged for now.
> 
> Signed-off-by: Jack Ma <jack4it@gmail.com>
> ---
>  include/net/nexthop.h        |  2 ++
>  include/uapi/linux/nexthop.h |  3 +++
>  net/ipv4/nexthop.c           | 20 +++++++++++++++++++-
>  3 files changed, 24 insertions(+), 1 deletion(-)
> 

Reviewed-by: David Ahern <dsahern@kernel.org>

Re: [PATCH net-next v4 2/3] vxlan: honor per-nexthop fdb destination port

[David Ahern]

On 7/22/26 7:17 PM, Jack Ma wrote:
> When an fdb entry points at a nexthop group, vxlan_fdb_nh_path_select()
> resolves the selected leg's remote IP but leaves the UDP destination port
> at the device default (vxlan->cfg.dst_port).
> 
> Extend nexthop_path_fdb_result() to also return the selected nexthop's
> NHA_DST_PORT (0 when unset) and have vxlan_fdb_nh_path_select() store it
> in rdst->remote_port.  vxlan_xmit_one() already prefers rdst->remote_port
> when non-zero and falls back to the device port otherwise, so nexthops
> without a port are unaffected.
> 
> This lets one fdb nexthop group load-balance a flow across legs that
> share an underlay IP but differ in UDP destination port.
> 
> Signed-off-by: Jack Ma <jack4it@gmail.com>
> ---
>  include/net/nexthop.h | 4 +++-
>  include/net/vxlan.h   | 5 ++++-
>  2 files changed, 7 insertions(+), 2 deletions(-)
> 

Reviewed-by: David Ahern <dsahern@kernel.org>

Re: [PATCH net-next v4 3/3] selftests: net: add coverage for fdb nexthop dst_port

[David Ahern]

On 7/22/26 7:17 PM, Jack Ma wrote:
> Add coverage for the new per-nexthop VXLAN destination port
> (NHA_DST_PORT).
> 
> In fib_nexthops.sh, new ipv4_fdb_port_fcnal() and ipv6_fdb_port_fcnal()
> tests check that a dst_port is accepted on an fdb nexthop that has a
> gateway and echoed back on dump, that it is rejected without a gateway
> and rejected when zero, that a group may hold legs that differ only in
> UDP port, and that a portless fdb nexthop omits the attribute.  The
> tests SKIP when iproute2 lacks the "dst_port" keyword.
> 
> In test_vxlan_nh.sh, basic_tx_common() gains a second fdb nexthop group
> whose nexthop carries a destination port that differs from the VXLAN
> device default, plus a flower filter keyed on that port, to confirm the
> per-nexthop port is used on the wire.  The test now requires an iproute2
> with dst_port support.
> 
> Signed-off-by: Jack Ma <jack4it@gmail.com>
> ---
>  tools/testing/selftests/net/fib_nexthops.sh  | 81 ++++++++++++++++++++++++++++
>  tools/testing/selftests/net/test_vxlan_nh.sh | 34 +++++++++++-
>  2 files changed, 113 insertions(+), 2 deletions(-)
> 

Reviewed-by: David Ahern <dsahern@kernel.org>