[PATCH net-next v3 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops

[PATCH net-next v3 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops

[Jack Ma]

FDB nexthops let a VXLAN fdb entry point at a group of remote VTEPs, with the
kernel flow-hashing across the group (commit 1274e1cc4226 ("vxlan: ecmp support
for mac fdb entries")).  Each leg carries its own remote IP, but the UDP
destination port is always taken from the VXLAN device (vxlan->cfg.dst_port)
and cannot be set per leg.

This series adds an optional per-nexthop UDP destination port for fdb nexthops,
so a group's legs can share a remote IP and differ only in UDP port.

Motivation

The deployment runs an overlay in which each tenant's traffic is terminated by
a "forwarder": a pod that hosts the VXLAN VTEP, decapsulates the tenant's
overlay, and relays it to and from that tenant's workload.  Forwarders for many
different tenants are packed onto the same receiver node behind one
mesh-routable underlay IP, and are demultiplexed purely by UDP destination
port.  The host does a stateless outer-UDP demux by port; it never terminates
the tunnel:

  receiver node -- one mesh-routable underlay IP (NodeIP_A)
  +----------------------------------------------------+
  |  host netns: stateless outer-UDP demux by dst port |
  |              (host does NOT terminate the tunnel)  |
  |                                                    |
  |     dst :40000        dst :40001        dst :40002 |
  |         |                 |                 |      |
  |   +-----v----+      +-----v----+      +-----v----+ |
  |   | pod0 ns  |      | pod1 ns  |      | pod2 ns  | |
  |   | vxlan    |      | vxlan    |      | vxlan    | |
  |   | VTEP     |      | VTEP     |      | VTEP     | |
  |   | decap    |      | decap    |      | decap    | |
  |   +----------+      +----------+      +----------+ |
  +----------------------------------------------------+
  (up to ~10 forwarder pods packed per node)

The packed pods are unrelated: each belongs to a different tenant on its own
VXLAN VNI, so the per-pod UDP port is node-level demux, not an HA construct.
The host, which only demuxes outer UDP, never has to reason about tenancy.

A single forwarder is made highly available by running replicas.  The replicas
of one forwarder share a single anycast overlay identity: one inner MAC and IP.
Clients address that one identity, and a sender spreads flows across the live
replicas with an fdb nexthop group.  Failover is transparent: a dead replica is
just dropped from the group, with no client re-resolution or route change.  The
single identity is deliberate; the endpoint is consumed one layer up as a
single stable address, so giving each replica its own address would push
multi-address handling and health-checking up into that consumer.

Anti-affinity keeps the two replicas of one HA set on different nodes, so a
group's legs land on distinct node IPs.  But each leg is still reachable only
at (node IP, that pod's UDP port), so within one group the legs differ in IP
*and* port.  A group can already carry a distinct IP per leg, but it takes the
UDP port from the device (a single value), so it cannot send each leg to its
own port.  That is the gap this series closes.

Zooming into one forwarder pod, there is nothing for the host to load-balance:
the tunnel terminates on a vxlan device inside the pod's own netns, and the pod
reaches its tenant through a separate NIC:

  one forwarder pod -- its own netns, tenant VNI X
  +-------------------------------------------------+
  |                                                 |
  |   on/off-ramp NIC   <--- customer data plane    |
  |   |   on-ramp (ingress) / off-ramp (egress)     |
  |   |   inner packet                              |
  |   vxlan (VTEP)   encap / decap for VNI X,       |
  |   |              listens on this pod's UDP port |
  |   |   outer VXLAN UDP                           |
  |   eth0 (underlay)   NodeIP:port                 |
  |   |   to peer VTEPs over the                    |
  |   v   mesh underlay                             |
  |                                                 |
  +-------------------------------------------------+

Existing mechanisms do not fit this shape:

  - L3 multipath in the overlay needs each leg to be a distinct routable
    nexthop with its own address.  Since an HA set is a single anycast address
    by design, there are no distinct per-leg addresses to route over; the fdb
    nexthop group bridging to that shared MAC is what load-balances.
  - Host-side fan-out (XDP / TC / SO_REUSEPORT) assumes a shared host datapath
    that is not there.  SO_REUSEPORT balances sockets within one netns, but the
    receivers are in different netns (in fact different tenants).  An XDP/TC
    fan-out would require the host to terminate the tunnel and re-dispatch
    inner traffic across netns and VNI boundaries, i.e. become a VTEP, which
    puts the host into the tenant datapath and largely duplicates what an fdb
    nexthop group already does.
  - Demuxing on VNI instead of port (one shared 4789 socket, multiple vxlan
    devices differing only in VNI, moved into each pod's netns) works when the
    co-located pods have different VNIs.  It does not help two same-VNI HA sets
    on one node: their outer headers are identical, so the host would again
    have to terminate the tunnel to tell them apart.  It also costs packing
    density: with one shared underlay IP, VNI demux allows at most one VTEP per
    (VNI, node), so N same-VNI HA sets of two replicas need 2N nodes, whereas a
    per-pod port fits them on two nodes with anti-affinity preserved.

This series adds the attribute:

  - Patch 1 adds a netlink attribute NHA_FDB_PORT (__be16, mirroring NDA_PORT),
    stored in struct nh_info and echoed back on dump.  It is only accepted
    together with NHA_FDB and NHA_GATEWAY.  Control-plane only; datapath
    behaviour is unchanged.
  - Patch 2 wires it into the VXLAN datapath: vxlan_fdb_nh_path_select() sets
    rdst->remote_port to the selected leg's port.  vxlan_xmit_one() already
    prefers rdst->remote_port when non-zero and otherwise falls back to the
    device port, so nexthops without a port are unaffected (backward
    compatible).
  - Patch 3 extends the fdb nexthop selftests.

On the uAPI: this does not add a new datapath concept.  A single fdb entry
already carries a per-destination UDP port (NDA_PORT), and vxlan_xmit_one()
already prefers rdst->remote_port when set.  NHA_FDB_PORT is the nexthop analog
of that existing attribute: control-plane only, no datapath change, and
backward compatible (a leg with no port falls back to the device port as
today).  It sits at the nexthop level rather than under NHA_ENCAP because fdb
nexthops do not use the NHA_ENCAP / LWT infrastructure.

Example:

  ip nexthop add id 1  via 192.0.2.10 fdb port 4789
  ip nexthop add id 2  via 192.0.2.10 fdb port 5789
  ip nexthop add id 10 group 1/2 fdb
  bridge fdb add 00:11:22:33:44:55 dev vxlan0 nhid 10

Both legs share gateway 192.0.2.10 and differ only in UDP port; the kernel
hashes flows across them.

Testing: kernel and iproute2 built on net-next.  The fdb nexthop cases in
fib_nexthops.sh pass, and the datapath test in test_vxlan_nh.sh confirms a
per-nexthop port is used on the wire in place of the device default.  The
series is bisectable: patches 1 and 2 each build individually.

A matching iproute2 change (the `ip nexthop ... fdb port N` keyword) is posted
separately to the iproute2 list.

Changes in v3 (all addressing Ido Schimmel's review):
- patch 1: type NHA_FDB_PORT with NLA_POLICY_MIN(NLA_BE16, 1) so the
  attribute is big-endian and a zero port is rejected by the policy.
- patch 1: reserve the NHA_FDB_PORT dump size based on nh_info->fdb_port
  instead of fdb_nh, so only nexthops that carry a port account for it.
- patch 2: fix the nexthop_path_fdb_result() continuation-line alignment
  and drop the redundant fdb_port NULL check; its only caller always passes
  a valid pointer.
- patch 3: fold the control-plane checks into ipv4_fdb_grp_fcnal() and
  ipv6_fdb_grp_fcnal() in fib_nexthops.sh rather than a standalone script,
  and add datapath coverage to test_vxlan_nh.sh.

Changes in v2:
- Expand the cover letter with the deployment model and the
  addressing-vs-load-balancing distinction that motivates the attribute, as
  requested (Ido Schimmel).
- Reword the problem statement: the UDP port is underlay demux (addressing),
  not what distinguishes otherwise-interchangeable load-balancing targets.
- Drop the speculative NHA_FDB_VNI follow-up note; there is no use case for a
  per-leg VNI in an fdb nexthop group.
- No functional change to the patches.

v2: https://lore.kernel.org/netdev/20260717-b4-vxlan-fdb-port-v2-0-f4862e8fe867@gmail.com/
v1: https://lore.kernel.org/netdev/20260712191218.236-1-jack4it@gmail.com/

Signed-off-by: Jack Ma <jack4it@gmail.com>
---
Jack Ma (3):
      net: nexthop: add NHA_FDB_PORT for fdb nexthops
      vxlan: honor per-nexthop fdb destination port
      selftests: net: add coverage for fdb nexthop dst port

 include/net/nexthop.h                        |  6 ++-
 include/net/vxlan.h                          |  5 ++-
 include/uapi/linux/nexthop.h                 |  3 ++
 net/ipv4/nexthop.c                           | 20 +++++++++-
 tools/testing/selftests/net/fib_nexthops.sh  | 59 ++++++++++++++++++++++++++++
 tools/testing/selftests/net/test_vxlan_nh.sh | 31 +++++++++++++++
 6 files changed, 121 insertions(+), 3 deletions(-)
---
base-commit: f6f3b36c15ed44de1fbb44e645e4fae8c4a4453e
change-id: 20260712-b4-vxlan-fdb-port-486bb215eab6

Best regards,
--  
Jack Ma <jack4it@gmail.com>

[PATCH net-next v3 1/3] net: nexthop: add NHA_FDB_PORT for fdb nexthops

[Jack Ma]

Commit 1274e1cc4226 ("vxlan: ecmp support for mac fdb entries") lets a
single inner MAC be reached through a group of remote VTEPs, with the
kernel flow-hashing across the group members.  Each member carries its
own remote IP, but the UDP destination port is always taken from the
VXLAN device (vxlan->cfg.dst_port) and cannot be set per member.

Some deployments pack several receivers behind one underlay IP and tell
them apart by UDP port, so they need a per-nexthop destination port to
spread flows across (IP, port) tuples rather than IP alone.

Add a netlink attribute NHA_FDB_PORT (__be16, mirroring NDA_PORT) that
carries an optional UDP destination port on an fdb nexthop.  It is only
accepted together with NHA_FDB and NHA_GATEWAY; it is stored in struct
nh_info and echoed back on dump.  This patch is control-plane plumbing
only; the VXLAN datapath is wired up in a follow-up patch, so behaviour
is unchanged for now.

Signed-off-by: Jack Ma <jack4it@gmail.com>
---
 include/net/nexthop.h        |  2 ++
 include/uapi/linux/nexthop.h |  3 +++
 net/ipv4/nexthop.c           | 20 +++++++++++++++++++-
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/include/net/nexthop.h b/include/net/nexthop.h
index 572e69cda476..9c822799634f 100644
--- a/include/net/nexthop.h
+++ b/include/net/nexthop.h
@@ -28,6 +28,7 @@ struct nh_config {
 	u8		nh_protocol;
 	u8		nh_blackhole;
 	u8		nh_fdb;
+	__be16		nh_fdb_port;
 	u32		nh_flags;
 
 	int		nh_ifindex;
@@ -63,6 +64,7 @@ struct nh_info {
 	u8			family;
 	bool			reject_nh;
 	bool			fdb_nh;
+	__be16			fdb_port;
 
 	union {
 		struct fib_nh_common	fib_nhc;
diff --git a/include/uapi/linux/nexthop.h b/include/uapi/linux/nexthop.h
index bc49baf4a267..e587bbf3b890 100644
--- a/include/uapi/linux/nexthop.h
+++ b/include/uapi/linux/nexthop.h
@@ -83,6 +83,9 @@ enum {
 	/* u32; read-only; whether any driver collects HW stats */
 	NHA_HW_STATS_USED,
 
+	/* be16; UDP destination port for an fdb nexthop (e.g. VXLAN) */
+	NHA_FDB_PORT,
+
 	__NHA_MAX,
 };
 
diff --git a/net/ipv4/nexthop.c b/net/ipv4/nexthop.c
index 6205bd57aa85..b82da3eff63c 100644
--- a/net/ipv4/nexthop.c
+++ b/net/ipv4/nexthop.c
@@ -39,6 +39,7 @@ static const struct nla_policy rtm_nh_policy_new[] = {
 	[NHA_ENCAP_TYPE]	= { .type = NLA_U16 },
 	[NHA_ENCAP]		= { .type = NLA_NESTED },
 	[NHA_FDB]		= { .type = NLA_FLAG },
+	[NHA_FDB_PORT]		= NLA_POLICY_MIN(NLA_BE16, 1),
 	[NHA_RES_GROUP]		= { .type = NLA_NESTED },
 	[NHA_HW_STATS_ENABLE]	= NLA_POLICY_MAX(NLA_U32, true),
 };
@@ -956,6 +957,9 @@ static int nh_fill_node(struct sk_buff *skb, struct nexthop *nh,
 	} else if (nhi->fdb_nh) {
 		if (nla_put_flag(skb, NHA_FDB))
 			goto nla_put_failure;
+		if (nhi->fdb_port &&
+		    nla_put_be16(skb, NHA_FDB_PORT, nhi->fdb_port))
+			goto nla_put_failure;
 	} else {
 		const struct net_device *dev;
 
@@ -1055,6 +1059,9 @@ static size_t nh_nlmsg_size_single(struct nexthop *nh)
 		break;
 	}
 
+	if (nhi->fdb_port)
+		sz += nla_total_size(2);	/* NHA_FDB_PORT */
+
 	if (nhi->fib_nhc.nhc_lwtstate) {
 		sz += lwtunnel_get_encap_size(nhi->fib_nhc.nhc_lwtstate);
 		sz += nla_total_size(2);  /* NHA_ENCAP_TYPE */
@@ -2956,8 +2963,10 @@ static struct nexthop *nexthop_create(struct net *net, struct nh_config *cfg,
 	nhi->family = cfg->nh_family;
 	nhi->fib_nhc.nhc_scope = RT_SCOPE_LINK;
 
-	if (cfg->nh_fdb)
+	if (cfg->nh_fdb) {
 		nhi->fdb_nh = 1;
+		nhi->fdb_port = cfg->nh_fdb_port;
+	}
 
 	if (cfg->nh_blackhole) {
 		nhi->reject_nh = 1;
@@ -3147,6 +3156,15 @@ static int rtm_to_nh_config(struct net *net, struct sk_buff *skb,
 		cfg->nh_fdb = nla_get_flag(tb[NHA_FDB]);
 	}
 
+	if (tb[NHA_FDB_PORT]) {
+		if (!tb[NHA_FDB] || !tb[NHA_GATEWAY]) {
+			NL_SET_ERR_MSG(extack,
+				       "FDB port can only be set on fdb nexthops that have a gateway");
+			goto out;
+		}
+		cfg->nh_fdb_port = nla_get_be16(tb[NHA_FDB_PORT]);
+	}
+
 	if (tb[NHA_GROUP]) {
 		if (nhm->nh_family != AF_UNSPEC) {
 			NL_SET_ERR_MSG(extack, "Invalid family for group");

-- 
2.43.0

[PATCH net-next v3 2/3] vxlan: honor per-nexthop fdb destination port

[Jack Ma]

When an fdb entry points at a nexthop group, vxlan_fdb_nh_path_select()
resolves the selected leg's remote IP but leaves the UDP destination port
at the device default (vxlan->cfg.dst_port).

Extend nexthop_path_fdb_result() to also return the selected nexthop's
NHA_FDB_PORT (0 when unset) and have vxlan_fdb_nh_path_select() store it
in rdst->remote_port.  vxlan_xmit_one() already prefers rdst->remote_port
when non-zero and falls back to the device port otherwise, so nexthops
without a port are unaffected.

This lets one fdb nexthop group load-balance a flow across legs that
share an underlay IP but differ in UDP destination port.

Signed-off-by: Jack Ma <jack4it@gmail.com>
---
 include/net/nexthop.h | 4 +++-
 include/net/vxlan.h   | 5 ++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/include/net/nexthop.h b/include/net/nexthop.h
index 9c822799634f..19e8670d964d 100644
--- a/include/net/nexthop.h
+++ b/include/net/nexthop.h
@@ -576,7 +576,8 @@ struct fib_nh_common *nexthop_fdb_nhc(struct nexthop *nh)
 }
 
 static inline struct fib_nh_common *nexthop_path_fdb_result(struct nexthop *nh,
-							    int hash)
+							    int hash,
+							    __be16 *fdb_port)
 {
 	struct nh_info *nhi;
 	struct nexthop *nhp;
@@ -585,6 +586,7 @@ static inline struct fib_nh_common *nexthop_path_fdb_result(struct nexthop *nh,
 	if (unlikely(!nhp))
 		return NULL;
 	nhi = rcu_dereference(nhp->nh_info);
+	*fdb_port = nhi->fdb_port;
 	return &nhi->fib_nhc;
 }
 #endif
diff --git a/include/net/vxlan.h b/include/net/vxlan.h
index dfba89695efc..de41b374688f 100644
--- a/include/net/vxlan.h
+++ b/include/net/vxlan.h
@@ -567,8 +567,9 @@ static inline bool vxlan_fdb_nh_path_select(struct nexthop *nh,
 					    struct vxlan_rdst *rdst)
 {
 	struct fib_nh_common *nhc;
+	__be16 fdb_port = 0;
 
-	nhc = nexthop_path_fdb_result(nh, hash >> 1);
+	nhc = nexthop_path_fdb_result(nh, hash >> 1, &fdb_port);
 	if (unlikely(!nhc))
 		return false;
 
@@ -583,6 +584,8 @@ static inline bool vxlan_fdb_nh_path_select(struct nexthop *nh,
 		break;
 	}
 
+	rdst->remote_port = fdb_port;
+
 	return true;
 }
 

-- 
2.43.0

[PATCH net-next v3 3/3] selftests: net: add coverage for fdb nexthop dst port

[Jack Ma]

Extend the existing fdb nexthop group tests with cases for the new
per-nexthop VXLAN destination port (NHA_FDB_PORT).

In fib_nexthops.sh, ipv4_fdb_grp_fcnal() and ipv6_fdb_grp_fcnal() now
check that a port is accepted on an fdb nexthop that has a gateway and
echoed back on dump, that it is rejected without a gateway and rejected
when zero, that a group may hold legs that differ only in UDP port, and
that a portless fdb nexthop omits the attribute.  The cases SKIP when
iproute2 lacks the "port" keyword.

In test_vxlan_nh.sh, basic_tx_common() gains a second fdb nexthop group
whose nexthop carries a destination port that differs from the VXLAN
device default, plus a flower filter keyed on that port, to confirm the
per-nexthop port is used on the wire.

Signed-off-by: Jack Ma <jack4it@gmail.com>
---
 tools/testing/selftests/net/fib_nexthops.sh  | 59 ++++++++++++++++++++++++++++
 tools/testing/selftests/net/test_vxlan_nh.sh | 31 +++++++++++++++
 2 files changed, 90 insertions(+)

diff --git a/tools/testing/selftests/net/fib_nexthops.sh b/tools/testing/selftests/net/fib_nexthops.sh
index ac868a731694..1ce4f45fa651 100755
--- a/tools/testing/selftests/net/fib_nexthops.sh
+++ b/tools/testing/selftests/net/fib_nexthops.sh
@@ -432,6 +432,15 @@ check_nexthop_fdb_support()
 	fi
 }
 
+check_nexthop_fdb_port_support()
+{
+	$IP nexthop help 2>&1 | grep -q "fdb \[ port"
+	if [ $? -ne 0 ]; then
+		echo "SKIP: iproute2 too old, missing fdb nexthop port support"
+		return $ksft_skip
+	fi
+}
+
 check_nexthop_res_support()
 {
 	$IP nexthop help 2>&1 | grep -q resilient
@@ -514,6 +523,31 @@ ipv6_fdb_grp_fcnal()
 	run_cmd "$IP nexthop replace id 72 via 2001:db8:91::2 fdb"
 	log_test $? 2 "Replace non-FDB nexthop to FDB nexthop while in a group"
 
+	# NHA_FDB_PORT: optional per-nexthop VXLAN destination UDP port,
+	# letting an fdb nexthop group balance a flow across legs that share
+	# an underlay IP but listen on different UDP ports.
+	if check_nexthop_fdb_port_support; then
+		run_cmd "$IP nexthop add id 80 via 2001:db8:91::2 fdb port 4790"
+		check_nexthop "id 80" "id 80 via 2001:db8:91::2 scope link fdb port 4790"
+		log_test $? 0 "Fdb nexthop with port"
+
+		run_cmd "$IP nexthop add id 81 fdb port 4790"
+		log_test $? 2 "Fdb nexthop with port but no gateway"
+
+		run_cmd "$IP nexthop add id 81 via 2001:db8:91::2 fdb port 0"
+		log_test $? 2 "Fdb nexthop with port 0"
+
+		run_cmd "$IP nexthop add id 82 via 2001:db8:91::2 fdb port 4789"
+		run_cmd "$IP nexthop add id 83 via 2001:db8:91::3 fdb port 5789"
+		run_cmd "$IP nexthop add id 106 group 82/83 fdb"
+		check_nexthop "id 106" "id 106 group 82/83 fdb"
+		log_test $? 0 "Fdb nexthop group with legs differing in port"
+
+		run_cmd "$IP nexthop add id 84 via 2001:db8:91::2 fdb"
+		check_nexthop "id 84" "id 84 via 2001:db8:91::2 scope link fdb"
+		log_test $? 0 "Fdb nexthop without port omits port"
+	fi
+
 	run_cmd "$IP link add name vx10 type vxlan id 1010 local 2001:db8:91::9 remote 2001:db8:91::10 dstport 4789 nolearning noudpcsum tos inherit ttl 100"
 	run_cmd "$BRIDGE fdb add 02:02:00:00:00:13 dev vx10 nhid 102 self"
 	log_test $? 0 "Fdb mac add with nexthop group"
@@ -614,6 +648,31 @@ ipv4_fdb_grp_fcnal()
 	run_cmd "$IP nexthop replace id 20 via 172.16.1.2 fdb"
 	log_test $? 2 "Replace non-FDB nexthop to FDB nexthop while in a group"
 
+	# NHA_FDB_PORT: optional per-nexthop VXLAN destination UDP port,
+	# letting an fdb nexthop group balance a flow across legs that share
+	# an underlay IP but listen on different UDP ports.
+	if check_nexthop_fdb_port_support; then
+		run_cmd "$IP nexthop add id 30 via 172.16.1.2 fdb port 4790"
+		check_nexthop "id 30" "id 30 via 172.16.1.2 scope link fdb port 4790"
+		log_test $? 0 "Fdb nexthop with port"
+
+		run_cmd "$IP nexthop add id 31 fdb port 4790"
+		log_test $? 2 "Fdb nexthop with port but no gateway"
+
+		run_cmd "$IP nexthop add id 31 via 172.16.1.2 fdb port 0"
+		log_test $? 2 "Fdb nexthop with port 0"
+
+		run_cmd "$IP nexthop add id 32 via 172.16.1.2 fdb port 4789"
+		run_cmd "$IP nexthop add id 33 via 172.16.1.3 fdb port 5789"
+		run_cmd "$IP nexthop add id 105 group 32/33 fdb"
+		check_nexthop "id 105" "id 105 group 32/33 fdb"
+		log_test $? 0 "Fdb nexthop group with legs differing in port"
+
+		run_cmd "$IP nexthop add id 34 via 172.16.1.2 fdb"
+		check_nexthop "id 34" "id 34 via 172.16.1.2 scope link fdb"
+		log_test $? 0 "Fdb nexthop without port omits port"
+	fi
+
 	run_cmd "$IP link add name vx10 type vxlan id 1010 local 10.0.0.1 remote 10.0.0.2 dstport 4789 nolearning noudpcsum tos inherit ttl 100"
 	run_cmd "$BRIDGE fdb add 02:02:00:00:00:13 dev vx10 nhid 102 self"
 	log_test $? 0 "Fdb mac add with nexthop group"
diff --git a/tools/testing/selftests/net/test_vxlan_nh.sh b/tools/testing/selftests/net/test_vxlan_nh.sh
index 20f3369f776b..34a24a4f95bf 100755
--- a/tools/testing/selftests/net/test_vxlan_nh.sh
+++ b/tools/testing/selftests/net/test_vxlan_nh.sh
@@ -56,6 +56,16 @@ tc_stats_get()
 	tc_rule_handle_stats_get "dev dummy1 egress" 101 ".packets" "-n $ns1"
 }
 
+nh_stats_get_port()
+{
+	ip -n "$ns1" -s -j nexthop show id 20 | jq ".[][\"group_stats\"][][\"packets\"]"
+}
+
+tc_stats_get_port()
+{
+	tc_rule_handle_stats_get "dev dummy1 egress" 102 ".packets" "-n $ns1"
+}
+
 basic_tx_common()
 {
 	local af_str=$1; shift
@@ -90,6 +100,27 @@ basic_tx_common()
 	busywait "$BUSYWAIT_TIMEOUT" until_counter_is "== 1" tc_stats_get > /dev/null
 	check_err $? "tc filter stats did not increase"
 
+	# Add a second FDB nexthop group whose nexthop carries a per-nexthop
+	# destination port (NHA_FDB_PORT) that differs from the VXLAN device
+	# default. Matching outer traffic must egress with that port, so a
+	# separate flower filter keyed on the new port catches it.
+	if ip nexthop help 2>&1 | grep -q "fdb \[ port"; then
+		run_cmd "tc -n $ns1 filter add dev dummy1 egress proto $proto pref 1 handle 102 flower ip_proto udp dst_ip $remote_addr dst_port 4790 action pass"
+
+		run_cmd "ip -n $ns1 nexthop add id 2 via $remote_addr fdb port 4790"
+		run_cmd "ip -n $ns1 nexthop add id 20 group 2 fdb"
+
+		run_cmd "bridge -n $ns1 fdb add 00:11:22:33:44:66 dev vx0 self static nhid 20"
+
+		run_cmd "ip netns exec $ns1 mausezahn vx0 -a own -b 00:11:22:33:44:66 -c 1 -q"
+
+		busywait "$BUSYWAIT_TIMEOUT" until_counter_is "== 1" nh_stats_get_port > /dev/null
+		check_err $? "FDB nexthop group stats did not increase (with port)"
+
+		busywait "$BUSYWAIT_TIMEOUT" until_counter_is "== 1" tc_stats_get_port > /dev/null
+		check_err $? "tc filter stats did not increase (with port)"
+	fi
+
 	log_test "VXLAN FDB nexthop: $af_str basic Tx"
 }
 

-- 
2.43.0

Re: [PATCH net-next v3 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops

[Jakub Kicinski]

On Tue, 21 Jul 2026 22:29:49 +0000 Jack Ma wrote:
> A matching iproute2 change (the `ip nexthop ... fdb port N` keyword) is posted
> separately to the iproute2 list.

Where is that?

Re: [PATCH net-next v3 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops

[Jack Ma]

On Tue, 21 Jul 2026 18:05:25 -0700 Jakub Kicinski wrote:
> On Tue, 21 Jul 2026 22:29:49 +0000 Jack Ma wrote:
> > A matching iproute2 change (the `ip nexthop ... fdb port N` keyword) is posted
> > separately to the iproute2 list.
>
> Where is that?

Apologies -- that was premature wording in the cover letter; the iproute2
patch had not actually gone out yet.  It is posted now:

  [PATCH iproute2-next] ip nexthop: support fdb destination port
  https://lore.kernel.org/netdev/20260722-b4-vxlan-fdb-port-iproute2-v1-1-45199b1ee240@gmail.com/

Thanks,
Jack

Re: [PATCH net-next v3 1/3] net: nexthop: add NHA_FDB_PORT for fdb nexthops

[Ido Schimmel]

On Tue, Jul 21, 2026 at 10:29:50PM +0000, Jack Ma wrote:
> Commit 1274e1cc4226 ("vxlan: ecmp support for mac fdb entries") lets a
> single inner MAC be reached through a group of remote VTEPs, with the
> kernel flow-hashing across the group members.  Each member carries its
> own remote IP, but the UDP destination port is always taken from the
> VXLAN device (vxlan->cfg.dst_port) and cannot be set per member.
> 
> Some deployments pack several receivers behind one underlay IP and tell
> them apart by UDP port, so they need a per-nexthop destination port to
> spread flows across (IP, port) tuples rather than IP alone.
> 
> Add a netlink attribute NHA_FDB_PORT (__be16, mirroring NDA_PORT) that
> carries an optional UDP destination port on an fdb nexthop.  It is only
> accepted together with NHA_FDB and NHA_GATEWAY; it is stored in struct
> nh_info and echoed back on dump.  This patch is control-plane plumbing
> only; the VXLAN datapath is wired up in a follow-up patch, so behaviour
> is unchanged for now.

I looked more closely at the test and the iproute2 patch and wanted to
comment that "port" seems too generic given that the attribute is
specific to FDB nexthops and called NHA_FDB_PORT. It made me think that
maybe we should use a more generic attribute name (e.g., NHA_DST_PORT)
and a corresponding "dst_port" keyword in iproute2. We can still limit
this to FDB nexthops during validation, but in the unlikely case that we
will need a destination port for other nexthops, we could just reuse
this attribute. It's not like we have NHA_FDB_GATEWAY.

David, WDYT?

> 
> Signed-off-by: Jack Ma <jack4it@gmail.com>
> ---
>  include/net/nexthop.h        |  2 ++
>  include/uapi/linux/nexthop.h |  3 +++
>  net/ipv4/nexthop.c           | 20 +++++++++++++++++++-
>  3 files changed, 24 insertions(+), 1 deletion(-)
> 
> diff --git a/include/net/nexthop.h b/include/net/nexthop.h
> index 572e69cda476..9c822799634f 100644
> --- a/include/net/nexthop.h
> +++ b/include/net/nexthop.h
> @@ -28,6 +28,7 @@ struct nh_config {
>  	u8		nh_protocol;
>  	u8		nh_blackhole;
>  	u8		nh_fdb;
> +	__be16		nh_fdb_port;

This will be nh_dst_port and similarly in other places throughout the
patchset

>  	u32		nh_flags;
>  
>  	int		nh_ifindex;

Re: [PATCH net-next v3 3/3] selftests: net: add coverage for fdb nexthop dst port

[Ido Schimmel]

On Tue, Jul 21, 2026 at 10:29:52PM +0000, Jack Ma wrote:
> Extend the existing fdb nexthop group tests with cases for the new
> per-nexthop VXLAN destination port (NHA_FDB_PORT).
> 
> In fib_nexthops.sh, ipv4_fdb_grp_fcnal() and ipv6_fdb_grp_fcnal() now
> check that a port is accepted on an fdb nexthop that has a gateway and
> echoed back on dump, that it is rejected without a gateway and rejected
> when zero, that a group may hold legs that differ only in UDP port, and
> that a portless fdb nexthop omits the attribute.  The cases SKIP when
> iproute2 lacks the "port" keyword.
> 
> In test_vxlan_nh.sh, basic_tx_common() gains a second fdb nexthop group
> whose nexthop carries a destination port that differs from the VXLAN
> device default, plus a flower filter keyed on that port, to confirm the
> per-nexthop port is used on the wire.
> 
> Signed-off-by: Jack Ma <jack4it@gmail.com>
> ---
>  tools/testing/selftests/net/fib_nexthops.sh  | 59 ++++++++++++++++++++++++++++
>  tools/testing/selftests/net/test_vxlan_nh.sh | 31 +++++++++++++++
>  2 files changed, 90 insertions(+)
> 
> diff --git a/tools/testing/selftests/net/fib_nexthops.sh b/tools/testing/selftests/net/fib_nexthops.sh
> index ac868a731694..1ce4f45fa651 100755
> --- a/tools/testing/selftests/net/fib_nexthops.sh
> +++ b/tools/testing/selftests/net/fib_nexthops.sh
> @@ -432,6 +432,15 @@ check_nexthop_fdb_support()
>  	fi
>  }
>  
> +check_nexthop_fdb_port_support()
> +{
> +	$IP nexthop help 2>&1 | grep -q "fdb \[ port"
> +	if [ $? -ne 0 ]; then
> +		echo "SKIP: iproute2 too old, missing fdb nexthop port support"
> +		return $ksft_skip
> +	fi
> +}
> +
>  check_nexthop_res_support()
>  {
>  	$IP nexthop help 2>&1 | grep -q resilient
> @@ -514,6 +523,31 @@ ipv6_fdb_grp_fcnal()
>  	run_cmd "$IP nexthop replace id 72 via 2001:db8:91::2 fdb"
>  	log_test $? 2 "Replace non-FDB nexthop to FDB nexthop while in a group"
>  
> +	# NHA_FDB_PORT: optional per-nexthop VXLAN destination UDP port,
> +	# letting an fdb nexthop group balance a flow across legs that share
> +	# an underlay IP but listen on different UDP ports.
> +	if check_nexthop_fdb_port_support; then
> +		run_cmd "$IP nexthop add id 80 via 2001:db8:91::2 fdb port 4790"
> +		check_nexthop "id 80" "id 80 via 2001:db8:91::2 scope link fdb port 4790"
> +		log_test $? 0 "Fdb nexthop with port"
> +
> +		run_cmd "$IP nexthop add id 81 fdb port 4790"
> +		log_test $? 2 "Fdb nexthop with port but no gateway"
> +
> +		run_cmd "$IP nexthop add id 81 via 2001:db8:91::2 fdb port 0"
> +		log_test $? 2 "Fdb nexthop with port 0"
> +
> +		run_cmd "$IP nexthop add id 82 via 2001:db8:91::2 fdb port 4789"
> +		run_cmd "$IP nexthop add id 83 via 2001:db8:91::3 fdb port 5789"
> +		run_cmd "$IP nexthop add id 106 group 82/83 fdb"
> +		check_nexthop "id 106" "id 106 group 82/83 fdb"
> +		log_test $? 0 "Fdb nexthop group with legs differing in port"
> +
> +		run_cmd "$IP nexthop add id 84 via 2001:db8:91::2 fdb"
> +		check_nexthop "id 84" "id 84 via 2001:db8:91::2 scope link fdb"
> +		log_test $? 0 "Fdb nexthop without port omits port"
> +	fi
> +
>  	run_cmd "$IP link add name vx10 type vxlan id 1010 local 2001:db8:91::9 remote 2001:db8:91::10 dstport 4789 nolearning noudpcsum tos inherit ttl 100"
>  	run_cmd "$BRIDGE fdb add 02:02:00:00:00:13 dev vx10 nhid 102 self"
>  	log_test $? 0 "Fdb mac add with nexthop group"
> @@ -614,6 +648,31 @@ ipv4_fdb_grp_fcnal()
>  	run_cmd "$IP nexthop replace id 20 via 172.16.1.2 fdb"
>  	log_test $? 2 "Replace non-FDB nexthop to FDB nexthop while in a group"
>  
> +	# NHA_FDB_PORT: optional per-nexthop VXLAN destination UDP port,
> +	# letting an fdb nexthop group balance a flow across legs that share
> +	# an underlay IP but listen on different UDP ports.
> +	if check_nexthop_fdb_port_support; then
> +		run_cmd "$IP nexthop add id 30 via 172.16.1.2 fdb port 4790"
> +		check_nexthop "id 30" "id 30 via 172.16.1.2 scope link fdb port 4790"
> +		log_test $? 0 "Fdb nexthop with port"
> +
> +		run_cmd "$IP nexthop add id 31 fdb port 4790"
> +		log_test $? 2 "Fdb nexthop with port but no gateway"
> +
> +		run_cmd "$IP nexthop add id 31 via 172.16.1.2 fdb port 0"
> +		log_test $? 2 "Fdb nexthop with port 0"
> +
> +		run_cmd "$IP nexthop add id 32 via 172.16.1.2 fdb port 4789"
> +		run_cmd "$IP nexthop add id 33 via 172.16.1.3 fdb port 5789"
> +		run_cmd "$IP nexthop add id 105 group 32/33 fdb"
> +		check_nexthop "id 105" "id 105 group 32/33 fdb"
> +		log_test $? 0 "Fdb nexthop group with legs differing in port"
> +
> +		run_cmd "$IP nexthop add id 34 via 172.16.1.2 fdb"
> +		check_nexthop "id 34" "id 34 via 172.16.1.2 scope link fdb"
> +		log_test $? 0 "Fdb nexthop without port omits port"
> +	fi

The indentation is sub-optimal. Maybe move the tests to
ipv4_fdb_port_fcnal() and ipv6_fdb_port_fcnal() (add to IPV4_TESTS and
IPV6_TESTS) that will be skipped if iproute2 is too old?

> +
>  	run_cmd "$IP link add name vx10 type vxlan id 1010 local 10.0.0.1 remote 10.0.0.2 dstport 4789 nolearning noudpcsum tos inherit ttl 100"
>  	run_cmd "$BRIDGE fdb add 02:02:00:00:00:13 dev vx10 nhid 102 self"
>  	log_test $? 0 "Fdb mac add with nexthop group"
> diff --git a/tools/testing/selftests/net/test_vxlan_nh.sh b/tools/testing/selftests/net/test_vxlan_nh.sh
> index 20f3369f776b..34a24a4f95bf 100755
> --- a/tools/testing/selftests/net/test_vxlan_nh.sh
> +++ b/tools/testing/selftests/net/test_vxlan_nh.sh
> @@ -56,6 +56,16 @@ tc_stats_get()
>  	tc_rule_handle_stats_get "dev dummy1 egress" 101 ".packets" "-n $ns1"
>  }
>  
> +nh_stats_get_port()
> +{
> +	ip -n "$ns1" -s -j nexthop show id 20 | jq ".[][\"group_stats\"][][\"packets\"]"
> +}
> +
> +tc_stats_get_port()
> +{
> +	tc_rule_handle_stats_get "dev dummy1 egress" 102 ".packets" "-n $ns1"
> +}
> +
>  basic_tx_common()
>  {
>  	local af_str=$1; shift
> @@ -90,6 +100,27 @@ basic_tx_common()
>  	busywait "$BUSYWAIT_TIMEOUT" until_counter_is "== 1" tc_stats_get > /dev/null
>  	check_err $? "tc filter stats did not increase"
>  
> +	# Add a second FDB nexthop group whose nexthop carries a per-nexthop
> +	# destination port (NHA_FDB_PORT) that differs from the VXLAN device
> +	# default. Matching outer traffic must egress with that port, so a
> +	# separate flower filter keyed on the new port catches it.
> +	if ip nexthop help 2>&1 | grep -q "fdb \[ port"; then
> +		run_cmd "tc -n $ns1 filter add dev dummy1 egress proto $proto pref 1 handle 102 flower ip_proto udp dst_ip $remote_addr dst_port 4790 action pass"
> +
> +		run_cmd "ip -n $ns1 nexthop add id 2 via $remote_addr fdb port 4790"
> +		run_cmd "ip -n $ns1 nexthop add id 20 group 2 fdb"
> +
> +		run_cmd "bridge -n $ns1 fdb add 00:11:22:33:44:66 dev vx0 self static nhid 20"
> +
> +		run_cmd "ip netns exec $ns1 mausezahn vx0 -a own -b 00:11:22:33:44:66 -c 1 -q"
> +
> +		busywait "$BUSYWAIT_TIMEOUT" until_counter_is "== 1" nh_stats_get_port > /dev/null
> +		check_err $? "FDB nexthop group stats did not increase (with port)"
> +
> +		busywait "$BUSYWAIT_TIMEOUT" until_counter_is "== 1" tc_stats_get_port > /dev/null
> +		check_err $? "tc filter stats did not increase (with port)"
> +	fi
> +
>  	log_test "VXLAN FDB nexthop: $af_str basic Tx"
>  }

This file only tests VXLAN with FDB nexthops so we can just do:

diff --git a/tools/testing/selftests/net/test_vxlan_nh.sh b/tools/testing/selftests/net/test_vxlan_nh.sh
index 34a24a4f95bf..ca905e82b7ca 100755
--- a/tools/testing/selftests/net/test_vxlan_nh.sh
+++ b/tools/testing/selftests/net/test_vxlan_nh.sh
@@ -241,8 +241,8 @@ require_command arping
 require_command ndisc6
 require_command jq
 
-if ! ip nexthop help 2>&1 | grep -q "stats"; then
-	echo "SKIP: iproute2 ip too old, missing nexthop stats support"
+if ! ip nexthop help 2>&1 | grep -q "dst_port"; then
+	echo "SKIP: iproute2 ip too old, missing nexthop dst_port support"
 	exit "$ksft_skip"
 fi