* [PATCH net-next 1/3] net: nexthop: add NHA_FDB_PORT for fdb nexthops
2026-07-12 19:12 [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops Jack Ma
@ 2026-07-12 19:12 ` Jack Ma
2026-07-12 19:12 ` [PATCH net-next 2/3] vxlan: honor per-nexthop fdb destination port Jack Ma
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Jack Ma @ 2026-07-12 19:12 UTC (permalink / raw)
To: netdev, David Ahern
Cc: Ido Schimmel, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Shuah Khan, linux-kselftest,
linux-kernel, Jack Ma
Commit 1274e1cc4226 ("vxlan: ecmp support for mac fdb entries") lets a
single inner MAC be reached through a group of remote VTEPs, with the
kernel flow-hashing across the group members. Each member carries its
own remote IP, but the UDP destination port is always taken from the
VXLAN device (vxlan->cfg.dst_port) and cannot be set per member.
Some deployments pack several receivers behind one underlay IP and tell
them apart by UDP port, so they need a per-nexthop destination port to
spread flows across (IP, port) tuples rather than IP alone.
Add a netlink attribute NHA_FDB_PORT (__be16, mirroring NDA_PORT) that
carries an optional UDP destination port on an fdb nexthop. It is only
accepted together with NHA_FDB and NHA_GATEWAY; it is stored in struct
nh_info and echoed back on dump. This patch is control-plane plumbing
only; the VXLAN datapath is wired up in a follow-up patch, so behaviour
is unchanged for now.
Signed-off-by: Jack Ma <jack4it@gmail.com>
---
include/net/nexthop.h | 2 ++
include/uapi/linux/nexthop.h | 3 +++
net/ipv4/nexthop.c | 20 +++++++++++++++++++-
3 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/include/net/nexthop.h b/include/net/nexthop.h
index 572e69cda..9c8227996 100644
--- a/include/net/nexthop.h
+++ b/include/net/nexthop.h
@@ -28,6 +28,7 @@ struct nh_config {
u8 nh_protocol;
u8 nh_blackhole;
u8 nh_fdb;
+ __be16 nh_fdb_port;
u32 nh_flags;
int nh_ifindex;
@@ -63,6 +64,7 @@ struct nh_info {
u8 family;
bool reject_nh;
bool fdb_nh;
+ __be16 fdb_port;
union {
struct fib_nh_common fib_nhc;
diff --git a/include/uapi/linux/nexthop.h b/include/uapi/linux/nexthop.h
index bc49baf4a..e587bbf3b 100644
--- a/include/uapi/linux/nexthop.h
+++ b/include/uapi/linux/nexthop.h
@@ -83,6 +83,9 @@ enum {
/* u32; read-only; whether any driver collects HW stats */
NHA_HW_STATS_USED,
+ /* be16; UDP destination port for an fdb nexthop (e.g. VXLAN) */
+ NHA_FDB_PORT,
+
__NHA_MAX,
};
diff --git a/net/ipv4/nexthop.c b/net/ipv4/nexthop.c
index 6205bd57a..5b27cc9a9 100644
--- a/net/ipv4/nexthop.c
+++ b/net/ipv4/nexthop.c
@@ -39,6 +39,7 @@ static const struct nla_policy rtm_nh_policy_new[] = {
[NHA_ENCAP_TYPE] = { .type = NLA_U16 },
[NHA_ENCAP] = { .type = NLA_NESTED },
[NHA_FDB] = { .type = NLA_FLAG },
+ [NHA_FDB_PORT] = { .type = NLA_U16 },
[NHA_RES_GROUP] = { .type = NLA_NESTED },
[NHA_HW_STATS_ENABLE] = NLA_POLICY_MAX(NLA_U32, true),
};
@@ -956,6 +957,9 @@ static int nh_fill_node(struct sk_buff *skb, struct nexthop *nh,
} else if (nhi->fdb_nh) {
if (nla_put_flag(skb, NHA_FDB))
goto nla_put_failure;
+ if (nhi->fdb_port &&
+ nla_put_be16(skb, NHA_FDB_PORT, nhi->fdb_port))
+ goto nla_put_failure;
} else {
const struct net_device *dev;
@@ -1055,6 +1059,9 @@ static size_t nh_nlmsg_size_single(struct nexthop *nh)
break;
}
+ if (nhi->fdb_nh)
+ sz += nla_total_size(2); /* NHA_FDB_PORT */
+
if (nhi->fib_nhc.nhc_lwtstate) {
sz += lwtunnel_get_encap_size(nhi->fib_nhc.nhc_lwtstate);
sz += nla_total_size(2); /* NHA_ENCAP_TYPE */
@@ -2956,8 +2963,10 @@ static struct nexthop *nexthop_create(struct net *net, struct nh_config *cfg,
nhi->family = cfg->nh_family;
nhi->fib_nhc.nhc_scope = RT_SCOPE_LINK;
- if (cfg->nh_fdb)
+ if (cfg->nh_fdb) {
nhi->fdb_nh = 1;
+ nhi->fdb_port = cfg->nh_fdb_port;
+ }
if (cfg->nh_blackhole) {
nhi->reject_nh = 1;
@@ -3147,6 +3156,15 @@ static int rtm_to_nh_config(struct net *net, struct sk_buff *skb,
cfg->nh_fdb = nla_get_flag(tb[NHA_FDB]);
}
+ if (tb[NHA_FDB_PORT]) {
+ if (!tb[NHA_FDB] || !tb[NHA_GATEWAY]) {
+ NL_SET_ERR_MSG(extack,
+ "FDB port can only be set on fdb nexthops that have a gateway");
+ goto out;
+ }
+ cfg->nh_fdb_port = nla_get_be16(tb[NHA_FDB_PORT]);
+ }
+
if (tb[NHA_GROUP]) {
if (nhm->nh_family != AF_UNSPEC) {
NL_SET_ERR_MSG(extack, "Invalid family for group");
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH net-next 2/3] vxlan: honor per-nexthop fdb destination port
2026-07-12 19:12 [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops Jack Ma
2026-07-12 19:12 ` [PATCH net-next 1/3] net: nexthop: add NHA_FDB_PORT for fdb nexthops Jack Ma
@ 2026-07-12 19:12 ` Jack Ma
2026-07-12 19:12 ` [PATCH net-next 3/3] selftests: net: add coverage for fdb nexthop dst port Jack Ma
2026-07-13 12:23 ` [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops Ido Schimmel
3 siblings, 0 replies; 6+ messages in thread
From: Jack Ma @ 2026-07-12 19:12 UTC (permalink / raw)
To: netdev, David Ahern
Cc: Ido Schimmel, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Shuah Khan, linux-kselftest,
linux-kernel, Jack Ma
When an fdb entry points at a nexthop group, vxlan_fdb_nh_path_select()
resolves the selected leg's remote IP but leaves the UDP destination port
at the device default (vxlan->cfg.dst_port).
Extend nexthop_path_fdb_result() to also return the selected nexthop's
NHA_FDB_PORT (0 when unset) and have vxlan_fdb_nh_path_select() store it
in rdst->remote_port. vxlan_xmit_one() already prefers rdst->remote_port
when non-zero and falls back to the device port otherwise, so nexthops
without a port are unaffected.
This lets one fdb nexthop group load-balance a flow across legs that
share an underlay IP but differ in UDP destination port.
Signed-off-by: Jack Ma <jack4it@gmail.com>
---
include/net/nexthop.h | 5 ++++-
include/net/vxlan.h | 5 ++++-
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/include/net/nexthop.h b/include/net/nexthop.h
index 9c8227996..7fb612a73 100644
--- a/include/net/nexthop.h
+++ b/include/net/nexthop.h
@@ -576,7 +576,8 @@ struct fib_nh_common *nexthop_fdb_nhc(struct nexthop *nh)
}
static inline struct fib_nh_common *nexthop_path_fdb_result(struct nexthop *nh,
- int hash)
+ int hash,
+ __be16 *fdb_port)
{
struct nh_info *nhi;
struct nexthop *nhp;
@@ -585,6 +586,8 @@ static inline struct fib_nh_common *nexthop_path_fdb_result(struct nexthop *nh,
if (unlikely(!nhp))
return NULL;
nhi = rcu_dereference(nhp->nh_info);
+ if (fdb_port)
+ *fdb_port = nhi->fdb_port;
return &nhi->fib_nhc;
}
#endif
diff --git a/include/net/vxlan.h b/include/net/vxlan.h
index dfba89695..de41b3746 100644
--- a/include/net/vxlan.h
+++ b/include/net/vxlan.h
@@ -567,8 +567,9 @@ static inline bool vxlan_fdb_nh_path_select(struct nexthop *nh,
struct vxlan_rdst *rdst)
{
struct fib_nh_common *nhc;
+ __be16 fdb_port = 0;
- nhc = nexthop_path_fdb_result(nh, hash >> 1);
+ nhc = nexthop_path_fdb_result(nh, hash >> 1, &fdb_port);
if (unlikely(!nhc))
return false;
@@ -583,6 +584,8 @@ static inline bool vxlan_fdb_nh_path_select(struct nexthop *nh,
break;
}
+ rdst->remote_port = fdb_port;
+
return true;
}
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH net-next 3/3] selftests: net: add coverage for fdb nexthop dst port
2026-07-12 19:12 [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops Jack Ma
2026-07-12 19:12 ` [PATCH net-next 1/3] net: nexthop: add NHA_FDB_PORT for fdb nexthops Jack Ma
2026-07-12 19:12 ` [PATCH net-next 2/3] vxlan: honor per-nexthop fdb destination port Jack Ma
@ 2026-07-12 19:12 ` Jack Ma
2026-07-13 12:23 ` [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops Ido Schimmel
3 siblings, 0 replies; 6+ messages in thread
From: Jack Ma @ 2026-07-12 19:12 UTC (permalink / raw)
To: netdev, David Ahern
Cc: Ido Schimmel, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Shuah Khan, linux-kselftest,
linux-kernel, Jack Ma
Add fib_nexthops_fdb_port.sh, which exercises the NHA_FDB_PORT rules:
accept a port on an fdb nexthop that has a gateway and echo it back on
dump, reject it on non-fdb or gateway-less nexthops, allow a group whose
legs differ only in UDP port, and confirm a portless fdb nexthop omits
the attribute. The test SKIPs cleanly on kernels or iproute2 without
NHA_FDB_PORT support.
Signed-off-by: Jack Ma <jack4it@gmail.com>
---
tools/testing/selftests/net/Makefile | 1 +
.../selftests/net/fib_nexthops_fdb_port.sh | 78 +++++++++++++++++++
2 files changed, 79 insertions(+)
create mode 100755 tools/testing/selftests/net/fib_nexthops_fdb_port.sh
diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index 708d960ae..c06eb4927 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -36,6 +36,7 @@ TEST_PROGS := \
fib_nexthop_multiprefix.sh \
fib_nexthop_nongw.sh \
fib_nexthops.sh \
+ fib_nexthops_fdb_port.sh \
fib_rule_tests.sh \
fib_tests.sh \
fin_ack_lat.sh \
diff --git a/tools/testing/selftests/net/fib_nexthops_fdb_port.sh b/tools/testing/selftests/net/fib_nexthops_fdb_port.sh
new file mode 100755
index 000000000..8b401c6d2
--- /dev/null
+++ b/tools/testing/selftests/net/fib_nexthops_fdb_port.sh
@@ -0,0 +1,78 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Control-plane selftest for per-nexthop VXLAN fdb destination port
+# (NHA_FDB_PORT). Verifies the accept/reject rules and the dump roundtrip.
+# No datapath traffic here -- see tests/integ/vxlan-fdb-port-integ.sh for the
+# real forwarding test.
+#
+# Requires: patched kernel (NHA_FDB_PORT) and patched iproute2 (the "port"
+# keyword on "ip nexthop ... fdb"). SKIPs cleanly otherwise.
+
+set -u
+
+ksft_skip=4
+NS="nhfdbport-$$"
+IP="ip -netns $NS"
+ret=0
+
+log_test() { # $1 actual_rc $2 expected_rc $3 name
+ if [ "$1" = "$2" ]; then
+ printf "TEST: %-58s [ OK ]\n" "$3"
+ else
+ printf "TEST: %-58s [FAIL] (rc=$1 want=$2)\n" "$3"
+ ret=1
+ fi
+}
+
+# passes (returns 0) iff the command FAILS
+expect_fail() {
+ if "$@" >/dev/null 2>&1; then return 1; else return 0; fi
+}
+
+cleanup() { ip netns del "$NS" 2>/dev/null; }
+
+command -v ip >/dev/null 2>&1 || { echo "SKIP: iproute2 not found"; exit $ksft_skip; }
+ip nexthop help 2>&1 | grep -q fdb || { echo "SKIP: no fdb nexthop support"; exit $ksft_skip; }
+
+trap cleanup EXIT
+cleanup
+ip netns add "$NS" || { echo "SKIP: cannot create netns"; exit $ksft_skip; }
+$IP link set lo up
+
+# Probe for "port" keyword + kernel NHA_FDB_PORT support; SKIP if missing.
+if ! $IP nexthop add id 1 via 10.0.0.1 fdb port 4790 2>/dev/null; then
+ echo "SKIP: 'ip nexthop ... fdb port' unsupported (needs patched kernel + iproute2)"
+ exit $ksft_skip
+fi
+log_test 0 0 "add fdb nexthop with port"
+
+# Dump roundtrip must echo the port back.
+$IP nexthop show id 1 | grep -qw "port 4790"
+log_test $? 0 "dump shows fdb port 4790"
+
+# Reject: port on a routed (non-fdb) nexthop.
+expect_fail $IP nexthop add id 2 via 10.0.0.1 dev lo port 4790
+log_test $? 0 "reject port on non-fdb nexthop"
+
+# Reject: fdb port without a gateway.
+expect_fail $IP nexthop add id 3 fdb port 4790
+log_test $? 0 "reject fdb port without gateway"
+
+# The HA case: a group whose legs share the gateway but differ in port.
+$IP nexthop add id 10 via 10.0.0.1 fdb port 4789 && \
+$IP nexthop add id 11 via 10.0.0.1 fdb port 5789 && \
+$IP nexthop add id 100 group 10/11 fdb
+log_test $? 0 "add fdb nexthop group with differing ports"
+
+# A fdb nexthop without a port must NOT emit one (backward compat).
+$IP nexthop add id 20 via 10.0.0.1 fdb
+$IP nexthop show id 20 | grep -qw "port"
+log_test $? 1 "fdb nexthop without port omits NHA_FDB_PORT"
+
+if [ $ret -eq 0 ]; then
+ echo "PASS: all NHA_FDB_PORT control-plane checks"
+else
+ echo "FAIL: one or more NHA_FDB_PORT checks failed"
+fi
+exit $ret
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops
2026-07-12 19:12 [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops Jack Ma
` (2 preceding siblings ...)
2026-07-12 19:12 ` [PATCH net-next 3/3] selftests: net: add coverage for fdb nexthop dst port Jack Ma
@ 2026-07-13 12:23 ` Ido Schimmel
2026-07-14 3:43 ` Jack Ma
3 siblings, 1 reply; 6+ messages in thread
From: Ido Schimmel @ 2026-07-13 12:23 UTC (permalink / raw)
To: Jack Ma
Cc: netdev, David Ahern, David S . Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Shuah Khan,
linux-kselftest, linux-kernel
On Sun, Jul 12, 2026 at 07:12:15PM +0000, Jack Ma wrote:
> FDB nexthops let a VXLAN fdb entry point at a group of remote VTEPs, with the
> kernel flow-hashing across the group (commit 1274e1cc4226 ("vxlan: ecmp support
> for mac fdb entries")). Each leg carries its own remote IP, but the UDP
> destination port is always taken from the VXLAN device (vxlan->cfg.dst_port)
> and cannot be set per leg.
>
> Some deployments pack several receivers behind a single underlay IP and tell
> them apart by UDP destination port. To spread flows across such receivers they
I don't understand the "tell them apart" phrasing. Aren't all of these
receivers interchangeable given you are load balancing between them?
> need a nexthop group whose legs share the remote IP but differ in UDP port,
> which is not currently expressible.
I think you will need to provide more details about the use case given
you are asking us to accept new uAPI which we will need to maintain
forever. The FDB nexthop groups make sense when you want to spread
overlay traffic between several distinct VTEPs. In the EVPN-MH use case
these VTEPs are different TOR switches. In your use case, why can't the
load balancing happen at the target host (e.g., using XDP / TC /
SO_REUSEPORT / flow classification)?
To be clear, I'm not strictly against the idea (we can talk about the
implementation later), but I would like to understand why your use case
can't be addressed using existing mechanisms (or why it's undesirable).
Thanks
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops
2026-07-13 12:23 ` [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops Ido Schimmel
@ 2026-07-14 3:43 ` Jack Ma
0 siblings, 0 replies; 6+ messages in thread
From: Jack Ma @ 2026-07-14 3:43 UTC (permalink / raw)
To: Ido Schimmel, netdev
Cc: David Ahern, David S . Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Shuah Khan, linux-kselftest,
linux-kernel, Jack Ma
On Mon, Jul 13, 2026 at 03:23:39PM +0300, Ido Schimmel wrote:
> > Some deployments pack several receivers behind a single underlay IP and tell
> > them apart by UDP destination port. To spread flows across such receivers they
>
> I don't understand the "tell them apart" phrasing. Aren't all of these
> receivers interchangeable given you are load balancing between them?
Sorry, that wording conflated two separate things. The load-balancing
targets are interchangeable; the UDP port is not what makes them
distinct services, it is how the underlay addresses each one:
- Addressing: several VTEPs are packed behind one underlay IP, each
reachable at a distinct UDP destination port. The port is the
underlay demux that delivers an encapsulated packet to the right
VTEP -- it selects *where*, not *which service*.
- Load-balancing: when those VTEPs form an HA set (shared inner
MAC/IP), a sender spreads flows across them with an fdb nexthop
group. The legs are interchangeable, but each one lives at a
different (underlay IP, UDP port).
So the legs are interchangeable endpoints that happen to sit at
different (IP, port) addresses. A group can already carry a distinct IP
per leg but not a distinct port, which is the gap here. I'll reword the
cover and patch 1 to say this explicitly.
> I think you will need to provide more details about the use case [...]
> In your use case, why can't the load balancing happen at the target
> host (e.g., using XDP / TC / SO_REUSEPORT / flow classification)?
I think the key difference is that there isn't really a shared
host-side datapath to load-balance in, and the UDP port is doing demux
rather than load-balancing. The shape is a bit unusual, so maybe a
couple of pictures help.
First, how forwarders sit on a receiver node. A forwarder here is
just the pod that terminates the overlay tunnel (it hosts the VXLAN
VTEP) and relays traffic to and from one tenant's workload. Many such
pods share one mesh-routable underlay IP, each demuxed by UDP port:
receiver node -- one mesh-routable underlay IP (NodeIP_A)
+----------------------------------------------------+
| host netns: stateless outer-UDP demux by dst port |
| (host does NOT terminate the tunnel) |
| |
| dst :40000 dst :40001 dst :40002 |
| | | | |
| +-----v----+ +-----v----+ +-----v----+ |
| | pod0 ns | | pod1 ns | | pod2 ns | |
| | vxlan | | vxlan | | vxlan | |
| | VTEP | | VTEP | | VTEP | |
| | decap | | decap | | decap | |
| +----------+ +----------+ +----------+ |
+----------------------------------------------------+
(up to ~10 forwarder pods packed per node)
These packed pods are unrelated -- different tenants on different
VNIs (below) -- so the per-pod UDP port is node-level demux, not an
HA construct. A single forwarder's HA replicas are the orthogonal
axis: anti-affinity spreads them across nodes (never co-resident),
so one member's nexthop group has its legs on distinct node IPs.
But each leg is still reachable only at (node IP, that pod's UDP
port), so within one group the legs differ in IP *and* port. A
group can already carry a distinct IP per leg, but it takes the UDP
port from the device (a single value), so it can't send each leg to
its own port. That last part is really the gap we keep running into.
The pods sharing a node need not be related: each one belongs to a
separate tenant on its own VXLAN VNI, and a node can pack forwarders
for many of them, each on its own UDP port. That per-pod port is what
lets different tenants co-locate without colliding, and the host --
which only demuxes outer UDP -- never has to reason about tenancy.
Zooming into one forwarder pod, there isn't really anything to
load-balance on the host: the tunnel terminates on a vxlan device
inside the pod's own netns, and the pod reaches the customer through a
separate NIC:
one forwarder pod -- its own netns, tenant VNI X
+-------------------------------------------------+
| |
| on/off-ramp NIC <--- customer data plane |
| | on-ramp (ingress) / off-ramp (egress) |
| | inner packet |
| vxlan (VTEP) encap / decap for VNI X, |
| | listens on this pod's UDP port |
| | outer VXLAN UDP |
| eth0 (underlay) NodeIP:port |
| | to peer VTEPs over the |
| v mesh underlay |
| |
+-------------------------------------------------+
The load-balancing already happens at the sender -- the fdb nexthop
group hashes the inner flow and picks a leg. The only missing piece is
that the selected leg can't carry its own UDP port, so the sender
reaches the right pod's IP but always the device's port.
The host-side alternatives seem to assume a shared datapath that isn't
really there in this design:
- SO_REUSEPORT balances sockets within a single netns; here the
receivers are in different netns -- in fact different tenants --
so there isn't a shared host socket to reuse.
- An XDP/TC host fan-out would mean the host has to become a VTEP --
terminate the tunnel and re-dispatch inner traffic across netns
and tenant/VNI boundaries -- which puts the host into the tenant
data path (today it only does a stateless outer-UDP demux) and
largely duplicates what an fdb nexthop group does: hashing an
inner flow across a set of endpoints. The group can already vary
the leg's IP; the port is the part it can't.
On the uAPI cost: I don't think this adds a new datapath concept. A
single fdb entry already carries a per-destination UDP port (NDA_PORT),
and vxlan_xmit_one() already prefers rdst->remote_port when it's set.
NHA_FDB_PORT is meant to be the nexthop analog of that existing
attribute: control-plane only, no datapath change, and backward
compatible (a leg with no port falls back to the device port as today).
Happy to go into the deployment/addressing constraints in more detail if
that would help.
Thanks for the review.
^ permalink raw reply [flat|nested] 6+ messages in thread