[PATCH net-next 0/2] Add support for externally validated neighbor entries

[PATCH net-next 0/2] Add support for externally validated neighbor entries

[Ido Schimmel]

Patch #1 adds a new neighbor flag ("extern_valid") that prevents the
kernel from invalidating or removing a neighbor entry, while allowing
the kernel to notify user space when the entry becomes reachable. See
motivation and implementation details in the commit message.

Patch #2 adds a selftest.

Ido Schimmel (2):
  neighbor: Add NTF_EXT_VALIDATED flag for externally validated entries
  selftests: net: Add a selftest for externally validated neighbor
    entries

 Documentation/netlink/specs/rt-neigh.yaml |   1 +
 include/net/neighbour.h                   |   4 +-
 include/uapi/linux/neighbour.h            |   5 +
 net/core/neighbour.c                      |  75 ++++-
 tools/testing/selftests/net/Makefile      |   1 +
 tools/testing/selftests/net/test_neigh.sh | 337 ++++++++++++++++++++++
 6 files changed, 413 insertions(+), 10 deletions(-)
 create mode 100755 tools/testing/selftests/net/test_neigh.sh

-- 
2.49.0

[PATCH net-next 1/2] neighbor: Add NTF_EXT_VALIDATED flag for externally validated entries

[Ido Schimmel]

tl;dr
=====

Add a new neighbor flag ("extern_valid") that can be used to indicate to
the kernel that a neighbor entry was learned and determined to be valid
externally. The kernel will not try to remove or invalidate such an
entry, leaving these decisions to the user space control plane. This is
needed for EVPN multi-homing where a neighbor entry for a multi-homed
host needs to be synced across all the VTEPs among which the host is
multi-homed.

Background
==========

In a typical EVPN multi-homing setup each host is multi-homed using a
set of links called ES (Ethernet Segment, i.e., LAG) to multiple leaf
switches (VTEPs). VTEPs that are connected to the same ES are called ES
peers.

When a neighbor entry is learned on a VTEP, it is distributed to both ES
peers and remote VTEPs using EVPN MAC/IP advertisement routes. ES peers
use the neighbor entry when routing traffic towards the multi-homed host
and remote VTEPs use it for ARP/NS suppression.

Motivation
==========

If the ES link between a host and the VTEP on which the neighbor entry
was locally learned goes down, the EVPN MAC/IP advertisement route will
be withdrawn and the neighbor entries will be removed from both ES peers
and remote VTEPs. Routing towards the multi-homed host and ARP/NS
suppression can fail until another ES peer locally learns the neighbor
entry and distributes it via an EVPN MAC/IP advertisement route.

"draft-rbickhart-evpn-ip-mac-proxy-adv-03" [1] suggests avoiding these
intermittent failures by having the ES peers install the neighbor
entries as before, but also injecting EVPN MAC/IP advertisement routes
with a proxy indication. When the previously mentioned ES link goes down
and the original EVPN MAC/IP advertisement route is withdrawn, the ES
peers will not withdraw their neighbor entries, but instead start aging
timers for the proxy indication.

If an ES peer locally learns the neighbor entry (i.e., it becomes
"reachable"), it will restart its aging timer for the entry and emit an
EVPN MAC/IP advertisement route without a proxy indication. An ES peer
will stop its aging timer for the proxy indication if it observes the
removal of the proxy indication from at least one of the ES peers
advertising the entry.

In the event that the aging timer for the proxy indication expired, an
ES peer will withdraw its EVPN MAC/IP advertisement route. If the timer
expired on all ES peers and they all withdrew their proxy
advertisements, the neighbor entry will be completely removed from the
EVPN fabric.

Implementation
==============

In the above scheme, when the control plane (e.g., FRR) advertises a
neighbor entry with a proxy indication, it expects the corresponding
entry in the data plane (i.e., the kernel) to remain valid and not be
removed due to garbage collection. The control plane also expects the
kernel to notify it if the entry was learned locally (i.e., became
"reachable") so that it will remove the proxy indication from the EVPN
MAC/IP advertisement route. That is why these entries cannot be
programmed with dummy states such as "permanent" or "noarp".

Instead, add a new neighbor flag ("extern_valid") which indicates that
the entry was learned and determined to be valid externally and should
not be removed or invalidated by the kernel. The kernel can probe the
entry and notify user space when it becomes "reachable". However, if the
kernel does not receive a confirmation, have it return the entry to the
"stale" state instead of the "failed" state.

In other words, an entry marked with the "extern_valid" flag behaves
like any other dynamically learned entry other than the fact that the
kernel cannot remove or invalidate it.

One can argue that the "extern_valid" flag should not prevent garbage
collection and that instead a neighbor entry should be programmed with
both the "extern_valid" and "extern_learn" flags. There are two reasons
for not doing that:

1. Unclear why a control plane would like to program an entry that the
   kernel cannot invalidate but can completely remove.

2. The "extern_learn" flag is used by FRR for neighbor entries learned
   on remote VTEPs (for ARP/NS suppression) whereas here we are
   concerned with local entries. This distinction is currently irrelevant
   for the kernel, but might be relevant in the future.

Given that the flag only makes sense when the neighbor has a valid
state, reject attempts to add a neighbor with an invalid state and with
this flag set. For example:

 # ip neigh add 192.0.2.1 nud none dev br0.10 extern_valid
 Error: Cannot create externally validated neighbor with an invalid state.
 # ip neigh add 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid
 # ip neigh replace 192.0.2.1 nud failed dev br0.10 extern_valid
 Error: Cannot mark neighbor as externally validated with an invalid state.

The above means that a neighbor cannot be created with the
"extern_valid" flag and flags such as "use" or "managed" as they result
in a neighbor being created with an invalid state ("none") and
immediately getting probed:

 # ip neigh add 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
 Error: Cannot create externally validated neighbor with an invalid state.

However, these flags can be used together with "extern_valid" after the
neighbor was created with a valid state:

 # ip neigh add 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid
 # ip neigh replace 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use

One consequence of preventing the kernel from invalidating a neighbor
entry is that by default it will only try to determine reachability
using unicast probes. This can be changed using the "mcast_resolicit"
sysctl:

 # sysctl net.ipv4.neigh.br0/10.mcast_resolicit
 0
 # tcpdump -nn -e -i br0.10 -Q out arp &
 # ip neigh replace 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
 62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
 62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
 62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
 # sysctl -wq net.ipv4.neigh.br0/10.mcast_resolicit=3
 # ip neigh replace 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
 62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
 62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
 62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
 62:50:1d:11:93:6f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
 62:50:1d:11:93:6f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
 62:50:1d:11:93:6f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28

iproute2 patches can be found here [2].

[1] https://datatracker.ietf.org/doc/html/draft-rbickhart-evpn-ip-mac-proxy-adv-03
[2] https://github.com/idosch/iproute2/tree/submit/extern_valid_v1

Signed-off-by: Ido Schimmel <idosch@nvidia.com>
---
 Documentation/netlink/specs/rt-neigh.yaml |  1 +
 include/net/neighbour.h                   |  4 +-
 include/uapi/linux/neighbour.h            |  5 ++
 net/core/neighbour.c                      | 75 ++++++++++++++++++++---
 4 files changed, 75 insertions(+), 10 deletions(-)

diff --git a/Documentation/netlink/specs/rt-neigh.yaml b/Documentation/netlink/specs/rt-neigh.yaml
index e9cba164e3d1..e263b8d60467 100644
--- a/Documentation/netlink/specs/rt-neigh.yaml
+++ b/Documentation/netlink/specs/rt-neigh.yaml
@@ -79,6 +79,7 @@ definitions:
     entries:
       - managed
       - locked
+      - ext-validated
   -
     name: rtm-type
     type: enum
diff --git a/include/net/neighbour.h b/include/net/neighbour.h
index 9a832cab5b1d..8d69d0f104d9 100644
--- a/include/net/neighbour.h
+++ b/include/net/neighbour.h
@@ -260,13 +260,15 @@ static inline void *neighbour_priv(const struct neighbour *n)
 #define NEIGH_UPDATE_F_EXT_LEARNED		BIT(5)
 #define NEIGH_UPDATE_F_ISROUTER			BIT(6)
 #define NEIGH_UPDATE_F_ADMIN			BIT(7)
+#define NEIGH_UPDATE_F_EXT_VALIDATED		BIT(8)
 
 /* In-kernel representation for NDA_FLAGS_EXT flags: */
 #define NTF_OLD_MASK		0xff
 #define NTF_EXT_SHIFT		8
-#define NTF_EXT_MASK		(NTF_EXT_MANAGED)
+#define NTF_EXT_MASK		(NTF_EXT_MANAGED | NTF_EXT_EXT_VALIDATED)
 
 #define NTF_MANAGED		(NTF_EXT_MANAGED << NTF_EXT_SHIFT)
+#define NTF_EXT_VALIDATED	(NTF_EXT_EXT_VALIDATED << NTF_EXT_SHIFT)
 
 extern const struct nla_policy nda_policy[];
 
diff --git a/include/uapi/linux/neighbour.h b/include/uapi/linux/neighbour.h
index b851c36ad25d..27bedfcce537 100644
--- a/include/uapi/linux/neighbour.h
+++ b/include/uapi/linux/neighbour.h
@@ -54,6 +54,7 @@ enum {
 /* Extended flags under NDA_FLAGS_EXT: */
 #define NTF_EXT_MANAGED		(1 << 0)
 #define NTF_EXT_LOCKED		(1 << 1)
+#define NTF_EXT_EXT_VALIDATED	(1 << 2)
 
 /*
  *	Neighbor Cache Entry States.
@@ -92,6 +93,10 @@ enum {
  * bridge in response to a host trying to communicate via a locked bridge port
  * with MAB enabled. Their purpose is to notify user space that a host requires
  * authentication.
+ *
+ * NTF_EXT_EXT_VALIDATED flagged neigbor entries were externally validated by a
+ * user space control plane. The kernel will not remove or invalidate them, but
+ * it can probe them and notify user space when they become reachable.
  */
 
 struct nda_cacheinfo {
diff --git a/net/core/neighbour.c b/net/core/neighbour.c
index a6e2c91ec3e7..be229f3f5a85 100644
--- a/net/core/neighbour.c
+++ b/net/core/neighbour.c
@@ -153,11 +153,12 @@ static void neigh_update_gc_list(struct neighbour *n)
 	if (n->dead)
 		goto out;
 
-	/* remove from the gc list if new state is permanent or if neighbor
-	 * is externally learned; otherwise entry should be on the gc list
+	/* remove from the gc list if new state is permanent or if neighbor is
+	 * externally learned / validated; otherwise entry should be on the gc
+	 * list
 	 */
 	exempt_from_gc = n->nud_state & NUD_PERMANENT ||
-			 n->flags & NTF_EXT_LEARNED;
+			 n->flags & (NTF_EXT_LEARNED | NTF_EXT_VALIDATED);
 	on_gc_list = !list_empty(&n->gc_list);
 
 	if (exempt_from_gc && on_gc_list) {
@@ -204,6 +205,7 @@ static void neigh_update_flags(struct neighbour *neigh, u32 flags, int *notify,
 
 	ndm_flags  = (flags & NEIGH_UPDATE_F_EXT_LEARNED) ? NTF_EXT_LEARNED : 0;
 	ndm_flags |= (flags & NEIGH_UPDATE_F_MANAGED) ? NTF_MANAGED : 0;
+	ndm_flags |= (flags & NEIGH_UPDATE_F_EXT_VALIDATED) ? NTF_EXT_VALIDATED : 0;
 
 	if ((old_flags ^ ndm_flags) & NTF_EXT_LEARNED) {
 		if (ndm_flags & NTF_EXT_LEARNED)
@@ -221,6 +223,14 @@ static void neigh_update_flags(struct neighbour *neigh, u32 flags, int *notify,
 		*notify = 1;
 		*managed_update = true;
 	}
+	if ((old_flags ^ ndm_flags) & NTF_EXT_VALIDATED) {
+		if (ndm_flags & NTF_EXT_VALIDATED)
+			neigh->flags |= NTF_EXT_VALIDATED;
+		else
+			neigh->flags &= ~NTF_EXT_VALIDATED;
+		*notify = 1;
+		*gc_update = true;
+	}
 }
 
 bool neigh_remove_one(struct neighbour *n)
@@ -937,7 +947,8 @@ static void neigh_periodic_work(struct work_struct *work)
 
 			state = n->nud_state;
 			if ((state & (NUD_PERMANENT | NUD_IN_TIMER)) ||
-			    (n->flags & NTF_EXT_LEARNED)) {
+			    (n->flags &
+			     (NTF_EXT_LEARNED | NTF_EXT_VALIDATED))) {
 				write_unlock(&n->lock);
 				continue;
 			}
@@ -1090,9 +1101,15 @@ static void neigh_timer_handler(struct timer_list *t)
 
 	if ((neigh->nud_state & (NUD_INCOMPLETE | NUD_PROBE)) &&
 	    atomic_read(&neigh->probes) >= neigh_max_probes(neigh)) {
-		WRITE_ONCE(neigh->nud_state, NUD_FAILED);
+		if (neigh->nud_state == NUD_PROBE &&
+		    neigh->flags & NTF_EXT_VALIDATED) {
+			WRITE_ONCE(neigh->nud_state, NUD_STALE);
+			neigh->updated = jiffies;
+		} else {
+			WRITE_ONCE(neigh->nud_state, NUD_FAILED);
+			neigh_invalidate(neigh);
+		}
 		notify = 1;
-		neigh_invalidate(neigh);
 		goto out;
 	}
 
@@ -1240,6 +1257,8 @@ static void neigh_update_hhs(struct neighbour *neigh)
 				NTF_ROUTER flag.
 	NEIGH_UPDATE_F_ISROUTER	indicates if the neighbour is known as
 				a router.
+	NEIGH_UPDATE_F_EXT_VALIDATED means that the entry will not be removed
+				or invalidated.
 
    Caller MUST hold reference count on the entry.
  */
@@ -1974,7 +1993,7 @@ static int neigh_add(struct sk_buff *skb, struct nlmsghdr *nlh,
 	if (ndm_flags & NTF_PROXY) {
 		struct pneigh_entry *pn;
 
-		if (ndm_flags & NTF_MANAGED) {
+		if (ndm_flags & (NTF_MANAGED | NTF_EXT_VALIDATED)) {
 			NL_SET_ERR_MSG(extack, "Invalid NTF_* flag combination");
 			goto out;
 		}
@@ -2004,7 +2023,8 @@ static int neigh_add(struct sk_buff *skb, struct nlmsghdr *nlh,
 	if (neigh == NULL) {
 		bool ndm_permanent  = ndm->ndm_state & NUD_PERMANENT;
 		bool exempt_from_gc = ndm_permanent ||
-				      ndm_flags & NTF_EXT_LEARNED;
+				      ndm_flags & (NTF_EXT_LEARNED |
+						   NTF_EXT_VALIDATED);
 
 		if (!(nlh->nlmsg_flags & NLM_F_CREATE)) {
 			err = -ENOENT;
@@ -2015,10 +2035,27 @@ static int neigh_add(struct sk_buff *skb, struct nlmsghdr *nlh,
 			err = -EINVAL;
 			goto out;
 		}
+		if (ndm_flags & NTF_EXT_VALIDATED) {
+			u8 state = ndm->ndm_state;
+
+			/* NTF_USE and NTF_MANAGED will result in the neighbor
+			 * being created with an invalid state (NUD_NONE).
+			 */
+			if (ndm_flags & (NTF_USE | NTF_MANAGED))
+				state = NUD_NONE;
+
+			if (!(state & NUD_VALID)) {
+				NL_SET_ERR_MSG(extack,
+					       "Cannot create externally validated neighbor with an invalid state");
+				err = -EINVAL;
+				goto out;
+			}
+		}
 
 		neigh = ___neigh_create(tbl, dst, dev,
 					ndm_flags &
-					(NTF_EXT_LEARNED | NTF_MANAGED),
+					(NTF_EXT_LEARNED | NTF_MANAGED |
+					 NTF_EXT_VALIDATED),
 					exempt_from_gc, true);
 		if (IS_ERR(neigh)) {
 			err = PTR_ERR(neigh);
@@ -2030,6 +2067,24 @@ static int neigh_add(struct sk_buff *skb, struct nlmsghdr *nlh,
 			neigh_release(neigh);
 			goto out;
 		}
+		if (ndm_flags & NTF_EXT_VALIDATED) {
+			u8 state = ndm->ndm_state;
+
+			/* NTF_USE and NTF_MANAGED do not update the existing
+			 * state other than clearing it if it was
+			 * NUD_PERMANENT.
+			 */
+			if (ndm_flags & (NTF_USE | NTF_MANAGED))
+				state = READ_ONCE(neigh->nud_state) & ~NUD_PERMANENT;
+
+			if (!(state & NUD_VALID)) {
+				NL_SET_ERR_MSG(extack,
+					       "Cannot mark neighbor as externally validated with an invalid state");
+				err = -EINVAL;
+				neigh_release(neigh);
+				goto out;
+			}
+		}
 
 		if (!(nlh->nlmsg_flags & NLM_F_REPLACE))
 			flags &= ~(NEIGH_UPDATE_F_OVERRIDE |
@@ -2046,6 +2101,8 @@ static int neigh_add(struct sk_buff *skb, struct nlmsghdr *nlh,
 		flags |= NEIGH_UPDATE_F_MANAGED;
 	if (ndm_flags & NTF_USE)
 		flags |= NEIGH_UPDATE_F_USE;
+	if (ndm_flags & NTF_EXT_VALIDATED)
+		flags |= NEIGH_UPDATE_F_EXT_VALIDATED;
 
 	err = __neigh_update(neigh, lladdr, ndm->ndm_state, flags,
 			     NETLINK_CB(skb).portid, extack);
-- 
2.49.0

[PATCH net-next 2/2] selftests: net: Add a selftest for externally validated neighbor entries

[Ido Schimmel]

Add test cases for externally validated neighbor entries, testing both
IPv4 and IPv6. Name the file "test_neigh.sh" so that it could be
possibly extended in the future with more neighbor test cases.

Example output:

 # ./test_neigh.sh
 TEST: IPv4 "extern_valid" flag: Add entry                           [ OK ]
 TEST: IPv4 "extern_valid" flag: Add with an invalid state           [ OK ]
 TEST: IPv4 "extern_valid" flag: Add with "use" flag                 [ OK ]
 TEST: IPv4 "extern_valid" flag: Replace entry                       [ OK ]
 TEST: IPv4 "extern_valid" flag: Replace entry with "managed" flag   [ OK ]
 TEST: IPv4 "extern_valid" flag: Replace with an invalid state       [ OK ]
 TEST: IPv4 "extern_valid" flag: Transition to "reachable" state     [ OK ]
 TEST: IPv4 "extern_valid" flag: Transition back to "stale" state    [ OK ]
 TEST: IPv4 "extern_valid" flag: Forced garbage collection           [ OK ]
 TEST: IPv4 "extern_valid" flag: Periodic garbage collection         [ OK ]
 TEST: IPv6 "extern_valid" flag: Add entry                           [ OK ]
 TEST: IPv6 "extern_valid" flag: Add with an invalid state           [ OK ]
 TEST: IPv6 "extern_valid" flag: Add with "use" flag                 [ OK ]
 TEST: IPv6 "extern_valid" flag: Replace entry                       [ OK ]
 TEST: IPv6 "extern_valid" flag: Replace entry with "managed" flag   [ OK ]
 TEST: IPv6 "extern_valid" flag: Replace with an invalid state       [ OK ]
 TEST: IPv6 "extern_valid" flag: Transition to "reachable" state     [ OK ]
 TEST: IPv6 "extern_valid" flag: Transition back to "stale" state    [ OK ]
 TEST: IPv6 "extern_valid" flag: Forced garbage collection           [ OK ]
 TEST: IPv6 "extern_valid" flag: Periodic garbage collection         [ OK ]

Signed-off-by: Ido Schimmel <idosch@nvidia.com>
---
 tools/testing/selftests/net/Makefile      |   1 +
 tools/testing/selftests/net/test_neigh.sh | 337 ++++++++++++++++++++++
 2 files changed, 338 insertions(+)
 create mode 100755 tools/testing/selftests/net/test_neigh.sh

diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index ea84b88bcb30..f7c8dac525de 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -98,6 +98,7 @@ TEST_PROGS += test_vxlan_mdb.sh
 TEST_PROGS += test_bridge_neigh_suppress.sh
 TEST_PROGS += test_vxlan_nolocalbypass.sh
 TEST_PROGS += test_bridge_backup_port.sh
+TEST_PROGS += test_neigh.sh
 TEST_PROGS += fdb_flush.sh fdb_notify.sh
 TEST_PROGS += fq_band_pktlimit.sh
 TEST_PROGS += vlan_hw_filter.sh
diff --git a/tools/testing/selftests/net/test_neigh.sh b/tools/testing/selftests/net/test_neigh.sh
new file mode 100755
index 000000000000..5bf9153aa7a6
--- /dev/null
+++ b/tools/testing/selftests/net/test_neigh.sh
@@ -0,0 +1,337 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+source lib.sh
+TESTS="
+	extern_valid_ipv4
+	extern_valid_ipv6
+"
+VERBOSE=0
+
+################################################################################
+# Utilities
+
+run_cmd()
+{
+	local cmd="$1"
+	local out
+	local stderr="2>/dev/null"
+
+	if [ "$VERBOSE" = "1" ]; then
+		printf "COMMAND: $cmd\n"
+		stderr=
+	fi
+
+	out=$(eval $cmd $stderr)
+	rc=$?
+	if [ "$VERBOSE" -eq 1 ] && [ -n "$out" ]; then
+		echo "    $out"
+	fi
+
+	return $rc
+}
+
+################################################################################
+# Setup
+
+setup()
+{
+	set -e
+
+	setup_ns ns1 ns2
+
+	ip -n $ns1 link add veth0 type veth peer name veth1 netns $ns2
+	ip -n $ns1 link set dev veth0 up
+	ip -n $ns2 link set dev veth1 up
+
+	ip -n $ns1 address add 192.0.2.1/24 dev veth0
+	ip -n $ns1 address add 2001:db8:1::1/64 dev veth0 nodad
+	ip -n $ns2 address add 192.0.2.2/24 dev veth1
+	ip -n $ns2 address add 2001:db8:1::2/64 dev veth1 nodad
+
+	sleep 5
+
+	set +e
+}
+
+exit_cleanup_all()
+{
+	cleanup_all_ns
+	exit "${EXIT_STATUS}"
+}
+
+################################################################################
+# Tests
+
+extern_valid_common()
+{
+	local af_str=$1; shift
+	local ip_addr=$1; shift
+	local tbl_name=$1; shift
+	local subnet=$1; shift
+	local mac
+
+	mac=$(ip -n $ns2 -j link show dev veth1 | jq -r '.[]["address"]')
+
+	RET=0
+
+	# Check that simple addition works.
+	run_cmd "ip -n $ns1 neigh add $ip_addr lladdr $mac nud stale dev veth0 extern_valid"
+	run_cmd "ip -n $ns1 neigh get $ip_addr dev veth0 | grep \"extern_valid\""
+	check_err $? "No \"extern_valid\" flag after addition"
+
+	log_test "$af_str \"extern_valid\" flag: Add entry"
+
+	RET=0
+
+	# Check that an entry cannot be added with "extern_valid" flag and an
+	# invalid state.
+	run_cmd "ip -n $ns1 neigh flush dev veth0"
+	run_cmd "ip -n $ns1 neigh add $ip_addr nud none dev veth0 extern_valid"
+	check_fail $? "Managed to add an entry with \"extern_valid\" flag and an invalid state"
+
+	log_test "$af_str \"extern_valid\" flag: Add with an invalid state"
+
+	RET=0
+
+	# Check that entry cannot be added with both "extern_valid" flag and
+	# "use" / "managed" flag.
+	run_cmd "ip -n $ns1 neigh flush dev veth0"
+	run_cmd "ip -n $ns1 neigh add $ip_addr lladdr $mac nud stale dev veth0 extern_valid use"
+	check_fail $? "Managed to add an entry with \"extern_valid\" flag and \"use\" flag"
+
+	log_test "$af_str \"extern_valid\" flag: Add with \"use\" flag"
+
+	RET=0
+
+	# Check that "extern_valid" flag can be toggled using replace.
+	run_cmd "ip -n $ns1 neigh flush dev veth0"
+	run_cmd "ip -n $ns1 neigh add $ip_addr lladdr $mac nud stale dev veth0"
+	run_cmd "ip -n $ns1 neigh replace $ip_addr lladdr $mac nud stale dev veth0 extern_valid"
+	run_cmd "ip -n $ns1 neigh get $ip_addr dev veth0 | grep \"extern_valid\""
+	check_err $? "Did not manage to set \"extern_valid\" flag with replace"
+	run_cmd "ip -n $ns1 neigh replace $ip_addr lladdr $mac nud stale dev veth0"
+	run_cmd "ip -n $ns1 neigh get $ip_addr dev veth0 | grep \"extern_valid\""
+	check_fail $? "Did not manage to clear \"extern_valid\" flag with replace"
+
+	log_test "$af_str \"extern_valid\" flag: Replace entry"
+
+	RET=0
+
+	# Check that an existing "extern_valid" entry can be marked as
+	# "managed".
+	run_cmd "ip -n $ns1 neigh flush dev veth0"
+	run_cmd "ip -n $ns1 neigh add $ip_addr lladdr $mac nud stale dev veth0 extern_valid"
+	run_cmd "ip -n $ns1 neigh replace $ip_addr lladdr $mac nud stale dev veth0 extern_valid managed"
+	check_err $? "Did not manage to add \"managed\" flag to an existing \"extern_valid\" entry"
+
+	log_test "$af_str \"extern_valid\" flag: Replace entry with \"managed\" flag"
+
+	RET=0
+
+	# Check that entry cannot be replaced with "extern_valid" flag and an
+	# invalid state.
+	run_cmd "ip -n $ns1 neigh flush dev veth0"
+	run_cmd "ip -n $ns1 neigh add $ip_addr lladdr $mac nud stale dev veth0 extern_valid"
+	run_cmd "ip -n $ns1 neigh replace $ip_addr nud none dev veth0 extern_valid"
+	check_fail $? "Managed to replace an entry with \"extern_valid\" flag and an invalid state"
+
+	log_test "$af_str \"extern_valid\" flag: Replace with an invalid state"
+
+	RET=0
+
+	# Check that when entry transitions to "reachable" state it maintains
+	# the "extern_valid" flag. Wait "delay_probe" seconds for ARP request /
+	# NS to be sent.
+	local delay_probe
+
+	delay_probe=$(ip -n $ns1 -j ntable show dev veth0 name $tbl_name | jq '.[]["delay_probe"]')
+	run_cmd "ip -n $ns1 neigh flush dev veth0"
+	run_cmd "ip -n $ns1 neigh add $ip_addr lladdr $mac nud stale dev veth0 extern_valid"
+	run_cmd "ip -n $ns1 neigh replace $ip_addr lladdr $mac nud stale dev veth0 extern_valid use"
+	run_cmd "sleep $((delay_probe / 1000 + 2))"
+	run_cmd "ip -n $ns1 neigh get $ip_addr dev veth0 | grep \"REACHABLE\""
+	check_err $? "Entry did not transition to \"reachable\" state"
+	run_cmd "ip -n $ns1 neigh get $ip_addr dev veth0 | grep \"extern_valid\""
+	check_err $? "Entry did not maintain \"extern_valid\" flag after transition to \"reachable\" state"
+
+	log_test "$af_str \"extern_valid\" flag: Transition to \"reachable\" state"
+
+	RET=0
+
+	# Drop all packets, trigger resolution and check that entry goes back
+	# to "stale" state instead of "failed".
+	local mcast_reprobes
+	local retrans_time
+	local ucast_probes
+	local app_probes
+	local probes
+	local delay
+
+	run_cmd "ip -n $ns1 neigh flush dev veth0"
+	run_cmd "tc -n $ns2 qdisc add dev veth1 clsact"
+	run_cmd "tc -n $ns2 filter add dev veth1 ingress proto all matchall action drop"
+	run_cmd "ip -n $ns1 neigh add $ip_addr lladdr $mac nud stale dev veth0 extern_valid"
+	run_cmd "ip -n $ns1 neigh replace $ip_addr lladdr $mac nud stale dev veth0 extern_valid use"
+	retrans_time=$(ip -n $ns1 -j ntable show dev veth0 name $tbl_name | jq '.[]["retrans"]')
+	ucast_probes=$(ip -n $ns1 -j ntable show dev veth0 name $tbl_name | jq '.[]["ucast_probes"]')
+	app_probes=$(ip -n $ns1 -j ntable show dev veth0 name $tbl_name | jq '.[]["app_probes"]')
+	mcast_reprobes=$(ip -n $ns1 -j ntable show dev veth0 name $tbl_name | jq '.[]["mcast_reprobes"]')
+	delay=$((delay_probe + (ucast_probes + app_probes + mcast_reprobes) * retrans_time))
+	run_cmd "sleep $((delay / 1000 + 2))"
+	run_cmd "ip -n $ns1 neigh get $ip_addr dev veth0 | grep \"STALE\""
+	check_err $? "Entry did not return to \"stale\" state"
+	run_cmd "ip -n $ns1 neigh get $ip_addr dev veth0 | grep \"extern_valid\""
+	check_err $? "Entry did not maintain \"extern_valid\" flag after returning to \"stale\" state"
+	probes=$(ip -n $ns1 -j -s neigh get $ip_addr dev veth0 | jq '.[]["probes"]')
+	if [[ $probes -eq 0 ]]; then
+		check_err 1 "No probes were sent"
+	fi
+
+	log_test "$af_str \"extern_valid\" flag: Transition back to \"stale\" state"
+
+	run_cmd "tc -n $ns2 qdisc del dev veth1 clsact"
+
+	RET=0
+
+	# Forced garbage collection runs whenever the number of entries is
+	# larger than "thresh3" and deletes stale entries that have not been
+	# updated in the last 5 seconds.
+	#
+	# Check that an "extern_valid" entry survives a forced garbage
+	# collection. Add an entry, wait 5 seconds and add more entries than
+	# "thresh3" so that forced garbage collection will run.
+	#
+	# Note that the garbage collection thresholds are global resources and
+	# that changes in the initial namespace affect all the namespaces.
+	local forced_gc_runs_t0
+	local forced_gc_runs_t1
+	local orig_thresh1
+	local orig_thresh2
+	local orig_thresh3
+
+	run_cmd "ip -n $ns1 neigh flush dev veth0"
+	orig_thresh1=$(ip -j ntable show name $tbl_name | jq '.[] | select(has("thresh1")) | .["thresh1"]')
+	orig_thresh2=$(ip -j ntable show name $tbl_name | jq '.[] | select(has("thresh2")) | .["thresh2"]')
+	orig_thresh3=$(ip -j ntable show name $tbl_name | jq '.[] | select(has("thresh3")) | .["thresh3"]')
+	run_cmd "ip ntable change name $tbl_name thresh3 10 thresh2 9 thresh1 8"
+	run_cmd "ip -n $ns1 neigh add $ip_addr lladdr $mac nud stale dev veth0 extern_valid"
+	run_cmd "ip -n $ns1 neigh add ${subnet}3 lladdr $mac nud stale dev veth0"
+	run_cmd "sleep 5"
+	forced_gc_runs_t0=$(ip -j -s ntable show name $tbl_name | jq '.[] | select(has("forced_gc_runs")) | .["forced_gc_runs"]')
+	for i in {1..20}; do
+		run_cmd "ip -n $ns1 neigh add ${subnet}$((i + 4)) nud none dev veth0"
+	done
+	forced_gc_runs_t1=$(ip -j -s ntable show name $tbl_name | jq '.[] | select(has("forced_gc_runs")) | .["forced_gc_runs"]')
+	if [[ $forced_gc_runs_t1 -eq $forced_gc_runs_t0 ]]; then
+		check_err 1 "Forced garbage collection did not run"
+	fi
+	run_cmd "ip -n $ns1 neigh get $ip_addr dev veth0 | grep \"extern_valid\""
+	check_err $? "Entry with \"extern_valid\" flag did not survive forced garbage collection"
+	run_cmd "ip -n $ns1 neigh get ${subnet}3 dev veth0"
+	check_fail $? "Entry without \"extern_valid\" flag survived forced garbage collection"
+
+	log_test "$af_str \"extern_valid\" flag: Forced garbage collection"
+
+	run_cmd "ip ntable change name $tbl_name thresh3 $orig_thresh3 thresh2 $orig_thresh2 thresh1 $orig_thresh1"
+
+	RET=0
+
+	# Periodic garbage collection runs every "base_reachable"/2 seconds and
+	# if the number of entries is larger than "thresh1", then it deletes
+	# stale entries that have not been used in the last "gc_stale" seconds.
+	#
+	# Check that an "extern_valid" entry survives a periodic garbage
+	# collection. Add an "extern_valid" entry, add more than "thresh1"
+	# regular entries, wait "base_reachable" (longer than "gc_stale")
+	# seconds and check that the "extern_valid" entry was not deleted.
+	#
+	# Note that the garbage collection thresholds and "base_reachable" are
+	# global resources and that changes in the initial namespace affect all
+	# the namespaces.
+	local periodic_gc_runs_t0
+	local periodic_gc_runs_t1
+	local orig_base_reachable
+	local orig_gc_stale
+
+	run_cmd "ip -n $ns1 neigh flush dev veth0"
+	orig_thresh1=$(ip -j ntable show name $tbl_name | jq '.[] | select(has("thresh1")) | .["thresh1"]')
+	orig_base_reachable=$(ip -j ntable show name $tbl_name | jq '.[] | select(has("thresh1")) | .["base_reachable"]')
+	run_cmd "ip ntable change name $tbl_name thresh1 10 base_reachable 10000"
+	orig_gc_stale=$(ip -n $ns1 -j ntable show name $tbl_name dev veth0 | jq '.[]["gc_stale"]')
+	run_cmd "ip -n $ns1 ntable change name $tbl_name dev veth0 gc_stale 5000"
+	# Wait orig_base_reachable/2 for the new interval to take effect.
+	run_cmd "sleep $(((orig_base_reachable / 1000) / 2 + 2))"
+	run_cmd "ip -n $ns1 neigh add $ip_addr lladdr $mac nud stale dev veth0 extern_valid"
+	run_cmd "ip -n $ns1 neigh add ${subnet}3 lladdr $mac nud stale dev veth0"
+	for i in {1..20}; do
+		run_cmd "ip -n $ns1 neigh add ${subnet}$((i + 4)) nud none dev veth0"
+	done
+	periodic_gc_runs_t0=$(ip -j -s ntable show name $tbl_name | jq '.[] | select(has("periodic_gc_runs")) | .["periodic_gc_runs"]')
+	run_cmd "sleep 10"
+	periodic_gc_runs_t1=$(ip -j -s ntable show name $tbl_name | jq '.[] | select(has("periodic_gc_runs")) | .["periodic_gc_runs"]')
+	[[ $periodic_gc_runs_t1 -ne $periodic_gc_runs_t0 ]]
+	check_err $? "Periodic garbage collection did not run"
+	run_cmd "ip -n $ns1 neigh get $ip_addr dev veth0 | grep \"extern_valid\""
+	check_err $? "Entry with \"extern_valid\" flag did not survive periodic garbage collection"
+	run_cmd "ip -n $ns1 neigh get ${subnet}3 dev veth0"
+	check_fail $? "Entry without \"extern_valid\" flag survived periodic garbage collection"
+
+	log_test "$af_str \"extern_valid\" flag: Periodic garbage collection"
+
+	run_cmd "ip -n $ns1 ntable change name $tbl_name dev veth0 gc_stale $orig_gc_stale"
+	run_cmd "ip ntable change name $tbl_name thresh1 $orig_thresh1 base_reachable $orig_base_reachable"
+}
+
+extern_valid_ipv4()
+{
+	extern_valid_common "IPv4" 192.0.2.2 "arp_cache" 192.0.2.
+}
+
+extern_valid_ipv6()
+{
+	extern_valid_common "IPv6" 2001:db8:1::2 "ndisc_cache" 2001:db8:1::
+}
+
+################################################################################
+# Usage
+
+usage()
+{
+	cat <<EOF
+usage: ${0##*/} OPTS
+
+        -t <test>   Test(s) to run (default: all)
+                    (options: $TESTS)
+        -p          Pause on fail
+        -v          Verbose mode (show commands and output)
+EOF
+}
+
+################################################################################
+# Main
+
+while getopts ":t:pvh" opt; do
+	case $opt in
+		t) TESTS=$OPTARG;;
+		p) PAUSE_ON_FAIL=yes;;
+		v) VERBOSE=$(($VERBOSE + 1));;
+		h) usage; exit 0;;
+		*) usage; exit 1;;
+	esac
+done
+
+require_command jq
+
+ip neigh help 2>&1 | grep -q "extern_valid"
+if [ $? -ne 0 ]; then
+   echo "SKIP: iproute2 ip too old, missing \"extern_valid\" support"
+   exit $ksft_skip
+fi
+
+trap exit_cleanup_all EXIT
+
+for t in $TESTS
+do
+	setup; $t; cleanup_all_ns;
+done
-- 
2.49.0

Re: [PATCH net-next 1/2] neighbor: Add NTF_EXT_VALIDATED flag for externally validated entries

[Nikolay Aleksandrov]

On 6/11/25 17:15, Ido Schimmel wrote:
> tl;dr
> =====
> 
> Add a new neighbor flag ("extern_valid") that can be used to indicate to
> the kernel that a neighbor entry was learned and determined to be valid
> externally. The kernel will not try to remove or invalidate such an
> entry, leaving these decisions to the user space control plane. This is
> needed for EVPN multi-homing where a neighbor entry for a multi-homed
> host needs to be synced across all the VTEPs among which the host is
> multi-homed.
> 
> Background
> ==========
> 
> In a typical EVPN multi-homing setup each host is multi-homed using a
> set of links called ES (Ethernet Segment, i.e., LAG) to multiple leaf
> switches (VTEPs). VTEPs that are connected to the same ES are called ES
> peers.
> 
> When a neighbor entry is learned on a VTEP, it is distributed to both ES
> peers and remote VTEPs using EVPN MAC/IP advertisement routes. ES peers
> use the neighbor entry when routing traffic towards the multi-homed host
> and remote VTEPs use it for ARP/NS suppression.
> 
> Motivation
> ==========
> 
> If the ES link between a host and the VTEP on which the neighbor entry
> was locally learned goes down, the EVPN MAC/IP advertisement route will
> be withdrawn and the neighbor entries will be removed from both ES peers
> and remote VTEPs. Routing towards the multi-homed host and ARP/NS
> suppression can fail until another ES peer locally learns the neighbor
> entry and distributes it via an EVPN MAC/IP advertisement route.
> 
> "draft-rbickhart-evpn-ip-mac-proxy-adv-03" [1] suggests avoiding these
> intermittent failures by having the ES peers install the neighbor
> entries as before, but also injecting EVPN MAC/IP advertisement routes
> with a proxy indication. When the previously mentioned ES link goes down
> and the original EVPN MAC/IP advertisement route is withdrawn, the ES
> peers will not withdraw their neighbor entries, but instead start aging
> timers for the proxy indication.
> 
> If an ES peer locally learns the neighbor entry (i.e., it becomes
> "reachable"), it will restart its aging timer for the entry and emit an
> EVPN MAC/IP advertisement route without a proxy indication. An ES peer
> will stop its aging timer for the proxy indication if it observes the
> removal of the proxy indication from at least one of the ES peers
> advertising the entry.
> 
> In the event that the aging timer for the proxy indication expired, an
> ES peer will withdraw its EVPN MAC/IP advertisement route. If the timer
> expired on all ES peers and they all withdrew their proxy
> advertisements, the neighbor entry will be completely removed from the
> EVPN fabric.
> 
> Implementation
> ==============
> 
> In the above scheme, when the control plane (e.g., FRR) advertises a
> neighbor entry with a proxy indication, it expects the corresponding
> entry in the data plane (i.e., the kernel) to remain valid and not be
> removed due to garbage collection. The control plane also expects the
> kernel to notify it if the entry was learned locally (i.e., became
> "reachable") so that it will remove the proxy indication from the EVPN
> MAC/IP advertisement route. That is why these entries cannot be
> programmed with dummy states such as "permanent" or "noarp".
> 
> Instead, add a new neighbor flag ("extern_valid") which indicates that
> the entry was learned and determined to be valid externally and should
> not be removed or invalidated by the kernel. The kernel can probe the
> entry and notify user space when it becomes "reachable". However, if the
> kernel does not receive a confirmation, have it return the entry to the
> "stale" state instead of the "failed" state.
> 
> In other words, an entry marked with the "extern_valid" flag behaves
> like any other dynamically learned entry other than the fact that the
> kernel cannot remove or invalidate it.
> 
> One can argue that the "extern_valid" flag should not prevent garbage
> collection and that instead a neighbor entry should be programmed with
> both the "extern_valid" and "extern_learn" flags. There are two reasons
> for not doing that:
> 
> 1. Unclear why a control plane would like to program an entry that the
>     kernel cannot invalidate but can completely remove.
> 
> 2. The "extern_learn" flag is used by FRR for neighbor entries learned
>     on remote VTEPs (for ARP/NS suppression) whereas here we are
>     concerned with local entries. This distinction is currently irrelevant
>     for the kernel, but might be relevant in the future.
> 
> Given that the flag only makes sense when the neighbor has a valid
> state, reject attempts to add a neighbor with an invalid state and with
> this flag set. For example:
> 
>   # ip neigh add 192.0.2.1 nud none dev br0.10 extern_valid
>   Error: Cannot create externally validated neighbor with an invalid state.
>   # ip neigh add 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid
>   # ip neigh replace 192.0.2.1 nud failed dev br0.10 extern_valid
>   Error: Cannot mark neighbor as externally validated with an invalid state.
> 
> The above means that a neighbor cannot be created with the
> "extern_valid" flag and flags such as "use" or "managed" as they result
> in a neighbor being created with an invalid state ("none") and
> immediately getting probed:
> 
>   # ip neigh add 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
>   Error: Cannot create externally validated neighbor with an invalid state.
> 
> However, these flags can be used together with "extern_valid" after the
> neighbor was created with a valid state:
> 
>   # ip neigh add 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid
>   # ip neigh replace 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
> 
> One consequence of preventing the kernel from invalidating a neighbor
> entry is that by default it will only try to determine reachability
> using unicast probes. This can be changed using the "mcast_resolicit"
> sysctl:
> 
>   # sysctl net.ipv4.neigh.br0/10.mcast_resolicit
>   0
>   # tcpdump -nn -e -i br0.10 -Q out arp &
>   # ip neigh replace 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   # sysctl -wq net.ipv4.neigh.br0/10.mcast_resolicit=3
>   # ip neigh replace 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
> 
> iproute2 patches can be found here [2].
> 
> [1] https://datatracker.ietf.org/doc/html/draft-rbickhart-evpn-ip-mac-proxy-adv-03
> [2] https://github.com/idosch/iproute2/tree/submit/extern_valid_v1
> 
> Signed-off-by: Ido Schimmel <idosch@nvidia.com>
> ---
>   Documentation/netlink/specs/rt-neigh.yaml |  1 +
>   include/net/neighbour.h                   |  4 +-
>   include/uapi/linux/neighbour.h            |  5 ++
>   net/core/neighbour.c                      | 75 ++++++++++++++++++++---
>   4 files changed, 75 insertions(+), 10 deletions(-)
> 

Reviewed-by: Nikolay Aleksandrov <razor@blackwall.org>

Re: [PATCH net-next 2/2] selftests: net: Add a selftest for externally validated neighbor entries

[Nikolay Aleksandrov]

On 6/11/25 17:15, Ido Schimmel wrote:
> Add test cases for externally validated neighbor entries, testing both
> IPv4 and IPv6. Name the file "test_neigh.sh" so that it could be
> possibly extended in the future with more neighbor test cases.
> 
> Example output:
> 
>   # ./test_neigh.sh
>   TEST: IPv4 "extern_valid" flag: Add entry                           [ OK ]
>   TEST: IPv4 "extern_valid" flag: Add with an invalid state           [ OK ]
>   TEST: IPv4 "extern_valid" flag: Add with "use" flag                 [ OK ]
>   TEST: IPv4 "extern_valid" flag: Replace entry                       [ OK ]
>   TEST: IPv4 "extern_valid" flag: Replace entry with "managed" flag   [ OK ]
>   TEST: IPv4 "extern_valid" flag: Replace with an invalid state       [ OK ]
>   TEST: IPv4 "extern_valid" flag: Transition to "reachable" state     [ OK ]
>   TEST: IPv4 "extern_valid" flag: Transition back to "stale" state    [ OK ]
>   TEST: IPv4 "extern_valid" flag: Forced garbage collection           [ OK ]
>   TEST: IPv4 "extern_valid" flag: Periodic garbage collection         [ OK ]
>   TEST: IPv6 "extern_valid" flag: Add entry                           [ OK ]
>   TEST: IPv6 "extern_valid" flag: Add with an invalid state           [ OK ]
>   TEST: IPv6 "extern_valid" flag: Add with "use" flag                 [ OK ]
>   TEST: IPv6 "extern_valid" flag: Replace entry                       [ OK ]
>   TEST: IPv6 "extern_valid" flag: Replace entry with "managed" flag   [ OK ]
>   TEST: IPv6 "extern_valid" flag: Replace with an invalid state       [ OK ]
>   TEST: IPv6 "extern_valid" flag: Transition to "reachable" state     [ OK ]
>   TEST: IPv6 "extern_valid" flag: Transition back to "stale" state    [ OK ]
>   TEST: IPv6 "extern_valid" flag: Forced garbage collection           [ OK ]
>   TEST: IPv6 "extern_valid" flag: Periodic garbage collection         [ OK ]
> 
> Signed-off-by: Ido Schimmel <idosch@nvidia.com>
> ---
>   tools/testing/selftests/net/Makefile      |   1 +
>   tools/testing/selftests/net/test_neigh.sh | 337 ++++++++++++++++++++++
>   2 files changed, 338 insertions(+)
>   create mode 100755 tools/testing/selftests/net/test_neigh.sh
> 

Reviewed-by: Nikolay Aleksandrov <razor@blackwall.org>

Re: [PATCH net-next 1/2] neighbor: Add NTF_EXT_VALIDATED flag for externally validated entries

[Daniel Borkmann]

Hi Ido,

On 6/11/25 4:15 PM, Ido Schimmel wrote:
> tl;dr
> =====
> 
> Add a new neighbor flag ("extern_valid") that can be used to indicate to
> the kernel that a neighbor entry was learned and determined to be valid
> externally. The kernel will not try to remove or invalidate such an
> entry, leaving these decisions to the user space control plane. This is
> needed for EVPN multi-homing where a neighbor entry for a multi-homed
> host needs to be synced across all the VTEPs among which the host is
> multi-homed.
> 
> Background
> ==========
> 
> In a typical EVPN multi-homing setup each host is multi-homed using a
> set of links called ES (Ethernet Segment, i.e., LAG) to multiple leaf
> switches (VTEPs). VTEPs that are connected to the same ES are called ES
> peers.
> 
> When a neighbor entry is learned on a VTEP, it is distributed to both ES
> peers and remote VTEPs using EVPN MAC/IP advertisement routes. ES peers
> use the neighbor entry when routing traffic towards the multi-homed host
> and remote VTEPs use it for ARP/NS suppression.
> 
> Motivation
> ==========
> 
> If the ES link between a host and the VTEP on which the neighbor entry
> was locally learned goes down, the EVPN MAC/IP advertisement route will
> be withdrawn and the neighbor entries will be removed from both ES peers
> and remote VTEPs. Routing towards the multi-homed host and ARP/NS
> suppression can fail until another ES peer locally learns the neighbor
> entry and distributes it via an EVPN MAC/IP advertisement route.
> 
> "draft-rbickhart-evpn-ip-mac-proxy-adv-03" [1] suggests avoiding these
> intermittent failures by having the ES peers install the neighbor
> entries as before, but also injecting EVPN MAC/IP advertisement routes
> with a proxy indication. When the previously mentioned ES link goes down
> and the original EVPN MAC/IP advertisement route is withdrawn, the ES
> peers will not withdraw their neighbor entries, but instead start aging
> timers for the proxy indication.
> 
> If an ES peer locally learns the neighbor entry (i.e., it becomes
> "reachable"), it will restart its aging timer for the entry and emit an
> EVPN MAC/IP advertisement route without a proxy indication. An ES peer
> will stop its aging timer for the proxy indication if it observes the
> removal of the proxy indication from at least one of the ES peers
> advertising the entry.
> 
> In the event that the aging timer for the proxy indication expired, an
> ES peer will withdraw its EVPN MAC/IP advertisement route. If the timer
> expired on all ES peers and they all withdrew their proxy
> advertisements, the neighbor entry will be completely removed from the
> EVPN fabric.
> 
> Implementation
> ==============
> 
> In the above scheme, when the control plane (e.g., FRR) advertises a
> neighbor entry with a proxy indication, it expects the corresponding
> entry in the data plane (i.e., the kernel) to remain valid and not be
> removed due to garbage collection. The control plane also expects the
> kernel to notify it if the entry was learned locally (i.e., became
> "reachable") so that it will remove the proxy indication from the EVPN
> MAC/IP advertisement route. That is why these entries cannot be
> programmed with dummy states such as "permanent" or "noarp".

Meaning, in contrast to "permanent" the initial user-provided lladdr
can still be updated by the kernel if it learned that there was a
migration, right?

> Instead, add a new neighbor flag ("extern_valid") which indicates that
> the entry was learned and determined to be valid externally and should
> not be removed or invalidated by the kernel. The kernel can probe the
> entry and notify user space when it becomes "reachable". However, if the
> kernel does not receive a confirmation, have it return the entry to the
> "stale" state instead of the "failed" state.
> 
> In other words, an entry marked with the "extern_valid" flag behaves
> like any other dynamically learned entry other than the fact that the
> kernel cannot remove or invalidate it.

How is the expected neigh_flush_dev() behavior? I presume in that case if
the neigh entry is in use and was NUD_STALE then we go into NUD_NONE state
right? (Asking as NUD_PERMANENT skips all that and whether that should be
similar or not for NTF_EXT_VALIDATED?)

> One can argue that the "extern_valid" flag should not prevent garbage
> collection and that instead a neighbor entry should be programmed with
> both the "extern_valid" and "extern_learn" flags. There are two reasons
> for not doing that:
> 
> 1. Unclear why a control plane would like to program an entry that the
>     kernel cannot invalidate but can completely remove.
> 
> 2. The "extern_learn" flag is used by FRR for neighbor entries learned
>     on remote VTEPs (for ARP/NS suppression) whereas here we are
>     concerned with local entries. This distinction is currently irrelevant
>     for the kernel, but might be relevant in the future.
> 
> Given that the flag only makes sense when the neighbor has a valid
> state, reject attempts to add a neighbor with an invalid state and with
> this flag set. For example:
> 
>   # ip neigh add 192.0.2.1 nud none dev br0.10 extern_valid
>   Error: Cannot create externally validated neighbor with an invalid state.
>   # ip neigh add 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid
>   # ip neigh replace 192.0.2.1 nud failed dev br0.10 extern_valid
>   Error: Cannot mark neighbor as externally validated with an invalid state.
> 
> The above means that a neighbor cannot be created with the
> "extern_valid" flag and flags such as "use" or "managed" as they result
> in a neighbor being created with an invalid state ("none") and
> immediately getting probed:
> 
>   # ip neigh add 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
>   Error: Cannot create externally validated neighbor with an invalid state.
> 
> However, these flags can be used together with "extern_valid" after the
> neighbor was created with a valid state:
> 
>   # ip neigh add 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid
>   # ip neigh replace 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
> 
> One consequence of preventing the kernel from invalidating a neighbor
> entry is that by default it will only try to determine reachability
> using unicast probes. This can be changed using the "mcast_resolicit"
> sysctl:
> 
>   # sysctl net.ipv4.neigh.br0/10.mcast_resolicit
>   0
>   # tcpdump -nn -e -i br0.10 -Q out arp &
>   # ip neigh replace 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   # sysctl -wq net.ipv4.neigh.br0/10.mcast_resolicit=3
>   # ip neigh replace 192.0.2.1 lladdr 00:11:22:33:44:55 nud stale dev br0.10 extern_valid use
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > 00:11:22:33:44:55, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
>   62:50:1d:11:93:6f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.1 tell 192.0.2.2, length 28
> 
> iproute2 patches can be found here [2].
> 
> [1] https://datatracker.ietf.org/doc/html/draft-rbickhart-evpn-ip-mac-proxy-adv-03
> [2] https://github.com/idosch/iproute2/tree/submit/extern_valid_v1
> 
> Signed-off-by: Ido Schimmel <idosch@nvidia.com>
[...]

Thanks,
Daniel

Re: [PATCH net-next 2/2] selftests: net: Add a selftest for externally validated neighbor entries

[Simon Horman]

On Wed, Jun 11, 2025 at 05:15:51PM +0300, Ido Schimmel wrote:
> Add test cases for externally validated neighbor entries, testing both
> IPv4 and IPv6. Name the file "test_neigh.sh" so that it could be
> possibly extended in the future with more neighbor test cases.
> 
> Example output:
> 
>  # ./test_neigh.sh
>  TEST: IPv4 "extern_valid" flag: Add entry                           [ OK ]
>  TEST: IPv4 "extern_valid" flag: Add with an invalid state           [ OK ]
>  TEST: IPv4 "extern_valid" flag: Add with "use" flag                 [ OK ]
>  TEST: IPv4 "extern_valid" flag: Replace entry                       [ OK ]
>  TEST: IPv4 "extern_valid" flag: Replace entry with "managed" flag   [ OK ]
>  TEST: IPv4 "extern_valid" flag: Replace with an invalid state       [ OK ]
>  TEST: IPv4 "extern_valid" flag: Transition to "reachable" state     [ OK ]
>  TEST: IPv4 "extern_valid" flag: Transition back to "stale" state    [ OK ]
>  TEST: IPv4 "extern_valid" flag: Forced garbage collection           [ OK ]
>  TEST: IPv4 "extern_valid" flag: Periodic garbage collection         [ OK ]
>  TEST: IPv6 "extern_valid" flag: Add entry                           [ OK ]
>  TEST: IPv6 "extern_valid" flag: Add with an invalid state           [ OK ]
>  TEST: IPv6 "extern_valid" flag: Add with "use" flag                 [ OK ]
>  TEST: IPv6 "extern_valid" flag: Replace entry                       [ OK ]
>  TEST: IPv6 "extern_valid" flag: Replace entry with "managed" flag   [ OK ]
>  TEST: IPv6 "extern_valid" flag: Replace with an invalid state       [ OK ]
>  TEST: IPv6 "extern_valid" flag: Transition to "reachable" state     [ OK ]
>  TEST: IPv6 "extern_valid" flag: Transition back to "stale" state    [ OK ]
>  TEST: IPv6 "extern_valid" flag: Forced garbage collection           [ OK ]
>  TEST: IPv6 "extern_valid" flag: Periodic garbage collection         [ OK ]
> 
> Signed-off-by: Ido Schimmel <idosch@nvidia.com>

...

> diff --git a/tools/testing/selftests/net/test_neigh.sh b/tools/testing/selftests/net/test_neigh.sh

...

> +################################################################################
> +# Utilities
> +
> +run_cmd()
> +{
> +	local cmd="$1"
> +	local out
> +	local stderr="2>/dev/null"
> +
> +	if [ "$VERBOSE" = "1" ]; then
> +		printf "COMMAND: $cmd\n"
> +		stderr=
> +	fi

Hi Ido,

shellcheck was recently added to NIPA and it warns about the above as follows.
Could you consider addressing this?

In tools/testing/selftests/net/test_neigh.sh line 21:
 		printf "COMMAND: $cmd\n"
                        ^---------------^ SC2059 (info): Don't use variables in the printf format string. Use printf '..%s..' "$foo".

> +
> +	out=$(eval $cmd $stderr)
> +	rc=$?
> +	if [ "$VERBOSE" -eq 1 ] && [ -n "$out" ]; then
> +		echo "    $out"
> +	fi
> +
> +	return $rc
> +}

...

> +################################################################################
> +# Main
> +
> +while getopts ":t:pvh" opt; do
> +	case $opt in
> +		t) TESTS=$OPTARG;;
> +		p) PAUSE_ON_FAIL=yes;;
> +		v) VERBOSE=$(($VERBOSE + 1));;
> +		h) usage; exit 0;;
> +		*) usage; exit 1;;
> +	esac
> +done

Likewise, shellcheck has the following to say about the above.

In tools/testing/selftests/net/test_neigh.sh line 318:
		v) VERBOSE=$(($VERBOSE + 1));;
                              ^------^ SC2004 (style): $/${} is unnecessary on arithmetic variables.

> +
> +require_command jq
> +
> +ip neigh help 2>&1 | grep -q "extern_valid"
> +if [ $? -ne 0 ]; then
> +   echo "SKIP: iproute2 ip too old, missing \"extern_valid\" support"
> +   exit $ksft_skip
> +fi
> +
> +trap exit_cleanup_all EXIT
> +
> +for t in $TESTS
> +do
> +	setup; $t; cleanup_all_ns;
> +done
> -- 
> 2.49.0
> 

-- 
pw-bot: cr

Re: [PATCH net-next 1/2] neighbor: Add NTF_EXT_VALIDATED flag for externally validated entries

[Simon Horman]

On Wed, Jun 11, 2025 at 05:15:50PM +0300, Ido Schimmel wrote:

...

> diff --git a/include/uapi/linux/neighbour.h b/include/uapi/linux/neighbour.h
> index b851c36ad25d..27bedfcce537 100644
> --- a/include/uapi/linux/neighbour.h
> +++ b/include/uapi/linux/neighbour.h
> @@ -54,6 +54,7 @@ enum {
>  /* Extended flags under NDA_FLAGS_EXT: */
>  #define NTF_EXT_MANAGED		(1 << 0)
>  #define NTF_EXT_LOCKED		(1 << 1)
> +#define NTF_EXT_EXT_VALIDATED	(1 << 2)
>  
>  /*
>   *	Neighbor Cache Entry States.
> @@ -92,6 +93,10 @@ enum {
>   * bridge in response to a host trying to communicate via a locked bridge port
>   * with MAB enabled. Their purpose is to notify user space that a host requires
>   * authentication.
> + *
> + * NTF_EXT_EXT_VALIDATED flagged neigbor entries were externally validated by a

nit: neighbor or neighbour

> + * user space control plane. The kernel will not remove or invalidate them, but
> + * it can probe them and notify user space when they become reachable.
>   */
>  
>  struct nda_cacheinfo {

...

Re: [PATCH net-next 1/2] neighbor: Add NTF_EXT_VALIDATED flag for externally validated entries

[Ido Schimmel]

On Fri, Jun 13, 2025 at 10:23:26AM +0200, Daniel Borkmann wrote:
> Hi Ido,

Hi Daniel,

> On 6/11/25 4:15 PM, Ido Schimmel wrote:
> > In the above scheme, when the control plane (e.g., FRR) advertises a
> > neighbor entry with a proxy indication, it expects the corresponding
> > entry in the data plane (i.e., the kernel) to remain valid and not be
> > removed due to garbage collection. The control plane also expects the
> > kernel to notify it if the entry was learned locally (i.e., became
> > "reachable") so that it will remove the proxy indication from the EVPN
> > MAC/IP advertisement route. That is why these entries cannot be
> > programmed with dummy states such as "permanent" or "noarp".
> 
> Meaning, in contrast to "permanent" the initial user-provided lladdr
> can still be updated by the kernel if it learned that there was a
> migration, right?

Yes. In addition, user space will be notified when the kernel locally
learns the entry. FRR installs such entries as "stale" and a
notification will be emitted when they transition to "reachable".

> > Instead, add a new neighbor flag ("extern_valid") which indicates that
> > the entry was learned and determined to be valid externally and should
> > not be removed or invalidated by the kernel. The kernel can probe the
> > entry and notify user space when it becomes "reachable". However, if the
> > kernel does not receive a confirmation, have it return the entry to the
> > "stale" state instead of the "failed" state.
> > 
> > In other words, an entry marked with the "extern_valid" flag behaves
> > like any other dynamically learned entry other than the fact that the
> > kernel cannot remove or invalidate it.
> 
> How is the expected neigh_flush_dev() behavior? I presume in that case if
> the neigh entry is in use and was NUD_STALE then we go into NUD_NONE state
> right? (Asking as NUD_PERMANENT skips all that and whether that should be
> similar or not for NTF_EXT_VALIDATED?)

Currently, unlike "permanent" entries, such entries will be flushed when
the interface loses its carrier. Given the description of "[...] behaves
like any other dynamically learned entry other than the fact that the
kernel cannot remove or invalidate it" I think it makes sense to not
flush such entries when the carrier goes down.

Like "permanent" entries, such entries will be flushed when the
interface is put administratively down or when its MAC changes, both of
which are user initiated actions.

IOW, I will squash the following diff and add test cases.

 static void neigh_flush_dev(struct neigh_table *tbl, struct net_device *dev,
-                           bool skip_perm)
+                           bool skip_perm_ext_valid)
 {
        struct hlist_head *dev_head;
        struct hlist_node *tmp;
@@ -378,7 +388,9 @@ static void neigh_flush_dev(struct neigh_table *tbl, struct net_device *dev,
        dev_head = neigh_get_dev_table(dev, tbl->family);
 
        hlist_for_each_entry_safe(n, tmp, dev_head, dev_list) {
-               if (skip_perm && n->nud_state & NUD_PERMANENT)
+               if (skip_perm_ext_valid &&
+                   (n->nud_state & NUD_PERMANENT ||
+                    n->flags & NTF_EXT_VALIDATED))
                        continue;
 
                hlist_del_rcu(&n->hash);
@@ -419,10 +431,10 @@ void neigh_changeaddr(struct neigh_table *tbl, struct net_device *dev)
 EXPORT_SYMBOL(neigh_changeaddr);
 
 static int __neigh_ifdown(struct neigh_table *tbl, struct net_device *dev,
-                         bool skip_perm)
+                         bool skip_perm_ext_valid)
 {
        write_lock_bh(&tbl->lock);
-       neigh_flush_dev(tbl, dev, skip_perm);
+       neigh_flush_dev(tbl, dev, skip_perm_ext_valid);
        pneigh_ifdown_and_unlock(tbl, dev);
        pneigh_queue_purge(&tbl->proxy_queue, dev ? dev_net(dev) : NULL,
                           tbl->family);

Re: [PATCH net-next 1/2] neighbor: Add NTF_EXT_VALIDATED flag for externally validated entries

[Daniel Borkmann]

Hi Ido,

On 6/17/25 4:09 PM, Ido Schimmel wrote:
> On Fri, Jun 13, 2025 at 10:23:26AM +0200, Daniel Borkmann wrote:
>> On 6/11/25 4:15 PM, Ido Schimmel wrote:
>>> In the above scheme, when the control plane (e.g., FRR) advertises a
>>> neighbor entry with a proxy indication, it expects the corresponding
>>> entry in the data plane (i.e., the kernel) to remain valid and not be
>>> removed due to garbage collection. The control plane also expects the
>>> kernel to notify it if the entry was learned locally (i.e., became
>>> "reachable") so that it will remove the proxy indication from the EVPN
>>> MAC/IP advertisement route. That is why these entries cannot be
>>> programmed with dummy states such as "permanent" or "noarp".
>>
>> Meaning, in contrast to "permanent" the initial user-provided lladdr
>> can still be updated by the kernel if it learned that there was a
>> migration, right?
> 
> Yes. In addition, user space will be notified when the kernel locally
> learns the entry. FRR installs such entries as "stale" and a
> notification will be emitted when they transition to "reachable".

Thanks for clarifying, perhaps makes sense to mention this bit in the
commit message also in v2.

>>> Instead, add a new neighbor flag ("extern_valid") which indicates that
>>> the entry was learned and determined to be valid externally and should
>>> not be removed or invalidated by the kernel. The kernel can probe the
>>> entry and notify user space when it becomes "reachable". However, if the
>>> kernel does not receive a confirmation, have it return the entry to the
>>> "stale" state instead of the "failed" state.
>>>
>>> In other words, an entry marked with the "extern_valid" flag behaves
>>> like any other dynamically learned entry other than the fact that the
>>> kernel cannot remove or invalidate it.
>>
>> How is the expected neigh_flush_dev() behavior? I presume in that case if
>> the neigh entry is in use and was NUD_STALE then we go into NUD_NONE state
>> right? (Asking as NUD_PERMANENT skips all that and whether that should be
>> similar or not for NTF_EXT_VALIDATED?)
> 
> Currently, unlike "permanent" entries, such entries will be flushed when
> the interface loses its carrier. Given the description of "[...] behaves
> like any other dynamically learned entry other than the fact that the
> kernel cannot remove or invalidate it" I think it makes sense to not
> flush such entries when the carrier goes down.

Yeah agree given the user intent that would make sense imho otherwise
you'd need some additional watcher to then reinstall if that happens.

> Like "permanent" entries, such entries will be flushed when the
> interface is put administratively down or when its MAC changes, both of
> which are user initiated actions.
> 
> IOW, I will squash the following diff and add test cases.

lgtm!

>   static void neigh_flush_dev(struct neigh_table *tbl, struct net_device *dev,
> -                           bool skip_perm)
> +                           bool skip_perm_ext_valid)
>   {
>          struct hlist_head *dev_head;
>          struct hlist_node *tmp;
> @@ -378,7 +388,9 @@ static void neigh_flush_dev(struct neigh_table *tbl, struct net_device *dev,
>          dev_head = neigh_get_dev_table(dev, tbl->family);
>   
>          hlist_for_each_entry_safe(n, tmp, dev_head, dev_list) {
> -               if (skip_perm && n->nud_state & NUD_PERMANENT)
> +               if (skip_perm_ext_valid &&
> +                   (n->nud_state & NUD_PERMANENT ||
> +                    n->flags & NTF_EXT_VALIDATED))
>                          continue;
>   
>                  hlist_del_rcu(&n->hash);
> @@ -419,10 +431,10 @@ void neigh_changeaddr(struct neigh_table *tbl, struct net_device *dev)
>   EXPORT_SYMBOL(neigh_changeaddr);
>   
>   static int __neigh_ifdown(struct neigh_table *tbl, struct net_device *dev,
> -                         bool skip_perm)
> +                         bool skip_perm_ext_valid)
>   {
>          write_lock_bh(&tbl->lock);
> -       neigh_flush_dev(tbl, dev, skip_perm);
> +       neigh_flush_dev(tbl, dev, skip_perm_ext_valid);
>          pneigh_ifdown_and_unlock(tbl, dev);
>          pneigh_queue_purge(&tbl->proxy_queue, dev ? dev_net(dev) : NULL,
>                             tbl->family);

Thanks,
Daniel