[PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Hangbin Liu]

This series makes IOAM insertion transparent to IPv6 ECMP, as required
by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
along the same path as the identical packet without them.

Patch 1 fixes the transit side: hash policy 0 hashed the raw IPv6
nexthdr, which becomes NEXTHDR_HOP once an IOAM Hop-by-Hop header is
inserted. The policy-0 key (and the ICMPv6-error flow selection) now
uses the nexthdr the packet would have had if the Hop-by-Hop header
carried only IOAM options and padding.

Patch 2 fixes the injecting node: the ioam6 output re-lookup on a
destination-cache miss hashed the transformed packet's NEXTHDR_HOP
instead of the original nexthdr (inline) or IPPROTO_IPV6 (encap), and
rebuilt the flow keys from the outer header only.

Patch 3 adds selftests/net/ioam6_ecmp.sh, covering both places: a
transit forwarder with a multipath route, and an encapsulating node
compared against a plain ip6ip6 baseline.

The issue was found while reviewing RFC9486. It can be reproduced and
verified by the newly added ioam6_ecmp.sh test.

Tested with:
ecmp_rehash.sh
fib_nexthops.sh
ioam6_ecmp.sh
tcp_ecmp_failover.sh
forwarding/gre_inner_v4_multipath.sh
forwarding/gre_inner_v6_multipath.sh
forwarding/gre_multipath_nh_res.sh
forwarding/gre_multipath_nh.sh
forwarding/gre_multipath.sh
forwarding/ip6gre_inner_v4_multipath.sh
forwarding/ip6gre_inner_v6_multipath.sh
forwarding/router_mpath_nh_res.sh
forwarding/router_mpath_nh.sh
forwarding/router_multipath.sh

Signed-off-by: Hangbin Liu <liuhangbin@kylinos.cn>
---
Hangbin Liu (3):
      ipv6: route: consider IOAM options in ECMP hash policy 0
      ipv6: ioam6: use transformed packet nexthdr for ECMP re-lookup
      selftests: net: add ioam6 ECMP test (RFC 9486 C2)

 include/uapi/linux/in6.h                  |   1 +
 net/ipv6/ioam6_iptunnel.c                 |   4 +-
 net/ipv6/route.c                          |  67 +++++-
 tools/testing/selftests/net/Makefile      |   1 +
 tools/testing/selftests/net/ioam6_ecmp.sh | 349 ++++++++++++++++++++++++++++++
 5 files changed, 419 insertions(+), 3 deletions(-)
---
base-commit: 5d7f3375cd574fc83f0cd30839b21455c0e79ae2
change-id: 20261020-ipv6_ioam_ecmp-ae4ef0a4bdae

Best regards,
--  
Hangbin Liu <liuhangbin@kylinos.cn>

[PATCH 1/3] ipv6: route: consider IOAM options in ECMP hash policy 0

[Hangbin Liu]

From: Hangbin Liu <liuhangbin@kylinos.cn>

RFC 9486 (C2) requires that the ECMP hash of a packet carrying IOAM
options is the same as the hash of the identical packet without them.

The default multipath hash policy (0) hashes basic.ip_proto, which
ip6_multipath_l3_keys() fills from the raw IPv6 nexthdr. An IOAM
Hop-by-Hop option is inserted ahead of the original next header, so a
transit router sees NEXTHDR_HOP (0) instead of e.g. IPPROTO_TCP and
may pick a different ECMP sibling than for the same packet without
IOAM.

Recover the nexthdr the packet would have had if its Hop-by-Hop
header carried only IOAM options (0x31, and 0x11 for DEX) plus
padding, and use it for the policy-0 key and for the ICMPv6-error
check, so errors that quote a flow keep hashing that flow. Any other
Hop-by-Hop option (Router Alert, Jumbo, ...) keeps hashing exactly as
before. The walk only runs for the rare nexthdr == NEXTHDR_HOP case
and is safe: ipv6_parse_hopopts() has already pulled and validated
the header before routing. ICMPv6-quoted inner headers are not
walked: they are payload and may be truncated.

Fixes: 3edede08ff37 ("ipv6: ioam: Support for IOAM injection with lwtunnels")
Signed-off-by: Hangbin Liu <liuhangbin@kylinos.cn>
---
The ECMP path with and without IOAM is verified by
tools/testing/selftests/net/ioam6_ecmp.sh in the next patch.
---
 include/uapi/linux/in6.h |  1 +
 net/ipv6/route.c         | 67 ++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 66 insertions(+), 2 deletions(-)

diff --git a/include/uapi/linux/in6.h b/include/uapi/linux/in6.h
index 5a47339ef7d7..2e66dc37e1e5 100644
--- a/include/uapi/linux/in6.h
+++ b/include/uapi/linux/in6.h
@@ -145,6 +145,7 @@ struct in6_flowlabel_req {
 #define IPV6_TLV_PADN		1
 #define IPV6_TLV_ROUTERALERT	5
 #define IPV6_TLV_CALIPSO	7	/* RFC 5570 */
+#define IPV6_TLV_IOAM_DEX	17	/* RFC 9486 */
 #define IPV6_TLV_IOAM		49	/* RFC 9486 */
 #define IPV6_TLV_JUMBO		194
 #define IPV6_TLV_HAO		201	/* home address option */
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 94e5ef1c4f83..d0761f05d326 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -2342,6 +2342,60 @@ struct dst_entry *ip6_route_input_lookup(struct net *net,
 }
 EXPORT_SYMBOL_GPL(ip6_route_input_lookup);
 
+/* RFC 9486 (C2) requires that the ECMP hash of a packet carrying IOAM
+ * options is the same as the hash of the identical packet without them.
+ * Recover the nexthdr if its Hop-by-Hop header carried only IOAM options
+ * and padding, every other packet is hashed exactly as before.
+ */
+static u8 ip6_nexthdr_without_ioam(const struct sk_buff *skb,
+				   const struct ipv6hdr *iph)
+{
+	const struct ipv6_opt_hdr *hhdr;
+	struct ipv6_opt_hdr hbuf;
+	bool has_ioam = false;
+	u8 obuf[2], *opt;
+	int off, len, i;
+
+	if (iph->nexthdr != NEXTHDR_HOP)
+		return iph->nexthdr;
+
+	/* The Hop-by-Hop header was already pulled and validated by
+	 * ipv6_parse_hopopts() before routing, so this walk is safe.
+	 */
+	off = sizeof(*iph);
+	hhdr = skb_header_pointer(skb, off, sizeof(hbuf), &hbuf);
+	if (!hhdr)
+		return iph->nexthdr;
+
+	len = (hhdr->hdrlen + 1) << 3;
+
+	for (i = 2; i < len; ) {
+		opt = skb_header_pointer(skb, off + i, sizeof(obuf), obuf);
+		if (!opt)
+			return iph->nexthdr;
+		if (opt[0] == IPV6_TLV_PAD1) {
+			i++;
+			continue;
+		}
+		if (opt[0] == IPV6_TLV_PADN) {
+			i += opt[1] + 2;
+			continue;
+		}
+		if (opt[0] != IPV6_TLV_IOAM && opt[0] != IPV6_TLV_IOAM_DEX)
+			return iph->nexthdr;
+
+		has_ioam = true;
+
+		/* Keep searching for the next TLV. As long as non-IOAM
+		 * options (Router Alert, Jumbo, etc.) exist in the HBH
+		 * header, we should preserve the original NEXTHDR_HOP.
+		 */
+		i += opt[1] + 2;
+	}
+
+	return has_ioam ? hhdr->nexthdr : iph->nexthdr;
+}
+
 static void ip6_multipath_l3_keys(const struct sk_buff *skb,
 				  struct flow_keys *keys,
 				  struct flow_keys *flkeys)
@@ -2353,8 +2407,10 @@ static void ip6_multipath_l3_keys(const struct sk_buff *skb,
 	const struct icmp6hdr *icmph;
 	struct ipv6hdr _inner_iph;
 	struct icmp6hdr _icmph;
+	u8 nexthdr;
 
-	if (likely(outer_iph->nexthdr != IPPROTO_ICMPV6))
+	nexthdr = ip6_nexthdr_without_ioam(skb, outer_iph);
+	if (likely(nexthdr != IPPROTO_ICMPV6))
 		goto out;
 
 	icmph = skb_header_pointer(skb, skb_transport_offset(skb),
@@ -2383,7 +2439,14 @@ static void ip6_multipath_l3_keys(const struct sk_buff *skb,
 		keys->addrs.v6addrs.src = key_iph->saddr;
 		keys->addrs.v6addrs.dst = key_iph->daddr;
 		keys->tags.flow_label = ip6_flowlabel(key_iph);
-		keys->basic.ip_proto = key_iph->nexthdr;
+		/* Only for the outer header: an ICMPv6-quoted inner header
+		 * is payload, and its (possibly truncated) extension
+		 * headers must not be walked here.
+		 */
+		if (key_iph == outer_iph)
+			keys->basic.ip_proto = nexthdr;
+		else
+			keys->basic.ip_proto = key_iph->nexthdr;
 	}
 }
 

-- 
2.56.0

[PATCH 2/3] ipv6: ioam6: use transformed packet nexthdr for ECMP re-lookup

[Hangbin Liu]

From: Hangbin Liu <liuhangbin@kylinos.cn>

RFC 9486 (C2) requires that the ECMP hash of a packet carrying IOAM
options is the same as the hash of the identical packet without them.

When the ioam6 output path misses the destination cache, it re-looks
up the route for the transformed packet. Under the default hash
policy 0 that lookup hashed NEXTHDR_HOP rather than the nexthdr the
identical packet without IOAM would carry, so it could pick a
different ECMP sibling.

Use the Hop-by-Hop nexthdr already stored by do_inline()/do_encap()
(eh.nexthdr) for the re-lookup, and precompute the multipath hash
from the packet: ip6_route_output() lookups carry no skb, so without
this ICMPv6 errors would stop hashing their quoted flow and L4 hash
policy packets would lose their ports.

Fixes: 8cb3bf8bff3c ("ipv6: ioam: Add support for the ip6ip6 encapsulation")
Signed-off-by: Hangbin Liu <liuhangbin@kylinos.cn>
---
 net/ipv6/ioam6_iptunnel.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/net/ipv6/ioam6_iptunnel.c b/net/ipv6/ioam6_iptunnel.c
index cfb2c41634a0..713e625bfa3a 100644
--- a/net/ipv6/ioam6_iptunnel.c
+++ b/net/ipv6/ioam6_iptunnel.c
@@ -13,6 +13,7 @@
 #include <linux/ioam6.h>
 #include <linux/ioam6_iptunnel.h>
 #include <net/dst.h>
+#include <net/ip6_route.h>
 #include <net/sock.h>
 #include <net/lwtunnel.h>
 #include <net/ioam6.h>
@@ -412,7 +413,8 @@ static int ioam6_output(struct net *net, struct sock *sk, struct sk_buff *skb)
 		fl6.saddr = hdr->saddr;
 		fl6.flowlabel = ip6_flowinfo(hdr);
 		fl6.flowi6_mark = skb->mark;
-		fl6.flowi6_proto = hdr->nexthdr;
+		fl6.flowi6_proto = ilwt->tuninfo.eh.nexthdr;
+		fl6.mp_hash = rt6_multipath_hash(net, &fl6, skb, NULL);
 
 		dst = ip6_route_output(net, NULL, &fl6);
 		if (dst->error) {

-- 
2.56.0

[PATCH 3/3] selftests: net: add ioam6 ECMP test (RFC 9486 C2)

[Hangbin Liu]

From: Hangbin Liu <liuhangbin@kylinos.cn>

Add new test ioam6_ecmp.sh, verifying that the ECMP path taken with
IOAM options is identical to the path of the same packet without them
under the default hash policy 0, as required by RFC 9486 (C2).

  - transit input hashing: the sender inserts an IOAM Hop-by-Hop option
    (inline mode), the forwarder has a multipath route;
  - encapsulating node re-lookup: the forwarder itself encapsulates
    (ioam6 encap mode, tunnel dst equal to the original dst) over a
    multipath route, with a plain ip6ip6 tunnel as the
    identical-packet-without-IOAM baseline.

Signed-off-by: Hangbin Liu <liuhangbin@kylinos.cn>
---
 tools/testing/selftests/net/Makefile      |   1 +
 tools/testing/selftests/net/ioam6_ecmp.sh | 349 ++++++++++++++++++++++++++++++
 2 files changed, 350 insertions(+)

diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index df215e1a6262..397b19799cc9 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -52,6 +52,7 @@ TEST_PROGS := \
 	icmp_redirect.sh \
 	io_uring_zerocopy_tx.sh \
 	ioam6.sh \
+	ioam6_ecmp.sh \
 	ip6_gre_headroom.sh \
 	ip6_tunnel.sh \
 	ip_defrag.sh \
diff --git a/tools/testing/selftests/net/ioam6_ecmp.sh b/tools/testing/selftests/net/ioam6_ecmp.sh
new file mode 100755
index 000000000000..c5c82884c757
--- /dev/null
+++ b/tools/testing/selftests/net/ioam6_ecmp.sh
@@ -0,0 +1,349 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+#
+# Verify RFC 9486 C2 for ECMP: with hash policy 0 (the default), the ECMP
+# nexthop chosen for a packet carrying IOAM options must be the same as for
+# the identical packet without them.
+#
+# Two scenarios, matching the two places where IOAM changed the hashed
+# IPv6 nexthdr:
+#
+# 1) transit node: alpha inserts an IOAM Hop-by-Hop option (inline mode);
+#    beta forwards with a multipath route. Policy 0 hashed the raw
+#    nexthdr (NEXTHDR_HOP vs IPPROTO_ICMPV6), so beta could pick a
+#    different ECMP sibling than for the same packet without IOAM.
+#
+# 2) encapsulating node: beta itself encapsulates (ioam6 encap mode,
+#    tunnel dst = original dst, tunnel src pinned) over a multipath
+#    route. The ioam6 re-lookup hashed NEXTHDR_HOP instead of the
+#    transformed packet's real nexthdr (IPPROTO_IPV6). The baseline is
+#    the same encapsulation without the IOAM option: a plain ip6ip6
+#    tunnel over the same multipath route, with the same outer
+#    addresses.
+#
+# Hash policy 0 hashes addresses, the flow label and the nexthdr (not L4
+# ports), so whether both variants land on the same ECMP sibling depends
+# on the per-netns hash seed. Each test therefore repeats with several
+# seeds and fails on the first divergence: pre-fix kernels fail with
+# high probability, fixed kernels always pass.
+#
+# Topology:
+#
+#   alpha                   beta                      gamma
+#   veth0 2001:db8:1::2 <-> veth0 2001:db8:1::1
+#                           veth1 2001:db8:2:1::1 <-> veth0 2001:db8:2:1::2
+#                           veth2 2001:db8:2:2::1 <-> veth1 2001:db8:2:2::2
+#
+# beta's ECMP route: 2001:db8:2:1::2/128
+#   nexthop via 2001:db8:2:1::2 dev veth1
+#   nexthop via 2001:db8:2:2::2 dev veth2
+#
+# Scenario 1 sends to 2001:db8:2:1::2 (gamma). Scenario 2 sends to
+# 2001:db8:3::2, which beta encapsulates towards 2001:db8:2:1::2. The
+# link gamma received the packets on is detected from tc flower
+# counters (read via jq) on the ingress of gamma's veth0/veth1. The
+# filters match the exact measurement packets: ICMPv6 echo requests
+# for scenario 1, and the outer addresses of the ip6ip6 packets for
+# scenario 2 (flower classifies outer headers only).
+
+# shellcheck disable=SC2154,SC2329
+source lib.sh
+
+readonly PKTS=3
+readonly SEEDS="1 2 3 4 5 6 7 8 9 10 11 12"
+
+ip6tnl_loaded=0
+
+cleanup()
+{
+	cleanup_all_ns
+
+	if [ "$ip6tnl_loaded" != 0 ]; then
+		modprobe -r ip6_tunnel &>/dev/null
+	fi
+}
+
+# Replace the tc measurement filters on gamma's ECMP links. $1 is the
+# flower match list, e.g. "ip_proto icmpv6 type 128 src_ip <sender>"
+# for plain IPv6 echo requests, or the outer addresses for ip6ip6
+# packets (flower classifies the outer header only, see
+# FLOW_DISSECTOR_F_STOP_BEFORE_ENCAP in cls_flower).
+set_measure_filters()
+{
+	local match=$1
+	local dev
+
+	for dev in veth0 veth1; do
+		tc -n "$ns_gamma" qdisc del dev "$dev" clsact &>/dev/null
+		tc -n "$ns_gamma" qdisc add dev "$dev" clsact || return 1
+		# shellcheck disable=SC2086
+		tc -n "$ns_gamma" filter add dev "$dev" ingress pref 1 \
+			protocol ipv6 flower skip_hw $match action pass || return 1
+	done
+}
+
+# Number of packets counted by the measurement filter on gamma's $1.
+icmp_packets()
+{
+	tc -n "$ns_gamma" -s -j filter show dev "$1" ingress |
+		jq '[.[] | .options.actions[]?.stats |
+		    (.packets // .basic.packets // 0)] | add // 0'
+}
+
+# Send PKTS pings from alpha to $1 and report which of gamma's links
+# received them: "A" (veth0), "B" (veth1), "TIE" or "LOST:<a>:<b>".
+measure_link()
+{
+	local dst=$1
+	local a0 b0 da db
+
+	a0=$(icmp_packets veth0)
+	b0=$(icmp_packets veth1)
+
+	ip netns exec "$ns_alpha" \
+		ping -6 -q -c "$PKTS" -i 0.2 -W 1 -w 2 "$dst" &>/dev/null || true
+
+	da=$(($(icmp_packets veth0) - a0))
+	db=$(($(icmp_packets veth1) - b0))
+
+	if [ $((da + db)) -lt "$PKTS" ]; then
+		echo "LOST:$da:$db"
+		return
+	fi
+
+	if [ "$da" -gt "$db" ]; then
+		echo "A"
+	elif [ "$db" -gt "$da" ]; then
+		echo "B"
+	else
+		echo "TIE"
+	fi
+}
+
+set_alpha_plain()
+{
+	ip -netns "$ns_alpha" -6 route replace 2001:db8:2:1::/64 \
+		via 2001:db8:1::1 dev veth0
+}
+
+set_alpha_ioam()
+{
+	ip -netns "$ns_alpha" -6 route replace 2001:db8:2:1::/64 \
+		encap ioam6 mode inline trace prealloc \
+		type 0x800000 ns 0 size 12 \
+		via 2001:db8:1::1 dev veth0
+}
+
+set_beta_tnl()
+{
+	# Recreate the tunnel: ip6_tnl caches the outer dst per CPU for a
+	# few seconds, and neither the route change below nor the
+	# fib_multipath_hash_seed write resets it. A stale cache would
+	# resolve the outer sibling under a previous seed (or on another
+	# CPU), making the baseline flip between attempts of one seed.
+	ip -netns "$ns_beta" link del ioamcmp &>/dev/null
+	ip -netns "$ns_beta" link add name ioamcmp type ip6tnl \
+		mode ip6ip6 local 2001:db8:2:1::1 remote 2001:db8:2:1::2
+	ip -netns "$ns_beta" link set ioamcmp up
+	ip -netns "$ns_beta" -6 route replace 2001:db8:3::2/128 dev ioamcmp
+}
+
+set_beta_ioam_encap()
+{
+	ip -netns "$ns_beta" -6 route replace 2001:db8:3::2/128 \
+		encap ioam6 mode encap tunsrc 2001:db8:2:1::1 tundst 2001:db8:2:1::2 \
+		trace prealloc type 0x800000 ns 0 size 12 dev veth1
+}
+
+# $1: description; $2: function installing the without-IOAM variant;
+# $3: function installing the with-IOAM variant; $4: destination.
+run_scenario()
+{
+	local desc=$1
+	local plain=$2
+	local ioam=$3
+	local dst=$4
+	local seed pl il
+
+	log_info "$desc"
+
+	for seed in $SEEDS; do
+		RET=0
+		ip netns exec "$ns_beta" \
+			sysctl -wq net.ipv4.fib_multipath_hash_seed="$seed"
+
+		if ! $plain; then
+			RET=$ksft_fail
+			log_test "$desc (plain route setup failed, seed $seed)"
+			continue
+		fi
+		pl=$(measure_link "$dst")
+		if ! $ioam; then
+			RET=$ksft_fail
+			log_test "$desc (ioam route setup failed, seed $seed)"
+			continue
+		fi
+		il=$(measure_link "$dst")
+
+		case "$pl$il" in
+		*LOST*)
+			RET=$ksft_fail
+			log_test "$desc (lost packets, seed $seed: without=$pl with=$il)"
+			continue
+			;;
+		*TIE*)
+			RET=$ksft_fail
+			log_test "$desc (no clear link, seed $seed)"
+			continue
+			;;
+		esac
+
+		if [ "$pl" = "$il" ]; then
+			log_test "$desc (seed $seed: same link $pl)"
+		else
+			RET=$ksft_fail
+			log_test "$desc (diverged on seed $seed: without IOAM via link $pl, with IOAM via link $il)"
+		fi
+	done
+}
+
+setup_topology()
+{
+	local dev
+
+	setup_ns ns_alpha ns_beta ns_gamma || return 1
+
+	ip link add veth0 netns "$ns_alpha" type veth \
+		peer name veth0 netns "$ns_beta" || return 1
+	ip link add veth1 netns "$ns_beta" type veth \
+		peer name veth0 netns "$ns_gamma" || return 1
+	ip link add veth2 netns "$ns_beta" type veth \
+		peer name veth1 netns "$ns_gamma" || return 1
+
+	ip -n "$ns_alpha" addr add 2001:db8:1::2/64 dev veth0 nodad
+	ip -n "$ns_beta" addr add 2001:db8:1::1/64 dev veth0 nodad
+	ip -n "$ns_beta" addr add 2001:db8:2:1::1/64 dev veth1 nodad
+	ip -n "$ns_beta" addr add 2001:db8:2:2::1/64 dev veth2 nodad
+	ip -n "$ns_gamma" addr add 2001:db8:2:1::2/64 dev veth0 nodad
+	ip -n "$ns_gamma" addr add 2001:db8:2:2::2/64 dev veth1 nodad
+
+	ip -n "$ns_alpha" link set veth0 up
+	ip -n "$ns_beta" link set veth0 up
+	ip -n "$ns_beta" link set veth1 up
+	ip -n "$ns_beta" link set veth2 up
+	ip -n "$ns_gamma" link set veth0 up
+	ip -n "$ns_gamma" link set veth1 up
+
+	ip netns exec "$ns_alpha" sysctl -wq net.ipv6.auto_flowlabels=0
+	ip netns exec "$ns_alpha" \
+		sysctl -wq net.ipv6.conf.veth0.accept_ra=0
+	ip netns exec "$ns_beta" sysctl -wq net.ipv6.conf.all.forwarding=1
+	ip netns exec "$ns_beta" \
+		sysctl -wq net.ipv6.fib_multipath_hash_policy=0
+	ip netns exec "$ns_gamma" sysctl -wq net.ipv6.conf.veth0.accept_ra=0
+	ip netns exec "$ns_gamma" sysctl -wq net.ipv6.conf.veth1.accept_ra=0
+
+	for dev in veth0 veth1 veth2
+	do
+		ip netns exec "$ns_beta" \
+			sysctl -wq net.ipv6.conf.$dev.accept_ra=0
+		ip netns exec "$ns_beta" \
+			sysctl -wq net.ipv6.conf.$dev.ioam6_enabled=1
+	done
+
+	# The ECMP route under test (scenario 1 hashes its selection at
+	# beta's input; scenario 2 at the encapsulating re-lookup).
+	ip -netns "$ns_beta" -6 route add 2001:db8:2:1::2/128 \
+		nexthop via 2001:db8:2:1::2 dev veth1 \
+		nexthop via 2001:db8:2:2::2 dev veth2
+
+	ip -netns "$ns_alpha" -6 route add 2001:db8:2:1::/64 \
+		via 2001:db8:1::1 dev veth0
+	ip -netns "$ns_alpha" -6 route add 2001:db8:3::/64 \
+		via 2001:db8:1::1 dev veth0
+	ip -netns "$ns_gamma" -6 route add 2001:db8:1::/64 \
+		via 2001:db8:2:1::1 dev veth0
+
+	# drop unreachable message when ping 2001:db8:3::2
+	tc -n "$ns_alpha" qdisc add dev veth0 clsact
+	tc -n "$ns_alpha" filter add dev veth0 ingress pref 1 \
+                protocol ipv6 flower skip_hw ip_proto icmpv6 type 1 action drop
+
+}
+
+setup_tunnel()
+{
+	if ! lsmod 2>/dev/null | grep -q "ip6_tunnel"; then
+		modprobe ip6_tunnel &>/dev/null
+		lsmod 2>/dev/null | grep -q "ip6_tunnel" || \
+			ip a 2>/dev/null | grep -q "ip6tnl0" || return 1
+		ip6tnl_loaded=1
+	fi
+
+	ip -netns "$ns_beta" link add name ioamcmp type ip6tnl \
+		mode ip6ip6 local 2001:db8:2:1::1 remote 2001:db8:2:1::2 \
+		2>/dev/null || return 1
+	ip -netns "$ns_beta" link set ioamcmp up
+}
+
+################################################################################
+#                                                                              #
+#                                     MAIN                                     #
+#                                                                              #
+################################################################################
+
+require_command ip
+require_command jq
+require_command ping
+require_command tc
+
+if [ ! -e /proc/sys/net/ipv4/fib_multipath_hash_seed ]; then
+	log_test_skip "Kernel without CONFIG_IP_ROUTE_MULTIPATH."
+	exit "${EXIT_STATUS}"
+fi
+
+setup_topology || { log_test_skip "Setup failed."; exit "${EXIT_STATUS}"; }
+trap cleanup EXIT
+
+# Compat check: can this kernel insert an IOAM trace into a route?
+if ! ip -netns "$ns_alpha" -6 route replace 2001:db8:2:1::/64 \
+	encap ioam6 mode inline trace prealloc type 0x800000 ns 0 size 12 \
+	via 2001:db8:1::1 dev veth0 &>/dev/null
+then
+	log_test_skip "Cannot attach an IOAM trace to a route"
+	exit "${EXIT_STATUS}"
+fi
+
+log_info "IOAM + ECMP (RFC 9486 C2, hash policy 0)"
+
+# Warm up both ECMP links (first-packet neighbor/RA activity) and check
+# basic connectivity before measuring.
+set_alpha_plain
+measure_link 2001:db8:2:1::2 >/dev/null
+
+# Scenario 1 counts the ICMPv6 echo requests sent by alpha.
+if ! set_measure_filters "ip_proto icmpv6 type 128 src_ip 2001:db8:1::2"; then
+	log_test_skip "Cannot install tc flower measurement filters"
+	exit "${EXIT_STATUS}"
+fi
+run_scenario \
+	"Transit input ECMP with/without IOAM" \
+	set_alpha_plain set_alpha_ioam 2001:db8:2:1::2
+
+if setup_tunnel; then
+	# Scenario 2 counts the outer ip6ip6 packets: the echo request is
+	# not visible to flower (outer headers only), so match the pinned
+	# tunnel addresses instead. No need to check return value again as
+	# we have done it at scenario 1
+	set_measure_filters "src_ip 2001:db8:2:1::1 dst_ip 2001:db8:2:1::2"
+	set_beta_tnl
+	measure_link 2001:db8:3::2 >/dev/null
+
+	run_scenario \
+		"Encapsulating node re-lookup ECMP with/without IOAM" \
+		set_beta_tnl set_beta_ioam_encap 2001:db8:3::2
+else
+	log_test_skip "Encapsulating node re-lookup ECMP with/without IOAM (no ip6_tunnel)"
+fi
+
+exit "${EXIT_STATUS}"

-- 
2.56.0

Re: [PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Hangbin Liu]

On Wed, Oct 28, 2026 at 01:45:28PM +0800, Hangbin Liu wrote:
> This series makes IOAM insertion transparent to IPv6 ECMP, as required
> by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
> along the same path as the identical packet without them.

Sorry, forgot to add target repo "net". I will add it in next version.

Thanks
Hangbin

Re: [PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Jakub Kicinski]

On Wed, 28 Oct 2026 16:01:13 +0800 Hangbin Liu wrote:
> On Wed, Oct 28, 2026 at 01:45:28PM +0800, Hangbin Liu wrote:
> > This series makes IOAM insertion transparent to IPv6 ECMP, as required
> > by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
> > along the same path as the identical packet without them.  
> 
> Sorry, forgot to add target repo "net". I will add it in next version.

CI bot says:

The new selftest added alongside this series (tools/testing/selftests/net/ioam6_ecmp.sh)
is failing in our CI on the "Transit input ECMP with/without IOAM" scenario:

  TEST: Transit input ECMP with/without IOAM (diverged on seed 1: without IOAM via
        link A, with IOAM via link B)   [FAIL]
  TEST: Transit input ECMP with/without IOAM (seed 2..12)                [ OK ]
  TEST: Encapsulating node re-lookup ECMP with/without IOAM (seed 1..12) [ OK ]

11 of the 12 "transit input" seeds pass, and the whole "encapsulating node re-lookup"
scenario passes on all seeds, so this looks like an edge case in the Hop-by-Hop
nexthdr-recovery logic used for the policy-0 multipath hash, rather than a test
infra/flake issue. Could you take a look at whether the recovered "virtual" nexthdr
(or the padding/option-length walk used to compute it) is correct for all IOAM option
layouts/trace sizes, since one specific packet still ends up hashed to a different ECMP
sibling than the equivalent packet without IOAM?

Re: [PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Hangbin Liu]

Hi Jakub,
On Wed, Oct 28, 2026 at 04:58:12PM -0700, Jakub Kicinski wrote:
> On Wed, 28 Oct 2026 16:01:13 +0800 Hangbin Liu wrote:
> > On Wed, Oct 28, 2026 at 01:45:28PM +0800, Hangbin Liu wrote:
> > > This series makes IOAM insertion transparent to IPv6 ECMP, as required
> > > by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
> > > along the same path as the identical packet without them.  
> > 
> > Sorry, forgot to add target repo "net". I will add it in next version.
> 
> CI bot says:
> 
> The new selftest added alongside this series (tools/testing/selftests/net/ioam6_ecmp.sh)
> is failing in our CI on the "Transit input ECMP with/without IOAM" scenario:
> 
>   TEST: Transit input ECMP with/without IOAM (diverged on seed 1: without IOAM via
>         link A, with IOAM via link B)   [FAIL]
>   TEST: Transit input ECMP with/without IOAM (seed 2..12)                [ OK ]
>   TEST: Encapsulating node re-lookup ECMP with/without IOAM (seed 1..12) [ OK ]
> 
> 11 of the 12 "transit input" seeds pass, and the whole "encapsulating node re-lookup"
> scenario passes on all seeds, so this looks like an edge case in the Hop-by-Hop
> nexthdr-recovery logic used for the policy-0 multipath hash, rather than a test
> infra/flake issue. Could you take a look at whether the recovered "virtual" nexthdr
> (or the padding/option-length walk used to compute it) is correct for all IOAM option
> layouts/trace sizes, since one specific packet still ends up hashed to a different ECMP
> sibling than the equivalent packet without IOAM?

Thanks for the report. I run it locally multi times and didn't got this issue.
I will try reproduce it first and check the reason.

Thanks
Hangbin

Re: [PATCH 0/3] net: ioam6: makes IOAM insertion transparent to IPv6 ECMP

[Hangbin Liu]

Hi Jakub,
On Wed, Oct 28, 2026 at 04:58:12PM -0700, Jakub Kicinski wrote:
> On Wed, 28 Oct 2026 16:01:13 +0800 Hangbin Liu wrote:
> > On Wed, Oct 28, 2026 at 01:45:28PM +0800, Hangbin Liu wrote:
> > > This series makes IOAM insertion transparent to IPv6 ECMP, as required
> > > by RFC 9486 (C2): a packet carrying IOAM options must be forwarded
> > > along the same path as the identical packet without them.  
> > 
> > Sorry, forgot to add target repo "net". I will add it in next version.
> 
> CI bot says:
> 
> The new selftest added alongside this series (tools/testing/selftests/net/ioam6_ecmp.sh)
> is failing in our CI on the "Transit input ECMP with/without IOAM" scenario:
> 
>   TEST: Transit input ECMP with/without IOAM (diverged on seed 1: without IOAM via
>         link A, with IOAM via link B)   [FAIL]
>   TEST: Transit input ECMP with/without IOAM (seed 2..12)                [ OK ]
>   TEST: Encapsulating node re-lookup ECMP with/without IOAM (seed 1..12) [ OK ]
> 
> 11 of the 12 "transit input" seeds pass, and the whole "encapsulating node re-lookup"
> scenario passes on all seeds, so this looks like an edge case in the Hop-by-Hop
> nexthdr-recovery logic used for the policy-0 multipath hash, rather than a test
> infra/flake issue. Could you take a look at whether the recovered "virtual" nexthdr
> (or the padding/option-length walk used to compute it) is correct for all IOAM option
> layouts/trace sizes, since one specific packet still ends up hashed to a different ECMP
> sibling than the equivalent packet without IOAM?

OK, after running the test 50 times on a debug kernel and over 70 times on a
normal kernel, I've finally reproduced the issue and identified the root cause.

The reason is the ping `-w` parameter. I originally thought it set the maximum
wait time, but it actually keeps sending packets regardless of how many packets
have been sent or received. Combined with `-i 0.2`, it sometimes sends 10 packets,
which carry over into the seed 1 test and affect the result.

That's why the debug kernel case worked and the normal kernel failed in Jakub's
environment, while it always passed in my laptop test environment.

After removing the "-W 1 -w 2" parameters, I ran the test 200 times and all
passed. I will post the fix in the next version.

TL;DR
--------------------

Here are my testing steps. After reproducing the issue, I dumped the tc output
and found 10 extra ICMPv6 echo requests.

filter protocol ipv6 pref 1 flower chain 0
filter protocol ipv6 pref 1 flower chain 0 handle 0x1
  eth_type ipv6
  ip_proto icmpv6
  src_ip 2001:db8:1::2
  icmp_type 128
  skip_hw
  not_in_hw
        action order 1: gact action pass
         random type none pass val 0
         index 1 ref 1 bind 1 installed 0 sec
        Action statistics:
        Sent 1040 bytes 10 pkt (dropped 0, overlimits 0 requeues 0)
        backlog 0b 0p requeues 0

on gamma veth0 at the start of seed 1 testing, but *a0* is 0. This means some
echo requests are still in flight during this window. I then captured traffic
with tcpdump and confirmed those echo requests are present on gamma veth0:

15:47:09.161066 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 1, length 64
15:47:09.161083 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 2, length 64
15:47:09.161086 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 3, length 64
15:47:09.161088 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 4, length 64
15:47:09.161091 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 5, length 64
15:47:09.161093 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 6, length 64
15:47:09.161095 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 7, length 64
15:47:09.161097 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 8, length 64
15:47:09.161100 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 9, length 64
15:47:09.161102 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3254, seq 10, length 64

The question is: where do these 10 packets come from, and why are they sent
within this narrow window?

We expect only 3 echo requests during the ECMP link warm-up, which does not
match the tcpdump capture. The timestamps, however, show the traffic starts
during the warm-up phase. The subsequent echo requests on veth1 are correctly
sent as 3 packets with a 0.2s interval.

15:47:09.221513 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3284, seq 1, length 64
15:47:09.426247 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3284, seq 2, length 64
15:47:09.633125 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3284, seq 3, length 64

It looks like the skbs were held in the unresolved neighbour queue and flushed
after receiving the NA, causing the packets to be delivered to gamma ingress later.

Then I tried to send 1 packet on before the warm up.
ip netns exec "$ns_alpha" ping -6 -q -c 1 -i 0.2 -W 1 -w 2 2001:db8:2:1::2  &> /dev/null

Now the packet shows

16:05:20.744986 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 1, length 64
16:05:20.744987 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 2, length 64
16:05:20.744988 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 3, length 64
16:05:20.744988 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 4, length 64
16:05:20.744989 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 5, length 64
16:05:20.744990 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 6, length 64
16:05:20.744990 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 7, length 64
16:05:20.744991 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 8, length 64
16:05:20.744992 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 9, length 64
16:05:20.744992 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3765, seq 10, length 64
16:05:20.744992 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3774, seq 1, length 64
16:05:20.936373 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3774, seq 2, length 64
16:05:21.137153 IP6 2001:db8:1::2 > 2001:db8:2:1::2: ICMP6, echo request, id 3774, seq 3, length 64

The warm-up packets are correctly back to 3 with a 0.2s interval. However, the
first `-c 1` ping still generates 10 packets. With set -x, it shows

+ ip netns exec ns_alpha-nBg602 ping -6 -q -c 1 -i 0.2 -W 1 -w 2 2001:db8:2:1::2
PING 2001:db8:2:1::2 (2001:db8:2:1::2) 56 data bytes

--- 2001:db8:2:1::2 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 1865ms

After checking, I finally got the answer:

-w deadline: Specify a timeout, in seconds, before ping exits regardless of
how many packets have been sent or received.

When `-w` is used, the `-c` packet count no longer acts as a hard limit. Ping
keeps sending packets until either the expected number of replies are received
or the timeout expires. In our case, the timeout was hit. All 10 packets
(sent at a 0.2s interval within the 2s window) got deferred and surfaced
during the seed 1 test, corrupting the result.

Thanks
Hangbin