Re: [PATCH rdma-next v1 0/2] RDMA: fix cross-NIC same-host IPv6 RDMA-CM connect

[Alex Timofeyev <sashka@ankey.net>]

> You need to setup policy routing or a VRF so these local routes don't
> show up the way they do. We shouldn't be mangling the loopback routes
> in the kernel, and removing the check is not correct.

You're right, and thanks for pushing back on this.

I've been testing VRF all day and it solves the problem completely,
with no kernel changes. Putting each NIC in its own VRF moves that
NIC's addresses out of the global v6 local table (255), so the
cross-NIC same-host destination no longer collapses to lo -- it
resolves on-wire to the real egress device:

  # enp49s0np0 -> vrfdata0 (table 100), enp193s0np0 -> vrfdata1 (101)
  $ ip -6 route get <dst-on-other-local-nic> from <src> oif enp49s0np0
  ... dev enp49s0np0 ...        # was: local ... dev lo

So neither addr_resolve_neigh() nor validate_ipv6_net_dev() ever hits
the local shortcut my patches were rewriting. The check stays correct
and the loopback routes stay untouched, exactly as you said.

I validated it with the actual workload that motivated the series, not
just rping: a 3-node DAOS cluster (two engines per host, one per NUMA
NIC) on a kernel WITHOUT these two patches -- only the upstream prereqs
fa29d1e8877b + c31e4038c97f. All 6 ranks join and stay joined, and a
pool created across all of them comes up healthy. That covers both the
same-host cross-NIC path and the cross-host path, all over v6 RoCEv2.

One observation in case it's useful to others who hit this: VRF
self-RPC relies on c31e4038c97f -- once an address is homed in a VRF
table, is_dst_local() needs the RTF_LOCAL test rather than the old
IFF_LOOPBACK one to still recognize it as local. That commit is in
v6.18 but not in linux-6.6.y, which is the stable base we (and
presumably other RoCE-on-6.6 users) are on; might be worth a stable
backport on its own merits, independent of this series.

Given all that, please drop this series -- I'm withdrawing it. The
right fix here is configuration, not a kernel patch. Appreciate the
review and the steer toward VRF.

Thanks,
Alex