[PATCH rdma-next v1 1/2] RDMA/core: use destination netdev MAC for cross-NIC same-host local dst

[Alex Timofeyev <sashka@ankey.net>]

addr_resolve_neigh() treats every is_dst_local() destination as loopback
and copies the source device's MAC into the path record's destination MAC
(dst_dev_addr <- src_dev_addr). That is correct for true loopback (source
and destination on the same netdev), but wrong when the local destination
address lives on a different netdev of the same host.

In that cross-NIC same-host case the destination NIC will not accept a
frame whose destination MAC is the source NIC's MAC, and drops it in
hardware before it reaches the peer. rdma_resolve_addr() and
ib_send_cm_req() both return success, but the CM REQ never arrives and the
connection times out.

Look up the netdev that owns the destination address and copy its MAC into
dst_dev_addr instead. Fall back to the source MAC when no netdev claims the
address (true loopback), preserving the existing behaviour.

This was observed with two RoCEv2 ConnectX-7 ports on the same host, each
holding a global IPv6 GID, when one process pinned per NUMA NIC connected
to the other over RDMA-CM: the resolved destination MAC was the source
port's MAC instead of the destination port's, and the REQ was silently
dropped. With the fix the resolved MAC is the destination port's and the
connection completes.

Fixes: c31e4038c97f ("RDMA/core: Use route entry flag to decide on loopback traffic")
Cc: stable@vger.kernel.org
Cc: Parav Pandit <parav@nvidia.com>
Signed-off-by: Alex Timofeyev <sashka@ankey.net>
---
 drivers/infiniband/core/addr.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c
index 7e62b5b1ffaa..84aa43436bfe 100644
--- a/drivers/infiniband/core/addr.c
+++ b/drivers/infiniband/core/addr.c
@@ -451,10 +451,26 @@ static int addr_resolve_neigh(const struct dst_entry *dst,
 			      u32 seq)
 {
 	if (is_dst_local(dst)) {
-		/* When the destination is local entry, source and destination
-		 * are same. Skip the neighbour lookup.
+		struct net_device *dst_ndev;
+
+		/* When the destination is local, source and destination are on
+		 * the same host. For true loopback (same netdev) the source and
+		 * destination MACs are equal, but when the destination address
+		 * lives on a different netdev of the same host the destination
+		 * MAC must be that netdev's MAC -- otherwise the destination NIC
+		 * silently drops the frame. Look up the netdev that owns the
+		 * destination address and copy its MAC; fall back to the source
+		 * MAC if no netdev claims the address.
 		 */
-		memcpy(addr->dst_dev_addr, addr->src_dev_addr, MAX_ADDR_LEN);
+		rcu_read_lock();
+		dst_ndev = rdma_find_ndev_for_src_ip_rcu(dev_net(dst->dev), dst_in);
+		if (!IS_ERR(dst_ndev))
+			memcpy(addr->dst_dev_addr, dst_ndev->dev_addr,
+			       MAX_ADDR_LEN);
+		else
+			memcpy(addr->dst_dev_addr, addr->src_dev_addr,
+			       MAX_ADDR_LEN);
+		rcu_read_unlock();
 		return 0;
 	}
 
-- 
2.40.4