net/mlx4: no ib device for Ethernet ports without IBoE since v6.6

net/mlx4: no ib device for Ethernet ports without IBoE since v6.6

[Peter Morrow]

Hi,

Since v6.6, mlx4 no longer creates an RDMA device for an Ethernet port
when the device doesn't advertise IBoE. On Azure, ConnectX-3 VFs are in
that state, so DPDK's mlx4 PMD can no longer open them:

  net_mlx4: cannot access device, is mlx4_ib loaded?

Petr, thanks for looking at this off-list; as you suggested, I'm reporting it here.

What changed
------------
Commit 7d22b1cb9d84 ("mlx4: Connect the infiniband part to the
auxiliary bus") creates the "ib" auxiliary device only when
is_ib_supported() sees an IB port or MLX4_DEV_CAP_FLAG_IBOE. Before it,
mlx4_ib_add() was called for every device and accepted any IB or
Ethernet port via mlx4_foreach_ib_transport_port(). An Ethernet port
without IBoE got a raw-packet-only port. mlx4_port_immutable() still
has that branch (RDMA_CORE_PORT_RAW_PACKET without RoCE), but nothing
can reach it now.

DPDK's mlx4 PMD needs only that: it opens the device through ibverbs
and creates IBV_QPT_RAW_PACKET QPs. It does not ask for RoCE.
mlx5_ib handles the equivalent case by creating the device with raw_eth_profile when
RoCE is disabled.

Evidence
--------
Azure VM, ConnectX-3 VF [15b3:1004], Debian kernel 6.12.107. Mainline
has the same check in is_ib_supported().

- mlx4_core and mlx4_en bind the VF and the netdev works. mlx4_ib is
  loaded, but there is mlx4_core.eth.N and no mlx4_core.ib.N, and
  mlx4_ib never probes: its version banner is missing from dmesg.
- With mlx4_core debug_level=1, the DEV_CAP flags dump goes from
  "UD multicast support" (bit 21) to "Unicast loopback support"
  (bit 32). "IBoE support" (bit 30) is absent on both VFs.

Tested change
-------------
This restores the pre-6.6 behaviour. I built it as an out-of-tree
mlx4_core.ko against the 6.12.107 Debian headers and loaded it at boot
on an Azure VM with two ConnectX-3 VFs. Both got an ib auxiliary device
and an RDMA device, DPDK's netvsc PMD attached both VFs, and traffic
flowed through them. On mainline I have only compile-tested it.

--- a/drivers/net/ethernet/mellanox/mlx4/intf.c
+++ b/drivers/net/ethernet/mellanox/mlx4/intf.c
@@ static bool is_ib_supported(struct mlx4_dev *dev)
  for (int port = 1; port <= dev->caps.num_ports; port++)
- if (dev->caps.port_type[port] == MLX4_PORT_TYPE_IB)
+ if (dev->caps.port_type[port] == MLX4_PORT_TYPE_IB ||
+    dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH)
  return true;
 
- if (dev->caps.flags & MLX4_DEV_CAP_FLAG_IBOE)
- return true;
-
  return false;

I'm not proposing this for merging as it stands as I really do not know this area, rather I am
a user of dpdk via the dpdk application vpp (the vector packet processor).

I'm very happy to test any patches that folks think are a better fix for this issue. Also note, I
used an LLM to assist with debugging/analysis here.

Thanks,
Peter