Re: SR-IOV with mlx4 on ConnectX-2 fails with DMAR errors

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Wed, Dec 14, 2016 at 10:06:13AM -0500, Joshua McBeth wrote:

> > Does this make your 4.8 kernel work? If yes, then I suspect mlx4 has
> > broken IB_DEVICE_LOCAL_DMA_LKEY with SRIOV.. Leon? mlx5 has this
> > broken, doesn't it?

> With 4.8.1 and the below applied to the SR-IOV host and guest kernels,
> SR-IOV functions in both the SR-IOV host and guests and there are no
> DMAR errors emitted.

So strange.

Looking at your original report you see these errors:

[  107.137484] DMAR: [DMA Read] Request device [05:06.1] fault addr

But I don't see where 05:06.01 is a PCI device. That seems like a big
problem.

Based on that this looks like a Mellanox bug where
IB_DEVICE_LOCAL_DMA_LKEY is causing the wrong PCI BDF to be provided
as the requestor. Mellanox will have to help you futher, you are
running the latest firmware, right?

> The NFS/RDMA client in the guest does not work on the SR-IOV virtual
> function with the NFS/RDMA server of the host on the SR-IOV physical
> function, but this may be something else I need to troubleshoot
> further, as both IPoIB and synthetic RDMA traffic passes between the
> guest, host, and remote node just fine.  The remote node's NFS/RDMA
> client is additionally able to function with the host's NFS/RDMA
> server on the SR-IOV physical function.

Try removing IB_DEVICE_LOCAL_DMA_LKEY from the mlx4 driver entirely..

> > It would also be very helpful to try and determine what memory the NIC is
> > trying to read.. If it is the ipoib packet or some mlx4 internal
> > thing.

> How can I determine this?

Print out the dma address of the skb when the SEND is submitted in
ipoib and see if it is similar to the DMAR region..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html