[PATCH 0/2] IB/hfi1: fix the PIO_CRED credit-return mmap

[Shuhei Takeshita <jyohuku.alterego@gmail.com>]

The PIO_CRED case of hfi1_file_mmap() hands user space the credit-return
page for its send context.  On a two-socket host with a translating IOMMU
it instead returns a page mapped from a frame above MAXPHYADDR, and the
first user read takes "Corrupted page table" / "Oops: Bad pagetable".
Correcting only the offset arithmetic replaces the Oops with a silent
wrong-page mapping, so every transfer that uses send PIO hangs instead.

The failure is intermittent: it depends on which of the credit-return
pages the context's entry lands on, which follows the hardware send
context index and so varies from boot to boot.

Patch 1 fixes the node used to resolve the credit-return buffer, which
has been wrong since the driver was merged.  Patch 2 fixes the byte
offset applied to a typed pointer and the use of dma_mmap_coherent(),
both introduced when this case was converted to the DMA API.  They are
separate patches because they have different Fixes: tags, but only the
two together make the mapping correct: patch 2 without patch 1 turns the
hang into an -ENXIO from iommu_dma_mmap()'s bounds check, because the
cross-node offset exceeds the buffer.

Found while debugging psm2_ep_open() Oopsing a Dell T7610 (Xeon E5-2650
v2, Intel IOMMU in DMA-FQ mode) talking to a Threadripper PRO 3995WX,
both Omni-Path 100.  With the series applied to both hosts,
psm2_ep_open() succeeds, send PIO, send DMA and the default mixed mode
all work, and MPI over the PSM2 MTL sustains 95 Gb/s.  Forcing send PIO
only (PSM2_SDMA=0), which hung indefinitely before, now completes.

Based on rdma/for-rc at 31b7c700670830a0e8a4cdcd451c88a13cc5dc48.

Shuhei Takeshita (2):
  IB/hfi1: Resolve the credit-return buffer through the send context's
    node
  IB/hfi1: Fix the PIO_CRED credit-return mmap

 drivers/infiniband/hw/hfi1/file_ops.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

-- 
2.43.0