mlx5: dma-buf MR on GH200 GPU memory wraps every 2^k bytes when the base is 2^k-aligned (k >= 32)

mlx5: dma-buf MR on GH200 GPU memory wraps every 2^k bytes when the base is 2^k-aligned (k >= 32)

[Dezhen Lu]

Hi,

On Grace Hopper (GH200) with a ConnectX HCA, a memory region registered
through ibv_reg_dmabuf_mr() on GPU memory behaves as if its mapping wraps
every 2^k bytes when the region's base address is aligned to exactly 2^k,
with k >= 32. RDMA operations complete successfully with the full byte
count and no error, but they target the wrong location:

   - local side, base aligned to 2^32: a READ into offset X lands at
     X mod 4 GiB. Destination blocks at offsets >= 4 GiB stay untouched
     (zero), and blocks below 4 GiB get overwritten;
   - remote side, base aligned to 2^33: a READ from offset X >= 8 GiB
     returns the data at X - 8 GiB.

The same memory registered through nvidia_peermem (plain ibv_reg_mr) is
correct at the same alignments, and bases aligned to 2^31 or less are
correct with dma-buf.

Environment:
   - 2 nodes, 1x NVIDIA GH200 480GB each, aarch64
   - Ubuntu kernel 6.8.0-136-generic, inbox mlx5_core / mlx5_ib
   - Mellanox device MT41692 (mlx5_0), RC transport; firmware 32.38.3056 on
     one node and 32.43.1014 on the other (the wrap occurs on both)
   - NVIDIA driver 595.58.03, CUDA 13.x
   - dma-buf fd from cuMemGetHandleForAddressRange() on cudaMalloc memory

Evidence:

1. UCX (bundled with NIXL 1.5.0), 15.4 GiB region per side, 9216 scattered
    64 KiB READs per iteration. The region is carved out of a larger
    allocation so that its base has exactly the chosen alignment; UCX's
    debug log shows that exact range registered with dma-buf.

      initiator base 2^32, dma-buf      2/2 iterations bad (x2 runs); the
                                        all-zero blocks are exactly those
                                        at local offsets >= 4 GiB
      target base 2^33, dma-buf         2/2 bad (x2 runs); exactly the
                                        blocks at remote offsets >= 8 GiB
      same two, UCX_CUDA_COPY_DMABUF=no 0 bad (same zero-copy rc path and
                                        bandwidth)
      initiator base 2^31, dma-buf      0 bad

    Without forcing anything, 4 of 30 fresh process pairs were bad. Each had
    a base aligned to exactly 2^32 or 2^33 on the failing side. I checked
    the wrapped values byte-for-byte against the fill pattern. Re-reading
    gives the same wrong result.

2. Without UCX: plain ibv_reg_dmabuf_mr() vs ibv_reg_mr() (Mooncake
    transfer engine, RDMA loopback through mlx5_0 on the node with
    firmware 32.43.1014). 12 GiB MRs carved from cudaMalloc so the base is
    aligned to exactly 2^k; 1152 scattered 64 KiB READ+WRITE blocks plus
    transfers ending at the last byte of the MR, all checked byte by byte:

      dma-buf, remote base 2^31           clean
      dma-buf, remote base 2^32           offsets 0-4 GiB correct, 4-12 GiB
                                          all wrong (READ: 781/1152 bad,
                                          776 all-zero)
      dma-buf, remote base 2^33           0-8 GiB correct, 8-12 GiB wrong
      dma-buf, local base 2^32            local offsets >= 4 GiB wrong
      ibv_reg_mr (peermem), base 2^32/33  clean
      dma-buf, base 2^32, MR split into
        regions of <= 2 GiB               clean

    An earlier unforced 16 GiB dma-buf run whose base happened to be
    0xf1b700000000 (exactly 2^32) showed the same pattern; runs with
    bases aligned to 2^29/2^30 were clean.

3. End to end: vLLM P/D with NIXL returns garbage when the decode KV
    cache's registered base is 2^32-aligned. Per-layer checksums show only
    the layers below the 4 GiB offset arrive correctly.

My guess is that the page size chosen for the dma-buf umem
(ib_umem_find_best_pgsz() on the dma-buf scatterlist, then the mlx5 mkey
setup) can reach 4 GiB or more here. GH200 exports large physically
contiguous GPU segments, so a 2^k-aligned IOVA allows a 2^k "page", and
something in the mlx5 path then handles page sizes >= 2^32 incorrectly.
I haven't verified this in the kernel source.

Reproducers and raw data:
   - NIXL-only script: https://github.com/ai-dynamo/nixl/issues/2512
   - vLLM report: https://github.com/vllm-project/vllm/issues/64048
   - Mooncake report (raw verbs, item 2):  
https://github.com/kvcache-ai/Mooncake/issues/5015

I can run a patch or collect more data (e.g. the page size chosen at
registration) on these nodes.

Thanks,
<name>

Re: mlx5: dma-buf MR on GH200 GPU memory wraps every 2^k bytes when the base is 2^k-aligned (k >= 32)

[Jason Gunthorpe]

On Wed, Oct 28, 2026 at 10:34:30AM +0100, Dezhen Lu wrote:
> Hi,
> 
> On Grace Hopper (GH200) with a ConnectX HCA, a memory region registered
> through ibv_reg_dmabuf_mr() on GPU memory behaves as if its mapping wraps
> every 2^k bytes when the region's base address is aligned to exactly 2^k,
> with k >= 32. RDMA operations complete successfully with the full byte
> count and no error, but they target the wrong location:

This was fixed a while ago, upgrade your software.

Jason