Re: NFS Server Issues with RDMA in Kernel 6.13.6

[Robin Murphy <robin.murphy@arm.com>]

On 2025-03-13 7:20 pm, Lucas via Bugspray Bot wrote:
[...]
> system: Suermicro AS-4124GS-TNR
> cpu: AMD EPYC 7H12 64-Core Processor
> ram: 512G
> rdma nic: Mellanox Technologies MT2910 Family [ConnectX-7]
> 
> 
>>> [  976.677373]  __dma_map_sg_attrs+0x139/0x1b0
>>> [  976.677380]  dma_map_sgtable+0x21/0x50
>>
>> So, here (and above) is where we leave the NFS server and venture into
>> the IOMMU layer. Adding the I/O folks for additional eyes.
>>
>> Can you give us the output of:
>>
>>    $ scripts/faddr2line drivers/iommu/iova.o alloc_iova+0x92
>>
> 
> 
> root@test:/usr/src/linux-6.13.6# scripts/faddr2line drivers/iommu/iova.o alloc_iova+0x92
> alloc_iova+0x92/0x290:
> __alloc_and_insert_iova_range at /usr/src/linux-6.13.6/drivers/iommu/iova.c:180
> (inlined by) alloc_iova at /usr/src/linux-6.13.6/drivers/iommu/iova.c:263
> root@test:/usr/src/linux-6.13.6#

OK so this is waiting for iova_rbtree_lock to get into the allocation 
slowpath since there was nothing suitable in the IOVA caches. Said 
slowpath under the lock is unfortunately prone to being quite slow, 
especially as the rbtree fills up with massive numbers of relatively 
small allocations (which I'm guessing I/O with a 4KB block size would 
tend towards). If you have 256 threads all contending the same path then 
they could certainly end up waiting a while, although they shouldn't be 
*permanently* stuck...

Thanks,
Robin.