Re: NFS over RDMA benchmark

[Tom Tucker <tom@opengridcomputing.com>]

On 4/30/13 9:38 AM, Yan Burman wrote:
>
>> -----Original Message-----
>> From: Tom Talpey [mailto:tom@talpey.com]
>> Sent: Tuesday, April 30, 2013 17:20
>> To: Yan Burman
>> Cc: J. Bruce Fields; Wendy Cheng; Atchley, Scott; Tom Tucker; linux-
>> rdma@vger.kernel.org; linux-nfs@vger.kernel.org; Or Gerlitz
>> Subject: Re: NFS over RDMA benchmark
>>
>> On 4/30/2013 1:09 AM, Yan Burman wrote:
>>> I now get up to ~95K IOPS and 4.1GB/sec bandwidth.
>>> ...
>>>   ib_send_bw with intel iommu enabled did get up to 4.5GB/sec
>> BTW, you may want to verify that these are the same GB. Many benchmarks
>> say KB/MB/GB when they really mean KiB/MiB/GiB.
>>
>> At GB/GiB, the difference is about 7.5%, very close to the difference between
>> 4.1 and 4.5.
>>
>> Just a thought.
> The question is not why there is 400MBps difference between ib_send_bw and NFSoRDMA.
> The question is why with IOMMU ib_send_bw got to the same bandwidth as without it while NFSoRDMA got half.
NFSRDMA is constantly registering and unregistering memory when you use 
FRMR mode. By contrast IPoIB has a descriptor ring that is set up once 
and re-used. I suspect this is the difference maker. Have you tried 
running the server in ALL_PHYSICAL mode, i.e. where it uses a DMA_MR for 
all of memory?

Tom
> >From some googling, it seems that when IOMMU is enabled, dma mapping functions get a lot more expensive.
> Perhaps that is the reason for the performance drop.
>
> Yan