From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Tucker Subject: Re: NFS over RDMA benchmark Date: Tue, 30 Apr 2013 13:58:34 -0500 Message-ID: <5180145A.5030305@opengridcomputing.com> References: <0EE9A1CDC8D6434DB00095CD7DB873462CF96C65@MTLDAG01.mtl.com> <62745258-4F3B-4C05-BFFD-03EA604576E4@ornl.gov> <0EE9A1CDC8D6434DB00095CD7DB873462CF9715B@MTLDAG01.mtl.com> <20130423210607.GJ3676@fieldses.org> <0EE9A1CDC8D6434DB00095CD7DB873462CF988C9@MTLDAG01.mtl.com> <20130424150540.GB20275@fieldses.org> <20130424152631.GC20275@fieldses.org> <0EE9A1CDC8D6434DB00095CD7DB873462CF9A820@MTLDAG01.mtl.com> <20130428144248.GA2037@fieldses.org> <0EE9A1CDC8D6434DB00095CD7DB873462CF9C90C@MTLDAG01.mtl.com> <517FD327.3060901@talpey.com> <0EE9A1CDC8D6434DB00095CD7DB873462CF9CBD0@MTLDAG01.mtl.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <0EE9A1CDC8D6434DB00095CD7DB873462CF9CBD0-fViJhHBwANKuSA5JZHE7gA@public.gmane.org> Sender: linux-nfs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Yan Burman Cc: Tom Talpey , "J. Bruce Fields" , Wendy Cheng , "Atchley, Scott" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Or Gerlitz List-Id: linux-rdma@vger.kernel.org On 4/30/13 9:38 AM, Yan Burman wrote: > >> -----Original Message----- >> From: Tom Talpey [mailto:tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org] >> Sent: Tuesday, April 30, 2013 17:20 >> To: Yan Burman >> Cc: J. Bruce Fields; Wendy Cheng; Atchley, Scott; Tom Tucker; linux- >> rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Or Gerlitz >> Subject: Re: NFS over RDMA benchmark >> >> On 4/30/2013 1:09 AM, Yan Burman wrote: >>> I now get up to ~95K IOPS and 4.1GB/sec bandwidth. >>> ... >>> ib_send_bw with intel iommu enabled did get up to 4.5GB/sec >> BTW, you may want to verify that these are the same GB. Many benchmarks >> say KB/MB/GB when they really mean KiB/MiB/GiB. >> >> At GB/GiB, the difference is about 7.5%, very close to the difference between >> 4.1 and 4.5. >> >> Just a thought. > The question is not why there is 400MBps difference between ib_send_bw and NFSoRDMA. > The question is why with IOMMU ib_send_bw got to the same bandwidth as without it while NFSoRDMA got half. NFSRDMA is constantly registering and unregistering memory when you use FRMR mode. By contrast IPoIB has a descriptor ring that is set up once and re-used. I suspect this is the difference maker. Have you tried running the server in ALL_PHYSICAL mode, i.e. where it uses a DMA_MR for all of memory? Tom > >From some googling, it seems that when IOMMU is enabled, dma mapping functions get a lot more expensive. > Perhaps that is the reason for the performance drop. > > Yan -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html