From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Gunthorpe Subject: Re: NFSoRDMA Fails for max_sge Less Than 18 Date: Wed, 11 Jan 2017 14:34:16 -0700 Message-ID: <20170111213416.GA30681@obsidianresearch.com> References: <027101d26c2c$c94467f0$5bcd37d0$@opengridcomputing.com> <28B0D906-7BDB-4B87-94E9-6BE263BFBFF7@oracle.com> <20170111211123.GD28917@obsidianresearch.com> <042601d26c4f$c7190e90$554b2bb0$@opengridcomputing.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <042601d26c4f$c7190e90$554b2bb0$@opengridcomputing.com> Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Steve Wise Cc: 'Chuck Lever' , "'Amrani, Ram'" , "'Elior, Ariel'" , "'Kalderon, Michal'" , linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, 'Hariprasad S' , 'Faisal Latif' , 'Doug Ledford' List-Id: linux-rdma@vger.kernel.org On Wed, Jan 11, 2017 at 03:15:04PM -0600, Steve Wise wrote: > > That would be good, it could bounce buffer, build temporary MRs, > > or repeat RDMA WRITE/READ as appropriate.. > > We're talking SEND opcodes where the ULP transfers data in the SEND that > includes the command request or reply instead of having the target/server side > issue RDMA opcodes. Yes, I know, handling READ/WRITE also would be the 'full generality' of a core API in this area. > > The ULP really is not the right place to put this trade off logic > > since it will be very device specific.. > > A temporary REG_MR would do the trick...assuming it can contain the SGE depth as > well. But issuing multiple SEND operations involves ULP protocol headers so > that couldn't be done by a core service... Right, for SEND the core API would need to pack the given SG list into whatever the device limit is, using bounce buffers if necessary. For NFS, since the data is coming out of the page cache, a MR should be always be buildable for the data? That means 1 send sge for the header and 1 for the MR? Basically the core API would be a packing API - input a SG list and a MR pool and you get back another SG list that fits within the device limits and can be used in a WR. The core figures out how to optimally compute the packed SG list. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html