Re: NFSoRDMA Fails for max_sge Less Than 18

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Wed, Jan 11, 2017 at 03:15:04PM -0600, Steve Wise wrote:

> > That would be good, it could bounce buffer, build temporary MRs,
> > or repeat RDMA WRITE/READ as appropriate..
> 
> We're talking SEND opcodes where the ULP transfers data in the SEND that
> includes the command request or reply instead of having the target/server side
> issue RDMA opcodes.

Yes, I know, handling READ/WRITE also would be the 'full generality'
of a core API in this area.

> > The ULP really is not the right place to put this trade off logic
> > since it will be very device specific..
> 
> A temporary REG_MR would do the trick...assuming it can contain the SGE depth as
> well.  But issuing multiple SEND operations involves ULP protocol headers so
> that couldn't be done by a core service...

Right, for SEND the core API would need to pack the given SG list into
whatever the device limit is, using bounce buffers if necessary.

For NFS, since the data is coming out of the page cache, a MR should
be always be buildable for the data? That means 1 send sge for the
header and 1 for the MR?

Basically the core API would be a packing API - input a SG list and a
MR pool and you get back another SG list that fits within the device
limits and can be used in a WR.

The core figures out how to optimally compute the packed SG list.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html