Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

[Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]

On Thu, Aug 20, 2015 at 01:04:13PM -0600, Jason Gunthorpe wrote:
> Trying to decouple the sub resources, ie by separately pooling the
> MR/SQE/etc, is just unnecessary complexity, IMHO.. NFS client already
> had serioues bugs in this area.
> 
> So, I turn to the idea that every ULP should work as the above, which
> means when it gets to working on a 'slot' that implies there is an
> actual struct ib_mr resource guaranteed available. This is why I
> suggested using the 'struct ib_mr' to guide the SG construction even if
> the actual HW MR isn't going to be used. The struct ib_mr is tied to
> the slot, so using it has no cost.

How is this going to work for drivers that might consumer multiple
MRs per request like SRP or similar upcoming block drivers?  Unless
you want to allocate a potentially large number of MRs for each
request that scheme doesn't work.

> This forces the ULP to deal with many of the issues. Having a slot
> means guarenteed minimum avaiable MR,SQE,CQE resources. That
> guarenteed minimum avoids the messy API struggle in my prior writings.
> 
> .. and maybe the above is even thinking too small, to Christoph's
> earlier musings, I wonder if a slot based middle API could hijack the
> entire SCQ processing and have a per-slot callback scheme
> instead. That kind of intervention is exactly what we'd need to
> trivially hide the FMR difference.

FYI, I have working early patches to do per-WR completion callback,
I'll post them after I get them into a slightly better shape.

As for your grand schemes:  I like some of the idea there, but we
need to get there gradually.  I'd much prefer to finish Sagi's simple
scheme, get my completion work in, add abstractions for RDMA READ and
WRITE scatterlist mapping and build things up slowly.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html