Re: [RFC 0/8] Reliably generate large request from SRP

David Dillow
2011-02-19T00:06:04Z

On Tue, 2011-01-18 at 21:31 -0800, Roland Dreier wrote:
> Hi Dave,
>
> > Now that at least one vendor is implementing full support for the SRP
> > indirect memory descriptor tables, we can safely expand the sg_tablesize,
> > and realize some performance gains, in many cases quite large. I don't
> > have vendor code that implements the full support needed for safety, but
> > the rareness of FMR mapping failures allows the mapping code to function,
> > at a risk, with existing targets.
>
> Have you considered using memory registration through a send queue (from
> the base memory management extensions)? mlx4 at least has support for
> this operation, which would let you pre-allocate everything and avoid
> the possibility of failure (I think).

I'm looking at this now, and it may not be too bad; I think I can re-use
the mapping machinery pretty easily without it getting too ugly.

I do have a few questions, though. It looks like I can only have up to
256 keys for each fast page registration MR, so that seems to limit the
mappings in flight -- I can see cases where we may run out and have to
push the request back to the mid-layer. We'd also be cycling through the
key space pretty quickly, negating some of the advantages of being able
to invalidate the mappings quickly.

I'm thinking about just allocating an MR per request -- this seems like
a simple way to avoid the restrictions, but would of course use more
MRs. mlx4 seems to allow just shy of 512 K of them, so that's probably
not a large concern.

What do you think? Is that overkill? Do I misunderstand the BMM
extensions?
--
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office