Re: [RFC 0/8] Reliably generate large request from SRP

David Dillow
2011-02-23T01:23:34Z

On Tue, 2011-02-22 at 15:38 -0800, Roland Dreier wrote:
> On Tue, Feb 22, 2011 at 6:49 AM, David Dillow <dillowda-1Heg1YXhbW8@public.gmane.org> wrote:
> > Thanks for the note Or -- you confirmed my suspensions after testing
> > last night. Looks like multiple MRs per request, then.
>
> Multiple MRs per request, or one fast reg MR per request?

Multiple fast reg MRs per request -- on mlx4, they max out at 511
entries, so I'd need 5 FRMRs to get to 8 MB requests. I'm scaling based
on max_sect and the device attributes, so if one doesn't want more than
1 MB requests, they don't pay for the extra MRs -- or if other cards
support larger/smaller fast reg MRs, then the code does the right thing.

> This really should be roughly the same as old-style FMRs -- in both
> cases, we need one "registration" per request that might be in flight.
> Old-style FMRs and new IB-spec fast reg MRs should be roughly
> the same in terms of resources consumed...

Yeah, it seems pretty close overall. Fast reg MRs put more of it in view
of SRP, where as old-style FMRs hid the bookkeeping in ib_pool_fmr.

As an aside, is there any guarantee that ib_post_send() will not modify
the list of work requests? I currently make no assumptions about that,
but it would avoid some work if I could re-use a work request, and just
change the fields I need instead of fully re-initializing it every time.
--
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office