Re: [PATCH 10/10] svcrdma: Documentation update for the FastReg memory model

["Talpey, Thomas" <Thomas.Talpey@netapp.com>]

At 03:18 PM 10/1/2008, Talpey, Thomas wrote:
>At 02:26 PM 10/1/2008, J. Bruce Fields wrote:
>>On Tue, Sep 30, 2008 at 03:04:26PM -0400, Talpey, Thomas wrote:
>>> At 02:44 PM 9/30/2008, J. Bruce Fields wrote:
>>> >On Mon, Sep 29, 2008 at 10:07:25PM -0500, Tom Tucker wrote:
>>> >> What's your perspective on the lifetime of bounce buffers, memory  
>>> >> windows, and the other strategies in client?
>>> >
>>> >I'm ignorant.  Pointer to something else I should read?
>>> >
>>> >I assume there are similar issues on the client?
>>> 
>>> Tom's asking about the client memory registration options and
>>> whether they should all remain in the client code going forward.
>>> There are different strategies in there, depending on what the
>>> hardware supports, how well it supports it, and how you want
>>> to run it all.
>>
>>Do any of them allow reading or writing memory other than that holding
>>rpc data?

BTW, there's a hopefully helpful slide on page 20 of the following presentation
which I gave back in April to the RDMA developers:

<http://openfabrics.org/archives/spring2008sonoma/Tuesday/NFS-RDMA-OFA-08.pdf>

FRMR's are, in part, a result of that request. Certainly they fulfill it!

Tom.

>
>Some of them, and the memory exposure varies. The only modes which
>expose additional memory are FMR's, which expose data on the same
>page as RPC data, and ALLPHYSICAL, which, well, exposes everything.
>
>The BOUNCEBUFFERS and REGISTER modes protect memory fiercely,
>but are relatively expensive. MEMWINDOWS are safe but not very fast,
>and only supported by obsolete devices. FRMR's are both safe and fast
> but a) still have a cost and b) are only supported by one device at present.
>
>Long term, we all hope FRMRs will be the default, logical, and only choice.
>
>I still owe feedback on Tom's new text... it's coming soon.
>
>Tom.
> 
>>
>>--b.
>>
>>> 
>>> The multiple strategies stem from the early days when no two
>>> adapters did quite the same thing. That said, at least one of
>>> them is no longer useful - no adapters support windows today,
>>> since the demise of Ammasso, although the kernel API to drive
>>> them is still there in the OFA stack below.
>>> 
>>> I think it's possible that some of the other modes can collapse, but
>>> not just now. There's a lot of older hardware out there, and newer
>>> hardware may appear to use the old stuff too. 
>>> 
>>> Another concern I have, frankly, is interoperability. If we collapse
>>> the modes, then the temptation to assume both ends have such-
>>> and-such a capability increase. If one side tries some RDMA operation
>>> that is only supported properly by a certain adapter, it's hard to detect
>>> that in a monoculture. Having the option to switch modes can help avert
>>> this, by testing for it.
>>> 
>>> In any case, if the current code doesn't work, it's a bug. Certainly
>>> bouncebuffers (non-RDMA mode) should work perfectly and I plan to
>>> check it asap.
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html