Re: [PATCH v1 09/12] xprtrdma: Prepare rpcrdma_ep_post() for RDMA_NOMSG calls

Chuck Lever
2015-07-10T14:53:40Z

On Jul 10, 2015, at 10:11 AM, Devesh Sharma <devesh.sharma-1wcpHE2jlwOV7V7JEmyr7g@public.gmane.orgm> wrote:

> On Fri, Jul 10, 2015 at 6:28 PM, Tom Talpey <tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> wrote:
>> On 7/10/2015 7:29 AM, Devesh Sharma wrote:
>>>
>>> we need to honor the max limits of device by checking
>>> dev_attr.max_sge? a vendor may not support 4 sges.
>>
>>
>> iWARP requires a minimum of 4 send SGEs (draft-hilland-verbs 8.1.3.2)
>>
>> An RI MUST support at least four Scatter/Gather Elements per
>> Scatter/Gather List when the Scatter/Gather List refers to the Data
>> Source of a Send Operation Type or the Data Sink of a Receive
>> Operation. An RI is NOT REQUIRED to support more than one
>> Scatter/Gather Element per Scatter/Gather List when the
>> Scatter/Gather List refers to the Data Source of an RDMA Write.
>>
>> I'm not certain if IB and RoCE state a similar minimum requirement,
>> but it seems a very bad idea to have fewer.
>
> To my knowledge IBTA Spec do not pose any such minimum requirement.
> RoCE also do not puts any minimum requirement. I think its fine if
> xprtrdma honors the device limits, thus covering iWARP devices because
> all iWARP devices would support minimum 4.
>
> Chuck would correct me if xprtrdma do have any minimum requirements

At least 2 SGEs are required for normal operation. The code in
rpcrdma_marshal_req() sets up an iovec for the RPC/RDMA header
and one for the RPC message itself. These are used in
rpcrdma_ep_post() to SEND each request.

Four SGEs are needed only for RDMA_MSGP type requests, but so far
I have never seen one of these used. I wonder if that logic can
be removed.

It is certainly possible to examine the device’s max_sge field
in rpcrdma_ep_create() and fail transport creation if the
device’s max_sge is less than RPC_MAX_IOVS.

--
Chuck Lever