Re: [PATCH for-3.11 7/7] IB/iser: Introduce fast memory registration model (FRWR)

[Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]

On 7/23/2013 2:58 PM, Bart Van Assche wrote:
> On 07/22/13 15:11, Sagi Grimberg wrote:
>> So just to clarify the flow:
>> . at connection establishment allocate pool of fastreg descriptors
>> . upon each IOP take a fastreg descriptor from the pool
>>      . if it is not invalidated - invalidate it.
>>      . register using FRWR.
>> . when cleanup_task is called - just return the fastreg descriptor to
>> the pool.
>> . at connection teardown free all resources.
>> Still to come:
>> . upon each IOP response, check if the target used remote invalidate -
>> if so mark relevant fastreg as valid.
>
> Hello Sagi and Or,
>
> Thanks for the clarifications. I have one more question though. My 
> interpretation of section "10.6 Memory Management" in the IB 
> specification is that memory registration maps a memory region that 
> either has contiguous virtual addresses or contiguous physical 
> addresses. However, there is no such requirement for an sg-list. As an 
> example, for direct I/O to a block device with a sector size of 512 
> bytes it is only required that I/O occurs in multiples of 512 bytes 
> and from memory aligned on 512-byte boundaries. So the use of direct 
> I/O can result in an sg-list where the second and subsequent sg-list 
> elements have a non-zero offset. Do you agree with this ? Are such 
> sg-lists mapped correctly by the FRWR code ?
>
> Bart.
>

Hey Bart,

You are on the money with this observation, like FMRs, FRWR cannot 
register any arbitrary SG-list. You have the same limitations.
Unlike SRP where the initiator will use multiple FMRs to register such 
"unaligned" SG-lists,
iSER uses a bounce buffer to copy the data to a nice physically 
contiguous memory area (see patch 5/7 fall_to_bounce_buf routine), thus 
will pass a single R_Key for each transaction.
An equivalent FRWR implementation for SRP will also use multiple FRWRs 
in-order to register such "un-aligned" SG-lists and publish the R_Keys 
in ib_sge.

Hope this helps,

-Sagi
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html