From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sagi Grimberg Subject: Re: [PATCH for-3.11 7/7] IB/iser: Introduce fast memory registration model (FRWR) Date: Tue, 23 Jul 2013 17:33:36 +0300 Message-ID: <51EE9440.7060707@mellanox.com> References: <1374153931-7313-1-git-send-email-ogerlitz@mellanox.com> <1374153931-7313-8-git-send-email-ogerlitz@mellanox.com> <51ED1B8E.7070703@acm.org> <51ED2F97.2080400@mellanox.com> <51EE6FFE.8040802@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <51EE6FFE.8040802-HInyCGIudOg@public.gmane.org> Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Bart Van Assche Cc: Or Gerlitz , roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, oren-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org List-Id: linux-rdma@vger.kernel.org On 7/23/2013 2:58 PM, Bart Van Assche wrote: > On 07/22/13 15:11, Sagi Grimberg wrote: >> So just to clarify the flow: >> . at connection establishment allocate pool of fastreg descriptors >> . upon each IOP take a fastreg descriptor from the pool >> . if it is not invalidated - invalidate it. >> . register using FRWR. >> . when cleanup_task is called - just return the fastreg descriptor to >> the pool. >> . at connection teardown free all resources. >> Still to come: >> . upon each IOP response, check if the target used remote invalidate - >> if so mark relevant fastreg as valid. > > Hello Sagi and Or, > > Thanks for the clarifications. I have one more question though. My > interpretation of section "10.6 Memory Management" in the IB > specification is that memory registration maps a memory region that > either has contiguous virtual addresses or contiguous physical > addresses. However, there is no such requirement for an sg-list. As an > example, for direct I/O to a block device with a sector size of 512 > bytes it is only required that I/O occurs in multiples of 512 bytes > and from memory aligned on 512-byte boundaries. So the use of direct > I/O can result in an sg-list where the second and subsequent sg-list > elements have a non-zero offset. Do you agree with this ? Are such > sg-lists mapped correctly by the FRWR code ? > > Bart. > Hey Bart, You are on the money with this observation, like FMRs, FRWR cannot register any arbitrary SG-list. You have the same limitations. Unlike SRP where the initiator will use multiple FMRs to register such "unaligned" SG-lists, iSER uses a bounce buffer to copy the data to a nice physically contiguous memory area (see patch 5/7 fall_to_bounce_buf routine), thus will pass a single R_Key for each transaction. An equivalent FRWR implementation for SRP will also use multiple FRWRs in-order to register such "un-aligned" SG-lists and publish the R_Keys in ib_sge. Hope this helps, -Sagi -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html