Re: linux rdma 3.14 merge plans

[Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>]

On 01/30/14 09:19, Or Gerlitz wrote:
> On 29/01/2014 19:56, Bart Van Assche wrote:
>> On 01/29/14 16:06, Sagi Grimberg wrote:
>>> Didn't understand why should it matter where the copy is done
>>> (iser/block)?
>> In the Linux kernel community it is considered important to avoid code
>> duplication. Hence the proposal to keep code that copies data buffers
>> in the block layer core and to avoid that such functionality has to be
>> reimplemented in every block driver or SCSI LLD.
> 
> Thanks for narrowing this down, I see your point, however the solution I
> propose if to remove this copy altogether... for those rare cases where
> fast-registration can't be done -- in SRP I think the code goes to
> indirect mode under which there no copy (right?). As for iSER, we will
> add to our TODO enhancing the code to avoid using bounce-buffer, few
> designs might be possible, one way would be to create set of
> FMRS/Fast-reg element which are capable to map chunks which are <
> PAGE_SIZE, e.g in the order of single block, and hence can map what ever
> arbitrary set of blocks provided by the upper layer.

I'm not sure how important this issue is on x86 systems since the page
size on these systems is 4 KB and since many applications use an I/O
size >= 4 KB. So if the I/O buffers are aligned on page boundaries the
buffer memory can be registered as a single memory region without having
to copy any data. However, on PPC systems, which have a page size of 64
KB, if e.g. a database issues a readv() or writev() call or if an I/O
scheduler coalesces several small I/O requests into a single I/O request
the data buffer of these I/O requests is discontiguous. I think it is
important that data copying can be avoided for such I/O requests.

By the way, is it documented somewhere what the alignment requirements
are for FMR and FR requests ? As far as I can see the mlx4 driver
requires page alignment for FMR requests but not for FR requests (see
e.g. mlx4_check_fmr()). I haven't found a page aligment requirement in
e.g. ehca_fmr_check_page_list(). Does this mean that alignment
restrictions for FMR are HCA-dependent ? Would it be a good idea to add
an attribute in e.g. ib_device_attr that allows ULP drivers to retrieve
FMR alignment requirements in a generic way ?

Thanks,

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html