Re: linux rdma 3.14 merge plans

Sagi Grimberg
2014-01-29T15:06:46Z

On 1/29/2014 4:13 PM, Bart Van Assche wrote:
> On 01/28/14 22:02, Or Gerlitz wrote:
>>>>> Roland, ping! the signature patches were posted > three months ago.
> I have a question about RDMA and T10-DIF support. The Linux block layer,
> the Linux SCSI core, Linux filesystems and block drivers all support
> discontiguous buffers. These are buffers that cannot be mapped onto a
> single contiguous range of virtual memory addresses. I think the RDMA
> FMR and FR memory registration mechanisms only support mapping a range
> of addresses that can be mapped onto a contiguous region of virtual
> addresses. This means that either multiple RDMA memory regions are
> needed to map a discontiguous buffer or that the buffer contents has to
> be copied before associating it with a memory region. What I understood
> from the mlx5 T10-DIF patch series is that the API introduced in that
> patch series is such that only a single memory region per data transfer
> operation is supported. This made me wonder how to support T10-DIF for
> discontiguous buffers.

Hey Bart, Thanks for the observation,

You are correct, the T10-PI RDMA offload API requires a single memory
region that describes the data
and a single memory region that describes the protection information.

Non-contiguous buffers are a well known problem/challenge for RDMA
transports, each handles them
differently: iSER uses bounce buffers, SRP registers multiple contiguous
regions. The T10-PI offload API does
not impose any other constraint on this situation, each ULP will act the
same: iSER will copy to contiguous
bounce buffers and SRP will register multiple "Signature" memory regions
(each for a contiguous chunk of data)
and send the rkeys to the target.

> Would one of the options below perhaps be an
> appropriate solution ?
> - Add a flag in struct request_queue that tells the block layer to use a
> bounce buffer (data copying) for I/O requests with a discontiguous
> data buffer. The block layer already supports copying data buffers for
> which e.g. DMA alignment requirements are not met. Rework patch
> "IB/iser: Introduce fast memory registration model" (commit
> 5587856c9659ac2d6ab201141aa8a5c2ff3be4cd) such that it takes advantage
> of that new flag. See also blk_rq_map_user_iov() for an example of
> how the block layer decides when copying a data buffer is necessary.

Didn't understand why should it matter where the copy is done (iser/block)?

> - Modify the patches that add T10-DIF such that discontiguous buffers
> are supported.

I thought about adding some kind of logic to T10-PI RDMA API to try and
solve the alignment
issue at the same time, but the fact is that the two are unrelated, and
I figured it is not
such a good idea. Non-contiguous memory registration is separate problem
(hint: under
development) and should be addressed separately, T10-PI RDMA offload
should play well
with it.

> Sorry that I had not realized this before.
>
> Bart.
>

Hops this helps,

Sagi.