Re: [PATCH v2 05/13] svcrdma: Introduce local rdma_rw API helpers

[Chuck Lever <chuck.lever@oracle.com>]

> On Mar 30, 2017, at 7:30 AM, Christoph Hellwig <hch@infradead.org> wrote:
> 
>> +	spinlock_t	     sc_rw_ctxt_lock;
>> +	struct list_head     sc_rw_ctxts;
> 
> It's a little sad that we always need a list and a spinlock when
> most requests should need a single context only.

The current code needs resources protected by several
spinlocks, some of which disable bottom-halfs. This rewrite
takes it down to just this one plain vanilla spinlock which
picks up all the svcrdma layer resources needed for the I/O
at once.

There are some common cases which can require more than one
of these.

My point is, I think this is better than trips to a memory
allocator, because those frequently require at least one
BH-disabled or irqsave spinlock, which helps prevent
latency outliers and, rarely, allocation failures.

That said, I will happily consider any solution that does
not require critical sections!


>> + * Each WR chain handles a single contiguous server-side buffer,
>> + * because some registration modes (eg. FRWR) do not support a
>> + * discontiguous scatterlist.
> 
> Both FRWR and FMR have no problem with a discontiguous page list,
> they only have a problem with any segment but the first not starting
> page aligned.  For NFS you'll need vectored direct I/O to hit that
> case.

I'll rewrite the comment.

For the Write chunk path, each RDMA segment in the chunk
can have a different R_key. So each non-empty segment gets
its own rdma_rw chain. If the client is good, it will use
a single large segment, but not all of them do.

The Reply chunk case occurs commonly, and can require
three or more separate scatterlists, due to the alignment
constraint. Each RPC Reply resides in an xdr_buf, each of
which has up to three portions:

1. A head, which is not necessarily page-aligned,
2. A page list, which does not have to be page-aligned, and
3. A tail, which is frequently but not always in the same page
as the head (and is thus not expected to be page-aligned).

The client can provide multiple segments, each with its
own R_key. The server has to fit the RDMA Writes into both
the alignment constraints of the xdr_buf components, and
the segments provided by the client.

This is why I organized the "write the reply chunk" path
this way.

Thanks to both you and Sagi for excellent review comments.


--
Chuck Lever