Re: [PATCH RFC 0/5] xprtrdma Send completion batching

Tom Talpey
2017-09-07T13:17:16Z

On 9/6/2017 3:39 PM, Jason Gunthorpe wrote:
> On Wed, Sep 06, 2017 at 02:33:50PM -0400, Chuck Lever wrote:
>
>> B. Force RPC completion to wait for Send completion, which
>> would allow the post-v4.6 scatter-gather code to work
>> safely. This would need some guarantee that Sends will
>> always complete in a short period.
>
> Why is waiting for the send completion so fundamentally different from
> waiting for the remote RPC reply?
>
> I would say that 99% of time the send completion and RPC reply
> completion will occure approximately concurrently.

Absolutely not. The RPC reply requires upper layer processing at
the server, which involves work requests, context switches, file
system and disk i/o operations, and plain old queuing. RPC replies
typically can be expected in milliseconds from traditional HDDs,
and hundreds of microseconds from SSDs. Storage class memory is
reducing these by two orders of magnitude, but it's still not in
the range of a network round trip.

> eg It is quite likely the RPC reply SEND carries an embeded ack
> for the requesting SEND..

That's not what we've seen, even on InfiniBand. On adapters such as
iWARP, I will note, the send completion is often generated when
the adapter buffers the outgoing message, and has almost nothing
to do with the remote transport ack.

Tom.