Re: [PATCH RFC 0/5] xprtrdma Send completion batching

Chuck Lever
2017-09-05T21:22:04Z

> On Sep 5, 2017, at 4:06 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>
> On Tue, Sep 05, 2017 at 01:00:10PM -0400, Chuck Lever wrote:
>
>> - Send completions are batched to reduce interrupts, but still
>> provide a periodic heartbeat signal for SQ housekeeping
>
> I would scrub this commentary, it is very misleading.
>
> The idea of a periodic completion does not match how verbs works at
> all, it was an incomplete root cause analysis from a HCA that uses
> different rules for freeing space in the SQ.

I think it does bear mentioning that, given this diagnosis, it is
still safe to remove the ib_post_send counting mechanism in 5/5,
which has been in xprtrdma for as long as I can recall, and has
been effective (with a few minor adjustments) at preventing SQ
overflow.

I'm not able to test this change with every HCA the Linux kernel
currently supports, unfortunately. The best I can do is offer a
"proof of correctness" and hope that vendors will jump on this
and try it out.

> Instead, I would say this series creates strong SQ accounting and
> properly guarentees the SQ can never overflow by only releasing SQ's
> back into the pool when the HCA has confirmed they are completed via a
> CQ.

I will adjust the cover letter (and patch descriptions as necessary)
next time I post this series. Thanks for your suggestions and review.

--
Chuck Lever