RE: [PATCH 7/8] xprtrdma: Split the completion queue

["Steve Wise" <swise@opengridcomputing.com>]

> -----Original Message-----
> From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma-owner@vger.kernel.org] On
> Behalf Of Chuck Lever
> Sent: Thursday, April 17, 2014 8:55 AM
> To: Sagi Grimberg
> Cc: Steve Wise; Linux NFS Mailing List; linux-rdma@vger.kernel.org
> Subject: Re: [PATCH 7/8] xprtrdma: Split the completion queue
> 
> 
> On Apr 17, 2014, at 3:06 AM, Sagi Grimberg <sagig@dev.mellanox.co.il> wrote:
> 
> > On 4/16/2014 9:21 PM, Chuck Lever wrote:
> >> Passing a small array to ip_poll_cq() is actually easy to do, and is
> >> exactly equivalent to a poll budget. The struct ib_wc should be taken
> >> off the stack anyway, IMO.
> >>
> >> The only other example I see in 3.15 right now is IPoIB, which seems
> >> to do exactly this.
> >>
> >> I'm testing a patch now. I'd like to start simple and make it more
> >> complex only if we need to.
> >
> > What array size are you using? Note that if you use a small array it may be an
overkill since
> > a lot more interrupts are invoked (-> more latency). I found that for a high workload
a
> budget
> > of 256/512/1024 keeps fairness and doesn't increase latency.
> 
> My array size is currently 4. It's a macro that can be changed easily.
> 
> By a very large majority, my workloads see only one WC per completion
> upcall. However, I'm using an older card with simple synthetic benchmarks.
> 
> I don't want to make the array large because struct ib_wc is at least
> 64 bytes on my systems - each WC array would be enormous and hardly ever
> used. But we can dial it in over time.

You could use a small array combined with a loop and a budget count.  So the code would
grab, say, 4 at a time, and keep looping polling up to 4 until the CQ is empty or the
desired budget is reached...

Stevo