Re: [PATCH 7/8] xprtrdma: Split the completion queue

[Sagi Grimberg <sagig@dev.mellanox.co.il>]

On 4/17/2014 4:55 PM, Chuck Lever wrote:
> On Apr 17, 2014, at 3:06 AM, Sagi Grimberg <sagig@dev.mellanox.co.il> wrote:
>
>> On 4/16/2014 9:21 PM, Chuck Lever wrote:
>>> Passing a small array to ip_poll_cq() is actually easy to do, and is
>>> exactly equivalent to a poll budget. The struct ib_wc should be taken
>>> off the stack anyway, IMO.
>>>
>>> The only other example I see in 3.15 right now is IPoIB, which seems
>>> to do exactly this.
>>>
>>> I’m testing a patch now. I’d like to start simple and make it more
>>> complex only if we need to.
>> What array size are you using? Note that if you use a small array it may be an overkill since
>> a lot more interrupts are invoked (-> more latency). I found that for a high workload a budget
>> of 256/512/1024 keeps fairness and doesn't increase latency.
> My array size is currently 4. It’s a macro that can be changed easily.
>
> By a very large majority, my workloads see only one WC per completion
> upcall. However, I’m using an older card with simple synthetic benchmarks.

It doesn't matter until it does matter...
We noticed this phenomenon under *extreme* stress.

> I don’t want to make the array large because struct ib_wc is at least
> 64 bytes on my systems — each WC array would be enormous and hardly ever
> used.

Right, large WC array would be annoying...

Sagi.