Re: [PATCH v2 12/12] IB/srp: Add multichannel support

[Sagi Grimberg <sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>]

On 10/30/2014 4:36 PM, Bart Van Assche wrote:
> On 10/30/14 15:19, Sagi Grimberg wrote:
>> My impression here that in the default settings, on a 1 NUMA node with
>> 8 cores, 2 different srp connections (using 4 channels each) will be
>> associated with comp vectors 0-3. while it could potentially use
>> vectors 4-7 and reduce possible mutual interference. right?
>
> Hello Sagi,
>
> That's correct. For this example if use of all completion vectors is
> desired additional configuration is required, e.g. by setting ch_count
> to 8 in /etc/modprobe.d/ib_srp.conf.

That is why I think that the user is still expected to be aware of
the configuration in order to get max performance. I would like to
see best performance to "just work". For example, I don't see any sort
of sw queue count to configure, it "just works".

Now I also agree with this may mean more (or sometimes way more)
resources, but I suggest that if we go with default of 4 per numa node
we should take care of such situations.

I'm not strict about this wrt to this patch set. But I think we should
consider this bit.

> By the way, I'm not sure it is
> possible to avoid manual configuration and tuning entirely. As an
> example, with a six core CPU at the initiator side and with
> hyperthreading enabled (12 CPU threads in total) I see higher IOPS
> results with ch_count=6 compared to ch_count=8 or ch_count=12.
> I have
> not tried to determine why but maybe this is because ch_count values
> that are below the number of CPU threads cause some interrupt coalescing.

I'm not aware of any implicit interrupt coalescing effect...
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html