Re: [PATCH v2 12/12] IB/srp: Add multichannel support

[Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>]

On 10/28/14 19:32, Sagi Grimberg wrote:
> On 10/21/2014 12:10 PM, Sagi Grimberg wrote:
>> On 10/20/2014 3:56 PM, Bart Van Assche wrote:
>>> On 10/19/14 19:36, Sagi Grimberg wrote:
>>>> On 10/7/2014 4:07 PM, Bart Van Assche wrote:
>>>>>           * comp_vector, a number in the range 0..n-1 specifying the
>>>>> -          MSI-X completion vector. Some HCA's allocate multiple (n)
>>>>> -          MSI-X vectors per HCA port. If the IRQ affinity masks of
>>>>> -          these interrupts have been configured such that each MSI-X
>>>>> -          interrupt is handled by a different CPU then the
>>>>> comp_vector
>>>>> -          parameter can be used to spread the SRP completion workload
>>>>> -          over multiple CPU's.
>>>>> +          MSI-X completion vector of the first RDMA channel. Some
>>>>> +          HCA's allocate multiple (n) MSI-X vectors per HCA port. If
>>>>> +          the IRQ affinity masks of these interrupts have been
>>>>> +          configured such that each MSI-X interrupt is handled by a
>>>>> +          different CPU then the comp_vector parameter can be used to
>>>>> +          spread the SRP completion workload over multiple CPU's.
>>>>
>>>> This is fairly not trivial for the user...
>>>>
>>>> Aren't we requesting a bit too much awareness here?
>>>> Can't we just "make it work"? The user hands out ch_count - why can't
>>>> you do some least-used logic here?
>>>>
>>>> Maybe we can even go with per-cpu QPs and discard comp_vector argument?
>>>> this would probably bring the best performance, wouldn't it?
>>>> (fallback to least-used logic in case HW support less vectors)
>>>
>>> The only reason the comp_vector parameter is still supported is because
>>> of backwards compatibility. What I expect is that users will set the
>>> ch_count parameter but not the comp_vector parameter.
>
> Another wander I have with this. Say you have 8 cores on a single numa
> node. First connection will attach to vectors 0-3 (ch_count=4) and so
> are all the connections. Don't we want to spread that a little?
>
> If we are not going per-cpu, why aren't we trying to spread vectors
> around to try and reduce the interference?

Hello Sagi,

Sorry but your question is not entirely clear to me. Are you referring 
to spreading the workload over CPU's or over completion vectors ? If a 
user wants to spread the completion workload maximally by using all 
completion vectors that can be achieved by setting ch_count to a value 
that is equal to or larger than the number of completion vectors.

As mentioned in the commit message, spreading the completion workload 
over CPU's is not entirely under control of the SRP initiator driver. It 
is assumed that a user assigns IRQ affinity such that the interrupts 
associated with different completion vectors are processed by different 
CPU threads. If there are more RDMA channels than completion vectors, 
the SRP initiator driver associates RDMA channels with completion 
vectors such that the workload is spread evenly over the completion vectors.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html