Re: [PATCH v3 11/13] IB/srp: Make HCA completion vector configurable

[Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>]

On 14/07/2013 3:43, Sagi Grimberg wrote:
> On 7/3/2013 3:58 PM, Bart Van Assche wrote:
>> Several InfiniBand HCA's allow to configure the completion vector
>> per queue pair. This allows to spread the workload created by IB
>> completion interrupts over multiple MSI-X vectors and hence over
>> multiple CPU cores. In other words, configuring the completion
>> vector properly not only allows to reduce latency on an initiator
>> connected to multiple SRP targets but also allows to improve
>> throughput.
>
> Hey Bart,
> Just wrote a small patch to allow srp_daemon spread connection across
> HCA's completion vectors.
> But re-thinking on this, is it really a good idea to give the user
> control over completion
> vectors for CQs he doesn't really owns. This way the user must retrieve
> the maximum completion
> vectors from the ib_device and consider this when adding a connection
> and In addition will need to set proper IRQ affinity.
>
> Perhaps the driver can manage this on it's own without involving the
> user, take the mlx4_en driver for
> example, it spreads it's CQs across HCAs completion vectors without
> involving the user. the user that
> opens a socket has no influence of the underlying cq<->comp-vector
> assignment.
>
> The only use-case I can think of is where the user will want to use only
> a subset of the completion-vectors
> if the user will want to reserve some completion-vectors for native IB
> applications but I don't know
> how common it is.
>
> Other from that, I think it is always better to spread the CQs across
> HCA completion-vectors, so perhaps the driver
> just assign connection CQs across comp-vecs without getting args from
> the user, but simply iterate over comp_vectors.
>
> What do you think?

Hello Sagi,

Sorry but I do not think it is a good idea to let srp_daemon assign the 
completion vector. While this might work well on single-socket systems 
this will result in suboptimal results on NUMA systems. For certain 
workloads on NUMA systems, and when a NUMA initiator system is connected 
to multiple target systems, the optimal configuration is to make sure 
that all processing that is associated with a single SCSI host occurs on 
the same NUMA node. This means configuring the completion vector value 
such that IB interrupts are generated on the same NUMA node where the 
associated SCSI host and applications are running.

More in general, performance tuning on NUMA systems requires system-wide 
knowledge of all applications that are running and also of which 
interrupt is processed by which NUMA node. So choosing a proper value 
for the completion vector is only possible once the system topology and 
the IRQ affinity masks are known. I don't think we should build 
knowledge of all this in srp_daemon.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html