Re: [PATCH 12/14] IB/srp: Make HCA completion vector configurable

[Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>]

On 06/27/13 23:24, David Dillow wrote:
> On Wed, 2013-06-12 at 15:35 +0200, Bart Van Assche wrote:
>> Several InfiniBand HCA's allow to configure the completion vector
>> per queue pair. This allows to spread the workload created by IB
>> completion interrupts over multiple MSI-X vectors and hence over
>> multiple CPU cores. In other words, configuring the completion
>> vector properly not only allows to reduce latency on an initiator
>> connected to multiple SRP targets but also allows to improve
>> throughput.
>
> Shouldn't there be some documentation about how to set this? What
> happens when you try to set it on an HCA that doesn't support this how
> does the user know what's wrong?

Hello Dave,

HCA drivers that support only one completion vector ignore the 
comp_vector parameter. See e.g. c2_create_cq() in 
drivers/infiniband/hw/amso1100/c2_provider.c for an example. The third 
argument of that function, called "vector", is ignored.

What will happen if an out-of-range value is chosen for this parameter 
depends on the HCA driver. This is how the mlx4 driver handles the 
comp_vector parameter:
	vector = dev->eq_table[vector % ibdev->num_comp_vectors];
So an out-of-range value for comp_vector won't cause a failure. And if 
ever a HCA driver would be added to the kernel tree that causes 
create_cq() to fail if the comp_vector parameter is out of range then 
the ib_srp driver will propagate the error code returned by 
ib_create_cq() to the process that wrote into the add_target sysfs 
attribute.

The procedure for choosing a value for this parameter is as follows:
* Look up the interrupt numbers that have been assigned to each HCA
   completion vector, e.g. via grep mlx4-ib- /proc/interrupts.
* Decide which CPU to bind each interrupt to and update
   /proc/irq/$irq/smp_affinity.
* Decide which CPU and completion vector should be used for processing
   SRP completions and specify that completion vector number in the SRP
   login string.

To me this looks like a standard procedure that applies to any HCA that 
supports multiple MSI-X vectors ?

Bart.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html