Re: [PATCH v2 3/4] rsockets: distribute completion queue vectors among multiple cores

[Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>]

On 09/08/14 16:27, Sreedhar Kodali wrote:
> Thanks for your insights.  Your suggestion of having a separate
> vector per process is really useful when multiple processes are
> simultaneously running in the same environment.  At the implementation
> level this requires an extended configuration file format. Can't
> we possibly think of this as a next iteration?

This patch introduces a new configuration file 
(/etc/rdma/rsocket/comp_vector). User expect that such configuration 
files remain supported after a software upgrade. In other words, 
introduction of a new configuration file deserves careful consideration. 
Do we really need this new configuration file ? Why not e.g. to change 
the default behavior of the rsockets library into spreading the 
interrupt processing workload over all completion vectors supported by 
the HCA ? Furthermore, the number of completion vectors that is 
supported by an HCA not only depends on the HCA firmware but also on the 
number of completion vectors a HCA could allocate during driver 
initialization. The number of MSI-X vectors available on an x86-64 
system is limited (224). Dual-socket x86-64 motherboards with up to 11 
PCIe slots are available commercially. If all PCIe-slots are populated 
it's almost sure that one of the PCIe cards won't be assigned as many 
MSI-X vectors as it could use.

Sorry but I'm not convinced that using the same completion vector mask 
for all HCA's and all processes is a good idea. Or is there perhaps 
something I have overlooked ?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html