From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bart Van Assche Subject: Re: [PATCH v2 3/4] rsockets: distribute completion queue vectors among multiple cores Date: Mon, 08 Sep 2014 20:22:04 +0200 Message-ID: <540DF3CC.1060304@acm.org> References: <5409DB0D.6080103@acm.org> <850a2835b5917f6da62af3d1ea0288fd@imap.linux.ibm.com> <540D50E9.5050709@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Sreedhar Kodali Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org, pradeeps-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org, linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: linux-rdma@vger.kernel.org On 09/08/14 16:27, Sreedhar Kodali wrote: > Thanks for your insights. Your suggestion of having a separate > vector per process is really useful when multiple processes are > simultaneously running in the same environment. At the implementation > level this requires an extended configuration file format. Can't > we possibly think of this as a next iteration? This patch introduces a new configuration file (/etc/rdma/rsocket/comp_vector). User expect that such configuration files remain supported after a software upgrade. In other words, introduction of a new configuration file deserves careful consideration. Do we really need this new configuration file ? Why not e.g. to change the default behavior of the rsockets library into spreading the interrupt processing workload over all completion vectors supported by the HCA ? Furthermore, the number of completion vectors that is supported by an HCA not only depends on the HCA firmware but also on the number of completion vectors a HCA could allocate during driver initialization. The number of MSI-X vectors available on an x86-64 system is limited (224). Dual-socket x86-64 motherboards with up to 11 PCIe slots are available commercially. If all PCIe-slots are populated it's almost sure that one of the PCIe cards won't be assigned as many MSI-X vectors as it could use. Sorry but I'm not convinced that using the same completion vector mask for all HCA's and all processes is a good idea. Or is there perhaps something I have overlooked ? Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html