From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sagi Grimberg Subject: Re: [PATCH v2 12/12] IB/srp: Add multichannel support Date: Tue, 28 Oct 2014 20:32:26 +0200 Message-ID: <544FE13A.60807@dev.mellanox.co.il> References: <5433E43D.3010107@acm.org> <5433E585.607@acm.org> <5443F69F.40606@dev.mellanox.co.il> <54450690.709@acm.org> <544622FE.5040906@dev.mellanox.co.il> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <544622FE.5040906-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Bart Van Assche , Christoph Hellwig Cc: Jens Axboe , Sagi Grimberg , Sebastian Parschauer , Robert Elliott , Ming Lei , "linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , linux-rdma List-Id: linux-rdma@vger.kernel.org On 10/21/2014 12:10 PM, Sagi Grimberg wrote: > On 10/20/2014 3:56 PM, Bart Van Assche wrote: >> On 10/19/14 19:36, Sagi Grimberg wrote: >>> On 10/7/2014 4:07 PM, Bart Van Assche wrote: >>>> * comp_vector, a number in the range 0..n-1 specifying the >>>> - MSI-X completion vector. Some HCA's allocate multiple (n) >>>> - MSI-X vectors per HCA port. If the IRQ affinity masks of >>>> - these interrupts have been configured such that each MSI-X >>>> - interrupt is handled by a different CPU then the comp_vector >>>> - parameter can be used to spread the SRP completion workload >>>> - over multiple CPU's. >>>> + MSI-X completion vector of the first RDMA channel. Some >>>> + HCA's allocate multiple (n) MSI-X vectors per HCA port. If >>>> + the IRQ affinity masks of these interrupts have been >>>> + configured such that each MSI-X interrupt is handled by a >>>> + different CPU then the comp_vector parameter can be used to >>>> + spread the SRP completion workload over multiple CPU's. >>> >>> This is fairly not trivial for the user... >>> >>> Aren't we requesting a bit too much awareness here? >>> Can't we just "make it work"? The user hands out ch_count - why can't >>> you do some least-used logic here? >>> >>> Maybe we can even go with per-cpu QPs and discard comp_vector argument? >>> this would probably bring the best performance, wouldn't it? >>> (fallback to least-used logic in case HW support less vectors) >> >> Hello Sagi, >> >> The only reason the comp_vector parameter is still supported is because >> of backwards compatibility. What I expect is that users will set the >> ch_count parameter but not the comp_vector parameter. > Hey Bart, Another wander I have with this. Say you have 8 cores on a single numa node. First connection will attach to vectors 0-3 (ch_count=4) and so are all the connections. Don't we want to spread that a little? If we are not going per-cpu, why aren't we trying to spread vectors around to try and reduce the interference? Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html