[PATCH] mlx4_core: request MSIX vectors as much as there CPU cores

[PATCH] mlx4_core: request MSIX vectors as much as there CPU cores

[Eli Cohen]

The current code requires num_possible_cpus() + 1 MSIX vectors. However,
num_possible_cpus() stands for the max number of supported CPUs by the kernel.
We should use num_online_cpus() which is the number of available CPUs for the
system.

Signed-off-by: Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
 drivers/net/mlx4/main.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c
index e3e0d54..0559df4 100644
--- a/drivers/net/mlx4/main.c
+++ b/drivers/net/mlx4/main.c
@@ -969,7 +969,7 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev)
 
 	if (msi_x) {
 		nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
-			     num_possible_cpus() + 1);
+			     num_online_cpus() + 1);
 		entries = kcalloc(nreq, sizeof *entries, GFP_KERNEL);
 		if (!entries)
 			goto no_msi;
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] mlx4_core: request MSIX vectors as much as there CPU cores

[Roland Dreier]

 > The current code requires num_possible_cpus() + 1 MSIX vectors. However,
 > num_possible_cpus() stands for the max number of supported CPUs by the kernel.

I believe this is wrong -- num_possible_cpus() is the maximum number of
CPUs for the running system, including hotplug.

 > We should use num_online_cpus() which is the number of available CPUs for the
 > system.

But then we don't have vectors for hotplugged CPUs.

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] mlx4_core: request MSIX vectors as much as there CPU cores

[Jason Gunthorpe]

On Wed, May 05, 2010 at 07:54:54AM -0700, Roland Dreier wrote:
>  > The current code requires num_possible_cpus() + 1 MSIX vectors. However,
>  > num_possible_cpus() stands for the max number of supported CPUs by the kernel.
> 
> I believe this is wrong -- num_possible_cpus() is the maximum number of
> CPUs for the running system, including hotplug.

FWIW, I've seen some systems running RH kernels that blow up here
because they run out of interrupt numbers if you have two IB cards in
them.

Mainline kernels have a reworked IRQ number allocator and don't have a
problem, so maybe this is an ofed only patch?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ewg] [PATCH] mlx4_core: request MSIX vectors as much as there CPU cores

[Tziporet Koren]

On 5/5/2010 7:32 PM, Jason Gunthorpe wrote:
> On Wed, May 05, 2010 at 07:54:54AM -0700, Roland Dreier wrote:
>    
>>   >  The current code requires num_possible_cpus() + 1 MSIX vectors. However,
>>   >  num_possible_cpus() stands for the max number of supported CPUs by the kernel.
>>
>> I believe this is wrong -- num_possible_cpus() is the maximum number of
>> CPUs for the running system, including hotplug.
>>      
> FWIW, I've seen some systems running RH kernels that blow up here
> because they run out of interrupt numbers if you have two IB cards in
> them.
>
> Mainline kernels have a reworked IRQ number allocator and don't have a
> problem, so maybe this is an ofed only patch?
>    
This is not related to OFED
We found it in performance work of our EN (10G) driver

Tziporet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ewg] [PATCH] mlx4_core: request MSIX vectors as much as there CPU cores

[Roland Dreier]

 > We found it in performance work of our EN (10G) driver

What does this have to do with performance?

Do you have a system where num_possible_cpus() differs significantly
from num_online_cpus()?  What kernel is that with?  As far as I can
tell, for x86 they should only be different if there genuinely are CPUs
that are available for hotplug.

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] mlx4_core: request MSIX vectors as much as there CPU cores

[Roland Dreier]

 > We found it in performance work of our EN (10G) driver

By the way, it would certainly make sense for the ethernet driver to use
a number of queues that matches num_online_cpus() at the time the
interface is brought up.  Since we can't change the # of MSI-X vectors
very easily I think we need to allow for the possible CPUs, but bouncing
a net interface seems lighter weight to me.

Although perhaps reloading a driver on CPU hotplug is OK too?

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

Re: [ewg] [PATCH] mlx4_core: request MSIX vectors as much as there CPU cores

[Eli Cohen]

On Wed, May 05, 2010 at 12:55:54PM -0700, Roland Dreier wrote:
>  > We found it in performance work of our EN (10G) driver
> 
> By the way, it would certainly make sense for the ethernet driver to use
> a number of queues that matches num_online_cpus() at the time the
> interface is brought up.  Since we can't change the # of MSI-X vectors
> very easily I think we need to allow for the possible CPUs, but bouncing
> a net interface seems lighter weight to me.
> 
> Although perhaps reloading a driver on CPU hotplug is OK too?
> 

Yes, we have a system where num_possible_cpus is 32 and
num_online_cpus is 16. It's a RH5.4 and the kernel has no problem
allocating 33 MSI-X vectors. The point is that using more than one EQ
per CPU core does not buy us anything; in fact it can contiribute to a
higher rate of interrupts since the same EQ serves less CQs and the
chances for coalescing EQEs are lower.

So what do you think about the following patch to mlx4_en:


diff --git a/drivers/net/mlx4/en_cq.c b/drivers/net/mlx4/en_cq.c
index 21786ad..07c0779 100644
--- a/drivers/net/mlx4/en_cq.c
+++ b/drivers/net/mlx4/en_cq.c
@@ -49,11 +49,12 @@ int mlx4_en_create_cq(struct mlx4_en_priv *priv,
 {
 	struct mlx4_en_dev *mdev = priv->mdev;
 	int err;
+	int num_active_vectors = min_t(int, num_online_cpus(), mdev->dev->caps.num_comp_vectors);
 
 	cq->size = entries;
 	if (mode == RX) {
 		cq->buf_size = cq->size * sizeof(struct mlx4_cqe);
-		cq->vector   = ring % mdev->dev->caps.num_comp_vectors;
+		cq->vector   = ring % num_active_vectors;
 	} else {
 		cq->buf_size = sizeof(struct mlx4_cqe);
 		cq->vector   = 0;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html