[PATCH] mlx4_core: module param to limit msix vec allocation

[PATCH] mlx4_core: module param to limit msix vec allocation

[Arthur Kepner]

(Resending to netdev. Previously sent to linux-rdma)

The mlx4_core driver allocates 'nreq' msix vectors (and irqs), 
where:

  nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
               num_possible_cpus() + 1);

ConnectX HCAs support 512 event queues (4 reserved). On a system 
with enough processors, we get:

  mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors available, trying again

Further attempts (by other drivers) to allocate interrupts fail, 
because mlx4_core got 'em all.

Use a module parameter to limit the number of MSI-X vectors that 
the mlx4_core driver will attempt to allocate.

Signed-off-by: Arthur Kepner <akepner@sgi.com>

---

diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c
index 5102ab1..3eba2f3 100644
--- a/drivers/net/mlx4/main.c
+++ b/drivers/net/mlx4/main.c
@@ -68,6 +68,10 @@ static int msi_x = 1;
 module_param(msi_x, int, 0444);
 MODULE_PARM_DESC(msi_x, "attempt to use MSI-X if nonzero");
 
+static int max_msi_x_vec = 64;
+module_param(max_msi_x_vec, int, 0444);
+MODULE_PARM_DESC(max_msi_x_vec, "max MSI-X vectors we'll attempt to allocate");
+
 #else /* CONFIG_PCI_MSI */
 
 #define msi_x (0)
@@ -968,8 +972,10 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev)
 	int i;
 
 	if (msi_x) {
+		nreq = min_t(int, num_possible_cpus() + 1, max_msi_x_vec);
 		nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
-			     num_possible_cpus() + 1);
+			     nreq);
+
 		entries = kcalloc(nreq, sizeof *entries, GFP_KERNEL);
 		if (!entries)
 			goto no_msi;

Re: [PATCH] mlx4_core: module param to limit msix vec allocation

[David Miller]

From: Arthur Kepner <akepner@sgi.com>
Date: Thu, 26 Aug 2010 10:06:46 -0700

> Use a module parameter to limit the number of MSI-X vectors that 
> the mlx4_core driver will attempt to allocate.
> 
> Signed-off-by: Arthur Kepner <akepner@sgi.com>

And what the heck happens if every single device driver tries to
handle this problem in the same way?  It makes no sense to make
the user specify N special module parameters just to get a working
system using MSI-X.

This must be fixed in a centralized way that allows coordination
between devices which compete for MSI-X vector resources, rather than
in a way that is specific to each and every device type.

I am not applying this patch, sorry.

Re: [PATCH] mlx4_core: module param to limit msix vec allocation

[Arthur Kepner]

On Thu, Aug 26, 2010 at 12:24:07PM -0700, David Miller wrote:
> ....
> And what the heck happens if every single device driver tries to
> handle this problem in the same way?  It makes no sense to make
> the user specify N special module parameters just to get a working
> system using MSI-X.
> 
> This must be fixed in a centralized way that allows coordination
> between devices which compete for MSI-X vector resources, rather than
> in a way that is specific to each and every device type.
> 

OK, how about this. Each time the kernel assigns a default 
affinity for an interrupt, it generates a netlink message, 
and a user-level interrupt balancer can decide whether and 
how to reassign the interrupt? 

I have that running on a (very small) system and it seems to 
work OK. Will send out the patch in just a minute.

-- 
Arthur

Re: [PATCH] mlx4_core: module param to limit msix vec allocation

[David Miller]

From: Arthur Kepner <akepner@sgi.com>
Date: Fri, 3 Sep 2010 13:30:44 -0700

> I have that running on a (very small) system and it seems to 
> work OK. Will send out the patch in just a minute.

Failure vs. non-failure of allocation of a kernel managed physical
resource can't be determined by a userland process which may or may
not be running.

In fact, if I build all of my drivers built-in and use NFS root which
will activate and bring up network devices, userland won't even be
present when the interrupts are requested.

No, you really can't do it this way, IRQ allocation management has to
be in the kernel.

Re: [PATCH] mlx4_core: module param to limit msix vec allocation

[Arthur Kepner]

On Fri, Sep 03, 2010 at 02:46:58PM -0700, David Miller wrote:
> From: Arthur Kepner <akepner@sgi.com>
> Date: Fri, 3 Sep 2010 13:30:44 -0700
> 
> > I have that running on a (very small) system and it seems to 
> > work OK. Will send out the patch in just a minute.
> 
> Failure vs. non-failure of allocation of a kernel managed physical
> resource can't be determined by a userland process which may or may
> not be running.

But even if there's no user process running, it's no worse 
than what we've got now.

> 
> In fact, if I build all of my drivers built-in and use NFS root which
> will activate and bring up network devices, userland won't even be
> present when the interrupts are requested.

Yep. This only works once the user-level irq balancer is 
available. 

> 
> No, you really can't do it this way, IRQ allocation management has to
> be in the kernel.

Really? I was specifically trying to avoid that, and let the 
policy about interrupt assignment be done in a user process. 

Do you have any specific ideas about how that'd look? 

-- 
Arthur

Re: [PATCH] mlx4_core: module param to limit msix vec allocation

[David Miller]

From: Arthur Kepner <akepner@sgi.com>
Date: Fri, 3 Sep 2010 15:13:03 -0700

> On Fri, Sep 03, 2010 at 02:46:58PM -0700, David Miller wrote:
>> No, you really can't do it this way, IRQ allocation management has to
>> be in the kernel.
> 
> Really? I was specifically trying to avoid that, and let the 
> policy about interrupt assignment be done in a user process. 
> 
> Do you have any specific ideas about how that'd look? 

Likely there would be a set of policies just like the cpu
power management layer we have.  And there would be a default,
and userland could override the default.

This also means there would have to be notifications sent to
drivers when the distribution and allocation policy of MSI-X
interrupts is changed.  This is so that, for example, network
drivers can reconfigure their network queues for the new set
of MSI-X vectors available to them after the policy change.

Re: [PATCH] mlx4_core: module param to limit msix vec allocation

[Arthur Kepner]

On Fri, Sep 03, 2010 at 07:30:46PM -0700, David Miller wrote:
> ...
> Likely there would be a set of policies just like the cpu
> power management layer we have.  And there would be a default,
> and userland could override the default.

Thanks. I had thought this over again, and come up with 
something like what you describe. Testing now, and plan 
to send a work-in-progress patch later today for comments.

> 
> This also means there would have to be notifications sent to
> drivers when the distribution and allocation policy of MSI-X
> interrupts is changed.  This is so that, for example, network
> drivers can reconfigure their network queues for the new set
> of MSI-X vectors available to them after the policy change.

Hadn't considered this. Will have to add it.

-- 
Arthur

Re: [PATCH] mlx4_core: module param to limit msix vec allocation

[David Miller]

From: Arthur Kepner <akepner@sgi.com>
Date: Mon, 6 Sep 2010 10:42:45 -0700

> On Fri, Sep 03, 2010 at 07:30:46PM -0700, David Miller wrote:
>> This also means there would have to be notifications sent to
>> drivers when the distribution and allocation policy of MSI-X
>> interrupts is changed.  This is so that, for example, network
>> drivers can reconfigure their network queues for the new set
>> of MSI-X vectors available to them after the policy change.
> 
> Hadn't considered this. Will have to add it.

BTW, reading this again made me consider another case that
needs to invoke a notifier.

Let's say that a policy specifies an even distribution of
MSI-X allocation amongst all present devices behind a PCI
domain.  Well, if a PCI-E device hot-plugs you'll have to
do notifications for that too because a card might now be
able to get more of the MSI-X vectors it wants.