MLX4 Interrupt Load-Balancing

public inbox for linux-rdma@vger.kernel.org
 help / color / mirror / Atom feed

* MLX4 Interrupt Load-Balancing
@ 2011-02-09 18:36 Nicolas Carlier
       [not found] ` <AANLkTi=Cguhde9-XXFMdrJ966SfGRKeb5K7KBH_ROZtC-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
  0 siblings, 1 reply; 2+ messages in thread
From: Nicolas Carlier @ 2011-02-09 18:36 UTC (permalink / raw)
  To: linux-rdma-u79uwXL29TY76Z2rM5mHXA

Hi,

I use a dual-proc X5670 box without hyperthreading, so I have 12
cores. I use a connectx2 card, with ofed-1.5.2, essentially to make
multicast on IpOIb. I'm on a kernel 2.6.18-164.el5.

First question:

Why I have this type message during the boot ?

"mlx4_core 0000:03:00.0: Requested 25 vectors, but only 14 MSI-X
vectors available, trying again"

I thought that the drivers doesn't try to get more vectors than cpu
cores. In spite of this message the driver is loaded with MSI-X
capabilities.


So my second question:

Why when I look at the "/proc/interrupts", I don't have the impression
that the load balancing is effective. I have the habit to use steering
on ethernet adapters, but the behaviour doesn't corresponding.  I
receive all the interrupts on the first "queue" as show the result of
"cat /proc/interrupts  | grep mlx".

 51:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-9
 59:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-10
 67:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-11
 75:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-12
 83:     223042          0          0          0          0          0
      1091          0          0          0          0          0
 PCI-MSI-X  mlx4_core(async)
170:         72   39800564   17641698      80102     769072  103351470
         0          0          0          0          0   17232132
 PCI-MSI-X  eth-mlx4-0
178:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-1
186:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-2
194:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-3
202:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-4
210:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-5
218:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-6
226:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-7
234:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-8



Thanks for the answer,

Best regards.
N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±ÙšŠ{ayº\x1dÊ‡Ú™ë,j\a¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

^ permalink raw reply	[flat|nested] 2+ messages in thread

[parent not found: <AANLkTi=Cguhde9-XXFMdrJ966SfGRKeb5K7KBH_ROZtC-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]

* Re: MLX4 Interrupt Load-Balancing
       [not found] ` <AANLkTi=Cguhde9-XXFMdrJ966SfGRKeb5K7KBH_ROZtC-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2011-02-11 12:41   ` Eli Cohen
  0 siblings, 0 replies; 2+ messages in thread
From: Eli Cohen @ 2011-02-11 12:41 UTC (permalink / raw)
  To: Nicolas Carlier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA

On Wed, Feb 09, 2011 at 07:36:10PM +0100, Nicolas Carlier wrote:
> Hi,
> 
> I use a dual-proc X5670 box without hyperthreading, so I have 12
> cores. I use a connectx2 card, with ofed-1.5.2, essentially to make
> multicast on IpOIb. I'm on a kernel 2.6.18-164.el5.
> 
> First question:
> 
> Why I have this type message during the boot ?
> 
> "mlx4_core 0000:03:00.0: Requested 25 vectors, but only 14 MSI-X
> vectors available, trying again"
That's because mlx4 will try to allocate two vectors per CPU core and
another one for asynchronous events. The kernel is "willing" to give
fewer vectors. So mlx4_core will try to allocate as much as kernel
gives. IPoIB does not make use of the multitude of vectors; they are
used by mlx4_en which is the ConnectX ethernet driver.

> 
> I thought that the drivers doesn't try to get more vectors than cpu
> cores. In spite of this message the driver is loaded with MSI-X
> capabilities.
> 
> 
> So my second question:
> 
> Why when I look at the "/proc/interrupts", I don't have the impression
> that the load balancing is effective. I have the habit to use steering
> on ethernet adapters, but the behaviour doesn't corresponding.  I
> receive all the interrupts on the first "queue" as show the result of
> "cat /proc/interrupts  | grep mlx".
> 
>  51:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-9
>  59:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-10
>  67:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-11
>  75:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-12
>  83:     223042          0          0          0          0          0
>       1091          0          0          0          0          0
>  PCI-MSI-X  mlx4_core(async)
> 170:         72   39800564   17641698      80102     769072  103351470
>          0          0          0          0          0   17232132
>  PCI-MSI-X  eth-mlx4-0
> 178:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-1
> 186:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-2
> 194:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-3
> 202:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-4
> 210:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-5
> 218:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-6
> 226:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-7
> 234:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-8
> 
> 
> 
> Thanks for the answer,
> 
> Best regards.
> N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±ÙšŠ{ayº\x1dÊ‡Ú™ë,j\a¢f£¢·hš‹àz¹\x1e®w¥¢¸\f¢·¦j:+v‰¨ŠwèjØm¶Ÿÿ¾\a«‘êçzZ+ƒùšŽŠÝ¢j"ú!
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2011-02-11 12:41 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-02-09 18:36 MLX4 Interrupt Load-Balancing Nicolas Carlier
     [not found] ` <AANLkTi=Cguhde9-XXFMdrJ966SfGRKeb5K7KBH_ROZtC-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2011-02-11 12:41   ` Eli Cohen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox