MLX4 Interrupt Load-Balancing

MLX4 Interrupt Load-Balancing

[Nicolas Carlier]

Hi,

I use a dual-proc X5670 box without hyperthreading, so I have 12
cores. I use a connectx2 card, with ofed-1.5.2, essentially to make
multicast on IpOIb. I'm on a kernel 2.6.18-164.el5.

First question:

Why I have this type message during the boot ?

"mlx4_core 0000:03:00.0: Requested 25 vectors, but only 14 MSI-X
vectors available, trying again"

I thought that the drivers doesn't try to get more vectors than cpu
cores. In spite of this message the driver is loaded with MSI-X
capabilities.


So my second question:

Why when I look at the "/proc/interrupts", I don't have the impression
that the load balancing is effective. I have the habit to use steering
on ethernet adapters, but the behaviour doesn't corresponding.  I
receive all the interrupts on the first "queue" as show the result of
"cat /proc/interrupts  | grep mlx".

 51:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-9
 59:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-10
 67:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-11
 75:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-12
 83:     223042          0          0          0          0          0
      1091          0          0          0          0          0
 PCI-MSI-X  mlx4_core(async)
170:         72   39800564   17641698      80102     769072  103351470
         0          0          0          0          0   17232132
 PCI-MSI-X  eth-mlx4-0
178:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-1
186:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-2
194:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-3
202:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-4
210:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-5
218:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-6
226:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-7
234:          0          0          0          0          0          0
         0          0          0          0          0          0
 PCI-MSI-X  eth-mlx4-8



Thanks for the answer,

Best regards.
N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

Re: MLX4 Interrupt Load-Balancing

[Eli Cohen]

On Wed, Feb 09, 2011 at 07:36:10PM +0100, Nicolas Carlier wrote:
> Hi,
> 
> I use a dual-proc X5670 box without hyperthreading, so I have 12
> cores. I use a connectx2 card, with ofed-1.5.2, essentially to make
> multicast on IpOIb. I'm on a kernel 2.6.18-164.el5.
> 
> First question:
> 
> Why I have this type message during the boot ?
> 
> "mlx4_core 0000:03:00.0: Requested 25 vectors, but only 14 MSI-X
> vectors available, trying again"
That's because mlx4 will try to allocate two vectors per CPU core and
another one for asynchronous events. The kernel is "willing" to give
fewer vectors. So mlx4_core will try to allocate as much as kernel
gives. IPoIB does not make use of the multitude of vectors; they are
used by mlx4_en which is the ConnectX ethernet driver.

> 
> I thought that the drivers doesn't try to get more vectors than cpu
> cores. In spite of this message the driver is loaded with MSI-X
> capabilities.
> 
> 
> So my second question:
> 
> Why when I look at the "/proc/interrupts", I don't have the impression
> that the load balancing is effective. I have the habit to use steering
> on ethernet adapters, but the behaviour doesn't corresponding.  I
> receive all the interrupts on the first "queue" as show the result of
> "cat /proc/interrupts  | grep mlx".
> 
>  51:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-9
>  59:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-10
>  67:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-11
>  75:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-12
>  83:     223042          0          0          0          0          0
>       1091          0          0          0          0          0
>  PCI-MSI-X  mlx4_core(async)
> 170:         72   39800564   17641698      80102     769072  103351470
>          0          0          0          0          0   17232132
>  PCI-MSI-X  eth-mlx4-0
> 178:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-1
> 186:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-2
> 194:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-3
> 202:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-4
> 210:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-5
> 218:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-6
> 226:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-7
> 234:          0          0          0          0          0          0
>          0          0          0          0          0          0
>  PCI-MSI-X  eth-mlx4-8
> 
> 
> 
> Thanks for the answer,
> 
> Best regards.
> N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹àz¹\x1e®w¥¢¸\f¢·¦j:+v‰¨ŠwèjØm¶Ÿÿ¾\a«‘êçzZ+ƒùšŽŠÝ¢j"ú!
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html