"NIG timer max" messages from bnx2x

"NIG timer max" messages from bnx2x

[Tore Anderson]

Hello list,

I'm having a strange error condition I'm hoping someone here might be
able to help me get to the bottom of.  I've got a HP blade enclosure
with a Nortel switch module, and one of the blades are running VMWare
ESX.  When I turn on this blade (or enable its switch port), the entire
network stops working after a short while.  Other server blades can no
longer communicate, for instance.  The only clue I've found so far is
the following error message from one of the other blades that's running
CentOS 5.4:

[bnx2x_hw_stats_update:3701(eth0)]NIG timer max (426)

The line is logged every two seconds and the number in parenthesis is
incrementing each time.  Once I disable the port to the ESX server, the
network instantly recovers.

I have no reason to believe the Linux kernel/bnx2x is doing anything
wrong here, but I'm wondering if anybody has any idea on what this error
message means and what could be causing it?  I can't see anything
unusual going on with tcpdump, and the error counters in "ip -s -s link
show eth0" are all zero.

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

Re: "NIG timer max" messages from bnx2x

[Eilon Greenstein]

On Wed, 2009-12-02 at 08:00 -0800, Tore Anderson wrote:
> Hello list,
> 
> I'm having a strange error condition I'm hoping someone here might be
> able to help me get to the bottom of.  I've got a HP blade enclosure
> with a Nortel switch module, and one of the blades are running VMWare
> ESX.  When I turn on this blade (or enable its switch port), the entire
> network stops working after a short while.  Other server blades can no
> longer communicate, for instance.  The only clue I've found so far is
> the following error message from one of the other blades that's running
> CentOS 5.4:
> 
> [bnx2x_hw_stats_update:3701(eth0)]NIG timer max (426)
> 
> The line is logged every two seconds and the number in parenthesis is
> incrementing each time.  Once I disable the port to the ESX server, the
> network instantly recovers.
> 
> I have no reason to believe the Linux kernel/bnx2x is doing anything
> wrong here, but I'm wondering if anybody has any idea on what this error
> message means and what could be causing it?  I can't see anything
> unusual going on with tcpdump, and the error counters in "ip -s -s link
> show eth0" are all zero.

This message indicates that for over 1 second, the device could not
transmit anything to the network (the same packet was not transmitted
for over one second) - this usually happens when the network is in fatal
condition - usually because some device is sending constant pause stream
that halts all traffic. In this case, the device is incrementing the
counter and drains outgoing traffic - without this action, the device
will not be able to unload since even the commands from the driver will
not be processed due to this excessive back pressure.

As you indicated - once you remove the server that halts all other
traffic, everything else resumes - that indicates that this server is
sending constant pause that ultimately halts the entire switch.

Regards,
Eilon

Re: "NIG timer max" messages from bnx2x

[Tore Anderson]

* Eilon Greenstein

> This message indicates that for over 1 second, the device could not 
> transmit anything to the network (the same packet was not
> transmitted for over one second) - this usually happens when the
> network is in fatal condition - usually because some device is
> sending constant pause stream that halts all traffic. In this case,
> the device is incrementing the counter and drains outgoing traffic -
> without this action, the device will not be able to unload since even
> the commands from the driver will not be processed due to this
> excessive back pressure.
> 
> As you indicated - once you remove the server that halts all other 
> traffic, everything else resumes - that indicates that this server
> is sending constant pause that ultimately halts the entire switch.

Thank you very much for your input Elion, you've definetively helped me
look in the right place.  I do see a constant stream of pause frames
being sent by the switch module to a F5 BIG-IP box that's connected to
external ports of the switch.  The stream appears instantly when the ESX
server is connected and vanishes just as fast when it's been
disconnected.  I haven't been able to verify what the ESX server is
doing to the switch yet, but your assumption that it too is spewing out
pause frames sounds very likely to me.

However I'm still a bit confused on why this is triggering the "NIG
timer max" message on the Linux blade.  It would have made perfect sense
to me for the bnx2x driver/NIC to stop transmitting outbound frames if
it was the recipient of an inbound pause frame flood, but tcpdump does
not show any such frames being received.  Yet it stalls all outbound
traffic.  Any idea why?

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

Re: "NIG timer max" messages from bnx2x

[Eilon Greenstein]

On Wed, 2009-12-02 at 23:36 -0800, Tore Anderson wrote:
> However I'm still a bit confused on why this is triggering the "NIG
> timer max" message on the Linux blade.  It would have made perfect sense
> to me for the bnx2x driver/NIC to stop transmitting outbound frames if
> it was the recipient of an inbound pause frame flood, but tcpdump does
> not show any such frames being received.  Yet it stalls all outbound
> traffic.  Any idea why?
> 
The pause frames are processed in the HW and therefore not visible to
tcpdump. You should be able to see the pause counter increases in
"ethtool -S".

Regards,
Eilon