r8169 crash

r8169 crash

[Denys]

Hi

Recently i start to use r8169 cards and faced some issues.

How to reproduce (kind of difficult, and i cannot that on my live server 
anymore):

1)Badly crimped cable.
2)PCI-Express R8169 card

Kernel 2.6.23.1

05:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit 
Ethernet (rev 10)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- 
Stepping- SERR- FastB2B-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
<TAbort- <MAbort- >SERR- <PERR-
        Latency: 64 (8000ns min, 16000ns max), Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 21
        Region 0: I/O ports at 1000 [size=256]
        Region 1: Memory at 55001000 (32-bit, non-prefetchable) [size=256]
        Expansion ROM at fffe0000 [disabled] [size=128K]
        Capabilities: [dc] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA 
PME(D0-,D1+,D2+,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-


r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
eth1: RTL8169sb/8110sb at 0xf881c000, 00:19:e0:72:49:88, XID 10000000 IRQ 21

To reproduce issue i need to shake faulty cable. Card will report:
Nov 15 22:09:36 vzone r8169: eth1: link up
So it is going up without going down!

Normally it is like this:
Nov  1 00:50:33 vzone r8169: eth1: link down
Nov  1 00:50:35 vzone r8169: eth1: link up

After that i have.
Nov 15 22:11:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:11:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:13 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:25 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:13:01 vzone NETDEV WATCHDOG: eth1: transmit timed out

And card completely non-functional (doesn't see traffic at all and doesn't 
send anything). Probably some functions (like NAPI enabling) is has to be 
disabled first, and if interface come twice up, without going down, card come 
crazy.


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

Re: r8169 crash

[Francois Romieu]

Denys <nuclearcat@nuclearcat.com> :
[...]
> After that i have.
> Nov 15 22:11:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
> Nov 15 22:11:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
> Nov 15 22:12:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
> Nov 15 22:12:13 vzone NETDEV WATCHDOG: eth1: transmit timed out
> Nov 15 22:12:25 vzone NETDEV WATCHDOG: eth1: transmit timed out
> Nov 15 22:12:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
> Nov 15 22:12:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
> Nov 15 22:13:01 vzone NETDEV WATCHDOG: eth1: transmit timed out

Is there a chance for you to verify that the network interface does
not recover if a real cable is plugged at this point ?

-- 
Ueimor

Re: r8169 crash

[Denys]

Before it happens on 2.6.22, i tried to attach good cable, plug-unplug, 
whatever, interface up/down - card still remains dead.

I try to plug cable to laptop with rtl8139 and pc with e100 - it is worked 
when i shake bad cable(interface was going up/down as well), and good cable 
also worked fine, and never crashed. On 2.6.22 i was able to reproduce it 
easily, but later i can't cause it is server in internet cafe, and customers 
going crazy.


On Sat, 17 Nov 2007 23:07:59 +0100, Francois Romieu wrote
> Denys <nuclearcat@nuclearcat.com> :
> [...]
> > After that i have.
> > Nov 15 22:11:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
> > Nov 15 22:11:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
> > Nov 15 22:12:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
> > Nov 15 22:12:13 vzone NETDEV WATCHDOG: eth1: transmit timed out
> > Nov 15 22:12:25 vzone NETDEV WATCHDOG: eth1: transmit timed out
> > Nov 15 22:12:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
> > Nov 15 22:12:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
> > Nov 15 22:13:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
> 
> Is there a chance for you to verify that the network interface does
> not recover if a real cable is plugged at this point ?
> 
> -- 
> Ueimor


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

Re: r8169 crash

[Francois Romieu]

Denys <nuclearcat@nuclearcat.com> :
> 
> Before it happens on 2.6.22, i tried to attach good cable, plug-unplug, 
> whatever, interface up/down - card still remains dead.

A few things have changed since 2.6.22 but I'll take that it is a
real bug and that 2.6.23 would not recover either. Can you fill a
PR at http://bugzilla.kernel.org, add a lspci -vvvxxxx and a dmesg ?

A dmesg including the watchdog message would be nice but if you can
not retrieve one, a dmesg from your current 2.6.23.x kernel should be
enough.

I'll take care of it once I am done with the sis190 watchdog failure
(nothing hard, simply ugly).

[...]
> I try to plug cable to laptop with rtl8139 and pc with e100 - it is worked 
> when i shake bad cable(interface was going up/down as well), and good cable 
> also worked fine, and never crashed. On 2.6.22 i was able to reproduce it 
> easily, but later i can't cause it is server in internet cafe, and customers 
> going crazy.

:o/

-- 
Ueimor

Re: r8169 crash

[Denys]

Maybe it is good idea to print some registers/etc in watchdog message?

I will try to reproduce problem with debug enabled, maybe it will be possible 
to catch problem with more debug. 

On Sun, 18 Nov 2007 22:00:38 +0100, Francois Romieu wrote
> Denys <nuclearcat@nuclearcat.com> :
> > 
> > Before it happens on 2.6.22, i tried to attach good cable, plug-unplug, 
> > whatever, interface up/down - card still remains dead.
> 
> A few things have changed since 2.6.22 but I'll take that it is a
> real bug and that 2.6.23 would not recover either. Can you fill a
> PR at http://bugzilla.kernel.org, add a lspci -vvvxxxx and a dmesg ?
> 
> A dmesg including the watchdog message would be nice but if you can
> not retrieve one, a dmesg from your current 2.6.23.x kernel should be
> enough.
> 
> I'll take care of it once I am done with the sis190 watchdog failure
> (nothing hard, simply ugly).
> 
> [...]
> > I try to plug cable to laptop with rtl8139 and pc with e100 - it is 
worked 
> > when i shake bad cable(interface was going up/down as well), and good 
cable 
> > also worked fine, and never crashed. On 2.6.22 i was able to reproduce it 
> > easily, but later i can't cause it is server in internet cafe, and 
customers 
> > going crazy.
> 
> :o/
> 
> -- 
> Ueimor
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

Re: r8169 crash

[Francois Romieu]

Denys <nuclearcat@nuclearcat.com> :
> Maybe it is good idea to print some registers/etc in watchdog message?
> 
> I will try to reproduce problem with debug enabled, maybe it will be possible 
> to catch problem with more debug. 

Your customers will not necessarily like it but it could be useful to
trace the Rx/Tx descriptor ring content and the irq status word.

If your make it dependent on tp->msg_enable (settable via
'ethtool -s ethX msglvl 65535), you can enable it at will and catch
some bit of normal traffic just before you experiment with the
magic cable.


> On Sun, 18 Nov 2007 22:00:38 +0100, Francois Romieu wrote
> > Denys <nuclearcat@nuclearcat.com> :
[...]
> > I'll take care of it once I am done with the sis190 watchdog failure
> > (nothing hard, simply ugly).

Actually I lack a reliable indication that the Tx process is stuck.
My hack seems able to recover but it can be quite slow to trigger.
A contact at SiS would be welcome.

-- 
Ueimor