rtl8139 nic dies with load (2.4.10, kt266)

rtl8139 nic dies with load (2.4.10, kt266)

[Oliver Seemann]

Hi,

my box got 2 rtl8139 nics (longshine iirc), eth0 connected to localnet
and eth1 for adsl (pppoe).
since i upgraded the hardware to via kt266 + athlon c 1.4 and kernel
to 2.4.9 all network activity dies when i copy big files over
localnet. with old hx chipset, cyrix 200mhz cpu and redhat 2.4.3
kernel i didn't have these problems.
no matter if ftp or smb, when transferring files over eth0 to a win2k
pc (3com nic), after some seconds or mbytes transfer, suddenly all
network activity on eth0 dies. the adsl connection over eth1 remains
alive.
after /etc/rc.d/init.d/network restart  all is fine again (rebooting
the win2k pc doesn't help ;). and it remains so until i make significant
traffic. even while watching an divx files over a samba share, the nic
died after 45 minutes of continuous watching.

meanwhile i updated to 2.4.10 but it does not fix it. also 2.4.9-ac16
shows the same behaviour.
i found no messages in /var/log/* or on console telling errors.
below some hw-specs (rtl8139-diag output at the end).
i'd be glad if i could help with more information or trying patches etc.

good luck,
Oliver Seemann

ps: i'm not subscribed, please reply directly. thx
----

redhat 7.1 system on msi k7t266 pro-r with 512mb pc266 ram.
kernel 2.4.10 with iptables-1.2.3
gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-85)

from dmesg:
8139too Fast Ethernet driver 0.9.18a
eth0: RealTek RTL8139 Fast Ethernet at 0xe086bf00, 00:e0:7d:c0:01:99, IRQ
12
eth0:  Identified 8139 chip type 'RTL-8139B'
eth1: RealTek RTL8139 Fast Ethernet at 0xe086de00, 00:e0:7d:9c:7d:f1, IRQ
5
eth1:  Identified 8139 chip type 'RTL-8139C'
eth0: Setting 100mbps full-duplex based on auto-negotiated partner ability
45e1.
task `ifconfig' exit_signal 17 in reparent_to_init
eth1: Setting half-duplex based on auto-negotiated partner ability 0000.
task `ifconfig' exit_signal 17 in reparent_to_init

from /proc/cpuinfo:
cpu family      : 6
model           : 4
model name      : AMD Athlon(tm) Processor
stepping        : 4
cpu MHz         : 1399.803

from /proc/pci:
PCI devices found:
  Bus  0, device   0, function  0:
    Host bridge: VIA Technologies, Inc. VT8367 [KT266] (rev 0).
      Master Capable.  Latency=8.  
      Prefetchable 32 bit memory at 0xe0000000 [0xe3ffffff].
  Bus  0, device   1, function  0:
    PCI bridge: VIA Technologies, Inc. VT8367 [KT266 AGP] (rev 0).
      Master Capable.  No bursts.  Min Gnt=4.
  Bus  0, device   7, function  0:
    VGA compatible controller: ATI Technologies Inc 264VT [Mach64 VT] (rev
64).
      Non-prefetchable 32 bit memory at 0xdf000000 [0xdfffffff].
      I/O at 0xd800 [0xd8ff].
  Bus  0, device   8, function  0:
    Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev
16).
      IRQ 12.
      Master Capable.  Latency=64.  Min Gnt=32.Max Lat=64.
      I/O at 0xc000 [0xc0ff].
      Non-prefetchable 32 bit memory at 0xdefbff00 [0xdefbffff].
  Bus  0, device   9, function  0:
    Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (#2)
(rev 16).
      IRQ 5.
      Master Capable.  Latency=64.  Min Gnt=32.Max Lat=64.
      I/O at 0xbc00 [0xbcff].
      Non-prefetchable 32 bit memory at 0xdefbfe00 [0xdefbfeff].
  Bus  0, device  12, function  0:
    RAID bus controller: Promise Technology, Inc. 20265 (rev 2).
      IRQ 11.
      Master Capable.  Latency=64.  
      I/O at 0xd400 [0xd407].
      I/O at 0xd000 [0xd003].
      I/O at 0xcc00 [0xcc07].
      I/O at 0xc800 [0xc803].
      I/O at 0xc400 [0xc43f].
      Non-prefetchable 32 bit memory at 0xdefc0000 [0xdefdffff].
  Bus  0, device  17, function  0:
    ISA bridge: VIA Technologies, Inc. VT8233 PCI to ISA Bridge (rev 0).
  Bus  0, device  17, function  1:
    IDE interface: VIA Technologies, Inc. Bus Master IDE (rev 6).
      Master Capable.  Latency=32.  
      I/O at 0xff00 [0xff0f].
  Bus  0, device  17, function  4:
    USB Controller: VIA Technologies, Inc. UHCI USB (#3) (rev 24).
      IRQ 12.
      Master Capable.  Latency=64.  
      I/O at 0xb400 [0xb41f].
  Bus  0, device  17, function  3:
    USB Controller: VIA Technologies, Inc. UHCI USB (#2) (rev 24).
      IRQ 12.
      Master Capable.  Latency=64.  
      I/O at 0xb000 [0xb01f].
  Bus  0, device  17, function  2:
    USB Controller: VIA Technologies, Inc. UHCI USB (rev 24).
      IRQ 12.
      Master Capable.  Latency=64.  
      I/O at 0xac00 [0xac1f].
  Bus  0, device  17, function  5:
    Multimedia audio controller: VIA Technologies, Inc. AC97 Audio
Controller (rev 16).
      IRQ 10.
      I/O at 0xb800 [0xb8ff].


rtl8139-diag -mmaaavvveef output, with eth0 ALIVE:

rtl8139-diag.c:v2.01 1/8/2001 Donald Becker (becker@scyld.com)
 http://www.scyld.com/diag/index.html
Index #1: Found a RealTek RTL8139 adapter at 0xc000.
RealTek chip registers at 0xc000
 0x000: c07de000 00009901 80000000 00000000 0008a04a 0008a04a 0008a04a
0008a05a
 0x020: 1c44e000 1c44e600 1c44ec00 1c44f200 1f300000 0d0a0000 96b096a0
0000c07f
 0x040: 78000600 0e00f78e 53539ad8 00000000 004d1000 00000000 0080c180
00100000
 0x060: 1100f00f 05e1782d 000145e1 00000000 00000004 000517c0 58fab388
a438d843.
  No interrupt sources are pending.
 The chip configuration is 0x10 0x4d, MII full-duplex mode.
EEPROM size test returned 6, 0x204a4 / 0x2.
Parsing the EEPROM of a RealTek chip:
  PCI IDs -- Vendor 0x10ec, Device 0x8139, Subsystem 0x10ec.
  PCI timer settings -- minimum grant 32, maximum latency 64.
  General purpose pins --  direction 0xe1  value 0x10.
  Station Address 00:E0:7D:C0:01:99.
  Configuration register 0/1 -- 0x4d / 0xc2.
 EEPROM active region checksum is 0a34.
EEPROM contents:
  8129 10ec 8139 10ec 8139 4020 e110 e000
  c07d 9901 4d10 f7c2 8001 b388 58fa 0708
  d843 a438 d843 a438 d843 a438 d843 a438
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
 The word-wide EEPROM checksum is 0xca6a.
Index #2: Found a RealTek RTL8139 adapter at 0xbc00.
RealTek chip registers at 0xbc00
 0x000: 9c7de000 0000f17d 80000000 00000000 9008a5c2 9008a5c2 9008a5c2
9008a5c2
 0x020: 1bc8a000 1bc8a600 1bc8ac00 1bc8b200 02dd0000 0d0a0000 fcd8fcc8
0000c07f
 0x040: 74000600 0e00f78e 53563b8a 00000000 004d1000 00000000 0088c118
00100000
 0x060: 1000f00f 01e1782d 00000000 00000000 00000005 000f77c0 b0f243b9
7a36d743.
  No interrupt sources are pending.
 The chip configuration is 0x10 0x4d, MII full-duplex mode.
EEPROM size test returned 6, 0x204a4 / 0x2.
Parsing the EEPROM of a RealTek chip:
  PCI IDs -- Vendor 0x10ec, Device 0x8139, Subsystem 0x10ec.
  PCI timer settings -- minimum grant 32, maximum latency 64.
  General purpose pins --  direction 0xe1  value 0x12.
  Station Address 00:E0:7D:9C:7D:F1.
  Configuration register 0/1 -- 0x4d / 0xc2.
 EEPROM active region checksum is 0ae6.
EEPROM contents:
  8129 10ec 8139 10ec 8139 4020 e112 e000
  9c7d f17d 4d10 f7c2 8801 43b9 b0f2 031a
  df43 8a36 df43 8a36 43b9 b0f2 1111 1111
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
 The word-wide EEPROM checksum is 0xe2f6.


rtl8139-diag -mmaaavvveef output, with eth0 DEAD:

rtl8139-diag.c:v2.01 1/8/2001 Donald Becker (becker@scyld.com)
 http://www.scyld.com/diag/index.html
Index #1: Found a RealTek RTL8139 adapter at 0xc000.
RealTek chip registers at 0xc000
 0x000: c07de000 00009901 80000000 00000000 0008a5ea 0008a5ea 0008a5ea
0008a5ea
 0x020: 1c44e000 1c44e600 1c44ec00 1c44f200 1f300000 0d000000 0000fff0
0000c07f
 0x040: 78000600 00000000 fc7278d1 00000000 004d10c6 00000000 0080c180
00100000
 0x060: 1100f00f 05e1782d 000145e1 00000000 00000004 000417c8 58fab388
a438d843.
  No interrupt sources are pending.
 The chip configuration is 0x10 0x4d, MII full-duplex mode.
EEPROM size test returned 6, 0x204a4 / 0x2.
Parsing the EEPROM of a RealTek chip:
  PCI IDs -- Vendor 0x10ec, Device 0x8139, Subsystem 0x10ec.
  PCI timer settings -- minimum grant 32, maximum latency 64.
  General purpose pins --  direction 0xe1  value 0x10.
  Station Address 00:E0:7D:C0:01:99.
  Configuration register 0/1 -- 0x4d / 0xc2.
 EEPROM active region checksum is 0a34.
EEPROM contents:
  8129 10ec 8139 10ec 8139 4020 e110 e000
  c07d 9901 4d10 f7c2 8001 b388 58fa 0708
  d843 a438 d843 a438 d843 a438 d843 a438
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
 The word-wide EEPROM checksum is 0xca6a.
Index #2: Found a RealTek RTL8139 adapter at 0xbc00.
RealTek chip registers at 0xbc00
 0x000: 9c7de000 0000f17d 80000000 00000000 9008a5c2 9008a5c2 9008a5c2
9008a5c2
 0x020: 1bc8a000 1bc8a600 1bc8ac00 1bc8b200 02dd0000 0d0a0000 fdf8fde8
0000c07f
 0x040: 74000600 0e00f78e fc751d5c 00000000 004d10c6 00000000 0088c118
00100000
 0x060: 1000f00f 01e1782d 00000000 00000000 00000005 000f77c0 b0f243b9
7a36d743.
  No interrupt sources are pending.
 The chip configuration is 0x10 0x4d, MII full-duplex mode.
EEPROM size test returned 6, 0x204a4 / 0x2.
Parsing the EEPROM of a RealTek chip:
  PCI IDs -- Vendor 0x10ec, Device 0x8139, Subsystem 0x10ec.
  PCI timer settings -- minimum grant 32, maximum latency 64.
  General purpose pins --  direction 0xe1  value 0x12.
  Station Address 00:E0:7D:9C:7D:F1.
  Configuration register 0/1 -- 0x4d / 0xc2.
 EEPROM active region checksum is 0ae6.
EEPROM contents:
  8129 10ec 8139 10ec 8139 4020 e112 e000
  9c7d f17d 4d10 f7c2 8801 43b9 b0f2 031a
  df43 8a36 df43 8a36 43b9 b0f2 1111 1111
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
  0000 0000 0000 0000 0000 0000 0000 0000
 The word-wide EEPROM checksum is 0xe2f6.


EOF

Re: rtl8139 nic dies with load (2.4.10, kt266)

[Manfred Spraul]

> no matter if ftp or smb, when transferring files over eth0 to a win2k
> pc (3com nic), after some seconds or mbytes transfer, suddenly all
> network activity on eth0 dies. the adsl connection over eth1 remains
> alive.

Could you enable debugging in the 8139too driver and send us the dmesg
log?

Around line 170 in linux/drivers/net/8139too.c
replace
	#undef RTL8139_DEBUG
with
	#define RTL8139_DEBUG	1

Recompile, reboot, load eth0 until it locks up, and send us the dmesg
output.

--
	Manfred

Re: rtl8139 nic dies with load (2.4.10, kt266)

[Oliver Seemann]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 494 bytes --]

hi,

> Recompile, reboot, load eth0 until it locks up, and send us the dmesg
> output.

ok done.

i've attached the zipped log (>200k).
the most interesting part ist maybe the following:

rtl8139_rx_err: eth0: Ethernet frame had errors, status 194f4571.
rtl8139_set_rx_mode: ENTER
rtl8139_set_rx_mode: eth0:   rtl8139_set_rx_mode(1003) done -- Rx config
00000000.
ether_crc: ENTER
ether_crc: EXIT, returning 2141400475
rtl8139_set_rx_mode: EXIT

hmm faulty ethernet frame, nic broken ?

Oliver

[-- Attachment #2: Type: APPLICATION/octet-stream, Size: 7155 bytes --]

Re: rtl8139 nic dies with load (2.4.10, kt266)

[Manfred Spraul]

 
Content-Type: multipart/mixed;
 boundary="------------E6DF444F67EB539614CC6DDF"

This is a multi-part message in MIME format.
--------------E6DF444F67EB539614CC6DDF
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

> 
> > Recompile, reboot, load eth0 until it locks up, and send us the dmesg
> > output.
> 
> ok done.
> 
> i've attached the zipped log (>200k).
> the most interesting part ist maybe the following:
> 
> rtl8139_rx_err: eth0: Ethernet frame had errors, status 194f4571.
> rtl8139_set_rx_mode: ENTER
> rtl8139_set_rx_mode: eth0:   rtl8139_set_rx_mode(1003) done -- Rx config
> 00000000.
> ether_crc: ENTER
> ether_crc: EXIT, returning 2141400475
> rtl8139_set_rx_mode: EXIT
> 
> hmm faulty ethernet frame, nic broken ?

No - a few frame errors are acceptable. Lots of electical noise, long
cable, bad cable, whatever.
The driver must recover from such errors.

But I think I found the error:
* set_rx_mode optimizes away the configuration change if it thinks that
the configuration didn't change.
* rtl_8139_rx_err changes the configuration of the NIC chip without
updating tp->rx_config.

Could you try the attached patch?

--
	Manfred
--------------E6DF444F67EB539614CC6DDF
Content-Type: text/plain; charset=us-ascii;
 name="patch-8139-rx_start"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="patch-8139-rx_start"

--- 2.4/drivers/net/8139too.c	Sun Sep 23 21:20:35 2001
+++ build-2.4/drivers/net/8139too.c	Sun Sep 30 17:57:40 2001
@@ -1865,6 +1865,7 @@
 
 	/* disable receive */
 	RTL_W8 (ChipCmd, CmdTxEnb);
+	tp->rx_config = 0;
 
 	/* A.C.: Reset the multicast list. */
 	rtl8139_set_rx_mode (dev);

--------------E6DF444F67EB539614CC6DDF--

Re: rtl8139 nic dies with load (2.4.10, kt266)

[Oliver Seemann]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 253 bytes --]

ok patch applied (and recompiled ;). but error still occurs.

after copying 300mb in 50 seconds over samba it's stuck again :/

dmesg output attached.

(meanwhile i exchanged the nic with another one from lcs with rtl8139
chip, but didn't help)

Oliver

[-- Attachment #2: Type: APPLICATION/octet-stream, Size: 934 bytes --]

Re: rtl8139 nic dies with load (2.4.10, kt266)

[Manfred Spraul]

[-- Attachment #1: Type: text/plain, Size: 183 bytes --]

New patch:
* deadlock in error handling fixed.
* error handling didn't restart the chip properly.

Could you try it?
It should now recover properly from the frame errors.

--
	Manfred

[-- Attachment #2: patch-8139-rx_start --]
[-- Type: text/plain, Size: 3872 bytes --]

--- 2.4/drivers/net/8139too.c	Sun Sep 23 21:20:35 2001
+++ build-2.4/drivers/net/8139too.c	Sun Sep 30 22:31:02 2001
@@ -618,6 +618,7 @@
 static struct net_device_stats *rtl8139_get_stats (struct net_device *dev);
 static inline u32 ether_crc (int length, unsigned char *data);
 static void rtl8139_set_rx_mode (struct net_device *dev);
+static void __set_rx_mode (struct net_device *dev);
 static void rtl8139_hw_start (struct net_device *dev);
 
 #ifdef USE_IO_OPS
@@ -1844,8 +1845,8 @@
 			    struct rtl8139_private *tp, void *ioaddr)
 {
 	u8 tmp8;
-	int tmp_work = 1000;
-
+	int tmp_work;
+    
 	DPRINTK ("%s: Ethernet frame had errors, status %8.8x.\n",
 	         dev->name, rx_status);
 	if (rx_status & RxTooLong) {
@@ -1860,33 +1861,62 @@
 		tp->stats.rx_length_errors++;
 	if (rx_status & RxCRCErr)
 		tp->stats.rx_crc_errors++;
+
 	/* Reset the receiver, based on RealTek recommendation. (Bug?) */
-	tp->cur_rx = 0;
 
 	/* disable receive */
-	RTL_W8 (ChipCmd, CmdTxEnb);
-
-	/* A.C.: Reset the multicast list. */
-	rtl8139_set_rx_mode (dev);
-
-	/* XXX potentially temporary hack to
-	 * restart hung receiver */
+	RTL_W8_F (ChipCmd, CmdTxEnb);
+	tmp_work = 200;
 	while (--tmp_work > 0) {
-		barrier();
+		udelay(1);
+		tmp8 = RTL_R8 (ChipCmd);
+		if (!(tmp8 & CmdRxEnb))
+			break;
+	}
+	if (tmp_work <= 0)
+		printk (KERN_WARNING PFX "rx stop wait too long\n");
+	/* restart receive */
+	tmp_work = 200;
+	while (--tmp_work > 0) {
+		RTL_W8_F (ChipCmd, CmdRxEnb | CmdTxEnb);
+		udelay(1);
 		tmp8 = RTL_R8 (ChipCmd);
 		if ((tmp8 & CmdRxEnb) && (tmp8 & CmdTxEnb))
 			break;
-		RTL_W8 (ChipCmd, CmdRxEnb | CmdTxEnb);
 	}
-
-	/* G.S.: Re-enable receiver */
-	/* XXX temporary hack to work around receiver hang */
-	rtl8139_set_rx_mode (dev);
-
 	if (tmp_work <= 0)
 		printk (KERN_WARNING PFX "tx/rx enable wait too long\n");
-}
 
+	/* and reinitialize all rx related registers */
+	{	/* I have no idea what I'm doing, just copied
+		 * from rtl8139_hw_start.
+		 *	Manfred Spraul
+		 */
+		RTL_W8_F (Cfg9346, Cfg9346_Unlock);
+		/* Must enable Tx/Rx before setting transfer thresholds! */
+		RTL_W8 (ChipCmd, CmdRxEnb | CmdTxEnb);
+
+		tp->rx_config = rtl8139_rx_config | AcceptBroadcast | AcceptMyPhys;
+		RTL_W32 (RxConfig, tp->rx_config);
+		tp->cur_rx = 0;
+
+		if (tp->chipset >= CH_8139B) {
+			/* disable magic packet scanning, which is enabled
+			 * when PM is enabled in Config1 */
+			RTL_W8 (Config3, RTL_R8 (Config3) & ~(1<<5));
+		}
+
+		DPRINTK("init buffer addresses\n");
+
+		/* Lock Config[01234] and BMCR register writes */
+		RTL_W8 (Cfg9346, Cfg9346_Lock);
+
+		/* init Rx ring buffer DMA address */
+		RTL_W32_F (RxBuf, tp->rx_ring_dma);
+	}
+	/* A.C.: Reset the multicast list. */
+	__set_rx_mode (dev);
+}
 
 static void rtl8139_rx_interrupt (struct net_device *dev,
 				  struct rtl8139_private *tp, void *ioaddr)
@@ -2309,11 +2339,10 @@
 }
 
 
-static void rtl8139_set_rx_mode (struct net_device *dev)
+static void __set_rx_mode (struct net_device *dev)
 {
 	struct rtl8139_private *tp = dev->priv;
 	void *ioaddr = tp->mmio_addr;
-	unsigned long flags;
 	u32 mc_filter[2];	/* Multicast hash filter */
 	int i, rx_mode;
 	u32 tmp;
@@ -2350,22 +2379,28 @@
 		}
 	}
 
-	spin_lock_irqsave (&tp->lock, flags);
-
 	/* We can safely update without stopping the chip. */
 	tmp = rtl8139_rx_config | rx_mode;
 	if (tp->rx_config != tmp) {
-		RTL_W32 (RxConfig, tmp);
+		RTL_W32_F (RxConfig, tmp);
 		tp->rx_config = tmp;
 	}
 	RTL_W32_F (MAR0 + 0, mc_filter[0]);
 	RTL_W32_F (MAR0 + 4, mc_filter[1]);
 
-	spin_unlock_irqrestore (&tp->lock, flags);
 
 	DPRINTK ("EXIT\n");
 }
 
+static void rtl8139_set_rx_mode (struct net_device *dev)
+{
+	unsigned long flags;
+	struct rtl8139_private *tp = dev->priv;
+
+	spin_lock_irqsave (&tp->lock, flags);
+	__set_rx_mode(dev);
+	spin_unlock_irqrestore (&tp->lock, flags);
+}
 
 #ifdef CONFIG_PM
 

Re: rtl8139 nic dies with load (2.4.10, kt266)

[Oliver Seemann]

Yes. Seems to work fine now :)

I put the nic on massive traffic and it remained stable.

Thanks a lot
Oliver