Transmit timeout with E1000

Transmit timeout with E1000

[Erik Mouw]

Hi,

I have lots of transmit timeouts with an Intel E1000 card during large
TCP transmissions (remotely viewing a 3000x2000 jpeg image using XV is
an excellent way to trigger it). This is what I get in linux-2.6.8.1:

Jan 10 15:24:41 zurix kernel: NETDEV WATCHDOG: eth0: transmit timed out
Jan 10 15:24:41 zurix kernel: e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
Jan 10 15:24:46 zurix kernel: nfs: server abra2 not responding, still trying
Jan 10 15:24:46 zurix kernel: nfs: server abra2 OK

And this is with linux-2.6.15:

Jan 10 06:53:27 zurix kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
Jan 10 06:53:27 zurix kernel:   TDH                  <b0>
Jan 10 06:53:27 zurix kernel:   TDT                  <b0>
Jan 10 06:53:27 zurix kernel:   next_to_use          <b0>
Jan 10 06:53:27 zurix kernel:   next_to_clean        <c3>
Jan 10 06:53:27 zurix kernel: buffer_info[next_to_clean]
Jan 10 06:53:27 zurix kernel:   dma                  <e938a5e>
Jan 10 06:53:27 zurix kernel:   time_stamp           <872de93>
Jan 10 06:53:27 zurix kernel:   next_to_watch        <c3>
Jan 10 06:53:27 zurix kernel:   jiffies              <872e086>
Jan 10 06:53:27 zurix kernel:   next_to_watch.status <0>
Jan 10 06:53:29 zurix kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
Jan 10 06:53:29 zurix kernel:   TDH                  <b0>
Jan 10 06:53:29 zurix kernel:   TDT                  <b0>
Jan 10 06:53:29 zurix kernel:   next_to_use          <b0>
Jan 10 06:53:29 zurix kernel:   next_to_clean        <c3>
Jan 10 06:53:29 zurix kernel: buffer_info[next_to_clean]
Jan 10 06:53:29 zurix kernel:   dma                  <e938a5e>
Jan 10 06:53:29 zurix kernel:   time_stamp           <872de93>
Jan 10 06:53:29 zurix kernel:   next_to_watch        <c3>
Jan 10 06:53:29 zurix kernel:   jiffies              <872e27a>
Jan 10 06:53:29 zurix kernel:   next_to_watch.status <0>
Jan 10 06:53:31 zurix kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
Jan 10 06:53:31 zurix kernel:   TDH                  <b0>
Jan 10 06:53:31 zurix kernel:   TDT                  <b0>
Jan 10 06:53:31 zurix kernel:   next_to_use          <b0>
Jan 10 06:53:31 zurix kernel:   next_to_clean        <c3>
Jan 10 06:53:31 zurix kernel: buffer_info[next_to_clean]
Jan 10 06:53:31 zurix kernel:   dma                  <e938a5e>
Jan 10 06:53:31 zurix kernel:   time_stamp           <872de93>
Jan 10 06:53:31 zurix kernel:   next_to_watch        <c3>
Jan 10 06:53:31 zurix kernel:   jiffies              <872e46e>
Jan 10 06:53:31 zurix kernel:   next_to_watch.status <0>
Jan 10 06:53:32 zurix kernel: nfs: server abra2 not responding, still trying
Jan 10 06:53:33 zurix kernel: NETDEV WATCHDOG: eth0: transmit timed out
Jan 10 06:53:36 zurix kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex
Jan 10 06:53:37 zurix kernel: nfs: server abra2 OK

The system is a an AMD Athlon XP 2000+ running at 1.666 GHz with a VIA
KT400 chipset (Asrock K7VT4APro).

Here's the relevant output from lspci:

0000:00:0b.0 Ethernet controller: Intel Corporation 82541PI Gigabit
Ethernet Controller (rev 05)
        Subsystem: Intel Corporation: Unknown device 1376
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (63750ns min), Cache Line Size: 0x08 (32 bytes)
        Interrupt: pin A routed to IRQ 19
        Region 0: Memory at dffc0000 (32-bit, non-prefetchable) [size=128K]
        Region 1: Memory at dffa0000 (32-bit, non-prefetchable) [size=128K]
        Region 2: I/O ports at d400 [size=64]
        Expansion ROM at fffe0000 [disabled] [size=128K]
        Capabilities: [dc] Power Management version 2
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [e4] PCI-X non-bridge device.
                Command: DPERE- ERO+ RBC=0 OST=0
                Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=2, DMOST=0, DMCRS=0, RSCEM-
00: 86 80 7c 10 17 00 30 02 05 00 00 02 08 20 00 00
10: 00 00 fc df 00 00 fa df 01 d4 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 76 13
30: 00 00 fe ff dc 00 00 00 00 00 00 00 0c 01 ff 00

Loaded modules (with 2.6.8.1): nfsd exportfs sd_mod sg lp sr_mod
autofs4 nfs lockd sunrpc ide_cd cdrom floppy parport_pc parport
8250_pnp 8250 serial_core snd_via82xx snd_ac97_codec snd_pcm_oss
snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart
snd_rawmidi snd_seq_device snd soundcore joydev evdev ehci_hcd usbhid
uhci_hcd usbcore sata_via libata e1000 reiserfs mga via_agp agpgart .

So far I have replaced the NIC, the motherboard, the power supply, RAM,
network cable, and gigE switch, but to no avail. I've tried three
different kernels (2.6.8.1, 2.6.11-ac7, and 2.6.15) but the problem
remains. I've been stress testing the system by continuously compiling
kernels (over NFS), but after 288 runs there hasn't been a single error
so I guess the CPU and RAM are OK. The amount of transmit timeouts is
less with linux-2.6.8.1, so for the moment I keep running that version.

We have about 15 other machines using the Intel E1000, but I haven't
seen these kind of problems on any of the other machines. I have run
out of ideas, so I hope somebody knows how to solve this. If you need
more information, just let me know.


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click

Re: Transmit timeout with E1000

[Erik Mouw]

On Tue, Jan 10, 2006 at 09:46:29AM -0800, Jesse Brandeburg wrote:
> sorry to hear you're having a problem, and cool, thanks for the test,
> we'll have to try it here.  We've classically had problems reproducing the
> athlon based hangs.

Athlon based or Athlon-on-VIA-KT400 based? We have an E1000 dual
interface server adapter on a dual Athlon with AMD 762 chipset running
fine, and also the same kind of adapter on a dual Athlon64 with
AMD-8111 chipset running fine.

> On Tue, 10 Jan 2006, Erik Mouw wrote:
> >And this is with linux-2.6.15:
> >
> >Jan 10 06:53:27 zurix kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx 
> >Unit Hang
> >Jan 10 06:53:27 zurix kernel:   TDH                  <b0>
> >Jan 10 06:53:27 zurix kernel:   TDT                  <b0>
> >Jan 10 06:53:27 zurix kernel:   next_to_use          <b0>
> >Jan 10 06:53:27 zurix kernel:   next_to_clean        <c3>
> >Jan 10 06:53:27 zurix kernel: buffer_info[next_to_clean]
> >Jan 10 06:53:27 zurix kernel:   dma                  <e938a5e>
> >Jan 10 06:53:27 zurix kernel:   time_stamp           <872de93>
> >Jan 10 06:53:27 zurix kernel:   next_to_watch        <c3>
> >Jan 10 06:53:27 zurix kernel:   jiffies              <872e086>
> >Jan 10 06:53:27 zurix kernel:   next_to_watch.status <0>
> 
> ugh, I don't get it, there is no way in the code that I know of that we 
> would not update TDT when we enqueued a transmit.

FWIW, I'm running a PREEMPT kernel with 4K stacks. Don't know if that's
relevant.

> These problems (for us) seem to be related to TSO, can you attempt to 
> disable it and try your test again, using
> ethtool -K eth0 tso off

OK (see below for results).

> >The system is a an AMD Athlon XP 2000+ running at 1.666 GHz with a VIA
> >KT400 chipset (Asrock K7VT4APro).
> 
> ah yes, this is the famous one that seems to get lots of problem reports. 
> You are running the latest bios, right?  Seems lame but that has actually 
> fixed problems here.

Hmm, that's one thing I didn't check. I wasn't running the latest BIOS,
I just upgraded from the 1.10 to the 1.50 version.

> >Here's the relevant output from lspci:
> 
> <snip>
> 
> >So far I have replaced the NIC, the motherboard, the power supply, RAM,
> >network cable, and gigE switch, but to no avail. I've tried three
> >different kernels (2.6.8.1, 2.6.11-ac7, and 2.6.15) but the problem
> >remains. I've been stress testing the system by continuously compiling
> >kernels (over NFS), but after 288 runs there hasn't been a single error
> >so I guess the CPU and RAM are OK. The amount of transmit timeouts is
> >less with linux-2.6.8.1, so for the moment I keep running that version.
> 
> wow, thats a lot of work, I'm almost at the point of a personal crusade 
> against these timeout issues.  The biggest block we have to solving them 
> is lack of reproduction locally.

If you can get hold of an Asrock K7VT4APro mainboard and an Athlon CPU,
you should be able to reproduce it. They're not too expensive, IIRC we
paid 48 EUR for it. See http://www.asrock.com/product/K7VT4A%20PRO.htm .

> like i said, try disabling TSO and see if that helps.  Please try driver 
> 6.3.9 from prdownloads.sf.net/e1000 and see if that changes anything too.

I upgraded the BIOS, installed the sf.net 6.3.9 driver, disabled TSO,
and used linux-2.6.15. I still get TX timeouts, but less. Right now the
amount is like linux-2.6.8.1 with old BIOS and kernel driver.

Enabling or disabling TSO doesn't make a difference, the TX timeouts
still happen. The 2.6.15 kernel driver or the sf.net 6.3.9 driver also
don't make a difference.


HTH,

Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
| Data lost? Stay calm and contact Harddisk-recovery.com


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click

Re: Transmit timeout with E1000

[Erik Mouw]

On Wed, Jan 11, 2006 at 01:59:46PM +0100, Erik Mouw wrote:
> On Tue, Jan 10, 2006 at 09:46:29AM -0800, Jesse Brandeburg wrote:
> > sorry to hear you're having a problem, and cool, thanks for the test,
> > we'll have to try it here.  We've classically had problems reproducing the
> > athlon based hangs.
> 
> Athlon based or Athlon-on-VIA-KT400 based? We have an E1000 dual
> interface server adapter on a dual Athlon with AMD 762 chipset running
> fine, and also the same kind of adapter on a dual Athlon64 with
> AMD-8111 chipset running fine.

FWIW: I just found out that we have a desktop with similar hardware
that doesn't have TX timeouts.

Same mainboard (Asrock K7VT4APro), same AMD Athlon XP 2000+ CPU (though
it appears to be underclocked at 1250 MHz instead of 1666 MHz, probably
a jumper problem), different E1000 adapter (8086:1076 vs. 8086:107c),
different kernel version (2.4.24 vs. 2.6.15).


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
| Data lost? Stay calm and contact Harddisk-recovery.com


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click

Re: Transmit timeout with E1000

[Erik Mouw]

On Wed, Jan 11, 2006 at 02:22:08PM +0100, Erik Mouw wrote:
> On Wed, Jan 11, 2006 at 01:59:46PM +0100, Erik Mouw wrote:
> > On Tue, Jan 10, 2006 at 09:46:29AM -0800, Jesse Brandeburg wrote:
> > > sorry to hear you're having a problem, and cool, thanks for the test,
> > > we'll have to try it here.  We've classically had problems reproducing the
> > > athlon based hangs.
> > 
> > Athlon based or Athlon-on-VIA-KT400 based? We have an E1000 dual
> > interface server adapter on a dual Athlon with AMD 762 chipset running
> > fine, and also the same kind of adapter on a dual Athlon64 with
> > AMD-8111 chipset running fine.
> 
> FWIW: I just found out that we have a desktop with similar hardware
> that doesn't have TX timeouts.
> 
> Same mainboard (Asrock K7VT4APro), same AMD Athlon XP 2000+ CPU (though
> it appears to be underclocked at 1250 MHz instead of 1666 MHz, probably
> a jumper problem), different E1000 adapter (8086:1076 vs. 8086:107c),
> different kernel version (2.4.24 vs. 2.6.15).

I just lowered the CPU speed of my own workstation to 1250 MHz, but
that doesn't make a difference.

Here's an easy way to reproduce the problem:

  rsh otherhost dd if=/dev/zero > /dev/null

That usually triggers a transmit timeout in less than 30 seconds.

Data transfers the other direction (rsh otherhost dd of=/dev/null <
/dev/zero) don't trigger the problem.

The system only recovers after the Netdev watchdog found out that the
transmit timed out. However, the e1000 register dump starts about 4 to
5 seconds earlier: a possible workaround would be to trigger the
timeout code path as soon as the register dump starts.


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click

Re: Transmit timeout with E1000

[Rogier Wolff]

On Wed, Jan 11, 2006 at 02:43:49PM +0100, Erik Mouw wrote:
> The system only recovers after the Netdev watchdog found out that the
> transmit timed out. However, the e1000 register dump starts about 4 to
> 5 seconds earlier: a possible workaround would be to trigger the
> timeout code path as soon as the register dump starts.

Found a typo. 

	Roger. 


--- e1000_main.c.orig	2006-01-11 14:53:23.000000000 +0100
+++ e1000_main.c	2006-01-11 14:53:38.000000000 +0100
@@ -3449,7 +3449,7 @@
 	}
 
 	for (i = 0; i < E1000_MAX_INTR; i++)
-		if (unlikely(!adapter->clean_rx(adapter, adapter->rx_ring) &
+		if (unlikely(!adapter->clean_rx(adapter, adapter->rx_ring) &&
 		   !e1000_clean_tx_irq(adapter, adapter->tx_ring)))
 			break;
 


-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click

Re: Transmit timeout with E1000

[Eric Dumazet]

Rogier Wolff a écrit :
> On Wed, Jan 11, 2006 at 02:43:49PM +0100, Erik Mouw wrote:
>> The system only recovers after the Netdev watchdog found out that the
>> transmit timed out. However, the e1000 register dump starts about 4 to
>> 5 seconds earlier: a possible workaround would be to trigger the
>> timeout code path as soon as the register dump starts.
> 
> Found a typo. 
> 
> 	Roger. 
> 
> 
> --- e1000_main.c.orig	2006-01-11 14:53:23.000000000 +0100
> +++ e1000_main.c	2006-01-11 14:53:38.000000000 +0100
> @@ -3449,7 +3449,7 @@
>  	}
>  
>  	for (i = 0; i < E1000_MAX_INTR; i++)
> -		if (unlikely(!adapter->clean_rx(adapter, adapter->rx_ring) &
> +		if (unlikely(!adapter->clean_rx(adapter, adapter->rx_ring) &&
>  		   !e1000_clean_tx_irq(adapter, adapter->tx_ring)))
>  			break;
>  
> 
> 

I believe it's not a typo.

The intention is to call both clean_rx() and e1000_clean_tx_irq(), and break 
of the loop if both said : There was no job pending.

Eric


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_idv37&alloc_id\x16865&op=click

Re: Transmit timeout with E1000

[Rogier Wolff]

On Wed, Jan 11, 2006 at 03:11:47PM +0100, Eric Dumazet wrote:
> Rogier Wolff a écrit :
> >On Wed, Jan 11, 2006 at 02:43:49PM +0100, Erik Mouw wrote:
> >>The system only recovers after the Netdev watchdog found out that the
> >>transmit timed out. However, the e1000 register dump starts about 4 to
> >>5 seconds earlier: a possible workaround would be to trigger the
> >>timeout code path as soon as the register dump starts.
> >
> >Found a typo. 
> >
> >	Roger. 
> >
> >
> >--- e1000_main.c.orig	2006-01-11 14:53:23.000000000 +0100
> >+++ e1000_main.c	2006-01-11 14:53:38.000000000 +0100
> >@@ -3449,7 +3449,7 @@
> > 	}
> > 
> > 	for (i = 0; i < E1000_MAX_INTR; i++)
> >-		if (unlikely(!adapter->clean_rx(adapter, adapter->rx_ring) &
> >+		if (unlikely(!adapter->clean_rx(adapter, adapter->rx_ring) &&
> > 		   !e1000_clean_tx_irq(adapter, adapter->tx_ring)))
> > 			break;
> > 
> >
> >
> 
> I believe it's not a typo.
> 
> The intention is to call both clean_rx() and e1000_clean_tx_irq(), and 
> break of the loop if both said : There was no job pending.

Although (one of) the prototypes state(s) that it returns a boolean,
it is valid C to return the number of items pending.

And if one reports "2 more pending" and the other reports "1 more
pending", the "&" between the two becomes 2 & 1 => 0 / FALSE, while 2
&& 1 => TRUE.

I consider this a low prio typo, not likely to be a bug "right now",
but it is inviting someone to make it into a bug later on.

	Roger. 

-- 
+-- Rogier Wolff -- www.harddisk-recovery.nl -- 0800 220 20 20 --
| Files foetsie, bestanden kwijt, alle data weg?!
| Blijf kalm en neem contact op met Harddisk-recovery.nl!


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click

Re: Transmit timeout with E1000

[Rogier Wolff]

On Wed, Jan 11, 2006 at 03:48:11PM +0100, Rogier Wolff wrote:
> On Wed, Jan 11, 2006 at 03:11:47PM +0100, Eric Dumazet wrote:
> > Rogier Wolff a écrit :
> > >On Wed, Jan 11, 2006 at 02:43:49PM +0100, Erik Mouw wrote:
> > >>The system only recovers after the Netdev watchdog found out that the
> > >>transmit timed out. However, the e1000 register dump starts about 4 to
> > >>5 seconds earlier: a possible workaround would be to trigger the
> > >>timeout code path as soon as the register dump starts.
> > >
> > >Found a typo. 
> > >
> > >	Roger. 
> > >
> > >
> > >--- e1000_main.c.orig	2006-01-11 14:53:23.000000000 +0100
> > >+++ e1000_main.c	2006-01-11 14:53:38.000000000 +0100
> > >@@ -3449,7 +3449,7 @@
> > > 	}
> > > 
> > > 	for (i = 0; i < E1000_MAX_INTR; i++)
> > >-		if (unlikely(!adapter->clean_rx(adapter, adapter->rx_ring) &
> > >+		if (unlikely(!adapter->clean_rx(adapter, adapter->rx_ring) &&
> > > 		   !e1000_clean_tx_irq(adapter, adapter->tx_ring)))
> > > 			break;
> > > 
> > >
> > >
> > 
> > I believe it's not a typo.
> > 
> > The intention is to call both clean_rx() and e1000_clean_tx_irq(), and 
> > break of the loop if both said : There was no job pending.
> 
> Although (one of) the prototypes state(s) that it returns a boolean,
> it is valid C to return the number of items pending.
> 
> And if one reports "2 more pending" and the other reports "1 more
> pending", the "&" between the two becomes 2 & 1 => 0 / FALSE, while 2
> && 1 => TRUE.
> 
> I consider this a low prio typo, not likely to be a bug "right now",
> but it is inviting someone to make it into a bug later on.

Oops. The line itself has "logical NOT" operators. So indeed it is NOW
not a bug, and slighlty less likely to become a bug later on, but
still bad programming practise.

	Roger. 

-- 
+-- Rogier Wolff -- www.harddisk-recovery.nl -- 0800 220 20 20 --
| Files foetsie, bestanden kwijt, alle data weg?!
| Blijf kalm en neem contact op met Harddisk-recovery.nl!


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click