Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Peter Zijlstra]

(netdev CC'ed)

On Tue, 2008-11-11 at 03:48 -0600, Roger Heflin wrote:
> I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
> extra modules, tg3 Gbit networking.   I have not yet tested
> earlier kernels to see if this has been around for a while.

How do more recent kernels do?

> So far I have had this error happen 5 times (MTBF is maybe
> 12 hours), 4 of the 5 times resulted in the networking being
> broken, one time things came back by itself without a reboot,
> I believe in this case the hang was traffic coming into the
> machine vs the other times going out of the machine.
> 
> Unloading all of the network modules and reloading them did
> not correct the problem.
> 
> Searching google finds a couple of other people getting the
> same error but they have a different network chipset (e1000
> and a rt811C chipset), which makes me thing that there is
> something interacting bad with the network.    Or does this
> error truly mean that the network chipset for some unknown reason
> locked itself up?
> 
> http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
> http://article.gmane.org/gmane.linux.network/110238
> 
> The changes I made recently were to upgrade my MB (old
> was E100 on a 100Mbit network,new is tg3 on a Gbit network,
> cpu and memory are the same, MB chipset is a intel 955
> chipset vs the old being a intel 915 chipset).
> 
> Autoneg is turned on all around, the GBit switch is a
> 8-port Dlink switch.   The network seems to otherwise be working
> correctly.
> 
> I did test the network under decent load and the error did not
> appear to be any more likely under load, and typically the network
> is under very light load 2-3MB/second.
> 
> The machine originally had 2 HT CPU's showing up, I turned off HT
> so that only one cpu was showing, but this did not change the error.
> 
> I am first turning off all offload capabilities on tg3 and going
> to see if that changes anything.
> 
> The next thing I am going to be doing is to turn of GB capability
> on the networking and see if that does anything.
> 
> I also have a second tg3 port that is slightly different, so I may
> try that eventually.
> 
> What else can I look at?
> 
> 
> 
> 
> 
> 
> Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
> Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
> dev_watchdog+0xfe/0x17e()
> Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
> Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs 
> w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor 
> async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb 
> wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner 
> mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa 
> cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat 
> i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 
> videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support 
> v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc 
> videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc 
> parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero 
> dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 
> jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
> Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
> Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
> Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
> Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
> Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
> Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
> Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
> Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
> Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
> Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
> Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
> Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0 
> [uhci_hcd]
> Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
> Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
> Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
> Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
> Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
> Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
> Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
> Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
> Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
> Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
> Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
> Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
> Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
> Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Roger Heflin]

Peter Zijlstra wrote:
> (netdev CC'ed)
> 
> On Tue, 2008-11-11 at 03:48 -0600, Roger Heflin wrote:
>> I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
>> extra modules, tg3 Gbit networking.   I have not yet tested
>> earlier kernels to see if this has been around for a while.
> 
> How do more recent kernels do?

I did not try more recent kernels, more testing seems to indicate
that at the very least it the bug depends on a certain version of
either tg3 and/or firmware to happen, as my second tg3 port does not
have it happen.    More about this below.

> 
>> So far I have had this error happen 5 times (MTBF is maybe
>> 12 hours), 4 of the 5 times resulted in the networking being
>> broken, one time things came back by itself without a reboot,
>> I believe in this case the hang was traffic coming into the
>> machine vs the other times going out of the machine.
>>
>> Unloading all of the network modules and reloading them did
>> not correct the problem.
>>
>> Searching google finds a couple of other people getting the
>> same error but they have a different network chipset (e1000
>> and a rt811C chipset), which makes me thing that there is
>> something interacting bad with the network.    Or does this
>> error truly mean that the network chipset for some unknown reason
>> locked itself up?
>>
>> http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
>> http://article.gmane.org/gmane.linux.network/110238
>>
>> The changes I made recently were to upgrade my MB (old
>> was E100 on a 100Mbit network,new is tg3 on a Gbit network,
>> cpu and memory are the same, MB chipset is a intel 955
>> chipset vs the old being a intel 915 chipset).
>>
>> Autoneg is turned on all around, the GBit switch is a
>> 8-port Dlink switch.   The network seems to otherwise be working
>> correctly.
>>
>> I did test the network under decent load and the error did not
>> appear to be any more likely under load, and typically the network
>> is under very light load 2-3MB/second.
>>
>> The machine originally had 2 HT CPU's showing up, I turned off HT
>> so that only one cpu was showing, but this did not change the error.
>>
>> I am first turning off all offload capabilities on tg3 and going
>> to see if that changes anything.

This made no difference in the error.

>>
>> The next thing I am going to be doing is to turn of GB capability
>> on the networking and see if that does anything.

Did not try.

>>
>> I also have a second tg3 port that is slightly different, so I may
>> try that eventually.

I tried this, and with the second port I don't appear to be getting
the error.   The first port is a 5789-v3.29a and the second port is a
5788-v3.04, I know the first port is faster (pcie-x1) than the second
port (pci bus-built-in, unknown exact connection).   The second port
will sustain about 50MB/second, were as the first port will get
 >90MB/second.

It seems to me to likely be the firmware on the tg3, and it would seem
unlikely that the driver could do anything more than work around the
issue that is in the firmware, and currently my system works on the
second port, and the second port is fast enough for my needs.

If someone else runs into this issue, since I have 2 ports I would be
able to do some testing on it, right now my first port is locked up, and
the machine is running fine on the second port.

lspci -vvv for the first (bad) port:

02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit 
Ethernet PCI Express (rev 11)
         Subsystem: Foxconn International, Inc. Unknown device 0cc1
         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B-
         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR-
         Latency: 0, Cache Line Size: 32 bytes
         Interrupt: pin A routed to IRQ 19
         Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K]
         Expansion ROM at <ignored> [disabled]
         Capabilities: [48] Power Management version 2
                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot+,D3cold+)
                 Status: D3 PME-Enable- DSel=0 DScale=1 PME-
         Capabilities: [50] Vital Product Data
         Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 
Enable-
                 Address: 0101b8102a0f7b0c  Data: f21e
         Capabilities: [d0] Express Endpoint IRQ 0
                 Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
                 Device: Latency L0s <4us, L1 unlimited
                 Device: AtnBtn- AtnInd- PwrInd-
                 Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
                 Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                 Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
                 Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
                 Link: Latency L0s <2us, L1 <64us
                 Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
                 Link: Speed 2.5Gb/s, Width x1
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [13c] Virtual Channel


>>
>> Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
>> Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
>> dev_watchdog+0xfe/0x17e()
>> Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
>> Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs 
>> w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor 
>> async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb 
>> wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner 
>> mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa 
>> cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat 
>> i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 
>> videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support 
>> v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc 
>> videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc 
>> parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero 
>> dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 
>> jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
>> Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
>> Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
>> Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
>> Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
>> Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
>> Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
>> Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
>> Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
>> Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
>> Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
>> Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
>> Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0 
>> [uhci_hcd]
>> Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
>> Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
>> Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
>> Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
>> Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
>> Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
>> Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
>> Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
>> Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
>> Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
>> Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
>> Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
>> Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
>> Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

Hello,

On Fri, Nov 14, 2008 at 10:01:46PM -0600, Roger Heflin wrote:
> Peter Zijlstra wrote:
> >(netdev CC'ed)
> >
> >On Tue, 2008-11-11 at 03:48 -0600, Roger Heflin wrote:
> >>I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
> >>extra modules, tg3 Gbit networking.   I have not yet tested
> >>earlier kernels to see if this has been around for a while.
> >
> >How do more recent kernels do?
> 
> I did not try more recent kernels, more testing seems to indicate
> that at the very least it the bug depends on a certain version of
> either tg3 and/or firmware to happen, as my second tg3 port does not
> have it happen.    More about this below.

I got the exact same one yesterday and wanted to fire a bug report,
so yours is saving me some time :-)

It's on my notebook (HP nc8000, Pentium-M, no SMP, no HT). It was
connected to a 100 Mbps switch and was sending nearly 100 Mbps of
traffic. It had auto-negociated 100-full. I noticed long pauses
on the other side, finally realizing that my notebook was ill.
It's on 2.6.27.7-rc1. I have never had any such problem on this
notebook with any earlier kernel up to 2.6.25.X. I've never run
2.6.26 on it.

My tg3 is just PCI-based, no PCIe in this beast. I can send more
info when I turn it on. I don't think that the tg3 driver changes
often, so most likely digging through the changes between 2.6.25
and 2.6.27 should not take much time. I just don't know if I can
reliably reproduce the issue right now.

> >>So far I have had this error happen 5 times (MTBF is maybe
> >>12 hours), 4 of the 5 times resulted in the networking being
> >>broken, one time things came back by itself without a reboot,
> >>I believe in this case the hang was traffic coming into the
> >>machine vs the other times going out of the machine.
> >>
> >>Unloading all of the network modules and reloading them did
> >>not correct the problem.
> >>
> >>Searching google finds a couple of other people getting the
> >>same error but they have a different network chipset (e1000
> >>and a rt811C chipset), which makes me thing that there is
> >>something interacting bad with the network.    Or does this
> >>error truly mean that the network chipset for some unknown reason
> >>locked itself up?
> >>
> >>http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
> >>http://article.gmane.org/gmane.linux.network/110238
> >>
> >>The changes I made recently were to upgrade my MB (old
> >>was E100 on a 100Mbit network,new is tg3 on a Gbit network,
> >>cpu and memory are the same, MB chipset is a intel 955
> >>chipset vs the old being a intel 915 chipset).
> >>
> >>Autoneg is turned on all around, the GBit switch is a
> >>8-port Dlink switch.   The network seems to otherwise be working
> >>correctly.
> >>
> >>I did test the network under decent load and the error did not
> >>appear to be any more likely under load, and typically the network
> >>is under very light load 2-3MB/second.
> >>
> >>The machine originally had 2 HT CPU's showing up, I turned off HT
> >>so that only one cpu was showing, but this did not change the error.
> >>
> >>I am first turning off all offload capabilities on tg3 and going
> >>to see if that changes anything.
> 
> This made no difference in the error.
> 
> >>
> >>The next thing I am going to be doing is to turn of GB capability
> >>on the networking and see if that does anything.
> 
> Did not try.
> 
> >>
> >>I also have a second tg3 port that is slightly different, so I may
> >>try that eventually.
> 
> I tried this, and with the second port I don't appear to be getting
> the error.   The first port is a 5789-v3.29a and the second port is a
> 5788-v3.04, I know the first port is faster (pcie-x1) than the second
> port (pci bus-built-in, unknown exact connection).   The second port
> will sustain about 50MB/second, were as the first port will get
> >90MB/second.
> 
> It seems to me to likely be the firmware on the tg3, and it would seem
> unlikely that the driver could do anything more than work around the
> issue that is in the firmware, and currently my system works on the
> second port, and the second port is fast enough for my needs.
> 
> If someone else runs into this issue, since I have 2 ports I would be
> able to do some testing on it, right now my first port is locked up, and
> the machine is running fine on the second port.
> 
> lspci -vvv for the first (bad) port:
> 
> 02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit 
> Ethernet PCI Express (rev 11)
>         Subsystem: Foxconn International, Inc. Unknown device 0cc1
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
>         <TAbort- <MAbort- >SERR- <PERR-
>         Latency: 0, Cache Line Size: 32 bytes
>         Interrupt: pin A routed to IRQ 19
>         Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K]
>         Expansion ROM at <ignored> [disabled]
>         Capabilities: [48] Power Management version 2
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>                 Status: D3 PME-Enable- DSel=0 DScale=1 PME-
>         Capabilities: [50] Vital Product Data
>         Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ 
>         Queue=0/3 Enable-
>                 Address: 0101b8102a0f7b0c  Data: f21e
>         Capabilities: [d0] Express Endpoint IRQ 0
>                 Device: Supported: MaxPayload 128 bytes, PhantFunc 0, 
>                 ExtTag+
>                 Device: Latency L0s <4us, L1 unlimited
>                 Device: AtnBtn- AtnInd- PwrInd-
>                 Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
>                 Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                 Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
>                 Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
>                 Link: Latency L0s <2us, L1 <64us
>                 Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
>                 Link: Speed 2.5Gb/s, Width x1
>         Capabilities: [100] Advanced Error Reporting
>         Capabilities: [13c] Virtual Channel
> 
> 
> >>
> >>Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
> >>Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
> >>dev_watchdog+0xfe/0x17e()
> >>Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit 
> >>timed out
> >>Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss 
> >>exportfs w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs 
> >>raid456 async_xor async_memcpy async_tx xor video output sbs sbshc 
> >>battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple 
> >>tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel 
> >>snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 
> >>snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit 
> >>cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb 
> >>i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common 
> >>snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc 
> >>videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm 
> >>parport_pc parport snd_timer snd soundcore button snd_page_alloc sg 
> >>dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic 
> >>libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last 
> >>unloaded: eeprom]
> >>Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 
> >>2.6.27.5 #2
> >>Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
> >>Nov 11 00:44:39 computer kernel:  [<c05663a4>] 
> >>usb_hcd_submit_urb+0x75c/0x811
> >>Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
> >>Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
> >>Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
> >>Nov 11 00:44:39 computer kernel:  [<c041cbe3>] 
> >>find_busiest_group+0x23e/0x672
> >>Nov 11 00:44:39 computer kernel:  [<c0439d1e>] 
> >>clocksource_get_next+0x39/0x3f
> >>Nov 11 00:44:39 computer kernel:  [<c0438e51>] 
> >>update_wall_time+0x567/0x70c
> >>Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
> >>Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
> >>Nov 11 00:44:39 computer kernel:  [<f8829a83>] 
> >>uhci_scan_schedule+0x11b/0x6b0 [uhci_hcd]
> >>Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
> >>Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
> >>Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
> >>Nov 11 00:44:39 computer kernel:  [<c05654ae>] 
> >>usb_hcd_poll_rh_status+0x12b/0x133
> >>Nov 11 00:44:39 computer kernel:  [<c043bca8>] 
> >>tick_dev_program_event+0x1e/0x81
> >>Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
> >>Nov 11 00:44:39 computer kernel:  [<c042c2b4>] 
> >>run_timer_softirq+0x10e/0x167
> >>Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
> >>Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
> >>Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
> >>Nov 11 00:44:39 computer kernel:  [<c0412939>] 
> >>smp_apic_timer_interrupt+0x6e/0x79
> >>Nov 11 00:44:39 computer kernel:  [<c040431c>] 
> >>apic_timer_interrupt+0x28/0x30
> >>Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
> >>Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5

Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Matt Carlson]

On Mon, Nov 17, 2008 at 10:50:07PM -0800, Willy Tarreau wrote:
> Hello,
> 
> On Fri, Nov 14, 2008 at 10:01:46PM -0600, Roger Heflin wrote:
> > Peter Zijlstra wrote:
> > >(netdev CC'ed)
> > >
> > >On Tue, 2008-11-11 at 03:48 -0600, Roger Heflin wrote:
> > >>I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
> > >>extra modules, tg3 Gbit networking.   I have not yet tested
> > >>earlier kernels to see if this has been around for a while.
> > >
> > >How do more recent kernels do?
> > 
> > I did not try more recent kernels, more testing seems to indicate
> > that at the very least it the bug depends on a certain version of
> > either tg3 and/or firmware to happen, as my second tg3 port does not
> > have it happen.    More about this below.
> 
> I got the exact same one yesterday and wanted to fire a bug report,
> so yours is saving me some time :-)
> 
> It's on my notebook (HP nc8000, Pentium-M, no SMP, no HT). It was
> connected to a 100 Mbps switch and was sending nearly 100 Mbps of
> traffic. It had auto-negociated 100-full. I noticed long pauses
> on the other side, finally realizing that my notebook was ill.
> It's on 2.6.27.7-rc1. I have never had any such problem on this
> notebook with any earlier kernel up to 2.6.25.X. I've never run
> 2.6.26 on it.
> 
> My tg3 is just PCI-based, no PCIe in this beast. I can send more
> info when I turn it on. I don't think that the tg3 driver changes
> often, so most likely digging through the changes between 2.6.25
> and 2.6.27 should not take much time. I just don't know if I can
> reliably reproduce the issue right now.

Willy, this problem description sounds a little different than the
original report.  There was a bug where the driver would wait 2.5
seconds for a firmware event that would never get serviced.  That
fix has already landed in the 2.6.27 tree though.

I glanced over the changes between 2.6.25 and 2.6.27.6.  There are quite
a few changes related to phylib support for an upcoming device, but not
so many changes that affect older devices.  What device are you using?

> > >>So far I have had this error happen 5 times (MTBF is maybe
> > >>12 hours), 4 of the 5 times resulted in the networking being
> > >>broken, one time things came back by itself without a reboot,
> > >>I believe in this case the hang was traffic coming into the
> > >>machine vs the other times going out of the machine.
> > >>
> > >>Unloading all of the network modules and reloading them did
> > >>not correct the problem.
> > >>
> > >>Searching google finds a couple of other people getting the
> > >>same error but they have a different network chipset (e1000
> > >>and a rt811C chipset), which makes me thing that there is
> > >>something interacting bad with the network.    Or does this
> > >>error truly mean that the network chipset for some unknown reason
> > >>locked itself up?
> > >>
> > >>http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
> > >>http://article.gmane.org/gmane.linux.network/110238
> > >>
> > >>The changes I made recently were to upgrade my MB (old
> > >>was E100 on a 100Mbit network,new is tg3 on a Gbit network,
> > >>cpu and memory are the same, MB chipset is a intel 955
> > >>chipset vs the old being a intel 915 chipset).
> > >>
> > >>Autoneg is turned on all around, the GBit switch is a
> > >>8-port Dlink switch.   The network seems to otherwise be working
> > >>correctly.
> > >>
> > >>I did test the network under decent load and the error did not
> > >>appear to be any more likely under load, and typically the network
> > >>is under very light load 2-3MB/second.
> > >>
> > >>The machine originally had 2 HT CPU's showing up, I turned off HT
> > >>so that only one cpu was showing, but this did not change the error.
> > >>
> > >>I am first turning off all offload capabilities on tg3 and going
> > >>to see if that changes anything.
> > 
> > This made no difference in the error.
> > 
> > >>
> > >>The next thing I am going to be doing is to turn of GB capability
> > >>on the networking and see if that does anything.
> > 
> > Did not try.
> > 
> > >>
> > >>I also have a second tg3 port that is slightly different, so I may
> > >>try that eventually.
> > 
> > I tried this, and with the second port I don't appear to be getting
> > the error.   The first port is a 5789-v3.29a and the second port is a
> > 5788-v3.04, I know the first port is faster (pcie-x1) than the second
> > port (pci bus-built-in, unknown exact connection).   The second port
> > will sustain about 50MB/second, were as the first port will get
> > >90MB/second.
> > 
> > It seems to me to likely be the firmware on the tg3, and it would seem
> > unlikely that the driver could do anything more than work around the
> > issue that is in the firmware, and currently my system works on the
> > second port, and the second port is fast enough for my needs.
> > 
> > If someone else runs into this issue, since I have 2 ports I would be
> > able to do some testing on it, right now my first port is locked up, and
> > the machine is running fine on the second port.
> > 
> > lspci -vvv for the first (bad) port:
> > 
> > 02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit 
> > Ethernet PCI Express (rev 11)
> >         Subsystem: Foxconn International, Inc. Unknown device 0cc1
> >         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> > Stepping- SERR- FastB2B-
> >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> >         <TAbort- <MAbort- >SERR- <PERR-
> >         Latency: 0, Cache Line Size: 32 bytes
> >         Interrupt: pin A routed to IRQ 19
> >         Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K]
> >         Expansion ROM at <ignored> [disabled]
> >         Capabilities: [48] Power Management version 2
> >                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
> > PME(D0-,D1-,D2-,D3hot+,D3cold+)
> >                 Status: D3 PME-Enable- DSel=0 DScale=1 PME-
> >         Capabilities: [50] Vital Product Data
> >         Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ 
> >         Queue=0/3 Enable-
> >                 Address: 0101b8102a0f7b0c  Data: f21e
> >         Capabilities: [d0] Express Endpoint IRQ 0
> >                 Device: Supported: MaxPayload 128 bytes, PhantFunc 0, 
> >                 ExtTag+
> >                 Device: Latency L0s <4us, L1 unlimited
> >                 Device: AtnBtn- AtnInd- PwrInd-
> >                 Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
> >                 Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> >                 Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
> >                 Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
> >                 Link: Latency L0s <2us, L1 <64us
> >                 Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
> >                 Link: Speed 2.5Gb/s, Width x1
> >         Capabilities: [100] Advanced Error Reporting
> >         Capabilities: [13c] Virtual Channel
> > 
> > 
> > >>
> > >>Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
> > >>Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
> > >>dev_watchdog+0xfe/0x17e()
> > >>Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit 
> > >>timed out
> > >>Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss 
> > >>exportfs w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs 
> > >>raid456 async_xor async_memcpy async_tx xor video output sbs sbshc 
> > >>battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple 
> > >>tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel 
> > >>snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 
> > >>snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit 
> > >>cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb 
> > >>i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common 
> > >>snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc 
> > >>videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm 
> > >>parport_pc parport snd_timer snd soundcore button snd_page_alloc sg 
> > >>dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic 
> > >>libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last 
> > >>unloaded: eeprom]
> > >>Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 
> > >>2.6.27.5 #2
> > >>Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
> > >>Nov 11 00:44:39 computer kernel:  [<c05663a4>] 
> > >>usb_hcd_submit_urb+0x75c/0x811
> > >>Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
> > >>Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
> > >>Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
> > >>Nov 11 00:44:39 computer kernel:  [<c041cbe3>] 
> > >>find_busiest_group+0x23e/0x672
> > >>Nov 11 00:44:39 computer kernel:  [<c0439d1e>] 
> > >>clocksource_get_next+0x39/0x3f
> > >>Nov 11 00:44:39 computer kernel:  [<c0438e51>] 
> > >>update_wall_time+0x567/0x70c
> > >>Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
> > >>Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
> > >>Nov 11 00:44:39 computer kernel:  [<f8829a83>] 
> > >>uhci_scan_schedule+0x11b/0x6b0 [uhci_hcd]
> > >>Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
> > >>Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
> > >>Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
> > >>Nov 11 00:44:39 computer kernel:  [<c05654ae>] 
> > >>usb_hcd_poll_rh_status+0x12b/0x133
> > >>Nov 11 00:44:39 computer kernel:  [<c043bca8>] 
> > >>tick_dev_program_event+0x1e/0x81
> > >>Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
> > >>Nov 11 00:44:39 computer kernel:  [<c042c2b4>] 
> > >>run_timer_softirq+0x10e/0x167
> > >>Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
> > >>Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
> > >>Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
> > >>Nov 11 00:44:39 computer kernel:  [<c0412939>] 
> > >>smp_apic_timer_interrupt+0x6e/0x79
> > >>Nov 11 00:44:39 computer kernel:  [<c040431c>] 
> > >>apic_timer_interrupt+0x28/0x30
> > >>Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
> > >>Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5
> 
> Willy
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

Hello Matt,

On Wed, Nov 19, 2008 at 07:11:01PM -0800, Matt Carlson wrote:
> > My tg3 is just PCI-based, no PCIe in this beast. I can send more
> > info when I turn it on. I don't think that the tg3 driver changes
> > often, so most likely digging through the changes between 2.6.25
> > and 2.6.27 should not take much time. I just don't know if I can
> > reliably reproduce the issue right now.
> 
> Willy, this problem description sounds a little different than the
> original report.  There was a bug where the driver would wait 2.5
> seconds for a firmware event that would never get serviced.  That
> fix has already landed in the 2.6.27 tree though.
> 
> I glanced over the changes between 2.6.25 and 2.6.27.6.  There are quite
> a few changes related to phylib support for an upcoming device, but not
> so many changes that affect older devices.  What device are you using?

I think it's a 5704, but I will check this this morning when I'm at
work. I also want to try to reliably reproduce the problem. After
that, I see only 29 patches which differ from the two kernels, it
should be pretty easy to spot the culprit.

If you think it's a different bug than original report (though I
really thought it was the same), I'll post my findings in a separate
thread not to mix investigations.

Thanks,
Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Matt Carlson]

On Wed, Nov 19, 2008 at 09:37:47PM -0800, Willy Tarreau wrote:
> Hello Matt,
> 
> On Wed, Nov 19, 2008 at 07:11:01PM -0800, Matt Carlson wrote:
> > > My tg3 is just PCI-based, no PCIe in this beast. I can send more
> > > info when I turn it on. I don't think that the tg3 driver changes
> > > often, so most likely digging through the changes between 2.6.25
> > > and 2.6.27 should not take much time. I just don't know if I can
> > > reliably reproduce the issue right now.
> > 
> > Willy, this problem description sounds a little different than the
> > original report.  There was a bug where the driver would wait 2.5
> > seconds for a firmware event that would never get serviced.  That
> > fix has already landed in the 2.6.27 tree though.
> > 
> > I glanced over the changes between 2.6.25 and 2.6.27.6.  There are quite
> > a few changes related to phylib support for an upcoming device, but not
> > so many changes that affect older devices.  What device are you using?
> 
> I think it's a 5704, but I will check this this morning when I'm at
> work. I also want to try to reliably reproduce the problem. After
> that, I see only 29 patches which differ from the two kernels, it
> should be pretty easy to spot the culprit.

O.K.  Let me know how it goes.

Could we clarify something though?  In your previous email, you said you
didn't have any problems on pre-2.6.25 kernels.  I'm wondering if the
problem goes back further than 2.6.25.  From 2.6.24 to 2.6.25, there was a
significant set of flow control changes that took place.  I suspect that
might have something to do with Roger's problem, and it may have
something to do with your problem too.  So, is it true that 2.6.25 works
for you?  If not, can you try disabling flow control and see if that
helps?

> If you think it's a different bug than original report (though I
> really thought it was the same), I'll post my findings in a separate
> thread not to mix investigations.

Right now, I think it is premature to say, so let's continue as if they
were the same problem.  We can always break it out into a separate
discussion later.

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

On Thu, Nov 20, 2008 at 10:43:10AM -0800, Matt Carlson wrote:
> On Wed, Nov 19, 2008 at 09:37:47PM -0800, Willy Tarreau wrote:
> > Hello Matt,
> > 
> > On Wed, Nov 19, 2008 at 07:11:01PM -0800, Matt Carlson wrote:
> > > > My tg3 is just PCI-based, no PCIe in this beast. I can send more
> > > > info when I turn it on. I don't think that the tg3 driver changes
> > > > often, so most likely digging through the changes between 2.6.25
> > > > and 2.6.27 should not take much time. I just don't know if I can
> > > > reliably reproduce the issue right now.
> > > 
> > > Willy, this problem description sounds a little different than the
> > > original report.  There was a bug where the driver would wait 2.5
> > > seconds for a firmware event that would never get serviced.  That
> > > fix has already landed in the 2.6.27 tree though.
> > > 
> > > I glanced over the changes between 2.6.25 and 2.6.27.6.  There are quite
> > > a few changes related to phylib support for an upcoming device, but not
> > > so many changes that affect older devices.  What device are you using?
> > 
> > I think it's a 5704, but I will check this this morning when I'm at
> > work. I also want to try to reliably reproduce the problem. After
> > that, I see only 29 patches which differ from the two kernels, it
> > should be pretty easy to spot the culprit.
> 
> O.K.  Let me know how it goes.

Today, with the notebook connected to a gig switch, I could not reproduce
the problem, even after one hour of approximately the same workload. I'll
retry with the original 100 Mbps switch on monday.

> Could we clarify something though?  In your previous email, you said you
> didn't have any problems on pre-2.6.25 kernels.  I'm wondering if the
> problem goes back further than 2.6.25.  From 2.6.24 to 2.6.25, there was a
> significant set of flow control changes that took place.  I suspect that
> might have something to do with Roger's problem, and it may have
> something to do with your problem too.  So, is it true that 2.6.25 works
> for you?  If not, can you try disabling flow control and see if that
> helps?

It works fine till 2.6.25.18 included. I have not tried any 2.6.26 on this
machine. Just 2.6.27.7-rc1 (with a few patches, none of which affect tg3).
My goal is 1) to reproduce with the exact same kernel on the original switch,
2) confirm that 2.6.25.18 does not exhibit the problem on the same switch,
3) to switch to plain 2.6.27.x to ease troubleshooting, and 4) to find which
of the 29 tg3 patches between 2.6.25.18 and 2.6.27.x brings the issue.

> > If you think it's a different bug than original report (though I
> > really thought it was the same), I'll post my findings in a separate
> > thread not to mix investigations.
> 
> Right now, I think it is premature to say, so let's continue as if they
> were the same problem.  We can always break it out into a separate
> discussion later.

OK that's fine for me.

Thanks Matt!
Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Matt Carlson]

On Thu, Nov 20, 2008 at 01:26:37PM -0800, Willy Tarreau wrote:
> On Thu, Nov 20, 2008 at 10:43:10AM -0800, Matt Carlson wrote:
> > On Wed, Nov 19, 2008 at 09:37:47PM -0800, Willy Tarreau wrote:
> > > Hello Matt,
> > > 
> > > On Wed, Nov 19, 2008 at 07:11:01PM -0800, Matt Carlson wrote:
> > > > > My tg3 is just PCI-based, no PCIe in this beast. I can send more
> > > > > info when I turn it on. I don't think that the tg3 driver changes
> > > > > often, so most likely digging through the changes between 2.6.25
> > > > > and 2.6.27 should not take much time. I just don't know if I can
> > > > > reliably reproduce the issue right now.
> > > > 
> > > > Willy, this problem description sounds a little different than the
> > > > original report.  There was a bug where the driver would wait 2.5
> > > > seconds for a firmware event that would never get serviced.  That
> > > > fix has already landed in the 2.6.27 tree though.
> > > > 
> > > > I glanced over the changes between 2.6.25 and 2.6.27.6.  There are quite
> > > > a few changes related to phylib support for an upcoming device, but not
> > > > so many changes that affect older devices.  What device are you using?
> > > 
> > > I think it's a 5704, but I will check this this morning when I'm at
> > > work. I also want to try to reliably reproduce the problem. After
> > > that, I see only 29 patches which differ from the two kernels, it
> > > should be pretty easy to spot the culprit.
> > 
> > O.K.  Let me know how it goes.
> 
> Today, with the notebook connected to a gig switch, I could not reproduce
> the problem, even after one hour of approximately the same workload. I'll
> retry with the original 100 Mbps switch on monday.

Was it really a 5704 that you were using?

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

Hi Matt,

On Thu, Nov 20, 2008 at 01:53:18PM -0800, Matt Carlson wrote:
> > Today, with the notebook connected to a gig switch, I could not reproduce
> > the problem, even after one hour of approximately the same workload. I'll
> > retry with the original 100 Mbps switch on monday.
> 
> Was it really a 5704 that you were using?

No, I was wrong, finally it's a 5705 :

tg3.c:v3.94 (August 14, 2008)
ACPI: PCI Interrupt Link [C0C5] enabled at IRQ 11
tg3 0000:02:0e.0: PCI INT A -> Link[C0C5] -> GSI 11 (level, low) -> IRQ 11
eth0: Tigon3 [partno(BMC5705mA3) rev 3003 PHY(5705)] (PCI:33MHz:32-bit) 10/100/1000Base-T Ethernet 00:0d:9d:91:ef:24
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TSOcap[1]
eth0: dma_rwctrl[763f0000] dma_mask[64-bit]
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

Here's the lspci :

02:0e.0 Class 0200: 14e4:165e (rev 03)
        Subsystem: 103c:088c
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 64 (16000ns min), cache line size 10
        Interrupt: pin A routed to IRQ 11
        Region 0: Memory at 90000000 (64-bit, non-prefetchable) [size=64K]
        Expansion ROM at 5c000000 [disabled] [size=64K]
        Capabilities: [48] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] Vital Product Data
        Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
                Address: fbff71ffffbfebfc  Data: edff
00: e4 14 5e 16 06 00 b0 02 03 00 00 02 10 40 00 00
10: 04 00 00 90 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 07 00 00 00 3c 10 8c 08
30: 00 00 00 00 48 00 00 00 00 00 00 00 0b 01 40 00

Regards,
Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

Hi Matt,

On Thu, Nov 20, 2008 at 01:53:18PM -0800, Matt Carlson wrote:
> > Today, with the notebook connected to a gig switch, I could not reproduce
> > the problem, even after one hour of approximately the same workload. I'll
> > retry with the original 100 Mbps switch on monday.

fairly easier now with the same switch. I just have to transfer 100k objects
over HTTP via this switch to see the problem happen :

tg3: eth0: The system may be re-ordering memory-mapped I/O cycles to the network device, attempting to recover. Please report the problem to the driver maintainer and include system chipset information.
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

The switch is an el-cheapo D-Link 10/100. Note that this time I did not see
any warning. Maybe I did not wait long enough though. Here are some stats
and information I could collect :

root@wtap:/# ethtool -i eth0
driver: tg3
version: 3.94
firmware-version: 5705-v3.14
bus-info: 0000:02:0e.0

root@wtap:/# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Advertised auto-negotiation: Yes
        Speed: 100Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x000000ff (255)
        Link detected: yes

root@wtap:/# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off

root@wtap:/# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  on
RX:             on
TX:             on

root@wtap:/# netstat -ni
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   1500   0   61665      0      0      0   65133      0      0      0 BMRU
lo    16436   0  945938      0      0      0  945938      0      0      0 LRU

root@wtap:/# ethtool -S eth0
NIC statistics:
     rx_octets: 52461785
     rx_fragments: 0
     rx_ucast_packets: 61018
     rx_mcast_packets: 0
     rx_bcast_packets: 476
     rx_fcs_errors: 0
     rx_align_errors: 0
     rx_xon_pause_rcvd: 0
     rx_xoff_pause_rcvd: 0
     rx_mac_ctrl_rcvd: 0
     rx_xoff_entered: 0
     rx_frame_too_long_errors: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_in_length_errors: 0
     rx_out_length_errors: 0
     rx_64_or_less_octet_packets: 0
     rx_65_to_127_octet_packets: 0
     rx_128_to_255_octet_packets: 0
     rx_256_to_511_octet_packets: 0
     rx_512_to_1023_octet_packets: 0
     rx_1024_to_1522_octet_packets: 0
     rx_1523_to_2047_octet_packets: 0
     rx_2048_to_4095_octet_packets: 0
     rx_4096_to_8191_octet_packets: 0
     rx_8192_to_9022_octet_packets: 0
     tx_octets: 54658258
     tx_collisions: 0
     tx_xon_sent: 0
     tx_xoff_sent: 0
     tx_flow_control: 0
     tx_mac_errors: 0
     tx_single_collisions: 0
     tx_mult_collisions: 0
     tx_deferred: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     tx_collide_2times: 0
     tx_collide_3times: 0
     tx_collide_4times: 0
     tx_collide_5times: 0
     tx_collide_6times: 0
     tx_collide_7times: 0
     tx_collide_8times: 0
     tx_collide_9times: 0
     tx_collide_10times: 0
     tx_collide_11times: 0
     tx_collide_12times: 0
     tx_collide_13times: 0
     tx_collide_14times: 0
     tx_collide_15times: 0
     tx_ucast_packets: 65033
     tx_mcast_packets: 0
     tx_bcast_packets: 0
     tx_carrier_sense_errors: 0
     tx_discards: 0
     tx_errors: 0
     dma_writeq_full: 0
     dma_write_prioq_full: 0
     rxbds_empty: 0
     rx_discards: 0
     rx_errors: 0
     rx_threshold_hit: 0
     dma_readq_full: 0
     dma_read_prioq_full: 0
     tx_comp_queue_full: 0
     ring_set_send_prod_index: 0
     ring_status_update: 0
     nic_irqs: 0
     nic_avoided_irqs: 0
     nic_tx_threshold_hit: 0

I will try to go back to older kernels then bisect tg3 changes. I
cannot do this right now since I'm using my laptop for work ;-)

Regards,
Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

Hi Matt,

just a follow-up.

On Mon, Nov 24, 2008 at 02:27:44PM +0100, Willy Tarreau wrote:
> Hi Matt,
> 
> On Thu, Nov 20, 2008 at 01:53:18PM -0800, Matt Carlson wrote:
> > > Today, with the notebook connected to a gig switch, I could not reproduce
> > > the problem, even after one hour of approximately the same workload. I'll
> > > retry with the original 100 Mbps switch on monday.
> 
> fairly easier now with the same switch. I just have to transfer 100k objects
> over HTTP via this switch to see the problem happen :
> 
> tg3: eth0: The system may be re-ordering memory-mapped I/O cycles to the network device, attempting to recover. Please report the problem to the driver maintainer and include system chipset information.
> tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
> tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
> tg3: eth0: Link is down.
> tg3: eth0: Link is up at 100 Mbps, full duplex.
> tg3: eth0: Flow control is on for TX and on for RX.
> 
> The switch is an el-cheapo D-Link 10/100. Note that this time I did not see
> any warning. Maybe I did not wait long enough though.

Got it again, just had to be patient to fire a second test :

WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0x1a4/0x1b0()
NETDEV WATCHDOG: eth0 (tg3): transmit timed out
Modules linked in: nfs lockd sunrpc mtdblock mtd_blkdevs slram mtd xt_tcpudp x_tables usbhid usb_storage ehci_hcd uhci_hcd usbcore snd_pcm_oss snd_mixer_oss snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc tg3 libphy ide_cs yenta_socket rsrc_nonstatic [last unloaded: ip_tables]
Pid: 0, comm: swapper Not tainted 2.6.27-wt2-wtap #1
 [<b01254a7>] warn_slowpath+0x67/0x90
 [<b01741a9>] ? get_slab+0x9/0x70
 [<b03d21af>] ? pskb_copy+0x2f/0x160
 [<b03aa332>] ? input_defuzz_abs_event+0x12/0xa0
 [<b03aa574>] ? input_handle_event+0x14/0x2a0
 [<b03b3d76>] ? synaptics_process_packet+0x2b6/0x3d0
 [<b0108a48>] ? native_io_delay+0x8/0x40
 [<b02ab4c9>] ? strlen+0x9/0x20
 [<b02a961e>] ? strlcpy+0x1e/0x60
 [<b03dbfbc>] ? netdev_drivername+0x3c/0x40
 [<b03e7c84>] dev_watchdog+0x1a4/0x1b0
 [<b013a27e>] ? run_hrtimer_pending+0xe/0xb0
 [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
 [<b012d548>] ? timer_stats_account_timer+0x38/0x40
 [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
 [<b012dbbc>] run_timer_softirq+0xac/0x170
 [<b013f863>] ? tick_periodic+0x33/0x70
 [<b013f8b7>] ? tick_handle_periodic+0x17/0x70
 [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
 [<b0129ae4>] __do_softirq+0x84/0xa0
 [<b0129b35>] do_softirq+0x35/0x40
 [<b0129bf6>] irq_exit+0x66/0x70
 [<b0105869>] do_IRQ+0x49/0x90
 [<b013bc30>] ? sched_clock_cpu+0xb0/0x100
 [<b010449b>] common_interrupt+0x23/0x28
 [<b0305158>] ? acpi_safe_halt+0x1b/0x29
 [<b0305b07>] acpi_idle_enter_c1+0xa6/0x117
 [<b03c096b>] cpuidle_idle_call+0x6b/0xa0
 [<b010206f>] cpu_idle+0x4f/0x70
 [<b04458dd>] rest_init+0x4d/0x50
 =======================
---[ end trace 1cc3b74458d87dab ]---
tg3: eth0: transmit timed out, resetting
tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

The ease with which I reproduce it here clearly indicates that this is
related to the switch, probably just the fact that it is at 100 Mbps.
Unfortunately this evening I must go, but I still have one 100 Mbps
switch somewhere at home, I'll reproduce the same test ASAP in order
to bisect the issue.

Regards,
Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Matt Carlson]

On Mon, Nov 24, 2008 at 01:52:47PM -0800, Willy Tarreau wrote:
> Hi Matt,
> 
> just a follow-up.
> 
> On Mon, Nov 24, 2008 at 02:27:44PM +0100, Willy Tarreau wrote:
> > Hi Matt,
> > 
> > On Thu, Nov 20, 2008 at 01:53:18PM -0800, Matt Carlson wrote:
> > > > Today, with the notebook connected to a gig switch, I could not reproduce
> > > > the problem, even after one hour of approximately the same workload. I'll
> > > > retry with the original 100 Mbps switch on monday.
> > 
> > fairly easier now with the same switch. I just have to transfer 100k objects
> > over HTTP via this switch to see the problem happen :
> > 
> > tg3: eth0: The system may be re-ordering memory-mapped I/O cycles to the network device, attempting to recover. Please report the problem to the driver maintainer and include system chipset information.
> > tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
> > tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
> > tg3: eth0: Link is down.
> > tg3: eth0: Link is up at 100 Mbps, full duplex.
> > tg3: eth0: Flow control is on for TX and on for RX.
> > 
> > The switch is an el-cheapo D-Link 10/100. Note that this time I did not see
> > any warning. Maybe I did not wait long enough though.
> 
> Got it again, just had to be patient to fire a second test :
> 
> WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0x1a4/0x1b0()
> NETDEV WATCHDOG: eth0 (tg3): transmit timed out
> Modules linked in: nfs lockd sunrpc mtdblock mtd_blkdevs slram mtd xt_tcpudp x_tables usbhid usb_storage ehci_hcd uhci_hcd usbcore snd_pcm_oss snd_mixer_oss snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc tg3 libphy ide_cs yenta_socket rsrc_nonstatic [last unloaded: ip_tables]
> Pid: 0, comm: swapper Not tainted 2.6.27-wt2-wtap #1
>  [<b01254a7>] warn_slowpath+0x67/0x90
>  [<b01741a9>] ? get_slab+0x9/0x70
>  [<b03d21af>] ? pskb_copy+0x2f/0x160
>  [<b03aa332>] ? input_defuzz_abs_event+0x12/0xa0
>  [<b03aa574>] ? input_handle_event+0x14/0x2a0
>  [<b03b3d76>] ? synaptics_process_packet+0x2b6/0x3d0
>  [<b0108a48>] ? native_io_delay+0x8/0x40
>  [<b02ab4c9>] ? strlen+0x9/0x20
>  [<b02a961e>] ? strlcpy+0x1e/0x60
>  [<b03dbfbc>] ? netdev_drivername+0x3c/0x40
>  [<b03e7c84>] dev_watchdog+0x1a4/0x1b0
>  [<b013a27e>] ? run_hrtimer_pending+0xe/0xb0
>  [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
>  [<b012d548>] ? timer_stats_account_timer+0x38/0x40
>  [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
>  [<b012dbbc>] run_timer_softirq+0xac/0x170
>  [<b013f863>] ? tick_periodic+0x33/0x70
>  [<b013f8b7>] ? tick_handle_periodic+0x17/0x70
>  [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
>  [<b0129ae4>] __do_softirq+0x84/0xa0
>  [<b0129b35>] do_softirq+0x35/0x40
>  [<b0129bf6>] irq_exit+0x66/0x70
>  [<b0105869>] do_IRQ+0x49/0x90
>  [<b013bc30>] ? sched_clock_cpu+0xb0/0x100
>  [<b010449b>] common_interrupt+0x23/0x28
>  [<b0305158>] ? acpi_safe_halt+0x1b/0x29
>  [<b0305b07>] acpi_idle_enter_c1+0xa6/0x117
>  [<b03c096b>] cpuidle_idle_call+0x6b/0xa0
>  [<b010206f>] cpu_idle+0x4f/0x70
>  [<b04458dd>] rest_init+0x4d/0x50
>  =======================
> ---[ end trace 1cc3b74458d87dab ]---
> tg3: eth0: transmit timed out, resetting
> tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
> tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
> tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
> tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
> tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
> tg3: eth0: Link is down.
> tg3: eth0: Link is up at 100 Mbps, full duplex.
> tg3: eth0: Flow control is on for TX and on for RX.
> 
> The ease with which I reproduce it here clearly indicates that this is
> related to the switch, probably just the fact that it is at 100 Mbps.
> Unfortunately this evening I must go, but I still have one 100 Mbps
> switch somewhere at home, I'll reproduce the same test ASAP in order
> to bisect the issue.
> 
> Regards,
> Willy

Does turning off flow control help at all?

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

On Mon, Nov 24, 2008 at 05:52:23PM -0800, Matt Carlson wrote:
(...)
> > tg3: eth0: transmit timed out, resetting
> > tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
> > tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
> > tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
> > tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
> > tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
> > tg3: eth0: Link is down.
> > tg3: eth0: Link is up at 100 Mbps, full duplex.
> > tg3: eth0: Flow control is on for TX and on for RX.
> > 
> > The ease with which I reproduce it here clearly indicates that this is
> > related to the switch, probably just the fact that it is at 100 Mbps.
> > Unfortunately this evening I must go, but I still have one 100 Mbps
> > switch somewhere at home, I'll reproduce the same test ASAP in order
> > to bisect the issue.
> > 
> > Regards,
> > Willy
> 
> Does turning off flow control help at all?

I have not tested but I will. I hope to be able to trigger the problem
on other similar switches, because I'm only once a week connected to
the culprit...

Regards,
Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Matt Carlson]

On Mon, Nov 24, 2008 at 09:31:28PM -0800, Willy Tarreau wrote:
> On Mon, Nov 24, 2008 at 05:52:23PM -0800, Matt Carlson wrote:
> (...)
> > > tg3: eth0: transmit timed out, resetting
> > > tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
> > > tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
> > > tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
> > > tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
> > > tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
> > > tg3: eth0: Link is down.
> > > tg3: eth0: Link is up at 100 Mbps, full duplex.
> > > tg3: eth0: Flow control is on for TX and on for RX.
> > > 
> > > The ease with which I reproduce it here clearly indicates that this is
> > > related to the switch, probably just the fact that it is at 100 Mbps.
> > > Unfortunately this evening I must go, but I still have one 100 Mbps
> > > switch somewhere at home, I'll reproduce the same test ASAP in order
> > > to bisect the issue.
> > > 
> > > Regards,
> > > Willy
> > 
> > Does turning off flow control help at all?
> 
> I have not tested but I will. I hope to be able to trigger the problem
> on other similar switches, because I'm only once a week connected to
> the culprit...

I can't say for certain, but I suspect the problem might be more
associated with the link speed than the particular switch you are using.
Can you try autoneg'ing down to a slower speed and see if that helps
make the problem more reproducable?

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

Hi Matt,

On Tue, Nov 25, 2008 at 09:54:13AM -0800, Matt Carlson wrote:
> On Mon, Nov 24, 2008 at 09:31:28PM -0800, Willy Tarreau wrote:
> > On Mon, Nov 24, 2008 at 05:52:23PM -0800, Matt Carlson wrote:
> > (...)
> > > > tg3: eth0: transmit timed out, resetting
> > > > tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
> > > > tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
> > > > tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
> > > > tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
> > > > tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
> > > > tg3: eth0: Link is down.
> > > > tg3: eth0: Link is up at 100 Mbps, full duplex.
> > > > tg3: eth0: Flow control is on for TX and on for RX.
> > > > 
> > > > The ease with which I reproduce it here clearly indicates that this is
> > > > related to the switch, probably just the fact that it is at 100 Mbps.
> > > > Unfortunately this evening I must go, but I still have one 100 Mbps
> > > > switch somewhere at home, I'll reproduce the same test ASAP in order
> > > > to bisect the issue.
> > > > 
> > > > Regards,
> > > > Willy
> > > 
> > > Does turning off flow control help at all?
> > 
> > I have not tested but I will. I hope to be able to trigger the problem
> > on other similar switches, because I'm only once a week connected to
> > the culprit...
> 
> I can't say for certain, but I suspect the problem might be more
> associated with the link speed than the particular switch you are using.
> Can you try autoneg'ing down to a slower speed and see if that helps
> make the problem more reproducable?

I've run a new test on a switch I have here at home (another el-cheapo,
non-manageable 100 Mbps, netgear this time). Unfortunately I cannot
reproduce the problem at all. I have disabled FC on my laptop, it did
not have any effect. I have disabled auto-neg and manually forced the
speed to 100/Full on my laptop, and could not reproduce the problem
either (though the speed was much lower due to the switch obviously
negociating 100/Half when not seeing my NWay frames).

I have tried unplugging the cable during transfers and changing negociation
during transfers, trying to trigger artifacts, but with no result. So I
think that I will really need to debug this on the "faulty" switch on
next monday. It does not surprize me much, because we don't see that
many reports for a similar problem, eventhough the tg3 is very common
in laptops. I just hope it's a recent regression, as I'd prefer avoid
having to bisect from a very old kernel.

I'll keep you informed,
Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Matt Carlson]

On Wed, Nov 26, 2008 at 01:12:20PM -0800, Willy Tarreau wrote:
> Hi Matt,
> 
> On Tue, Nov 25, 2008 at 09:54:13AM -0800, Matt Carlson wrote:
> > On Mon, Nov 24, 2008 at 09:31:28PM -0800, Willy Tarreau wrote:
> > > On Mon, Nov 24, 2008 at 05:52:23PM -0800, Matt Carlson wrote:
> > > (...)
> > > > > tg3: eth0: transmit timed out, resetting
> > > > > tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
> > > > > tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
> > > > > tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
> > > > > tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
> > > > > tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
> > > > > tg3: eth0: Link is down.
> > > > > tg3: eth0: Link is up at 100 Mbps, full duplex.
> > > > > tg3: eth0: Flow control is on for TX and on for RX.
> > > > > 
> > > > > The ease with which I reproduce it here clearly indicates that this is
> > > > > related to the switch, probably just the fact that it is at 100 Mbps.
> > > > > Unfortunately this evening I must go, but I still have one 100 Mbps
> > > > > switch somewhere at home, I'll reproduce the same test ASAP in order
> > > > > to bisect the issue.
> > > > > 
> > > > > Regards,
> > > > > Willy
> > > > 
> > > > Does turning off flow control help at all?
> > > 
> > > I have not tested but I will. I hope to be able to trigger the problem
> > > on other similar switches, because I'm only once a week connected to
> > > the culprit...
> > 
> > I can't say for certain, but I suspect the problem might be more
> > associated with the link speed than the particular switch you are using.
> > Can you try autoneg'ing down to a slower speed and see if that helps
> > make the problem more reproducable?
> 
> I've run a new test on a switch I have here at home (another el-cheapo,
> non-manageable 100 Mbps, netgear this time). Unfortunately I cannot
> reproduce the problem at all. I have disabled FC on my laptop, it did
> not have any effect.

Disabling FC should have a positive effect, not a negative one.  It
might be the case that the switch does not advertise nor support FC.  If
that is true, you might not be able to repro the problem no matter what
you did (if your problem is what I think it is).  Can you check your
link messages and see if it really is negotiated to off?  (I see the
message above, but I don't think that is with the current switch.)

> I have disabled auto-neg and manually forced the
> speed to 100/Full on my laptop, and could not reproduce the problem
> either (though the speed was much lower due to the switch obviously
> negociating 100/Half when not seeing my NWay frames).

Yes.  If you force the link, both sides must be forced.  The switch
rightly assumes HD when bringing the link up.

> I have tried unplugging the cable during transfers and changing negociation
> during transfers, trying to trigger artifacts, but with no result. So I
> think that I will really need to debug this on the "faulty" switch on
> next monday.  It does not surprize me much, because we don't see that
> many reports for a similar problem, eventhough the tg3 is very common
> in laptops. I just hope it's a recent regression, as I'd prefer avoid
> having to bisect from a very old kernel.
> 
> I'll keep you informed,
> Willy

O.K.

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

On Wed, Nov 26, 2008 at 02:54:21PM -0800, Matt Carlson wrote:
(...)
> > I've run a new test on a switch I have here at home (another el-cheapo,
> > non-manageable 100 Mbps, netgear this time). Unfortunately I cannot
> > reproduce the problem at all. I have disabled FC on my laptop, it did
> > not have any effect.
> 
> Disabling FC should have a positive effect, not a negative one.  It
> might be the case that the switch does not advertise nor support FC.  If
> that is true, you might not be able to repro the problem no matter what
> you did (if your problem is what I think it is).  Can you check your
> link messages and see if it really is negotiated to off?  (I see the
> message above, but I don't think that is with the current switch.)

yes the switch does advertuse FC :

willy@wtap:~$ dmesg|grep eth0
eth0: Tigon3 [partno(BMC5705mA3) rev 3003 PHY(5705)] (PCI:33MHz:32-bit) 10/100/1000Base-T Ethernet 00:0d:9d:91:ef:24
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TSOcap[1]
eth0: dma_rwctrl[763f0000] dma_mask[64-bit]
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

> > I have disabled auto-neg and manually forced the
> > speed to 100/Full on my laptop, and could not reproduce the problem
> > either (though the speed was much lower due to the switch obviously
> > negociating 100/Half when not seeing my NWay frames).
> 
> Yes.  If you force the link, both sides must be forced.  The switch
> rightly assumes HD when bringing the link up.

I know ;-) but not seeing the problem, I started to suspect that the other
switch was a little bit ill and tried to reproduce some problems I might
incidently have been encountering on it.

Regards,
Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Frantisek Hanzlik]

Willy Tarreau wrote:
> On Wed, Nov 26, 2008 at 02:54:21PM -0800, Matt Carlson wrote:
> (...)
>>> I've run a new test on a switch I have here at home (another el-cheapo,
>>> non-manageable 100 Mbps, netgear this time). Unfortunately I cannot
>>> reproduce the problem at all. I have disabled FC on my laptop, it did
>>> not have any effect.
>> Disabling FC should have a positive effect, not a negative one.  It
>> might be the case that the switch does not advertise nor support FC.  If
>> that is true, you might not be able to repro the problem no matter what
>> you did (if your problem is what I think it is).  Can you check your
>> link messages and see if it really is negotiated to off?  (I see the
>> message above, but I don't think that is with the current switch.)
>
> yes the switch does advertuse FC :
>
> willy@wtap:~$ dmesg|grep eth0
> eth0: Tigon3 [partno(BMC5705mA3) rev 3003 PHY(5705)] (PCI:33MHz:32-bit) 10/100/1000Base-T Ethernet 00:0d:9d:91:ef:24
> eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TSOcap[1]
> eth0: dma_rwctrl[763f0000] dma_mask[64-bit]
> tg3: eth0: Link is up at 100 Mbps, full duplex.
> tg3: eth0: Flow control is on for TX and on for RX.
>
>>> I have disabled auto-neg and manually forced the
>>> speed to 100/Full on my laptop, and could not reproduce the problem
>>> either (though the speed was much lower due to the switch obviously
>>> negociating 100/Half when not seeing my NWay frames).
>> Yes.  If you force the link, both sides must be forced.  The switch
>> rightly assumes HD when bringing the link up.
>
> I know ;-) but not seeing the problem, I started to suspect that the other
> switch was a little bit ill and tried to reproduce some problems I might
> incidently have been encountering on it.
>
> Regards,
> Willy
>
I got same issue two days ago on PCI-X Fiber 1000BASE-SX D-Link Adapter
DGE-550SX/dl2k driver (I report it yesterday). This card work fine in
old DEC Alphaserver 800/Fedora Core 5/kernel 2.6.17. We use it for our
internet connections, which is limited to approx. 50 Mb/sec by ISP,
then I think there should be no bottleneck in LAN card.
We just tried install new i386 based router (Core2Duo E8500/4GB RAM/
4x Realtek 8111C and this D-Link from old Alphaserver, on Fedora 10,
kernel 2.6.27.5). Router is connected to several gigabit backbones
over other ois gigabit cards, then I'm not sure when any crapped internal
machines could overload this DGE-550SX. Card in new router stop working
after several secs/minutes - packed transmitting freeze (what is
interesting, in one case after it sent exactly 8192 packets, and other
cases are some multiples of 8 too) and stop transmit packets.
Only workaround (but for a while) is rmmod + modprobe its dl2k driver.

/var/log/messages contains following messages:

...
Nov 25 19:04:52 ns kernel: Sundance Technology IPG Triple-Speed Ethernet 0000:09:00.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Nov 25 19:04:52 ns kernel: 0000:09:00.0: D-Link NIC
...
Nov 25 19:05:12 ns kernel: ------------[ cut here ]------------
Nov 25 19:05:12 ns kernel: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xda/0x12d()
Nov 25 19:05:12 ns kernel: NETDEV WATCHDOG: eth1 (Sundance Technology IPG Triple-Speed Ethernet): transmit timed out
Nov 25 19:05:12 ns kernel: Modules linked in: hwmon_vid hwmon nf_nat_ftp nf_conntrack_ftp xt_comment iptable_nat nf_nat cpufreq_ondemand acpi_cpufreq dm_multipath uinput snd_hda_intel 
snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device dl2k snd_pcm_oss snd_mixer_oss snd_pcm ipg snd_timer snd_page_alloc snd_hwdep snd i2c_i801 firewire_ohci i2c_core 
soundcore pcspkr r8169 mii firewire_core crc_itu_t raid456 async_xor async_memcpy async_tx xor raid1 [last unloaded: microcode]
Nov 25 19:05:12 ns kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5-117.fc10.i686 #1
Nov 25 19:05:12 ns kernel: [<c042bb3a>] warn_slowpath+0x4b/0x6c
Nov 25 19:05:12 ns kernel: [<c046fb00>] ? mempool_resize+0x15c/0x183
Nov 25 19:05:12 ns kernel: [<c048ca25>] ? __slab_free+0x63/0x26e
Nov 25 19:05:12 ns kernel: [<c048ce4e>] ? kmem_cache_free+0x71/0xa7
Nov 25 19:05:12 ns kernel: [<c048ca25>] ? __slab_free+0x63/0x26e
Nov 25 19:05:12 ns kernel: [<c050cb98>] ? blk_remove_plug+0x66/0x92
Nov 25 19:05:12 ns kernel: [<c050a19f>] ? elv_queue_empty+0x20/0x22
Nov 25 19:05:12 ns kernel: [<c050d29a>] ? blk_run_queue+0x28/0x2c
Nov 25 19:05:12 ns kernel: [<c059f8c5>] ? scsi_run_queue+0x250/0x27c
Nov 25 19:05:12 ns kernel: [<c0518eb6>] ? kobject_put+0x37/0x3c
Nov 25 19:05:12 ns kernel: [<c051bf6e>] ? strlcpy+0x17/0x49
Nov 25 19:05:12 ns kernel: [<c063e554>] dev_watchdog+0xda/0x12d
Nov 25 19:05:12 ns kernel: [<c05a0cef>] ? scsi_device_unbusy+0x6b/0x70
Nov 25 19:05:12 ns kernel: [<c04341f5>] run_timer_softirq+0x14b/0x1bb
Nov 25 19:05:12 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:05:12 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:05:12 ns kernel: [<c043076f>] __do_softirq+0x84/0x109
Nov 25 19:05:12 ns kernel: [<c04306eb>] ? __do_softirq+0x0/0x109
Nov 25 19:05:12 ns kernel: [<c0405eec>] do_softirq+0x77/0xdb
Nov 25 19:05:12 ns kernel: [<c04640bb>] ? handle_fasteoi_irq+0x0/0xc0
Nov 25 19:05:12 ns kernel: [<c04303d6>] irq_exit+0x44/0x83
Nov 25 19:05:12 ns kernel: [<c0405e5e>] do_IRQ+0xe7/0xfe
Nov 25 19:05:12 ns kernel: [<c0404654>] common_interrupt+0x28/0x30
Nov 25 19:05:12 ns kernel: [<c0564b95>] ? acpi_idle_enter_simple+0x162/0x19d
Nov 25 19:05:12 ns kernel: [<c0613efd>] cpuidle_idle_call+0x67/0x97
Nov 25 19:05:12 ns kernel: [<c0402c4d>] cpu_idle+0x101/0x134
Nov 25 19:05:12 ns kernel: [<c06a32de>] start_secondary+0x197/0x19f
Nov 25 19:05:12 ns kernel: =======================
Nov 25 19:05:12 ns kernel: ---[ end trace fd7fa9607e312047 ]---

=====

Nov 25 19:29:33 ns kernel: ------------[ cut here ]------------
Nov 25 19:29:33 ns kernel: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xda/0x12d()
Nov 25 19:29:33 ns kernel: NETDEV WATCHDOG: eth1 (Sundance Technology IPG Triple-Speed Ethernet): transmit timed out
Nov 25 19:29:33 ns kernel: Modules linked in: hwmon_vid hwmon nf_nat_ftp nf_conntrack_ftp xt_comment iptable_nat nf_nat cpufreq_ondemand acpi_cpufreq dm_multipath uinput dl2k i2c_i801 pcspkr 
r8169 ipg mii i2c_core firewire_ohci firewire_core crc_itu_t raid456 async_xor async_memcpy async_tx xor raid1 [last unloaded: microcode]
Nov 25 19:29:33 ns kernel: Pid: 9941, comm: mail Not tainted 2.6.27.5-117.fc10.i686 #1
Nov 25 19:29:33 ns kernel: [<c042bb3a>] warn_slowpath+0x4b/0x6c
Nov 25 19:29:33 ns kernel: [<c046fb00>] ? mempool_resize+0x15c/0x183
Nov 25 19:29:33 ns kernel: [<c04b0b3b>] ? bio_free+0x40/0x44
Nov 25 19:29:33 ns kernel: [<c04b0b4d>] ? bio_fs_destructor+0xe/0x11
Nov 25 19:29:33 ns kernel: [<c04af8a5>] ? bio_put+0x26/0x28
Nov 25 19:29:33 ns kernel: [<c048a971>] ? virt_to_head_page+0x22/0x2e
Nov 25 19:29:33 ns kernel: [<c050c0d4>] ? queue_flag_clear+0x18/0x54
Nov 25 19:29:33 ns kernel: [<c050c17b>] ? __freed_request+0x6b/0x72
Nov 25 19:29:33 ns kernel: [<c0420940>] ? __enqueue_entity+0xe3/0xeb
Nov 25 19:29:33 ns kernel: [<c04223aa>] ? enqueue_entity+0x203/0x20b
Nov 25 19:29:33 ns kernel: [<c051bf6e>] ? strlcpy+0x17/0x49
Nov 25 19:29:33 ns kernel: [<c063e554>] dev_watchdog+0xda/0x12d
Nov 25 19:29:33 ns kernel: [<c0405e5e>] ? do_IRQ+0xe7/0xfe
Nov 25 19:29:33 ns kernel: [<c04341f5>] run_timer_softirq+0x14b/0x1bb
Nov 25 19:29:33 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:29:33 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:29:33 ns kernel: [<c043076f>] __do_softirq+0x84/0x109
Nov 25 19:29:33 ns kernel: [<c04306eb>] ? __do_softirq+0x0/0x109
Nov 25 19:29:33 ns kernel: [<c0405eec>] do_softirq+0x77/0xdb
Nov 25 19:29:33 ns kernel: [<c04303d6>] irq_exit+0x44/0x83
Nov 25 19:29:33 ns kernel: [<c0413ee9>] smp_apic_timer_interrupt+0x6e/0x7c
Nov 25 19:29:33 ns kernel: [<c0404759>] apic_timer_interrupt+0x2d/0x34
Nov 25 19:29:33 ns kernel: [<c041007b>] ? speedstep_target+0x23/0x7e
Nov 25 19:29:33 ns kernel: [<c0419e89>] ? native_flush_tlb_single+0x6/0x8
Nov 25 19:29:33 ns kernel: [<c041d37d>] kunmap_atomic+0x67/0xa7
Nov 25 19:29:33 ns kernel: [<c047a368>] follow_page+0x1c5/0x23b
Nov 25 19:29:33 ns kernel: [<c047b98e>] get_user_pages+0x289/0x2fe
Nov 25 19:29:33 ns kernel: [<c0493d9e>] get_arg_page+0x2d/0x80
Nov 25 19:29:33 ns kernel: [<c051dcb9>] ? strnlen_user+0x2f/0x4d
Nov 25 19:29:33 ns kernel: [<c0493eb4>] copy_strings+0xc3/0x160
Nov 25 19:29:33 ns kernel: [<c0495079>] do_execve+0x14e/0x215
Nov 25 19:29:33 ns kernel: [<c04023a3>] sys_execve+0x29/0x50
Nov 25 19:29:33 ns kernel: [<c0403c76>] syscall_call+0x7/0xb
Nov 25 19:29:33 ns kernel: [<c06a007b>] ? init_intel_cacheinfo+0x0/0x421
Nov 25 19:29:33 ns kernel: =======================
Nov 25 19:29:33 ns kernel: ---[ end trace 45c41aa8555c94fa ]---

=====

Nov 25 19:41:00 ns kernel: ------------[ cut here ]------------
Nov 25 19:41:00 ns kernel: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xda/0x12d()
Nov 25 19:41:00 ns kernel: NETDEV WATCHDOG: eth1 (Sundance Technology IPG Triple-Speed Ethernet): transmit timed out
Nov 25 19:41:00 ns kernel: Modules linked in: hwmon_vid hwmon nf_nat_ftp nf_conntrack_ftp xt_comment iptable_nat nf_nat cpufreq_ondemand acpi_cpufreq dm_multipath uinput dl2k ipg 
firewire_ohci firewire_core crc_itu_t pcspkr r8169 mii i2c_i801 i2c_core raid456 async_xor async_memcpy async_tx xor raid1 [last unloaded: microcode]
Nov 25 19:41:00 ns kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5-117.fc10.i686 #1
Nov 25 19:41:00 ns kernel: [<c042bb3a>] warn_slowpath+0x4b/0x6c
Nov 25 19:41:00 ns kernel: [<c0420900>] ? __enqueue_entity+0xa3/0xeb
Nov 25 19:41:00 ns kernel: [<c04223aa>] ? enqueue_entity+0x203/0x20b
Nov 25 19:41:00 ns kernel: [<c04223ed>] ? enqueue_task_fair+0x3b/0x3f
Nov 25 19:41:00 ns kernel: [<c041d92c>] ? resched_task+0x3a/0x6e
Nov 25 19:41:00 ns kernel: [<c06a76b3>] ? _spin_unlock_irqrestore+0x22/0x38
Nov 25 19:41:00 ns kernel: [<c0426237>] ? try_to_wake_up+0x221/0x22b
Nov 25 19:41:00 ns kernel: [<c06a77cc>] ? _spin_lock_irqsave+0x29/0x30
Nov 25 19:41:00 ns kernel: [<c051bf6e>] ? strlcpy+0x17/0x49
Nov 25 19:41:00 ns kernel: [<c063e554>] dev_watchdog+0xda/0x12d
Nov 25 19:41:00 ns kernel: [<c043a716>] ? __queue_work+0x26/0x2b
Nov 25 19:41:00 ns kernel: [<c04341f5>] run_timer_softirq+0x14b/0x1bb
Nov 25 19:41:00 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:41:00 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:41:00 ns kernel: [<c043076f>] __do_softirq+0x84/0x109
Nov 25 19:41:00 ns kernel: [<c04306eb>] ? __do_softirq+0x0/0x109
Nov 25 19:41:00 ns kernel: [<c0405eec>] do_softirq+0x77/0xdb
Nov 25 19:41:00 ns kernel: [<c04303d6>] irq_exit+0x44/0x83
Nov 25 19:41:00 ns kernel: [<c0413ee9>] smp_apic_timer_interrupt+0x6e/0x7c
Nov 25 19:41:00 ns kernel: [<c0404759>] apic_timer_interrupt+0x2d/0x34
Nov 25 19:41:00 ns kernel: [<c05649e4>] ? acpi_idle_enter_bm+0x277/0x2c6
Nov 25 19:41:00 ns kernel: [<c0613efd>] cpuidle_idle_call+0x67/0x97
Nov 25 19:41:00 ns kernel: [<c0402c4d>] cpu_idle+0x101/0x134
Nov 25 19:41:00 ns kernel: [<c0697406>] rest_init+0x4e/0x50
Nov 25 19:41:00 ns kernel: =======================
Nov 25 19:41:00 ns kernel: ---[ end trace 9ce93cc9159b5214 ]---

It's there any thing for help with it?

Thx, Franta Hanzlik

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

On Thu, Nov 27, 2008 at 11:06:58AM +0100, Frantisek Hanzlik wrote:
> Willy Tarreau wrote:
> >On Wed, Nov 26, 2008 at 02:54:21PM -0800, Matt Carlson wrote:
> >(...)
> >>>I've run a new test on a switch I have here at home (another el-cheapo,
> >>>non-manageable 100 Mbps, netgear this time). Unfortunately I cannot
> >>>reproduce the problem at all. I have disabled FC on my laptop, it did
> >>>not have any effect.
> >>Disabling FC should have a positive effect, not a negative one.  It
> >>might be the case that the switch does not advertise nor support FC.  If
> >>that is true, you might not be able to repro the problem no matter what
> >>you did (if your problem is what I think it is).  Can you check your
> >>link messages and see if it really is negotiated to off?  (I see the
> >>message above, but I don't think that is with the current switch.)
> >
> >yes the switch does advertuse FC :
> >
> >willy@wtap:~$ dmesg|grep eth0
> >eth0: Tigon3 [partno(BMC5705mA3) rev 3003 PHY(5705)] (PCI:33MHz:32-bit) 
> >10/100/1000Base-T Ethernet 00:0d:9d:91:ef:24
> >eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TSOcap[1]
> >eth0: dma_rwctrl[763f0000] dma_mask[64-bit]
> >tg3: eth0: Link is up at 100 Mbps, full duplex.
> >tg3: eth0: Flow control is on for TX and on for RX.
> >
> >>>I have disabled auto-neg and manually forced the
> >>>speed to 100/Full on my laptop, and could not reproduce the problem
> >>>either (though the speed was much lower due to the switch obviously
> >>>negociating 100/Half when not seeing my NWay frames).
> >>Yes.  If you force the link, both sides must be forced.  The switch
> >>rightly assumes HD when bringing the link up.
> >
> >I know ;-) but not seeing the problem, I started to suspect that the other
> >switch was a little bit ill and tried to reproduce some problems I might
> >incidently have been encountering on it.
> >
> >Regards,
> >Willy
> >
> I got same issue two days ago on PCI-X Fiber 1000BASE-SX D-Link Adapter
> DGE-550SX/dl2k driver (I report it yesterday). This card work fine in
> old DEC Alphaserver 800/Fedora Core 5/kernel 2.6.17. We use it for our
> internet connections, which is limited to approx. 50 Mb/sec by ISP,
> then I think there should be no bottleneck in LAN card.
> We just tried install new i386 based router (Core2Duo E8500/4GB RAM/
> 4x Realtek 8111C and this D-Link from old Alphaserver, on Fedora 10,
> kernel 2.6.27.5). Router is connected to several gigabit backbones
> over other ois gigabit cards, then I'm not sure when any crapped internal
> machines could overload this DGE-550SX. Card in new router stop working
> after several secs/minutes - packed transmitting freeze (what is
> interesting, in one case after it sent exactly 8192 packets, and other
> cases are some multiples of 8 too) and stop transmit packets.
> Only workaround (but for a while) is rmmod + modprobe its dl2k driver.

I had been facing this same issue 5-6 years ago when I tried my first
DL2K. It always stopped transmitting after a few hundreds of packets
on my machine while it was OK on another one. Mine was dual-athlon
with an AMD chipset. A working one was a P3-933 with a ServerWorks
chipset. I noticed that it would only lock up when plugged into a
64-bit slot, and it was fine in 32-bit slots (though it did work well
on the P3 in a 64-bit slot). It was such a crappy chip that I finally
threw it away, so I could not compare anymore. I'd bet that your new
machine is too fast for this beast and it triggers some bugs in the
chip, probably on the PCI side.

Regards,
Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Willy Tarreau]

Hi Matt,

I ran a lot of tests last night. I have a few more information.
The issue sometimes takes longer to reproduce so it caused me
to identify wrong culprits among the 29 patches affecting tg3
between 2.6.25 and 2.6.27.7. I was finally able to reproduce
the issue by running the plain 2.6.25 driver (v3.90) on 2.6.27.7,
but not at all when running on 2.6.25, even after ten minutes
(in 2.6.27.7, it takes between 5s and 1mn to get a tx timeout).

Later, I noticed that 2.6.27's driver uses libphy, which was
never removed between tests. I wonder if it can interfer with
my tests. Maybe it initializes the phy differently from plain
2.6.25, causing delayed issues, I don't know. Unfortunately,
I cannot run 2.6.27's driver on 2.6.25 because of the libphy
dependency (that's how I discovered it).

I'm also now 100% certain that enabling/disabling FC does not
change anything with either kernel. So unless the hardware still
interpretes pause frames when disabled, it should not come from
there.

I suspect that the switch is getting ill : The problem happens
more often when it's been transfering at full speed for some
time. Since it's a cheap one lying on a desk, it might have
burned out capacitors in it causing some randomly corrupt
frames to go out from time to time (maybe even pause frames
preventing the NIC from sending). That was also a problem
for my tests, because after patching/unpatching and compilation
phases, it had some time to rest and took longer to reproduce
the issue.

I will re-run some tests on 2.6.27 + tg3 v3.90 (from 2.6.25)
without ever loading libphy from the power up, in order to
clearly identify if the problem is caused by the driver or
something else in the kernel. If it's something else, the
bisect will take a few weeks since I'm not there long enough
to run about 15 full builds and wait long enough for the
problem to (not) occur.

But I'm keeping hope, there's no reason not to find it!

Regards,
Willy

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Matt Carlson]

On Fri, Nov 14, 2008 at 08:01:46PM -0800, Roger Heflin wrote:
> Peter Zijlstra wrote:
> > (netdev CC'ed)
> > 
> > On Tue, 2008-11-11 at 03:48 -0600, Roger Heflin wrote:
> >> I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
> >> extra modules, tg3 Gbit networking.   I have not yet tested
> >> earlier kernels to see if this has been around for a while.
> > 
> > How do more recent kernels do?
> 
> I did not try more recent kernels, more testing seems to indicate
> that at the very least it the bug depends on a certain version of
> either tg3 and/or firmware to happen, as my second tg3 port does not
> have it happen.    More about this below.
> 
> > 
> >> So far I have had this error happen 5 times (MTBF is maybe
> >> 12 hours), 4 of the 5 times resulted in the networking being
> >> broken, one time things came back by itself without a reboot,
> >> I believe in this case the hang was traffic coming into the
> >> machine vs the other times going out of the machine.
> >>
> >> Unloading all of the network modules and reloading them did
> >> not correct the problem.
> >>
> >> Searching google finds a couple of other people getting the
> >> same error but they have a different network chipset (e1000
> >> and a rt811C chipset), which makes me thing that there is
> >> something interacting bad with the network.    Or does this
> >> error truly mean that the network chipset for some unknown reason
> >> locked itself up?
> >>
> >> http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
> >> http://article.gmane.org/gmane.linux.network/110238
> >>
> >> The changes I made recently were to upgrade my MB (old
> >> was E100 on a 100Mbit network,new is tg3 on a Gbit network,
> >> cpu and memory are the same, MB chipset is a intel 955
> >> chipset vs the old being a intel 915 chipset).
> >>
> >> Autoneg is turned on all around, the GBit switch is a
> >> 8-port Dlink switch.   The network seems to otherwise be working
> >> correctly.
> >>
> >> I did test the network under decent load and the error did not
> >> appear to be any more likely under load, and typically the network
> >> is under very light load 2-3MB/second.
> >>
> >> The machine originally had 2 HT CPU's showing up, I turned off HT
> >> so that only one cpu was showing, but this did not change the error.
> >>
> >> I am first turning off all offload capabilities on tg3 and going
> >> to see if that changes anything.
> 
> This made no difference in the error.
> 
> >>
> >> The next thing I am going to be doing is to turn of GB capability
> >> on the networking and see if that does anything.
> 
> Did not try.
> 
> >>
> >> I also have a second tg3 port that is slightly different, so I may
> >> try that eventually.
> 
> I tried this, and with the second port I don't appear to be getting
> the error.   The first port is a 5789-v3.29a and the second port is a
> 5788-v3.04, I know the first port is faster (pcie-x1) than the second
> port (pci bus-built-in, unknown exact connection).   The second port
> will sustain about 50MB/second, were as the first port will get
>  >90MB/second.

The 5789 is a PCIe device, the 5788 is a PCI device.  There's a lot of
room for differences between the two devices.

> It seems to me to likely be the firmware on the tg3, and it would seem
> unlikely that the driver could do anything more than work around the
> issue that is in the firmware, and currently my system works on the
> second port, and the second port is fast enough for my needs.

I don't think the 5789 should have any firmware running at the time of
the failure.  There's always the possibility that there is a problem
with the bootcode, but I'm guessing the problem is elsewhere.

> If someone else runs into this issue, since I have 2 ports I would be
> able to do some testing on it, right now my first port is locked up, and
> the machine is running fine on the second port.
> 
> lspci -vvv for the first (bad) port:

Ah.  There it is.

> 02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit 
> Ethernet PCI Express (rev 11)
>          Subsystem: Foxconn International, Inc. Unknown device 0cc1
>          Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B-
>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
> <MAbort- >SERR- <PERR-
>          Latency: 0, Cache Line Size: 32 bytes
>          Interrupt: pin A routed to IRQ 19
>          Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K]
>          Expansion ROM at <ignored> [disabled]
>          Capabilities: [48] Power Management version 2
>                  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>                  Status: D3 PME-Enable- DSel=0 DScale=1 PME-
>          Capabilities: [50] Vital Product Data
>          Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 
> Enable-
>                  Address: 0101b8102a0f7b0c  Data: f21e
>          Capabilities: [d0] Express Endpoint IRQ 0
>                  Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
>                  Device: Latency L0s <4us, L1 unlimited
>                  Device: AtnBtn- AtnInd- PwrInd-
>                  Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
>                  Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                  Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
>                  Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
>                  Link: Latency L0s <2us, L1 <64us
>                  Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
>                  Link: Speed 2.5Gb/s, Width x1
>          Capabilities: [100] Advanced Error Reporting
>          Capabilities: [13c] Virtual Channel

Hmmm.  No smoking gun.  Perhaps the register dump will help.


> >> Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
> >> Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
> >> dev_watchdog+0xfe/0x17e()
> >> Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out

Usually the tg3_tx_timeout function dumps a few registers before
resetting the chip, but I don't see that here.  Have you seen any dumps
since then?

> >> Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs 
> >> w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor 
> >> async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb 
> >> wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner 
> >> mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa 
> >> cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat 
> >> i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 
> >> videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support 
> >> v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc 
> >> videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc 
> >> parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero 
> >> dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 
> >> jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
> >> Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
> >> Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
> >> Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
> >> Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
> >> Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
> >> Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
> >> Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
> >> Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
> >> Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
> >> Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
> >> Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
> >> Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0 
> >> [uhci_hcd]
> >> Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
> >> Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
> >> Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
> >> Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
> >> Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
> >> Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
> >> Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
> >> Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
> >> Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
> >> Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
> >> Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
> >> Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
> >> Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
> >> Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> Please read the FAQ at  http://www.tux.org/lkml/
> > 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Roger Heflin]

Matt Carlson wrote:
> On Fri, Nov 14, 2008 at 08:01:46PM -0800, Roger Heflin wrote:
>> Peter Zijlstra wrote:
>>> (netdev CC'ed)
>>>
>>> On Tue, 2008-11-11 at 03:48 -0600, Roger Heflin wrote:
>>>> I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
>>>> extra modules, tg3 Gbit networking.   I have not yet tested
>>>> earlier kernels to see if this has been around for a while.
>>> How do more recent kernels do?
>> I did not try more recent kernels, more testing seems to indicate
>> that at the very least it the bug depends on a certain version of
>> either tg3 and/or firmware to happen, as my second tg3 port does not
>> have it happen.    More about this below.
>>
>>>> So far I have had this error happen 5 times (MTBF is maybe
>>>> 12 hours), 4 of the 5 times resulted in the networking being
>>>> broken, one time things came back by itself without a reboot,
>>>> I believe in this case the hang was traffic coming into the
>>>> machine vs the other times going out of the machine.
>>>>
>>>> Unloading all of the network modules and reloading them did
>>>> not correct the problem.
>>>>
>>>> Searching google finds a couple of other people getting the
>>>> same error but they have a different network chipset (e1000
>>>> and a rt811C chipset), which makes me thing that there is
>>>> something interacting bad with the network.    Or does this
>>>> error truly mean that the network chipset for some unknown reason
>>>> locked itself up?
>>>>
>>>> http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
>>>> http://article.gmane.org/gmane.linux.network/110238
>>>>
>>>> The changes I made recently were to upgrade my MB (old
>>>> was E100 on a 100Mbit network,new is tg3 on a Gbit network,
>>>> cpu and memory are the same, MB chipset is a intel 955
>>>> chipset vs the old being a intel 915 chipset).
>>>>
>>>> Autoneg is turned on all around, the GBit switch is a
>>>> 8-port Dlink switch.   The network seems to otherwise be working
>>>> correctly.
>>>>
>>>> I did test the network under decent load and the error did not
>>>> appear to be any more likely under load, and typically the network
>>>> is under very light load 2-3MB/second.
>>>>
>>>> The machine originally had 2 HT CPU's showing up, I turned off HT
>>>> so that only one cpu was showing, but this did not change the error.
>>>>
>>>> I am first turning off all offload capabilities on tg3 and going
>>>> to see if that changes anything.
>> This made no difference in the error.
>>
>>>> The next thing I am going to be doing is to turn of GB capability
>>>> on the networking and see if that does anything.
>> Did not try.
>>
>>>> I also have a second tg3 port that is slightly different, so I may
>>>> try that eventually.
>> I tried this, and with the second port I don't appear to be getting
>> the error.   The first port is a 5789-v3.29a and the second port is a
>> 5788-v3.04, I know the first port is faster (pcie-x1) than the second
>> port (pci bus-built-in, unknown exact connection).   The second port
>> will sustain about 50MB/second, were as the first port will get
>>  >90MB/second.
> 
> The 5789 is a PCIe device, the 5788 is a PCI device.  There's a lot of
> room for differences between the two devices.

I expected that there were going to be large changes between the two.
> 
>> It seems to me to likely be the firmware on the tg3, and it would seem
>> unlikely that the driver could do anything more than work around the
>> issue that is in the firmware, and currently my system works on the
>> second port, and the second port is fast enough for my needs.
> 
> I don't think the 5789 should have any firmware running at the time of
> the failure.  There's always the possibility that there is a problem
> with the bootcode, but I'm guessing the problem is elsewhere.

Where does the offload code and other similar stuff run?

I know on a tg3 before I had difficulties when a BMC card was in the system as 
the tg3 had a bug that it grabbed UDP packets that it though were going to the 
BMC card, in that case it was incorrectly thinking that UDP data was a port
number (this is only the case on the first UDP packet of a set) when it was 
actually data (on the UDP packets after the first one) in some of the udp 
packets.   Is that done from the firmware, or is that completely done in the 
hardware and basically not fixable?

> 
>> If someone else runs into this issue, since I have 2 ports I would be
>> able to do some testing on it, right now my first port is locked up, and
>> the machine is running fine on the second port.
>>
>> lspci -vvv for the first (bad) port:
> 
> Ah.  There it is.
> 
>> 02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit 
>> Ethernet PCI Express (rev 11)
>>          Subsystem: Foxconn International, Inc. Unknown device 0cc1
>>          Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
>> Stepping- SERR- FastB2B-
>>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
>> <MAbort- >SERR- <PERR-
>>          Latency: 0, Cache Line Size: 32 bytes
>>          Interrupt: pin A routed to IRQ 19
>>          Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K]
>>          Expansion ROM at <ignored> [disabled]
>>          Capabilities: [48] Power Management version 2
>>                  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
>> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>>                  Status: D3 PME-Enable- DSel=0 DScale=1 PME-
>>          Capabilities: [50] Vital Product Data
>>          Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 
>> Enable-
>>                  Address: 0101b8102a0f7b0c  Data: f21e
>>          Capabilities: [d0] Express Endpoint IRQ 0
>>                  Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
>>                  Device: Latency L0s <4us, L1 unlimited
>>                  Device: AtnBtn- AtnInd- PwrInd-
>>                  Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
>>                  Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>>                  Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
>>                  Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
>>                  Link: Latency L0s <2us, L1 <64us
>>                  Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
>>                  Link: Speed 2.5Gb/s, Width x1
>>          Capabilities: [100] Advanced Error Reporting
>>          Capabilities: [13c] Virtual Channel
> 
> Hmmm.  No smoking gun.  Perhaps the register dump will help.
> 

driver: tg3
version: 3.94
firmware-version: 5789-v3.29a
bus-info: 0000:02:00.0

tg3.c:v3.94 (August 14, 2008)
tg3 0000:02:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
tg3 0000:02:00.0: setting latency timer to 64
tg3 0000:05:01.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

Right now I brought the interface back up (it is still broken) and setup a 
network ip on it that other machines can ping.

The registers are included at the end of the email.


> 
>>>> Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
>>>> Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
>>>> dev_watchdog+0xfe/0x17e()
>>>> Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
> 
> Usually the tg3_tx_timeout function dumps a few registers before
> resetting the chip, but I don't see that here.  Have you seen any dumps
> since then?

Is this the dump?

Nov 12 14:58:13 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 14:58:13 computer kernel: tg3: DEBUG: MAC_TX_STATUS[00000008] 
MAC_RX_STATUS[00000006]
Nov 12 14:58:13 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000010] 
WDMAC_STATUS[00000000]
Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=2c00 
enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=1400 
enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 
enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: eth0: Link is down.
Nov 12 14:58:16 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 14:58:16 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 15:20:37 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 15:20:37 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] 
MAC_RX_STATUS[00000000]
Nov 12 15:20:37 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] 
WDMAC_STATUS[00000000]
Nov 12 15:20:37 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 
enable_bit=2
Nov 12 15:20:37 computer kernel: tg3: eth0: Link is down.
Nov 12 15:20:40 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 15:20:40 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 15:48:16 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 15:48:16 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] 
MAC_RX_STATUS[00000000]
Nov 12 15:48:16 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] 
WDMAC_STATUS[00000000]
Nov 12 15:48:16 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 
enable_bit=2
Nov 12 15:48:16 computer kernel: tg3: eth0: Link is down.
Nov 12 15:48:19 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 15:48:19 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:10:44 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:10:44 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] 
MAC_RX_STATUS[00000000]
Nov 12 16:10:44 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] 
WDMAC_STATUS[00000000]
Nov 12 16:10:44 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 
enable_bit=2
Nov 12 16:10:44 computer kernel: tg3: eth0: Link is down.
Nov 12 16:10:47 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:10:47 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:18:43 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:18:43 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] 
MAC_RX_STATUS[00000000]
Nov 12 16:18:43 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] 
WDMAC_STATUS[00000000]
Nov 12 16:18:43 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 
enable_bit=2
Nov 12 16:18:43 computer kernel: tg3: eth0: Link is down.
Nov 12 16:18:46 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:18:46 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:27:02 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:27:02 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] 
MAC_RX_STATUS[00000000]
Nov 12 16:27:02 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] 
WDMAC_STATUS[00000000]
Nov 12 16:27:02 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 
enable_bit=2
Nov 12 16:27:02 computer kernel: tg3: eth0: Link is down.
Nov 12 16:27:06 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:27:06 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:35:17 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:35:17 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] 
MAC_RX_STATUS[00000000]
Nov 12 16:35:17 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] 
WDMAC_STATUS[00000000]
Nov 12 16:35:17 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 
enable_bit=2
Nov 12 16:35:17 computer kernel: tg3: eth0: Link is down.
Nov 12 16:35:20 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:35:20 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.

> 
>>>> Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs 
>>>> w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor 
>>>> async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb 
>>>> wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner 
>>>> mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa 
>>>> cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat 
>>>> i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 
>>>> videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support 
>>>> v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc 
>>>> videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc 
>>>> parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero 
>>>> dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 
>>>> jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
>>>> Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
>>>> Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
>>>> Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
>>>> Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
>>>> Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
>>>> Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
>>>> Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
>>>> Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
>>>> Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
>>>> Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
>>>> Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
>>>> Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0 
>>>> [uhci_hcd]
>>>> Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
>>>> Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
>>>> Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
>>>> Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
>>>> Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
>>>> Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
>>>> Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
>>>> Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
>>>> Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
>>>> Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
>>>> Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
>>>> Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
>>>> Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
>>>> Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5

> 

Offset	Value
------	----------
0x0000	0x169d14e4
0x0004	0x00100006
0x0008	0x02000011
0x000c	0x00000008
0x0010	0xfd8f0004
0x0014	0x00000000
0x0018	0x00000000
0x001c	0x00000000
0x0020	0x00000000
0x0024	0x00000000
0x0028	0x00000007
0x002c	0x0cc1105b
0x0030	0x02aa0000
0x0034	0x00000048
0x0038	0x00000000
0x003c	0x0000010b
0x0040	0x00000000
0x0044	0x00000000
0x0048	0xc0025001
0x004c	0x64002000
0x0050	0x00d05803
0x0054	0x587d4b8a
0x0058	0x0086d005
0x005c	0x2a0f7b0c
0x0060	0x0101b810
0x0064	0x0000f21e
0x0068	0x41010298
0x006c	0x76180000
0x0070	0x000010f2
0x0074	0x000000c0
0x0078	0x00000050
0x007c	0x00000000
0x0080	0x00d05803
0x0084	0x00000000
0x0088	0x04130034
0x008c	0x3c081082
0x0090	0x01000609
0x0094	0x02000000
0x0098	0x00000000
0x009c	0x000000c8
0x00a0	0x00000000
0x00a4	0x00000000
0x00a8	0x00000000
0x00ac	0x00000000
0x00b0	0x00000000
0x00b4	0x00000000
0x00b8	0x00000000
0x00bc	0x00000000
0x00c0	0x00000000
0x00c4	0x00000000
0x00c8	0x00000000
0x00cc	0x00000000
0x00d0	0x00000000
0x00d4	0x00000000
0x00d8	0x00000000
0x00dc	0x00000000
0x00e0	0x00000000
0x00e4	0x00000000
0x00e8	0x00000000
0x00ec	0x00000000
0x00f0	0x00000000
0x00f4	0x00000000
0x00f8	0x00000000
0x00fc	0x00000000
0x0100	0x00000000
0x0104	0x00000000
0x0108	0x00000000
0x010c	0x00000000
0x0110	0x00000000
0x0114	0x00000000
0x0118	0x00000000
0x011c	0x00000000
0x0120	0x00000000
0x0124	0x00000000
0x0128	0x00000000
0x012c	0x00000000
0x0130	0x00000000
0x0134	0x00000000
0x0138	0x00000000
0x013c	0x00000000
0x0140	0x00000000
0x0144	0x00000000
0x0148	0x00000000
0x014c	0x00000000
0x0150	0x00000000
0x0154	0x00000000
0x0158	0x00000000

0x0200	0x00000000
0x0204	0x02000000
0x0208	0x00000000
0x020c	0x02000000
0x0210	0x00000000
0x0214	0x02000000
0x0218	0x00000000
0x021c	0x02000000
0x0220	0x00000000
0x0224	0x02000000
0x0228	0x00000000
0x022c	0x02000000
0x0230	0x00000000
0x0234	0x02000000
0x0238	0x00000000
0x023c	0x02000000
0x0240	0x00000000
0x0244	0x02000000
0x0248	0x00000000
0x024c	0x02000000
0x0250	0x00000000
0x0254	0x02000000
0x0258	0x00000000
0x025c	0x02000000
0x0260	0x00000000
0x0264	0x02000000
0x0268	0x00000000
0x026c	0x000000c8
0x0270	0x00000000
0x0274	0x02000000
0x0278	0x00000000
0x027c	0x02000000
0x0280	0x00000000
0x0284	0x00000000
0x0288	0x00000000
0x028c	0x02000000
0x0290	0x00000000
0x0294	0x02000000
0x0298	0x00000000
0x029c	0x02000000
0x02a0	0x00000000
0x02a4	0x02000000
0x02a8	0x00000000
0x02ac	0x02000000
0x02b0	0x00000000
0x02b4	0x02000000
0x02b8	0x00000000
0x02bc	0x02000000
0x02c0	0x00000000
0x02c4	0x02000000
0x02c8	0x00000000
0x02cc	0x02000000
0x02d0	0x00000000
0x02d4	0x02000000
0x02d8	0x00000000
0x02dc	0x02000000
0x02e0	0x00000000
0x02e4	0x02000000
0x02e8	0x00000000
0x02ec	0x02000000
0x02f0	0x00000000
0x02f4	0x02000000
0x02f8	0x00000000
0x02fc	0x02000000
0x0300	0x00000000
0x0304	0x00000024
0x0308	0x00000000
0x030c	0x02000024
0x0310	0x00000000
0x0314	0x02000024
0x0318	0x00000000
0x031c	0x02000024
0x0320	0x00000000
0x0324	0x02000024
0x0328	0x00000000
0x032c	0x02000024
0x0330	0x00000000
0x0334	0x02000024
0x0338	0x00000000
0x033c	0x02000024
0x0340	0x00000000
0x0344	0x02000024
0x0348	0x00000000
0x034c	0x02000024
0x0350	0x00000000
0x0354	0x02000024
0x0358	0x00000000
0x035c	0x02000024
0x0360	0x00000000
0x0364	0x02000024
0x0368	0x00000000
0x036c	0x02000024
0x0370	0x00000000
0x0374	0x02000024
0x0378	0x00000000
0x037c	0x02000024
0x0380	0x00000000
0x0384	0x00000000
0x0388	0x00000000
0x038c	0x02000000
0x0390	0x00000000
0x0394	0x02000000
0x0398	0x00000000
0x039c	0x02000000
0x03a0	0x00000000
0x03a4	0x02000000
0x03a8	0x00000000
0x03ac	0x02000000
0x03b0	0x00000000
0x03b4	0x02000000
0x03b8	0x00000000
0x03bc	0x02000000
0x03c0	0x00000000
0x03c4	0x02000000
0x03c8	0x00000000
0x03cc	0x02000000
0x03d0	0x00000000
0x03d4	0x02000000
0x03d8	0x00000000
0x03dc	0x02000000
0x03e0	0x00000000
0x03e4	0x02000000
0x03e8	0x00000000
0x03ec	0x02000000
0x03f0	0x00000000
0x03f4	0x02000000
0x03f8	0x00000000
0x03fc	0x02000000

0x0400	0x00e04808
0x0404	0x00400000
0x0408	0x00001000
0x040c	0x00000880
0x0410	0x00000001
0x0414	0x6ce59c51
0x0418	0x00000001
0x041c	0x6ce59c51
0x0420	0x00000001
0x0424	0x6ce59c51
0x0428	0x00000001
0x042c	0x6ce59c51
0x0430	0x00000000
0x0434	0x00000000
0x0438	0x000000da
0x043c	0x000005f2
0x0440	0x00000000
0x0444	0x00000000
0x0448	0x00000000
0x044c	0x0825c5e1
0x0450	0x00000001
0x0454	0x000c0000
0x0458	0x00000000
0x045c	0x00000012
0x0460	0x00000008
0x0464	0x00002620
0x0468	0x00000406
0x046c	0x00000000
0x0470	0x80000000
0x0474	0x00000000
0x0478	0x00000080
0x047c	0x40200000
0x0480	0x42000000
0x0484	0x7fffffff
0x0488	0x06000004
0x048c	0x7fffffff
0x0490	0x00000000
0x0494	0x00000000
0x0498	0x00000000
0x049c	0x00000000
0x04a0	0x00000000
0x04a4	0x00000000
0x04a8	0x00000000
0x04ac	0x00000000
0x04b0	0x00000000
0x04b4	0x00000000
0x04b8	0x00000000
0x04bc	0x00000000
0x04c0	0x00000000
0x04c4	0x00000000
0x04c8	0x00000000
0x04cc	0x00000000
0x04d0	0x00000000
0x04d4	0x00000000
0x04d8	0x00000000
0x04dc	0x00000000
0x04e0	0x00000000
0x04e4	0x00000000
0x04e8	0x00000000
0x04ec	0x00000000
0x04f0	0x00000000
0x04f4	0x00000000
0x04f8	0x00000000
0x04fc	0x00000000
0x0500	0x00000008
0x0504	0x00000002
0x0508	0x00000000
0x050c	0x00000000
0x0510	0x00000000
0x0514	0x00000000
0x0518	0x00000000
0x051c	0x00000000
0x0520	0x00000000
0x0524	0x00000000
0x0528	0x00000000
0x052c	0x00000000
0x0530	0x00000000
0x0534	0x00000000
0x0538	0x00000000
0x053c	0x00000000
0x0540	0x00000000
0x0544	0x00000000
0x0548	0x00000000
0x054c	0x00000000
0x0550	0x00000000
0x0554	0x00000000
0x0558	0x00000000
0x055c	0x00000000
0x0560	0x00000000
0x0564	0x00000000
0x0568	0x00000000
0x056c	0x00000000
0x0570	0x00000000
0x0574	0x00000000
0x0578	0x00000000
0x057c	0x00000000
0x0580	0x00000000
0x0584	0x00000000
0x0588	0x00000000
0x058c	0x00000000
0x0590	0x00e00000
0x0594	0x00000000
0x0598	0x00000000
0x059c	0x00000000
0x05a0	0x00000000
0x05a4	0x00000000
0x05a8	0x00000000
0x05ac	0x00000000
0x05b0	0x00000000
0x05b4	0x00000000
0x05b8	0x00000000
0x05bc	0x00000000
0x05c0	0x00000000
0x05c4	0x00000000
0x05c8	0x00000000
0x05cc	0x00000000
0x05d0	0x00000000
0x05d4	0x00000000
0x05d8	0x00000000
0x05dc	0x00000000
0x05e0	0x00000000
0x05e4	0x00000000
0x05e8	0x00000000
0x05ec	0x00000000
0x05f0	0x00000000
0x05f4	0x00000000
0x05f8	0x00000000
0x05fc	0x00000000
0x0600	0xffffffff
0x0604	0x00f80011
0x0608	0x00000000
0x060c	0x00001f04
0x0610	0xffffffff
0x0614	0x00000000
0x0618	0x07c00004
0x061c	0x30000000
0x0620	0x00000040
0x0624	0x00000000
0x0628	0x00000000
0x062c	0x00000000
0x0630	0x00000000
0x0634	0x00000000
0x0638	0x00000000
0x063c	0x00000000
0x0640	0x00000000
0x0644	0x00000000
0x0648	0x00000000
0x064c	0x00000000
0x0650	0x00000000
0x0654	0x00000000
0x0658	0x00000000
0x065c	0x00000000
0x0660	0x00000000
0x0664	0x00000000
0x0668	0x00000000
0x066c	0x00000000
0x0670	0x00000000
0x0674	0x00000000
0x0678	0x00000000
0x067c	0x00000000
0x0680	0x00000000
0x0684	0x00000000
0x0688	0x00000000
0x068c	0x00000000
0x0690	0x00000000
0x0694	0x00000000
0x0698	0x00000000
0x069c	0x00000000
0x06a0	0x00000000
0x06a4	0x00000000
0x06a8	0x00000000
0x06ac	0x00000000
0x06b0	0x00000000
0x06b4	0x00000000
0x06b8	0x00000000
0x06bc	0x00000000
0x06c0	0x00000000
0x06c4	0x00000000
0x06c8	0x00000000
0x06cc	0x00000000
0x06d0	0x00000000
0x06d4	0x00000000
0x06d8	0x00000000
0x06dc	0x00000000
0x06e0	0x00000000
0x06e4	0x00000000
0x06e8	0x00000000
0x06ec	0x00000000
0x06f0	0x00000000
0x06f4	0x00000000
0x06f8	0x00000000
0x06fc	0x00000000
0x0700	0x00000000
0x0704	0x00000000
0x0708	0x00000000
0x070c	0x00000000
0x0710	0x00000000
0x0714	0x00000000
0x0718	0x00000000
0x071c	0x00000000
0x0720	0x00000000
0x0724	0x00000000
0x0728	0x00000000
0x072c	0x00000000
0x0730	0x00000000
0x0734	0x00000000
0x0738	0x00000000
0x073c	0x00000000
0x0740	0x00000000
0x0744	0x00000000
0x0748	0x00000000
0x074c	0x00000000
0x0750	0x00000000
0x0754	0x00000000
0x0758	0x00000000
0x075c	0x00000000
0x0760	0x00000000
0x0764	0x00000000
0x0768	0x00000000
0x076c	0x00000000
0x0770	0x00000000
0x0774	0x00000000
0x0778	0x00000000
0x077c	0x00000000
0x0780	0x00000000
0x0784	0x00000000
0x0788	0x00000000
0x078c	0x00000000
0x0790	0x00000000
0x0794	0x00000000
0x0798	0x00000000
0x079c	0x00000000
0x07a0	0x00000000
0x07a4	0x00000000
0x07a8	0x00000000
0x07ac	0x00000000
0x07b0	0x00000000
0x07b4	0x00000000
0x07b8	0x00000000
0x07bc	0x00000000
0x07c0	0x00000000
0x07c4	0x00000000
0x07c8	0x00000000
0x07cc	0x00000000
0x07d0	0x00000000
0x07d4	0x00000000
0x07d8	0x00000000
0x07dc	0x00000000
0x07e0	0x00000000
0x07e4	0x00000000
0x07e8	0x00000000
0x07ec	0x00000000
0x07f0	0x00000000
0x07f4	0x00000000
0x07f8	0x00000000
0x07fc	0x00000000
0x0800	0x00000000
0x0804	0xffffffff
0x0808	0x00000000
0x080c	0x00000000
0x0810	0x00000000
0x0814	0xffffffff
0x0818	0x00000000
0x081c	0x00000000
0x0820	0x00000000
0x0824	0x00000000
0x0828	0xffffffff
0x082c	0x00000000
0x0830	0x00000000
0x0834	0xffffffff
0x0838	0xffffffff
0x083c	0xffffffff
0x0840	0xffffffff
0x0844	0xffffffff
0x0848	0xffffffff
0x084c	0xffffffff
0x0850	0xffffffff
0x0854	0xffffffff
0x0858	0xffffffff
0x085c	0xffffffff
0x0860	0xffffffff
0x0864	0xffffffff
0x0868	0xffffffff
0x086c	0x00000000
0x0870	0x00000000
0x0874	0x00000000
0x0878	0x00000000
0x087c	0x00000000
0x0880	0x00000000
0x0884	0x00000c80
0x0888	0x00000000
0x088c	0x00000000
0x0890	0x00000000
0x0894	0x00000000
0x0898	0x00000000
0x089c	0x00000000
0x08a0	0x00000000
0x08a4	0x00000000
0x08a8	0x00000000
0x08ac	0x00000000
0x08b0	0x00000000
0x08b4	0x00000000
0x08b8	0x00000000
0x08bc	0x00000000
0x08c0	0x00000000
0x08c4	0x00000000
0x08c8	0x00000000
0x08cc	0x00000000
0x08d0	0x00000000
0x08d4	0x00000000
0x08d8	0x00000000
0x08dc	0x00000000
0x08e0	0x00000000
0x08e4	0x00000000
0x08e8	0x00000000
0x08ec	0x00000000

0x0c00	0x0000000a
0x0c04	0x00000000
0x0c08	0x00000003
0x0c0c	0x00000001
0x0c10	0x00000000
0x0c14	0x00000000
0x0c18	0x00000000
0x0c1c	0x00000000
0x0c20	0x00000000
0x0c24	0x00000000
0x0c28	0x00000000
0x0c2c	0x00000000
0x0c30	0x00000000
0x0c34	0x00000000
0x0c38	0x00000000
0x0c3c	0x00000000
0x0c40	0x00000000
0x0c44	0x00000000
0x0c48	0x00000000
0x0c4c	0x00000000
0x0c50	0x00000000
0x0c54	0x00000000
0x0c58	0x00000000
0x0c5c	0x00000000
0x0c60	0x00000000
0x0c64	0x00000000
0x0c68	0x00000000
0x0c6c	0x00000000
0x0c70	0x00000000
0x0c74	0x00000000
0x0c78	0x00000000
0x0c7c	0x00000000
0x0c80	0x00000000
0x0c84	0x00000000
0x0c88	0x00000000
0x0c8c	0x00000000
0x0c90	0x00000000
0x0c94	0x00000000
0x0c98	0x00000000
0x0c9c	0x00000000
0x0ca0	0x00000000
0x0ca4	0x00000000
0x0ca8	0x00000000
0x0cac	0x00000000
0x0cb0	0x00000000
0x0cb4	0x00000000
0x0cb8	0x00000000
0x0cbc	0x00000000
0x0cc0	0x00000000
0x0cc4	0x00000000
0x0cc8	0x00000000
0x0ccc	0x00000000
0x0cd0	0x00000000
0x0cd4	0x00000000
0x0cd8	0x00000000
0x0cdc	0x00000000

0x1000	0x00000002

0x1400	0x00000006
0x1404	0x00000000
0x1408	0x00000000
0x140c	0x00000000
0x1410	0x00000000
0x1414	0x00000000
0x1418	0x00000000
0x141c	0x00000000
0x1420	0x00000000
0x1424	0x00000000
0x1428	0x00000000
0x142c	0x00000000
0x1430	0x00000000
0x1434	0x00000000
0x1438	0x00000000
0x143c	0x00000000
0x1440	0x00000000
0x1444	0x00000000
0x1448	0x00000000
0x144c	0x00000000
0x1450	0x00000000
0x1454	0x00000000
0x1458	0x00000000
0x145c	0x00000000
0x1460	0x00000000
0x1464	0x00000000
0x1468	0x00000000
0x146c	0x00000000
0x1470	0x00000000
0x1474	0x00000000
0x1478	0x00000000
0x147c	0x00000000

0x1800	0x00000016
0x1804	0x00000000
0x1808	0x00000001
0x180c	0x00000000
0x1810	0x00000000
0x1814	0x00000000
0x1818	0x00000000
0x181c	0x00000000
0x1820	0x00000000
0x1824	0x00000000
0x1828	0x00000000
0x182c	0x00000000
0x1830	0x00000000
0x1834	0x00000000
0x1838	0x00000000
0x183c	0x00000000
0x1840	0x00000000
0x1844	0x00000000

0x1c00	0x00000002

0x2000	0x00000002
0x2004	0x00000000
0x2008	0x00000000
0x200c	0x00000000
0x2010	0x00000181
0x2014	0x00000001
0x2018	0x00790407
0x201c	0x00000000
0x2020	0x00000000
0x2024	0x00000000
0x2028	0x00000000
0x202c	0x00000000
0x2030	0x00000000
0x2034	0x00000000
0x2038	0x00000000
0x203c	0x00000000
0x2040	0x00000000
0x2044	0x00000000
0x2048	0x00000000
0x204c	0x00000000
0x2050	0x00000000
0x2054	0x00000000
0x2058	0x00000000
0x205c	0x00000000
0x2060	0x00000000
0x2064	0x00000000
0x2068	0x00000000
0x206c	0x00000000
0x2070	0x00000000
0x2074	0x00000000
0x2078	0x00000000
0x207c	0x00000000
0x2080	0x00000000
0x2084	0x00000000
0x2088	0x00000000
0x208c	0x00000000
0x2090	0x00000000
0x2094	0x00000000
0x2098	0x00000000
0x209c	0x00000000
0x20a0	0x00000000
0x20a4	0x00000000
0x20a8	0x00000000
0x20ac	0x00000000
0x20b0	0x00000000
0x20b4	0x00000000
0x20b8	0x00000000
0x20bc	0x00000000
0x20c0	0x00000000
0x20c4	0x00000000
0x20c8	0x00000000
0x20cc	0x00000000
0x20d0	0x00000000
0x20d4	0x00000000
0x20d8	0x00000000
0x20dc	0x00000000
0x20e0	0x00000000
0x20e4	0x00000000
0x20e8	0x00000000
0x20ec	0x00000000
0x20f0	0x00000000
0x20f4	0x00000000
0x20f8	0x00000000
0x20fc	0x00000000
0x2100	0x00086633
0x2104	0x00090683
0x2108	0x0000004d
0x210c	0x00000000
0x2110	0x00086633
0x2114	0x00090683
0x2118	0x0000004d
0x211c	0x00000000
0x2120	0x00086633
0x2124	0x00090683
0x2128	0x0000004d
0x212c	0x00000000
0x2130	0x00086633
0x2134	0x00090683
0x2138	0x0000004d
0x213c	0x00000000
0x2140	0x00086633
0x2144	0x00090683
0x2148	0x0000004d
0x214c	0x00000000
0x2150	0x00086633
0x2154	0x00090683
0x2158	0x0000004d
0x215c	0x00000000
0x2160	0x00086633
0x2164	0x00090683
0x2168	0x0000004d
0x216c	0x00000000
0x2170	0x00086633
0x2174	0x00090683
0x2178	0x0000004d
0x217c	0x00000000
0x2180	0x00086633
0x2184	0x00090683
0x2188	0x0000004d
0x218c	0x00000000
0x2190	0x00086633
0x2194	0x00090683
0x2198	0x0000004d
0x219c	0x00000000
0x21a0	0x00086633
0x21a4	0x00090683
0x21a8	0x0000004d
0x21ac	0x00000000
0x21b0	0x00086633
0x21b4	0x00090683
0x21b8	0x0000004d
0x21bc	0x00000000
0x21c0	0x00086633
0x21c4	0x00090683
0x21c8	0x0000004d
0x21cc	0x00000000
0x21d0	0x00086633
0x21d4	0x00090683
0x21d8	0x0000004d
0x21dc	0x00000000
0x21e0	0x00086633
0x21e4	0x00090683
0x21e8	0x0000004d
0x21ec	0x00000000
0x21f0	0x00086633
0x21f4	0x00090683
0x21f8	0x0000004d
0x21fc	0x00000000
0x2200	0x00000037
0x2204	0x00000000
0x2208	0x00000000
0x220c	0x00000000
0x2210	0x00000000
0x2214	0x00000000
0x2218	0x00000000
0x221c	0x00000000
0x2220	0x00000000
0x2224	0x00000000
0x2228	0x00000000
0x222c	0x00000000
0x2230	0x00000000
0x2234	0x00000000
0x2238	0x00000000
0x223c	0x00000000
0x2240	0x00000000
0x2244	0x00000000
0x2248	0x00000000
0x224c	0x00000000
0x2250	0x00000000
0x2254	0x00000000
0x2258	0x00000000

0x2400	0x00000012
0x2404	0x00000000
0x2408	0x00000000
0x240c	0x00000000
0x2410	0x00000000
0x2414	0x00000000
0x2418	0x00000000
0x241c	0x00000000
0x2420	0x00000000
0x2424	0x00000000
0x2428	0x00000000
0x242c	0x00000000
0x2430	0x00000000
0x2434	0x00000000
0x2438	0x00000000
0x243c	0x00000000
0x2440	0x00000000
0x2444	0x00000000
0x2448	0x00000000
0x244c	0x00000000
0x2450	0x00000000
0x2454	0x030d4000
0x2458	0x02000000
0x245c	0x00006000
0x2460	0x00000000
0x2464	0x00000000
0x2468	0x00000000
0x246c	0x00000000
0x2470	0x00000000
0x2474	0x00000000
0x2478	0x00000000
0x247c	0x00000000
0x2480	0x00000000
0x2484	0x00000000
0x2488	0x00000000
0x248c	0x00000000
0x2490	0x00000000
0x2494	0x00000000
0x2498	0x00000000
0x249c	0x00000000
0x24a0	0x00000000
0x24a4	0x00000000
0x24a8	0x00000000
0x24ac	0x00000000
0x24b0	0x00000000
0x24b4	0x00000000
0x24b8	0x00000000
0x24bc	0x00000000
0x24c0	0x08000010

0x2800	0x00000006

0x2c00	0x00000006
0x2c04	0x00000000
0x2c08	0x00000000
0x2c0c	0x0000007f
0x2c10	0x00000000
0x2c14	0x00000000
0x2c18	0x00000008
0x2c1c	0x00000000

0x3000	0x00000006
0x3004	0x00000000
0x3008	0x00000000
0x300c	0x00000000
0x3010	0x00000000

0x3400	0x00000000
0x3404	0x00000000

0x3800	0x00000000
0x3804	0x00000000

0x3c00	0x00000302
0x3c04	0x00000000
0x3c08	0x00000064
0x3c0c	0x00000048
0x3c10	0x0000000a
0x3c14	0x00000035
0x3c18	0x00000000
0x3c1c	0x00000000
0x3c20	0x0000000a
0x3c24	0x00000005
0x3c28	0x00000000
0x3c2c	0x00000000
0x3c30	0x00000000
0x3c34	0x00000000
0x3c38	0x00000000
0x3c3c	0x03270000
0x3c40	0x00000000
0x3c44	0x00000b00
0x3c48	0x00000000
0x3c4c	0x00000000
0x3c50	0x00000000
0x3c54	0x00000000
0x3c58	0x00000000
0x3c5c	0x00000000
0x3c60	0x00000000
0x3c64	0x00000000
0x3c68	0x00000000
0x3c6c	0x00000000
0x3c70	0x00000000
0x3c74	0x00000000
0x3c78	0x00000000
0x3c7c	0x00000000
0x3c80	0x00000000
0x3c84	0x00000000
0x3c88	0x00000000
0x3c8c	0x00000000
0x3c90	0x00000000
0x3c94	0x00000000
0x3c98	0x00000000
0x3c9c	0x00000000
0x3ca0	0x00000000
0x3ca4	0x00000000
0x3ca8	0x00000000
0x3cac	0x00000000
0x3cb0	0x00000000
0x3cb4	0x00000000
0x3cb8	0x00000000
0x3cbc	0x00000000
0x3cc0	0x00000000
0x3cc4	0x00000000
0x3cc8	0x00000000
0x3ccc	0x00000000
0x3cd0	0x00000000
0x3cd4	0x00000000
0x3cd8	0x00000000
0x3cdc	0x00000000
0x3ce0	0x00000000
0x3ce4	0x00000000
0x3ce8	0x00000000
0x3cec	0x00000000
0x3cf0	0x00000000
0x3cf4	0x00000000
0x3cf8	0x00000000
0x3cfc	0x00000000

0x4000	0x00000002
0x4004	0x00000000
0x4008	0x000b72fd
0x400c	0x001d174b

0x4400	0x00000006
0x4404	0x00000000
0x4408	0x00010f00
0x440c	0x0000e100
0x4410	0x00000000
0x4414	0x00000010
0x4418	0x00000060
0x441c	0x00000000
0x4420	0x0000003d
0x4424	0x00000000
0x4428	0x00000000
0x442c	0x00000000
0x4430	0x00000000
0x4434	0x00000000
0x4438	0x00000000
0x443c	0x00000000
0x4440	0x00000000
0x4444	0x00000000
0x4448	0x00000000
0x444c	0x01207401
0x4450	0x005e0364
0x4454	0x00090067

0x4800	0x080303fe
0x4804	0x00000000

0x4c00	0x000003fe
0x4c04	0x00000000

0x5000	0x00009820
0x5004	0x80008000
0x5008	0x00000000
0x500c	0x00000000
0x5010	0x00000000
0x5014	0x00000000
0x5018	0x00000000
0x501c	0x00010d24
0x5020	0x00000000
0x5024	0x00000000
0x5028	0x00000000
0x502c	0x00000000
0x5030	0x00000000
0x5034	0x0000001d
0x5038	0x00000000
0x503c	0x00000000
0x5040	0x00000000
0x5044	0x00000000
0x5048	0x00000000
0x504c	0x00000000
0x5050	0x00000000
0x5054	0x00000000
0x5058	0x00000000
0x505c	0x00000000
0x5060	0x00000000
0x5064	0x00000000
0x5068	0x00000000
0x506c	0x00000000
0x5070	0x00000000
0x5074	0x00000000
0x5078	0x00000000
0x507c	0x00000000
0x5080	0x00000000
0x5084	0x00000000
0x5088	0x00000000
0x508c	0x00000000
0x5090	0x00000000
0x5094	0x00000000
0x5098	0x00000000
0x509c	0x00000000
0x50a0	0x00000000
0x50a4	0x00000000
0x50a8	0x00000000
0x50ac	0x00000000
0x50b0	0x00000000
0x50b4	0x00000000
0x50b8	0x00000000
0x50bc	0x00000000
0x50c0	0x00000000
0x50c4	0x00000000
0x50c8	0x00000000
0x50cc	0x00000000
0x50d0	0x00000000
0x50d4	0x00000000
0x50d8	0x00000000
0x50dc	0x00000000
0x50e0	0x00000000
0x50e4	0x00000000
0x50e8	0x00000000
0x50ec	0x00000000
0x50f0	0x00000000
0x50f4	0x00000000
0x50f8	0x00000000
0x50fc	0x00000000
0x5100	0x00000000
0x5104	0x00000000
0x5108	0x00000000
0x510c	0x00000000
0x5110	0x00000000
0x5114	0x00000000
0x5118	0x00000000
0x511c	0x00000000
0x5120	0x00000000
0x5124	0x00000000
0x5128	0x00000000
0x512c	0x00000000
0x5130	0x00000000
0x5134	0x00000000
0x5138	0x00000000
0x513c	0x00000000
0x5140	0x00000000
0x5144	0x00000000
0x5148	0x00000000
0x514c	0x00000000
0x5150	0x00000000
0x5154	0x00000000
0x5158	0x00000000
0x515c	0x00000000
0x5160	0x00000000
0x5164	0x00000000
0x5168	0x00000000
0x516c	0x00000000
0x5170	0x00000000
0x5174	0x00000000
0x5178	0x00000000
0x517c	0x00000000
0x5180	0x00000000
0x5184	0x00000000
0x5188	0x00000000
0x518c	0x00000000
0x5190	0x00000000
0x5194	0x00000000
0x5198	0x00000000
0x519c	0x00000000
0x51a0	0x00000000
0x51a4	0x00000000
0x51a8	0x00000000
0x51ac	0x00000000
0x51b0	0x00000000
0x51b4	0x00000000
0x51b8	0x00000000
0x51bc	0x00000000
0x51c0	0x00000000
0x51c4	0x00000000
0x51c8	0x00000000
0x51cc	0x00000000
0x51d0	0x00000000
0x51d4	0x00000000
0x51d8	0x00000000
0x51dc	0x00000000
0x51e0	0x00000000
0x51e4	0x00000000
0x51e8	0x00000000
0x51ec	0x00000000
0x51f0	0x00000000
0x51f4	0x00000000
0x51f8	0x00000000
0x51fc	0x00000000
0x5200	0x00000000
0x5204	0x00000000
0x5208	0x00000000
0x520c	0x00000000
0x5210	0x00000000
0x5214	0x00000000
0x5218	0x00000000
0x521c	0x00000000
0x5220	0x00000000
0x5224	0x00000000
0x5228	0x00000000
0x522c	0x00000000
0x5230	0x00000000
0x5234	0x00000000
0x5238	0x00000000
0x523c	0x00000000
0x5240	0x00000000
0x5244	0x00000000
0x5248	0x00000000
0x524c	0x00000000
0x5250	0x00000000
0x5254	0x00000000
0x5258	0x00000000
0x525c	0x00000000
0x5260	0x00000000
0x5264	0x00000000
0x5268	0x00000000
0x526c	0x00000000
0x5270	0x00000000
0x5274	0x00000000
0x5278	0x00000000
0x527c	0x00000000

0x5400	0x00000000
0x5404	0x00000000
0x5408	0x00000000
0x540c	0x00000000
0x5410	0x00000000
0x5414	0x00000000
0x5418	0x00000000
0x541c	0x00000000
0x5420	0x00000000
0x5424	0x00000000
0x5428	0x00000000
0x542c	0x00000000
0x5430	0x00000000
0x5434	0x00000000
0x5438	0x00000000
0x543c	0x00000000
0x5440	0x00000000
0x5444	0x00000000
0x5448	0x00000000
0x544c	0x00000000
0x5450	0x00000000
0x5454	0x00000000
0x5458	0x00000000
0x545c	0x00000000
0x5460	0x00000000
0x5464	0x00000000
0x5468	0x00000000
0x546c	0x00000000
0x5470	0x00000000
0x5474	0x00000000
0x5478	0x00000000
0x547c	0x00000000
0x5480	0x00000000
0x5484	0x00000000
0x5488	0x00000000
0x548c	0x00000000
0x5490	0x00000000
0x5494	0x00000000
0x5498	0x00000000
0x549c	0x00000000
0x54a0	0x00000000
0x54a4	0x00000000
0x54a8	0x00000000
0x54ac	0x00000000
0x54b0	0x00000000
0x54b4	0x00000000
0x54b8	0x00000000
0x54bc	0x00000000
0x54c0	0x00000000
0x54c4	0x00000000
0x54c8	0x00000000
0x54cc	0x00000000
0x54d0	0x00000000
0x54d4	0x00000000
0x54d8	0x00000000
0x54dc	0x00000000
0x54e0	0x00000000
0x54e4	0x00000000
0x54e8	0x00000000
0x54ec	0x00000000
0x54f0	0x00000000
0x54f4	0x00000000
0x54f8	0x00000000
0x54fc	0x00000000
0x5500	0x00000000
0x5504	0x00000000
0x5508	0x00000000
0x550c	0x00000000
0x5510	0x00000000
0x5514	0x00000000
0x5518	0x00000000
0x551c	0x00000000
0x5520	0x00000000
0x5524	0x00000000
0x5528	0x00000000
0x552c	0x00000000
0x5530	0x00000000
0x5534	0x00000000
0x5538	0x00000000
0x553c	0x00000000
0x5540	0x00000000
0x5544	0x00000000
0x5548	0x00000000
0x554c	0x00000000
0x5550	0x00000000
0x5554	0x00000000
0x5558	0x00000000
0x555c	0x00000000
0x5560	0x00000000
0x5564	0x00000000
0x5568	0x00000000
0x556c	0x00000000
0x5570	0x00000000
0x5574	0x00000000
0x5578	0x00000000
0x557c	0x00000000
0x5580	0x00000000
0x5584	0x00000000
0x5588	0x00000000
0x558c	0x00000000
0x5590	0x00000000
0x5594	0x00000000
0x5598	0x00000000
0x559c	0x00000000
0x55a0	0x00000000
0x55a4	0x00000000
0x55a8	0x00000000
0x55ac	0x00000000
0x55b0	0x00000000
0x55b4	0x00000000
0x55b8	0x00000000
0x55bc	0x00000000
0x55c0	0x00000000
0x55c4	0x00000000
0x55c8	0x00000000
0x55cc	0x00000000
0x55d0	0x00000000
0x55d4	0x00000000
0x55d8	0x00000000
0x55dc	0x00000000
0x55e0	0x00000000
0x55e4	0x00000000
0x55e8	0x00000000
0x55ec	0x00000000
0x55f0	0x00000000
0x55f4	0x00000000
0x55f8	0x00000000
0x55fc	0x00000000
0x5600	0x00000000
0x5604	0x00000000
0x5608	0x00000000
0x560c	0x00000000
0x5610	0x00000000
0x5614	0x00000000
0x5618	0x00000000
0x561c	0x00000000
0x5620	0x00000000
0x5624	0x00000000
0x5628	0x00000000
0x562c	0x00000000
0x5630	0x00000000
0x5634	0x00000000
0x5638	0x00000000
0x563c	0x00000000
0x5640	0x00000000
0x5644	0x00000000
0x5648	0x00000000
0x564c	0x00000000
0x5650	0x00000000
0x5654	0x00000000
0x5658	0x00000000
0x565c	0x00000000
0x5660	0x00000000
0x5664	0x00000000
0x5668	0x00000000
0x566c	0x00000000
0x5670	0x00000000
0x5674	0x00000000
0x5678	0x00000000
0x567c	0x00000000

0x5800	0x00000000
0x5804	0x02000000
0x5808	0x00000000
0x580c	0x02000000
0x5810	0x00000000
0x5814	0x02000000
0x5818	0x00000000
0x581c	0x02000000
0x5820	0x00000000
0x5824	0x02000000
0x5828	0x00000000
0x582c	0x02000000
0x5830	0x00000000
0x5834	0x02000000
0x5838	0x00000000
0x583c	0x02000000
0x5840	0x00000000
0x5844	0x02000000
0x5848	0x00000000
0x584c	0x02000000
0x5850	0x00000000
0x5854	0x02000000
0x5858	0x00000000
0x585c	0x02000000
0x5860	0x00000000
0x5864	0x02000000
0x5868	0x00000000
0x586c	0x000000c8
0x5870	0x00000000
0x5874	0x02000000
0x5878	0x00000000
0x587c	0x02000000
0x5880	0x00000000
0x5884	0x00000000
0x5888	0x00000000
0x588c	0x02000000
0x5890	0x00000000
0x5894	0x02000000
0x5898	0x00000000
0x589c	0x02000000
0x58a0	0x00000000
0x58a4	0x02000000
0x58a8	0x00000000
0x58ac	0x02000000
0x58b0	0x00000000
0x58b4	0x02000000
0x58b8	0x00000000
0x58bc	0x02000000
0x58c0	0x00000000
0x58c4	0x02000000
0x58c8	0x00000000
0x58cc	0x02000000
0x58d0	0x00000000
0x58d4	0x02000000
0x58d8	0x00000000
0x58dc	0x02000000
0x58e0	0x00000000
0x58e4	0x02000000
0x58e8	0x00000000
0x58ec	0x02000000
0x58f0	0x00000000
0x58f4	0x02000000
0x58f8	0x00000000
0x58fc	0x02000000
0x5900	0x00000000
0x5904	0x00000024
0x5908	0x00000000
0x590c	0x02000000
0x5910	0x00000000
0x5914	0x00000000
0x5918	0x00000000
0x591c	0x00000000
0x5920	0x00000000
0x5924	0x00000000
0x5928	0x00000000
0x592c	0x00000000
0x5930	0x00000000
0x5934	0x00000000
0x5938	0x00000000
0x593c	0x00000000
0x5940	0x00000000
0x5944	0x00000000
0x5948	0x00000000
0x594c	0x00000000
0x5950	0x00000000
0x5954	0x00000000
0x5958	0x00000000
0x595c	0x00000000
0x5960	0x00000000
0x5964	0x00000000
0x5968	0x00000000
0x596c	0x00000000
0x5970	0x00000000
0x5974	0x00000000
0x5978	0x00000000
0x597c	0x00000000
0x5980	0x00000000
0x5984	0x00000000
0x5988	0x00000000
0x598c	0x00000000
0x5990	0x00000000
0x5994	0x00000000
0x5998	0x00000000
0x599c	0x00000000
0x59a0	0x00000000
0x59a4	0x00000000
0x59a8	0x00000000
0x59ac	0x00000000
0x59b0	0x00000000
0x59b4	0x00000000
0x59b8	0x00000000
0x59bc	0x00000000
0x59c0	0x00000000
0x59c4	0x00000000
0x59c8	0x00000000
0x59cc	0x00000000
0x59d0	0x00000000
0x59d4	0x00000000
0x59d8	0x00000000
0x59dc	0x00000000
0x59e0	0x00000000
0x59e4	0x00000000
0x59e8	0x00000000
0x59ec	0x00000000
0x59f0	0x00000000
0x59f4	0x00000000
0x59f8	0x00000000
0x59fc	0x00000000
0x5a00	0x00000000
0x5a04	0x00000000
0x5a08	0x00000000
0x5a0c	0x00000000

0x5c00	0x00000000
0x5c04	0x00000000
0x5c08	0x00000000
0x5c0c	0x00000000
0x5c10	0x00000000
0x5c14	0x00000000
0x5c18	0x00000000
0x5c1c	0x00000000
0x5c20	0x00000000
0x5c24	0x00000000
0x5c28	0x00000000
0x5c2c	0x00000000
0x5c30	0x00000000
0x5c34	0x00000000
0x5c38	0x00000000
0x5c3c	0x00000000
0x5c40	0x00000000
0x5c44	0x00000000
0x5c48	0x00000000
0x5c4c	0x00000000
0x5c50	0x00000000
0x5c54	0x00000000
0x5c58	0x00000000
0x5c5c	0x00000000
0x5c60	0x00000000
0x5c64	0x00000000
0x5c68	0x00000000
0x5c6c	0x00000000
0x5c70	0x00000000
0x5c74	0x00000000
0x5c78	0x00000000
0x5c7c	0x00000000
0x5c80	0x00000000
0x5c84	0x00000000
0x5c88	0x00000000
0x5c8c	0x00000000
0x5c90	0x00000000
0x5c94	0x00000000
0x5c98	0x00000000
0x5c9c	0x00000000
0x5ca0	0x00000000
0x5ca4	0x00000000
0x5ca8	0x00000000
0x5cac	0x00000000
0x5cb0	0x00000000
0x5cb4	0x00000000
0x5cb8	0x00000000
0x5cbc	0x00000000
0x5cc0	0x00000000
0x5cc4	0x00000000
0x5cc8	0x00000000
0x5ccc	0x00000000
0x5cd0	0x00000000
0x5cd4	0x00000000
0x5cd8	0x00000000
0x5cdc	0x00000000
0x5ce0	0x00000000
0x5ce4	0x00000000
0x5ce8	0x00000000
0x5cec	0x00000000
0x5cf0	0x00000000
0x5cf4	0x00000000
0x5cf8	0x00000000
0x5cfc	0x00000000
0x5d00	0x00000000
0x5d04	0x00000000
0x5d08	0x00000000
0x5d0c	0x00000000
0x5d10	0x00000000
0x5d14	0x00000000
0x5d18	0x00000000
0x5d1c	0x00000000

0x6000	0x00000000
0x6004	0x00000000
0x6008	0x00000004

0x6800	0x04130034
0x6804	0x3c081082
0x6808	0x01000609
0x680c	0x16695a00
0x6810	0x81020010
0x6814	0xffffffff
0x6818	0x00000000
0x681c	0x00000000
0x6820	0x00000000
0x6824	0x00000000
0x6828	0x00000000
0x682c	0x00000000
0x6830	0xfffc3ccf
0x6834	0xfffc0fff
0x6838	0x00000000
0x683c	0x00000000
0x6840	0x00000004
0x6844	0x00000000

0x7000	0x00000188
0x7004	0x00000000
0x7008	0x00000000
0x700c	0x000000c0
0x7010	0x0a000064
0x7014	0x02008273
0x7018	0x00570081
0x701c	0x68848353
0x7020	0x00000000
0x7024	0x00000000
0x7028	0x00000000
0x702c	0x00000000
0x7030	0x00000000

0x7c00	0x00000000
0x7c04	0x00000000
0x7c08	0x00000000
0x7c0c	0x00000000
0x7c10	0x00000000
0x7c14	0x00000000
0x7c18	0x00000000
0x7c1c	0x00000000
0x7c20	0x00000000
0x7c24	0x00000000
0x7c28	0x00000000
0x7c2c	0x00000000
0x7c30	0x00000000
0x7c34	0x00000000
0x7c38	0x00000000
0x7c3c	0x00000000
0x7c40	0x00000000
0x7c44	0x00000000
0x7c48	0x00000000
0x7c4c	0x00000000
0x7c50	0x00000000
0x7c54	0x00000000
0x7c58	0x00000000
0x7c5c	0x00000000
0x7c60	0x00000000
0x7c64	0x00000000
0x7c68	0x00000000
0x7c6c	0x00000000
0x7c70	0x00000000
0x7c74	0x00000000
0x7c78	0x00000000
0x7c7c	0x00000000
0x7c80	0x00000000
0x7c84	0x00000000
0x7c88	0x00000000
0x7c8c	0x00000000
0x7c90	0x00000000
0x7c94	0x00000000
0x7c98	0x00000000
0x7c9c	0x00000000
0x7ca0	0x00000000
0x7ca4	0x00000000
0x7ca8	0x00000000
0x7cac	0x00000000
0x7cb0	0x00000000
0x7cb4	0x00000000
0x7cb8	0x00000000
0x7cbc	0x00000000
0x7cc0	0x00000000
0x7cc4	0x00000000
0x7cc8	0x00000000
0x7ccc	0x00000000
0x7cd0	0x00000000
0x7cd4	0x00000000
0x7cd8	0x00000000
0x7cdc	0x00000000
0x7ce0	0x00000000
0x7ce4	0x00000000
0x7ce8	0x00000000
0x7cec	0x00000000
0x7cf0	0x00000000
0x7cf4	0x00000000
0x7cf8	0x00000000
0x7cfc	0x00000000
0x7d00	0x00000000
0x7d04	0x00000000
0x7d08	0x00000000
0x7d0c	0x00000000
0x7d10	0x00000000
0x7d14	0x00000000
0x7d18	0x00000000
0x7d1c	0x00000000
0x7d20	0x00000000
0x7d24	0x00000000
0x7d28	0x00000000
0x7d2c	0x00000000
0x7d30	0x00000000
0x7d34	0x00000000
0x7d38	0x00000000
0x7d3c	0x00000000
0x7d40	0x00000000
0x7d44	0x00000000
0x7d48	0x00000000
0x7d4c	0x00000000
0x7d50	0x00000000
0x7d54	0x00000000
0x7d58	0x00000000
0x7d5c	0x00000000
0x7d60	0x00000000
0x7d64	0x00000000
0x7d68	0x00000000
0x7d6c	0x00000000
0x7d70	0x00000000
0x7d74	0x00000000
0x7d78	0x00000000
0x7d7c	0x00000000
0x7d80	0x00000000
0x7d84	0x00000000
0x7d88	0x00000000
0x7d8c	0x00000000
0x7d90	0x00000000
0x7d94	0x00000000
0x7d98	0x00000000
0x7d9c	0x00000000
0x7da0	0x00000000
0x7da4	0x00000000
0x7da8	0x00000000
0x7dac	0x00000000
0x7db0	0x00000000
0x7db4	0x00000000
0x7db8	0x00000000
0x7dbc	0x00000000
0x7dc0	0x00000000
0x7dc4	0x00000000
0x7dc8	0x00000000
0x7dcc	0x00000000
0x7dd0	0x00000000
0x7dd4	0x00000000
0x7dd8	0x00000000
0x7ddc	0x00000000
0x7de0	0x00000000
0x7de4	0x00000000
0x7de8	0x00000000
0x7dec	0x00000000
0x7df0	0x00000000
0x7df4	0x00000000
0x7df8	0x00000000
0x7dfc	0x00000000
0x7e00	0x00000000
0x7e04	0x00000000
0x7e08	0x00000000
0x7e0c	0x00000000
0x7e10	0x00000000
0x7e14	0x00000000
0x7e18	0x00000000
0x7e1c	0x00000000
0x7e20	0x00000000
0x7e24	0x00000000
0x7e28	0x00000000
0x7e2c	0x00000000
0x7e30	0x00000000
0x7e34	0x00000000
0x7e38	0x00000000
0x7e3c	0x00000000

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Matt Carlson]

On Thu, Nov 20, 2008 at 02:07:42AM -0800, Roger Heflin wrote:
> Matt Carlson wrote:
> > On Fri, Nov 14, 2008 at 08:01:46PM -0800, Roger Heflin wrote:
> >> Peter Zijlstra wrote:
> >>> (netdev CC'ed)
> >>>
> >>> On Tue, 2008-11-11 at 03:48 -0600, Roger Heflin wrote:
> >>>> I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
> >>>> extra modules, tg3 Gbit networking.   I have not yet tested
> >>>> earlier kernels to see if this has been around for a while.
> >>> How do more recent kernels do?
> >> I did not try more recent kernels, more testing seems to indicate
> >> that at the very least it the bug depends on a certain version of
> >> either tg3 and/or firmware to happen, as my second tg3 port does not
> >> have it happen.    More about this below.
> >>
> >>>> So far I have had this error happen 5 times (MTBF is maybe
> >>>> 12 hours), 4 of the 5 times resulted in the networking being
> >>>> broken, one time things came back by itself without a reboot,
> >>>> I believe in this case the hang was traffic coming into the
> >>>> machine vs the other times going out of the machine.
> >>>>
> >>>> Unloading all of the network modules and reloading them did
> >>>> not correct the problem.
> >>>>
> >>>> Searching google finds a couple of other people getting the
> >>>> same error but they have a different network chipset (e1000
> >>>> and a rt811C chipset), which makes me thing that there is
> >>>> something interacting bad with the network.    Or does this
> >>>> error truly mean that the network chipset for some unknown reason
> >>>> locked itself up?
> >>>>
> >>>> http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
> >>>> http://article.gmane.org/gmane.linux.network/110238
> >>>>
> >>>> The changes I made recently were to upgrade my MB (old
> >>>> was E100 on a 100Mbit network,new is tg3 on a Gbit network,
> >>>> cpu and memory are the same, MB chipset is a intel 955
> >>>> chipset vs the old being a intel 915 chipset).
> >>>>
> >>>> Autoneg is turned on all around, the GBit switch is a
> >>>> 8-port Dlink switch.   The network seems to otherwise be working
> >>>> correctly.
> >>>>
> >>>> I did test the network under decent load and the error did not
> >>>> appear to be any more likely under load, and typically the network
> >>>> is under very light load 2-3MB/second.
> >>>>
> >>>> The machine originally had 2 HT CPU's showing up, I turned off HT
> >>>> so that only one cpu was showing, but this did not change the error.
> >>>>
> >>>> I am first turning off all offload capabilities on tg3 and going
> >>>> to see if that changes anything.
> >> This made no difference in the error.
> >>
> >>>> The next thing I am going to be doing is to turn of GB capability
> >>>> on the networking and see if that does anything.
> >> Did not try.
> >>
> >>>> I also have a second tg3 port that is slightly different, so I may
> >>>> try that eventually.
> >> I tried this, and with the second port I don't appear to be getting
> >> the error.   The first port is a 5789-v3.29a and the second port is a
> >> 5788-v3.04, I know the first port is faster (pcie-x1) than the second
> >> port (pci bus-built-in, unknown exact connection).   The second port
> >> will sustain about 50MB/second, were as the first port will get
> >>  >90MB/second.
> >
> > The 5789 is a PCIe device, the 5788 is a PCI device.  There's a lot of
> > room for differences between the two devices.
> 
> I expected that there were going to be large changes between the two.
> >
> >> It seems to me to likely be the firmware on the tg3, and it would seem
> >> unlikely that the driver could do anything more than work around the
> >> issue that is in the firmware, and currently my system works on the
> >> second port, and the second port is fast enough for my needs.
> >
> > I don't think the 5789 should have any firmware running at the time of
> > the failure.  There's always the possibility that there is a problem
> > with the bootcode, but I'm guessing the problem is elsewhere.
> 
> Where does the offload code and other similar stuff run?

On the 5789, LSO is implemented directly in the hardware.  Checksum
offload was always in the hardware.

> I know on a tg3 before I had difficulties when a BMC card was in the system as
> the tg3 had a bug that it grabbed UDP packets that it though were going to the
> BMC card, in that case it was incorrectly thinking that UDP data was a port
> number (this is only the case on the first UDP packet of a set) when it was
> actually data (on the UDP packets after the first one) in some of the udp
> packets.   Is that done from the firmware, or is that completely done in the
> hardware and basically not fixable?

Yes, I remember hearing something about this problem too.  That is a firmware
problem though.  The 5789 does not have any management firmware, so that
shouldn't be the case here.

> >> If someone else runs into this issue, since I have 2 ports I would be
> >> able to do some testing on it, right now my first port is locked up, and
> >> the machine is running fine on the second port.
> >>
> >> lspci -vvv for the first (bad) port:
> >
> > Ah.  There it is.
> >
> >> 02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit
> >> Ethernet PCI Express (rev 11)
> >>          Subsystem: Foxconn International, Inc. Unknown device 0cc1
> >>          Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> >> Stepping- SERR- FastB2B-
> >>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> >> <MAbort- >SERR- <PERR-
> >>          Latency: 0, Cache Line Size: 32 bytes
> >>          Interrupt: pin A routed to IRQ 19
> >>          Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K]
> >>          Expansion ROM at <ignored> [disabled]
> >>          Capabilities: [48] Power Management version 2
> >>                  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> >> PME(D0-,D1-,D2-,D3hot+,D3cold+)
> >>                  Status: D3 PME-Enable- DSel=0 DScale=1 PME-
> >>          Capabilities: [50] Vital Product Data
> >>          Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
> >> Enable-
> >>                  Address: 0101b8102a0f7b0c  Data: f21e
> >>          Capabilities: [d0] Express Endpoint IRQ 0
> >>                  Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
> >>                  Device: Latency L0s <4us, L1 unlimited
> >>                  Device: AtnBtn- AtnInd- PwrInd-
> >>                  Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
> >>                  Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> >>                  Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
> >>                  Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
> >>                  Link: Latency L0s <2us, L1 <64us
> >>                  Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
> >>                  Link: Speed 2.5Gb/s, Width x1
> >>          Capabilities: [100] Advanced Error Reporting
> >>          Capabilities: [13c] Virtual Channel
> >
> > Hmmm.  No smoking gun.  Perhaps the register dump will help.
> >
> 
> driver: tg3
> version: 3.94
> firmware-version: 5789-v3.29a
> bus-info: 0000:02:00.0

O.K.  I'll see if I can find any problems like this in the firmware
archives.

> tg3.c:v3.94 (August 14, 2008)
> tg3 0000:02:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
> tg3 0000:02:00.0: setting latency timer to 64
> tg3 0000:05:01.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
> tg3: eth0: Link is up at 1000 Mbps, full duplex.
> tg3: eth0: Flow control is on for TX and on for RX.
> 
> Right now I brought the interface back up (it is still broken) and setup a
> network ip on it that other machines can ping.
> 
> The registers are included at the end of the email.

O.K.  I'll pour over the dump and get back to you.

More below.

> >>>> Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
> >>>> Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219
> >>>> dev_watchdog+0xfe/0x17e()
> >>>> Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
> >
> > Usually the tg3_tx_timeout function dumps a few registers before
> > resetting the chip, but I don't see that here.  Have you seen any dumps
> > since then?
> 
> Is this the dump?

This would be it.  Thanks.

> Nov 12 14:58:13 computer kernel: tg3: eth0: transmit timed out, resetting
> Nov 12 14:58:13 computer kernel: tg3: DEBUG: MAC_TX_STATUS[00000008]
> MAC_RX_STATUS[00000006]
> Nov 12 14:58:13 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000010]
> WDMAC_STATUS[00000000]

Here the Read DMA Status register is reporting a Read DMA PCI Parity Error.
I've seen this before...very recently in fact.  The problem was that the
chipset was not programmed by the BIOS correctly.  In that particular case,
a BIOS upgrade solved the problem.  YMMV.

> Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=2c00
> enable_bit=2
> Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=1400
> enable_bit=2
> Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
> enable_bit=2
> Nov 12 14:58:13 computer kernel: tg3: eth0: Link is down.
> Nov 12 14:58:16 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
> Nov 12 14:58:16 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
> Nov 12 15:20:37 computer kernel: tg3: eth0: transmit timed out, resetting
> Nov 12 15:20:37 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
> MAC_RX_STATUS[00000000]
> Nov 12 15:20:37 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
> WDMAC_STATUS[00000000]

Here the MAC TX Status register is reporting that the link is up, but
the device is sending pause frames and rx is currently rx off'd.

Does the same problem happen if flow control is disabled?

> Nov 12 15:20:37 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
> enable_bit=2
> Nov 12 15:20:37 computer kernel: tg3: eth0: Link is down.
> Nov 12 15:20:40 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
> Nov 12 15:20:40 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
> Nov 12 15:48:16 computer kernel: tg3: eth0: transmit timed out, resetting
> Nov 12 15:48:16 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
> MAC_RX_STATUS[00000000]
> Nov 12 15:48:16 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
> WDMAC_STATUS[00000000]
> Nov 12 15:48:16 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
> enable_bit=2
> Nov 12 15:48:16 computer kernel: tg3: eth0: Link is down.
> Nov 12 15:48:19 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
> Nov 12 15:48:19 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
> Nov 12 16:10:44 computer kernel: tg3: eth0: transmit timed out, resetting
> Nov 12 16:10:44 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
> MAC_RX_STATUS[00000000]
> Nov 12 16:10:44 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
> WDMAC_STATUS[00000000]
> Nov 12 16:10:44 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
> enable_bit=2
> Nov 12 16:10:44 computer kernel: tg3: eth0: Link is down.
> Nov 12 16:10:47 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
> Nov 12 16:10:47 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
> Nov 12 16:18:43 computer kernel: tg3: eth0: transmit timed out, resetting
> Nov 12 16:18:43 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
> MAC_RX_STATUS[00000000]
> Nov 12 16:18:43 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
> WDMAC_STATUS[00000000]
> Nov 12 16:18:43 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
> enable_bit=2
> Nov 12 16:18:43 computer kernel: tg3: eth0: Link is down.
> Nov 12 16:18:46 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
> Nov 12 16:18:46 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
> Nov 12 16:27:02 computer kernel: tg3: eth0: transmit timed out, resetting
> Nov 12 16:27:02 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
> MAC_RX_STATUS[00000000]
> Nov 12 16:27:02 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
> WDMAC_STATUS[00000000]
> Nov 12 16:27:02 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
> enable_bit=2
> Nov 12 16:27:02 computer kernel: tg3: eth0: Link is down.
> Nov 12 16:27:06 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
> Nov 12 16:27:06 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
> Nov 12 16:35:17 computer kernel: tg3: eth0: transmit timed out, resetting
> Nov 12 16:35:17 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
> MAC_RX_STATUS[00000000]
> Nov 12 16:35:17 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
> WDMAC_STATUS[00000000]
> Nov 12 16:35:17 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
> enable_bit=2
> Nov 12 16:35:17 computer kernel: tg3: eth0: Link is down.
> Nov 12 16:35:20 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
> Nov 12 16:35:20 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
> 
> >
> >>>> Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs
> >>>> w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor
> >>>> async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb
> >>>> wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner
> >>>> mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa
> >>>> cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat
> >>>> i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32
> >>>> videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support
> >>>> v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc
> >>>> videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc
> >>>> parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero
> >>>> dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3
> >>>> jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
> >>>> Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
> >>>> Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
> >>>> Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
> >>>> Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
> >>>> Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
> >>>> Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
> >>>> Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
> >>>> Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
> >>>> Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
> >>>> Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
> >>>> Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
> >>>> Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0
> >>>> [uhci_hcd]
> >>>> Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
> >>>> Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
> >>>> Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
> >>>> Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
> >>>> Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
> >>>> Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
> >>>> Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
> >>>> Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
> >>>> Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
> >>>> Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
> >>>> Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
> >>>> Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
> >>>> Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
> >>>> Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5
> 
> >
> 
> Offset  Value
> ------  ----------
> 0x0000  0x169d14e4
> 0x0004  0x00100006
> 0x0008  0x02000011
> 0x000c  0x00000008
> 0x0010  0xfd8f0004
> 0x0014  0x00000000
> 0x0018  0x00000000
> 0x001c  0x00000000
> 0x0020  0x00000000
> 0x0024  0x00000000
> 0x0028  0x00000007
> 0x002c  0x0cc1105b
> 0x0030  0x02aa0000
> 0x0034  0x00000048
> 0x0038  0x00000000
> 0x003c  0x0000010b
> 0x0040  0x00000000
> 0x0044  0x00000000
> 0x0048  0xc0025001
> 0x004c  0x64002000
> 0x0050  0x00d05803
> 0x0054  0x587d4b8a
> 0x0058  0x0086d005
> 0x005c  0x2a0f7b0c
> 0x0060  0x0101b810
> 0x0064  0x0000f21e
> 0x0068  0x41010298
> 0x006c  0x76180000
> 0x0070  0x000010f2
> 0x0074  0x000000c0
> 0x0078  0x00000050
> 0x007c  0x00000000
> 0x0080  0x00d05803
> 0x0084  0x00000000
> 0x0088  0x04130034
> 0x008c  0x3c081082
> 0x0090  0x01000609
> 0x0094  0x02000000
> 0x0098  0x00000000
> 0x009c  0x000000c8
> 0x00a0  0x00000000
> 0x00a4  0x00000000
> 0x00a8  0x00000000
> 0x00ac  0x00000000
> 0x00b0  0x00000000
> 0x00b4  0x00000000
> 0x00b8  0x00000000
> 0x00bc  0x00000000
> 0x00c0  0x00000000
> 0x00c4  0x00000000
> 0x00c8  0x00000000
> 0x00cc  0x00000000
> 0x00d0  0x00000000
> 0x00d4  0x00000000
> 0x00d8  0x00000000
> 0x00dc  0x00000000
> 0x00e0  0x00000000
> 0x00e4  0x00000000
> 0x00e8  0x00000000
> 0x00ec  0x00000000
> 0x00f0  0x00000000
> 0x00f4  0x00000000
> 0x00f8  0x00000000
> 0x00fc  0x00000000
> 0x0100  0x00000000
> 0x0104  0x00000000
> 0x0108  0x00000000
> 0x010c  0x00000000
> 0x0110  0x00000000
> 0x0114  0x00000000
> 0x0118  0x00000000
> 0x011c  0x00000000
> 0x0120  0x00000000
> 0x0124  0x00000000
> 0x0128  0x00000000
> 0x012c  0x00000000
> 0x0130  0x00000000
> 0x0134  0x00000000
> 0x0138  0x00000000
> 0x013c  0x00000000
> 0x0140  0x00000000
> 0x0144  0x00000000
> 0x0148  0x00000000
> 0x014c  0x00000000
> 0x0150  0x00000000
> 0x0154  0x00000000
> 0x0158  0x00000000
> 
> 0x0200  0x00000000
> 0x0204  0x02000000
> 0x0208  0x00000000
> 0x020c  0x02000000
> 0x0210  0x00000000
> 0x0214  0x02000000
> 0x0218  0x00000000
> 0x021c  0x02000000
> 0x0220  0x00000000
> 0x0224  0x02000000
> 0x0228  0x00000000
> 0x022c  0x02000000
> 0x0230  0x00000000
> 0x0234  0x02000000
> 0x0238  0x00000000
> 0x023c  0x02000000
> 0x0240  0x00000000
> 0x0244  0x02000000
> 0x0248  0x00000000
> 0x024c  0x02000000
> 0x0250  0x00000000
> 0x0254  0x02000000
> 0x0258  0x00000000
> 0x025c  0x02000000
> 0x0260  0x00000000
> 0x0264  0x02000000
> 0x0268  0x00000000
> 0x026c  0x000000c8
> 0x0270  0x00000000
> 0x0274  0x02000000
> 0x0278  0x00000000
> 0x027c  0x02000000
> 0x0280  0x00000000
> 0x0284  0x00000000
> 0x0288  0x00000000
> 0x028c  0x02000000
> 0x0290  0x00000000
> 0x0294  0x02000000
> 0x0298  0x00000000
> 0x029c  0x02000000
> 0x02a0  0x00000000
> 0x02a4  0x02000000
> 0x02a8  0x00000000
> 0x02ac  0x02000000
> 0x02b0  0x00000000
> 0x02b4  0x02000000
> 0x02b8  0x00000000
> 0x02bc  0x02000000
> 0x02c0  0x00000000
> 0x02c4  0x02000000
> 0x02c8  0x00000000
> 0x02cc  0x02000000
> 0x02d0  0x00000000
> 0x02d4  0x02000000
> 0x02d8  0x00000000
> 0x02dc  0x02000000
> 0x02e0  0x00000000
> 0x02e4  0x02000000
> 0x02e8  0x00000000
> 0x02ec  0x02000000
> 0x02f0  0x00000000
> 0x02f4  0x02000000
> 0x02f8  0x00000000
> 0x02fc  0x02000000
> 0x0300  0x00000000
> 0x0304  0x00000024
> 0x0308  0x00000000
> 0x030c  0x02000024
> 0x0310  0x00000000
> 0x0314  0x02000024
> 0x0318  0x00000000
> 0x031c  0x02000024
> 0x0320  0x00000000
> 0x0324  0x02000024
> 0x0328  0x00000000
> 0x032c  0x02000024
> 0x0330  0x00000000
> 0x0334  0x02000024
> 0x0338  0x00000000
> 0x033c  0x02000024
> 0x0340  0x00000000
> 0x0344  0x02000024
> 0x0348  0x00000000
> 0x034c  0x02000024
> 0x0350  0x00000000
> 0x0354  0x02000024
> 0x0358  0x00000000
> 0x035c  0x02000024
> 0x0360  0x00000000
> 0x0364  0x02000024
> 0x0368  0x00000000
> 0x036c  0x02000024
> 0x0370  0x00000000
> 0x0374  0x02000024
> 0x0378  0x00000000
> 0x037c  0x02000024
> 0x0380  0x00000000
> 0x0384  0x00000000
> 0x0388  0x00000000
> 0x038c  0x02000000
> 0x0390  0x00000000
> 0x0394  0x02000000
> 0x0398  0x00000000
> 0x039c  0x02000000
> 0x03a0  0x00000000
> 0x03a4  0x02000000
> 0x03a8  0x00000000
> 0x03ac  0x02000000
> 0x03b0  0x00000000
> 0x03b4  0x02000000
> 0x03b8  0x00000000
> 0x03bc  0x02000000
> 0x03c0  0x00000000
> 0x03c4  0x02000000
> 0x03c8  0x00000000
> 0x03cc  0x02000000
> 0x03d0  0x00000000
> 0x03d4  0x02000000
> 0x03d8  0x00000000
> 0x03dc  0x02000000
> 0x03e0  0x00000000
> 0x03e4  0x02000000
> 0x03e8  0x00000000
> 0x03ec  0x02000000
> 0x03f0  0x00000000
> 0x03f4  0x02000000
> 0x03f8  0x00000000
> 0x03fc  0x02000000
> 
> 0x0400  0x00e04808
> 0x0404  0x00400000
> 0x0408  0x00001000
> 0x040c  0x00000880
> 0x0410  0x00000001
> 0x0414  0x6ce59c51
> 0x0418  0x00000001
> 0x041c  0x6ce59c51
> 0x0420  0x00000001
> 0x0424  0x6ce59c51
> 0x0428  0x00000001
> 0x042c  0x6ce59c51
> 0x0430  0x00000000
> 0x0434  0x00000000
> 0x0438  0x000000da
> 0x043c  0x000005f2
> 0x0440  0x00000000
> 0x0444  0x00000000
> 0x0448  0x00000000
> 0x044c  0x0825c5e1
> 0x0450  0x00000001
> 0x0454  0x000c0000
> 0x0458  0x00000000
> 0x045c  0x00000012
> 0x0460  0x00000008
> 0x0464  0x00002620
> 0x0468  0x00000406
> 0x046c  0x00000000
> 0x0470  0x80000000
> 0x0474  0x00000000
> 0x0478  0x00000080
> 0x047c  0x40200000
> 0x0480  0x42000000
> 0x0484  0x7fffffff
> 0x0488  0x06000004
> 0x048c  0x7fffffff
> 0x0490  0x00000000
> 0x0494  0x00000000
> 0x0498  0x00000000
> 0x049c  0x00000000
> 0x04a0  0x00000000
> 0x04a4  0x00000000
> 0x04a8  0x00000000
> 0x04ac  0x00000000
> 0x04b0  0x00000000
> 0x04b4  0x00000000
> 0x04b8  0x00000000
> 0x04bc  0x00000000
> 0x04c0  0x00000000
> 0x04c4  0x00000000
> 0x04c8  0x00000000
> 0x04cc  0x00000000
> 0x04d0  0x00000000
> 0x04d4  0x00000000
> 0x04d8  0x00000000
> 0x04dc  0x00000000
> 0x04e0  0x00000000
> 0x04e4  0x00000000
> 0x04e8  0x00000000
> 0x04ec  0x00000000
> 0x04f0  0x00000000
> 0x04f4  0x00000000
> 0x04f8  0x00000000
> 0x04fc  0x00000000
> 0x0500  0x00000008
> 0x0504  0x00000002
> 0x0508  0x00000000
> 0x050c  0x00000000
> 0x0510  0x00000000
> 0x0514  0x00000000
> 0x0518  0x00000000
> 0x051c  0x00000000
> 0x0520  0x00000000
> 0x0524  0x00000000
> 0x0528  0x00000000
> 0x052c  0x00000000
> 0x0530  0x00000000
> 0x0534  0x00000000
> 0x0538  0x00000000
> 0x053c  0x00000000
> 0x0540  0x00000000
> 0x0544  0x00000000
> 0x0548  0x00000000
> 0x054c  0x00000000
> 0x0550  0x00000000
> 0x0554  0x00000000
> 0x0558  0x00000000
> 0x055c  0x00000000
> 0x0560  0x00000000
> 0x0564  0x00000000
> 0x0568  0x00000000
> 0x056c  0x00000000
> 0x0570  0x00000000
> 0x0574  0x00000000
> 0x0578  0x00000000
> 0x057c  0x00000000
> 0x0580  0x00000000
> 0x0584  0x00000000
> 0x0588  0x00000000
> 0x058c  0x00000000
> 0x0590  0x00e00000
> 0x0594  0x00000000
> 0x0598  0x00000000
> 0x059c  0x00000000
> 0x05a0  0x00000000
> 0x05a4  0x00000000
> 0x05a8  0x00000000
> 0x05ac  0x00000000
> 0x05b0  0x00000000
> 0x05b4  0x00000000
> 0x05b8  0x00000000
> 0x05bc  0x00000000
> 0x05c0  0x00000000
> 0x05c4  0x00000000
> 0x05c8  0x00000000
> 0x05cc  0x00000000
> 0x05d0  0x00000000
> 0x05d4  0x00000000
> 0x05d8  0x00000000
> 0x05dc  0x00000000
> 0x05e0  0x00000000
> 0x05e4  0x00000000
> 0x05e8  0x00000000
> 0x05ec  0x00000000
> 0x05f0  0x00000000
> 0x05f4  0x00000000
> 0x05f8  0x00000000
> 0x05fc  0x00000000
> 0x0600  0xffffffff
> 0x0604  0x00f80011
> 0x0608  0x00000000
> 0x060c  0x00001f04
> 0x0610  0xffffffff
> 0x0614  0x00000000
> 0x0618  0x07c00004
> 0x061c  0x30000000
> 0x0620  0x00000040
> 0x0624  0x00000000
> 0x0628  0x00000000
> 0x062c  0x00000000
> 0x0630  0x00000000
> 0x0634  0x00000000
> 0x0638  0x00000000
> 0x063c  0x00000000
> 0x0640  0x00000000
> 0x0644  0x00000000
> 0x0648  0x00000000
> 0x064c  0x00000000
> 0x0650  0x00000000
> 0x0654  0x00000000
> 0x0658  0x00000000
> 0x065c  0x00000000
> 0x0660  0x00000000
> 0x0664  0x00000000
> 0x0668  0x00000000
> 0x066c  0x00000000
> 0x0670  0x00000000
> 0x0674  0x00000000
> 0x0678  0x00000000
> 0x067c  0x00000000
> 0x0680  0x00000000
> 0x0684  0x00000000
> 0x0688  0x00000000
> 0x068c  0x00000000
> 0x0690  0x00000000
> 0x0694  0x00000000
> 0x0698  0x00000000
> 0x069c  0x00000000
> 0x06a0  0x00000000
> 0x06a4  0x00000000
> 0x06a8  0x00000000
> 0x06ac  0x00000000
> 0x06b0  0x00000000
> 0x06b4  0x00000000
> 0x06b8  0x00000000
> 0x06bc  0x00000000
> 0x06c0  0x00000000
> 0x06c4  0x00000000
> 0x06c8  0x00000000
> 0x06cc  0x00000000
> 0x06d0  0x00000000
> 0x06d4  0x00000000
> 0x06d8  0x00000000
> 0x06dc  0x00000000
> 0x06e0  0x00000000
> 0x06e4  0x00000000
> 0x06e8  0x00000000
> 0x06ec  0x00000000
> 0x06f0  0x00000000
> 0x06f4  0x00000000
> 0x06f8  0x00000000
> 0x06fc  0x00000000
> 0x0700  0x00000000
> 0x0704  0x00000000
> 0x0708  0x00000000
> 0x070c  0x00000000
> 0x0710  0x00000000
> 0x0714  0x00000000
> 0x0718  0x00000000
> 0x071c  0x00000000
> 0x0720  0x00000000
> 0x0724  0x00000000
> 0x0728  0x00000000
> 0x072c  0x00000000
> 0x0730  0x00000000
> 0x0734  0x00000000
> 0x0738  0x00000000
> 0x073c  0x00000000
> 0x0740  0x00000000
> 0x0744  0x00000000
> 0x0748  0x00000000
> 0x074c  0x00000000
> 0x0750  0x00000000
> 0x0754  0x00000000
> 0x0758  0x00000000
> 0x075c  0x00000000
> 0x0760  0x00000000
> 0x0764  0x00000000
> 0x0768  0x00000000
> 0x076c  0x00000000
> 0x0770  0x00000000
> 0x0774  0x00000000
> 0x0778  0x00000000
> 0x077c  0x00000000
> 0x0780  0x00000000
> 0x0784  0x00000000
> 0x0788  0x00000000
> 0x078c  0x00000000
> 0x0790  0x00000000
> 0x0794  0x00000000
> 0x0798  0x00000000
> 0x079c  0x00000000
> 0x07a0  0x00000000
> 0x07a4  0x00000000
> 0x07a8  0x00000000
> 0x07ac  0x00000000
> 0x07b0  0x00000000
> 0x07b4  0x00000000
> 0x07b8  0x00000000
> 0x07bc  0x00000000
> 0x07c0  0x00000000
> 0x07c4  0x00000000
> 0x07c8  0x00000000
> 0x07cc  0x00000000
> 0x07d0  0x00000000
> 0x07d4  0x00000000
> 0x07d8  0x00000000
> 0x07dc  0x00000000
> 0x07e0  0x00000000
> 0x07e4  0x00000000
> 0x07e8  0x00000000
> 0x07ec  0x00000000
> 0x07f0  0x00000000
> 0x07f4  0x00000000
> 0x07f8  0x00000000
> 0x07fc  0x00000000
> 0x0800  0x00000000
> 0x0804  0xffffffff
> 0x0808  0x00000000
> 0x080c  0x00000000
> 0x0810  0x00000000
> 0x0814  0xffffffff
> 0x0818  0x00000000
> 0x081c  0x00000000
> 0x0820  0x00000000
> 0x0824  0x00000000
> 0x0828  0xffffffff
> 0x082c  0x00000000
> 0x0830  0x00000000
> 0x0834  0xffffffff
> 0x0838  0xffffffff
> 0x083c  0xffffffff
> 0x0840  0xffffffff
> 0x0844  0xffffffff
> 0x0848  0xffffffff
> 0x084c  0xffffffff
> 0x0850  0xffffffff
> 0x0854  0xffffffff
> 0x0858  0xffffffff
> 0x085c  0xffffffff
> 0x0860  0xffffffff
> 0x0864  0xffffffff
> 0x0868  0xffffffff
> 0x086c  0x00000000
> 0x0870  0x00000000
> 0x0874  0x00000000
> 0x0878  0x00000000
> 0x087c  0x00000000
> 0x0880  0x00000000
> 0x0884  0x00000c80
> 0x0888  0x00000000
> 0x088c  0x00000000
> 0x0890  0x00000000
> 0x0894  0x00000000
> 0x0898  0x00000000
> 0x089c  0x00000000
> 0x08a0  0x00000000
> 0x08a4  0x00000000
> 0x08a8  0x00000000
> 0x08ac  0x00000000
> 0x08b0  0x00000000
> 0x08b4  0x00000000
> 0x08b8  0x00000000
> 0x08bc  0x00000000
> 0x08c0  0x00000000
> 0x08c4  0x00000000
> 0x08c8  0x00000000
> 0x08cc  0x00000000
> 0x08d0  0x00000000
> 0x08d4  0x00000000
> 0x08d8  0x00000000
> 0x08dc  0x00000000
> 0x08e0  0x00000000
> 0x08e4  0x00000000
> 0x08e8  0x00000000
> 0x08ec  0x00000000
> 
> 0x0c00  0x0000000a
> 0x0c04  0x00000000
> 0x0c08  0x00000003
> 0x0c0c  0x00000001
> 0x0c10  0x00000000
> 0x0c14  0x00000000
> 0x0c18  0x00000000
> 0x0c1c  0x00000000
> 0x0c20  0x00000000
> 0x0c24  0x00000000
> 0x0c28  0x00000000
> 0x0c2c  0x00000000
> 0x0c30  0x00000000
> 0x0c34  0x00000000
> 0x0c38  0x00000000
> 0x0c3c  0x00000000
> 0x0c40  0x00000000
> 0x0c44  0x00000000
> 0x0c48  0x00000000
> 0x0c4c  0x00000000
> 0x0c50  0x00000000
> 0x0c54  0x00000000
> 0x0c58  0x00000000
> 0x0c5c  0x00000000
> 0x0c60  0x00000000
> 0x0c64  0x00000000
> 0x0c68  0x00000000
> 0x0c6c  0x00000000
> 0x0c70  0x00000000
> 0x0c74  0x00000000
> 0x0c78  0x00000000
> 0x0c7c  0x00000000
> 0x0c80  0x00000000
> 0x0c84  0x00000000
> 0x0c88  0x00000000
> 0x0c8c  0x00000000
> 0x0c90  0x00000000
> 0x0c94  0x00000000
> 0x0c98  0x00000000
> 0x0c9c  0x00000000
> 0x0ca0  0x00000000
> 0x0ca4  0x00000000
> 0x0ca8  0x00000000
> 0x0cac  0x00000000
> 0x0cb0  0x00000000
> 0x0cb4  0x00000000
> 0x0cb8  0x00000000
> 0x0cbc  0x00000000
> 0x0cc0  0x00000000
> 0x0cc4  0x00000000
> 0x0cc8  0x00000000
> 0x0ccc  0x00000000
> 0x0cd0  0x00000000
> 0x0cd4  0x00000000
> 0x0cd8  0x00000000
> 0x0cdc  0x00000000
> 
> 0x1000  0x00000002
> 
> 0x1400  0x00000006
> 0x1404  0x00000000
> 0x1408  0x00000000
> 0x140c  0x00000000
> 0x1410  0x00000000
> 0x1414  0x00000000
> 0x1418  0x00000000
> 0x141c  0x00000000
> 0x1420  0x00000000
> 0x1424  0x00000000
> 0x1428  0x00000000
> 0x142c  0x00000000
> 0x1430  0x00000000
> 0x1434  0x00000000
> 0x1438  0x00000000
> 0x143c  0x00000000
> 0x1440  0x00000000
> 0x1444  0x00000000
> 0x1448  0x00000000
> 0x144c  0x00000000
> 0x1450  0x00000000
> 0x1454  0x00000000
> 0x1458  0x00000000
> 0x145c  0x00000000
> 0x1460  0x00000000
> 0x1464  0x00000000
> 0x1468  0x00000000
> 0x146c  0x00000000
> 0x1470  0x00000000
> 0x1474  0x00000000
> 0x1478  0x00000000
> 0x147c  0x00000000
> 
> 0x1800  0x00000016
> 0x1804  0x00000000
> 0x1808  0x00000001
> 0x180c  0x00000000
> 0x1810  0x00000000
> 0x1814  0x00000000
> 0x1818  0x00000000
> 0x181c  0x00000000
> 0x1820  0x00000000
> 0x1824  0x00000000
> 0x1828  0x00000000
> 0x182c  0x00000000
> 0x1830  0x00000000
> 0x1834  0x00000000
> 0x1838  0x00000000
> 0x183c  0x00000000
> 0x1840  0x00000000
> 0x1844  0x00000000
> 
> 0x1c00  0x00000002
> 
> 0x2000  0x00000002
> 0x2004  0x00000000
> 0x2008  0x00000000
> 0x200c  0x00000000
> 0x2010  0x00000181
> 0x2014  0x00000001
> 0x2018  0x00790407
> 0x201c  0x00000000
> 0x2020  0x00000000
> 0x2024  0x00000000
> 0x2028  0x00000000
> 0x202c  0x00000000
> 0x2030  0x00000000
> 0x2034  0x00000000
> 0x2038  0x00000000
> 0x203c  0x00000000
> 0x2040  0x00000000
> 0x2044  0x00000000
> 0x2048  0x00000000
> 0x204c  0x00000000
> 0x2050  0x00000000
> 0x2054  0x00000000
> 0x2058  0x00000000
> 0x205c  0x00000000
> 0x2060  0x00000000
> 0x2064  0x00000000
> 0x2068  0x00000000
> 0x206c  0x00000000
> 0x2070  0x00000000
> 0x2074  0x00000000
> 0x2078  0x00000000
> 0x207c  0x00000000
> 0x2080  0x00000000
> 0x2084  0x00000000
> 0x2088  0x00000000
> 0x208c  0x00000000
> 0x2090  0x00000000
> 0x2094  0x00000000
> 0x2098  0x00000000
> 0x209c  0x00000000
> 0x20a0  0x00000000
> 0x20a4  0x00000000
> 0x20a8  0x00000000
> 0x20ac  0x00000000
> 0x20b0  0x00000000
> 0x20b4  0x00000000
> 0x20b8  0x00000000
> 0x20bc  0x00000000
> 0x20c0  0x00000000
> 0x20c4  0x00000000
> 0x20c8  0x00000000
> 0x20cc  0x00000000
> 0x20d0  0x00000000
> 0x20d4  0x00000000
> 0x20d8  0x00000000
> 0x20dc  0x00000000
> 0x20e0  0x00000000
> 0x20e4  0x00000000
> 0x20e8  0x00000000
> 0x20ec  0x00000000
> 0x20f0  0x00000000
> 0x20f4  0x00000000
> 0x20f8  0x00000000
> 0x20fc  0x00000000
> 0x2100  0x00086633
> 0x2104  0x00090683
> 0x2108  0x0000004d
> 0x210c  0x00000000
> 0x2110  0x00086633
> 0x2114  0x00090683
> 0x2118  0x0000004d
> 0x211c  0x00000000
> 0x2120  0x00086633
> 0x2124  0x00090683
> 0x2128  0x0000004d
> 0x212c  0x00000000
> 0x2130  0x00086633
> 0x2134  0x00090683
> 0x2138  0x0000004d
> 0x213c  0x00000000
> 0x2140  0x00086633
> 0x2144  0x00090683
> 0x2148  0x0000004d
> 0x214c  0x00000000
> 0x2150  0x00086633
> 0x2154  0x00090683
> 0x2158  0x0000004d
> 0x215c  0x00000000
> 0x2160  0x00086633
> 0x2164  0x00090683
> 0x2168  0x0000004d
> 0x216c  0x00000000
> 0x2170  0x00086633
> 0x2174  0x00090683
> 0x2178  0x0000004d
> 0x217c  0x00000000
> 0x2180  0x00086633
> 0x2184  0x00090683
> 0x2188  0x0000004d
> 0x218c  0x00000000
> 0x2190  0x00086633
> 0x2194  0x00090683
> 0x2198  0x0000004d
> 0x219c  0x00000000
> 0x21a0  0x00086633
> 0x21a4  0x00090683
> 0x21a8  0x0000004d
> 0x21ac  0x00000000
> 0x21b0  0x00086633
> 0x21b4  0x00090683
> 0x21b8  0x0000004d
> 0x21bc  0x00000000
> 0x21c0  0x00086633
> 0x21c4  0x00090683
> 0x21c8  0x0000004d
> 0x21cc  0x00000000
> 0x21d0  0x00086633
> 0x21d4  0x00090683
> 0x21d8  0x0000004d
> 0x21dc  0x00000000
> 0x21e0  0x00086633
> 0x21e4  0x00090683
> 0x21e8  0x0000004d
> 0x21ec  0x00000000
> 0x21f0  0x00086633
> 0x21f4  0x00090683
> 0x21f8  0x0000004d
> 0x21fc  0x00000000
> 0x2200  0x00000037
> 0x2204  0x00000000
> 0x2208  0x00000000
> 0x220c  0x00000000
> 0x2210  0x00000000
> 0x2214  0x00000000
> 0x2218  0x00000000
> 0x221c  0x00000000
> 0x2220  0x00000000
> 0x2224  0x00000000
> 0x2228  0x00000000
> 0x222c  0x00000000
> 0x2230  0x00000000
> 0x2234  0x00000000
> 0x2238  0x00000000
> 0x223c  0x00000000
> 0x2240  0x00000000
> 0x2244  0x00000000
> 0x2248  0x00000000
> 0x224c  0x00000000
> 0x2250  0x00000000
> 0x2254  0x00000000
> 0x2258  0x00000000
> 
> 0x2400  0x00000012
> 0x2404  0x00000000
> 0x2408  0x00000000
> 0x240c  0x00000000
> 0x2410  0x00000000
> 0x2414  0x00000000
> 0x2418  0x00000000
> 0x241c  0x00000000
> 0x2420  0x00000000
> 0x2424  0x00000000
> 0x2428  0x00000000
> 0x242c  0x00000000
> 0x2430  0x00000000
> 0x2434  0x00000000
> 0x2438  0x00000000
> 0x243c  0x00000000
> 0x2440  0x00000000
> 0x2444  0x00000000
> 0x2448  0x00000000
> 0x244c  0x00000000
> 0x2450  0x00000000
> 0x2454  0x030d4000
> 0x2458  0x02000000
> 0x245c  0x00006000
> 0x2460  0x00000000
> 0x2464  0x00000000
> 0x2468  0x00000000
> 0x246c  0x00000000
> 0x2470  0x00000000
> 0x2474  0x00000000
> 0x2478  0x00000000
> 0x247c  0x00000000
> 0x2480  0x00000000
> 0x2484  0x00000000
> 0x2488  0x00000000
> 0x248c  0x00000000
> 0x2490  0x00000000
> 0x2494  0x00000000
> 0x2498  0x00000000
> 0x249c  0x00000000
> 0x24a0  0x00000000
> 0x24a4  0x00000000
> 0x24a8  0x00000000
> 0x24ac  0x00000000
> 0x24b0  0x00000000
> 0x24b4  0x00000000
> 0x24b8  0x00000000
> 0x24bc  0x00000000
> 0x24c0  0x08000010
> 
> 0x2800  0x00000006
> 
> 0x2c00  0x00000006
> 0x2c04  0x00000000
> 0x2c08  0x00000000
> 0x2c0c  0x0000007f
> 0x2c10  0x00000000
> 0x2c14  0x00000000
> 0x2c18  0x00000008
> 0x2c1c  0x00000000
> 
> 0x3000  0x00000006
> 0x3004  0x00000000
> 0x3008  0x00000000
> 0x300c  0x00000000
> 0x3010  0x00000000
> 
> 0x3400  0x00000000
> 0x3404  0x00000000
> 
> 0x3800  0x00000000
> 0x3804  0x00000000
> 
> 0x3c00  0x00000302
> 0x3c04  0x00000000
> 0x3c08  0x00000064
> 0x3c0c  0x00000048
> 0x3c10  0x0000000a
> 0x3c14  0x00000035
> 0x3c18  0x00000000
> 0x3c1c  0x00000000
> 0x3c20  0x0000000a
> 0x3c24  0x00000005
> 0x3c28  0x00000000
> 0x3c2c  0x00000000
> 0x3c30  0x00000000
> 0x3c34  0x00000000
> 0x3c38  0x00000000
> 0x3c3c  0x03270000
> 0x3c40  0x00000000
> 0x3c44  0x00000b00
> 0x3c48  0x00000000
> 0x3c4c  0x00000000
> 0x3c50  0x00000000
> 0x3c54  0x00000000
> 0x3c58  0x00000000
> 0x3c5c  0x00000000
> 0x3c60  0x00000000
> 0x3c64  0x00000000
> 0x3c68  0x00000000
> 0x3c6c  0x00000000
> 0x3c70  0x00000000
> 0x3c74  0x00000000
> 0x3c78  0x00000000
> 0x3c7c  0x00000000
> 0x3c80  0x00000000
> 0x3c84  0x00000000
> 0x3c88  0x00000000
> 0x3c8c  0x00000000
> 0x3c90  0x00000000
> 0x3c94  0x00000000
> 0x3c98  0x00000000
> 0x3c9c  0x00000000
> 0x3ca0  0x00000000
> 0x3ca4  0x00000000
> 0x3ca8  0x00000000
> 0x3cac  0x00000000
> 0x3cb0  0x00000000
> 0x3cb4  0x00000000
> 0x3cb8  0x00000000
> 0x3cbc  0x00000000
> 0x3cc0  0x00000000
> 0x3cc4  0x00000000
> 0x3cc8  0x00000000
> 0x3ccc  0x00000000
> 0x3cd0  0x00000000
> 0x3cd4  0x00000000
> 0x3cd8  0x00000000
> 0x3cdc  0x00000000
> 0x3ce0  0x00000000
> 0x3ce4  0x00000000
> 0x3ce8  0x00000000
> 0x3cec  0x00000000
> 0x3cf0  0x00000000
> 0x3cf4  0x00000000
> 0x3cf8  0x00000000
> 0x3cfc  0x00000000
> 
> 0x4000  0x00000002
> 0x4004  0x00000000
> 0x4008  0x000b72fd
> 0x400c  0x001d174b
> 
> 0x4400  0x00000006
> 0x4404  0x00000000
> 0x4408  0x00010f00
> 0x440c  0x0000e100
> 0x4410  0x00000000
> 0x4414  0x00000010
> 0x4418  0x00000060
> 0x441c  0x00000000
> 0x4420  0x0000003d
> 0x4424  0x00000000
> 0x4428  0x00000000
> 0x442c  0x00000000
> 0x4430  0x00000000
> 0x4434  0x00000000
> 0x4438  0x00000000
> 0x443c  0x00000000
> 0x4440  0x00000000
> 0x4444  0x00000000
> 0x4448  0x00000000
> 0x444c  0x01207401
> 0x4450  0x005e0364
> 0x4454  0x00090067
> 
> 0x4800  0x080303fe
> 0x4804  0x00000000
> 
> 0x4c00  0x000003fe
> 0x4c04  0x00000000
> 
> 0x5000  0x00009820
> 0x5004  0x80008000
> 0x5008  0x00000000
> 0x500c  0x00000000
> 0x5010  0x00000000
> 0x5014  0x00000000
> 0x5018  0x00000000
> 0x501c  0x00010d24
> 0x5020  0x00000000
> 0x5024  0x00000000
> 0x5028  0x00000000
> 0x502c  0x00000000
> 0x5030  0x00000000
> 0x5034  0x0000001d
> 0x5038  0x00000000
> 0x503c  0x00000000
> 0x5040  0x00000000
> 0x5044  0x00000000
> 0x5048  0x00000000
> 0x504c  0x00000000
> 0x5050  0x00000000
> 0x5054  0x00000000
> 0x5058  0x00000000
> 0x505c  0x00000000
> 0x5060  0x00000000
> 0x5064  0x00000000
> 0x5068  0x00000000
> 0x506c  0x00000000
> 0x5070  0x00000000
> 0x5074  0x00000000
> 0x5078  0x00000000
> 0x507c  0x00000000
> 0x5080  0x00000000
> 0x5084  0x00000000
> 0x5088  0x00000000
> 0x508c  0x00000000
> 0x5090  0x00000000
> 0x5094  0x00000000
> 0x5098  0x00000000
> 0x509c  0x00000000
> 0x50a0  0x00000000
> 0x50a4  0x00000000
> 0x50a8  0x00000000
> 0x50ac  0x00000000
> 0x50b0  0x00000000
> 0x50b4  0x00000000
> 0x50b8  0x00000000
> 0x50bc  0x00000000
> 0x50c0  0x00000000
> 0x50c4  0x00000000
> 0x50c8  0x00000000
> 0x50cc  0x00000000
> 0x50d0  0x00000000
> 0x50d4  0x00000000
> 0x50d8  0x00000000
> 0x50dc  0x00000000
> 0x50e0  0x00000000
> 0x50e4  0x00000000
> 0x50e8  0x00000000
> 0x50ec  0x00000000
> 0x50f0  0x00000000
> 0x50f4  0x00000000
> 0x50f8  0x00000000
> 0x50fc  0x00000000
> 0x5100  0x00000000
> 0x5104  0x00000000
> 0x5108  0x00000000
> 0x510c  0x00000000
> 0x5110  0x00000000
> 0x5114  0x00000000
> 0x5118  0x00000000
> 0x511c  0x00000000
> 0x5120  0x00000000
> 0x5124  0x00000000
> 0x5128  0x00000000
> 0x512c  0x00000000
> 0x5130  0x00000000
> 0x5134  0x00000000
> 0x5138  0x00000000
> 0x513c  0x00000000
> 0x5140  0x00000000
> 0x5144  0x00000000
> 0x5148  0x00000000
> 0x514c  0x00000000
> 0x5150  0x00000000
> 0x5154  0x00000000
> 0x5158  0x00000000
> 0x515c  0x00000000
> 0x5160  0x00000000
> 0x5164  0x00000000
> 0x5168  0x00000000
> 0x516c  0x00000000
> 0x5170  0x00000000
> 0x5174  0x00000000
> 0x5178  0x00000000
> 0x517c  0x00000000
> 0x5180  0x00000000
> 0x5184  0x00000000
> 0x5188  0x00000000
> 0x518c  0x00000000
> 0x5190  0x00000000
> 0x5194  0x00000000
> 0x5198  0x00000000
> 0x519c  0x00000000
> 0x51a0  0x00000000
> 0x51a4  0x00000000
> 0x51a8  0x00000000
> 0x51ac  0x00000000
> 0x51b0  0x00000000
> 0x51b4  0x00000000
> 0x51b8  0x00000000
> 0x51bc  0x00000000
> 0x51c0  0x00000000
> 0x51c4  0x00000000
> 0x51c8  0x00000000
> 0x51cc  0x00000000
> 0x51d0  0x00000000
> 0x51d4  0x00000000
> 0x51d8  0x00000000
> 0x51dc  0x00000000
> 0x51e0  0x00000000
> 0x51e4  0x00000000
> 0x51e8  0x00000000
> 0x51ec  0x00000000
> 0x51f0  0x00000000
> 0x51f4  0x00000000
> 0x51f8  0x00000000
> 0x51fc  0x00000000
> 0x5200  0x00000000
> 0x5204  0x00000000
> 0x5208  0x00000000
> 0x520c  0x00000000
> 0x5210  0x00000000
> 0x5214  0x00000000
> 0x5218  0x00000000
> 0x521c  0x00000000
> 0x5220  0x00000000
> 0x5224  0x00000000
> 0x5228  0x00000000
> 0x522c  0x00000000
> 0x5230  0x00000000
> 0x5234  0x00000000
> 0x5238  0x00000000
> 0x523c  0x00000000
> 0x5240  0x00000000
> 0x5244  0x00000000
> 0x5248  0x00000000
> 0x524c  0x00000000
> 0x5250  0x00000000
> 0x5254  0x00000000
> 0x5258  0x00000000
> 0x525c  0x00000000
> 0x5260  0x00000000
> 0x5264  0x00000000
> 0x5268  0x00000000
> 0x526c  0x00000000
> 0x5270  0x00000000
> 0x5274  0x00000000
> 0x5278  0x00000000
> 0x527c  0x00000000
> 
> 0x5400  0x00000000
> 0x5404  0x00000000
> 0x5408  0x00000000
> 0x540c  0x00000000
> 0x5410  0x00000000
> 0x5414  0x00000000
> 0x5418  0x00000000
> 0x541c  0x00000000
> 0x5420  0x00000000
> 0x5424  0x00000000
> 0x5428  0x00000000
> 0x542c  0x00000000
> 0x5430  0x00000000
> 0x5434  0x00000000
> 0x5438  0x00000000
> 0x543c  0x00000000
> 0x5440  0x00000000
> 0x5444  0x00000000
> 0x5448  0x00000000
> 0x544c  0x00000000
> 0x5450  0x00000000
> 0x5454  0x00000000
> 0x5458  0x00000000
> 0x545c  0x00000000
> 0x5460  0x00000000
> 0x5464  0x00000000
> 0x5468  0x00000000
> 0x546c  0x00000000
> 0x5470  0x00000000
> 0x5474  0x00000000
> 0x5478  0x00000000
> 0x547c  0x00000000
> 0x5480  0x00000000
> 0x5484  0x00000000
> 0x5488  0x00000000
> 0x548c  0x00000000
> 0x5490  0x00000000
> 0x5494  0x00000000
> 0x5498  0x00000000
> 0x549c  0x00000000
> 0x54a0  0x00000000
> 0x54a4  0x00000000
> 0x54a8  0x00000000
> 0x54ac  0x00000000
> 0x54b0  0x00000000
> 0x54b4  0x00000000
> 0x54b8  0x00000000
> 0x54bc  0x00000000
> 0x54c0  0x00000000
> 0x54c4  0x00000000
> 0x54c8  0x00000000
> 0x54cc  0x00000000
> 0x54d0  0x00000000
> 0x54d4  0x00000000
> 0x54d8  0x00000000
> 0x54dc  0x00000000
> 0x54e0  0x00000000
> 0x54e4  0x00000000
> 0x54e8  0x00000000
> 0x54ec  0x00000000
> 0x54f0  0x00000000
> 0x54f4  0x00000000
> 0x54f8  0x00000000
> 0x54fc  0x00000000
> 0x5500  0x00000000
> 0x5504  0x00000000
> 0x5508  0x00000000
> 0x550c  0x00000000
> 0x5510  0x00000000
> 0x5514  0x00000000
> 0x5518  0x00000000
> 0x551c  0x00000000
> 0x5520  0x00000000
> 0x5524  0x00000000
> 0x5528  0x00000000
> 0x552c  0x00000000
> 0x5530  0x00000000
> 0x5534  0x00000000
> 0x5538  0x00000000
> 0x553c  0x00000000
> 0x5540  0x00000000
> 0x5544  0x00000000
> 0x5548  0x00000000
> 0x554c  0x00000000
> 0x5550  0x00000000
> 0x5554  0x00000000
> 0x5558  0x00000000
> 0x555c  0x00000000
> 0x5560  0x00000000
> 0x5564  0x00000000
> 0x5568  0x00000000
> 0x556c  0x00000000
> 0x5570  0x00000000
> 0x5574  0x00000000
> 0x5578  0x00000000
> 0x557c  0x00000000
> 0x5580  0x00000000
> 0x5584  0x00000000
> 0x5588  0x00000000
> 0x558c  0x00000000
> 0x5590  0x00000000
> 0x5594  0x00000000
> 0x5598  0x00000000
> 0x559c  0x00000000
> 0x55a0  0x00000000
> 0x55a4  0x00000000
> 0x55a8  0x00000000
> 0x55ac  0x00000000
> 0x55b0  0x00000000
> 0x55b4  0x00000000
> 0x55b8  0x00000000
> 0x55bc  0x00000000
> 0x55c0  0x00000000
> 0x55c4  0x00000000
> 0x55c8  0x00000000
> 0x55cc  0x00000000
> 0x55d0  0x00000000
> 0x55d4  0x00000000
> 0x55d8  0x00000000
> 0x55dc  0x00000000
> 0x55e0  0x00000000
> 0x55e4  0x00000000
> 0x55e8  0x00000000
> 0x55ec  0x00000000
> 0x55f0  0x00000000
> 0x55f4  0x00000000
> 0x55f8  0x00000000
> 0x55fc  0x00000000
> 0x5600  0x00000000
> 0x5604  0x00000000
> 0x5608  0x00000000
> 0x560c  0x00000000
> 0x5610  0x00000000
> 0x5614  0x00000000
> 0x5618  0x00000000
> 0x561c  0x00000000
> 0x5620  0x00000000
> 0x5624  0x00000000
> 0x5628  0x00000000
> 0x562c  0x00000000
> 0x5630  0x00000000
> 0x5634  0x00000000
> 0x5638  0x00000000
> 0x563c  0x00000000
> 0x5640  0x00000000
> 0x5644  0x00000000
> 0x5648  0x00000000
> 0x564c  0x00000000
> 0x5650  0x00000000
> 0x5654  0x00000000
> 0x5658  0x00000000
> 0x565c  0x00000000
> 0x5660  0x00000000
> 0x5664  0x00000000
> 0x5668  0x00000000
> 0x566c  0x00000000
> 0x5670  0x00000000
> 0x5674  0x00000000
> 0x5678  0x00000000
> 0x567c  0x00000000
> 
> 0x5800  0x00000000
> 0x5804  0x02000000
> 0x5808  0x00000000
> 0x580c  0x02000000
> 0x5810  0x00000000
> 0x5814  0x02000000
> 0x5818  0x00000000
> 0x581c  0x02000000
> 0x5820  0x00000000
> 0x5824  0x02000000
> 0x5828  0x00000000
> 0x582c  0x02000000
> 0x5830  0x00000000
> 0x5834  0x02000000
> 0x5838  0x00000000
> 0x583c  0x02000000
> 0x5840  0x00000000
> 0x5844  0x02000000
> 0x5848  0x00000000
> 0x584c  0x02000000
> 0x5850  0x00000000
> 0x5854  0x02000000
> 0x5858  0x00000000
> 0x585c  0x02000000
> 0x5860  0x00000000
> 0x5864  0x02000000
> 0x5868  0x00000000
> 0x586c  0x000000c8
> 0x5870  0x00000000
> 0x5874  0x02000000
> 0x5878  0x00000000
> 0x587c  0x02000000
> 0x5880  0x00000000
> 0x5884  0x00000000
> 0x5888  0x00000000
> 0x588c  0x02000000
> 0x5890  0x00000000
> 0x5894  0x02000000
> 0x5898  0x00000000
> 0x589c  0x02000000
> 0x58a0  0x00000000
> 0x58a4  0x02000000
> 0x58a8  0x00000000
> 0x58ac  0x02000000
> 0x58b0  0x00000000
> 0x58b4  0x02000000
> 0x58b8  0x00000000
> 0x58bc  0x02000000
> 0x58c0  0x00000000
> 0x58c4  0x02000000
> 0x58c8  0x00000000
> 0x58cc  0x02000000
> 0x58d0  0x00000000
> 0x58d4  0x02000000
> 0x58d8  0x00000000
> 0x58dc  0x02000000
> 0x58e0  0x00000000
> 0x58e4  0x02000000
> 0x58e8  0x00000000
> 0x58ec  0x02000000
> 0x58f0  0x00000000
> 0x58f4  0x02000000
> 0x58f8  0x00000000
> 0x58fc  0x02000000
> 0x5900  0x00000000
> 0x5904  0x00000024
> 0x5908  0x00000000
> 0x590c  0x02000000
> 0x5910  0x00000000
> 0x5914  0x00000000
> 0x5918  0x00000000
> 0x591c  0x00000000
> 0x5920  0x00000000
> 0x5924  0x00000000
> 0x5928  0x00000000
> 0x592c  0x00000000
> 0x5930  0x00000000
> 0x5934  0x00000000
> 0x5938  0x00000000
> 0x593c  0x00000000
> 0x5940  0x00000000
> 0x5944  0x00000000
> 0x5948  0x00000000
> 0x594c  0x00000000
> 0x5950  0x00000000
> 0x5954  0x00000000
> 0x5958  0x00000000
> 0x595c  0x00000000
> 0x5960  0x00000000
> 0x5964  0x00000000
> 0x5968  0x00000000
> 0x596c  0x00000000
> 0x5970  0x00000000
> 0x5974  0x00000000
> 0x5978  0x00000000
> 0x597c  0x00000000
> 0x5980  0x00000000
> 0x5984  0x00000000
> 0x5988  0x00000000
> 0x598c  0x00000000
> 0x5990  0x00000000
> 0x5994  0x00000000
> 0x5998  0x00000000
> 0x599c  0x00000000
> 0x59a0  0x00000000
> 0x59a4  0x00000000
> 0x59a8  0x00000000
> 0x59ac  0x00000000
> 0x59b0  0x00000000
> 0x59b4  0x00000000
> 0x59b8  0x00000000
> 0x59bc  0x00000000
> 0x59c0  0x00000000
> 0x59c4  0x00000000
> 0x59c8  0x00000000
> 0x59cc  0x00000000
> 0x59d0  0x00000000
> 0x59d4  0x00000000
> 0x59d8  0x00000000
> 0x59dc  0x00000000
> 0x59e0  0x00000000
> 0x59e4  0x00000000
> 0x59e8  0x00000000
> 0x59ec  0x00000000
> 0x59f0  0x00000000
> 0x59f4  0x00000000
> 0x59f8  0x00000000
> 0x59fc  0x00000000
> 0x5a00  0x00000000
> 0x5a04  0x00000000
> 0x5a08  0x00000000
> 0x5a0c  0x00000000
> 
> 0x5c00  0x00000000
> 0x5c04  0x00000000
> 0x5c08  0x00000000
> 0x5c0c  0x00000000
> 0x5c10  0x00000000
> 0x5c14  0x00000000
> 0x5c18  0x00000000
> 0x5c1c  0x00000000
> 0x5c20  0x00000000
> 0x5c24  0x00000000
> 0x5c28  0x00000000
> 0x5c2c  0x00000000
> 0x5c30  0x00000000
> 0x5c34  0x00000000
> 0x5c38  0x00000000
> 0x5c3c  0x00000000
> 0x5c40  0x00000000
> 0x5c44  0x00000000
> 0x5c48  0x00000000
> 0x5c4c  0x00000000
> 0x5c50  0x00000000
> 0x5c54  0x00000000
> 0x5c58  0x00000000
> 0x5c5c  0x00000000
> 0x5c60  0x00000000
> 0x5c64  0x00000000
> 0x5c68  0x00000000
> 0x5c6c  0x00000000
> 0x5c70  0x00000000
> 0x5c74  0x00000000
> 0x5c78  0x00000000
> 0x5c7c  0x00000000
> 0x5c80  0x00000000
> 0x5c84  0x00000000
> 0x5c88  0x00000000
> 0x5c8c  0x00000000
> 0x5c90  0x00000000
> 0x5c94  0x00000000
> 0x5c98  0x00000000
> 0x5c9c  0x00000000
> 0x5ca0  0x00000000
> 0x5ca4  0x00000000
> 0x5ca8  0x00000000
> 0x5cac  0x00000000
> 0x5cb0  0x00000000
> 0x5cb4  0x00000000
> 0x5cb8  0x00000000
> 0x5cbc  0x00000000
> 0x5cc0  0x00000000
> 0x5cc4  0x00000000
> 0x5cc8  0x00000000
> 0x5ccc  0x00000000
> 0x5cd0  0x00000000
> 0x5cd4  0x00000000
> 0x5cd8  0x00000000
> 0x5cdc  0x00000000
> 0x5ce0  0x00000000
> 0x5ce4  0x00000000
> 0x5ce8  0x00000000
> 0x5cec  0x00000000
> 0x5cf0  0x00000000
> 0x5cf4  0x00000000
> 0x5cf8  0x00000000
> 0x5cfc  0x00000000
> 0x5d00  0x00000000
> 0x5d04  0x00000000
> 0x5d08  0x00000000
> 0x5d0c  0x00000000
> 0x5d10  0x00000000
> 0x5d14  0x00000000
> 0x5d18  0x00000000
> 0x5d1c  0x00000000
> 
> 0x6000  0x00000000
> 0x6004  0x00000000
> 0x6008  0x00000004
> 
> 0x6800  0x04130034
> 0x6804  0x3c081082
> 0x6808  0x01000609
> 0x680c  0x16695a00
> 0x6810  0x81020010
> 0x6814  0xffffffff
> 0x6818  0x00000000
> 0x681c  0x00000000
> 0x6820  0x00000000
> 0x6824  0x00000000
> 0x6828  0x00000000
> 0x682c  0x00000000
> 0x6830  0xfffc3ccf
> 0x6834  0xfffc0fff
> 0x6838  0x00000000
> 0x683c  0x00000000
> 0x6840  0x00000004
> 0x6844  0x00000000
> 
> 0x7000  0x00000188
> 0x7004  0x00000000
> 0x7008  0x00000000
> 0x700c  0x000000c0
> 0x7010  0x0a000064
> 0x7014  0x02008273
> 0x7018  0x00570081
> 0x701c  0x68848353
> 0x7020  0x00000000
> 0x7024  0x00000000
> 0x7028  0x00000000
> 0x702c  0x00000000
> 0x7030  0x00000000
> 
> 0x7c00  0x00000000
> 0x7c04  0x00000000
> 0x7c08  0x00000000
> 0x7c0c  0x00000000
> 0x7c10  0x00000000
> 0x7c14  0x00000000
> 0x7c18  0x00000000
> 0x7c1c  0x00000000
> 0x7c20  0x00000000
> 0x7c24  0x00000000
> 0x7c28  0x00000000
> 0x7c2c  0x00000000
> 0x7c30  0x00000000
> 0x7c34  0x00000000
> 0x7c38  0x00000000
> 0x7c3c  0x00000000
> 0x7c40  0x00000000
> 0x7c44  0x00000000
> 0x7c48  0x00000000
> 0x7c4c  0x00000000
> 0x7c50  0x00000000
> 0x7c54  0x00000000
> 0x7c58  0x00000000
> 0x7c5c  0x00000000
> 0x7c60  0x00000000
> 0x7c64  0x00000000
> 0x7c68  0x00000000
> 0x7c6c  0x00000000
> 0x7c70  0x00000000
> 0x7c74  0x00000000
> 0x7c78  0x00000000
> 0x7c7c  0x00000000
> 0x7c80  0x00000000
> 0x7c84  0x00000000
> 0x7c88  0x00000000
> 0x7c8c  0x00000000
> 0x7c90  0x00000000
> 0x7c94  0x00000000
> 0x7c98  0x00000000
> 0x7c9c  0x00000000
> 0x7ca0  0x00000000
> 0x7ca4  0x00000000
> 0x7ca8  0x00000000
> 0x7cac  0x00000000
> 0x7cb0  0x00000000
> 0x7cb4  0x00000000
> 0x7cb8  0x00000000
> 0x7cbc  0x00000000
> 0x7cc0  0x00000000
> 0x7cc4  0x00000000
> 0x7cc8  0x00000000
> 0x7ccc  0x00000000
> 0x7cd0  0x00000000
> 0x7cd4  0x00000000
> 0x7cd8  0x00000000
> 0x7cdc  0x00000000
> 0x7ce0  0x00000000
> 0x7ce4  0x00000000
> 0x7ce8  0x00000000
> 0x7cec  0x00000000
> 0x7cf0  0x00000000
> 0x7cf4  0x00000000
> 0x7cf8  0x00000000
> 0x7cfc  0x00000000
> 0x7d00  0x00000000
> 0x7d04  0x00000000
> 0x7d08  0x00000000
> 0x7d0c  0x00000000
> 0x7d10  0x00000000
> 0x7d14  0x00000000
> 0x7d18  0x00000000
> 0x7d1c  0x00000000
> 0x7d20  0x00000000
> 0x7d24  0x00000000
> 0x7d28  0x00000000
> 0x7d2c  0x00000000
> 0x7d30  0x00000000
> 0x7d34  0x00000000
> 0x7d38  0x00000000
> 0x7d3c  0x00000000
> 0x7d40  0x00000000
> 0x7d44  0x00000000
> 0x7d48  0x00000000
> 0x7d4c  0x00000000
> 0x7d50  0x00000000
> 0x7d54  0x00000000
> 0x7d58  0x00000000
> 0x7d5c  0x00000000
> 0x7d60  0x00000000
> 0x7d64  0x00000000
> 0x7d68  0x00000000
> 0x7d6c  0x00000000
> 0x7d70  0x00000000
> 0x7d74  0x00000000
> 0x7d78  0x00000000
> 0x7d7c  0x00000000
> 0x7d80  0x00000000
> 0x7d84  0x00000000
> 0x7d88  0x00000000
> 0x7d8c  0x00000000
> 0x7d90  0x00000000
> 0x7d94  0x00000000
> 0x7d98  0x00000000
> 0x7d9c  0x00000000
> 0x7da0  0x00000000
> 0x7da4  0x00000000
> 0x7da8  0x00000000
> 0x7dac  0x00000000
> 0x7db0  0x00000000
> 0x7db4  0x00000000
> 0x7db8  0x00000000
> 0x7dbc  0x00000000
> 0x7dc0  0x00000000
> 0x7dc4  0x00000000
> 0x7dc8  0x00000000
> 0x7dcc  0x00000000
> 0x7dd0  0x00000000
> 0x7dd4  0x00000000
> 0x7dd8  0x00000000
> 0x7ddc  0x00000000
> 0x7de0  0x00000000
> 0x7de4  0x00000000
> 0x7de8  0x00000000
> 0x7dec  0x00000000
> 0x7df0  0x00000000
> 0x7df4  0x00000000
> 0x7df8  0x00000000
> 0x7dfc  0x00000000
> 0x7e00  0x00000000
> 0x7e04  0x00000000
> 0x7e08  0x00000000
> 0x7e0c  0x00000000
> 0x7e10  0x00000000
> 0x7e14  0x00000000
> 0x7e18  0x00000000
> 0x7e1c  0x00000000
> 0x7e20  0x00000000
> 0x7e24  0x00000000
> 0x7e28  0x00000000
> 0x7e2c  0x00000000
> 0x7e30  0x00000000
> 0x7e34  0x00000000
> 0x7e38  0x00000000
> 0x7e3c  0x00000000
> 
> 

Re: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

[Roger Heflin]

Matt Carlson wrote:
> On Thu, Nov 20, 2008 at 02:07:42AM -0800, Roger Heflin wrote:
>> Matt Carlson wrote:

> 
> Yes, I remember hearing something about this problem too.  That is a firmware
> problem though.  The 5789 does not have any management firmware, so that
> shouldn't be the case here.
> 

Gotcha.

>>>> If someone else runs into this issue, since I have 2 ports I would be
>>>> able to do some testing on it, right now my first port is locked up, and
>>>> the machine is running fine on the second port.
>>>>
>>>> lspci -vvv for the first (bad) port:
>>> Ah.  There it is.
>>>
>>>> 02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit
>>>> Ethernet PCI Express (rev 11)
>>>>          Subsystem: Foxconn International, Inc. Unknown device 0cc1
>>>>          Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B-
>>>>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
>>>> <MAbort- >SERR- <PERR-
>>>>          Latency: 0, Cache Line Size: 32 bytes
>>>>          Interrupt: pin A routed to IRQ 19
>>>>          Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K]
>>>>          Expansion ROM at <ignored> [disabled]
>>>>          Capabilities: [48] Power Management version 2
>>>>                  Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>>>>                  Status: D3 PME-Enable- DSel=0 DScale=1 PME-
>>>>          Capabilities: [50] Vital Product Data
>>>>          Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
>>>> Enable-
>>>>                  Address: 0101b8102a0f7b0c  Data: f21e
>>>>          Capabilities: [d0] Express Endpoint IRQ 0
>>>>                  Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
>>>>                  Device: Latency L0s <4us, L1 unlimited
>>>>                  Device: AtnBtn- AtnInd- PwrInd-
>>>>                  Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
>>>>                  Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>>>>                  Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
>>>>                  Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
>>>>                  Link: Latency L0s <2us, L1 <64us
>>>>                  Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
>>>>                  Link: Speed 2.5Gb/s, Width x1
>>>>          Capabilities: [100] Advanced Error Reporting
>>>>          Capabilities: [13c] Virtual Channel
>>> Hmmm.  No smoking gun.  Perhaps the register dump will help.
>>>
>> driver: tg3
>> version: 3.94
>> firmware-version: 5789-v3.29a
>> bus-info: 0000:02:00.0
> 
> O.K.  I'll see if I can find any problems like this in the firmware
> archives.
> 
>> tg3.c:v3.94 (August 14, 2008)
>> tg3 0000:02:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
>> tg3 0000:02:00.0: setting latency timer to 64
>> tg3 0000:05:01.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
>> tg3: eth0: Link is up at 1000 Mbps, full duplex.
>> tg3: eth0: Flow control is on for TX and on for RX.
>>
>> Right now I brought the interface back up (it is still broken) and setup a
>> network ip on it that other machines can ping.
>>
>> The registers are included at the end of the email.
> 
> O.K.  I'll pour over the dump and get back to you.
> 
> More below.
> 
>>>>>> Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
>>>>>> Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219
>>>>>> dev_watchdog+0xfe/0x17e()
>>>>>> Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
>>> Usually the tg3_tx_timeout function dumps a few registers before
>>> resetting the chip, but I don't see that here.  Have you seen any dumps
>>> since then?
>> Is this the dump?
> 
> This would be it.  Thanks.
> 
>> Nov 12 14:58:13 computer kernel: tg3: eth0: transmit timed out, resetting
>> Nov 12 14:58:13 computer kernel: tg3: DEBUG: MAC_TX_STATUS[00000008]
>> MAC_RX_STATUS[00000006]
>> Nov 12 14:58:13 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000010]
>> WDMAC_STATUS[00000000]
> 
> Here the Read DMA Status register is reporting a Read DMA PCI Parity Error.
> I've seen this before...very recently in fact.  The problem was that the
> chipset was not programmed by the BIOS correctly.  In that particular case,
> a BIOS upgrade solved the problem.  YMMV.

The board I have is a OLD board (but new to me) and I have what appears to be 
the last bios that was officially released for it, and cannot find any newer 
updates that what I have.

> 
>> Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=2c00
>> enable_bit=2
>> Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=1400
>> enable_bit=2
>> Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
>> enable_bit=2
>> Nov 12 14:58:13 computer kernel: tg3: eth0: Link is down.
>> Nov 12 14:58:16 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
>> Nov 12 14:58:16 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
>> Nov 12 15:20:37 computer kernel: tg3: eth0: transmit timed out, resetting
>> Nov 12 15:20:37 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
>> MAC_RX_STATUS[00000000]
>> Nov 12 15:20:37 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
>> WDMAC_STATUS[00000000]
> 
> Here the MAC TX Status register is reporting that the link is up, but
> the device is sending pause frames and rx is currently rx off'd.
> 
> Does the same problem happen if flow control is disabled?
> 

I have disabled flow control (live) but not rebooted yet I won't have time to 
reboot and test until sometime tomorrow.