* Re: Detected Hardware Unit Hang on Intel Wired Ethernet [not found] <CAHM4w1=i5dYy+ReqMO=m6W96CNaQ+O=jw+pKQok0nmjb5-fNAw@mail.gmail.com> @ 2012-01-04 11:01 ` Pratyush Anand 2012-01-04 19:22 ` Greg KH 0 siblings, 1 reply; 13+ messages in thread From: Pratyush Anand @ 2012-01-04 11:01 UTC (permalink / raw) To: Pratyush Anand Cc: e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci Adding PCI mailing list too, as problem is coming only when MSI is enabled. If I connect an PCIe analyzer, I see that at the time of issue MRd(64) for 32 words has been issued with a wrong 64 bit address from ethernet card to my RC. In the normal course it always issues MRd(32) only. Regards Pratyush On 1/4/2012 3:18 PM, Pratyush Anand wrote: > Hi All, > > I am trying to use PCIe based Intel PRO/1000 PT Server Adapter card on > an ARM Corterx A-9 based platform. > I am using linux 2.6.37. > > I am trying to mount my rootfile system on NFS using interface of this card. > > I see follwing issue while nfs mount > ---------------------------------------------------------------------------------------------------------------------------------------------------------- > IP-Config: Complete: > device=eth0, addr=192.168.1.10, mask=255.255.255.0, gw=255.255.255.255, > host=192.168.1.10, domain=, nis-domain=(none), > bootserver=192.168.1.1, rootserver=192.168.1.1, rootpath= > NFS:1. attempt to mount rootVFS: Mounted root (nfs filesystem) on device 0:13. > Freeing init memory: 184K > nfs: server 192.168.1.1 not responding, still trying > e1000e 0000:03:00.0: eth0: Detected Hardware Unit Hang: > TDH<40> > TDT<43> > next_to_use<43> > next_to_clean<3f> > buffer_info[next_to_clean]: > time_stamp<ffff984a> > next_to_watch<40> > jiffies<ffff9948> > next_to_watch.status<0> > MAC Status<80383> > PHY Status<792d> > PHY 1000BASE-T Status<3800> > PHY Extended Status<3000> > PCI Status<4010> > e1000e 0000:03:00.0: eth0: Detected Hardware Unit Hang: > TDH<40> > TDT<43> > next_to_use<43> > next_to_clean<3f> > buffer_info[next_to_clean]: > time_stamp<ffff984a> > next_to_watch<40> > jiffies<ffff9a10> > next_to_watch.status<0> > MAC Status<80383> > PHY Status<792d> > PHY 1000BASE-T Status<3800> > PHY Extended Status<3000> > PCI Status<4010> > e1000e 0000:03:00.0: eth0: Detected Hardware Unit Hang: > TDH<40> > TDT<43> > next_to_use<43> > next_to_clean<3f> > buffer_info[next_to_clean]: > time_stamp<ffff984a> > next_to_watch<40> > jiffies<ffff9ba0> > next_to_watch.status<0> > MAC Status<80383> > PHY Status<792d> > PHY 1000BASE-T Status<3800> > PHY Extended Status<3000> > PCI Status<4010> > e1000e 0000:03:00.0: eth0: Detected Hardware Unit Hang: > TDH<40> > TDT<43> > next_to_use<43> > next_to_clean<3f> > buffer_info[next_to_clean]: > time_stamp<ffff984a> > next_to_watch<40> > jiffies<ffff9c68> > next_to_watch.status<0> > MAC Status<80383> > PHY Status<792d> > PHY 1000BASE-T Status<3800> > PHY Extended Status<3000> > PCI Status<4010> > ------------[ cut here ]------------ > WARNING: at /data/csd_sw/spear/drives_os/pratyusha/spear/kernel/linux-2.6/net/sched/sch_generic.c:258 > dev_watchdog+0x168/0x280() > NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out > Modules linked in: > Backtrace: > [<8003f9fc>] (dump_backtrace+0x0/0x10c) from [<803ede3c>] (dump_stack+0x18/0x1c) > r6:804f3833 r5:00000102 r4:8e83dc18 r3:60000113 > [<803ede24>] (dump_stack+0x0/0x1c) from [<8005e2b8>] > (warn_slowpath_common+0x54/0x6c) > [<8005e264>] (warn_slowpath_common+0x0/0x6c) from [<8005e374>] > (warn_slowpath_fmt+0x38/0x40) > r8:00000001 r7:00000000 r6:807164c0 r5:8eaf01d4 r4:8eaf0000 > r3:00000009 > [<8005e33c>] (warn_slowpath_fmt+0x0/0x40) from [<8036d7bc>] > (dev_watchdog+0x168/0x280) > r3:8eaf0000 r2:804f3889 > [<8036d654>] (dev_watchdog+0x0/0x280) from [<80069680>] > (run_timer_softirq+0x158/0x210) > [<80069528>] (run_timer_softirq+0x0/0x210) from [<80063cb8>] > (__do_softirq+0xb8/0x160) > r8:0000000a r7:00000100 r6:80525044 r5:00000141 r4:8e83c000 > [<80063c00>] (__do_softirq+0x0/0x160) from [<80064144>] (irq_exit+0x4c/0x54) > [<800640f8>] (irq_exit+0x0/0x54) from [<80040e2c>] (ipi_timer+0x40/0x4c) > [<80040dec>] (ipi_timer+0x0/0x4c) from [<80036260>] (do_local_timer+0x5c/0x88) > r4:800348b4 r3:00001179 > [<80036204>] (do_local_timer+0x0/0x88) from [<8003b714>] (__irq_svc+0x34/0xc0) > Exception stack(0x8e83dd68 to 0x8e83ddb0) > dd60: 8054d648 60000093 8054d640 60000013 00000206 00000001 > dd80: 80585216 00000000 8054d5b8 8e83de3c 804ca994 8e83de1c 8e83dd88 8e83ddb0 > dda0: 8005f0b0 8005f588 60000013 ffffffff > r6:0000001d r5:fec80100 r4:ffffffff r3:60000013 > [<8005f1f4>] (vprintk+0x0/0x3f0) from [<803edfe0>] (printk+0x24/0x2c) > [<803edfbc>] (printk+0x0/0x2c) from [<8020db74>] (__dev_printk+0x58/0x68) > r3:8e886dc0 r2:8056e0fc r1:804b70c9 r0:804ca994 > [<8020db1c>] (__dev_printk+0x0/0x68) from [<8020ddb0>] (dev_printk+0x34/0x3c) > r6:00000040 r5:ffff984a r4:8eaf0360 > [<8020dd7c>] (dev_printk+0x0/0x3c) from [<803589fc>] (__netdev_printk+0x4c/0x94) > r3:8eaf0000 r2:804ca999 > [<803589b0>] (__netdev_printk+0x0/0x94) from [<80358b58>] (netdev_err+0x3c/0x48) > r4:8eaf0360 > [<80358b1c>] (netdev_err+0x0/0x48) from [<8028cb34>] > (e1000_print_hw_hang+0x124/0x134) > r3:00000043 r2:00000040 r1:804d85ab > [<8028ca10>] (e1000_print_hw_hang+0x0/0x134) from [<80072d6c>] > (process_one_work+0x1f0/0x324) > [<80072b7c>] (process_one_work+0x0/0x324) from [<800733a4>] > (worker_thread+0x1c0/0x300) > [<800731e4>] (worker_thread+0x0/0x300) from [<80076e04>] (kthread+0x90/0x98) > [<80076d74>] (kthread+0x0/0x98) from [<8006169c>] (do_exit+0x0/0x5f8) > r6:8006169c r5:80076d74 r4:8e831ee0 > ---[ end trace ea1efd5a579b2b9e ]--- > e1000e 0000:03:00.0: eth0: Reset adapter > e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > nfs: server 192.168.1.1 not responding, still trying > nfs: server 192.168.1.1 not responding, still trying > ---------------------------------------------------------------------------------------------------------------------------------------------------------- > > However, If I pass pci=nomsi in bootargs then it works fine. > > I see similar issue discussed earlier at following link: > http://sourceforge.net/tracker/index.php?func=detail&aid=2896629&group_id=42302&atid=447449 > > Reading above link, it seems that bug should have been resolved in linux 2.6.32. > But I still see it in 2.6.37. > Any suggestion to resolve? > > Regards > Pratyush > . > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-04 11:01 ` Detected Hardware Unit Hang on Intel Wired Ethernet Pratyush Anand @ 2012-01-04 19:22 ` Greg KH 2012-01-05 4:30 ` Pratyush Anand 0 siblings, 1 reply; 13+ messages in thread From: Greg KH @ 2012-01-04 19:22 UTC (permalink / raw) To: Pratyush Anand Cc: Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: > Adding PCI mailing list too, as problem is coming only when MSI is enabled. > > If I connect an PCIe analyzer, I see that at the time of issue > MRd(64) for 32 words has been issued with a wrong 64 bit address > from ethernet card to my RC. > In the normal course it always issues MRd(32) only. Bug in your pcie firmware controller? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-04 19:22 ` Greg KH @ 2012-01-05 4:30 ` Pratyush Anand 2012-01-05 16:46 ` Greg KH 2012-01-05 23:54 ` Dave, Tushar N 0 siblings, 2 replies; 13+ messages in thread From: Pratyush Anand @ 2012-01-05 4:30 UTC (permalink / raw) To: Greg KH Cc: Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, linux.nics On 1/5/2012 12:52 AM, Greg KH wrote: > On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: >> Adding PCI mailing list too, as problem is coming only when MSI is enabled. >> >> If I connect an PCIe analyzer, I see that at the time of issue >> MRd(64) for 32 words has been issued with a wrong 64 bit address >> from ethernet card to my RC. >> In the normal course it always issues MRd(32) only. > > Bug in your pcie firmware controller? > > . > when you say "Bug in your pcie firmware controller?", is it RC's software or EP's software? Here I am pasting a part of analyzer log converted into text. Packet(177940), is an upstream request for MSI. Whenever any device writes at address 0x58A8F8, my PCIe RC considers it as MSI and generates an interrupt. So I receive MSI interrupt correctly in my software. Also MSI controller is correctly able to point me that the interrupt is from ethernet card. Now in Packet(178010), ethernet controller sends another upstream request for MRd(64) of 32 dwords with Address(AFECEB87:A9D88B00).Since, this address does not exist in my RC's world so, an UR is returned and hence the problem occurs. Now, question is, why ethernet card is generating inbound request with such a wrong address. I have taken log of all the tx_desc->buffer_addr programmed by software in function e1000_tx_queue. None of them is 64 bit or any invalid address. _______|_______________________________________________________________________ Packet(177916) Upstream 2.5(x1) TLP(1475) Mem MWr(32)(10:00000) Length(4) _______| RequesterID(003:00:0) Tag(2) Address(0EB00200) 1st BE(1111) _______| Last BE(1111) Data(4 dwords) LCRC(0x44E0407C) _______| Time Stamp(0013 . 460 549 544 s) _______|_______________________________________________________________________ Packet(177918) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1475) _______| CRC 16(0x0EB7) Time Stamp(0013 . 460 551 144 s) _______|_______________________________________________________________________ Packet(177940) Upstream 2.5(x1) TLP(1476) Mem MWr(32)(10:00000) Length(1) _______| RequesterID(003:00:0) Tag(30) Address(0058A8F8) 1st BE(0011) _______| Last BE(0000) Data(1 dword) LCRC(0xC21F32B6) _______| Time Stamp(0013 . 460 588 544 s) _______|_______________________________________________________________________ Packet(177942) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1476) _______| CRC 16(0x69F5) Time Stamp(0013 . 460 590 088 s) _______|_______________________________________________________________________ Packet(177946) Downstream 2.5(x1) TLP(309) Mem MRd(32)(00:00000) Length(1) _______| RequesterID(002:00:0) Tag(19) Address(C01000C0) 1st BE(1111) _______| Last BE(0000) LCRC(0x91BDA1F5) Time Stamp(0013 . 460 595 936 s) _______|_______________________________________________________________________ Packet(177947) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(309) _______| CRC 16(0x25C6) Time Stamp(0013 . 460 596 368 s) _______|_______________________________________________________________________ Packet(177950) Upstream 2.5(x1) TLP(1477) Cpl CplD(10:01010) Length(1) _______| RequesterID(002:00:0) Tag(19) CompleterID(003:00:0) Status(SC) BCM(0) _______| Byte Cnt(4) Lwr Addr(0x40) Data(1 dword) LCRC(0x8FE0D922) _______| Time Stamp(0013 . 460 597 304 s) _______|_______________________________________________________________________ Packet(177952) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1477) _______| CRC 16(0xC8EE) Time Stamp(0013 . 460 598 840 s) _______|_______________________________________________________________________ Packet(177999) Downstream 2.5(x1) TLP(310) Mem MWr(32)(10:00000) Length(1) _______| RequesterID(002:00:0) Tag(0) Address(C0103818) 1st BE(1111) _______| Last BE(0000) Data(1 dword) LCRC(0xA898D9A1) _______| Time Stamp(0013 . 460 687 936 s) _______|_______________________________________________________________________ Packet(178001) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(310) _______| CRC 16(0xC6EA) Time Stamp(0013 . 460 688 384 s) _______|_______________________________________________________________________ Packet(178004) Upstream 2.5(x1) TLP(1478) Mem MRd(32)(00:00000) Length(4) _______| RequesterID(003:00:0) Tag(4) Address(0EAFB990) 1st BE(1111) _______| Last BE(1111) LCRC(0xB54722D2) Time Stamp(0013 . 460 689 312 s) _______|_______________________________________________________________________ Packet(178006) Downstream 2.5(x1) TLP(311) Cpl CplD(10:01010) Length(4) _______| RequesterID(003:00:0) Tag(4) CompleterID(002:00:0) Status(SC) BCM(0) _______| Byte Cnt(16) Lwr Addr(0x10) Data(4 dwords) LCRC(0xFE303776) _______| Time Stamp(0013 . 460 690 288 s) _______|_______________________________________________________________________ Packet(178007) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(311) _______| CRC 16(0x67F1) Time Stamp(0013 . 460 690 776 s) _______|_______________________________________________________________________ Packet(178008) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1478) _______| CRC 16(0x2BC2) Time Stamp(0013 . 460 690 824 s) _______|_______________________________________________________________________ Packet(178010) Upstream 2.5(x1) TLP(1479) Mem MRd(64)(01:00000) Length(32) _______| RequesterID(003:00:0) Tag(11) Address(AFECEB87:A9D88B00) 1st BE(1100) _______| Last BE(0011) LCRC(0x6BE341C9) Time Stamp(0013 . 460 691 680 s) _______|_______________________________________________________________________ Packet(178011) Upstream 2.5(x1) TLP(1480) Mem MRd(64)(01:00000) Length(32) _______| RequesterID(003:00:0) Tag(8) Address(AFECEB87:A9D88B7C) 1st BE(1100) _______| Last BE(0011) LCRC(0xAA5647BD) Time Stamp(0013 . 460 691 808 s) _______|_______________________________________________________________________ Packet(178012) Upstream 2.5(x1) TLP(1481) Mem MRd(64)(01:00000) Length(32) _______| RequesterID(003:00:0) Tag(9) Address(AFECEB87:A9D88BF8) 1st BE(1100) _______| Last BE(0011) LCRC(0xEEB1F63F) Time Stamp(0013 . 460 692 120 s) _______|_______________________________________________________________________ Packet(178013) Upstream 2.5(x1) TLP(1482) Mem MRd(64)(01:00000) Length(32) _______| RequesterID(003:00:0) Tag(10) Address(AFECEB87:A9D88C74) 1st BE(1100) _______| Last BE(0011) LCRC(0xA508142C) Time Stamp(0013 . 460 692 248 s) _______|_______________________________________________________________________ Packet(178014) Downstream 2.5(x1) TLP(312) Cpl Cpl(00:01010) Length(0) _______| RequesterID(003:00:0) Tag(11) CompleterID(002:00:0) Status(UR)-BAD _______| BCM(0) Byte Cnt(124) Lwr Addr(0x02) LCRC(0xCE5540D2) _______| Time Stamp(0013 . 460 692 328 s) _______|_______________________________________________________________________ Packet(178015) Downstream 2.5(x1) TLP(313) Cpl Cpl(00:01010) Length(0) _______| RequesterID(003:00:0) Tag(8) CompleterID(002:00:0) Status(UR)-BAD _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7E) LCRC(0x9FE2487D) _______| Time Stamp(0013 . 460 692 456 s) _______|_______________________________________________________________________ Packet(178016) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(312) _______| CRC 16(0x086E) Time Stamp(0013 . 460 692 760 s) _______|_______________________________________________________________________ Packet(178017) Downstream 2.5(x1) TLP(314) Cpl Cpl(00:01010) Length(0) _______| RequesterID(003:00:0) Tag(9) CompleterID(002:00:0) Status(UR)-BAD _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7A) LCRC(0x097BF4DE) _______| Time Stamp(0013 . 460 692 776 s) _______|_______________________________________________________________________ Packet(178018) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(313) _______| CRC 16(0xA975) Time Stamp(0013 . 460 692 888 s) _______|_______________________________________________________________________ Packet(178019) Downstream 2.5(x1) TLP(315) Cpl Cpl(00:01010) Length(0) _______| RequesterID(003:00:0) Tag(10) CompleterID(002:00:0) Status(UR)-BAD _______| BCM(0) Byte Cnt(124) Lwr Addr(0x76) LCRC(0x64BDF921) _______| Time Stamp(0013 . 460 692 904 s) _______|_______________________________________________________________________ Packet(178020) Upstream 2.5(x1) TLP(1483) Msg Msg(01:10000) _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) _______| Message Code(ERR_FATAL) LCRC(0xCDA53E96) _______| Time Stamp(0013 . 460 693 184 s) _______|_______________________________________________________________________ Packet(178021) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1482) _______| CRC 16(0xA771) Time Stamp(0013 . 460 693 208 s) _______|_______________________________________________________________________ Packet(178023) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(314) _______| CRC 16(0x4A59) Time Stamp(0013 . 460 693 280 s) _______|_______________________________________________________________________ Packet(178024) Upstream 2.5(x1) TLP(1484) Msg Msg(01:10000) _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) _______| Message Code(ERR_FATAL) LCRC(0x86D9ACB6) _______| Time Stamp(0013 . 460 693 312 s) _______|_______________________________________________________________________ Packet(178025) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(315) _______| CRC 16(0xEB42) Time Stamp(0013 . 460 693 408 s) _______|_______________________________________________________________________ Packet(178026) Upstream 2.5(x1) TLP(1485) Msg Msg(01:10000) _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) _______| Message Code(ERR_FATAL) LCRC(0xC5120A31) _______| Time Stamp(0013 . 460 693 632 s) _______|_______________________________________________________________________ Packet(178028) Upstream 2.5(x1) TLP(1486) Msg Msg(01:10000) _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) _______| Message Code(ERR_FATAL) LCRC(0x41499062) _______| Time Stamp(0013 . 460 693 792 s) _______|_______________________________________________________________________ Packet(178029) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1486) _______| CRC 16(0x231F) Time Stamp(0013 . 460 694 704 s) _______|_______________________________________________________________________ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-05 4:30 ` Pratyush Anand @ 2012-01-05 16:46 ` Greg KH 2012-01-05 23:54 ` Dave, Tushar N 1 sibling, 0 replies; 13+ messages in thread From: Greg KH @ 2012-01-05 16:46 UTC (permalink / raw) To: Pratyush Anand Cc: Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, linux.nics On Thu, Jan 05, 2012 at 10:00:39AM +0530, Pratyush Anand wrote: > On 1/5/2012 12:52 AM, Greg KH wrote: > >On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: > >>Adding PCI mailing list too, as problem is coming only when MSI is enabled. > >> > >>If I connect an PCIe analyzer, I see that at the time of issue > >>MRd(64) for 32 words has been issued with a wrong 64 bit address > >>from ethernet card to my RC. > >>In the normal course it always issues MRd(32) only. > > > >Bug in your pcie firmware controller? > > > >. > > > > when you say "Bug in your pcie firmware controller?", is it RC's > software or EP's software? I don't know, but I would place the bet on your motherboard controller, not the device. greg k-h ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-05 4:30 ` Pratyush Anand 2012-01-05 16:46 ` Greg KH @ 2012-01-05 23:54 ` Dave, Tushar N 2012-01-06 4:25 ` Pratyush Anand 1 sibling, 1 reply; 13+ messages in thread From: Dave, Tushar N @ 2012-01-05 23:54 UTC (permalink / raw) To: Pratyush Anand, Greg KH Cc: Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, Linux NICS, Dave, Tushar N Anand, Sorry to hear that you have this issue with card. And yeah, thanks for doing the debugging and providing the bus trace. I think we should run the debug driver that prints the HW ring details when hang occurs. I can provide you a debug driver. You can then install debug driver and also let the bus tracer running. Once the issue occurs, provide me the full dmesg output (that has HW ring details) and bus trace. Tell me which card you have, 1gig or 10gig? Which driver are you running e1000e or igb or ixgbe? Can you also provide ethtool -i ethx output. Once I know which driver, I send you debug driver. Thanks. -Tushar -----Original Message----- From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Pratyush Anand Sent: Wednesday, January 04, 2012 8:31 PM To: Greg KH Cc: Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet On 1/5/2012 12:52 AM, Greg KH wrote: > On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: >> Adding PCI mailing list too, as problem is coming only when MSI is enabled. >> >> If I connect an PCIe analyzer, I see that at the time of issue >> MRd(64) for 32 words has been issued with a wrong 64 bit address >> from ethernet card to my RC. >> In the normal course it always issues MRd(32) only. > > Bug in your pcie firmware controller? > > . > when you say "Bug in your pcie firmware controller?", is it RC's software or EP's software? Here I am pasting a part of analyzer log converted into text. Packet(177940), is an upstream request for MSI. Whenever any device writes at address 0x58A8F8, my PCIe RC considers it as MSI and generates an interrupt. So I receive MSI interrupt correctly in my software. Also MSI controller is correctly able to point me that the interrupt is from ethernet card. Now in Packet(178010), ethernet controller sends another upstream request for MRd(64) of 32 dwords with Address(AFECEB87:A9D88B00).Since, this address does not exist in my RC's world so, an UR is returned and hence the problem occurs. Now, question is, why ethernet card is generating inbound request with such a wrong address. I have taken log of all the tx_desc->buffer_addr programmed by software in function e1000_tx_queue. None of them is 64 bit or any invalid address. _______|_______________________________________________________________________ Packet(177916) Upstream 2.5(x1) TLP(1475) Mem MWr(32)(10:00000) Length(4) _______| RequesterID(003:00:0) Tag(2) Address(0EB00200) 1st BE(1111) _______| Last BE(1111) Data(4 dwords) LCRC(0x44E0407C) _______| Time Stamp(0013 . 460 549 544 s) _______|_______________________________________________________________________ Packet(177918) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1475) _______| CRC 16(0x0EB7) Time Stamp(0013 . 460 551 144 s) _______|_______________________________________________________________________ Packet(177940) Upstream 2.5(x1) TLP(1476) Mem MWr(32)(10:00000) Length(1) _______| RequesterID(003:00:0) Tag(30) Address(0058A8F8) 1st BE(0011) _______| Last BE(0000) Data(1 dword) LCRC(0xC21F32B6) _______| Time Stamp(0013 . 460 588 544 s) _______|_______________________________________________________________________ Packet(177942) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1476) _______| CRC 16(0x69F5) Time Stamp(0013 . 460 590 088 s) _______|_______________________________________________________________________ Packet(177946) Downstream 2.5(x1) TLP(309) Mem MRd(32)(00:00000) Length(1) _______| RequesterID(002:00:0) Tag(19) Address(C01000C0) 1st BE(1111) _______| Last BE(0000) LCRC(0x91BDA1F5) Time Stamp(0013 . 460 595 936 s) _______|_______________________________________________________________________ Packet(177947) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(309) _______| CRC 16(0x25C6) Time Stamp(0013 . 460 596 368 s) _______|_______________________________________________________________________ Packet(177950) Upstream 2.5(x1) TLP(1477) Cpl CplD(10:01010) Length(1) _______| RequesterID(002:00:0) Tag(19) CompleterID(003:00:0) Status(SC) BCM(0) _______| Byte Cnt(4) Lwr Addr(0x40) Data(1 dword) LCRC(0x8FE0D922) _______| Time Stamp(0013 . 460 597 304 s) _______|_______________________________________________________________________ Packet(177952) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1477) _______| CRC 16(0xC8EE) Time Stamp(0013 . 460 598 840 s) _______|_______________________________________________________________________ Packet(177999) Downstream 2.5(x1) TLP(310) Mem MWr(32)(10:00000) Length(1) _______| RequesterID(002:00:0) Tag(0) Address(C0103818) 1st BE(1111) _______| Last BE(0000) Data(1 dword) LCRC(0xA898D9A1) _______| Time Stamp(0013 . 460 687 936 s) _______|_______________________________________________________________________ Packet(178001) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(310) _______| CRC 16(0xC6EA) Time Stamp(0013 . 460 688 384 s) _______|_______________________________________________________________________ Packet(178004) Upstream 2.5(x1) TLP(1478) Mem MRd(32)(00:00000) Length(4) _______| RequesterID(003:00:0) Tag(4) Address(0EAFB990) 1st BE(1111) _______| Last BE(1111) LCRC(0xB54722D2) Time Stamp(0013 . 460 689 312 s) _______|_______________________________________________________________________ Packet(178006) Downstream 2.5(x1) TLP(311) Cpl CplD(10:01010) Length(4) _______| RequesterID(003:00:0) Tag(4) CompleterID(002:00:0) Status(SC) BCM(0) _______| Byte Cnt(16) Lwr Addr(0x10) Data(4 dwords) LCRC(0xFE303776) _______| Time Stamp(0013 . 460 690 288 s) _______|_______________________________________________________________________ Packet(178007) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(311) _______| CRC 16(0x67F1) Time Stamp(0013 . 460 690 776 s) _______|_______________________________________________________________________ Packet(178008) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1478) _______| CRC 16(0x2BC2) Time Stamp(0013 . 460 690 824 s) _______|_______________________________________________________________________ Packet(178010) Upstream 2.5(x1) TLP(1479) Mem MRd(64)(01:00000) Length(32) _______| RequesterID(003:00:0) Tag(11) Address(AFECEB87:A9D88B00) 1st BE(1100) _______| Last BE(0011) LCRC(0x6BE341C9) Time Stamp(0013 . 460 691 680 s) _______|_______________________________________________________________________ Packet(178011) Upstream 2.5(x1) TLP(1480) Mem MRd(64)(01:00000) Length(32) _______| RequesterID(003:00:0) Tag(8) Address(AFECEB87:A9D88B7C) 1st BE(1100) _______| Last BE(0011) LCRC(0xAA5647BD) Time Stamp(0013 . 460 691 808 s) _______|_______________________________________________________________________ Packet(178012) Upstream 2.5(x1) TLP(1481) Mem MRd(64)(01:00000) Length(32) _______| RequesterID(003:00:0) Tag(9) Address(AFECEB87:A9D88BF8) 1st BE(1100) _______| Last BE(0011) LCRC(0xEEB1F63F) Time Stamp(0013 . 460 692 120 s) _______|_______________________________________________________________________ Packet(178013) Upstream 2.5(x1) TLP(1482) Mem MRd(64)(01:00000) Length(32) _______| RequesterID(003:00:0) Tag(10) Address(AFECEB87:A9D88C74) 1st BE(1100) _______| Last BE(0011) LCRC(0xA508142C) Time Stamp(0013 . 460 692 248 s) _______|_______________________________________________________________________ Packet(178014) Downstream 2.5(x1) TLP(312) Cpl Cpl(00:01010) Length(0) _______| RequesterID(003:00:0) Tag(11) CompleterID(002:00:0) Status(UR)-BAD _______| BCM(0) Byte Cnt(124) Lwr Addr(0x02) LCRC(0xCE5540D2) _______| Time Stamp(0013 . 460 692 328 s) _______|_______________________________________________________________________ Packet(178015) Downstream 2.5(x1) TLP(313) Cpl Cpl(00:01010) Length(0) _______| RequesterID(003:00:0) Tag(8) CompleterID(002:00:0) Status(UR)-BAD _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7E) LCRC(0x9FE2487D) _______| Time Stamp(0013 . 460 692 456 s) _______|_______________________________________________________________________ Packet(178016) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(312) _______| CRC 16(0x086E) Time Stamp(0013 . 460 692 760 s) _______|_______________________________________________________________________ Packet(178017) Downstream 2.5(x1) TLP(314) Cpl Cpl(00:01010) Length(0) _______| RequesterID(003:00:0) Tag(9) CompleterID(002:00:0) Status(UR)-BAD _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7A) LCRC(0x097BF4DE) _______| Time Stamp(0013 . 460 692 776 s) _______|_______________________________________________________________________ Packet(178018) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(313) _______| CRC 16(0xA975) Time Stamp(0013 . 460 692 888 s) _______|_______________________________________________________________________ Packet(178019) Downstream 2.5(x1) TLP(315) Cpl Cpl(00:01010) Length(0) _______| RequesterID(003:00:0) Tag(10) CompleterID(002:00:0) Status(UR)-BAD _______| BCM(0) Byte Cnt(124) Lwr Addr(0x76) LCRC(0x64BDF921) _______| Time Stamp(0013 . 460 692 904 s) _______|_______________________________________________________________________ Packet(178020) Upstream 2.5(x1) TLP(1483) Msg Msg(01:10000) _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) _______| Message Code(ERR_FATAL) LCRC(0xCDA53E96) _______| Time Stamp(0013 . 460 693 184 s) _______|_______________________________________________________________________ Packet(178021) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1482) _______| CRC 16(0xA771) Time Stamp(0013 . 460 693 208 s) _______|_______________________________________________________________________ Packet(178023) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(314) _______| CRC 16(0x4A59) Time Stamp(0013 . 460 693 280 s) _______|_______________________________________________________________________ Packet(178024) Upstream 2.5(x1) TLP(1484) Msg Msg(01:10000) _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) _______| Message Code(ERR_FATAL) LCRC(0x86D9ACB6) _______| Time Stamp(0013 . 460 693 312 s) _______|_______________________________________________________________________ Packet(178025) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(315) _______| CRC 16(0xEB42) Time Stamp(0013 . 460 693 408 s) _______|_______________________________________________________________________ Packet(178026) Upstream 2.5(x1) TLP(1485) Msg Msg(01:10000) _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) _______| Message Code(ERR_FATAL) LCRC(0xC5120A31) _______| Time Stamp(0013 . 460 693 632 s) _______|_______________________________________________________________________ Packet(178028) Upstream 2.5(x1) TLP(1486) Msg Msg(01:10000) _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) _______| Message Code(ERR_FATAL) LCRC(0x41499062) _______| Time Stamp(0013 . 460 693 792 s) _______|_______________________________________________________________________ Packet(178029) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1486) _______| CRC 16(0x231F) Time Stamp(0013 . 460 694 704 s) _______|_______________________________________________________________________ -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-05 23:54 ` Dave, Tushar N @ 2012-01-06 4:25 ` Pratyush Anand 2012-01-06 18:55 ` Dave, Tushar N 0 siblings, 1 reply; 13+ messages in thread From: Pratyush Anand @ 2012-01-06 4:25 UTC (permalink / raw) To: Dave, Tushar N Cc: Greg KH, Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, Linux NICS Thanks Tushar, On 1/6/2012 5:24 AM, Dave, Tushar N wrote: > Anand, > > Sorry to hear that you have this issue with card. And yeah, thanks for doing the debugging and providing the bus trace. > I think we should run the debug driver that prints the HW ring details when hang occurs. I can provide you a debug driver. You can then install debug driver and also let the bus tracer running. Once the issue occurs, provide me the full dmesg output (that has HW ring details) and bus trace. > > Tell me which card you have, 1gig or 10gig? Which driver are you running e1000e or igb or ixgbe? > Can you also provide ethtool -i ethx output. > > Once I know which driver, I send you debug driver. I am using Intel PRO/1000 PT Server Adapter. http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/pro-1000-pt.html I am using e1000e driver. I see the problem when I try to mount rootfilesystem using NFS and use MSI interrupt. I see this issue even before I can have cell prompt. Please see first mail in this thread. http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg04894.html Here, you can also see tx ring details when issue occur. Please let me know, if you need any more info. Regards Pratyush > > Thanks. > > -Tushar > > -----Original Message----- > From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Pratyush Anand > Sent: Wednesday, January 04, 2012 8:31 PM > To: Greg KH > Cc: Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS > Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet > > On 1/5/2012 12:52 AM, Greg KH wrote: >> On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: >>> Adding PCI mailing list too, as problem is coming only when MSI is enabled. >>> >>> If I connect an PCIe analyzer, I see that at the time of issue >>> MRd(64) for 32 words has been issued with a wrong 64 bit address >>> from ethernet card to my RC. >>> In the normal course it always issues MRd(32) only. >> >> Bug in your pcie firmware controller? >> >> . >> > > when you say "Bug in your pcie firmware controller?", is it RC's > software or EP's software? > > Here I am pasting a part of analyzer log converted into text. > Packet(177940), is an upstream request for MSI. Whenever any device > writes at address 0x58A8F8, my PCIe RC considers it as MSI and generates > an interrupt. So I receive MSI interrupt correctly in my software. Also > MSI controller is correctly able to point me that the interrupt is from > ethernet card. > > Now in Packet(178010), ethernet controller sends another upstream > request for MRd(64) of 32 dwords with Address(AFECEB87:A9D88B00).Since, > this address does not exist in my RC's world so, an UR is returned and > hence the problem occurs. > > Now, question is, why ethernet card is generating inbound request with > such a wrong address. I have taken log of all the tx_desc->buffer_addr > programmed by software in function e1000_tx_queue. None of them is 64 > bit or any invalid address. > > _______|_______________________________________________________________________ > Packet(177916) Upstream 2.5(x1) TLP(1475) Mem MWr(32)(10:00000) Length(4) > _______| RequesterID(003:00:0) Tag(2) Address(0EB00200) 1st BE(1111) > _______| Last BE(1111) Data(4 dwords) LCRC(0x44E0407C) > _______| Time Stamp(0013 . 460 549 544 s) > _______|_______________________________________________________________________ > Packet(177918) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1475) > _______| CRC 16(0x0EB7) Time Stamp(0013 . 460 551 144 s) > _______|_______________________________________________________________________ > Packet(177940) Upstream 2.5(x1) TLP(1476) Mem MWr(32)(10:00000) Length(1) > _______| RequesterID(003:00:0) Tag(30) Address(0058A8F8) 1st BE(0011) > _______| Last BE(0000) Data(1 dword) LCRC(0xC21F32B6) > _______| Time Stamp(0013 . 460 588 544 s) > _______|_______________________________________________________________________ > Packet(177942) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1476) > _______| CRC 16(0x69F5) Time Stamp(0013 . 460 590 088 s) > _______|_______________________________________________________________________ > Packet(177946) Downstream 2.5(x1) TLP(309) Mem MRd(32)(00:00000) Length(1) > _______| RequesterID(002:00:0) Tag(19) Address(C01000C0) 1st BE(1111) > _______| Last BE(0000) LCRC(0x91BDA1F5) Time Stamp(0013 . 460 595 936 s) > _______|_______________________________________________________________________ > Packet(177947) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(309) > _______| CRC 16(0x25C6) Time Stamp(0013 . 460 596 368 s) > _______|_______________________________________________________________________ > Packet(177950) Upstream 2.5(x1) TLP(1477) Cpl CplD(10:01010) Length(1) > _______| RequesterID(002:00:0) Tag(19) CompleterID(003:00:0) Status(SC) > BCM(0) > _______| Byte Cnt(4) Lwr Addr(0x40) Data(1 dword) LCRC(0x8FE0D922) > _______| Time Stamp(0013 . 460 597 304 s) > _______|_______________________________________________________________________ > Packet(177952) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1477) > _______| CRC 16(0xC8EE) Time Stamp(0013 . 460 598 840 s) > _______|_______________________________________________________________________ > Packet(177999) Downstream 2.5(x1) TLP(310) Mem MWr(32)(10:00000) Length(1) > _______| RequesterID(002:00:0) Tag(0) Address(C0103818) 1st BE(1111) > _______| Last BE(0000) Data(1 dword) LCRC(0xA898D9A1) > _______| Time Stamp(0013 . 460 687 936 s) > _______|_______________________________________________________________________ > Packet(178001) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(310) > _______| CRC 16(0xC6EA) Time Stamp(0013 . 460 688 384 s) > _______|_______________________________________________________________________ > Packet(178004) Upstream 2.5(x1) TLP(1478) Mem MRd(32)(00:00000) Length(4) > _______| RequesterID(003:00:0) Tag(4) Address(0EAFB990) 1st BE(1111) > _______| Last BE(1111) LCRC(0xB54722D2) Time Stamp(0013 . 460 689 312 s) > _______|_______________________________________________________________________ > Packet(178006) Downstream 2.5(x1) TLP(311) Cpl CplD(10:01010) Length(4) > _______| RequesterID(003:00:0) Tag(4) CompleterID(002:00:0) Status(SC) > BCM(0) > _______| Byte Cnt(16) Lwr Addr(0x10) Data(4 dwords) LCRC(0xFE303776) > _______| Time Stamp(0013 . 460 690 288 s) > _______|_______________________________________________________________________ > Packet(178007) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(311) > _______| CRC 16(0x67F1) Time Stamp(0013 . 460 690 776 s) > _______|_______________________________________________________________________ > Packet(178008) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1478) > _______| CRC 16(0x2BC2) Time Stamp(0013 . 460 690 824 s) > _______|_______________________________________________________________________ > Packet(178010) Upstream 2.5(x1) TLP(1479) Mem MRd(64)(01:00000) Length(32) > _______| RequesterID(003:00:0) Tag(11) Address(AFECEB87:A9D88B00) 1st > BE(1100) > _______| Last BE(0011) LCRC(0x6BE341C9) Time Stamp(0013 . 460 691 680 s) > _______|_______________________________________________________________________ > Packet(178011) Upstream 2.5(x1) TLP(1480) Mem MRd(64)(01:00000) Length(32) > _______| RequesterID(003:00:0) Tag(8) Address(AFECEB87:A9D88B7C) 1st > BE(1100) > _______| Last BE(0011) LCRC(0xAA5647BD) Time Stamp(0013 . 460 691 808 s) > _______|_______________________________________________________________________ > Packet(178012) Upstream 2.5(x1) TLP(1481) Mem MRd(64)(01:00000) Length(32) > _______| RequesterID(003:00:0) Tag(9) Address(AFECEB87:A9D88BF8) 1st > BE(1100) > _______| Last BE(0011) LCRC(0xEEB1F63F) Time Stamp(0013 . 460 692 120 s) > _______|_______________________________________________________________________ > Packet(178013) Upstream 2.5(x1) TLP(1482) Mem MRd(64)(01:00000) Length(32) > _______| RequesterID(003:00:0) Tag(10) Address(AFECEB87:A9D88C74) 1st > BE(1100) > _______| Last BE(0011) LCRC(0xA508142C) Time Stamp(0013 . 460 692 248 s) > _______|_______________________________________________________________________ > Packet(178014) Downstream 2.5(x1) TLP(312) Cpl Cpl(00:01010) Length(0) > _______| RequesterID(003:00:0) Tag(11) CompleterID(002:00:0) Status(UR)-BAD > _______| BCM(0) Byte Cnt(124) Lwr Addr(0x02) LCRC(0xCE5540D2) > _______| Time Stamp(0013 . 460 692 328 s) > _______|_______________________________________________________________________ > Packet(178015) Downstream 2.5(x1) TLP(313) Cpl Cpl(00:01010) Length(0) > _______| RequesterID(003:00:0) Tag(8) CompleterID(002:00:0) Status(UR)-BAD > _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7E) LCRC(0x9FE2487D) > _______| Time Stamp(0013 . 460 692 456 s) > _______|_______________________________________________________________________ > Packet(178016) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(312) > _______| CRC 16(0x086E) Time Stamp(0013 . 460 692 760 s) > _______|_______________________________________________________________________ > Packet(178017) Downstream 2.5(x1) TLP(314) Cpl Cpl(00:01010) Length(0) > _______| RequesterID(003:00:0) Tag(9) CompleterID(002:00:0) Status(UR)-BAD > _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7A) LCRC(0x097BF4DE) > _______| Time Stamp(0013 . 460 692 776 s) > _______|_______________________________________________________________________ > Packet(178018) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(313) > _______| CRC 16(0xA975) Time Stamp(0013 . 460 692 888 s) > _______|_______________________________________________________________________ > Packet(178019) Downstream 2.5(x1) TLP(315) Cpl Cpl(00:01010) Length(0) > _______| RequesterID(003:00:0) Tag(10) CompleterID(002:00:0) Status(UR)-BAD > _______| BCM(0) Byte Cnt(124) Lwr Addr(0x76) LCRC(0x64BDF921) > _______| Time Stamp(0013 . 460 692 904 s) > _______|_______________________________________________________________________ > Packet(178020) Upstream 2.5(x1) TLP(1483) Msg Msg(01:10000) > _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) > _______| Message Code(ERR_FATAL) LCRC(0xCDA53E96) > _______| Time Stamp(0013 . 460 693 184 s) > _______|_______________________________________________________________________ > Packet(178021) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1482) > _______| CRC 16(0xA771) Time Stamp(0013 . 460 693 208 s) > _______|_______________________________________________________________________ > Packet(178023) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(314) > _______| CRC 16(0x4A59) Time Stamp(0013 . 460 693 280 s) > _______|_______________________________________________________________________ > Packet(178024) Upstream 2.5(x1) TLP(1484) Msg Msg(01:10000) > _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) > _______| Message Code(ERR_FATAL) LCRC(0x86D9ACB6) > _______| Time Stamp(0013 . 460 693 312 s) > _______|_______________________________________________________________________ > Packet(178025) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(315) > _______| CRC 16(0xEB42) Time Stamp(0013 . 460 693 408 s) > _______|_______________________________________________________________________ > Packet(178026) Upstream 2.5(x1) TLP(1485) Msg Msg(01:10000) > _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) > _______| Message Code(ERR_FATAL) LCRC(0xC5120A31) > _______| Time Stamp(0013 . 460 693 632 s) > _______|_______________________________________________________________________ > Packet(178028) Upstream 2.5(x1) TLP(1486) Msg Msg(01:10000) > _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) > _______| Message Code(ERR_FATAL) LCRC(0x41499062) > _______| Time Stamp(0013 . 460 693 792 s) > _______|_______________________________________________________________________ > Packet(178029) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1486) > _______| CRC 16(0x231F) Time Stamp(0013 . 460 694 704 s) > _______|_______________________________________________________________________ > > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > . > ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-06 4:25 ` Pratyush Anand @ 2012-01-06 18:55 ` Dave, Tushar N 2012-01-10 4:20 ` Pratyush Anand 0 siblings, 1 reply; 13+ messages in thread From: Dave, Tushar N @ 2012-01-06 18:55 UTC (permalink / raw) To: Pratyush Anand Cc: Greg KH, Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, Linux NICS, Dave, Tushar N Pratyush, Sorry I got your name reversed. Are you using in-kernel driver or one from Sourceforge. Please send me output of ethtool -i ethx. -Tushar -----Original Message----- From: Pratyush Anand [mailto:pratyush.anand@st.com] Sent: Thursday, January 05, 2012 8:25 PM To: Dave, Tushar N Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet Thanks Tushar, On 1/6/2012 5:24 AM, Dave, Tushar N wrote: > Anand, > > Sorry to hear that you have this issue with card. And yeah, thanks for doing the debugging and providing the bus trace. > I think we should run the debug driver that prints the HW ring details when hang occurs. I can provide you a debug driver. You can then install debug driver and also let the bus tracer running. Once the issue occurs, provide me the full dmesg output (that has HW ring details) and bus trace. > > Tell me which card you have, 1gig or 10gig? Which driver are you running e1000e or igb or ixgbe? > Can you also provide ethtool -i ethx output. > > Once I know which driver, I send you debug driver. I am using Intel PRO/1000 PT Server Adapter. http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/pro-1000-pt.html I am using e1000e driver. I see the problem when I try to mount rootfilesystem using NFS and use MSI interrupt. I see this issue even before I can have cell prompt. Please see first mail in this thread. http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg04894.html Here, you can also see tx ring details when issue occur. Please let me know, if you need any more info. Regards Pratyush > > Thanks. > > -Tushar > > -----Original Message----- > From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Pratyush Anand > Sent: Wednesday, January 04, 2012 8:31 PM > To: Greg KH > Cc: Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS > Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet > > On 1/5/2012 12:52 AM, Greg KH wrote: >> On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: >>> Adding PCI mailing list too, as problem is coming only when MSI is enabled. >>> >>> If I connect an PCIe analyzer, I see that at the time of issue >>> MRd(64) for 32 words has been issued with a wrong 64 bit address >>> from ethernet card to my RC. >>> In the normal course it always issues MRd(32) only. >> >> Bug in your pcie firmware controller? >> >> . >> > > when you say "Bug in your pcie firmware controller?", is it RC's > software or EP's software? > > Here I am pasting a part of analyzer log converted into text. > Packet(177940), is an upstream request for MSI. Whenever any device > writes at address 0x58A8F8, my PCIe RC considers it as MSI and generates > an interrupt. So I receive MSI interrupt correctly in my software. Also > MSI controller is correctly able to point me that the interrupt is from > ethernet card. > > Now in Packet(178010), ethernet controller sends another upstream > request for MRd(64) of 32 dwords with Address(AFECEB87:A9D88B00).Since, > this address does not exist in my RC's world so, an UR is returned and > hence the problem occurs. > > Now, question is, why ethernet card is generating inbound request with > such a wrong address. I have taken log of all the tx_desc->buffer_addr > programmed by software in function e1000_tx_queue. None of them is 64 > bit or any invalid address. > > _______|_______________________________________________________________________ > Packet(177916) Upstream 2.5(x1) TLP(1475) Mem MWr(32)(10:00000) Length(4) > _______| RequesterID(003:00:0) Tag(2) Address(0EB00200) 1st BE(1111) > _______| Last BE(1111) Data(4 dwords) LCRC(0x44E0407C) > _______| Time Stamp(0013 . 460 549 544 s) > _______|_______________________________________________________________________ > Packet(177918) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1475) > _______| CRC 16(0x0EB7) Time Stamp(0013 . 460 551 144 s) > _______|_______________________________________________________________________ > Packet(177940) Upstream 2.5(x1) TLP(1476) Mem MWr(32)(10:00000) Length(1) > _______| RequesterID(003:00:0) Tag(30) Address(0058A8F8) 1st BE(0011) > _______| Last BE(0000) Data(1 dword) LCRC(0xC21F32B6) > _______| Time Stamp(0013 . 460 588 544 s) > _______|_______________________________________________________________________ > Packet(177942) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1476) > _______| CRC 16(0x69F5) Time Stamp(0013 . 460 590 088 s) > _______|_______________________________________________________________________ > Packet(177946) Downstream 2.5(x1) TLP(309) Mem MRd(32)(00:00000) Length(1) > _______| RequesterID(002:00:0) Tag(19) Address(C01000C0) 1st BE(1111) > _______| Last BE(0000) LCRC(0x91BDA1F5) Time Stamp(0013 . 460 595 936 s) > _______|_______________________________________________________________________ > Packet(177947) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(309) > _______| CRC 16(0x25C6) Time Stamp(0013 . 460 596 368 s) > _______|_______________________________________________________________________ > Packet(177950) Upstream 2.5(x1) TLP(1477) Cpl CplD(10:01010) Length(1) > _______| RequesterID(002:00:0) Tag(19) CompleterID(003:00:0) Status(SC) > BCM(0) > _______| Byte Cnt(4) Lwr Addr(0x40) Data(1 dword) LCRC(0x8FE0D922) > _______| Time Stamp(0013 . 460 597 304 s) > _______|_______________________________________________________________________ > Packet(177952) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1477) > _______| CRC 16(0xC8EE) Time Stamp(0013 . 460 598 840 s) > _______|_______________________________________________________________________ > Packet(177999) Downstream 2.5(x1) TLP(310) Mem MWr(32)(10:00000) Length(1) > _______| RequesterID(002:00:0) Tag(0) Address(C0103818) 1st BE(1111) > _______| Last BE(0000) Data(1 dword) LCRC(0xA898D9A1) > _______| Time Stamp(0013 . 460 687 936 s) > _______|_______________________________________________________________________ > Packet(178001) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(310) > _______| CRC 16(0xC6EA) Time Stamp(0013 . 460 688 384 s) > _______|_______________________________________________________________________ > Packet(178004) Upstream 2.5(x1) TLP(1478) Mem MRd(32)(00:00000) Length(4) > _______| RequesterID(003:00:0) Tag(4) Address(0EAFB990) 1st BE(1111) > _______| Last BE(1111) LCRC(0xB54722D2) Time Stamp(0013 . 460 689 312 s) > _______|_______________________________________________________________________ > Packet(178006) Downstream 2.5(x1) TLP(311) Cpl CplD(10:01010) Length(4) > _______| RequesterID(003:00:0) Tag(4) CompleterID(002:00:0) Status(SC) > BCM(0) > _______| Byte Cnt(16) Lwr Addr(0x10) Data(4 dwords) LCRC(0xFE303776) > _______| Time Stamp(0013 . 460 690 288 s) > _______|_______________________________________________________________________ > Packet(178007) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(311) > _______| CRC 16(0x67F1) Time Stamp(0013 . 460 690 776 s) > _______|_______________________________________________________________________ > Packet(178008) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1478) > _______| CRC 16(0x2BC2) Time Stamp(0013 . 460 690 824 s) > _______|_______________________________________________________________________ > Packet(178010) Upstream 2.5(x1) TLP(1479) Mem MRd(64)(01:00000) Length(32) > _______| RequesterID(003:00:0) Tag(11) Address(AFECEB87:A9D88B00) 1st > BE(1100) > _______| Last BE(0011) LCRC(0x6BE341C9) Time Stamp(0013 . 460 691 680 s) > _______|_______________________________________________________________________ > Packet(178011) Upstream 2.5(x1) TLP(1480) Mem MRd(64)(01:00000) Length(32) > _______| RequesterID(003:00:0) Tag(8) Address(AFECEB87:A9D88B7C) 1st > BE(1100) > _______| Last BE(0011) LCRC(0xAA5647BD) Time Stamp(0013 . 460 691 808 s) > _______|_______________________________________________________________________ > Packet(178012) Upstream 2.5(x1) TLP(1481) Mem MRd(64)(01:00000) Length(32) > _______| RequesterID(003:00:0) Tag(9) Address(AFECEB87:A9D88BF8) 1st > BE(1100) > _______| Last BE(0011) LCRC(0xEEB1F63F) Time Stamp(0013 . 460 692 120 s) > _______|_______________________________________________________________________ > Packet(178013) Upstream 2.5(x1) TLP(1482) Mem MRd(64)(01:00000) Length(32) > _______| RequesterID(003:00:0) Tag(10) Address(AFECEB87:A9D88C74) 1st > BE(1100) > _______| Last BE(0011) LCRC(0xA508142C) Time Stamp(0013 . 460 692 248 s) > _______|_______________________________________________________________________ > Packet(178014) Downstream 2.5(x1) TLP(312) Cpl Cpl(00:01010) Length(0) > _______| RequesterID(003:00:0) Tag(11) CompleterID(002:00:0) Status(UR)-BAD > _______| BCM(0) Byte Cnt(124) Lwr Addr(0x02) LCRC(0xCE5540D2) > _______| Time Stamp(0013 . 460 692 328 s) > _______|_______________________________________________________________________ > Packet(178015) Downstream 2.5(x1) TLP(313) Cpl Cpl(00:01010) Length(0) > _______| RequesterID(003:00:0) Tag(8) CompleterID(002:00:0) Status(UR)-BAD > _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7E) LCRC(0x9FE2487D) > _______| Time Stamp(0013 . 460 692 456 s) > _______|_______________________________________________________________________ > Packet(178016) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(312) > _______| CRC 16(0x086E) Time Stamp(0013 . 460 692 760 s) > _______|_______________________________________________________________________ > Packet(178017) Downstream 2.5(x1) TLP(314) Cpl Cpl(00:01010) Length(0) > _______| RequesterID(003:00:0) Tag(9) CompleterID(002:00:0) Status(UR)-BAD > _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7A) LCRC(0x097BF4DE) > _______| Time Stamp(0013 . 460 692 776 s) > _______|_______________________________________________________________________ > Packet(178018) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(313) > _______| CRC 16(0xA975) Time Stamp(0013 . 460 692 888 s) > _______|_______________________________________________________________________ > Packet(178019) Downstream 2.5(x1) TLP(315) Cpl Cpl(00:01010) Length(0) > _______| RequesterID(003:00:0) Tag(10) CompleterID(002:00:0) Status(UR)-BAD > _______| BCM(0) Byte Cnt(124) Lwr Addr(0x76) LCRC(0x64BDF921) > _______| Time Stamp(0013 . 460 692 904 s) > _______|_______________________________________________________________________ > Packet(178020) Upstream 2.5(x1) TLP(1483) Msg Msg(01:10000) > _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) > _______| Message Code(ERR_FATAL) LCRC(0xCDA53E96) > _______| Time Stamp(0013 . 460 693 184 s) > _______|_______________________________________________________________________ > Packet(178021) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1482) > _______| CRC 16(0xA771) Time Stamp(0013 . 460 693 208 s) > _______|_______________________________________________________________________ > Packet(178023) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(314) > _______| CRC 16(0x4A59) Time Stamp(0013 . 460 693 280 s) > _______|_______________________________________________________________________ > Packet(178024) Upstream 2.5(x1) TLP(1484) Msg Msg(01:10000) > _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) > _______| Message Code(ERR_FATAL) LCRC(0x86D9ACB6) > _______| Time Stamp(0013 . 460 693 312 s) > _______|_______________________________________________________________________ > Packet(178025) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(315) > _______| CRC 16(0xEB42) Time Stamp(0013 . 460 693 408 s) > _______|_______________________________________________________________________ > Packet(178026) Upstream 2.5(x1) TLP(1485) Msg Msg(01:10000) > _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) > _______| Message Code(ERR_FATAL) LCRC(0xC5120A31) > _______| Time Stamp(0013 . 460 693 632 s) > _______|_______________________________________________________________________ > Packet(178028) Upstream 2.5(x1) TLP(1486) Msg Msg(01:10000) > _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) > _______| Message Code(ERR_FATAL) LCRC(0x41499062) > _______| Time Stamp(0013 . 460 693 792 s) > _______|_______________________________________________________________________ > Packet(178029) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1486) > _______| CRC 16(0x231F) Time Stamp(0013 . 460 694 704 s) > _______|_______________________________________________________________________ > > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > . > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-06 18:55 ` Dave, Tushar N @ 2012-01-10 4:20 ` Pratyush Anand 2012-01-11 1:10 ` Dave, Tushar N 0 siblings, 1 reply; 13+ messages in thread From: Pratyush Anand @ 2012-01-10 4:20 UTC (permalink / raw) To: Dave, Tushar N Cc: Greg KH, Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, Linux NICS On 1/7/2012 12:25 AM, Dave, Tushar N wrote: > Pratyush, > > Sorry I got your name reversed. > Are you using in-kernel driver or one from Sourceforge. I am using in-kernel driver from kernel 2.6.37. > Please send me output of ethtool -i ethx. root@192.168.1.10:~# ethtool -i eth0 driver: e1000e version: 1.2.7-k2 firmware-version: 5.11-8 bus-info: 0000:01:00.0 Regards Pratyush > > -Tushar > > -----Original Message----- > From: Pratyush Anand [mailto:pratyush.anand@st.com] > Sent: Thursday, January 05, 2012 8:25 PM > To: Dave, Tushar N > Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS > Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet > > Thanks Tushar, > > On 1/6/2012 5:24 AM, Dave, Tushar N wrote: >> Anand, >> >> Sorry to hear that you have this issue with card. And yeah, thanks for doing the debugging and providing the bus trace. >> I think we should run the debug driver that prints the HW ring details when hang occurs. I can provide you a debug driver. You can then install debug driver and also let the bus tracer running. Once the issue occurs, provide me the full dmesg output (that has HW ring details) and bus trace. >> >> Tell me which card you have, 1gig or 10gig? Which driver are you running e1000e or igb or ixgbe? >> Can you also provide ethtool -i ethx output. >> >> Once I know which driver, I send you debug driver. > > I am using Intel PRO/1000 PT Server Adapter. > http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/pro-1000-pt.html > > I am using e1000e driver. > > I see the problem when I try to mount rootfilesystem using NFS and use > MSI interrupt. I see this issue even before I can have cell prompt. > Please see first mail in this thread. > > http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg04894.html > > Here, you can also see tx ring details when issue occur. > Please let me know, if you need any more info. > > Regards > Pratyush > >> >> Thanks. >> >> -Tushar >> >> -----Original Message----- >> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Pratyush Anand >> Sent: Wednesday, January 04, 2012 8:31 PM >> To: Greg KH >> Cc: Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS >> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >> >> On 1/5/2012 12:52 AM, Greg KH wrote: >>> On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: >>>> Adding PCI mailing list too, as problem is coming only when MSI is enabled. >>>> >>>> If I connect an PCIe analyzer, I see that at the time of issue >>>> MRd(64) for 32 words has been issued with a wrong 64 bit address >>>> from ethernet card to my RC. >>>> In the normal course it always issues MRd(32) only. >>> >>> Bug in your pcie firmware controller? >>> >>> . >>> >> >> when you say "Bug in your pcie firmware controller?", is it RC's >> software or EP's software? >> >> Here I am pasting a part of analyzer log converted into text. >> Packet(177940), is an upstream request for MSI. Whenever any device >> writes at address 0x58A8F8, my PCIe RC considers it as MSI and generates >> an interrupt. So I receive MSI interrupt correctly in my software. Also >> MSI controller is correctly able to point me that the interrupt is from >> ethernet card. >> >> Now in Packet(178010), ethernet controller sends another upstream >> request for MRd(64) of 32 dwords with Address(AFECEB87:A9D88B00).Since, >> this address does not exist in my RC's world so, an UR is returned and >> hence the problem occurs. >> >> Now, question is, why ethernet card is generating inbound request with >> such a wrong address. I have taken log of all the tx_desc->buffer_addr >> programmed by software in function e1000_tx_queue. None of them is 64 >> bit or any invalid address. >> >> _______|_______________________________________________________________________ >> Packet(177916) Upstream 2.5(x1) TLP(1475) Mem MWr(32)(10:00000) Length(4) >> _______| RequesterID(003:00:0) Tag(2) Address(0EB00200) 1st BE(1111) >> _______| Last BE(1111) Data(4 dwords) LCRC(0x44E0407C) >> _______| Time Stamp(0013 . 460 549 544 s) >> _______|_______________________________________________________________________ >> Packet(177918) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1475) >> _______| CRC 16(0x0EB7) Time Stamp(0013 . 460 551 144 s) >> _______|_______________________________________________________________________ >> Packet(177940) Upstream 2.5(x1) TLP(1476) Mem MWr(32)(10:00000) Length(1) >> _______| RequesterID(003:00:0) Tag(30) Address(0058A8F8) 1st BE(0011) >> _______| Last BE(0000) Data(1 dword) LCRC(0xC21F32B6) >> _______| Time Stamp(0013 . 460 588 544 s) >> _______|_______________________________________________________________________ >> Packet(177942) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1476) >> _______| CRC 16(0x69F5) Time Stamp(0013 . 460 590 088 s) >> _______|_______________________________________________________________________ >> Packet(177946) Downstream 2.5(x1) TLP(309) Mem MRd(32)(00:00000) Length(1) >> _______| RequesterID(002:00:0) Tag(19) Address(C01000C0) 1st BE(1111) >> _______| Last BE(0000) LCRC(0x91BDA1F5) Time Stamp(0013 . 460 595 936 s) >> _______|_______________________________________________________________________ >> Packet(177947) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(309) >> _______| CRC 16(0x25C6) Time Stamp(0013 . 460 596 368 s) >> _______|_______________________________________________________________________ >> Packet(177950) Upstream 2.5(x1) TLP(1477) Cpl CplD(10:01010) Length(1) >> _______| RequesterID(002:00:0) Tag(19) CompleterID(003:00:0) Status(SC) >> BCM(0) >> _______| Byte Cnt(4) Lwr Addr(0x40) Data(1 dword) LCRC(0x8FE0D922) >> _______| Time Stamp(0013 . 460 597 304 s) >> _______|_______________________________________________________________________ >> Packet(177952) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1477) >> _______| CRC 16(0xC8EE) Time Stamp(0013 . 460 598 840 s) >> _______|_______________________________________________________________________ >> Packet(177999) Downstream 2.5(x1) TLP(310) Mem MWr(32)(10:00000) Length(1) >> _______| RequesterID(002:00:0) Tag(0) Address(C0103818) 1st BE(1111) >> _______| Last BE(0000) Data(1 dword) LCRC(0xA898D9A1) >> _______| Time Stamp(0013 . 460 687 936 s) >> _______|_______________________________________________________________________ >> Packet(178001) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(310) >> _______| CRC 16(0xC6EA) Time Stamp(0013 . 460 688 384 s) >> _______|_______________________________________________________________________ >> Packet(178004) Upstream 2.5(x1) TLP(1478) Mem MRd(32)(00:00000) Length(4) >> _______| RequesterID(003:00:0) Tag(4) Address(0EAFB990) 1st BE(1111) >> _______| Last BE(1111) LCRC(0xB54722D2) Time Stamp(0013 . 460 689 312 s) >> _______|_______________________________________________________________________ >> Packet(178006) Downstream 2.5(x1) TLP(311) Cpl CplD(10:01010) Length(4) >> _______| RequesterID(003:00:0) Tag(4) CompleterID(002:00:0) Status(SC) >> BCM(0) >> _______| Byte Cnt(16) Lwr Addr(0x10) Data(4 dwords) LCRC(0xFE303776) >> _______| Time Stamp(0013 . 460 690 288 s) >> _______|_______________________________________________________________________ >> Packet(178007) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(311) >> _______| CRC 16(0x67F1) Time Stamp(0013 . 460 690 776 s) >> _______|_______________________________________________________________________ >> Packet(178008) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1478) >> _______| CRC 16(0x2BC2) Time Stamp(0013 . 460 690 824 s) >> _______|_______________________________________________________________________ >> Packet(178010) Upstream 2.5(x1) TLP(1479) Mem MRd(64)(01:00000) Length(32) >> _______| RequesterID(003:00:0) Tag(11) Address(AFECEB87:A9D88B00) 1st >> BE(1100) >> _______| Last BE(0011) LCRC(0x6BE341C9) Time Stamp(0013 . 460 691 680 s) >> _______|_______________________________________________________________________ >> Packet(178011) Upstream 2.5(x1) TLP(1480) Mem MRd(64)(01:00000) Length(32) >> _______| RequesterID(003:00:0) Tag(8) Address(AFECEB87:A9D88B7C) 1st >> BE(1100) >> _______| Last BE(0011) LCRC(0xAA5647BD) Time Stamp(0013 . 460 691 808 s) >> _______|_______________________________________________________________________ >> Packet(178012) Upstream 2.5(x1) TLP(1481) Mem MRd(64)(01:00000) Length(32) >> _______| RequesterID(003:00:0) Tag(9) Address(AFECEB87:A9D88BF8) 1st >> BE(1100) >> _______| Last BE(0011) LCRC(0xEEB1F63F) Time Stamp(0013 . 460 692 120 s) >> _______|_______________________________________________________________________ >> Packet(178013) Upstream 2.5(x1) TLP(1482) Mem MRd(64)(01:00000) Length(32) >> _______| RequesterID(003:00:0) Tag(10) Address(AFECEB87:A9D88C74) 1st >> BE(1100) >> _______| Last BE(0011) LCRC(0xA508142C) Time Stamp(0013 . 460 692 248 s) >> _______|_______________________________________________________________________ >> Packet(178014) Downstream 2.5(x1) TLP(312) Cpl Cpl(00:01010) Length(0) >> _______| RequesterID(003:00:0) Tag(11) CompleterID(002:00:0) Status(UR)-BAD >> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x02) LCRC(0xCE5540D2) >> _______| Time Stamp(0013 . 460 692 328 s) >> _______|_______________________________________________________________________ >> Packet(178015) Downstream 2.5(x1) TLP(313) Cpl Cpl(00:01010) Length(0) >> _______| RequesterID(003:00:0) Tag(8) CompleterID(002:00:0) Status(UR)-BAD >> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7E) LCRC(0x9FE2487D) >> _______| Time Stamp(0013 . 460 692 456 s) >> _______|_______________________________________________________________________ >> Packet(178016) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(312) >> _______| CRC 16(0x086E) Time Stamp(0013 . 460 692 760 s) >> _______|_______________________________________________________________________ >> Packet(178017) Downstream 2.5(x1) TLP(314) Cpl Cpl(00:01010) Length(0) >> _______| RequesterID(003:00:0) Tag(9) CompleterID(002:00:0) Status(UR)-BAD >> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7A) LCRC(0x097BF4DE) >> _______| Time Stamp(0013 . 460 692 776 s) >> _______|_______________________________________________________________________ >> Packet(178018) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(313) >> _______| CRC 16(0xA975) Time Stamp(0013 . 460 692 888 s) >> _______|_______________________________________________________________________ >> Packet(178019) Downstream 2.5(x1) TLP(315) Cpl Cpl(00:01010) Length(0) >> _______| RequesterID(003:00:0) Tag(10) CompleterID(002:00:0) Status(UR)-BAD >> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x76) LCRC(0x64BDF921) >> _______| Time Stamp(0013 . 460 692 904 s) >> _______|_______________________________________________________________________ >> Packet(178020) Upstream 2.5(x1) TLP(1483) Msg Msg(01:10000) >> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >> _______| Message Code(ERR_FATAL) LCRC(0xCDA53E96) >> _______| Time Stamp(0013 . 460 693 184 s) >> _______|_______________________________________________________________________ >> Packet(178021) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1482) >> _______| CRC 16(0xA771) Time Stamp(0013 . 460 693 208 s) >> _______|_______________________________________________________________________ >> Packet(178023) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(314) >> _______| CRC 16(0x4A59) Time Stamp(0013 . 460 693 280 s) >> _______|_______________________________________________________________________ >> Packet(178024) Upstream 2.5(x1) TLP(1484) Msg Msg(01:10000) >> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >> _______| Message Code(ERR_FATAL) LCRC(0x86D9ACB6) >> _______| Time Stamp(0013 . 460 693 312 s) >> _______|_______________________________________________________________________ >> Packet(178025) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(315) >> _______| CRC 16(0xEB42) Time Stamp(0013 . 460 693 408 s) >> _______|_______________________________________________________________________ >> Packet(178026) Upstream 2.5(x1) TLP(1485) Msg Msg(01:10000) >> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >> _______| Message Code(ERR_FATAL) LCRC(0xC5120A31) >> _______| Time Stamp(0013 . 460 693 632 s) >> _______|_______________________________________________________________________ >> Packet(178028) Upstream 2.5(x1) TLP(1486) Msg Msg(01:10000) >> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >> _______| Message Code(ERR_FATAL) LCRC(0x41499062) >> _______| Time Stamp(0013 . 460 693 792 s) >> _______|_______________________________________________________________________ >> Packet(178029) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1486) >> _______| CRC 16(0x231F) Time Stamp(0013 . 460 694 704 s) >> _______|_______________________________________________________________________ >> >> -- >> To unsubscribe from this list: send the line "unsubscribe netdev" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> . >> > > . > ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-10 4:20 ` Pratyush Anand @ 2012-01-11 1:10 ` Dave, Tushar N 2012-01-11 3:33 ` Pratyush Anand 0 siblings, 1 reply; 13+ messages in thread From: Dave, Tushar N @ 2012-01-11 1:10 UTC (permalink / raw) To: Pratyush Anand Cc: Greg KH, Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, Linux NICS, Dave, Tushar N Thanks for driver info. Because you are running in-kernel driver, we can enable the debug message level via ethtool. That will print HW ring info when issue occurs. Here is the ethtool command to enable debug messages. # ethtool -s ethx msglvl 0x3c00 This will enable tx_done, rx_status, pktdata and hw message levels. You can confirm it by typing ethtool ethx , this will show you 'Current message level' Next time when issue occurs, please send me the full dmesg log after the issue occurred along with the bus trace. Thanks. -Tushar -----Original Message----- From: Pratyush Anand [mailto:pratyush.anand@st.com] Sent: Monday, January 09, 2012 8:21 PM To: Dave, Tushar N Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet On 1/7/2012 12:25 AM, Dave, Tushar N wrote: > Pratyush, > > Sorry I got your name reversed. > Are you using in-kernel driver or one from Sourceforge. I am using in-kernel driver from kernel 2.6.37. > Please send me output of ethtool -i ethx. root@192.168.1.10:~# ethtool -i eth0 driver: e1000e version: 1.2.7-k2 firmware-version: 5.11-8 bus-info: 0000:01:00.0 Regards Pratyush > > -Tushar > > -----Original Message----- > From: Pratyush Anand [mailto:pratyush.anand@st.com] > Sent: Thursday, January 05, 2012 8:25 PM > To: Dave, Tushar N > Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS > Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet > > Thanks Tushar, > > On 1/6/2012 5:24 AM, Dave, Tushar N wrote: >> Anand, >> >> Sorry to hear that you have this issue with card. And yeah, thanks for doing the debugging and providing the bus trace. >> I think we should run the debug driver that prints the HW ring details when hang occurs. I can provide you a debug driver. You can then install debug driver and also let the bus tracer running. Once the issue occurs, provide me the full dmesg output (that has HW ring details) and bus trace. >> >> Tell me which card you have, 1gig or 10gig? Which driver are you running e1000e or igb or ixgbe? >> Can you also provide ethtool -i ethx output. >> >> Once I know which driver, I send you debug driver. > > I am using Intel PRO/1000 PT Server Adapter. > http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/pro-1000-pt.html > > I am using e1000e driver. > > I see the problem when I try to mount rootfilesystem using NFS and use > MSI interrupt. I see this issue even before I can have cell prompt. > Please see first mail in this thread. > > http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg04894.html > > Here, you can also see tx ring details when issue occur. > Please let me know, if you need any more info. > > Regards > Pratyush > >> >> Thanks. >> >> -Tushar >> >> -----Original Message----- >> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Pratyush Anand >> Sent: Wednesday, January 04, 2012 8:31 PM >> To: Greg KH >> Cc: Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS >> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >> >> On 1/5/2012 12:52 AM, Greg KH wrote: >>> On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: >>>> Adding PCI mailing list too, as problem is coming only when MSI is enabled. >>>> >>>> If I connect an PCIe analyzer, I see that at the time of issue >>>> MRd(64) for 32 words has been issued with a wrong 64 bit address >>>> from ethernet card to my RC. >>>> In the normal course it always issues MRd(32) only. >>> >>> Bug in your pcie firmware controller? >>> >>> . >>> >> >> when you say "Bug in your pcie firmware controller?", is it RC's >> software or EP's software? >> >> Here I am pasting a part of analyzer log converted into text. >> Packet(177940), is an upstream request for MSI. Whenever any device >> writes at address 0x58A8F8, my PCIe RC considers it as MSI and generates >> an interrupt. So I receive MSI interrupt correctly in my software. Also >> MSI controller is correctly able to point me that the interrupt is from >> ethernet card. >> >> Now in Packet(178010), ethernet controller sends another upstream >> request for MRd(64) of 32 dwords with Address(AFECEB87:A9D88B00).Since, >> this address does not exist in my RC's world so, an UR is returned and >> hence the problem occurs. >> >> Now, question is, why ethernet card is generating inbound request with >> such a wrong address. I have taken log of all the tx_desc->buffer_addr >> programmed by software in function e1000_tx_queue. None of them is 64 >> bit or any invalid address. >> >> _______|_______________________________________________________________________ >> Packet(177916) Upstream 2.5(x1) TLP(1475) Mem MWr(32)(10:00000) Length(4) >> _______| RequesterID(003:00:0) Tag(2) Address(0EB00200) 1st BE(1111) >> _______| Last BE(1111) Data(4 dwords) LCRC(0x44E0407C) >> _______| Time Stamp(0013 . 460 549 544 s) >> _______|_______________________________________________________________________ >> Packet(177918) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1475) >> _______| CRC 16(0x0EB7) Time Stamp(0013 . 460 551 144 s) >> _______|_______________________________________________________________________ >> Packet(177940) Upstream 2.5(x1) TLP(1476) Mem MWr(32)(10:00000) Length(1) >> _______| RequesterID(003:00:0) Tag(30) Address(0058A8F8) 1st BE(0011) >> _______| Last BE(0000) Data(1 dword) LCRC(0xC21F32B6) >> _______| Time Stamp(0013 . 460 588 544 s) >> _______|_______________________________________________________________________ >> Packet(177942) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1476) >> _______| CRC 16(0x69F5) Time Stamp(0013 . 460 590 088 s) >> _______|_______________________________________________________________________ >> Packet(177946) Downstream 2.5(x1) TLP(309) Mem MRd(32)(00:00000) Length(1) >> _______| RequesterID(002:00:0) Tag(19) Address(C01000C0) 1st BE(1111) >> _______| Last BE(0000) LCRC(0x91BDA1F5) Time Stamp(0013 . 460 595 936 s) >> _______|_______________________________________________________________________ >> Packet(177947) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(309) >> _______| CRC 16(0x25C6) Time Stamp(0013 . 460 596 368 s) >> _______|_______________________________________________________________________ >> Packet(177950) Upstream 2.5(x1) TLP(1477) Cpl CplD(10:01010) Length(1) >> _______| RequesterID(002:00:0) Tag(19) CompleterID(003:00:0) Status(SC) >> BCM(0) >> _______| Byte Cnt(4) Lwr Addr(0x40) Data(1 dword) LCRC(0x8FE0D922) >> _______| Time Stamp(0013 . 460 597 304 s) >> _______|_______________________________________________________________________ >> Packet(177952) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1477) >> _______| CRC 16(0xC8EE) Time Stamp(0013 . 460 598 840 s) >> _______|_______________________________________________________________________ >> Packet(177999) Downstream 2.5(x1) TLP(310) Mem MWr(32)(10:00000) Length(1) >> _______| RequesterID(002:00:0) Tag(0) Address(C0103818) 1st BE(1111) >> _______| Last BE(0000) Data(1 dword) LCRC(0xA898D9A1) >> _______| Time Stamp(0013 . 460 687 936 s) >> _______|_______________________________________________________________________ >> Packet(178001) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(310) >> _______| CRC 16(0xC6EA) Time Stamp(0013 . 460 688 384 s) >> _______|_______________________________________________________________________ >> Packet(178004) Upstream 2.5(x1) TLP(1478) Mem MRd(32)(00:00000) Length(4) >> _______| RequesterID(003:00:0) Tag(4) Address(0EAFB990) 1st BE(1111) >> _______| Last BE(1111) LCRC(0xB54722D2) Time Stamp(0013 . 460 689 312 s) >> _______|_______________________________________________________________________ >> Packet(178006) Downstream 2.5(x1) TLP(311) Cpl CplD(10:01010) Length(4) >> _______| RequesterID(003:00:0) Tag(4) CompleterID(002:00:0) Status(SC) >> BCM(0) >> _______| Byte Cnt(16) Lwr Addr(0x10) Data(4 dwords) LCRC(0xFE303776) >> _______| Time Stamp(0013 . 460 690 288 s) >> _______|_______________________________________________________________________ >> Packet(178007) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(311) >> _______| CRC 16(0x67F1) Time Stamp(0013 . 460 690 776 s) >> _______|_______________________________________________________________________ >> Packet(178008) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1478) >> _______| CRC 16(0x2BC2) Time Stamp(0013 . 460 690 824 s) >> _______|_______________________________________________________________________ >> Packet(178010) Upstream 2.5(x1) TLP(1479) Mem MRd(64)(01:00000) Length(32) >> _______| RequesterID(003:00:0) Tag(11) Address(AFECEB87:A9D88B00) 1st >> BE(1100) >> _______| Last BE(0011) LCRC(0x6BE341C9) Time Stamp(0013 . 460 691 680 s) >> _______|_______________________________________________________________________ >> Packet(178011) Upstream 2.5(x1) TLP(1480) Mem MRd(64)(01:00000) Length(32) >> _______| RequesterID(003:00:0) Tag(8) Address(AFECEB87:A9D88B7C) 1st >> BE(1100) >> _______| Last BE(0011) LCRC(0xAA5647BD) Time Stamp(0013 . 460 691 808 s) >> _______|_______________________________________________________________________ >> Packet(178012) Upstream 2.5(x1) TLP(1481) Mem MRd(64)(01:00000) Length(32) >> _______| RequesterID(003:00:0) Tag(9) Address(AFECEB87:A9D88BF8) 1st >> BE(1100) >> _______| Last BE(0011) LCRC(0xEEB1F63F) Time Stamp(0013 . 460 692 120 s) >> _______|_______________________________________________________________________ >> Packet(178013) Upstream 2.5(x1) TLP(1482) Mem MRd(64)(01:00000) Length(32) >> _______| RequesterID(003:00:0) Tag(10) Address(AFECEB87:A9D88C74) 1st >> BE(1100) >> _______| Last BE(0011) LCRC(0xA508142C) Time Stamp(0013 . 460 692 248 s) >> _______|_______________________________________________________________________ >> Packet(178014) Downstream 2.5(x1) TLP(312) Cpl Cpl(00:01010) Length(0) >> _______| RequesterID(003:00:0) Tag(11) CompleterID(002:00:0) Status(UR)-BAD >> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x02) LCRC(0xCE5540D2) >> _______| Time Stamp(0013 . 460 692 328 s) >> _______|_______________________________________________________________________ >> Packet(178015) Downstream 2.5(x1) TLP(313) Cpl Cpl(00:01010) Length(0) >> _______| RequesterID(003:00:0) Tag(8) CompleterID(002:00:0) Status(UR)-BAD >> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7E) LCRC(0x9FE2487D) >> _______| Time Stamp(0013 . 460 692 456 s) >> _______|_______________________________________________________________________ >> Packet(178016) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(312) >> _______| CRC 16(0x086E) Time Stamp(0013 . 460 692 760 s) >> _______|_______________________________________________________________________ >> Packet(178017) Downstream 2.5(x1) TLP(314) Cpl Cpl(00:01010) Length(0) >> _______| RequesterID(003:00:0) Tag(9) CompleterID(002:00:0) Status(UR)-BAD >> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7A) LCRC(0x097BF4DE) >> _______| Time Stamp(0013 . 460 692 776 s) >> _______|_______________________________________________________________________ >> Packet(178018) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(313) >> _______| CRC 16(0xA975) Time Stamp(0013 . 460 692 888 s) >> _______|_______________________________________________________________________ >> Packet(178019) Downstream 2.5(x1) TLP(315) Cpl Cpl(00:01010) Length(0) >> _______| RequesterID(003:00:0) Tag(10) CompleterID(002:00:0) Status(UR)-BAD >> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x76) LCRC(0x64BDF921) >> _______| Time Stamp(0013 . 460 692 904 s) >> _______|_______________________________________________________________________ >> Packet(178020) Upstream 2.5(x1) TLP(1483) Msg Msg(01:10000) >> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >> _______| Message Code(ERR_FATAL) LCRC(0xCDA53E96) >> _______| Time Stamp(0013 . 460 693 184 s) >> _______|_______________________________________________________________________ >> Packet(178021) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1482) >> _______| CRC 16(0xA771) Time Stamp(0013 . 460 693 208 s) >> _______|_______________________________________________________________________ >> Packet(178023) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(314) >> _______| CRC 16(0x4A59) Time Stamp(0013 . 460 693 280 s) >> _______|_______________________________________________________________________ >> Packet(178024) Upstream 2.5(x1) TLP(1484) Msg Msg(01:10000) >> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >> _______| Message Code(ERR_FATAL) LCRC(0x86D9ACB6) >> _______| Time Stamp(0013 . 460 693 312 s) >> _______|_______________________________________________________________________ >> Packet(178025) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(315) >> _______| CRC 16(0xEB42) Time Stamp(0013 . 460 693 408 s) >> _______|_______________________________________________________________________ >> Packet(178026) Upstream 2.5(x1) TLP(1485) Msg Msg(01:10000) >> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >> _______| Message Code(ERR_FATAL) LCRC(0xC5120A31) >> _______| Time Stamp(0013 . 460 693 632 s) >> _______|_______________________________________________________________________ >> Packet(178028) Upstream 2.5(x1) TLP(1486) Msg Msg(01:10000) >> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >> _______| Message Code(ERR_FATAL) LCRC(0x41499062) >> _______| Time Stamp(0013 . 460 693 792 s) >> _______|_______________________________________________________________________ >> Packet(178029) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1486) >> _______| CRC 16(0x231F) Time Stamp(0013 . 460 694 704 s) >> _______|_______________________________________________________________________ >> >> -- >> To unsubscribe from this list: send the line "unsubscribe netdev" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> . >> > > . > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-11 1:10 ` Dave, Tushar N @ 2012-01-11 3:33 ` Pratyush Anand 2012-01-11 3:48 ` Dave, Tushar N 2012-01-26 21:27 ` Dave, Tushar N 0 siblings, 2 replies; 13+ messages in thread From: Pratyush Anand @ 2012-01-11 3:33 UTC (permalink / raw) To: Dave, Tushar N Cc: Greg KH, Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, Linux NICS On 1/11/2012 6:40 AM, Dave, Tushar N wrote: > Thanks for driver info. > Because you are running in-kernel driver, we can enable the debug message level via ethtool. That will print HW ring info when issue occurs. > > Here is the ethtool command to enable debug messages. > # ethtool -s ethx msglvl 0x3c00 > This will enable tx_done, rx_status, pktdata and hw message levels. > You can confirm it by typing ethtool ethx , this will show you 'Current message level' > > Next time when issue occurs, please send me the full dmesg log after the issue occurred along with the bus trace. As I said earlier, issue is reproducible if I try to keep my rootfilesystem over NFS. So, after the booting, kernel tries to mount rootfs over NFS and it crashes. So, I see issue even before I can reach to # prompt. How can I use "ethtool -s ethx msglvl 0x3c00" to enable any debug message. May be I can directly change in kernel code to enable this. Regards Pratyush > > Thanks. > > -Tushar > > > -----Original Message----- > From: Pratyush Anand [mailto:pratyush.anand@st.com] > Sent: Monday, January 09, 2012 8:21 PM > To: Dave, Tushar N > Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS > Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet > > On 1/7/2012 12:25 AM, Dave, Tushar N wrote: >> Pratyush, >> >> Sorry I got your name reversed. >> Are you using in-kernel driver or one from Sourceforge. > > I am using in-kernel driver from kernel 2.6.37. > >> Please send me output of ethtool -i ethx. > > root@192.168.1.10:~# ethtool -i eth0 > driver: e1000e > version: 1.2.7-k2 > firmware-version: 5.11-8 > bus-info: 0000:01:00.0 > > Regards > Pratyush > >> >> -Tushar >> >> -----Original Message----- >> From: Pratyush Anand [mailto:pratyush.anand@st.com] >> Sent: Thursday, January 05, 2012 8:25 PM >> To: Dave, Tushar N >> Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS >> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >> >> Thanks Tushar, >> >> On 1/6/2012 5:24 AM, Dave, Tushar N wrote: >>> Anand, >>> >>> Sorry to hear that you have this issue with card. And yeah, thanks for doing the debugging and providing the bus trace. >>> I think we should run the debug driver that prints the HW ring details when hang occurs. I can provide you a debug driver. You can then install debug driver and also let the bus tracer running. Once the issue occurs, provide me the full dmesg output (that has HW ring details) and bus trace. >>> >>> Tell me which card you have, 1gig or 10gig? Which driver are you running e1000e or igb or ixgbe? >>> Can you also provide ethtool -i ethx output. >>> >>> Once I know which driver, I send you debug driver. >> >> I am using Intel PRO/1000 PT Server Adapter. >> http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/pro-1000-pt.html >> >> I am using e1000e driver. >> >> I see the problem when I try to mount rootfilesystem using NFS and use >> MSI interrupt. I see this issue even before I can have cell prompt. >> Please see first mail in this thread. >> >> http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg04894.html >> >> Here, you can also see tx ring details when issue occur. >> Please let me know, if you need any more info. >> >> Regards >> Pratyush >> >>> >>> Thanks. >>> >>> -Tushar >>> >>> -----Original Message----- >>> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Pratyush Anand >>> Sent: Wednesday, January 04, 2012 8:31 PM >>> To: Greg KH >>> Cc: Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS >>> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >>> >>> On 1/5/2012 12:52 AM, Greg KH wrote: >>>> On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: >>>>> Adding PCI mailing list too, as problem is coming only when MSI is enabled. >>>>> >>>>> If I connect an PCIe analyzer, I see that at the time of issue >>>>> MRd(64) for 32 words has been issued with a wrong 64 bit address >>>>> from ethernet card to my RC. >>>>> In the normal course it always issues MRd(32) only. >>>> >>>> Bug in your pcie firmware controller? >>>> >>>> . >>>> >>> >>> when you say "Bug in your pcie firmware controller?", is it RC's >>> software or EP's software? >>> >>> Here I am pasting a part of analyzer log converted into text. >>> Packet(177940), is an upstream request for MSI. Whenever any device >>> writes at address 0x58A8F8, my PCIe RC considers it as MSI and generates >>> an interrupt. So I receive MSI interrupt correctly in my software. Also >>> MSI controller is correctly able to point me that the interrupt is from >>> ethernet card. >>> >>> Now in Packet(178010), ethernet controller sends another upstream >>> request for MRd(64) of 32 dwords with Address(AFECEB87:A9D88B00).Since, >>> this address does not exist in my RC's world so, an UR is returned and >>> hence the problem occurs. >>> >>> Now, question is, why ethernet card is generating inbound request with >>> such a wrong address. I have taken log of all the tx_desc->buffer_addr >>> programmed by software in function e1000_tx_queue. None of them is 64 >>> bit or any invalid address. >>> >>> _______|_______________________________________________________________________ >>> Packet(177916) Upstream 2.5(x1) TLP(1475) Mem MWr(32)(10:00000) Length(4) >>> _______| RequesterID(003:00:0) Tag(2) Address(0EB00200) 1st BE(1111) >>> _______| Last BE(1111) Data(4 dwords) LCRC(0x44E0407C) >>> _______| Time Stamp(0013 . 460 549 544 s) >>> _______|_______________________________________________________________________ >>> Packet(177918) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1475) >>> _______| CRC 16(0x0EB7) Time Stamp(0013 . 460 551 144 s) >>> _______|_______________________________________________________________________ >>> Packet(177940) Upstream 2.5(x1) TLP(1476) Mem MWr(32)(10:00000) Length(1) >>> _______| RequesterID(003:00:0) Tag(30) Address(0058A8F8) 1st BE(0011) >>> _______| Last BE(0000) Data(1 dword) LCRC(0xC21F32B6) >>> _______| Time Stamp(0013 . 460 588 544 s) >>> _______|_______________________________________________________________________ >>> Packet(177942) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1476) >>> _______| CRC 16(0x69F5) Time Stamp(0013 . 460 590 088 s) >>> _______|_______________________________________________________________________ >>> Packet(177946) Downstream 2.5(x1) TLP(309) Mem MRd(32)(00:00000) Length(1) >>> _______| RequesterID(002:00:0) Tag(19) Address(C01000C0) 1st BE(1111) >>> _______| Last BE(0000) LCRC(0x91BDA1F5) Time Stamp(0013 . 460 595 936 s) >>> _______|_______________________________________________________________________ >>> Packet(177947) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(309) >>> _______| CRC 16(0x25C6) Time Stamp(0013 . 460 596 368 s) >>> _______|_______________________________________________________________________ >>> Packet(177950) Upstream 2.5(x1) TLP(1477) Cpl CplD(10:01010) Length(1) >>> _______| RequesterID(002:00:0) Tag(19) CompleterID(003:00:0) Status(SC) >>> BCM(0) >>> _______| Byte Cnt(4) Lwr Addr(0x40) Data(1 dword) LCRC(0x8FE0D922) >>> _______| Time Stamp(0013 . 460 597 304 s) >>> _______|_______________________________________________________________________ >>> Packet(177952) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1477) >>> _______| CRC 16(0xC8EE) Time Stamp(0013 . 460 598 840 s) >>> _______|_______________________________________________________________________ >>> Packet(177999) Downstream 2.5(x1) TLP(310) Mem MWr(32)(10:00000) Length(1) >>> _______| RequesterID(002:00:0) Tag(0) Address(C0103818) 1st BE(1111) >>> _______| Last BE(0000) Data(1 dword) LCRC(0xA898D9A1) >>> _______| Time Stamp(0013 . 460 687 936 s) >>> _______|_______________________________________________________________________ >>> Packet(178001) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(310) >>> _______| CRC 16(0xC6EA) Time Stamp(0013 . 460 688 384 s) >>> _______|_______________________________________________________________________ >>> Packet(178004) Upstream 2.5(x1) TLP(1478) Mem MRd(32)(00:00000) Length(4) >>> _______| RequesterID(003:00:0) Tag(4) Address(0EAFB990) 1st BE(1111) >>> _______| Last BE(1111) LCRC(0xB54722D2) Time Stamp(0013 . 460 689 312 s) >>> _______|_______________________________________________________________________ >>> Packet(178006) Downstream 2.5(x1) TLP(311) Cpl CplD(10:01010) Length(4) >>> _______| RequesterID(003:00:0) Tag(4) CompleterID(002:00:0) Status(SC) >>> BCM(0) >>> _______| Byte Cnt(16) Lwr Addr(0x10) Data(4 dwords) LCRC(0xFE303776) >>> _______| Time Stamp(0013 . 460 690 288 s) >>> _______|_______________________________________________________________________ >>> Packet(178007) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(311) >>> _______| CRC 16(0x67F1) Time Stamp(0013 . 460 690 776 s) >>> _______|_______________________________________________________________________ >>> Packet(178008) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1478) >>> _______| CRC 16(0x2BC2) Time Stamp(0013 . 460 690 824 s) >>> _______|_______________________________________________________________________ >>> Packet(178010) Upstream 2.5(x1) TLP(1479) Mem MRd(64)(01:00000) Length(32) >>> _______| RequesterID(003:00:0) Tag(11) Address(AFECEB87:A9D88B00) 1st >>> BE(1100) >>> _______| Last BE(0011) LCRC(0x6BE341C9) Time Stamp(0013 . 460 691 680 s) >>> _______|_______________________________________________________________________ >>> Packet(178011) Upstream 2.5(x1) TLP(1480) Mem MRd(64)(01:00000) Length(32) >>> _______| RequesterID(003:00:0) Tag(8) Address(AFECEB87:A9D88B7C) 1st >>> BE(1100) >>> _______| Last BE(0011) LCRC(0xAA5647BD) Time Stamp(0013 . 460 691 808 s) >>> _______|_______________________________________________________________________ >>> Packet(178012) Upstream 2.5(x1) TLP(1481) Mem MRd(64)(01:00000) Length(32) >>> _______| RequesterID(003:00:0) Tag(9) Address(AFECEB87:A9D88BF8) 1st >>> BE(1100) >>> _______| Last BE(0011) LCRC(0xEEB1F63F) Time Stamp(0013 . 460 692 120 s) >>> _______|_______________________________________________________________________ >>> Packet(178013) Upstream 2.5(x1) TLP(1482) Mem MRd(64)(01:00000) Length(32) >>> _______| RequesterID(003:00:0) Tag(10) Address(AFECEB87:A9D88C74) 1st >>> BE(1100) >>> _______| Last BE(0011) LCRC(0xA508142C) Time Stamp(0013 . 460 692 248 s) >>> _______|_______________________________________________________________________ >>> Packet(178014) Downstream 2.5(x1) TLP(312) Cpl Cpl(00:01010) Length(0) >>> _______| RequesterID(003:00:0) Tag(11) CompleterID(002:00:0) Status(UR)-BAD >>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x02) LCRC(0xCE5540D2) >>> _______| Time Stamp(0013 . 460 692 328 s) >>> _______|_______________________________________________________________________ >>> Packet(178015) Downstream 2.5(x1) TLP(313) Cpl Cpl(00:01010) Length(0) >>> _______| RequesterID(003:00:0) Tag(8) CompleterID(002:00:0) Status(UR)-BAD >>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7E) LCRC(0x9FE2487D) >>> _______| Time Stamp(0013 . 460 692 456 s) >>> _______|_______________________________________________________________________ >>> Packet(178016) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(312) >>> _______| CRC 16(0x086E) Time Stamp(0013 . 460 692 760 s) >>> _______|_______________________________________________________________________ >>> Packet(178017) Downstream 2.5(x1) TLP(314) Cpl Cpl(00:01010) Length(0) >>> _______| RequesterID(003:00:0) Tag(9) CompleterID(002:00:0) Status(UR)-BAD >>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7A) LCRC(0x097BF4DE) >>> _______| Time Stamp(0013 . 460 692 776 s) >>> _______|_______________________________________________________________________ >>> Packet(178018) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(313) >>> _______| CRC 16(0xA975) Time Stamp(0013 . 460 692 888 s) >>> _______|_______________________________________________________________________ >>> Packet(178019) Downstream 2.5(x1) TLP(315) Cpl Cpl(00:01010) Length(0) >>> _______| RequesterID(003:00:0) Tag(10) CompleterID(002:00:0) Status(UR)-BAD >>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x76) LCRC(0x64BDF921) >>> _______| Time Stamp(0013 . 460 692 904 s) >>> _______|_______________________________________________________________________ >>> Packet(178020) Upstream 2.5(x1) TLP(1483) Msg Msg(01:10000) >>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>> _______| Message Code(ERR_FATAL) LCRC(0xCDA53E96) >>> _______| Time Stamp(0013 . 460 693 184 s) >>> _______|_______________________________________________________________________ >>> Packet(178021) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1482) >>> _______| CRC 16(0xA771) Time Stamp(0013 . 460 693 208 s) >>> _______|_______________________________________________________________________ >>> Packet(178023) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(314) >>> _______| CRC 16(0x4A59) Time Stamp(0013 . 460 693 280 s) >>> _______|_______________________________________________________________________ >>> Packet(178024) Upstream 2.5(x1) TLP(1484) Msg Msg(01:10000) >>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>> _______| Message Code(ERR_FATAL) LCRC(0x86D9ACB6) >>> _______| Time Stamp(0013 . 460 693 312 s) >>> _______|_______________________________________________________________________ >>> Packet(178025) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(315) >>> _______| CRC 16(0xEB42) Time Stamp(0013 . 460 693 408 s) >>> _______|_______________________________________________________________________ >>> Packet(178026) Upstream 2.5(x1) TLP(1485) Msg Msg(01:10000) >>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>> _______| Message Code(ERR_FATAL) LCRC(0xC5120A31) >>> _______| Time Stamp(0013 . 460 693 632 s) >>> _______|_______________________________________________________________________ >>> Packet(178028) Upstream 2.5(x1) TLP(1486) Msg Msg(01:10000) >>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>> _______| Message Code(ERR_FATAL) LCRC(0x41499062) >>> _______| Time Stamp(0013 . 460 693 792 s) >>> _______|_______________________________________________________________________ >>> Packet(178029) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1486) >>> _______| CRC 16(0x231F) Time Stamp(0013 . 460 694 704 s) >>> _______|_______________________________________________________________________ >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe netdev" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> . >>> >> >> . >> > > . > ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-11 3:33 ` Pratyush Anand @ 2012-01-11 3:48 ` Dave, Tushar N 2012-01-26 21:27 ` Dave, Tushar N 1 sibling, 0 replies; 13+ messages in thread From: Dave, Tushar N @ 2012-01-11 3:48 UTC (permalink / raw) To: Pratyush Anand Cc: Greg KH, Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, Linux NICS >As I said earlier, issue is reproducible if I try to keep my >rootfilesystem over NFS. So, after the booting, kernel tries to mount >rootfs over NFS and it crashes. So, I see issue even before I can reach >to # prompt. How can I use "ethtool -s ethx msglvl 0x3c00" to enable any >debug message. May be I can directly change in kernel code to enable this. Yes either you can modify in-kernel driver code Or I will provide you the latest e1000e out of tree driver from http://sourceforge.net/projects/e1000/files/ with debug info enabled. -Tushar > > Thanks. > > -Tushar > > > -----Original Message----- > From: Pratyush Anand [mailto:pratyush.anand@st.com] > Sent: Monday, January 09, 2012 8:21 PM > To: Dave, Tushar N > Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS > Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet > > On 1/7/2012 12:25 AM, Dave, Tushar N wrote: >> Pratyush, >> >> Sorry I got your name reversed. >> Are you using in-kernel driver or one from Sourceforge. > > I am using in-kernel driver from kernel 2.6.37. > >> Please send me output of ethtool -i ethx. > > root@192.168.1.10:~# ethtool -i eth0 > driver: e1000e > version: 1.2.7-k2 > firmware-version: 5.11-8 > bus-info: 0000:01:00.0 > > Regards > Pratyush > >> >> -Tushar >> >> -----Original Message----- >> From: Pratyush Anand [mailto:pratyush.anand@st.com] >> Sent: Thursday, January 05, 2012 8:25 PM >> To: Dave, Tushar N >> Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS >> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >> >> Thanks Tushar, >> >> On 1/6/2012 5:24 AM, Dave, Tushar N wrote: >>> Anand, >>> >>> Sorry to hear that you have this issue with card. And yeah, thanks for doing the debugging and providing the bus trace. >>> I think we should run the debug driver that prints the HW ring details when hang occurs. I can provide you a debug driver. You can then install debug driver and also let the bus tracer running. Once the issue occurs, provide me the full dmesg output (that has HW ring details) and bus trace. >>> >>> Tell me which card you have, 1gig or 10gig? Which driver are you running e1000e or igb or ixgbe? >>> Can you also provide ethtool -i ethx output. >>> >>> Once I know which driver, I send you debug driver. >> >> I am using Intel PRO/1000 PT Server Adapter. >> http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/pro-1000-pt.html >> >> I am using e1000e driver. >> >> I see the problem when I try to mount rootfilesystem using NFS and use >> MSI interrupt. I see this issue even before I can have cell prompt. >> Please see first mail in this thread. >> >> http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg04894.html >> >> Here, you can also see tx ring details when issue occur. >> Please let me know, if you need any more info. >> >> Regards >> Pratyush >> >>> >>> Thanks. >>> >>> -Tushar >>> >>> -----Original Message----- >>> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Pratyush Anand >>> Sent: Wednesday, January 04, 2012 8:31 PM >>> To: Greg KH >>> Cc: Pratyush Anand; e1000-devel@lists.sourceforge.net; netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci@vger.kernel.org; Linux NICS >>> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >>> >>> On 1/5/2012 12:52 AM, Greg KH wrote: >>>> On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: >>>>> Adding PCI mailing list too, as problem is coming only when MSI is enabled. >>>>> >>>>> If I connect an PCIe analyzer, I see that at the time of issue >>>>> MRd(64) for 32 words has been issued with a wrong 64 bit address >>>>> from ethernet card to my RC. >>>>> In the normal course it always issues MRd(32) only. >>>> >>>> Bug in your pcie firmware controller? >>>> >>>> . >>>> >>> >>> when you say "Bug in your pcie firmware controller?", is it RC's >>> software or EP's software? >>> >>> Here I am pasting a part of analyzer log converted into text. >>> Packet(177940), is an upstream request for MSI. Whenever any device >>> writes at address 0x58A8F8, my PCIe RC considers it as MSI and generates >>> an interrupt. So I receive MSI interrupt correctly in my software. Also >>> MSI controller is correctly able to point me that the interrupt is from >>> ethernet card. >>> >>> Now in Packet(178010), ethernet controller sends another upstream >>> request for MRd(64) of 32 dwords with Address(AFECEB87:A9D88B00).Since, >>> this address does not exist in my RC's world so, an UR is returned and >>> hence the problem occurs. >>> >>> Now, question is, why ethernet card is generating inbound request with >>> such a wrong address. I have taken log of all the tx_desc->buffer_addr >>> programmed by software in function e1000_tx_queue. None of them is 64 >>> bit or any invalid address. >>> >>> _______|_______________________________________________________________________ >>> Packet(177916) Upstream 2.5(x1) TLP(1475) Mem MWr(32)(10:00000) Length(4) >>> _______| RequesterID(003:00:0) Tag(2) Address(0EB00200) 1st BE(1111) >>> _______| Last BE(1111) Data(4 dwords) LCRC(0x44E0407C) >>> _______| Time Stamp(0013 . 460 549 544 s) >>> _______|_______________________________________________________________________ >>> Packet(177918) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1475) >>> _______| CRC 16(0x0EB7) Time Stamp(0013 . 460 551 144 s) >>> _______|_______________________________________________________________________ >>> Packet(177940) Upstream 2.5(x1) TLP(1476) Mem MWr(32)(10:00000) Length(1) >>> _______| RequesterID(003:00:0) Tag(30) Address(0058A8F8) 1st BE(0011) >>> _______| Last BE(0000) Data(1 dword) LCRC(0xC21F32B6) >>> _______| Time Stamp(0013 . 460 588 544 s) >>> _______|_______________________________________________________________________ >>> Packet(177942) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1476) >>> _______| CRC 16(0x69F5) Time Stamp(0013 . 460 590 088 s) >>> _______|_______________________________________________________________________ >>> Packet(177946) Downstream 2.5(x1) TLP(309) Mem MRd(32)(00:00000) Length(1) >>> _______| RequesterID(002:00:0) Tag(19) Address(C01000C0) 1st BE(1111) >>> _______| Last BE(0000) LCRC(0x91BDA1F5) Time Stamp(0013 . 460 595 936 s) >>> _______|_______________________________________________________________________ >>> Packet(177947) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(309) >>> _______| CRC 16(0x25C6) Time Stamp(0013 . 460 596 368 s) >>> _______|_______________________________________________________________________ >>> Packet(177950) Upstream 2.5(x1) TLP(1477) Cpl CplD(10:01010) Length(1) >>> _______| RequesterID(002:00:0) Tag(19) CompleterID(003:00:0) Status(SC) >>> BCM(0) >>> _______| Byte Cnt(4) Lwr Addr(0x40) Data(1 dword) LCRC(0x8FE0D922) >>> _______| Time Stamp(0013 . 460 597 304 s) >>> _______|_______________________________________________________________________ >>> Packet(177952) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1477) >>> _______| CRC 16(0xC8EE) Time Stamp(0013 . 460 598 840 s) >>> _______|_______________________________________________________________________ >>> Packet(177999) Downstream 2.5(x1) TLP(310) Mem MWr(32)(10:00000) Length(1) >>> _______| RequesterID(002:00:0) Tag(0) Address(C0103818) 1st BE(1111) >>> _______| Last BE(0000) Data(1 dword) LCRC(0xA898D9A1) >>> _______| Time Stamp(0013 . 460 687 936 s) >>> _______|_______________________________________________________________________ >>> Packet(178001) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(310) >>> _______| CRC 16(0xC6EA) Time Stamp(0013 . 460 688 384 s) >>> _______|_______________________________________________________________________ >>> Packet(178004) Upstream 2.5(x1) TLP(1478) Mem MRd(32)(00:00000) Length(4) >>> _______| RequesterID(003:00:0) Tag(4) Address(0EAFB990) 1st BE(1111) >>> _______| Last BE(1111) LCRC(0xB54722D2) Time Stamp(0013 . 460 689 312 s) >>> _______|_______________________________________________________________________ >>> Packet(178006) Downstream 2.5(x1) TLP(311) Cpl CplD(10:01010) Length(4) >>> _______| RequesterID(003:00:0) Tag(4) CompleterID(002:00:0) Status(SC) >>> BCM(0) >>> _______| Byte Cnt(16) Lwr Addr(0x10) Data(4 dwords) LCRC(0xFE303776) >>> _______| Time Stamp(0013 . 460 690 288 s) >>> _______|_______________________________________________________________________ >>> Packet(178007) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(311) >>> _______| CRC 16(0x67F1) Time Stamp(0013 . 460 690 776 s) >>> _______|_______________________________________________________________________ >>> Packet(178008) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1478) >>> _______| CRC 16(0x2BC2) Time Stamp(0013 . 460 690 824 s) >>> _______|_______________________________________________________________________ >>> Packet(178010) Upstream 2.5(x1) TLP(1479) Mem MRd(64)(01:00000) Length(32) >>> _______| RequesterID(003:00:0) Tag(11) Address(AFECEB87:A9D88B00) 1st >>> BE(1100) >>> _______| Last BE(0011) LCRC(0x6BE341C9) Time Stamp(0013 . 460 691 680 s) >>> _______|_______________________________________________________________________ >>> Packet(178011) Upstream 2.5(x1) TLP(1480) Mem MRd(64)(01:00000) Length(32) >>> _______| RequesterID(003:00:0) Tag(8) Address(AFECEB87:A9D88B7C) 1st >>> BE(1100) >>> _______| Last BE(0011) LCRC(0xAA5647BD) Time Stamp(0013 . 460 691 808 s) >>> _______|_______________________________________________________________________ >>> Packet(178012) Upstream 2.5(x1) TLP(1481) Mem MRd(64)(01:00000) Length(32) >>> _______| RequesterID(003:00:0) Tag(9) Address(AFECEB87:A9D88BF8) 1st >>> BE(1100) >>> _______| Last BE(0011) LCRC(0xEEB1F63F) Time Stamp(0013 . 460 692 120 s) >>> _______|_______________________________________________________________________ >>> Packet(178013) Upstream 2.5(x1) TLP(1482) Mem MRd(64)(01:00000) Length(32) >>> _______| RequesterID(003:00:0) Tag(10) Address(AFECEB87:A9D88C74) 1st >>> BE(1100) >>> _______| Last BE(0011) LCRC(0xA508142C) Time Stamp(0013 . 460 692 248 s) >>> _______|_______________________________________________________________________ >>> Packet(178014) Downstream 2.5(x1) TLP(312) Cpl Cpl(00:01010) Length(0) >>> _______| RequesterID(003:00:0) Tag(11) CompleterID(002:00:0) Status(UR)-BAD >>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x02) LCRC(0xCE5540D2) >>> _______| Time Stamp(0013 . 460 692 328 s) >>> _______|_______________________________________________________________________ >>> Packet(178015) Downstream 2.5(x1) TLP(313) Cpl Cpl(00:01010) Length(0) >>> _______| RequesterID(003:00:0) Tag(8) CompleterID(002:00:0) Status(UR)-BAD >>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7E) LCRC(0x9FE2487D) >>> _______| Time Stamp(0013 . 460 692 456 s) >>> _______|_______________________________________________________________________ >>> Packet(178016) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(312) >>> _______| CRC 16(0x086E) Time Stamp(0013 . 460 692 760 s) >>> _______|_______________________________________________________________________ >>> Packet(178017) Downstream 2.5(x1) TLP(314) Cpl Cpl(00:01010) Length(0) >>> _______| RequesterID(003:00:0) Tag(9) CompleterID(002:00:0) Status(UR)-BAD >>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7A) LCRC(0x097BF4DE) >>> _______| Time Stamp(0013 . 460 692 776 s) >>> _______|_______________________________________________________________________ >>> Packet(178018) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(313) >>> _______| CRC 16(0xA975) Time Stamp(0013 . 460 692 888 s) >>> _______|_______________________________________________________________________ >>> Packet(178019) Downstream 2.5(x1) TLP(315) Cpl Cpl(00:01010) Length(0) >>> _______| RequesterID(003:00:0) Tag(10) CompleterID(002:00:0) Status(UR)-BAD >>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x76) LCRC(0x64BDF921) >>> _______| Time Stamp(0013 . 460 692 904 s) >>> _______|_______________________________________________________________________ >>> Packet(178020) Upstream 2.5(x1) TLP(1483) Msg Msg(01:10000) >>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>> _______| Message Code(ERR_FATAL) LCRC(0xCDA53E96) >>> _______| Time Stamp(0013 . 460 693 184 s) >>> _______|_______________________________________________________________________ >>> Packet(178021) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1482) >>> _______| CRC 16(0xA771) Time Stamp(0013 . 460 693 208 s) >>> _______|_______________________________________________________________________ >>> Packet(178023) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(314) >>> _______| CRC 16(0x4A59) Time Stamp(0013 . 460 693 280 s) >>> _______|_______________________________________________________________________ >>> Packet(178024) Upstream 2.5(x1) TLP(1484) Msg Msg(01:10000) >>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>> _______| Message Code(ERR_FATAL) LCRC(0x86D9ACB6) >>> _______| Time Stamp(0013 . 460 693 312 s) >>> _______|_______________________________________________________________________ >>> Packet(178025) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(315) >>> _______| CRC 16(0xEB42) Time Stamp(0013 . 460 693 408 s) >>> _______|_______________________________________________________________________ >>> Packet(178026) Upstream 2.5(x1) TLP(1485) Msg Msg(01:10000) >>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>> _______| Message Code(ERR_FATAL) LCRC(0xC5120A31) >>> _______| Time Stamp(0013 . 460 693 632 s) >>> _______|_______________________________________________________________________ >>> Packet(178028) Upstream 2.5(x1) TLP(1486) Msg Msg(01:10000) >>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>> _______| Message Code(ERR_FATAL) LCRC(0x41499062) >>> _______| Time Stamp(0013 . 460 693 792 s) >>> _______|_______________________________________________________________________ >>> Packet(178029) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1486) >>> _______| CRC 16(0x231F) Time Stamp(0013 . 460 694 704 s) >>> _______|_______________________________________________________________________ >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe netdev" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> . >>> >> >> . >> > > . > ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-11 3:33 ` Pratyush Anand 2012-01-11 3:48 ` Dave, Tushar N @ 2012-01-26 21:27 ` Dave, Tushar N 2012-01-27 8:03 ` Pratyush Anand 1 sibling, 1 reply; 13+ messages in thread From: Dave, Tushar N @ 2012-01-26 21:27 UTC (permalink / raw) To: Pratyush Anand Cc: Greg KH, Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, Linux NICS, Dave, Tushar N >-----Original Message----- >From: Pratyush Anand [mailto:pratyush.anand@st.com] >Sent: Tuesday, January 10, 2012 7:34 PM >To: Dave, Tushar N >Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; >netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux- >pci@vger.kernel.org; Linux NICS >Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet > >As I said earlier, issue is reproducible if I try to keep my >rootfilesystem over NFS. So, after the booting, kernel tries to mount >rootfs over NFS and it crashes. So, I see issue even before I can reach >to # prompt. How can I use "ethtool -s ethx msglvl 0x3c00" to enable any >debug message. May be I can directly change in kernel code to enable this. Any update on this? Did you change in-kernel driver source to print the driver HW ring? If you did and had reproduced the issue please send me the full dmesg log along with bus trace and I'll take a look. -Tushar >> -----Original Message----- >> From: Pratyush Anand [mailto:pratyush.anand@st.com] >> Sent: Monday, January 09, 2012 8:21 PM >> To: Dave, Tushar N >> Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; >netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux- >pci@vger.kernel.org; Linux NICS >> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >> >> On 1/7/2012 12:25 AM, Dave, Tushar N wrote: >>> Pratyush, >>> >>> Sorry I got your name reversed. >>> Are you using in-kernel driver or one from Sourceforge. >> >> I am using in-kernel driver from kernel 2.6.37. >> >>> Please send me output of ethtool -i ethx. >> >> root@192.168.1.10:~# ethtool -i eth0 >> driver: e1000e >> version: 1.2.7-k2 >> firmware-version: 5.11-8 >> bus-info: 0000:01:00.0 >> >> Regards >> Pratyush >> >>> >>> -Tushar >>> >>> -----Original Message----- >>> From: Pratyush Anand [mailto:pratyush.anand@st.com] >>> Sent: Thursday, January 05, 2012 8:25 PM >>> To: Dave, Tushar N >>> Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; >netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux- >pci@vger.kernel.org; Linux NICS >>> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >>> >>> Thanks Tushar, >>> >>> On 1/6/2012 5:24 AM, Dave, Tushar N wrote: >>>> Anand, >>>> >>>> Sorry to hear that you have this issue with card. And yeah, thanks for >doing the debugging and providing the bus trace. >>>> I think we should run the debug driver that prints the HW ring details >when hang occurs. I can provide you a debug driver. You can then install >debug driver and also let the bus tracer running. Once the issue occurs, >provide me the full dmesg output (that has HW ring details) and bus trace. >>>> >>>> Tell me which card you have, 1gig or 10gig? Which driver are you >running e1000e or igb or ixgbe? >>>> Can you also provide ethtool -i ethx output. >>>> >>>> Once I know which driver, I send you debug driver. >>> >>> I am using Intel PRO/1000 PT Server Adapter. >>> http://www.intel.com/content/www/us/en/network-adapters/gigabit- >network-adapters/pro-1000-pt.html >>> >>> I am using e1000e driver. >>> >>> I see the problem when I try to mount rootfilesystem using NFS and use >>> MSI interrupt. I see this issue even before I can have cell prompt. >>> Please see first mail in this thread. >>> >>> http://www.mail-archive.com/e1000- >devel@lists.sourceforge.net/msg04894.html >>> >>> Here, you can also see tx ring details when issue occur. >>> Please let me know, if you need any more info. >>> >>> Regards >>> Pratyush >>> >>>> >>>> Thanks. >>>> >>>> -Tushar >>>> >>>> -----Original Message----- >>>> From: netdev-owner@vger.kernel.org [mailto:netdev- >owner@vger.kernel.org] On Behalf Of Pratyush Anand >>>> Sent: Wednesday, January 04, 2012 8:31 PM >>>> To: Greg KH >>>> Cc: Pratyush Anand; e1000-devel@lists.sourceforge.net; >netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux- >pci@vger.kernel.org; Linux NICS >>>> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >>>> >>>> On 1/5/2012 12:52 AM, Greg KH wrote: >>>>> On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: >>>>>> Adding PCI mailing list too, as problem is coming only when MSI is >enabled. >>>>>> >>>>>> If I connect an PCIe analyzer, I see that at the time of issue >>>>>> MRd(64) for 32 words has been issued with a wrong 64 bit address >>>>>> from ethernet card to my RC. >>>>>> In the normal course it always issues MRd(32) only. >>>>> >>>>> Bug in your pcie firmware controller? >>>>> >>>>> . >>>>> >>>> >>>> when you say "Bug in your pcie firmware controller?", is it RC's >>>> software or EP's software? >>>> >>>> Here I am pasting a part of analyzer log converted into text. >>>> Packet(177940), is an upstream request for MSI. Whenever any device >>>> writes at address 0x58A8F8, my PCIe RC considers it as MSI and >generates >>>> an interrupt. So I receive MSI interrupt correctly in my software. >Also >>>> MSI controller is correctly able to point me that the interrupt is >from >>>> ethernet card. >>>> >>>> Now in Packet(178010), ethernet controller sends another upstream >>>> request for MRd(64) of 32 dwords with >Address(AFECEB87:A9D88B00).Since, >>>> this address does not exist in my RC's world so, an UR is returned and >>>> hence the problem occurs. >>>> >>>> Now, question is, why ethernet card is generating inbound request with >>>> such a wrong address. I have taken log of all the tx_desc->buffer_addr >>>> programmed by software in function e1000_tx_queue. None of them is 64 >>>> bit or any invalid address. >>>> >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(177916) Upstream 2.5(x1) TLP(1475) Mem MWr(32)(10:00000) >Length(4) >>>> _______| RequesterID(003:00:0) Tag(2) Address(0EB00200) 1st BE(1111) >>>> _______| Last BE(1111) Data(4 dwords) LCRC(0x44E0407C) >>>> _______| Time Stamp(0013 . 460 549 544 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(177918) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1475) >>>> _______| CRC 16(0x0EB7) Time Stamp(0013 . 460 551 144 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(177940) Upstream 2.5(x1) TLP(1476) Mem MWr(32)(10:00000) >Length(1) >>>> _______| RequesterID(003:00:0) Tag(30) Address(0058A8F8) 1st BE(0011) >>>> _______| Last BE(0000) Data(1 dword) LCRC(0xC21F32B6) >>>> _______| Time Stamp(0013 . 460 588 544 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(177942) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1476) >>>> _______| CRC 16(0x69F5) Time Stamp(0013 . 460 590 088 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(177946) Downstream 2.5(x1) TLP(309) Mem MRd(32)(00:00000) >Length(1) >>>> _______| RequesterID(002:00:0) Tag(19) Address(C01000C0) 1st BE(1111) >>>> _______| Last BE(0000) LCRC(0x91BDA1F5) Time Stamp(0013 . 460 595 936 >s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(177947) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(309) >>>> _______| CRC 16(0x25C6) Time Stamp(0013 . 460 596 368 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(177950) Upstream 2.5(x1) TLP(1477) Cpl CplD(10:01010) Length(1) >>>> _______| RequesterID(002:00:0) Tag(19) CompleterID(003:00:0) >Status(SC) >>>> BCM(0) >>>> _______| Byte Cnt(4) Lwr Addr(0x40) Data(1 dword) LCRC(0x8FE0D922) >>>> _______| Time Stamp(0013 . 460 597 304 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(177952) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1477) >>>> _______| CRC 16(0xC8EE) Time Stamp(0013 . 460 598 840 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(177999) Downstream 2.5(x1) TLP(310) Mem MWr(32)(10:00000) >Length(1) >>>> _______| RequesterID(002:00:0) Tag(0) Address(C0103818) 1st BE(1111) >>>> _______| Last BE(0000) Data(1 dword) LCRC(0xA898D9A1) >>>> _______| Time Stamp(0013 . 460 687 936 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178001) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(310) >>>> _______| CRC 16(0xC6EA) Time Stamp(0013 . 460 688 384 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178004) Upstream 2.5(x1) TLP(1478) Mem MRd(32)(00:00000) >Length(4) >>>> _______| RequesterID(003:00:0) Tag(4) Address(0EAFB990) 1st BE(1111) >>>> _______| Last BE(1111) LCRC(0xB54722D2) Time Stamp(0013 . 460 689 312 >s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178006) Downstream 2.5(x1) TLP(311) Cpl CplD(10:01010) >Length(4) >>>> _______| RequesterID(003:00:0) Tag(4) CompleterID(002:00:0) Status(SC) >>>> BCM(0) >>>> _______| Byte Cnt(16) Lwr Addr(0x10) Data(4 dwords) LCRC(0xFE303776) >>>> _______| Time Stamp(0013 . 460 690 288 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178007) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(311) >>>> _______| CRC 16(0x67F1) Time Stamp(0013 . 460 690 776 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178008) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1478) >>>> _______| CRC 16(0x2BC2) Time Stamp(0013 . 460 690 824 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178010) Upstream 2.5(x1) TLP(1479) Mem MRd(64)(01:00000) >Length(32) >>>> _______| RequesterID(003:00:0) Tag(11) Address(AFECEB87:A9D88B00) 1st >>>> BE(1100) >>>> _______| Last BE(0011) LCRC(0x6BE341C9) Time Stamp(0013 . 460 691 680 >s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178011) Upstream 2.5(x1) TLP(1480) Mem MRd(64)(01:00000) >Length(32) >>>> _______| RequesterID(003:00:0) Tag(8) Address(AFECEB87:A9D88B7C) 1st >>>> BE(1100) >>>> _______| Last BE(0011) LCRC(0xAA5647BD) Time Stamp(0013 . 460 691 808 >s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178012) Upstream 2.5(x1) TLP(1481) Mem MRd(64)(01:00000) >Length(32) >>>> _______| RequesterID(003:00:0) Tag(9) Address(AFECEB87:A9D88BF8) 1st >>>> BE(1100) >>>> _______| Last BE(0011) LCRC(0xEEB1F63F) Time Stamp(0013 . 460 692 120 >s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178013) Upstream 2.5(x1) TLP(1482) Mem MRd(64)(01:00000) >Length(32) >>>> _______| RequesterID(003:00:0) Tag(10) Address(AFECEB87:A9D88C74) 1st >>>> BE(1100) >>>> _______| Last BE(0011) LCRC(0xA508142C) Time Stamp(0013 . 460 692 248 >s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178014) Downstream 2.5(x1) TLP(312) Cpl Cpl(00:01010) Length(0) >>>> _______| RequesterID(003:00:0) Tag(11) CompleterID(002:00:0) >Status(UR)-BAD >>>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x02) LCRC(0xCE5540D2) >>>> _______| Time Stamp(0013 . 460 692 328 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178015) Downstream 2.5(x1) TLP(313) Cpl Cpl(00:01010) Length(0) >>>> _______| RequesterID(003:00:0) Tag(8) CompleterID(002:00:0) >Status(UR)-BAD >>>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7E) LCRC(0x9FE2487D) >>>> _______| Time Stamp(0013 . 460 692 456 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178016) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(312) >>>> _______| CRC 16(0x086E) Time Stamp(0013 . 460 692 760 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178017) Downstream 2.5(x1) TLP(314) Cpl Cpl(00:01010) Length(0) >>>> _______| RequesterID(003:00:0) Tag(9) CompleterID(002:00:0) >Status(UR)-BAD >>>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7A) LCRC(0x097BF4DE) >>>> _______| Time Stamp(0013 . 460 692 776 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178018) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(313) >>>> _______| CRC 16(0xA975) Time Stamp(0013 . 460 692 888 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178019) Downstream 2.5(x1) TLP(315) Cpl Cpl(00:01010) Length(0) >>>> _______| RequesterID(003:00:0) Tag(10) CompleterID(002:00:0) >Status(UR)-BAD >>>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x76) LCRC(0x64BDF921) >>>> _______| Time Stamp(0013 . 460 692 904 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178020) Upstream 2.5(x1) TLP(1483) Msg Msg(01:10000) >>>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>>> _______| Message Code(ERR_FATAL) LCRC(0xCDA53E96) >>>> _______| Time Stamp(0013 . 460 693 184 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178021) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1482) >>>> _______| CRC 16(0xA771) Time Stamp(0013 . 460 693 208 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178023) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(314) >>>> _______| CRC 16(0x4A59) Time Stamp(0013 . 460 693 280 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178024) Upstream 2.5(x1) TLP(1484) Msg Msg(01:10000) >>>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>>> _______| Message Code(ERR_FATAL) LCRC(0x86D9ACB6) >>>> _______| Time Stamp(0013 . 460 693 312 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178025) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(315) >>>> _______| CRC 16(0xEB42) Time Stamp(0013 . 460 693 408 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178026) Upstream 2.5(x1) TLP(1485) Msg Msg(01:10000) >>>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>>> _______| Message Code(ERR_FATAL) LCRC(0xC5120A31) >>>> _______| Time Stamp(0013 . 460 693 632 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178028) Upstream 2.5(x1) TLP(1486) Msg Msg(01:10000) >>>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>>> _______| Message Code(ERR_FATAL) LCRC(0x41499062) >>>> _______| Time Stamp(0013 . 460 693 792 s) >>>> >_______|__________________________________________________________________ >_____ >>>> Packet(178029) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1486) >>>> _______| CRC 16(0x231F) Time Stamp(0013 . 460 694 704 s) >>>> >_______|__________________________________________________________________ >_____ >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe netdev" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> . >>>> >>> >>> . >>> >> >> . >> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Detected Hardware Unit Hang on Intel Wired Ethernet 2012-01-26 21:27 ` Dave, Tushar N @ 2012-01-27 8:03 ` Pratyush Anand 0 siblings, 0 replies; 13+ messages in thread From: Pratyush Anand @ 2012-01-27 8:03 UTC (permalink / raw) To: Dave, Tushar N Cc: Greg KH, Pratyush Anand, e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org, Shiraz HASHIM, Deepak SIKRI, Bhavna YADAV, linux-pci@vger.kernel.org, Linux NICS Hello Tushar, On 1/27/2012 2:57 AM, Dave, Tushar N wrote: >> -----Original Message----- >> From: Pratyush Anand [mailto:pratyush.anand@st.com] >> Sent: Tuesday, January 10, 2012 7:34 PM >> To: Dave, Tushar N >> Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; >> netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux- >> pci@vger.kernel.org; Linux NICS >> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >> >> As I said earlier, issue is reproducible if I try to keep my >> rootfilesystem over NFS. So, after the booting, kernel tries to mount >> rootfs over NFS and it crashes. So, I see issue even before I can reach >> to # prompt. How can I use "ethtool -s ethx msglvl 0x3c00" to enable any >> debug message. May be I can directly change in kernel code to enable this. > > Any update on this? Did you change in-kernel driver source to print the driver HW ring? > If you did and had reproduced the issue please send me the full dmesg log along with bus trace and I'll take a look. I am not able to work with this. Busy with some other work. Will get back ,when I again start working with this issue. Thanks for your support. Regards Pratyush > > -Tushar > >>> -----Original Message----- >>> From: Pratyush Anand [mailto:pratyush.anand@st.com] >>> Sent: Monday, January 09, 2012 8:21 PM >>> To: Dave, Tushar N >>> Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; >> netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux- >> pci@vger.kernel.org; Linux NICS >>> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >>> >>> On 1/7/2012 12:25 AM, Dave, Tushar N wrote: >>>> Pratyush, >>>> >>>> Sorry I got your name reversed. >>>> Are you using in-kernel driver or one from Sourceforge. >>> >>> I am using in-kernel driver from kernel 2.6.37. >>> >>>> Please send me output of ethtool -i ethx. >>> >>> root@192.168.1.10:~# ethtool -i eth0 >>> driver: e1000e >>> version: 1.2.7-k2 >>> firmware-version: 5.11-8 >>> bus-info: 0000:01:00.0 >>> >>> Regards >>> Pratyush >>> >>>> >>>> -Tushar >>>> >>>> -----Original Message----- >>>> From: Pratyush Anand [mailto:pratyush.anand@st.com] >>>> Sent: Thursday, January 05, 2012 8:25 PM >>>> To: Dave, Tushar N >>>> Cc: Greg KH; Pratyush Anand; e1000-devel@lists.sourceforge.net; >> netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux- >> pci@vger.kernel.org; Linux NICS >>>> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >>>> >>>> Thanks Tushar, >>>> >>>> On 1/6/2012 5:24 AM, Dave, Tushar N wrote: >>>>> Anand, >>>>> >>>>> Sorry to hear that you have this issue with card. And yeah, thanks for >> doing the debugging and providing the bus trace. >>>>> I think we should run the debug driver that prints the HW ring details >> when hang occurs. I can provide you a debug driver. You can then install >> debug driver and also let the bus tracer running. Once the issue occurs, >> provide me the full dmesg output (that has HW ring details) and bus trace. >>>>> >>>>> Tell me which card you have, 1gig or 10gig? Which driver are you >> running e1000e or igb or ixgbe? >>>>> Can you also provide ethtool -i ethx output. >>>>> >>>>> Once I know which driver, I send you debug driver. >>>> >>>> I am using Intel PRO/1000 PT Server Adapter. >>>> http://www.intel.com/content/www/us/en/network-adapters/gigabit- >> network-adapters/pro-1000-pt.html >>>> >>>> I am using e1000e driver. >>>> >>>> I see the problem when I try to mount rootfilesystem using NFS and use >>>> MSI interrupt. I see this issue even before I can have cell prompt. >>>> Please see first mail in this thread. >>>> >>>> http://www.mail-archive.com/e1000- >> devel@lists.sourceforge.net/msg04894.html >>>> >>>> Here, you can also see tx ring details when issue occur. >>>> Please let me know, if you need any more info. >>>> >>>> Regards >>>> Pratyush >>>> >>>>> >>>>> Thanks. >>>>> >>>>> -Tushar >>>>> >>>>> -----Original Message----- >>>>> From: netdev-owner@vger.kernel.org [mailto:netdev- >> owner@vger.kernel.org] On Behalf Of Pratyush Anand >>>>> Sent: Wednesday, January 04, 2012 8:31 PM >>>>> To: Greg KH >>>>> Cc: Pratyush Anand; e1000-devel@lists.sourceforge.net; >> netdev@vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux- >> pci@vger.kernel.org; Linux NICS >>>>> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet >>>>> >>>>> On 1/5/2012 12:52 AM, Greg KH wrote: >>>>>> On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote: >>>>>>> Adding PCI mailing list too, as problem is coming only when MSI is >> enabled. >>>>>>> >>>>>>> If I connect an PCIe analyzer, I see that at the time of issue >>>>>>> MRd(64) for 32 words has been issued with a wrong 64 bit address >>>>>>> from ethernet card to my RC. >>>>>>> In the normal course it always issues MRd(32) only. >>>>>> >>>>>> Bug in your pcie firmware controller? >>>>>> >>>>>> . >>>>>> >>>>> >>>>> when you say "Bug in your pcie firmware controller?", is it RC's >>>>> software or EP's software? >>>>> >>>>> Here I am pasting a part of analyzer log converted into text. >>>>> Packet(177940), is an upstream request for MSI. Whenever any device >>>>> writes at address 0x58A8F8, my PCIe RC considers it as MSI and >> generates >>>>> an interrupt. So I receive MSI interrupt correctly in my software. >> Also >>>>> MSI controller is correctly able to point me that the interrupt is >> from >>>>> ethernet card. >>>>> >>>>> Now in Packet(178010), ethernet controller sends another upstream >>>>> request for MRd(64) of 32 dwords with >> Address(AFECEB87:A9D88B00).Since, >>>>> this address does not exist in my RC's world so, an UR is returned and >>>>> hence the problem occurs. >>>>> >>>>> Now, question is, why ethernet card is generating inbound request with >>>>> such a wrong address. I have taken log of all the tx_desc->buffer_addr >>>>> programmed by software in function e1000_tx_queue. None of them is 64 >>>>> bit or any invalid address. >>>>> >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(177916) Upstream 2.5(x1) TLP(1475) Mem MWr(32)(10:00000) >> Length(4) >>>>> _______| RequesterID(003:00:0) Tag(2) Address(0EB00200) 1st BE(1111) >>>>> _______| Last BE(1111) Data(4 dwords) LCRC(0x44E0407C) >>>>> _______| Time Stamp(0013 . 460 549 544 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(177918) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1475) >>>>> _______| CRC 16(0x0EB7) Time Stamp(0013 . 460 551 144 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(177940) Upstream 2.5(x1) TLP(1476) Mem MWr(32)(10:00000) >> Length(1) >>>>> _______| RequesterID(003:00:0) Tag(30) Address(0058A8F8) 1st BE(0011) >>>>> _______| Last BE(0000) Data(1 dword) LCRC(0xC21F32B6) >>>>> _______| Time Stamp(0013 . 460 588 544 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(177942) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1476) >>>>> _______| CRC 16(0x69F5) Time Stamp(0013 . 460 590 088 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(177946) Downstream 2.5(x1) TLP(309) Mem MRd(32)(00:00000) >> Length(1) >>>>> _______| RequesterID(002:00:0) Tag(19) Address(C01000C0) 1st BE(1111) >>>>> _______| Last BE(0000) LCRC(0x91BDA1F5) Time Stamp(0013 . 460 595 936 >> s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(177947) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(309) >>>>> _______| CRC 16(0x25C6) Time Stamp(0013 . 460 596 368 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(177950) Upstream 2.5(x1) TLP(1477) Cpl CplD(10:01010) Length(1) >>>>> _______| RequesterID(002:00:0) Tag(19) CompleterID(003:00:0) >> Status(SC) >>>>> BCM(0) >>>>> _______| Byte Cnt(4) Lwr Addr(0x40) Data(1 dword) LCRC(0x8FE0D922) >>>>> _______| Time Stamp(0013 . 460 597 304 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(177952) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1477) >>>>> _______| CRC 16(0xC8EE) Time Stamp(0013 . 460 598 840 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(177999) Downstream 2.5(x1) TLP(310) Mem MWr(32)(10:00000) >> Length(1) >>>>> _______| RequesterID(002:00:0) Tag(0) Address(C0103818) 1st BE(1111) >>>>> _______| Last BE(0000) Data(1 dword) LCRC(0xA898D9A1) >>>>> _______| Time Stamp(0013 . 460 687 936 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178001) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(310) >>>>> _______| CRC 16(0xC6EA) Time Stamp(0013 . 460 688 384 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178004) Upstream 2.5(x1) TLP(1478) Mem MRd(32)(00:00000) >> Length(4) >>>>> _______| RequesterID(003:00:0) Tag(4) Address(0EAFB990) 1st BE(1111) >>>>> _______| Last BE(1111) LCRC(0xB54722D2) Time Stamp(0013 . 460 689 312 >> s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178006) Downstream 2.5(x1) TLP(311) Cpl CplD(10:01010) >> Length(4) >>>>> _______| RequesterID(003:00:0) Tag(4) CompleterID(002:00:0) Status(SC) >>>>> BCM(0) >>>>> _______| Byte Cnt(16) Lwr Addr(0x10) Data(4 dwords) LCRC(0xFE303776) >>>>> _______| Time Stamp(0013 . 460 690 288 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178007) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(311) >>>>> _______| CRC 16(0x67F1) Time Stamp(0013 . 460 690 776 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178008) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1478) >>>>> _______| CRC 16(0x2BC2) Time Stamp(0013 . 460 690 824 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178010) Upstream 2.5(x1) TLP(1479) Mem MRd(64)(01:00000) >> Length(32) >>>>> _______| RequesterID(003:00:0) Tag(11) Address(AFECEB87:A9D88B00) 1st >>>>> BE(1100) >>>>> _______| Last BE(0011) LCRC(0x6BE341C9) Time Stamp(0013 . 460 691 680 >> s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178011) Upstream 2.5(x1) TLP(1480) Mem MRd(64)(01:00000) >> Length(32) >>>>> _______| RequesterID(003:00:0) Tag(8) Address(AFECEB87:A9D88B7C) 1st >>>>> BE(1100) >>>>> _______| Last BE(0011) LCRC(0xAA5647BD) Time Stamp(0013 . 460 691 808 >> s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178012) Upstream 2.5(x1) TLP(1481) Mem MRd(64)(01:00000) >> Length(32) >>>>> _______| RequesterID(003:00:0) Tag(9) Address(AFECEB87:A9D88BF8) 1st >>>>> BE(1100) >>>>> _______| Last BE(0011) LCRC(0xEEB1F63F) Time Stamp(0013 . 460 692 120 >> s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178013) Upstream 2.5(x1) TLP(1482) Mem MRd(64)(01:00000) >> Length(32) >>>>> _______| RequesterID(003:00:0) Tag(10) Address(AFECEB87:A9D88C74) 1st >>>>> BE(1100) >>>>> _______| Last BE(0011) LCRC(0xA508142C) Time Stamp(0013 . 460 692 248 >> s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178014) Downstream 2.5(x1) TLP(312) Cpl Cpl(00:01010) Length(0) >>>>> _______| RequesterID(003:00:0) Tag(11) CompleterID(002:00:0) >> Status(UR)-BAD >>>>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x02) LCRC(0xCE5540D2) >>>>> _______| Time Stamp(0013 . 460 692 328 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178015) Downstream 2.5(x1) TLP(313) Cpl Cpl(00:01010) Length(0) >>>>> _______| RequesterID(003:00:0) Tag(8) CompleterID(002:00:0) >> Status(UR)-BAD >>>>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7E) LCRC(0x9FE2487D) >>>>> _______| Time Stamp(0013 . 460 692 456 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178016) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(312) >>>>> _______| CRC 16(0x086E) Time Stamp(0013 . 460 692 760 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178017) Downstream 2.5(x1) TLP(314) Cpl Cpl(00:01010) Length(0) >>>>> _______| RequesterID(003:00:0) Tag(9) CompleterID(002:00:0) >> Status(UR)-BAD >>>>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x7A) LCRC(0x097BF4DE) >>>>> _______| Time Stamp(0013 . 460 692 776 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178018) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(313) >>>>> _______| CRC 16(0xA975) Time Stamp(0013 . 460 692 888 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178019) Downstream 2.5(x1) TLP(315) Cpl Cpl(00:01010) Length(0) >>>>> _______| RequesterID(003:00:0) Tag(10) CompleterID(002:00:0) >> Status(UR)-BAD >>>>> _______| BCM(0) Byte Cnt(124) Lwr Addr(0x76) LCRC(0x64BDF921) >>>>> _______| Time Stamp(0013 . 460 692 904 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178020) Upstream 2.5(x1) TLP(1483) Msg Msg(01:10000) >>>>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>>>> _______| Message Code(ERR_FATAL) LCRC(0xCDA53E96) >>>>> _______| Time Stamp(0013 . 460 693 184 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178021) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1482) >>>>> _______| CRC 16(0xA771) Time Stamp(0013 . 460 693 208 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178023) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(314) >>>>> _______| CRC 16(0x4A59) Time Stamp(0013 . 460 693 280 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178024) Upstream 2.5(x1) TLP(1484) Msg Msg(01:10000) >>>>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>>>> _______| Message Code(ERR_FATAL) LCRC(0x86D9ACB6) >>>>> _______| Time Stamp(0013 . 460 693 312 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178025) Upstream 2.5(x1) DLLP ACK AckNak_Seq_Num(315) >>>>> _______| CRC 16(0xEB42) Time Stamp(0013 . 460 693 408 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178026) Upstream 2.5(x1) TLP(1485) Msg Msg(01:10000) >>>>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>>>> _______| Message Code(ERR_FATAL) LCRC(0xC5120A31) >>>>> _______| Time Stamp(0013 . 460 693 632 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178028) Upstream 2.5(x1) TLP(1486) Msg Msg(01:10000) >>>>> _______| Msg Routing(To RC) Length(0) RequesterID(003:00:0) Tag(31) >>>>> _______| Message Code(ERR_FATAL) LCRC(0x41499062) >>>>> _______| Time Stamp(0013 . 460 693 792 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> Packet(178029) Downstream 2.5(x1) DLLP ACK AckNak_Seq_Num(1486) >>>>> _______| CRC 16(0x231F) Time Stamp(0013 . 460 694 704 s) >>>>> >> _______|__________________________________________________________________ >> _____ >>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> . >>>>> >>>> >>>> . >>>> >>> >>> . >>> > > . > ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2012-01-27 8:04 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAHM4w1=i5dYy+ReqMO=m6W96CNaQ+O=jw+pKQok0nmjb5-fNAw@mail.gmail.com>
2012-01-04 11:01 ` Detected Hardware Unit Hang on Intel Wired Ethernet Pratyush Anand
2012-01-04 19:22 ` Greg KH
2012-01-05 4:30 ` Pratyush Anand
2012-01-05 16:46 ` Greg KH
2012-01-05 23:54 ` Dave, Tushar N
2012-01-06 4:25 ` Pratyush Anand
2012-01-06 18:55 ` Dave, Tushar N
2012-01-10 4:20 ` Pratyush Anand
2012-01-11 1:10 ` Dave, Tushar N
2012-01-11 3:33 ` Pratyush Anand
2012-01-11 3:48 ` Dave, Tushar N
2012-01-26 21:27 ` Dave, Tushar N
2012-01-27 8:03 ` Pratyush Anand
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).