* i.Mx6Quad - eth0: tx queue full! @ 2013-01-28 17:39 Vikram Narayanan 2013-01-28 19:47 ` Troy Kisky 0 siblings, 1 reply; 13+ messages in thread From: Vikram Narayanan @ 2013-01-28 17:39 UTC (permalink / raw) To: netdev; +Cc: LAK, shawn.guo, Uwe Kleine-König, Greg Ungerer Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform gives me the below error when flooded with ping requests. == Start log == [ 2555.004031] ------------[ cut here ]------------ [ 2555.009740] WARNING: at net/sched/sch_generic.c:254 dev_watchdog+0x298/0x2b8() [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out [ 2555.026733] Modules linked in: [ 2555.030598] Backtrace: [ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from [<803b8494>] (dump_stack+0x18/0x1c) [ 2555.044438] r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80 [ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>] (warn_slowpath_common+0x54/0x6c) [ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from [<8001dfc4>] (warn_slowpath_fmt+0x38/0x40) [ 2555.073936] r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4 r4:8f977000 r3:00000009 [ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from [<80302f64>] (dev_watchdog+0x298/0x2b8) [ 2555.095535] r3:8f977000 r2:8049f6d4 [ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>] (call_timer_fn.isra.33+0x2c/0x8c) [ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from [<8002af48>] (run_timer_softirq+0x1f0/0x204) [ 2555.122240] r7:80571114 r6:805040c0 r5:00000000 r4:80570900 [ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from [<80025750>] (__do_softirq+0xc8/0x180) [ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>] (irq_exit+0x88/0x90) [ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>] (handle_IRQ+0x44/0x98) [ 2555.160112] r4:804ffde0 r3:00000220 [ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>] (gic_handle_irq+0x30/0x64) [ 2555.174956] r6:80503f28 r5:8050a518 r4:f400010c r3:00000000 [ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from [<8000df80>] (__irq_svc+0x40/0x50) [ 2555.192737] Exception stack(0x80503f28 to 0x80503f70) [ 2555.198639] 3f20: 8052f150 a0000093 00000000 00000000 80502000 8052ed08 [ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000 80503f7c 80503f80 80503f70 [ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff [ 2555.224730] r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8 [ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>] (cpu_idle+0xcc/0x114) [ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>] (rest_init+0x64/0x7c) [ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>] (start_kernel+0x258/0x298) [ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>] (0x10008078) [ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]--- [ 2555.277270] eth0: tx queue full!. [ 2555.288776] eth0: tx queue full!. [ 2555.293594] eth0: tx queue full!. [ 2555.297944] eth0: tx queue full!. [ 2555.302229] eth0: tx queue full!. ...... continues indefinitely and needs a reboot to recover the system. == End log == i.Mx6Quad's MAC is connected to SMSC LAN88730 PHY via an MII interface. I found a similar thread [1], but that solution didn't work for me. Any ideas on why the fec driver is unhappy? Thanks, Vikram [1] https://community.freescale.com/thread/281457 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-01-28 17:39 i.Mx6Quad - eth0: tx queue full! Vikram Narayanan @ 2013-01-28 19:47 ` Troy Kisky 2013-01-29 16:33 ` Vikram Narayanan 2013-01-29 16:34 ` Vikram Narayanan 0 siblings, 2 replies; 13+ messages in thread From: Troy Kisky @ 2013-01-28 19:47 UTC (permalink / raw) To: Vikram Narayanan Cc: netdev, Greg Ungerer, shawn.guo, LAK, Uwe Kleine-König On 1/28/2013 10:39 AM, Vikram Narayanan wrote: > Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform > gives me the below error when flooded with ping requests. > > == Start log == > [ 2555.004031] ------------[ cut here ]------------ > [ 2555.009740] WARNING: at net/sched/sch_generic.c:254 dev_watchdog+0x298/0x2b8() > [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out I think the tx interrupt status bit was lost. The packets were transmitted, but the interrupt never happened. The controller should have been reset here, but perhaps a bug with the reset code. Are you using the mainline kernel, or a version Freescale's kernel. mainline fec_restart does not reset tx_full You can try adding fep->tx_full = 0; to fec_restart, though it would be better to call fec_enet_tx in fec_timeout and skip the call to fec_restart if it returns some packets. > [ 2555.026733] Modules linked in: > [ 2555.030598] Backtrace: > [ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from [<803b8494>] (dump_stack+0x18/0x1c) > [ 2555.044438] r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80 > [ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>] (warn_slowpath_common+0x54/0x6c) > [ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from [<8001dfc4>] (warn_slowpath_fmt+0x38/0x40) > [ 2555.073936] r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4 r4:8f977000 > r3:00000009 > [ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from [<80302f64>] (dev_watchdog+0x298/0x2b8) > [ 2555.095535] r3:8f977000 r2:8049f6d4 > [ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>] (call_timer_fn.isra.33+0x2c/0x8c) > [ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from [<8002af48>] (run_timer_softirq+0x1f0/0x204) > [ 2555.122240] r7:80571114 r6:805040c0 r5:00000000 r4:80570900 > [ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from [<80025750>] (__do_softirq+0xc8/0x180) > [ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>] (irq_exit+0x88/0x90) > [ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>] (handle_IRQ+0x44/0x98) > [ 2555.160112] r4:804ffde0 r3:00000220 > [ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>] (gic_handle_irq+0x30/0x64) > [ 2555.174956] r6:80503f28 r5:8050a518 r4:f400010c r3:00000000 > [ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from [<8000df80>] (__irq_svc+0x40/0x50) > [ 2555.192737] Exception stack(0x80503f28 to 0x80503f70) > [ 2555.198639] 3f20: 8052f150 a0000093 00000000 00000000 80502000 8052ed08 > [ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000 80503f7c 80503f80 80503f70 > [ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff > [ 2555.224730] r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8 > [ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>] (cpu_idle+0xcc/0x114) > [ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>] (rest_init+0x64/0x7c) > [ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>] (start_kernel+0x258/0x298) > [ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>] (0x10008078) > [ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]--- > [ 2555.277270] eth0: tx queue full!. > [ 2555.288776] eth0: tx queue full!. > [ 2555.293594] eth0: tx queue full!. > [ 2555.297944] eth0: tx queue full!. > [ 2555.302229] eth0: tx queue full!. All the packet have been transmitted but the transmit queue is full, so no more tx interrupts can happen to replace the previously lost tx interrupt. I've seen MII interrupts clear the TX interrupt status bit. > ...... continues indefinitely and needs a reboot to recover the system. > == End log == > > i.Mx6Quad's MAC is connected to SMSC LAN88730 PHY via an MII interface. > > I found a similar thread [1], but that solution didn't work for me. > Any ideas on why the fec driver is unhappy? > > Thanks, > Vikram > > [1] https://community.freescale.com/thread/281457 > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-01-28 19:47 ` Troy Kisky @ 2013-01-29 16:33 ` Vikram Narayanan 2013-01-29 16:45 ` Fabio Estevam 2013-01-29 16:34 ` Vikram Narayanan 1 sibling, 1 reply; 13+ messages in thread From: Vikram Narayanan @ 2013-01-29 16:33 UTC (permalink / raw) To: Troy Kisky; +Cc: netdev, Greg Ungerer, shawn.guo, LAK, Uwe Kleine-König On 1/29/2013 1:17 AM, Troy Kisky wrote: > On 1/28/2013 10:39 AM, Vikram Narayanan wrote: >> Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform >> gives me the below error when flooded with ping requests. >> >> == Start log == >> [ 2555.004031] ------------[ cut here ]------------ >> [ 2555.009740] WARNING: at net/sched/sch_generic.c:254 >> dev_watchdog+0x298/0x2b8() >> [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out > > I think the tx interrupt status bit was lost. The packets were > transmitted, but the interrupt never > happened. The controller should have been reset here, but perhaps a bug > with the reset code. > Are you using the mainline kernel, or a version Freescale's kernel. I tried with both the kernels. Freescale's and mainline results in the same error. > mainline fec_restart does not reset tx_full > > You can try adding > fep->tx_full = 0; With this there was no improvement. > to fec_restart, though it would be better to call fec_enet_tx in > fec_timeout > and skip the call to fec_restart if it returns some packets. fec_enet_tx returns void, how do I check for the return packets. I'm sorry, I couldn't get your point here. > >> [ 2555.026733] Modules linked in: >> [ 2555.030598] Backtrace: >> [ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from >> [<803b8494>] (dump_stack+0x18/0x1c) >> [ 2555.044438] r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80 >> [ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>] >> (warn_slowpath_common+0x54/0x6c) >> [ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from >> [<8001dfc4>] (warn_slowpath_fmt+0x38/0x40) >> [ 2555.073936] r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4 >> r4:8f977000 >> r3:00000009 >> [ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from >> [<80302f64>] (dev_watchdog+0x298/0x2b8) >> [ 2555.095535] r3:8f977000 r2:8049f6d4 >> [ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>] >> (call_timer_fn.isra.33+0x2c/0x8c) >> [ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from >> [<8002af48>] (run_timer_softirq+0x1f0/0x204) >> [ 2555.122240] r7:80571114 r6:805040c0 r5:00000000 r4:80570900 >> [ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from >> [<80025750>] (__do_softirq+0xc8/0x180) >> [ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>] >> (irq_exit+0x88/0x90) >> [ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>] >> (handle_IRQ+0x44/0x98) >> [ 2555.160112] r4:804ffde0 r3:00000220 >> [ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>] >> (gic_handle_irq+0x30/0x64) >> [ 2555.174956] r6:80503f28 r5:8050a518 r4:f400010c r3:00000000 >> [ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from >> [<8000df80>] (__irq_svc+0x40/0x50) >> [ 2555.192737] Exception stack(0x80503f28 to 0x80503f70) >> [ 2555.198639] 3f20: 8052f150 a0000093 00000000 >> 00000000 80502000 8052ed08 >> [ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000 >> 80503f7c 80503f80 80503f70 >> [ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff >> [ 2555.224730] r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8 >> [ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>] >> (cpu_idle+0xcc/0x114) >> [ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>] >> (rest_init+0x64/0x7c) >> [ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>] >> (start_kernel+0x258/0x298) >> [ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>] >> (0x10008078) >> [ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]--- >> [ 2555.277270] eth0: tx queue full!. >> [ 2555.288776] eth0: tx queue full!. >> [ 2555.293594] eth0: tx queue full!. >> [ 2555.297944] eth0: tx queue full!. >> [ 2555.302229] eth0: tx queue full!. > All the packet have been transmitted but the transmit queue is full, so > no more > tx interrupts can happen to replace the previously lost tx interrupt. > > I've seen MII interrupts clear the TX interrupt status bit. > Do you suspect the MAC <-> PHY interconnect? I'm having an MII interface. Regards, Vikram ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-01-29 16:33 ` Vikram Narayanan @ 2013-01-29 16:45 ` Fabio Estevam 2013-01-29 16:55 ` Vikram Narayanan 0 siblings, 1 reply; 13+ messages in thread From: Fabio Estevam @ 2013-01-29 16:45 UTC (permalink / raw) To: Vikram Narayanan Cc: Greg Ungerer, netdev, Troy Kisky, Uwe Kleine-König, shawn.guo, LAK On Tue, Jan 29, 2013 at 2:33 PM, Vikram Narayanan <vikram186@gmail.com> wrote: > I tried with both the kernels. Freescale's and mainline results in the same > error. Do you also see this issue with a 3.6 or 3.7 stable kernels? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-01-29 16:45 ` Fabio Estevam @ 2013-01-29 16:55 ` Vikram Narayanan 0 siblings, 0 replies; 13+ messages in thread From: Vikram Narayanan @ 2013-01-29 16:55 UTC (permalink / raw) To: Fabio Estevam; +Cc: Troy Kisky, netdev On 1/29/2013 10:15 PM, Fabio Estevam wrote: > On Tue, Jan 29, 2013 at 2:33 PM, Vikram Narayanan <vikram186@gmail.com> wrote: > >> I tried with both the kernels. Freescale's and mainline results in the same >> error. > > Do you also see this issue with a 3.6 or 3.7 stable kernels? > I didn't try with the 3.6 or 3.7 stable. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-01-28 19:47 ` Troy Kisky 2013-01-29 16:33 ` Vikram Narayanan @ 2013-01-29 16:34 ` Vikram Narayanan 2013-01-29 18:46 ` Troy Kisky 1 sibling, 1 reply; 13+ messages in thread From: Vikram Narayanan @ 2013-01-29 16:34 UTC (permalink / raw) To: Troy Kisky; +Cc: netdev, Greg Ungerer, shawn.guo, LAK, Uwe Kleine-König On 1/29/2013 1:17 AM, Troy Kisky wrote: > On 1/28/2013 10:39 AM, Vikram Narayanan wrote: >> Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform >> gives me the below error when flooded with ping requests. >> >> == Start log == >> [ 2555.004031] ------------[ cut here ]------------ >> [ 2555.009740] WARNING: at net/sched/sch_generic.c:254 >> dev_watchdog+0x298/0x2b8() >> [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out > > I think the tx interrupt status bit was lost. The packets were > transmitted, but the interrupt never > happened. The controller should have been reset here, but perhaps a bug > with the reset code. > Are you using the mainline kernel, or a version Freescale's kernel. I tried with both the kernels. Freescale's and mainline results in the same error. > mainline fec_restart does not reset tx_full > > You can try adding > fep->tx_full = 0; With this there was no improvement. > to fec_restart, though it would be better to call fec_enet_tx in > fec_timeout > and skip the call to fec_restart if it returns some packets. fec_enet_tx returns void, how do I check for the return packets. I'm sorry, I couldn't get your point here. > >> [ 2555.026733] Modules linked in: >> [ 2555.030598] Backtrace: >> [ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from >> [<803b8494>] (dump_stack+0x18/0x1c) >> [ 2555.044438] r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80 >> [ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>] >> (warn_slowpath_common+0x54/0x6c) >> [ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from >> [<8001dfc4>] (warn_slowpath_fmt+0x38/0x40) >> [ 2555.073936] r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4 >> r4:8f977000 >> r3:00000009 >> [ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from >> [<80302f64>] (dev_watchdog+0x298/0x2b8) >> [ 2555.095535] r3:8f977000 r2:8049f6d4 >> [ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>] >> (call_timer_fn.isra.33+0x2c/0x8c) >> [ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from >> [<8002af48>] (run_timer_softirq+0x1f0/0x204) >> [ 2555.122240] r7:80571114 r6:805040c0 r5:00000000 r4:80570900 >> [ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from >> [<80025750>] (__do_softirq+0xc8/0x180) >> [ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>] >> (irq_exit+0x88/0x90) >> [ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>] >> (handle_IRQ+0x44/0x98) >> [ 2555.160112] r4:804ffde0 r3:00000220 >> [ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>] >> (gic_handle_irq+0x30/0x64) >> [ 2555.174956] r6:80503f28 r5:8050a518 r4:f400010c r3:00000000 >> [ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from >> [<8000df80>] (__irq_svc+0x40/0x50) >> [ 2555.192737] Exception stack(0x80503f28 to 0x80503f70) >> [ 2555.198639] 3f20: 8052f150 a0000093 00000000 >> 00000000 80502000 8052ed08 >> [ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000 >> 80503f7c 80503f80 80503f70 >> [ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff >> [ 2555.224730] r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8 >> [ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>] >> (cpu_idle+0xcc/0x114) >> [ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>] >> (rest_init+0x64/0x7c) >> [ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>] >> (start_kernel+0x258/0x298) >> [ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>] >> (0x10008078) >> [ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]--- >> [ 2555.277270] eth0: tx queue full!. >> [ 2555.288776] eth0: tx queue full!. >> [ 2555.293594] eth0: tx queue full!. >> [ 2555.297944] eth0: tx queue full!. >> [ 2555.302229] eth0: tx queue full!. > All the packet have been transmitted but the transmit queue is full, so > no more > tx interrupts can happen to replace the previously lost tx interrupt. > > I've seen MII interrupts clear the TX interrupt status bit. > Do you suspect the MAC <-> PHY interconnect? I'm having an MII interface. Regards, Vikram ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-01-29 16:34 ` Vikram Narayanan @ 2013-01-29 18:46 ` Troy Kisky 2013-01-30 8:37 ` Duan Fugang-B38611 2013-01-30 15:13 ` Vikram Narayanan 0 siblings, 2 replies; 13+ messages in thread From: Troy Kisky @ 2013-01-29 18:46 UTC (permalink / raw) To: Vikram Narayanan Cc: netdev, Greg Ungerer, shawn.guo, LAK, Uwe Kleine-König, Fabio Estevam On 1/29/2013 9:34 AM, Vikram Narayanan wrote: > On 1/29/2013 1:17 AM, Troy Kisky wrote: >> On 1/28/2013 10:39 AM, Vikram Narayanan wrote: >>> Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform >>> gives me the below error when flooded with ping requests. >>> >>> == Start log == >>> [ 2555.004031] ------------[ cut here ]------------ >>> [ 2555.009740] WARNING: at net/sched/sch_generic.c:254 >>> dev_watchdog+0x298/0x2b8() >>> [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out >> >> I think the tx interrupt status bit was lost. The packets were >> transmitted, but the interrupt never >> happened. The controller should have been reset here, but perhaps a bug >> with the reset code. >> Are you using the mainline kernel, or a version Freescale's kernel. > > I tried with both the kernels. Freescale's and mainline results in the > same error. > >> mainline fec_restart does not reset tx_full >> >> You can try adding >> fep->tx_full = 0; > > With this there was no improvement. I have fixed this bug (and more) on Freescale's kernel (imx-3.0.35_1.1.0). I created a branch you can try. Feel free to port to mainline. This is the patch that should fix your problem fec: clear TX_FULL in fec_restart git://github.com/boundarydevices/linux-imx6.git ethernet_test Please let me know results. Thanks Troy ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: i.Mx6Quad - eth0: tx queue full! 2013-01-29 18:46 ` Troy Kisky @ 2013-01-30 8:37 ` Duan Fugang-B38611 2013-01-30 15:18 ` Vikram Narayanan 2013-01-30 15:13 ` Vikram Narayanan 1 sibling, 1 reply; 13+ messages in thread From: Duan Fugang-B38611 @ 2013-01-30 8:37 UTC (permalink / raw) To: Troy Kisky, Vikram Narayanan Cc: netdev@vger.kernel.org, Greg Ungerer, shawn.guo@linaro.org, LAK, Uwe Kleine-König, Fabio Estevam Hi, all The issue cannot be found kernel 3.0.35 branch: (even run stress test with IPU and VPU) ssh://sw-git01-tx30/git/sw_git/repos/linux-2.6-imx.git branch name: imx_3.0.35 The patch as below: From 91a0c892263e57ecde9e9ff38be3acdb7f66a17f Mon Sep 17 00:00:00 2001 From: Fugang Duan <B38611@freescale.com> Date: Thu, 9 Aug 2012 17:59:44 +0800 Subject: [PATCH] ENGR00180288 - FEC : Fix kernel dump about eth0 Kernel dump when do wifi stress test with suspend and resume as below: eth0: tx queue full!. remove wake up source irq 103 PM: resume of devices complete after 348.934 msecs Restarting tasks ... done. ------------[ cut here ]------------ WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x284/0x2a8() NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out Modules linked in: ar6000 [<8004482c>] (unwind_backtrace+0x0/0xf8) from [<80068cd0>] (warn_slowpath_common+0x4c/0x64) [<80068cd0>] (warn_slowpath_common+0x4c/0x64)from [<80068d7c>] (warn_slowpath_fmt+0x30/0x40) [<80068d7c>] (warn_slowpath_fmt+0x30/0x40) from [<803f0c50>] (dev_watchdog+0x284/0x2a8) [<803f0c50>] (dev_watchdog+0x284/0x2a8) from [<80074430>] (run_timer_softirq+0xec/0x214) [<80074430>] (run_timer_softirq+0xec/0x214) from [<8006e524>] (__do_softirq+0xac/0x140) [<8006e524>] (__do_softirq+0xac/0x140) from [<8006ea60>] (irq_exit+0x94/0x9c) [<8006ea60>] (irq_exit+0x94/0x9c) from [<80039240>] (do_local_timer+0x54/0x70) [<80039240>] (do_local_timer+0x54/0x70) from [<8003ea0c>] (__irq_svc+0x4c/0xe8) Exception stack(0x80a2bf68 to 0x80a2bfb0) bf60: 0000001f 80a3babc 80a2bfb0 00000000 80a2a000 80a7b8e4 bf80: 804befcc 80a3ee7c 1000406a 412fc09a 00000000 00000000 80a81440 80a2bfb0 bfa0: 8003fa64 8003fa68 60000013 ffffffff [<8003ea0c>] (__irq_svc+0x4c/0xe8) from [<8003fa68>] (default_idle+0x24/0x28) [<8003fa68>] (default_idle+0x24/0x28) from [<8003fc60>] (cpu_idle+0xbc/0xfc) [<8003fc60>] (cpu_idle+0xbc/0xfc) from [<80008878>] (start_kernel+0x258/0x29c) [<80008878>] (start_kernel+0x258/0x29c) from [<10008040>] (0x10008040) ---[ end trace 30671ac42e272c2d ]--- But ethernet and system still be alive. In sometime,the issue will cause system hang like "nfs: server 10.192.242.179 not responding, still trying". The root cause is tx buffer descriptors are not cleaned when ethernet resume back. Signed-off-by: Fugang Duan <B38611@freescale.com> drivers/net/fec.c | 39 ++++++++++++++++++++++++++------------- 1 files changed, 26 insertions(+), 13 deletions(-) diff --git a/drivers/net/fec.c b/drivers/net/fec.c index f007bf0..b1fa464 100755 --- a/drivers/net/fec.c +++ b/drivers/net/fec.c @@ -1456,6 +1456,28 @@ static const struct net_device_ops fec_netdev_ops = { #endif }; +/* Init TX buffer descriptors + */ +static void fec_enet_txbd_init(struct net_device *dev) +{ + struct fec_enet_private *fep = netdev_priv(dev); + struct bufdesc *bdp; + int i; + + /* ...and the same for transmit */ + bdp = fep->tx_bd_base; + for (i = 0; i < TX_RING_SIZE; i++) { + + /* Initialize the BD for every fragment in the page. */ + bdp->cbd_sc = 0; + bdp++; + } + + /* Set the last buffer to wrap */ + bdp--; + bdp->cbd_sc |= BD_SC_WRAP; +} + /* * XXX: We need to clean up on failure exits here. * @@ -1512,19 +1534,8 @@ static int fec_enet_init(struct net_device *ndev) bdp--; bdp->cbd_sc |= BD_SC_WRAP; - /* ...and the same for transmit */ - bdp = fep->tx_bd_base; - for (i = 0; i < TX_RING_SIZE; i++) { - - /* Initialize the BD for every fragment in the page. */ - bdp->cbd_sc = 0; - bdp->cbd_bufaddr = 0; - bdp++; - } - - /* Set the last buffer to wrap */ - bdp--; - bdp->cbd_sc |= BD_SC_WRAP; + /* Init transmit descriptors */ + fec_enet_txbd_init(ndev); fec_restart(ndev, 0); @@ -1575,6 +1586,8 @@ fec_restart(struct net_device *dev, int duplex) writel(fep->bd_dma, fep->hwp + FEC_R_DES_START); writel((unsigned long)fep->bd_dma + sizeof(struct bufdesc) * RX_RING_SIZE, fep->hwp + FEC_X_DES_START); + /* Reinit transmit descriptors */ + fec_enet_txbd_init(dev); fep->dirty_tx = fep->cur_tx = fep->tx_bd_base; fep->cur_rx = fep->rx_bd_base; -- 1.7.0.4 Thanks, Andy -----Original Message----- From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Troy Kisky Sent: Wednesday, January 30, 2013 2:47 AM To: Vikram Narayanan Cc: netdev@vger.kernel.org; Greg Ungerer; shawn.guo@linaro.org; LAK; Uwe Kleine-König; Fabio Estevam Subject: Re: i.Mx6Quad - eth0: tx queue full! On 1/29/2013 9:34 AM, Vikram Narayanan wrote: > On 1/29/2013 1:17 AM, Troy Kisky wrote: >> On 1/28/2013 10:39 AM, Vikram Narayanan wrote: >>> Running the latest head <linux-2.6.git> on an i.Mx6Quad based >>> platform gives me the below error when flooded with ping requests. >>> >>> == Start log == >>> [ 2555.004031] ------------[ cut here ]------------ [ 2555.009740] >>> WARNING: at net/sched/sch_generic.c:254 >>> dev_watchdog+0x298/0x2b8() >>> [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed >>> out >> >> I think the tx interrupt status bit was lost. The packets were >> transmitted, but the interrupt never happened. The controller should >> have been reset here, but perhaps a bug with the reset code. >> Are you using the mainline kernel, or a version Freescale's kernel. > > I tried with both the kernels. Freescale's and mainline results in the > same error. > >> mainline fec_restart does not reset tx_full >> >> You can try adding >> fep->tx_full = 0; > > With this there was no improvement. I have fixed this bug (and more) on Freescale's kernel (imx-3.0.35_1.1.0). I created a branch you can try. Feel free to port to mainline. This is the patch that should fix your problem fec: clear TX_FULL in fec_restart git://github.com/boundarydevices/linux-imx6.git ethernet_test Please let me know results. Thanks Troy -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-01-30 8:37 ` Duan Fugang-B38611 @ 2013-01-30 15:18 ` Vikram Narayanan 0 siblings, 0 replies; 13+ messages in thread From: Vikram Narayanan @ 2013-01-30 15:18 UTC (permalink / raw) To: Duan Fugang-B38611 Cc: Troy Kisky, netdev@vger.kernel.org, shawn.guo@linaro.org, LAK, Fabio Estevam Hello Duan, On 1/30/2013 2:07 PM, Duan Fugang-B38611 wrote: > Hi, all > > The issue cannot be found kernel 3.0.35 branch: (even run stress test with IPU and VPU) > ssh://sw-git01-tx30/git/sw_git/repos/linux-2.6-imx.git Where is this git located? > branch name: imx_3.0.35 Troy's branch (located in github) works fine for me. I've kept it for a long run with heavy network load. Should I encounter some errors, I'll post back. > The patch as below: > > From 91a0c892263e57ecde9e9ff38be3acdb7f66a17f Mon Sep 17 00:00:00 2001 > From: Fugang Duan <B38611@freescale.com> > Date: Thu, 9 Aug 2012 17:59:44 +0800 > Subject: [PATCH] ENGR00180288 - FEC : Fix kernel dump about eth0 Thanks, I'll give a try with this patch as well. Regards, Vikram ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-01-29 18:46 ` Troy Kisky 2013-01-30 8:37 ` Duan Fugang-B38611 @ 2013-01-30 15:13 ` Vikram Narayanan 2013-01-30 21:04 ` Troy Kisky 1 sibling, 1 reply; 13+ messages in thread From: Vikram Narayanan @ 2013-01-30 15:13 UTC (permalink / raw) To: Troy Kisky; +Cc: netdev, LAK, Fabio Estevam On 1/30/2013 12:16 AM, Troy Kisky wrote: > I have fixed this bug (and more) on Freescale's kernel > (imx-3.0.35_1.1.0). I created a branch you can try. > Feel free to port to mainline. Thanks a lot for the branch. I saw a lot of differences between your driver and the mainline. I should spend some time If I had to port this to mainline. > This is the patch that should fix your problem > fec: clear TX_FULL in fec_restart > > > Please let me know results. When NAPI is disabled I saw a severe packet loss (around 40% in a flood ping) which spitted out this, [ 470.390928] net eth0: missed rxf 2000000 1c000000 0 [ 470.419098] net eth0: missed rxf 2000000 1c000000 0 [ 470.443800] net eth0: missed rxf 2000000 1c000000 0 [ 470.450315] net eth0: missed rxf 2000000 1c000000 0 When NAPI is enabled, it was working good. Though it resulted in the same error, it recovered by itself, which is perfectly fine for me. :) [ 507.359857] ------------[ cut here ]------------ [ 507.365841] WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x298/0x2c0() [ 507.375655] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out [ 507.384521] Modules linked in: [ 507.388907] [<80033e6c>] (unwind_backtrace+0x0/0xf8) from [<800558d0>] (warn_slowpath_common+0x4c/0x64) [ 507.401045] [<800558d0>] (warn_slowpath_common+0x4c/0x64) from [<8005597c>] (warn_slowpath_fmt+0x30/0x40) [ 507.413387] [<8005597c>] (warn_slowpath_fmt+0x30/0x40) from [<802db42c>] (dev_watchdog+0x298/0x2c0) [ 507.425223] [<802db42c>] (dev_watchdog+0x298/0x2c0) from [<80060ea4>] (run_timer_softirq+0xf4/0x23c) [ 507.437170] [<80060ea4>] (run_timer_softirq+0xf4/0x23c) from [<8005aea0>] (__do_softirq+0x98/0x130) [ 507.449063] [<8005aea0>] (__do_softirq+0x98/0x130) from [<8005b354>] (irq_exit+0x8c/0x94) [ 507.460155] [<8005b354>] (irq_exit+0x8c/0x94) from [<8002f3f0>] (handle_IRQ+0x34/0x84) [ 507.471019] [<8002f3f0>] (handle_IRQ+0x34/0x84) from [<8002e70c>] (__irq_svc+0x4c/0xa4) [ 507.482010] [<8002e70c>] (__irq_svc+0x4c/0xa4) from [<8002f500>] (default_idle+0x24/0x28) [ 507.493222] [<8002f500>] (default_idle+0x24/0x28) from [<8002f6a8>] (cpu_idle+0x84/0xb8) [ 507.504384] [<8002f6a8>] (cpu_idle+0x84/0xb8) from [<80008824>] (start_kernel+0x22c/0x26c) [ 507.515724] [<80008824>] (start_kernel+0x22c/0x26c) from [<1000803c>] (0x1000803c) [ 507.526419] ---[ end trace 1b75b31a2719ed1e ]--- [ 507.532613] net eth0: fec_timeout tx_full=1 tx_insert=5a tx_remove=5a 9c00 [ 507.542567] net eth0: 12000000 8c00000, 12000000 [ 507.548754] net eth0: calling restart [ 509.250736] PHY: 1:00 - Link is Up - 100/Full Thank you so much for your time and help. Regards, Vikram ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-01-30 15:13 ` Vikram Narayanan @ 2013-01-30 21:04 ` Troy Kisky 2013-02-18 7:29 ` Frank Li 0 siblings, 1 reply; 13+ messages in thread From: Troy Kisky @ 2013-01-30 21:04 UTC (permalink / raw) To: Vikram Narayanan; +Cc: netdev, LAK, Fabio Estevam On 1/30/2013 8:13 AM, Vikram Narayanan wrote: > On 1/30/2013 12:16 AM, Troy Kisky wrote: >> I have fixed this bug (and more) on Freescale's kernel >> (imx-3.0.35_1.1.0). I created a branch you can try. >> Feel free to port to mainline. > Thanks a lot for the branch. I saw a lot of differences between > your driver and the mainline. I should spend some time If I had > to port this to mainline. > >> This is the patch that should fix your problem >> fec: clear TX_FULL in fec_restart >> >> >> Please let me know results. > When NAPI is disabled I saw a severe packet loss > (around 40% in a flood ping) which spitted out this, > > [ 470.390928] net eth0: missed rxf 2000000 1c000000 0 > [ 470.419098] net eth0: missed rxf 2000000 1c000000 0 > [ 470.443800] net eth0: missed rxf 2000000 1c000000 0 > [ 470.450315] net eth0: missed rxf 2000000 1c000000 0 > > When NAPI is enabled, it was working good. Though it > resulted in the same error, it recovered by itself, > which is perfectly fine for me. :) > You'll also have better performance if you pass "enable_wait_mode=off" in bootargs. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-01-30 21:04 ` Troy Kisky @ 2013-02-18 7:29 ` Frank Li 2013-02-21 17:20 ` Vikram Narayanan 0 siblings, 1 reply; 13+ messages in thread From: Frank Li @ 2013-02-18 7:29 UTC (permalink / raw) To: Troy Kisky; +Cc: Vikram Narayanan, netdev, LAK, Fabio Estevam Do you have a way to duplicate this issue consistently? best regards Frank Li ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: i.Mx6Quad - eth0: tx queue full! 2013-02-18 7:29 ` Frank Li @ 2013-02-21 17:20 ` Vikram Narayanan 0 siblings, 0 replies; 13+ messages in thread From: Vikram Narayanan @ 2013-02-21 17:20 UTC (permalink / raw) To: Frank Li; +Cc: Troy Kisky, netdev, Fabio Estevam On 2/18/2013 12:59 PM, Frank Li wrote: > Do you have a way to duplicate this issue consistently? I think you meant "reproduce". Yes. Almost everytime in my hardware. Any network load would trigger this. On the NFS, I did something like for i in {1..10000};do;ls -lR *;done and did a flood ping to the target as well. Hope this helps, Vikram ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2013-02-21 17:21 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-01-28 17:39 i.Mx6Quad - eth0: tx queue full! Vikram Narayanan 2013-01-28 19:47 ` Troy Kisky 2013-01-29 16:33 ` Vikram Narayanan 2013-01-29 16:45 ` Fabio Estevam 2013-01-29 16:55 ` Vikram Narayanan 2013-01-29 16:34 ` Vikram Narayanan 2013-01-29 18:46 ` Troy Kisky 2013-01-30 8:37 ` Duan Fugang-B38611 2013-01-30 15:18 ` Vikram Narayanan 2013-01-30 15:13 ` Vikram Narayanan 2013-01-30 21:04 ` Troy Kisky 2013-02-18 7:29 ` Frank Li 2013-02-21 17:20 ` Vikram Narayanan
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).