i.Mx6Quad - eth0: tx queue full!

i.Mx6Quad - eth0: tx queue full!

[Vikram Narayanan]

Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform
gives me the below error when flooded with ping requests.

== Start log ==
[ 2555.004031] ------------[ cut here ]------------
[ 2555.009740] WARNING: at net/sched/sch_generic.c:254 dev_watchdog+0x298/0x2b8()
[ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
[ 2555.026733] Modules linked in:
[ 2555.030598] Backtrace:
[ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from [<803b8494>] (dump_stack+0x18/0x1c)
[ 2555.044438]  r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80
[ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>] (warn_slowpath_common+0x54/0x6c)
[ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from [<8001dfc4>] (warn_slowpath_fmt+0x38/0x40)
[ 2555.073936]  r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4 r4:8f977000
r3:00000009
[ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from [<80302f64>] (dev_watchdog+0x298/0x2b8)
[ 2555.095535]  r3:8f977000 r2:8049f6d4
[ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>] (call_timer_fn.isra.33+0x2c/0x8c)
[ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from [<8002af48>] (run_timer_softirq+0x1f0/0x204)
[ 2555.122240]  r7:80571114 r6:805040c0 r5:00000000 r4:80570900
[ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from [<80025750>] (__do_softirq+0xc8/0x180)
[ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>] (irq_exit+0x88/0x90)
[ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>] (handle_IRQ+0x44/0x98)
[ 2555.160112]  r4:804ffde0 r3:00000220
[ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>] (gic_handle_irq+0x30/0x64)
[ 2555.174956]  r6:80503f28 r5:8050a518 r4:f400010c r3:00000000
[ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from [<8000df80>] (__irq_svc+0x40/0x50)
[ 2555.192737] Exception stack(0x80503f28 to 0x80503f70)
[ 2555.198639] 3f20:                   8052f150 a0000093 00000000 00000000 80502000 8052ed08
[ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000 80503f7c 80503f80 80503f70
[ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff
[ 2555.224730]  r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8
[ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>] (cpu_idle+0xcc/0x114)
[ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>] (rest_init+0x64/0x7c)
[ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>] (start_kernel+0x258/0x298)
[ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>] (0x10008078)
[ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]---
[ 2555.277270] eth0: tx queue full!.
[ 2555.288776] eth0: tx queue full!.
[ 2555.293594] eth0: tx queue full!.
[ 2555.297944] eth0: tx queue full!.
[ 2555.302229] eth0: tx queue full!.

...... continues indefinitely and needs a reboot to recover the system.
== End log ==

i.Mx6Quad's MAC is connected to SMSC LAN88730 PHY via an MII interface.

I found a similar thread [1], but that solution didn't work for me.
Any ideas on why the fec driver is unhappy?

Thanks,
Vikram

[1] https://community.freescale.com/thread/281457

Re: i.Mx6Quad - eth0: tx queue full!

[Troy Kisky]

On 1/28/2013 10:39 AM, Vikram Narayanan wrote:
> Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform
> gives me the below error when flooded with ping requests.
>
> == Start log ==
> [ 2555.004031] ------------[ cut here ]------------
> [ 2555.009740] WARNING: at net/sched/sch_generic.c:254 dev_watchdog+0x298/0x2b8()
> [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out

I think the tx interrupt status bit was lost. The packets were 
transmitted, but the interrupt never
happened. The controller should have been reset here, but perhaps a bug 
with the reset code.
Are you using the mainline kernel, or a version Freescale's kernel.

mainline fec_restart does not reset tx_full

You can try adding
fep->tx_full = 0;

to fec_restart, though it would be better to call fec_enet_tx in fec_timeout
and skip the call to fec_restart if it returns some packets.

> [ 2555.026733] Modules linked in:
> [ 2555.030598] Backtrace:
> [ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from [<803b8494>] (dump_stack+0x18/0x1c)
> [ 2555.044438]  r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80
> [ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>] (warn_slowpath_common+0x54/0x6c)
> [ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from [<8001dfc4>] (warn_slowpath_fmt+0x38/0x40)
> [ 2555.073936]  r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4 r4:8f977000
> r3:00000009
> [ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from [<80302f64>] (dev_watchdog+0x298/0x2b8)
> [ 2555.095535]  r3:8f977000 r2:8049f6d4
> [ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>] (call_timer_fn.isra.33+0x2c/0x8c)
> [ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from [<8002af48>] (run_timer_softirq+0x1f0/0x204)
> [ 2555.122240]  r7:80571114 r6:805040c0 r5:00000000 r4:80570900
> [ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from [<80025750>] (__do_softirq+0xc8/0x180)
> [ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>] (irq_exit+0x88/0x90)
> [ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>] (handle_IRQ+0x44/0x98)
> [ 2555.160112]  r4:804ffde0 r3:00000220
> [ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>] (gic_handle_irq+0x30/0x64)
> [ 2555.174956]  r6:80503f28 r5:8050a518 r4:f400010c r3:00000000
> [ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from [<8000df80>] (__irq_svc+0x40/0x50)
> [ 2555.192737] Exception stack(0x80503f28 to 0x80503f70)
> [ 2555.198639] 3f20:                   8052f150 a0000093 00000000 00000000 80502000 8052ed08
> [ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000 80503f7c 80503f80 80503f70
> [ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff
> [ 2555.224730]  r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8
> [ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>] (cpu_idle+0xcc/0x114)
> [ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>] (rest_init+0x64/0x7c)
> [ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>] (start_kernel+0x258/0x298)
> [ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>] (0x10008078)
> [ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]---
> [ 2555.277270] eth0: tx queue full!.
> [ 2555.288776] eth0: tx queue full!.
> [ 2555.293594] eth0: tx queue full!.
> [ 2555.297944] eth0: tx queue full!.
> [ 2555.302229] eth0: tx queue full!.
All the packet have been transmitted but the transmit queue is full, so 
no more
tx interrupts can happen to replace the previously lost tx interrupt.

I've seen MII interrupts clear the TX interrupt status bit.

> ...... continues indefinitely and needs a reboot to recover the system.
> == End log ==
>
> i.Mx6Quad's MAC is connected to SMSC LAN88730 PHY via an MII interface.
>
> I found a similar thread [1], but that solution didn't work for me.
> Any ideas on why the fec driver is unhappy?
>
> Thanks,
> Vikram
>
> [1] https://community.freescale.com/thread/281457
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>

Re: i.Mx6Quad - eth0: tx queue full!

[Vikram Narayanan]

On 1/29/2013 1:17 AM, Troy Kisky wrote:
> On 1/28/2013 10:39 AM, Vikram Narayanan wrote:
>> Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform
>> gives me the below error when flooded with ping requests.
>>
>> == Start log ==
>> [ 2555.004031] ------------[ cut here ]------------
>> [ 2555.009740] WARNING: at net/sched/sch_generic.c:254
>> dev_watchdog+0x298/0x2b8()
>> [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
>
> I think the tx interrupt status bit was lost. The packets were
> transmitted, but the interrupt never
> happened. The controller should have been reset here, but perhaps a bug
> with the reset code.
> Are you using the mainline kernel, or a version Freescale's kernel.

I tried with both the kernels. Freescale's and mainline results in the 
same error.

> mainline fec_restart does not reset tx_full
>
> You can try adding
> fep->tx_full = 0;

With this there was no improvement.

> to fec_restart, though it would be better to call fec_enet_tx in
> fec_timeout
> and skip the call to fec_restart if it returns some packets.

fec_enet_tx returns void, how do I check for the return packets.
I'm sorry, I couldn't get your point here.

>
>> [ 2555.026733] Modules linked in:
>> [ 2555.030598] Backtrace:
>> [ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from
>> [<803b8494>] (dump_stack+0x18/0x1c)
>> [ 2555.044438]  r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80
>> [ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>]
>> (warn_slowpath_common+0x54/0x6c)
>> [ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from
>> [<8001dfc4>] (warn_slowpath_fmt+0x38/0x40)
>> [ 2555.073936]  r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4
>> r4:8f977000
>> r3:00000009
>> [ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from
>> [<80302f64>] (dev_watchdog+0x298/0x2b8)
>> [ 2555.095535]  r3:8f977000 r2:8049f6d4
>> [ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>]
>> (call_timer_fn.isra.33+0x2c/0x8c)
>> [ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from
>> [<8002af48>] (run_timer_softirq+0x1f0/0x204)
>> [ 2555.122240]  r7:80571114 r6:805040c0 r5:00000000 r4:80570900
>> [ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from
>> [<80025750>] (__do_softirq+0xc8/0x180)
>> [ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>]
>> (irq_exit+0x88/0x90)
>> [ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>]
>> (handle_IRQ+0x44/0x98)
>> [ 2555.160112]  r4:804ffde0 r3:00000220
>> [ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>]
>> (gic_handle_irq+0x30/0x64)
>> [ 2555.174956]  r6:80503f28 r5:8050a518 r4:f400010c r3:00000000
>> [ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from
>> [<8000df80>] (__irq_svc+0x40/0x50)
>> [ 2555.192737] Exception stack(0x80503f28 to 0x80503f70)
>> [ 2555.198639] 3f20:                   8052f150 a0000093 00000000
>> 00000000 80502000 8052ed08
>> [ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000
>> 80503f7c 80503f80 80503f70
>> [ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff
>> [ 2555.224730]  r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8
>> [ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>]
>> (cpu_idle+0xcc/0x114)
>> [ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>]
>> (rest_init+0x64/0x7c)
>> [ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>]
>> (start_kernel+0x258/0x298)
>> [ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>]
>> (0x10008078)
>> [ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]---
>> [ 2555.277270] eth0: tx queue full!.
>> [ 2555.288776] eth0: tx queue full!.
>> [ 2555.293594] eth0: tx queue full!.
>> [ 2555.297944] eth0: tx queue full!.
>> [ 2555.302229] eth0: tx queue full!.
> All the packet have been transmitted but the transmit queue is full, so
> no more
> tx interrupts can happen to replace the previously lost tx interrupt.
>
> I've seen MII interrupts clear the TX interrupt status bit.
>

Do you suspect the MAC <-> PHY interconnect? I'm having an MII interface.

Regards,
Vikram

Re: i.Mx6Quad - eth0: tx queue full!

[Vikram Narayanan]

On 1/29/2013 1:17 AM, Troy Kisky wrote:
> On 1/28/2013 10:39 AM, Vikram Narayanan wrote:
>> Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform
>> gives me the below error when flooded with ping requests.
>>
>> == Start log ==
>> [ 2555.004031] ------------[ cut here ]------------
>> [ 2555.009740] WARNING: at net/sched/sch_generic.c:254
>> dev_watchdog+0x298/0x2b8()
>> [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
>
> I think the tx interrupt status bit was lost. The packets were
> transmitted, but the interrupt never
> happened. The controller should have been reset here, but perhaps a bug
> with the reset code.
> Are you using the mainline kernel, or a version Freescale's kernel.

I tried with both the kernels. Freescale's and mainline results in the 
same error.

> mainline fec_restart does not reset tx_full
>
> You can try adding
> fep->tx_full = 0;

With this there was no improvement.

> to fec_restart, though it would be better to call fec_enet_tx in
> fec_timeout
> and skip the call to fec_restart if it returns some packets.

fec_enet_tx returns void, how do I check for the return packets.
I'm sorry, I couldn't get your point here.

>
>> [ 2555.026733] Modules linked in:
>> [ 2555.030598] Backtrace:
>> [ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from
>> [<803b8494>] (dump_stack+0x18/0x1c)
>> [ 2555.044438]  r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80
>> [ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>]
>> (warn_slowpath_common+0x54/0x6c)
>> [ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from
>> [<8001dfc4>] (warn_slowpath_fmt+0x38/0x40)
>> [ 2555.073936]  r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4
>> r4:8f977000
>> r3:00000009
>> [ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from
>> [<80302f64>] (dev_watchdog+0x298/0x2b8)
>> [ 2555.095535]  r3:8f977000 r2:8049f6d4
>> [ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>]
>> (call_timer_fn.isra.33+0x2c/0x8c)
>> [ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from
>> [<8002af48>] (run_timer_softirq+0x1f0/0x204)
>> [ 2555.122240]  r7:80571114 r6:805040c0 r5:00000000 r4:80570900
>> [ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from
>> [<80025750>] (__do_softirq+0xc8/0x180)
>> [ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>]
>> (irq_exit+0x88/0x90)
>> [ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>]
>> (handle_IRQ+0x44/0x98)
>> [ 2555.160112]  r4:804ffde0 r3:00000220
>> [ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>]
>> (gic_handle_irq+0x30/0x64)
>> [ 2555.174956]  r6:80503f28 r5:8050a518 r4:f400010c r3:00000000
>> [ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from
>> [<8000df80>] (__irq_svc+0x40/0x50)
>> [ 2555.192737] Exception stack(0x80503f28 to 0x80503f70)
>> [ 2555.198639] 3f20:                   8052f150 a0000093 00000000
>> 00000000 80502000 8052ed08
>> [ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000
>> 80503f7c 80503f80 80503f70
>> [ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff
>> [ 2555.224730]  r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8
>> [ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>]
>> (cpu_idle+0xcc/0x114)
>> [ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>]
>> (rest_init+0x64/0x7c)
>> [ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>]
>> (start_kernel+0x258/0x298)
>> [ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>]
>> (0x10008078)
>> [ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]---
>> [ 2555.277270] eth0: tx queue full!.
>> [ 2555.288776] eth0: tx queue full!.
>> [ 2555.293594] eth0: tx queue full!.
>> [ 2555.297944] eth0: tx queue full!.
>> [ 2555.302229] eth0: tx queue full!.
> All the packet have been transmitted but the transmit queue is full, so
> no more
> tx interrupts can happen to replace the previously lost tx interrupt.
>
> I've seen MII interrupts clear the TX interrupt status bit.
>

Do you suspect the MAC <-> PHY interconnect? I'm having an MII interface.

Regards,
Vikram

Re: i.Mx6Quad - eth0: tx queue full!

[Fabio Estevam]

On Tue, Jan 29, 2013 at 2:33 PM, Vikram Narayanan <vikram186@gmail.com> wrote:

> I tried with both the kernels. Freescale's and mainline results in the same
> error.

Do you also see this issue with a 3.6 or 3.7 stable kernels?

Re: i.Mx6Quad - eth0: tx queue full!

[Vikram Narayanan]

On 1/29/2013 10:15 PM, Fabio Estevam wrote:
> On Tue, Jan 29, 2013 at 2:33 PM, Vikram Narayanan <vikram186@gmail.com> wrote:
>
>> I tried with both the kernels. Freescale's and mainline results in the same
>> error.
>
> Do you also see this issue with a 3.6 or 3.7 stable kernels?
>

I didn't try with the 3.6 or 3.7 stable.

Re: i.Mx6Quad - eth0: tx queue full!

[Troy Kisky]

On 1/29/2013 9:34 AM, Vikram Narayanan wrote:
> On 1/29/2013 1:17 AM, Troy Kisky wrote:
>> On 1/28/2013 10:39 AM, Vikram Narayanan wrote:
>>> Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform
>>> gives me the below error when flooded with ping requests.
>>>
>>> == Start log ==
>>> [ 2555.004031] ------------[ cut here ]------------
>>> [ 2555.009740] WARNING: at net/sched/sch_generic.c:254
>>> dev_watchdog+0x298/0x2b8()
>>> [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
>>
>> I think the tx interrupt status bit was lost. The packets were
>> transmitted, but the interrupt never
>> happened. The controller should have been reset here, but perhaps a bug
>> with the reset code.
>> Are you using the mainline kernel, or a version Freescale's kernel.
>
> I tried with both the kernels. Freescale's and mainline results in the 
> same error.
>
>> mainline fec_restart does not reset tx_full
>>
>> You can try adding
>> fep->tx_full = 0;
>
> With this there was no improvement.

I have fixed this bug (and more) on Freescale's kernel 
(imx-3.0.35_1.1.0). I created a branch you can try.
Feel free to port to mainline.

This is the patch that should fix your problem
fec: clear TX_FULL in fec_restart

git://github.com/boundarydevices/linux-imx6.git ethernet_test



Please let me know results.
Thanks
Troy

RE: i.Mx6Quad - eth0: tx queue full!

[Duan Fugang-B38611]

Hi, all

The issue cannot be found kernel 3.0.35 branch: (even run stress test with IPU and VPU)
ssh://sw-git01-tx30/git/sw_git/repos/linux-2.6-imx.git  
branch name: imx_3.0.35


The patch as below:


From 91a0c892263e57ecde9e9ff38be3acdb7f66a17f Mon Sep 17 00:00:00 2001
From: Fugang Duan <B38611@freescale.com>
Date: Thu, 9 Aug 2012 17:59:44 +0800
Subject: [PATCH] ENGR00180288 - FEC : Fix kernel dump about eth0

Kernel dump when do wifi stress test with suspend and resume as below:
        eth0: tx queue full!.
        remove wake up source irq 103 
        PM: resume of devices complete after 348.934 msecs
        Restarting tasks ... done.
        ------------[ cut here ]------------
        WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x284/0x2a8()
        NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out 
        Modules linked in: ar6000
        [<8004482c>] (unwind_backtrace+0x0/0xf8) from
        [<80068cd0>] (warn_slowpath_common+0x4c/0x64)
        [<80068cd0>] (warn_slowpath_common+0x4c/0x64)from
        [<80068d7c>] (warn_slowpath_fmt+0x30/0x40)
        [<80068d7c>] (warn_slowpath_fmt+0x30/0x40) from
        [<803f0c50>] (dev_watchdog+0x284/0x2a8)
        [<803f0c50>] (dev_watchdog+0x284/0x2a8) from
        [<80074430>] (run_timer_softirq+0xec/0x214)
        [<80074430>] (run_timer_softirq+0xec/0x214) from
        [<8006e524>] (__do_softirq+0xac/0x140)
        [<8006e524>] (__do_softirq+0xac/0x140) from
        [<8006ea60>] (irq_exit+0x94/0x9c)
        [<8006ea60>] (irq_exit+0x94/0x9c) from
        [<80039240>] (do_local_timer+0x54/0x70)
        [<80039240>] (do_local_timer+0x54/0x70) from
        [<8003ea0c>] (__irq_svc+0x4c/0xe8)
        Exception stack(0x80a2bf68 to 0x80a2bfb0)
        bf60:                   0000001f 80a3babc 80a2bfb0 00000000 80a2a000 80a7b8e4
        bf80: 804befcc 80a3ee7c 1000406a 412fc09a 00000000 00000000 80a81440 80a2bfb0
        bfa0: 8003fa64 8003fa68 60000013 ffffffff
        [<8003ea0c>] (__irq_svc+0x4c/0xe8) from [<8003fa68>] (default_idle+0x24/0x28)
        [<8003fa68>] (default_idle+0x24/0x28) from [<8003fc60>] (cpu_idle+0xbc/0xfc)
        [<8003fc60>] (cpu_idle+0xbc/0xfc) from [<80008878>] (start_kernel+0x258/0x29c)
        [<80008878>] (start_kernel+0x258/0x29c) from [<10008040>] (0x10008040)
        ---[ end trace 30671ac42e272c2d ]---

But ethernet and system still be alive. In sometime,the issue
will cause system hang like "nfs: server 10.192.242.179 not 
responding, still trying".

The root cause is tx buffer descriptors are not cleaned when
ethernet resume back.

Signed-off-by: Fugang Duan <B38611@freescale.com>

drivers/net/fec.c |   39 ++++++++++++++++++++++++++-------------
 1 files changed, 26 insertions(+), 13 deletions(-)

diff --git a/drivers/net/fec.c b/drivers/net/fec.c
index f007bf0..b1fa464 100755
--- a/drivers/net/fec.c
+++ b/drivers/net/fec.c
@@ -1456,6 +1456,28 @@ static const struct net_device_ops fec_netdev_ops = {
 #endif
 };
 
+/* Init TX buffer descriptors
+ */
+static void fec_enet_txbd_init(struct net_device *dev)
+{
+       struct fec_enet_private *fep = netdev_priv(dev);
+       struct bufdesc *bdp;
+       int i;
+
+       /* ...and the same for transmit */
+       bdp = fep->tx_bd_base;
+       for (i = 0; i < TX_RING_SIZE; i++) {
+
+               /* Initialize the BD for every fragment in the page. */
+               bdp->cbd_sc = 0;
+               bdp++;
+       }
+
+       /* Set the last buffer to wrap */
+       bdp--;
+       bdp->cbd_sc |= BD_SC_WRAP;
+}
+
  /*
   * XXX:  We need to clean up on failure exits here.
   *
@@ -1512,19 +1534,8 @@ static int fec_enet_init(struct net_device *ndev)
        bdp--;
        bdp->cbd_sc |= BD_SC_WRAP;
 
-       /* ...and the same for transmit */
-       bdp = fep->tx_bd_base;
-       for (i = 0; i < TX_RING_SIZE; i++) {
-
-               /* Initialize the BD for every fragment in the page. */
-               bdp->cbd_sc = 0;
-               bdp->cbd_bufaddr = 0;
-               bdp++;
-       }
-
-       /* Set the last buffer to wrap */
-       bdp--;
-       bdp->cbd_sc |= BD_SC_WRAP;
+       /* Init transmit descriptors */
+       fec_enet_txbd_init(ndev);
 
        fec_restart(ndev, 0);
 
@@ -1575,6 +1586,8 @@ fec_restart(struct net_device *dev, int duplex)
        writel(fep->bd_dma, fep->hwp + FEC_R_DES_START);
        writel((unsigned long)fep->bd_dma + sizeof(struct bufdesc) * RX_RING_SIZE,
                        fep->hwp + FEC_X_DES_START);
+       /* Reinit transmit descriptors */
+       fec_enet_txbd_init(dev);
 
        fep->dirty_tx = fep->cur_tx = fep->tx_bd_base;
        fep->cur_rx = fep->rx_bd_base;
-- 
1.7.0.4

Thanks,
Andy




-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Troy Kisky
Sent: Wednesday, January 30, 2013 2:47 AM
To: Vikram Narayanan
Cc: netdev@vger.kernel.org; Greg Ungerer; shawn.guo@linaro.org; LAK; Uwe Kleine-König; Fabio Estevam
Subject: Re: i.Mx6Quad - eth0: tx queue full!

On 1/29/2013 9:34 AM, Vikram Narayanan wrote:
> On 1/29/2013 1:17 AM, Troy Kisky wrote:
>> On 1/28/2013 10:39 AM, Vikram Narayanan wrote:
>>> Running the latest head <linux-2.6.git> on an i.Mx6Quad based 
>>> platform gives me the below error when flooded with ping requests.
>>>
>>> == Start log ==
>>> [ 2555.004031] ------------[ cut here ]------------ [ 2555.009740] 
>>> WARNING: at net/sched/sch_generic.c:254
>>> dev_watchdog+0x298/0x2b8()
>>> [ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed 
>>> out
>>
>> I think the tx interrupt status bit was lost. The packets were 
>> transmitted, but the interrupt never happened. The controller should 
>> have been reset here, but perhaps a bug with the reset code.
>> Are you using the mainline kernel, or a version Freescale's kernel.
>
> I tried with both the kernels. Freescale's and mainline results in the 
> same error.
>
>> mainline fec_restart does not reset tx_full
>>
>> You can try adding
>> fep->tx_full = 0;
>
> With this there was no improvement.

I have fixed this bug (and more) on Freescale's kernel (imx-3.0.35_1.1.0). I created a branch you can try.
Feel free to port to mainline.

This is the patch that should fix your problem
fec: clear TX_FULL in fec_restart

git://github.com/boundarydevices/linux-imx6.git ethernet_test



Please let me know results.
Thanks
Troy


--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: i.Mx6Quad - eth0: tx queue full!

[Vikram Narayanan]

On 1/30/2013 12:16 AM, Troy Kisky wrote:
> I have fixed this bug (and more) on Freescale's kernel 
> (imx-3.0.35_1.1.0). I created a branch you can try.
> Feel free to port to mainline.

Thanks a lot for the branch. I saw a lot of differences between
your driver and the mainline. I should spend some time If I had
to port this to mainline.
 
> This is the patch that should fix your problem
> fec: clear TX_FULL in fec_restart
> 
> 
> Please let me know results.

When NAPI is disabled I saw a severe packet loss 
(around 40% in a flood ping) which spitted out this,

[  470.390928] net eth0: missed rxf 2000000 1c000000 0
[  470.419098] net eth0: missed rxf 2000000 1c000000 0
[  470.443800] net eth0: missed rxf 2000000 1c000000 0
[  470.450315] net eth0: missed rxf 2000000 1c000000 0

When NAPI is enabled, it was working good. Though it
resulted in the same error, it recovered by itself,
which is perfectly fine for me. :)

[  507.359857] ------------[ cut here ]------------
[  507.365841] WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x298/0x2c0()
[  507.375655] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
[  507.384521] Modules linked in:
[  507.388907] [<80033e6c>] (unwind_backtrace+0x0/0xf8) from [<800558d0>] (warn_slowpath_common+0x4c/0x64)
[  507.401045] [<800558d0>] (warn_slowpath_common+0x4c/0x64) from [<8005597c>] (warn_slowpath_fmt+0x30/0x40)
[  507.413387] [<8005597c>] (warn_slowpath_fmt+0x30/0x40) from [<802db42c>] (dev_watchdog+0x298/0x2c0)
[  507.425223] [<802db42c>] (dev_watchdog+0x298/0x2c0) from [<80060ea4>] (run_timer_softirq+0xf4/0x23c)
[  507.437170] [<80060ea4>] (run_timer_softirq+0xf4/0x23c) from [<8005aea0>] (__do_softirq+0x98/0x130)
[  507.449063] [<8005aea0>] (__do_softirq+0x98/0x130) from [<8005b354>] (irq_exit+0x8c/0x94)
[  507.460155] [<8005b354>] (irq_exit+0x8c/0x94) from [<8002f3f0>] (handle_IRQ+0x34/0x84)
[  507.471019] [<8002f3f0>] (handle_IRQ+0x34/0x84) from [<8002e70c>] (__irq_svc+0x4c/0xa4)
[  507.482010] [<8002e70c>] (__irq_svc+0x4c/0xa4) from [<8002f500>] (default_idle+0x24/0x28)
[  507.493222] [<8002f500>] (default_idle+0x24/0x28) from [<8002f6a8>] (cpu_idle+0x84/0xb8)
[  507.504384] [<8002f6a8>] (cpu_idle+0x84/0xb8) from [<80008824>] (start_kernel+0x22c/0x26c)
[  507.515724] [<80008824>] (start_kernel+0x22c/0x26c) from [<1000803c>] (0x1000803c)
[  507.526419] ---[ end trace 1b75b31a2719ed1e ]---
[  507.532613] net eth0: fec_timeout tx_full=1 tx_insert=5a tx_remove=5a 9c00
[  507.542567] net eth0: 12000000 8c00000, 12000000
[  507.548754] net eth0: calling restart
[  509.250736] PHY: 1:00 - Link is Up - 100/Full

Thank you so much for your time and help.

Regards,
Vikram

Re: i.Mx6Quad - eth0: tx queue full!

[Vikram Narayanan]

Hello Duan,

On 1/30/2013 2:07 PM, Duan Fugang-B38611 wrote:
> Hi, all
>
> The issue cannot be found kernel 3.0.35 branch: (even run stress test with IPU and VPU)
> ssh://sw-git01-tx30/git/sw_git/repos/linux-2.6-imx.git

Where is this git located?

> branch name: imx_3.0.35

Troy's branch (located in github) works fine for me. I've kept it for a 
long run with heavy network load. Should I encounter some errors, I'll 
post back.

> The patch as below:
>
>  From 91a0c892263e57ecde9e9ff38be3acdb7f66a17f Mon Sep 17 00:00:00 2001
> From: Fugang Duan <B38611@freescale.com>
> Date: Thu, 9 Aug 2012 17:59:44 +0800
> Subject: [PATCH] ENGR00180288 - FEC : Fix kernel dump about eth0

Thanks, I'll give a try with this patch as well.

Regards,
Vikram

Re: i.Mx6Quad - eth0: tx queue full!

[Troy Kisky]

On 1/30/2013 8:13 AM, Vikram Narayanan wrote:
> On 1/30/2013 12:16 AM, Troy Kisky wrote:
>> I have fixed this bug (and more) on Freescale's kernel
>> (imx-3.0.35_1.1.0). I created a branch you can try.
>> Feel free to port to mainline.
> Thanks a lot for the branch. I saw a lot of differences between
> your driver and the mainline. I should spend some time If I had
> to port this to mainline.
>   
>> This is the patch that should fix your problem
>> fec: clear TX_FULL in fec_restart
>>
>>
>> Please let me know results.
> When NAPI is disabled I saw a severe packet loss
> (around 40% in a flood ping) which spitted out this,
>
> [  470.390928] net eth0: missed rxf 2000000 1c000000 0
> [  470.419098] net eth0: missed rxf 2000000 1c000000 0
> [  470.443800] net eth0: missed rxf 2000000 1c000000 0
> [  470.450315] net eth0: missed rxf 2000000 1c000000 0
>
> When NAPI is enabled, it was working good. Though it
> resulted in the same error, it recovered by itself,
> which is perfectly fine for me. :)
>

You'll also have better performance if you pass "enable_wait_mode=off" 
in bootargs.

Re: i.Mx6Quad - eth0: tx queue full!

[Frank Li]

Do you have a way to duplicate this issue consistently?

best regards
Frank Li

Re: i.Mx6Quad - eth0: tx queue full!

[Vikram Narayanan]

On 2/18/2013 12:59 PM, Frank Li wrote:
> Do you have a way to duplicate this issue consistently?

I think you meant "reproduce".

Yes. Almost everytime in my hardware.
Any network load would trigger this.

On the NFS, I did something like
for i in {1..10000};do;ls -lR *;done

and did a flood ping to the target as well.

Hope this helps,
Vikram