Problems in the DaVinci EMAC driver & AM35xx?

CF Adad
2012-06-13T22:38:54Z

All,

I believe I may have uncovered a problem in the DaVinci EMAC driver as it is used on the AM35xx. I suspect this is likely related to the slab crashing I reported here (http://thread.gmane.org/gmane.linux.ports.arm.omap/78039/). Your help and thoughts on this would be greatly appreciated.
and
*A QUICK BACKSTORY:*

We've been working with AM3517 modules (CM-T3517 and TAM3517) for about 6 months now, and in that time have tried many different versions of OS software. We started with the 2.6.37 kernel supported by TI's PSP and quickly migrated through 3.2, 3.3, 3.4, and as of yeterday, 3.5rc1. In that same time, we've tried several versions of both bootloaders as well. One consistent factor we've noticed is a relatively odd DaVinci EMAC performance issue. Using just basic iperf ("iperf -s" on one side, "iperf -c <IP>" on the other), we noted that whenever we used our DaVinci EMAC port on the AM3517, we saw poor performance when connecting AM3517 EMAC <=> AM3517 EMAC, but great performance when connecting the EMAC to another, non-EMAC Ethernet device. Here is a real example capture:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
{{ DaVinci EMAC server, PC client }}

[  4] local 192.168.2.192 port 5001 connected with 192.168.2.40 port 44452
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.1 sec    114 MBytes  94.1 Mbits/sec
[  5] local 192.168.2.192 port 5001 connected with 192.168.2.40 port 44459
[  5]  0.0-10.1 sec    114 MBytes  94.0 Mbits/sec
[  4] local 192.168.2.192 port 5001 connected with 192.168.2.40 port 44467
[  4]  0.0-10.1 sec    113 MBytes  94.1 Mbits/sec
[  5] local 192.168.2.192 port 5001 connected with 192.168.2.40 port 44474
[  5]  0.0-10.1 sec    114 MBytes  94.1 Mbits/sec
[  4] local 192.168.2.192 port 5001 connected with 192.168.2.40 port 44481
[  4]  0.0-10.1 sec    114 MBytes  94.1 Mbits/sec
[  5] local 192.168.2.192 port 5001 connected with 192.168.2.40 port 44488
[  5]  0.0- 9.8 sec    110 MBytes  94.1 Mbits/sec

{{ same DaVinci EMAC server, still running, just changed to DaVinci EMAC client }}

[  5] local 192.168.2.192 port 5001 connected with 192.168.2.74 port 37325
[  5]  0.0-10.4 sec  73.9 MBytes  59.6 Mbits/sec
[  6] local 192.168.2.192 port 5001 connected with 192.168.2.74 port 37326
[  6]  0.0-10.8 sec  61.5 MBytes  47.8 Mbits/sec
[  5] local 192.168.2.192 port 5001 connected with 192.168.2.74 port 37327
[  5]  0.0-10.3 sec  64.6 MBytes  52.6 Mbits/sec
[  6] local 192.168.2.192 port 5001 connected with 192.168.2.74 port 37328
[  6]  0.0-10.0 sec  78.2 MBytes  65.5 Mbits/sec

See how the data rates drop from high 80s - 90s+ Mbps down into the 50s+ Mbps?  Despite that data rate slowdown, we only show having dropped 1 packet. There are no errors or xruns reported. We've long been frustrated by this, but not having time to invest in it, we were willing to accept the slower performance for the time being. As we upgraded versions of Linux, we hoped that perhaps it would simply get better. Unfortunately it did not.

*NOW FOR THE "BUG"?:*

Several months ago, we started seeing that slab error that is being discussed on this list (http://thread.gmane.org/gmane.linux.ports.arm.omap/78039/). The error was generic enough that we could not put our finger on it. We could not figure out a way of reliably reproducing it. We assumed something in our software immediately, and spent the last several months constantly updating our BSP to include the latest bootloaders and Linux sources. However, that never made it go away. It was a very sporadic error. Some days the boards would crash constantly with it, and then mysteriously weeks would pass without incident. Right when we started to think our latest kernel update had killed it, it would come roaring back

Last week was one of those weeks where the issues came roaring back. Since they coincided with us enabling the EMAC from within u-boot, we turned our attention to the EMAC immediately and started running constant iperf loop tests against it. For a while, it seemed it could create the slab error relatively easily, but then as fast as it came, it disappeared again.

Today I was running a combination test of iperf and the stress utility (http://weather.ou.edu/~apw/projects/stress/), and even with relatively mild stress loads I was seeing other memory allocation crashes being blamed on the EMAC. (For those that have read the other thread, my SMSC LAN911x port has been disabled in the bootloaders and kernel for these tests. So, it cannot play a role here.)  We saw several errors on two different platforms and across both the 3.4rc6 and 3.5rc1 kernels. All the different errors are reported below.

I'm definitely not a Linux memory management expert, but it's looking like the Davinci EMAC driver, at least as it pertains to the AM35xx platform, may be malfunctioning a bit???  I can provide any additional information that you all would find helpful.

Our tests are simple:
For iperf: One side does "iperf -s" or "iperf -s -D" if running in the back ground is desired, while the other side runs "iperf -c <IP of server>".  We are looping them in a script like:

1.  #!/bin/sh
2.  set -x
3.  while [ 1 ]
4.  do
5.     iperf -c <IP>
6.     sleep 1
7.  done

For stress:  We're running the command lines you see (such as "stress --cpu 8 --io 4 --vm 1 --vm-bytes 150M --hdd 2 --hdd-bytes 150M --timeout 60s").  We're also running those in a script just like above, with the stress line replacing line #5 obviously.

THANKS IN ADVANCE!!!

*CRASHES AND DUMPS:*

This one happened on BOTH Linux 3.4rc6 and 3.5rc1. You can see the stress command for this particular error there. We eventually eased that back to:
"stress --cpu 8 --io 4 --vm 1 --vm-bytes 150M --hdd 2 --hdd-bytes 150M --timeout 60s".
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

+ /home/root/stress --cpu 24 --io 10 --vm 9 --vm-bytes 25M --hdd 2 --hdd-bytes 150M --timeout 60
stress: info: [21812] dispatching hogs: 24 cpu, 10 io, 9 vm, 2 hdd
[61010.621826] stress: page allocation failure: order:0, mode:0x120
[61010.628173] Backtrace:
[61010.630767] [<c0011d6c>] (dump_backtrace+0x0/0x114) from [<c039cb8c>] (dump_stack+0x20/0x24)
[61010.639648]  r6:cd9fc000 r5:00000000 r4:00000120 r3:c052fed8
[61010.645599] [<c039cb6c>] (dump_stack+0x0/0x24) from [<c009c304>] (warn_alloc_failed+0xd8/0x11c)
[61010.654754] [<c009c22c>] (warn_alloc_failed+0x0/0x11c) from [<c009e918>] (__alloc_pages_nodemask+0x508/0x678)
[61010.665161]  r3:cd9fdcf4 r2:00000000
[61010.668914]  r7:c051ef60 r6:00000000 r5:00000000 r4:00000120
[61010.674896] [<c009e410>] (__alloc_pages_nodemask+0x0/0x678) from [<c0309458>] (netdev_alloc_frag+0xa4/0xdc)
[61010.685119] [<c03093b4>] (netdev_alloc_frag+0x0/0xdc) from [<c030a428>] (__netdev_alloc_skb+0x78/0xd0)
[61010.694885]  r7:00000000 r6:7b4c2a43 r5:cfa50000 r4:00000700
[61010.700866] [<c030a3b0>] (__netdev_alloc_skb+0x0/0xd0) from [<c02a0770>] (emac_rx_alloc+0x28/0x64)
[61010.710266]  r6:7b4c2a43 r5:cd95e300 r4:cfa50000 r3:cfa50440
[61010.716217] [<c02a0748>] (emac_rx_alloc+0x0/0x64) from [<c02a1670>] (emac_rx_handler+0x74/0x11c)
[61010.725463]  r4:cfa50000 r3:059e8859
[61010.729217] [<c02a15fc>] (emac_rx_handler+0x0/0x11c) from [<c02a2968>] (__cpdma_chan_free+0xc8/0xe0)
[61010.738830]  r6:cfa6f5c0 r5:60000113 r4:cfa5bac0
[61010.743682] [<c02a28a0>] (__cpdma_chan_free+0x0/0xe0) from [<c02a2a4c>] (__cpdma_chan_process+0xcc/0x104)
[61010.753723] [<c02a2980>] (__cpdma_chan_process+0x0/0x104) from [<c02a35b8>] (cpdma_chan_process+0x4c/0x64)
[61010.763885]  r7:00000040 r6:00000040 r5:cfa6f5c0 r4:00000000
[61010.769836] [<c02a356c>] (cpdma_chan_process+0x0/0x64) from [<c02a1b44>] (emac_poll+0x9c/0x208)
[61010.778961]  r6:00000001 r5:00000001 r4:cfa5044c r3:00000001
[61010.784912] [<c02a1aa8>] (emac_poll+0x0/0x208) from [<c0314340>] (net_rx_action+0xb0/0x1a8)
[61010.793701]  r8:c0531130 r7:0000012c r6:00000040 r5:00000001 r4:cfa5044c
[61010.793701] r3:c02a1aa8
[61010.803314] [<c0314290>] (net_rx_action+0x0/0x1a8) from [<c0035778>] (__do_softirq+0xb0/0x1d8)
[61010.812347] [<c00356c8>] (__do_softirq+0x0/0x1d8) from [<c0035c7c>] (irq_exit+0x8c/0x94)
[61010.820861] [<c0035bf0>] (irq_exit+0x0/0x94) from [<c000f010>] (handle_IRQ+0x44/0x94)
[61010.829101]  r4:c05499b8 r3:c00764a4
[61010.832855] [<c000efcc>] (handle_IRQ+0x0/0x94) from [<c00085b4>] (omap3_intc_handle_irq+0x68/0x78)
[61010.842285]  r6:c055cbbc r5:cd9fdfb0 r4:fa200000 r3:00000044
[61010.848236] [<c000854c>] (omap3_intc_handle_irq+0x0/0x78) from [<c000e480>] (__irq_usr+0x40/0x60)
[61010.857543] Exception stack(0xcd9fdfb0 to 0xcd9fdff8)
[61010.862854] dfa0:                                     b6e474c0 00000000 00000001 00000000
[61010.871459] dfc0: 00000000 00000000 0000e538 0000e530 0000000e 0000e478 00000014 00000000
[61010.880035] dfe0: 0000e4a8 befbbb58 00008adc b6d4d71c 60000010 ffffffff
[61010.886993]  r7:0000e530 r6:ffffffff r5:60000010 r4:b6d4d71c
[61010.892913] Mem-info:
[61010.895324] Normal per-cpu:
[61010.898254] CPU    0: hi:   90, btch:  15 usd:  10
[61010.903289] active_anon:57090 inactive_anon:60 isolated_anon:0
[61010.903289]  active_file:853 inactive_file:324 isolated_file:0
[61010.903289]  unevictable:0 dirty:0 writeback:0 unstable:0
[61010.903289]  free:2251 slab_reclaimable:679 slab_unreclaimable:692
[61010.903289]  mapped:245 shmem:77 pagetables:350 bounce:0
[61010.933258] Normal free:9004kB min:2028kB low:2532kB high:3040kB active_anon:228360kB inactive_anon:240kB active_file:3412kB inacto
[61010.974212] lowmem_reserve[]: 0 0 0
[61010.977905] Normal: 1589*4kB 331*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9004kB
[61010.989044] 1254 total pagecache pages
[61011.003723] 65536 pages of RAM
[61011.006927] 2574 free pages
[61011.009857] 2234 reserved pages
[61011.013153] 1371 slab pages
[61011.016082] 4367 pages shared
[61011.019195] 0 pages swap cached
[61011.022613] ------------[ cut here ]------------
[61011.027557] WARNING: at drivers/net/ethernet/ti/davinci_emac.c:997 emac_rx_alloc+0x5c/0x64()
[61011.036437] Modules linked in:
[61011.039642] Backtrace:
[61011.042236] [<c0011d6c>] (dump_backtrace+0x0/0x114) from [<c039cb8c>] (dump_stack+0x20/0x24)
[61011.051147]  r6:c02a07a4 r5:00000009 r4:00000000 r3:c052fed8
[61011.057098] [<c039cb6c>] (dump_stack+0x0/0x24) from [<c002eb3c>] (warn_slowpath_common+0x5c/0x74)
[61011.066467] [<c002eae0>] (warn_slowpath_common+0x0/0x74) from [<c002eb80>] (warn_slowpath_null+0x2c/0x34)
[61011.076538]  r8:0000003c r7:00000000 r6:7b4c2a43 r5:cd95e300 r4:00000000
[61011.076538] r3:00000009
[61011.086151] [<c002eb54>] (warn_slowpath_null+0x0/0x34) from [<c02a07a4>] (emac_rx_alloc+0x5c/0x64)
[61011.095611] [<c02a0748>] (emac_rx_alloc+0x0/0x64) from [<c02a1670>] (emac_rx_handler+0x74/0x11c)
[61011.104858]  r4:cfa50000 r3:059e8859
[61011.108642] [<c02a15fc>] (emac_rx_handler+0x0/0x11c) from [<c02a2968>] (__cpdma_chan_free+0xc8/0xe0)
[61011.118255]  r6:cfa6f5c0 r5:60000113 r4:cfa5bac0
[61011.123138] [<c02a28a0>] (__cpdma_chan_free+0x0/0xe0) from [<c02a2a4c>] (__cpdma_chan_process+0xcc/0x104)
[61011.133239] [<c02a2980>] (__cpdma_chan_process+0x0/0x104) from [<c02a35b8>] (cpdma_chan_process+0x4c/0x64)
[61011.143402]  r7:00000040 r6:00000040 r5:cfa6f5c0 r4:00000000
[61011.149383] [<c02a356c>] (cpdma_chan_process+0x0/0x64) from [<c02a1b44>] (emac_poll+0x9c/0x208)
[61011.158538]  r6:00000001 r5:00000001 r4:cfa5044c r3:00000001
[61011.164520] [<c02a1aa8>] (emac_poll+0x0/0x208) from [<c0314340>] (net_rx_action+0xb0/0x1a8)
[61011.173339]  r8:c0531130 r7:0000012c r6:00000040 r5:00000001 r4:cfa5044c
[61011.173339] r3:c02a1aa8
[61011.182952] [<c0314290>] (net_rx_action+0x0/0x1a8) from [<c0035778>] (__do_softirq+0xb0/0x1d8)
[61011.192016] [<c00356c8>] (__do_softirq+0x0/0x1d8) from [<c0035c7c>] (irq_exit+0x8c/0x94)
[61011.200561] [<c0035bf0>] (irq_exit+0x0/0x94) from [<c000f010>] (handle_IRQ+0x44/0x94)
[61011.208801]  r4:c05499b8 r3:c00764a4
[61011.212585] [<c000efcc>] (handle_IRQ+0x0/0x94) from [<c00085b4>] (omap3_intc_handle_irq+0x68/0x78)
[61011.222015]  r6:c055cbbc r5:cd9fdfb0 r4:fa200000 r3:00000044
[61011.227996] [<c000854c>] (omap3_intc_handle_irq+0x0/0x78) from [<c000e480>] (__irq_usr+0x40/0x60)
[61011.237335] Exception stack(0xcd9fdfb0 to 0xcd9fdff8)
[61011.242645] dfa0:                                     b6e474c0 00000000 00000001 00000000
[61011.251281] dfc0: 00000000 00000000 0000e538 0000e530 0000000e 0000e478 00000014 00000000
[61011.259887] dfe0: 0000e4a8 befbbb58 00008adc b6d4d71c 60000010 ffffffff
[61011.266845]  r7:0000e530 r6:ffffffff r5:60000010 r4:b6d4d71c
[61011.272827] ---[ end trace c901cd47c92f77fb ]---

Then on our Development Platform (Technexion Twister board in this case), running the same 3.4rc6, this happened IMMEDIATELY after rebooting:
---------------------------------------------------------------------------------------------------------------------------------------------------------

[   40.027679] Unable to handle kernel NULL pointer dereference at virtual addc
[   40.036285] pgd = ce568000
[   40.039154] [0000000c] *pgd=8e4b9831, *pte=00000000, *ppte=00000000
[   40.045776] Internal error: Oops: 17 [#1] ARM
[   40.050354] Modules linked in:
[   40.053588] CPU: 0    Not tainted  (3.4.0-rc6 #4)
[   40.058532] PC is at tcp_rcv_established+0x34/0x65c
[   40.063690] LR is at tcp_v4_do_rcv+0xbc/0x1bc
[   40.068267] pc : [<c032b4d8>]    lr : [<c0331ee0>]    psr: 600f0013
[   40.068267] sp : cfb71c98  ip : cfb71cc8  fp : cfb71cc4
[   40.080322] r10: 00000001  r9 : 00002000  r8 : c052b256
[   40.085815] r7 : 00000000  r6 : ce62d0c0  r5 : 00000000  r4 : ce4b2500
[   40.092681] r3 : 00000000  r2 : 0000003a  r1 : c9011080  r0 : ce4b2500
[   40.099548] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   40.107025] Control: 10c5387d  Table: 8e568019  DAC: 00000015
[   40.113067] Process iperf (pid: 807, stack limit = 0xcfb702f0)
[   40.119201] Stack: (0xcfb71c98 to 0xcfb72000)
[   40.123779] 1c80:                                                       ce62dc00 000005c8
[   40.132385] 1ca0: c052b256 ce62d0c0 ce4b2500 00000000 00000000 c052b256 cfb71cfc cfb71cc8
[   40.140991] 1cc0: c0331ee0 c032b4b0 cfb71cec cfb71cd8 c0055a20 c037d5a4 00000000 c037d9a0
[   40.149597] 1ce0: ce62d0c0 ce4b2500 00000000 00000000 cfb71d24 cfb71d00 c02e2ba8 c0331e30
[   40.158172] 1d00: cfb2c0c0 000003b8 000003b8 ce4b281c ce4b2500 00000000 cfb71d84 cfb71d28
[   40.166778] 1d20: c0322244 c02e2b3c c028668c c02871f8 00000001 c0026964 cfb71d54 00000001
[   40.175384] 1d40: cfb71ecc ce42f3c0 00000000 00000000 c028565c 7fffffff 04000000 c05280d8
[   40.183990] 1d60: cfb71ecc 00000000 cf5e6200 00000000 cfb71ecc 00002000 cfb71dbc cfb71d88
[   40.192596] 1d80: c03401cc c0321fd0 00000000 00000000 cfb71d9c c0285718 00000001 00000000
[   40.201171] 1da0: cfb71dec 00000000 00000000 00000000 cfb71eb4 cfb71dc0 c02df95c c034018c
[   40.209777] 1dc0: 00000000 cfb71dd0 c0034a8c c0034128 00000000 00002000 cf5e6200 0000000a
[   40.218383] 1de0: 00000000 cfb71ecc c007602c cfb70000 00000044 c0034f18 c0503338 cfb70000
[   40.226989] 1e00: 00000044 00000000 cfb71e2c cfb71e18 00000000 00000001 ffffffff 00000000
[   40.235595] 1e20: 00000000 00000000 00000000 00000000 ce42f3c0 fa200000 00000000 00000000
[   40.244171] 1e40: cfb71e6c cfb71e50 cfb71dc8 c000f098 c02e1780 a0000013 ffffffff cfb71ea4
[   40.252777] 1e60: cfb71f8c cfb71e70 c000e3c0 c0008564 cf5e6200 c02e17f8 cfb71ec4 cfae30c0
[   40.261383] 1e80: 00002000 c02dfb04 cfb71ebc 00002000 cf5e6200 00000000 cfb71ee8 00000000
[   40.269989] 1ea0: cfb70000 0001c470 cfb71f8c cfb71eb8 c02e17b4 c02df8b0 cfb71f0c fffffff7
[   40.278594] 1ec0: 00000001 0001e470 00000000 cfb71ee8 00000080 cfb71ec4 00000001 00000000
[   40.287170] 1ee0: 00000000 cfb70000 00000044 c0034f18 c0503338 cfb70000 00000044 00000000
[   40.295776] 1f00: cfb71f24 cfb71f10 c0034f18 c0079244 c007603c c051ab2c cfb71f44 cfb71f28
[   40.304382] 1f20: c000f0d0 c0034ea0 00000044 fa200000 cfb71f68 c052d560 cfb71f64 cfb71f48
[   40.312988] 1f40: c00085cc c000f098 c000e938 80000013 ffffffff cfb71f9c 00000000 cfb71f68
[   40.321594] 1f60: c000e3c0 c0008564 0001c458 0001e478 0001c470 00000123 c000e9c4 00000000
[   40.330169] 1f80: cfb71fa4 cfb71f90 c02e182c c02e1728 00000000 00000000 00000000 cfb71fa8
[   40.338775] 1fa0: c000e780 c02e1810 0001c458 0001e478 00000000 0001c470 00002000 00000000
[   40.347381] 1fc0: 0001c458 0001e478 0001c470 00000123 00fea9f8 00000000 00000fb8 b64d6f9c
[   40.355987] 1fe0: 00000000 b64d6da0 b6e85fe4 b6e85ff4 80000010 00000000 00000000 00000000
[   40.364562] Backtrace:
[   40.367156] [<c032b4a4>] (tcp_rcv_established+0x0/0x65c) from [<c0331ee0>] (tcp_v4_do_rcv+0xbc/0x1bc)
[   40.376861]  r8:c052b256 r7:00000000 r6:00000000 r5:ce4b2500 r4:ce62d0c0
[   40.383941] [<c0331e24>] (tcp_v4_do_rcv+0x0/0x1bc) from [<c02e2ba8>] (release_sock+0x78/0xe0)
[   40.392883]  r7:00000000 r6:00000000 r5:ce4b2500 r4:ce62d0c0
[   40.398864] [<c02e2b30>] (release_sock+0x0/0xe0) from [<c0322244>] (tcp_recvmsg+0x280/0x864)
[   40.407745] [<c0321fc4>] (tcp_recvmsg+0x0/0x864) from [<c03401cc>] (inet_recvmsg+0x4c/0x60)
[   40.416564] [<c0340180>] (inet_recvmsg+0x0/0x60) from [<c02df95c>] (sock_recvmsg+0xb8/0xd8)
[   40.425323]  r6:00000000 r5:00000000 r4:00000000
[   40.430206] [<c02df8a4>] (sock_recvmsg+0x0/0xd8) from [<c02e17b4>] (sys_recvfrom+0x98/0xe8)
[   40.438995] [<c02e171c>] (sys_recvfrom+0x0/0xe8) from [<c02e182c>] (sys_recv+0x28/0x30)
[   40.447418] [<c02e1804>] (sys_recv+0x0/0x30) from [<c000e780>] (ret_fast_syscall+0x0/0x30)
[   40.456115] Code: e5901314 e1a04000 e7c0201f e5c023ac (e595200c)
[   40.462615] ---[ end trace b41eca898abf2dd4 ]---
[   40.467468] Kernel panic - not syncing: Fatal exception in interrupt

After a power-cycle reboot, as soon as I re-enabled the iperf server, which had a client already waiting to run, it threw the dreaded SLAB error, and faulted the EMAC for it:
---------------------------------------------------------------------------------------------------------------------------------------------------------

$ iperf -s -D
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
Running Iperf Server as a daemon
The Iperf daemon process ID : 814
root@sc-hd1u-tam3517:~# [   36.567047] ------------[ cut here ]------------
[   36.571929] kernel BUG at mm/slab.c:3175!
[   36.576141] Internal error: Oops - BUG: 0 [#1] ARM
[   36.581176] Modules linked in:
[   36.584381] CPU: 0    Not tainted  (3.4.0-rc6 #4)
[   36.589355] PC is at cache_alloc_refill+0x13c/0x534
[   36.594482] LR is at kmem_cache_alloc+0x10c/0x11c
[   36.599426] pc : [<c037a610>]    lr : [<c00c3b84>]    psr: 60000093
[   36.599456] sp : ce945a60  ip : 0000003e  fp : ce945aa4
[   36.611511] r10: 00000014  r9 : cf80d7c0  r8 : 0000000a
[   36.617004] r7 : 00100100  r6 : cf813480  r5 : 00200200  r4 : cf829400
[   36.623870] r3 : cfb0e040  r2 : 00000001  r1 : 00000014  r0 : 00000014
[   36.630706] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   36.638305] Control: 10c5387d  Table: 8e910019  DAC: 00000015
[   36.644348] Process iperf (pid: 815, stack limit = 0xce9442f0)
[   36.650482] Stack: (0xce945a60 to 0xce946000)
[   36.655059] 5a60: cf813490 00000020 00000020 00000000 00000020 cf813488 c03141f0 00000020
[   36.663665] 5a80: 60000013 cf80d7c0 00000020 c02e7bac c052b080 cf80d7c0 ce945ad4 ce945aa8
[   36.672271] 5aa0: c00c3b84 c037a4e0 cf9ee000 00000008 cf9ee000 00000020 cf80d7c0 00000634
[   36.680877] 5ac0: c02e812c 00000000 ce945afc ce945ad8 c02e7bac c00c3a84 cf9ee000 cfb33e40
[   36.689453] 5ae0: 039586f6 00000000 000005ea cfb33e40 ce945b14 ce945b00 c02e812c c02e7b7c
[   36.698059] 5b00: cf9ee480 cf9ee000 ce945b2c ce945b18 c02843d4 c02e810c 00009b52 cf9ee000
[   36.706665] 5b20: ce945b54 ce945b30 c02852d4 c02843b8 c0017b30 c02e8828 cfa2c940 cfa2c940
[   36.715270] 5b40: 60000013 cfa31440 ce945b7c ce945b58 c02865b4 c028526c cfa2c940 00000000
[   36.723876] 5b60: d26d0fc0 d08d0660 cf9ee000 ce944000 ce945b9c ce945b80 c028668c c0286510
[   36.732452] 5b80: 00000023 cfa31440 00000040 00000040 ce945bbc ce945ba0 c02871f8 c02865cc
[   36.741058] 5ba0: 00000001 cf9ee48c 00010001 00000001 ce945be4 ce945bc0 c02857a8 c02871b8
[   36.749664] 5bc0: c028570c cf9ee48c 00000001 00000040 0000012c c0501978 ce945c1c ce945be8
[   36.758270] 5be0: c02f1dbc c0285718 ce945be8 ffff7c46 ffffffff 00000001 00000003 0000000c
[   36.766876] 5c00: c053e390 c053e38c 00000b50 ce944000 ce945c64 ce945c20 c0034a1c c02f1d18
[   36.775451] 5c20: cfb0e180 ce8d5084 17762052 0000000a c053e380 00000101 ce945c68 60000013
[   36.784057] 5c40: ce929864 ce894d40 00000020 c052b256 00000b50 00000001 ce945c7c ce945c68
[   36.792663] 5c60: c0034ca0 c0034978 0000000a ce944000 ce945c94 ce945c80 c0034e1c c0034c54
[   36.801269] 5c80: ce9c9500 ce929864 ce945cc4 ce945c98 c032b9bc c0034d78 ce87e2c0 000005c8
[   36.809875] 5ca0: c053e380 ce894d40 ce9c9500 cea3c3c0 00000000 c052b256 ce945cfc ce945cc8
[   36.818481] 5cc0: c0331ee0 c032b4b0 ce945cec ce945cd8 c0034ca0 c0034978 00000008 ce944000
[   36.827056] 5ce0: ce894d40 ce9c9500 cea3c3c0 00000000 ce945d24 ce945d00 c02e2ba8 c0331e30
[   36.835662] 5d00: ce944000 cea66440 ce9c953c ce9c981c ce9c9500 00000000 ce945d84 ce945d28
[   36.844268] 5d20: c0322300 c02e2b3c c0026964 c0017fe0 ce945d54 ce945d40 c028565c 00000001
[   36.852874] 5d40: ce945ecc cea66440 00000000 000014b0 c0285794 7fffffff c028570c c05280d8
[   36.861480] 5d60: ce945ecc 00000000 cf5e36c0 00000000 ce945ecc 00002000 ce945dbc ce945d88
[   36.870056] 5d80: c03401cc c0321fd0 00000000 00000000 ce945d9c c0034a8c 00000100 00000000
[   36.878662] 5da0: 00000003 00000000 00000000 00000000 ce945eb4 ce945dc0 c02df95c c034018c
[   36.887268] 5dc0: 00000000 0000000a c053e380 00000100 00000000 00002000 cf5e36c0 c0034f18
[   36.895874] 5de0: 00000000 ce945ecc 00000044 00000000 ce945e14 ce945e00 c0034f18 c0079244
[   36.904449] 5e00: c007603c c051ab2c ce945e34 ce945e18 00000000 00000001 ffffffff 00000000
[   36.913055] 5e20: 00000000 00000000 00000000 00000000 cea66440 c000f098 00000000 00000000
[   36.921661] 5e40: ffffffff ce945e8c ce945dc8 ce945e58 c000e3c0 c0008564 cfba7ac0 c03ad080
[   36.930267] 5e60: c03ad080 cfba7ac0 ce945ebc ce945ec0 00000000 c02e17f8 00000000 ce944000
[   36.938873] 5e80: 0001c470 c02dfb04 ce945ebc 00002000 cf5e36c0 00000000 ce945ee8 00000000
[   36.947448] 5ea0: ce944000 0001c470 ce945f8c ce945eb8 c02e17b4 c02df8b0 c0034a8c fffffff7
[   36.956054] 5ec0: 00000001 0001e0b8 000003b8 ce945ee8 00000080 ce945ec4 00000001 00000000
[   36.964660] 5ee0: 00000000 c0034f18 c0503338 ce944000 00000044 00000000 ce945f1c ce945f08
[   36.973266] 5f00: c0034f18 c0079244 c007603c c051ab2c ce945f3c ce945f20 c000f0d0 c0034ea0
[   36.981872] 5f20: 00000044 fa200000 ce945f60 c052d560 ce945f5c ce945f40 c00085cc c000f098
[   36.990447] 5f40: c000e780 20000013 ffffffff ce945f94 00000000 ce945f60 c000e3c0 c0008564
[   36.999053] 5f60: 00002000 00000000 0001c458 0001e478 0001c470 00000123 c000e9c4 00000000
[   37.007659] 5f80: ce945fa4 ce945f90 c02e182c c02e1728 00000000 00000000 00000000 ce945fa8
[   37.016265] 5fa0: c000e780 c02e1810 0001c458 0001e478 00000000 0001c470 00002000 00000000
[   37.024871] 5fc0: 0001c458 0001e478 0001c470 00000123 036b4730 00000000 000010f8 b6472f9c
[   37.033447] 5fe0: 00000000 b6472da0 b6e21fe4 b6e21ff4 80000010 00000000 18631b25 4609c480
[   37.042053] Backtrace:
[   37.044647] [<c037a4d4>] (cache_alloc_refill+0x0/0x534) from [<c00c3b84>] (kmem_cache_alloc+0x10c/0x11c)
[   37.054626] [<c00c3a78>] (kmem_cache_alloc+0x0/0x11c) from [<c02e7bac>] (__alloc_skb+0x3c/0xfc)
[   37.063781] [<c02e7b70>] (__alloc_skb+0x0/0xfc) from [<c02e812c>] (__netdev_alloc_skb+0x2c/0x54)
[   37.073059] [<c02e8100>] (__netdev_alloc_skb+0x0/0x54) from [<c02843d4>] (emac_rx_alloc+0x28/0x64)
[   37.082458]  r4:cf9ee000 r3:cf9ee480
[   37.086242] [<c02843ac>] (emac_rx_alloc+0x0/0x64) from [<c02852d4>] (emac_rx_handler+0x74/0x11c)
[   37.095489]  r4:cf9ee000 r3:00009b52
[   37.099273] [<c0285260>] (emac_rx_handler+0x0/0x11c) from [<c02865b4>] (__cpdma_chan_free+0xb0/0xbc)
[   37.108856]  r6:cfa31440 r5:60000013 r4:cfa2c940
[   37.113739] [<c0286504>] (__cpdma_chan_free+0x0/0xbc) from [<c028668c>] (__cpdma_chan_process+0xcc/0x104)
[   37.123809] [<c02865c0>] (__cpdma_chan_process+0x0/0x104) from [<c02871f8>] (cpdma_chan_process+0x4c/0x64)
[   37.133941]  r7:00000040 r6:00000040 r5:cfa31440 r4:00000023
[   37.139923] [<c02871ac>] (cpdma_chan_process+0x0/0x64) from [<c02857a8>] (emac_poll+0x9c/0x208)
[   37.149078]  r6:00000001 r5:00010001 r4:cf9ee48c r3:00000001
[   37.155059] [<c028570c>] (emac_poll+0x0/0x208) from [<c02f1dbc>] (net_rx_action+0xb0/0x1a8)
[   37.163818]  r8:c0501978 r7:0000012c r6:00000040 r5:00000001 r4:cf9ee48c
[   37.170715] r3:c028570c
[   37.173461] [<c02f1d0c>] (net_rx_action+0x0/0x1a8) from [<c0034a1c>] (__do_softirq+0xb0/0x1d8)
[   37.182525] [<c003496c>] (__do_softirq+0x0/0x1d8) from [<c0034ca0>] (do_softirq+0x58/0x64)
[   37.191223] [<c0034c48>] (do_softirq+0x0/0x64) from [<c0034e1c>] (local_bh_enable+0xb0/0xc0)
[   37.200073]  r4:ce944000 r3:0000000a
[   37.203857] [<c0034d6c>] (local_bh_enable+0x0/0xc0) from [<c032b9bc>] (tcp_rcv_established+0x518/0x65c)
[   37.213745]  r5:ce929864 r4:ce9c9500
[   37.217529] [<c032b4a4>] (tcp_rcv_established+0x0/0x65c) from [<c0331ee0>] (tcp_v4_do_rcv+0xbc/0x1bc)
[   37.227203]  r8:c052b256 r7:00000000 r6:cea3c3c0 r5:ce9c9500 r4:ce894d40
[   37.234283] [<c0331e24>] (tcp_v4_do_rcv+0x0/0x1bc) from [<c02e2ba8>] (release_sock+0x78/0xe0)
[   37.243225]  r7:00000000 r6:cea3c3c0 r5:ce9c9500 r4:ce894d40
[   37.249206] [<c02e2b30>] (release_sock+0x0/0xe0) from [<c0322300>] (tcp_recvmsg+0x33c/0x864)
[   37.258087] [<c0321fc4>] (tcp_recvmsg+0x0/0x864) from [<c03401cc>] (inet_recvmsg+0x4c/0x60)
[   37.266906] [<c0340180>] (inet_recvmsg+0x0/0x60) from [<c02df95c>] (sock_recvmsg+0xb8/0xd8)
[   37.275665]  r6:00000000 r5:00000000 r4:00000000
[   37.280548] [<c02df8a4>] (sock_recvmsg+0x0/0xd8) from [<c02e17b4>] (sys_recvfrom+0x98/0xe8)
[   37.289337] [<c02e171c>] (sys_recvfrom+0x0/0xe8) from [<c02e182c>] (sys_recv+0x28/0x30)
[   37.297760] [<c02e1804>] (sys_recv+0x0/0x30) from [<c000e780>] (ret_fast_syscall+0x0/0x30)
[   37.306457] Code: e5930010 e5991018 e1500001 3a00000e (e7f001f2)
[   37.312927] ---[ end trace 3d0484470996ac0b ]---
[   37.317810] Kernel panic - not syncing: Fatal exception in interrupt