* memory clobber in rx path, maybe related to ath9k. @ 2010-10-05 17:00 Ben Greear 2010-10-05 17:16 ` Luis R. Rodriguez 2010-10-05 17:22 ` Johannes Berg 0 siblings, 2 replies; 84+ messages in thread From: Ben Greear @ 2010-10-05 17:00 UTC (permalink / raw) To: linux-wireless@vger.kernel.org I started seeing this very soon after creating interfaces. Maybe caused by a write-after-free? This is on the ath9k system, and I enabled SLUB debugging, etc. Oct 5 09:50:18 localhost kernel: ============================================================================= Oct 5 09:50:18 localhost kernel: BUG kmalloc-8192: Poison overwritten Oct 5 09:50:18 localhost kernel: ----------------------------------------------------------------------------- Oct 5 09:50:18 localhost kernel: Oct 5 09:50:18 localhost kernel: INFO: 0xf5b18040-0xf5b1812f. First byte 0x80 instead of 0x6b Oct 5 09:50:18 localhost kernel: INFO: Allocated in ath_rxbuf_alloc+0x1d/0x74 [ath] age=54091 cpu=0 pid=613 Oct 5 09:50:18 localhost kernel: INFO: Freed in skb_release_data+0x8c/0x90 age=89 cpu=0 pid=4029 Oct 5 09:50:18 localhost kernel: INFO: Slab 0xc1fef300 objects=3 used=2 fp=0xf5b18000 flags=0x400040c1 Oct 5 09:50:18 localhost kernel: INFO: Object 0xf5b18000 @offset=0 fp=0x(null) Oct 5 09:50:18 localhost kernel: Oct 5 09:50:18 localhost kernel: Object 0xf5b18000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18020: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18040: 80 00 00 00 ff ff ff ff ff ff c8 4c 75 20 e8 fd ....ÿÿÿÿÿÿÈLu.èý Oct 5 09:50:18 localhost kernel: Object 0xf5b18050: c8 4c 75 20 e8 fd d0 88 80 c1 e1 03 00 00 00 00 ÈLu.èýÐ..Áá..... Oct 5 09:50:18 localhost kernel: Object 0xf5b18060: 90 01 21 04 00 09 63 69 73 63 6f 73 62 2d 31 01 ..!...ciscosb-1. Oct 5 09:50:18 localhost kernel: Object 0xf5b18070: 08 82 84 8b 96 0c 12 18 24 03 01 0b 05 04 00 01 ........$....... Oct 5 09:50:18 localhost kernel: Object 0xf5b18080: 00 00 2a 01 00 32 04 30 48 60 6c dd 18 00 50 f2 ..*..2.0H`lÝ..Pò Oct 5 09:50:18 localhost kernel: Object 0xf5b18090: 02 01 01 84 00 03 a4 00 00 27 a4 00 00 42 43 5e ......¤..'¤..BC^ Oct 5 09:50:18 localhost kernel: Object 0xf5b180a0: 00 62 32 2f 00 dd 1e 00 90 4c 33 4c 10 1b ff ff .b2/.Ý...L3L..ÿÿ Oct 5 09:50:18 localhost kernel: Object 0xf5b180b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ Oct 5 09:50:18 localhost kernel: Object 0xf5b180c0: 00 00 00 00 00 2d 1a 4c 10 1b ff ff 00 00 00 00 .....-.L..ÿÿ.... Oct 5 09:50:18 localhost kernel: Object 0xf5b180d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ Oct 5 09:50:18 localhost kernel: Object 0xf5b180e0: 00 dd 1a 00 90 4c 34 0b 08 08 00 00 00 00 00 00 .Ý...L4......... Oct 5 09:50:18 localhost kernel: Object 0xf5b180f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 3d 16 0b .............=.. Oct 5 09:50:18 localhost kernel: Object 0xf5b18100: 08 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ Oct 5 09:50:18 localhost kernel: Object 0xf5b18110: 00 00 00 00 00 dd 09 00 03 7f 01 01 00 00 ff 7f .....Ý........ÿ. Oct 5 09:50:18 localhost kernel: Object 0xf5b18120: dd 0a 00 03 7f 04 01 00 00 00 00 00 c9 74 23 19 Ý...........Ét#. Oct 5 09:50:18 localhost kernel: Object 0xf5b18130: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18140: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18150: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18160: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18170: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18180: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18190: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b181a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b181b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b181c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b181d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b181e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b181f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18200: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18210: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18220: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18230: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18240: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18250: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18260: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18270: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18280: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18290: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b182a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b182b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b182c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b182d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b182e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b182f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18300: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18310: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18320: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18330: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18340: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18350: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18360: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18370: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18380: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18390: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b183a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b183b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b183c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b183d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b183e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b183f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18400: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18440: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18450: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18460: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18470: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18480: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18490: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b184a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b184b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b184c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b184d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b184e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b184f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18500: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18510: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18520: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18530: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18540: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18550: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18560: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18570: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18580: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18590: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b185a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b185b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b185c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b185d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b185e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b185f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18600: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18610: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18620: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18630: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18640: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18650: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18660: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18670: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18680: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18690: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b186a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b186b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b186c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b186d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b186e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b186f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b187a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b187b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b187c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b187d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b187e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b187f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b188a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b188b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b188c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b188d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b188e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b188f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18900: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18910: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18920: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18930: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18940: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18950: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18960: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18970: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18980: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18990: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b189a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b189b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b189c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b189d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b189e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b189f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18a00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18a10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18a20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18a30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18a40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18a50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18a60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18a70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18a80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18a90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18aa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ab0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ac0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ad0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ae0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18af0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18b00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18b10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18b20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18b30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18b40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18b50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18b60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18b70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18b80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18bc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18bd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18be0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18bf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18c00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18c10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18c20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18c30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18c40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18c50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18c60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18c70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18c80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18c90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ca0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18cb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18cc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18cd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ce0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18cf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18d00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18d10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18d30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18d40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18d50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18d60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18d70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18d80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18d90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Object 0xf5b18ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 09:50:18 localhost kernel: Redzone 0xf5b1a000: bb bb bb bb »»»» Oct 5 09:50:18 localhost kernel: Padding 0xf5b1a028: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ Oct 5 09:50:18 localhost kernel: Pid: 4049, comm: wpa_supplicant Not tainted 2.6.36-rc6-wl+ #5 Oct 5 09:50:18 localhost kernel: Call Trace: Oct 5 09:50:18 localhost kernel: [<c04ab513>] print_trailer+0xdc/0xe4 Oct 5 09:50:18 localhost kernel: [<c04ab8e9>] check_bytes_and_report+0x96/0xcc Oct 5 09:50:18 localhost kernel: [<c04ab9c6>] check_object+0xa7/0x15d Oct 5 09:50:18 localhost kernel: [<c04acc37>] __slab_alloc+0x339/0x3dd Oct 5 09:50:18 localhost kernel: [<c0455df5>] ? mark_held_locks+0x47/0x5f Oct 5 09:50:18 localhost kernel: [<c04ad0d2>] ? kmem_cache_alloc+0xa0/0xc5 Oct 5 09:50:18 localhost kernel: [<c04ad6fd>] __kmalloc_track_caller+0xa1/0xf2 Oct 5 09:50:18 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83 Oct 5 09:50:18 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83 Oct 5 09:50:18 localhost kernel: [<c06c6d6a>] __alloc_skb+0x58/0xf4 Oct 5 09:50:18 localhost kernel: [<c06c7331>] skb_copy+0x2e/0x83 Oct 5 09:50:18 localhost kernel: [<f8d706dd>] ieee80211_prepare_and_rx_handle+0x3b1/0x86d [mac80211] Oct 5 09:50:18 localhost kernel: [<c0455bee>] ? mark_lock+0x1e/0x1de Oct 5 09:50:18 localhost kernel: [<f8d5f8cf>] ? rcu_read_lock_held+0x1d/0x1f [mac80211] Oct 5 09:50:18 localhost kernel: [<f8d5facf>] ? sta_info_get_bss+0xcb/0x10c [mac80211] Oct 5 09:50:18 localhost kernel: [<f8d71352>] ieee80211_rx+0x70f/0x7b5 [mac80211] Oct 5 09:50:18 localhost kernel: [<c04ad729>] ? __kmalloc_track_caller+0xcd/0xf2 Oct 5 09:50:18 localhost kernel: [<c0456038>] ? trace_hardirqs_on_caller+0xeb/0x125 Oct 5 09:50:18 localhost kernel: [<f8c55836>] ath_rx_send_to_mac80211+0x5a/0x60 [ath9k] Oct 5 09:50:18 localhost kernel: [<c045607d>] ? trace_hardirqs_on+0xb/0xd Oct 5 09:50:18 localhost kernel: [<f8c57066>] ath_rx_tasklet+0x1253/0x130e [ath9k] Oct 5 09:50:18 localhost kernel: [<f8c55232>] ath9k_tasklet+0x9a/0x12a [ath9k] Oct 5 09:50:18 localhost kernel: [<c0438fd1>] tasklet_action+0x73/0xc6 Oct 5 09:50:18 localhost kernel: [<c043945f>] __do_softirq+0x86/0x111 Oct 5 09:50:18 localhost kernel: [<c0439520>] do_softirq+0x36/0x5a Oct 5 09:50:18 localhost kernel: [<c0439659>] irq_exit+0x35/0x69 Oct 5 09:50:18 localhost kernel: [<c0403fb9>] do_IRQ+0x86/0x9a Oct 5 09:50:18 localhost kernel: [<c04034ee>] common_interrupt+0x2e/0x40 Oct 5 09:50:18 localhost kernel: [<c045007b>] ? do_adjtimex+0x217/0x55e Oct 5 09:50:18 localhost kernel: [<c04ad0d7>] ? kmem_cache_alloc+0xa5/0xc5 Oct 5 09:50:18 localhost kernel: [<c04bfe3a>] ? getname+0x1b/0xb8 Oct 5 09:50:18 localhost kernel: [<c04bfe3a>] getname+0x1b/0xb8 Oct 5 09:50:18 localhost kernel: [<c04b5f3e>] do_sys_open+0x14/0xd0 Oct 5 09:50:18 localhost kernel: [<c04b603c>] sys_open+0x1e/0x26 Oct 5 09:50:18 localhost kernel: [<c0760585>] syscall_call+0x7/0xb Oct 5 09:50:18 localhost kernel: FIX kmalloc-8192: Restoring 0xf5b18040-0xf5b1812f=0x6b Oct 5 09:50:18 localhost kernel: -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 17:00 memory clobber in rx path, maybe related to ath9k Ben Greear @ 2010-10-05 17:16 ` Luis R. Rodriguez 2010-10-05 17:24 ` Ben Greear 2010-10-05 17:22 ` Johannes Berg 1 sibling, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-05 17:16 UTC (permalink / raw) To: Ben Greear; +Cc: linux-wireless@vger.kernel.org On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear <greearb@candelatech.com> wrote: > I started seeing this very soon after creating interfaces. Can you be more specific how one can reproduce? Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 17:16 ` Luis R. Rodriguez @ 2010-10-05 17:24 ` Ben Greear 2010-10-05 17:36 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-05 17:24 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-wireless@vger.kernel.org On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote: > On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<greearb@candelatech.com> wrote: >> I started seeing this very soon after creating interfaces. > > Can you be more specific how one can reproduce? Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations), lockdep, pre-empt. Please try creating a bunch of stations on one ath9k phy, and configure them to use WPA (we're using one supplicant per STA). If you don't hit it within 2 minutes of creating STAs and starting WPA, do let me know and I'll try to come up with something more specific. But, at least for us, it seems extremely easy to hit this issue. The same kernel boots fine with no such errors on a system with ath5k, btw. I wish there was a way to get slub debugging to print the backtrace for the code that deleted the memory, but so far, I haven't found anything. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 17:24 ` Ben Greear @ 2010-10-05 17:36 ` Luis R. Rodriguez 2010-10-05 17:38 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-05 17:36 UTC (permalink / raw) To: Ben Greear; +Cc: linux-wireless@vger.kernel.org On Tue, Oct 5, 2010 at 10:24 AM, Ben Greear <greearb@candelatech.com> wrote: > On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote: >> >> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<greearb@candelatech.com> >> wrote: >>> >>> I started seeing this very soon after creating interfaces. >> >> Can you be more specific how one can reproduce? > > Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations), > lockdep, pre-empt. OK I just enabled PAGE_ALLOC thingy, it wasn't enabled on my kernel. > Please try creating a bunch of stations on one ath9k phy, How many? Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 17:36 ` Luis R. Rodriguez @ 2010-10-05 17:38 ` Ben Greear 2010-10-05 17:43 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-05 17:38 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-wireless@vger.kernel.org On 10/05/2010 10:36 AM, Luis R. Rodriguez wrote: > On Tue, Oct 5, 2010 at 10:24 AM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote: >>> >>> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<greearb@candelatech.com> >>> wrote: >>>> >>>> I started seeing this very soon after creating interfaces. >>> >>> Can you be more specific how one can reproduce? >> >> Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations), >> lockdep, pre-empt. > > OK I just enabled PAGE_ALLOC thingy, it wasn't enabled on my kernel. > >> Please try creating a bunch of stations on one ath9k phy, > > How many? 130 is a nice round number :) I'll try with a smaller number and see if I can hit it. Thanks, Ben > > Luis -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 17:38 ` Ben Greear @ 2010-10-05 17:43 ` Luis R. Rodriguez 2010-10-05 17:47 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-05 17:43 UTC (permalink / raw) To: Ben Greear; +Cc: linux-wireless@vger.kernel.org On Tue, Oct 5, 2010 at 10:38 AM, Ben Greear <greearb@candelatech.com> wrote: > On 10/05/2010 10:36 AM, Luis R. Rodriguez wrote: >> >> On Tue, Oct 5, 2010 at 10:24 AM, Ben Greear<greearb@candelatech.com> >> wrote: >>> >>> On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote: >>>> >>>> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<greearb@candelatech.com> >>>> wrote: >>>>> >>>>> I started seeing this very soon after creating interfaces. >>>> >>>> Can you be more specific how one can reproduce? >>> >>> Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations), >>> lockdep, pre-empt. >> >> OK I just enabled PAGE_ALLOC thingy, it wasn't enabled on my kernel. >> >>> Please try creating a bunch of stations on one ath9k phy, >> >> How many? > > 130 is a nice round number :) Let me get this straight. You are creating 130 STAs with ath9k using iw, then running 150 instances of wpa_supplicant to connect to the same AP? > I'll try with a smaller number and see if I can hit it. Please do, you can use the log2n approach, should take you 7 tries, each by a power of 2, start at the middle of course. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 17:43 ` Luis R. Rodriguez @ 2010-10-05 17:47 ` Ben Greear 2010-10-05 17:55 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-05 17:47 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-wireless@vger.kernel.org On 10/05/2010 10:43 AM, Luis R. Rodriguez wrote: > On Tue, Oct 5, 2010 at 10:38 AM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/05/2010 10:36 AM, Luis R. Rodriguez wrote: >>> >>> On Tue, Oct 5, 2010 at 10:24 AM, Ben Greear<greearb@candelatech.com> >>> wrote: >>>> >>>> On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote: >>>>> >>>>> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<greearb@candelatech.com> >>>>> wrote: >>>>>> >>>>>> I started seeing this very soon after creating interfaces. >>>>> >>>>> Can you be more specific how one can reproduce? >>>> >>>> Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations), >>>> lockdep, pre-empt. >>> >>> OK I just enabled PAGE_ALLOC thingy, it wasn't enabled on my kernel. >>> >>>> Please try creating a bunch of stations on one ath9k phy, >>> >>> How many? >> >> 130 is a nice round number :) > > Let me get this straight. You are creating 130 STAs with ath9k using > iw, then running 150 instances of wpa_supplicant to connect to the > same AP? Well, 130 instances, yes :) >> I'll try with a smaller number and see if I can hit it. > > Please do, you can use the log2n approach, should take you 7 tries, > each by a power of 2, start at the middle of course. Sure...will be a few minutes. If it's any sane bug, then hopefully I can hit it with a small number. Out of curiosity, is there any particular number of STAs > 1 you guys test with? Thanks, Ben > > Luis -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 17:47 ` Ben Greear @ 2010-10-05 17:55 ` Luis R. Rodriguez 2010-10-05 18:14 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-05 17:55 UTC (permalink / raw) To: Ben Greear; +Cc: linux-wireless@vger.kernel.org On Tue, Oct 5, 2010 at 10:47 AM, Ben Greear <greearb@candelatech.com> wrote: > On 10/05/2010 10:43 AM, Luis R. Rodriguez wrote: >> >> On Tue, Oct 5, 2010 at 10:38 AM, Ben Greear<greearb@candelatech.com> >> wrote: >>> >>> On 10/05/2010 10:36 AM, Luis R. Rodriguez wrote: >>>> >>>> On Tue, Oct 5, 2010 at 10:24 AM, Ben Greear<greearb@candelatech.com> >>>> wrote: >>>>> >>>>> On 10/05/2010 10:16 AM, Luis R. Rodriguez wrote: >>>>>> >>>>>> On Tue, Oct 5, 2010 at 10:00 AM, Ben Greear<greearb@candelatech.com> >>>>>> wrote: >>>>>>> >>>>>>> I started seeing this very soon after creating interfaces. >>>>>> >>>>>> Can you be more specific how one can reproduce? >>>>> >>>>> Enable SLUB debugging, DEBUG_PAGEALLOC (Debug page memory allocations), >>>>> lockdep, pre-empt. >>>> >>>> OK I just enabled PAGE_ALLOC thingy, it wasn't enabled on my kernel. >>>> >>>>> Please try creating a bunch of stations on one ath9k phy, >>>> >>>> How many? >>> >>> 130 is a nice round number :) >> >> Let me get this straight. You are creating 130 STAs with ath9k using >> iw, then running 150 instances of wpa_supplicant to connect to the >> same AP? > > Well, 130 instances, yes :) > >>> I'll try with a smaller number and see if I can hit it. >> >> Please do, you can use the log2n approach, should take you 7 tries, >> each by a power of 2, start at the middle of course. > > Sure...will be a few minutes. > > If it's any sane bug, then hopefully I can hit it with a small > number. > > Out of curiosity, is there any particular number of STAs > 1 you guys > test with? We don't test this :) Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 17:55 ` Luis R. Rodriguez @ 2010-10-05 18:14 ` Ben Greear 2010-10-05 21:12 ` Ben Greear 2010-10-07 17:33 ` Ben Greear 0 siblings, 2 replies; 84+ messages in thread From: Ben Greear @ 2010-10-05 18:14 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-wireless@vger.kernel.org On 10/05/2010 10:55 AM, Luis R. Rodriguez wrote: > On Tue, Oct 5, 2010 at 10:47 AM, Ben Greear<greearb@candelatech.com> wrote: >>> Please do, you can use the log2n approach, should take you 7 tries, >>> each by a power of 2, start at the middle of course. >> >> Sure...will be a few minutes. >> >> If it's any sane bug, then hopefully I can hit it with a small >> number. >> >> Out of curiosity, is there any particular number of STAs> 1 you guys >> test with? > > We don't test this :) Heh, I sort of supposed so :) Anyway, I started with one, doubled each time, and when I added the 4 to bring the system to 8 STA, it crapped itself. I ran 64kbps UDP data streams on pairs of STA interfaces for 5 minutes between STA increments. I didn't get a chance to start any traffic for the 4 that I had just added. Earlier, I wasn't running any traffic at all, but of course there is still noise on the network. Note I'm running 'iwconfig foo' and 'cat /proc/net/wireless' in the background as well, and that may have something to do with things. Oct 5 11:05:38 localhost kernel: ============================================================================= Oct 5 11:05:38 localhost kernel: BUG kmalloc-8192: Poison overwritten Oct 5 11:05:38 localhost kernel: ----------------------------------------------------------------------------- Oct 5 11:05:38 localhost kernel: Oct 5 11:05:38 localhost kernel: INFO: 0xf6362100-0xf63624d3. First byte 0x96 instead of 0x6b Oct 5 11:05:38 localhost kernel: INFO: Allocated in ath_rxbuf_alloc+0x1d/0x74 [ath] age=11216 cpu=1 pid=0 Oct 5 11:05:38 localhost kernel: INFO: Freed in skb_release_data+0x8c/0x90 age=143 cpu=1 pid=8407 Oct 5 11:05:38 localhost kernel: INFO: Slab 0xc1fffc00 objects=3 used=2 fp=0xf6362030 flags=0x400040c1 Oct 5 11:05:38 localhost kernel: INFO: Object 0xf6362030 @offset=8240 fp=0x(null) Oct 5 11:05:38 localhost kernel: Oct 5 11:05:38 localhost kernel: Bytes b4 0xf6362020: 2a 00 00 00 67 fd 0b 00 5a 5a 5a 5a 5a 5a 5a 5a *...g�..ZZZZZZZZ Oct 5 11:05:38 localhost kernel: Object 0xf6362030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362040: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362050: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362080: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362090: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63620a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63620b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63620c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63620d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63620e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63620f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362100: 96 72 50 c0 11 2c 8e 7e 51 78 b2 6f 4c a2 58 ce .rP�.,.~Qx�oL�X Oct 5 11:05:38 localhost kernel: Object 0xf6362110: b3 34 36 e4 a0 b3 d7 fb bb d9 c2 a7 04 9c 62 46 �46�.�����������٧..bF Oct 5 11:05:38 localhost kernel: Object 0xf6362120: 32 b9 8f 72 0b aa 38 13 22 9a fa 68 5d eb 11 55 2�.r.�8.".�h]�.U Oct 5 11:05:38 localhost kernel: Object 0xf6362130: 2f a3 95 08 16 bb 7e f4 45 45 77 2d c9 c1 63 0b /�...�~�EEw-��c. Oct 5 11:05:38 localhost kernel: Object 0xf6362140: c4 e6 cf 9b 3c cb 88 6e a0 fc 16 d7 2a eb de b3 ���.<�.n.�.�*� Oct 5 11:05:38 localhost kernel: Object 0xf6362150: 19 67 dd d6 ea c5 8e f1 e9 a1 23 0b 91 a4 49 24 .g����.�������#..�I$ Oct 5 11:05:38 localhost kernel: Object 0xf6362160: e0 b7 07 ce 68 fd b4 77 99 1b 6f 2d e4 3d 10 2f ��.�h�෴w..o-�=./ Oct 5 11:05:38 localhost kernel: Object 0xf6362170: a5 0e 46 8b 4f c3 75 2d 63 8e dd f5 25 9e da bb �.F.O�u-c.��%.�䥻 Oct 5 11:05:38 localhost kernel: Object 0xf6362180: 64 db d3 f2 e2 3a ce bd 7d b5 2d de 75 c5 7a 48 d����:�۽}�-�u�zH Oct 5 11:05:38 localhost kernel: Object 0xf6362190: 17 f7 81 94 c4 48 5a 7a 6c ae a1 93 e5 74 98 67 .�..�HZzlޮ�.�t.g Oct 5 11:05:38 localhost kernel: Object 0xf63621a0: 1c a9 e3 80 35 65 24 c7 ce 7f 37 1b a7 a3 35 b5 .��.5e$��.7.婧�5� Oct 5 11:05:38 localhost kernel: Object 0xf63621b0: ee a2 7e 9c 30 f2 a4 8d e1 f7 e6 8c 0c c7 79 71 ��~.0�.���..�yq Oct 5 11:05:38 localhost kernel: Object 0xf63621c0: 76 99 68 e5 e1 03 34 03 21 ff a9 35 98 20 0a ec v.h��.4.!��5... Oct 5 11:05:38 localhost kernel: Object 0xf63621d0: 0f fc 03 3d dc 88 75 b2 39 f2 65 68 fd 27 de 83 .�.=�.uᩲ9�eh�'�. Oct 5 11:05:38 localhost kernel: Object 0xf63621e0: 99 7a 61 a6 82 6d a9 af 79 c3 fa c3 20 fd 6d 2b .za�.m�y���.�m+ Oct 5 11:05:38 localhost kernel: Object 0xf63621f0: b9 fc 0d f7 be 4f b9 4f c3 a1 97 00 0a ad 0a e2 ù�.��O�Oá...�. Oct 5 11:05:38 localhost kernel: Object 0xf6362200: a6 88 03 42 5b c5 c7 cf 69 a5 42 5c b8 1b d7 6d ������..B[���iťB\�.�m Oct 5 11:05:38 localhost kernel: Object 0xf6362210: 92 7a 64 04 1e 39 77 26 7f 22 11 69 86 60 fe aa .zd..9w&.".i.`�ת Oct 5 11:05:38 localhost kernel: Object 0xf6362220: 1c d8 49 9a 9d a8 47 dd ae eb 33 53 cc ba 21 17 .�I..بG�ݮ�3S̺!. Oct 5 11:05:38 localhost kernel: Object 0xf6362230: 62 78 92 99 ae f6 91 f8 71 da a4 35 6b 8c fd ce bx..뺮�.�qڤ5k.� Oct 5 11:05:38 localhost kernel: Object 0xf6362240: 95 28 a1 7f af 14 fc 7a b7 6e 52 81 bd 22 35 72 .(�.����.�z�nR.�"5r Oct 5 11:05:38 localhost kernel: Object 0xf6362250: 33 25 be 93 65 fd 1e 18 4c 4e 85 b5 55 e9 ed 53 3%�.e�..LN.�U��S Oct 5 11:05:38 localhost kernel: Object 0xf6362260: 52 a2 3f 5d 92 64 0e ec 45 ce 0c 3d 98 56 fe 1e R������?].d.�E�.=.V�. Oct 5 11:05:38 localhost kernel: Object 0xf6362270: 9d 95 0d 49 b2 a7 cb c8 2a 7d 2b 8a 9a 59 36 51 ...I�첧��*}+..Y6Q Oct 5 11:05:38 localhost kernel: Object 0xf6362280: ca 8c cf 4e c0 df 09 42 5f 61 40 a5 7f 92 d8 88 �.�N��.B_a@˥..�. Oct 5 11:05:38 localhost kernel: Object 0xf6362290: 0d 51 75 4b c3 2e 48 e1 14 21 7f 00 23 be 42 55 .QuK�.H�.!..#ؾBU Oct 5 11:05:38 localhost kernel: Object 0xf63622a0: 9f 6b 88 67 d2 e1 ce 1c 6c 1f 39 b6 9b 73 e9 8c .k.g���.l.9Ҷ.s�. Oct 5 11:05:38 localhost kernel: Object 0xf63622b0: 3b 44 70 de 0d 4b 78 40 18 a3 3c fc 50 4c a6 89 ;Dp�.Kx@.�<�PL飦. Oct 5 11:05:38 localhost kernel: Object 0xf63622c0: 78 6e d1 c8 87 81 0c c0 41 2a d9 d1 98 10 48 b1 xn��...�A*��..Hѱ Oct 5 11:05:38 localhost kernel: Object 0xf63622d0: e2 ef 37 77 98 e0 d3 61 67 9c 0c 5b 63 72 1c d3 ��7w.��ag..[cr. Oct 5 11:05:38 localhost kernel: Object 0xf63622e0: ee 8b cf 29 7f 4b 18 b5 46 18 2e ba 43 e1 6a bb �.�).K.�F..C�j� Oct 5 11:05:38 localhost kernel: Object 0xf63622f0: 4d 92 86 63 05 4b e9 f0 0d 9e 36 23 11 8a 38 8f M..c.K��..6#..8. Oct 5 11:05:38 localhost kernel: Object 0xf6362300: e4 4d 26 4b 64 51 53 05 c5 f9 01 8c 34 95 71 76 �M&KdQS.��..4.qv Oct 5 11:05:38 localhost kernel: Object 0xf6362310: 2d 57 a8 3e ef 8c 98 5c 36 d1 04 30 91 cf 51 5f -WỨ>�..\6�.0.�Q_ Oct 5 11:05:38 localhost kernel: Object 0xf6362320: b9 f4 ab b5 3a dc a7 e6 ae 98 23 21 80 50 c6 70 ��﹫�:�ܧ��.#!.P�p Oct 5 11:05:38 localhost kernel: Object 0xf6362330: 1c c7 14 65 af 48 54 53 76 6f bd 02 e4 b0 13 2a .�.e殯HTSvo�.��.* Oct 5 11:05:38 localhost kernel: Object 0xf6362340: c6 37 eb 50 bc 20 35 d1 e0 6c 50 9c 1b d2 0e ed �7�P䰼.5��lP..�. Oct 5 11:05:38 localhost kernel: Object 0xf6362350: 8f d1 b9 9f b6 7c 9c 5f ac f3 86 c1 b6 c0 31 86 .�ѹ.�|._��.���1. Oct 5 11:05:38 localhost kernel: Object 0xf6362360: a7 e6 c2 e4 78 f6 20 41 71 16 f8 7e bf 1d 47 6e ����x�.Aq.�~.Gn Oct 5 11:05:38 localhost kernel: Object 0xf6362370: 82 b9 df 67 d3 b2 fe f3 08 b0 e4 b4 53 b8 23 d8 .��g�߲��.���S# Oct 5 11:05:38 localhost kernel: Object 0xf6362380: ad 46 3e 7b f4 6e 1d 00 de f7 c6 a9 9a 2f 4f 75 حF>{�n..��Ʃ./Ou Oct 5 11:05:38 localhost kernel: Object 0xf6362390: a7 e3 99 8e b6 a5 f2 0d 73 ac 32 47 96 c1 cf 0b ��..������.s�2G.��. Oct 5 11:05:38 localhost kernel: Object 0xf63623a0: 79 80 7e cd ac 3d 42 41 32 af 62 79 3d f8 98 ce y.~ͬ=BA2by=�. Oct 5 11:05:38 localhost kernel: Object 0xf63623b0: ab df d0 0c 24 36 8c 7c 66 78 18 c1 25 2a 06 95 ���.$6.|fx.�%*.. Oct 5 11:05:38 localhost kernel: Object 0xf63623c0: ce 9c ae 2c 84 71 3e 4c 70 4c d3 f2 f4 cc 5b b9 �.�,.q>LpL����[� Oct 5 11:05:38 localhost kernel: Object 0xf63623d0: 42 a0 a7 73 a1 73 1b b8 5c 7d 3c 76 b9 78 bd 6b B.�����s�s.�\}<v�x�k Oct 5 11:05:38 localhost kernel: Object 0xf63623e0: 30 ca b2 11 1d b3 30 6b 0e 5a f6 53 64 99 24 58 0�ʲ..�0k.Z�Sd.$X Oct 5 11:05:38 localhost kernel: Object 0xf63623f0: a8 7a 47 e0 2e 1b cf b4 49 ea 3d 77 90 6f f2 87 �zG�..ϴI�=w.o�. Oct 5 11:05:38 localhost kernel: Object 0xf6362400: e2 36 0f 51 68 9d d3 70 84 bf 7e e6 c9 12 47 29 �6.Qh.�p.����~��.G) Oct 5 11:05:38 localhost kernel: Object 0xf6362410: 3b fb f2 32 34 58 fd 6d 2a a8 9d 47 bb b4 90 c2 ;��24X�m*�.G樻�. Oct 5 11:05:38 localhost kernel: Object 0xf6362420: 8b a5 fc 42 d7 a7 e3 44 9c 36 37 0b b5 51 4b cc .¥�Bק�D.67.�QK Oct 5 11:05:38 localhost kernel: Object 0xf6362430: 2a eb 55 e2 fe 98 cb 4f 34 29 b8 18 59 f9 af c1 *�U��.�O4)�.Y�� Oct 5 11:05:38 localhost kernel: Object 0xf6362440: e7 a0 3a e8 1e 01 37 80 4d 6d 9c 37 e6 8f ae 82 �.:�..7.Mm.7�.������. Oct 5 11:05:38 localhost kernel: Object 0xf6362450: 59 5d e4 99 3f 6e f3 cf d6 26 c5 21 1b aa ce fd Y]�.?n���&�!.�� Oct 5 11:05:38 localhost kernel: Object 0xf6362460: 01 d3 59 20 38 99 5c 85 95 de d8 01 14 1a 8f 65 .�Y.8.\..��....e Oct 5 11:05:38 localhost kernel: Object 0xf6362470: dc 54 2a f6 cd 3b 9a 54 73 0d 34 1f 73 19 ac 5c �T*��;.Ts.4.s.䪬\ Oct 5 11:05:38 localhost kernel: Object 0xf6362480: af 38 9c 4d 2a a3 7b ec b3 1a 21 c3 fe 42 4e 8f �8.M*�{��.!��BN. Oct 5 11:05:38 localhost kernel: Object 0xf6362490: 03 a1 14 38 f2 bf a3 da 60 69 08 42 85 9e 8c 01 .쳡.8����`i.B.... Oct 5 11:05:38 localhost kernel: Object 0xf63624a0: 18 e4 b7 6e a3 f6 b9 11 bb ee 0d 07 0d 2a f2 7c .�n���.��...*�| Oct 5 11:05:38 localhost kernel: Object 0xf63624b0: 9c 13 2c 56 56 d5 05 8d 2a e3 45 83 26 09 50 3d ..,VV�..*�E.&.P= Oct 5 11:05:38 localhost kernel: Object 0xf63624c0: 42 7c 49 5a bd ac 2d ca 49 90 27 2c e7 42 ef e1 B|IZ�����-�I.',�B� Oct 5 11:05:38 localhost kernel: Object 0xf63624d0: 42 2f fa 99 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b B/�.kkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63624e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63624f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362500: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362510: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362520: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362530: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362540: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362550: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362560: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362570: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362580: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362590: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63625a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63625b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63625c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63625d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63625e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63625f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362600: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362610: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362620: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362630: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362640: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362650: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362660: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362670: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362680: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362690: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63626a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63626b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63626c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63626d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63626e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63626f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63627a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63627b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63627c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63627d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63627e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63627f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63628a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63628b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63628c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63628d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63628e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63628f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362900: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362910: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362920: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362930: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362940: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362950: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362960: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362970: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362980: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362990: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63629a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63629b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63629c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63629d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63629e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf63629f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362a00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362a10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362a20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362a30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362a40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362a50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362a60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362a70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362a80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362a90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362aa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ab0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ac0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ad0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ae0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362af0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362b00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362b10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362b20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362b30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362b40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362b50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362b60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362b70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362b80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362bc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362bd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362be0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362bf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362c00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362c10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362c20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362c30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362c40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362c50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362c60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362c70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362c80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362c90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ca0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362cb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362cc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362cd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ce0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362cf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362d00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362d10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362d30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362d40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362d50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362d60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362d70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362d80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362d90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6362ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6363000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6363010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Object 0xf6363020: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:38 localhost kernel: Redzone 0xf6364030: bb bb bb bb ʻ��� Oct 5 11:05:38 localhost kernel: Padding 0xf6364058: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ Oct 5 11:05:38 localhost kernel: Pid: 8416, comm: parse-iwconfig. Not tainted 2.6.36-rc6-wl+ #5 Oct 5 11:05:38 localhost kernel: Call Trace: Oct 5 11:05:38 localhost kernel: [<c04ab513>] print_trailer+0xdc/0xe4 Oct 5 11:05:38 localhost kernel: [<c04ab8e9>] check_bytes_and_report+0x96/0xcc Oct 5 11:05:38 localhost kernel: [<c04ab9c6>] check_object+0xa7/0x15d Oct 5 11:05:38 localhost kernel: [<c04acc37>] __slab_alloc+0x339/0x3dd Oct 5 11:05:38 localhost kernel: [<c0455df5>] ? mark_held_locks+0x47/0x5f Oct 5 11:05:38 localhost kernel: [<c04ad0d2>] ? kmem_cache_alloc+0xa0/0xc5 Oct 5 11:05:38 localhost kernel: [<c04ad6fd>] __kmalloc_track_caller+0xa1/0xf2 Oct 5 11:05:38 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83 Oct 5 11:05:38 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83 Oct 5 11:05:38 localhost kernel: [<c06c6d6a>] __alloc_skb+0x58/0xf4 Oct 5 11:05:38 localhost kernel: [<c06c7331>] skb_copy+0x2e/0x83 Oct 5 11:05:38 localhost kernel: [<f8d426dd>] ieee80211_prepare_and_rx_handle+0x3b1/0x86d [mac80211] Oct 5 11:05:38 localhost kernel: [<c0455bee>] ? mark_lock+0x1e/0x1de Oct 5 11:05:38 localhost kernel: [<c0455bee>] ? mark_lock+0x1e/0x1de Oct 5 11:05:38 localhost kernel: [<f8d31ab4>] ? sta_info_get_bss+0xb0/0x10c [mac80211] Oct 5 11:05:38 localhost kernel: [<f8d43352>] ieee80211_rx+0x70f/0x7b5 [mac80211] Oct 5 11:05:38 localhost kernel: [<c04ad729>] ? __kmalloc_track_caller+0xcd/0xf2 Oct 5 11:05:38 localhost kernel: [<c0456038>] ? trace_hardirqs_on_caller+0xeb/0x125 Oct 5 11:05:38 localhost kernel: [<f86e3836>] ath_rx_send_to_mac80211+0x5a/0x60 [ath9k] Oct 5 11:05:38 localhost kernel: [<c045607d>] ? trace_hardirqs_on+0xb/0xd Oct 5 11:05:38 localhost kernel: [<f86e5066>] ath_rx_tasklet+0x1253/0x130e [ath9k] Oct 5 11:05:38 localhost kernel: [<f86e3232>] ath9k_tasklet+0x9a/0x12a [ath9k] Oct 5 11:05:38 localhost kernel: [<c0438fd1>] tasklet_action+0x73/0xc6 Oct 5 11:05:38 localhost kernel: [<c043945f>] __do_softirq+0x86/0x111 Oct 5 11:05:38 localhost kernel: [<c0439520>] do_softirq+0x36/0x5a Oct 5 11:05:38 localhost kernel: [<c0439659>] irq_exit+0x35/0x69 Oct 5 11:05:38 localhost kernel: [<c0403fb9>] do_IRQ+0x86/0x9a Oct 5 11:05:38 localhost kernel: [<c04034ee>] common_interrupt+0x2e/0x40 Oct 5 11:05:38 localhost kernel: FIX kmalloc-8192: Restoring 0xf6362100-0xf63624d3=0x6b Oct 5 11:05:38 localhost kernel: Oct 5 11:05:38 localhost kernel: FIX kmalloc-8192: Marking all objects used Oct 5 11:05:39 localhost kernel: sta5: deauthenticating from 00:14:d1:c6:d2:54 by local choice (reason=3) Oct 5 11:05:39 localhost kernel: ieee80211 phy0: Removed STA 00:14:d1:c6:d2:54 Oct 5 11:05:39 localhost kernel: ieee80211 phy0: Destroyed STA 00:14:d1:c6:d2:54 Oct 5 11:05:41 localhost kernel: sta6: no IPv6 routers present Oct 5 11:05:41 localhost kernel: ============================================================================= Oct 5 11:05:41 localhost kernel: BUG kmalloc-8192: Poison overwritten Oct 5 11:05:41 localhost kernel: ----------------------------------------------------------------------------- Oct 5 11:05:41 localhost kernel: Oct 5 11:05:41 localhost kernel: INFO: 0xf34d40a0-0xf34d411f. First byte 0x80 instead of 0x6b Oct 5 11:05:41 localhost kernel: INFO: Allocated in ath_rxbuf_alloc+0x1d/0x74 [ath] age=9852 cpu=1 pid=0 Oct 5 11:05:41 localhost kernel: INFO: Freed in skb_release_data+0x8c/0x90 age=66 cpu=1 pid=0 Oct 5 11:05:41 localhost kernel: INFO: Slab 0xc1fa2a00 objects=3 used=2 fp=0xf34d4060 flags=0x400040c1 Oct 5 11:05:41 localhost kernel: INFO: Object 0xf34d4060 @offset=16480 fp=0x(null) Oct 5 11:05:41 localhost kernel: Oct 5 11:05:41 localhost kernel: Bytes b4 0xf34d4050: 00 00 00 00 fe 02 0c 00 5a 5a 5a 5a 5a 5a 5a 5a ....�...ZZZZZZZZ Oct 5 11:05:41 localhost kernel: Object 0xf34d4060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4080: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4090: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d40a0: 80 00 00 00 ff ff ff ff ff ff 00 1d 7e 23 46 52 ....������..~#FR Oct 5 11:05:41 localhost kernel: Object 0xf34d40b0: 00 1d 7e 23 46 52 00 ad 87 a1 f9 6a 90 06 00 00 ..~#FR.�.��j.... Oct 5 11:05:41 localhost kernel: Object 0xf34d40c0: 64 00 11 04 00 0f 46 65 72 6e 64 61 6c 65 43 68 d.....FerndaleCh Oct 5 11:05:41 localhost kernel: Object 0xf34d40d0: 61 6d 62 65 72 01 08 82 84 8b 96 24 30 48 6c 03 amber......$0Hl. Oct 5 11:05:41 localhost kernel: Object 0xf34d40e0: 01 06 05 04 00 01 00 00 2a 01 04 2f 01 04 32 04 ........*../..2. Oct 5 11:05:41 localhost kernel: Object 0xf34d40f0: 0c 12 18 60 dd 09 00 10 18 02 01 f4 00 00 00 dd ...`�......�... Oct 5 11:05:41 localhost kernel: Object 0xf34d4100: 18 00 50 f2 01 01 00 00 50 f2 02 01 00 00 50 f2 ..P�....P�....P Oct 5 11:05:41 localhost kernel: Object 0xf34d4110: 02 01 00 00 50 f2 02 00 00 51 f0 2b 8f 51 f0 8f ....P�...Q�+.Q�. Oct 5 11:05:41 localhost kernel: Object 0xf34d4120: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4130: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4140: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4150: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4160: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4170: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4180: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4190: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d41a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d41b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d41c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d41d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d41e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d41f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4200: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4210: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4220: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4230: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4240: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4250: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4260: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4270: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4280: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4290: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d42a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d42b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d42c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d42d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d42e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d42f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4300: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4310: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4320: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4330: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4340: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4350: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4360: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4370: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4380: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4390: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d43a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d43b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d43c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d43d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d43e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d43f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4400: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4440: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4450: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4460: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4470: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4480: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4490: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d44a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d44b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d44c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d44d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d44e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d44f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4500: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4510: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4520: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4530: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4540: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4550: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4560: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4570: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4580: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4590: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d45a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d45b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d45c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d45d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d45e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d45f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4600: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4610: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4620: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4630: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4640: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4650: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4660: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4670: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4680: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4690: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d46a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d46b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d46c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d46d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d46e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d46f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d47a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d47b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d47c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d47d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d47e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d47f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d48a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d48b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d48c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d48d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d48e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d48f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4900: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4910: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4920: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4930: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4940: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4950: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4960: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4970: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4980: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4990: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d49a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d49b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d49c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d49d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d49e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d49f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4a00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4a10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4a20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4a30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4a40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4a50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4a60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4a70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4a80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4a90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4aa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ab0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ac0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ad0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ae0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4af0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4b00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4b10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4b20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4b30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4b40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4b50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4b60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4b70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4b80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4bc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4bd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4be0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4bf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4c00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4c10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4c20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4c30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4c40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4c50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4c60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4c70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4c80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4c90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ca0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4cb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4cc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4cd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ce0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4cf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4d00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4d10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4d30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4d40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4d50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4d60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4d70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4d80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4d90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d4ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d5000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d5010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d5020: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d5030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d5040: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Object 0xf34d5050: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Oct 5 11:05:41 localhost kernel: Redzone 0xf34d6060: bb bb bb bb �������� Oct 5 11:05:41 localhost kernel: Padding 0xf34d6088: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ Oct 5 11:05:41 localhost kernel: Pid: 0, comm: kworker/0:0 Not tainted 2.6.36-rc6-wl+ #5 Oct 5 11:05:41 localhost kernel: Call Trace: Oct 5 11:05:41 localhost kernel: [<c04ab513>] print_trailer+0xdc/0xe4 Oct 5 11:05:41 localhost kernel: [<c04ab8e9>] check_bytes_and_report+0x96/0xcc Oct 5 11:05:41 localhost kernel: [<c04ab9c6>] check_object+0xa7/0x15d Oct 5 11:05:41 localhost kernel: [<c04acc37>] __slab_alloc+0x339/0x3dd Oct 5 11:05:41 localhost kernel: [<c0455df5>] ? mark_held_locks+0x47/0x5f Oct 5 11:05:41 localhost kernel: [<c04ad0d2>] ? kmem_cache_alloc+0xa0/0xc5 Oct 5 11:05:41 localhost kernel: [<c04ad6fd>] __kmalloc_track_caller+0xa1/0xf2 Oct 5 11:05:41 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83 Oct 5 11:05:41 localhost kernel: [<c06c7331>] ? skb_copy+0x2e/0x83 Oct 5 11:05:41 localhost kernel: [<c06c6d6a>] __alloc_skb+0x58/0xf4 Oct 5 11:05:41 localhost kernel: [<c06c7331>] skb_copy+0x2e/0x83 Oct 5 11:05:41 localhost kernel: [<f8d426dd>] ieee80211_prepare_and_rx_handle+0x3b1/0x86d [mac80211] Oct 5 11:05:41 localhost kernel: [<c0455bee>] ? mark_lock+0x1e/0x1de Oct 5 11:05:41 localhost kernel: [<f8d31ab4>] ? sta_info_get_bss+0xb0/0x10c [mac80211] Oct 5 11:05:41 localhost kernel: [<f8d43352>] ieee80211_rx+0x70f/0x7b5 [mac80211] Oct 5 11:05:41 localhost kernel: [<c04ad729>] ? __kmalloc_track_caller+0xcd/0xf2 Oct 5 11:05:41 localhost kernel: [<c0456038>] ? trace_hardirqs_on_caller+0xeb/0x125 Oct 5 11:05:41 localhost kernel: [<f86e3836>] ath_rx_send_to_mac80211+0x5a/0x60 [ath9k] Oct 5 11:05:41 localhost kernel: [<c045607d>] ? trace_hardirqs_on+0xb/0xd Oct 5 11:05:41 localhost kernel: [<f86e5066>] ath_rx_tasklet+0x1253/0x130e [ath9k] Oct 5 11:05:41 localhost kernel: [<f86e3232>] ath9k_tasklet+0x9a/0x12a [ath9k] Oct 5 11:05:41 localhost kernel: [<c0438fd1>] tasklet_action+0x73/0xc6 Oct 5 11:05:41 localhost kernel: [<c043945f>] __do_softirq+0x86/0x111 Oct 5 11:05:41 localhost kernel: [<c0439520>] do_softirq+0x36/0x5a Oct 5 11:05:41 localhost kernel: [<c0439659>] irq_exit+0x35/0x69 Oct 5 11:05:41 localhost kernel: [<c0403fb9>] do_IRQ+0x86/0x9a Oct 5 11:05:41 localhost kernel: [<c04034ee>] common_interrupt+0x2e/0x40 Oct 5 11:05:41 localhost kernel: [<c045007b>] ? do_adjtimex+0x217/0x55e Oct 5 11:05:41 localhost kernel: [<c0408245>] ? mwait_idle+0x5c/0x6c Oct 5 11:05:41 localhost kernel: [<c040227f>] cpu_idle+0x4e/0x6b Oct 5 11:05:41 localhost kernel: [<c075b591>] start_secondary+0x1a8/0x1ad Oct 5 11:05:41 localhost kernel: FIX kmalloc-8192: Restoring 0xf34d40a0-0xf34d411f=0x6b Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 18:14 ` Ben Greear @ 2010-10-05 21:12 ` Ben Greear 2010-10-07 17:33 ` Ben Greear 1 sibling, 0 replies; 84+ messages in thread From: Ben Greear @ 2010-10-05 21:12 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-wireless@vger.kernel.org On 10/05/2010 11:14 AM, Ben Greear wrote: > On 10/05/2010 10:55 AM, Luis R. Rodriguez wrote: >> On Tue, Oct 5, 2010 at 10:47 AM, Ben Greear<greearb@candelatech.com> >> wrote: > >>>> Please do, you can use the log2n approach, should take you 7 tries, >>>> each by a power of 2, start at the middle of course. >>> >>> Sure...will be a few minutes. >>> >>> If it's any sane bug, then hopefully I can hit it with a small >>> number. >>> >>> Out of curiosity, is there any particular number of STAs> 1 you guys >>> test with? >> >> We don't test this :) > > Heh, I sort of supposed so :) > > Anyway, I started with one, doubled each time, > and when I added the 4 to bring the system to > 8 STA, it crapped itself. > > I ran 64kbps UDP data streams on pairs of STA interfaces > for 5 minutes between STA increments. I didn't get a chance to > start any traffic for the 4 that I had just added. > > Earlier, I wasn't running any traffic at all, but of course > there is still noise on the network. > > Note I'm running 'iwconfig foo' and 'cat /proc/net/wireless' in > the background as well, and that may have something to do with > things. I tried with the wmb() patch, and the problem still exists, so it must be something else. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 18:14 ` Ben Greear 2010-10-05 21:12 ` Ben Greear @ 2010-10-07 17:33 ` Ben Greear 2010-10-07 18:14 ` Johannes Berg 1 sibling, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-07 17:33 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-wireless@vger.kernel.org In case it helps, here is a dump of where the corrupted SKB was deleted. I added debugging to slub to get this information, but it looks like it's correct to me. Reading symbols from /home/greearb/kernel/2.6/wireless-testing-dbg.p4s/net/mac80211/mac80211.ko...done. (gdb) l *(ieee80211_rx+0x74d) 0x13751 is in ieee80211_rx (/home/greearb/git/linux.wireless-testing/include/linux/rcupdate.h:346). 341 * 342 * See rcu_read_lock() for more information. 343 */ 344 static inline void rcu_read_unlock(void) 345 { 346 rcu_read_release(); 347 __release(RCU); 348 __rcu_read_unlock(); 349 } 350 (gdb) # I don't really know what that second address means, but just in case it's useful, # I printed it out here: (gdb) l *(ieee80211_rx+0x7b4) 0x137b8 is in ieee80211_process_measurement_req (/home/greearb/git/linux.wireless-testing/net/mac80211/spectmgmt.c:74). 69 } 70 71 void ieee80211_process_measurement_req(struct ieee80211_sub_if_data *sdata, 72 struct ieee80211_mgmt *mgmt, 73 size_t len) 74 { 75 /* 76 * Ignoring measurement request is spec violation. 77 * Mandatory measurements must be reported optional 78 * measurements might be refused or reported incapable INFO: Freed in skb_release_data+0x8c/0x90 age=122 cpu=1 pid=0 set_track+0x3c/0x89 __slab_free+0x17f/0x1ba skb_release_data+0x8c/0x90 kfree+0xaf/0xdf skb_release_data+0x8c/0x90 skb_release_data+0x8c/0x90 skb_release_data+0x8c/0x90 __kfree_skb+0x12/0x6d consume_skb+0x2a/0x2c ieee80211_rx+0x74d/0x7b4 [mac80211] __kmalloc_track_caller+0xcd/0xf2 trace_hardirqs_on_caller+0xeb/0x125 ath_rx_send_to_mac80211+0x5a/0x60 [ath9k] trace_hardirqs_on+0xb/0xd Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 17:33 ` Ben Greear @ 2010-10-07 18:14 ` Johannes Berg 2010-10-07 18:29 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Johannes Berg @ 2010-10-07 18:14 UTC (permalink / raw) To: Ben Greear; +Cc: Luis R. Rodriguez, linux-wireless@vger.kernel.org On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote: > In case it helps, here is a dump of where the corrupted SKB was deleted. I wonder, do you have a machine with a decent IOMMU? Adding IOMMU debugging into the mix could help you figure out if it's a DMA problem. johannes ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 18:14 ` Johannes Berg @ 2010-10-07 18:29 ` Luis R. Rodriguez 2010-10-07 18:39 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-07 18:29 UTC (permalink / raw) To: Johannes Berg; +Cc: Ben Greear, linux-wireless@vger.kernel.org On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg <johannes@sipsolutions.net> wrote: > On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote: >> In case it helps, here is a dump of where the corrupted SKB was deleted. > > I wonder, do you have a machine with a decent IOMMU? Adding IOMMU > debugging into the mix could help you figure out if it's a DMA problem. Ben, how much traffic are you RX'ing on these virtual interfaces? Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 18:29 ` Luis R. Rodriguez @ 2010-10-07 18:39 ` Ben Greear 2010-10-07 18:42 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-07 18:39 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/07/2010 11:29 AM, Luis R. Rodriguez wrote: > On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg > <johannes@sipsolutions.net> wrote: >> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote: >>> In case it helps, here is a dump of where the corrupted SKB was deleted. >> >> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU >> debugging into the mix could help you figure out if it's a DMA problem. > > Ben, how much traffic are you RX'ing on these virtual interfaces? I disabled my user-space application, and this script alone can reproduce the problem fairly quickly on my system. You will need to change some of those first variables. Just start it and wait a few minutes and watch the splats show on the console :) Note that I am not generating any traffic, but the wpa_supplicants are doing their thing of course... I'm using the kernel found here: http://dmz2.candelatech.com/git/gitweb.cgi?p=linux.wireless-testing.ct/.git;a=summary It's latest wireless-testing with some of my own patches, and some I've gathered from here an there. I doubt I'm causing this problem, but if you can't reproduce it with this script on your kernels, I can try with base wireless-testing or whatever you are using. #!/usr/bin/perl use strict; my $iw = "./local/sbin/iw"; my $ip = "./local/sbin/ip"; my $wpa_s = "./local/bin/wpa_supplicant"; my $ssid = "candela-n"; my $key = "wpadmz123"; my $phy = "phy0"; my $max = 32; my $i; my $bmac = "00:01:02:03:04:"; my $cmd; # Create stations for ($i = 0; $i<$max; $i++) { runCmd("$iw phy $phy interface add sta$i type station"); my $mc5 = "$i"; if (length($mc5) == 1) { $mc5 = "0$mc5"; # pad mac octet } my $mac = "$bmac$mc5"; runCmd("$ip link set sta$i address $mac"); runCmd("$iw dev sta$i set power_save off"); } # Bring them up with WPA for ($i = 0; $i<$max; $i++) { open(FD, ">sta$i" . "_wpa.conf") || die("Couldn't open file: $!\n"); print FD " ctrl_interface=/var/run/wpa_supplicant fast_reauth=1 #can_scan_one=1 network={ ssid=\"$ssid\" proto=WPA key_mgmt=WPA-PSK psk=\"$key\" pairwise=TKIP CCMP group=TKIP CCMP } "; runCmd("$wpa_s -B -i sta$i -c sta$i" . "_wpa.conf -P sta$i" . "_wpa.pid -t -f sta$i" . "_wpa.log"); } sub runCmd { my $cmd = shift; print "$cmd\n"; `$cmd`; } -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 18:39 ` Ben Greear @ 2010-10-07 18:42 ` Luis R. Rodriguez 2010-10-07 18:45 ` Ben Greear 2010-10-07 19:22 ` Ben Greear 0 siblings, 2 replies; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-07 18:42 UTC (permalink / raw) To: Ben Greear; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On Thu, Oct 7, 2010 at 11:39 AM, Ben Greear <greearb@candelatech.com> wrote: > On 10/07/2010 11:29 AM, Luis R. Rodriguez wrote: >> >> On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg >> <johannes@sipsolutions.net> wrote: >>> >>> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote: >>>> >>>> In case it helps, here is a dump of where the corrupted SKB was deleted. >>> >>> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU >>> debugging into the mix could help you figure out if it's a DMA problem. >> >> Ben, how much traffic are you RX'ing on these virtual interfaces? > > I disabled my user-space application, and this script alone can reproduce > the problem fairly quickly on my system. You will need to change some > of those first variables. Just start it and wait a few minutes and > watch the splats show on the console :) > > Note that I am not generating any traffic, but the wpa_supplicants are > doing their thing of course... > > I'm using the kernel found here: > http://dmz2.candelatech.com/git/gitweb.cgi?p=linux.wireless-testing.ct/.git;a=summary > > It's latest wireless-testing with some of my own patches, and some > I've gathered from here an there. I doubt I'm causing this problem, > but if you can't reproduce it with this script on your kernels, > I can try with base wireless-testing or whatever you are using. I'll run this now, but can you try a vanilla wireless-testing? I hear the latest wireless-testing is borked so maybe try (git reset --hard master-2010-09-29), its what I'm on. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 18:42 ` Luis R. Rodriguez @ 2010-10-07 18:45 ` Ben Greear 2010-10-07 19:14 ` Ben Greear 2010-10-07 19:22 ` Ben Greear 1 sibling, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-07 18:45 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/07/2010 11:42 AM, Luis R. Rodriguez wrote: > On Thu, Oct 7, 2010 at 11:39 AM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/07/2010 11:29 AM, Luis R. Rodriguez wrote: >>> >>> On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg >>> <johannes@sipsolutions.net> wrote: >>>> >>>> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote: >>>>> >>>>> In case it helps, here is a dump of where the corrupted SKB was deleted. >>>> >>>> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU >>>> debugging into the mix could help you figure out if it's a DMA problem. >>> >>> Ben, how much traffic are you RX'ing on these virtual interfaces? >> >> I disabled my user-space application, and this script alone can reproduce >> the problem fairly quickly on my system. You will need to change some >> of those first variables. Just start it and wait a few minutes and >> watch the splats show on the console :) >> >> Note that I am not generating any traffic, but the wpa_supplicants are >> doing their thing of course... >> >> I'm using the kernel found here: >> http://dmz2.candelatech.com/git/gitweb.cgi?p=linux.wireless-testing.ct/.git;a=summary >> >> It's latest wireless-testing with some of my own patches, and some >> I've gathered from here an there. I doubt I'm causing this problem, >> but if you can't reproduce it with this script on your kernels, >> I can try with base wireless-testing or whatever you are using. > > I'll run this now, but can you try a vanilla wireless-testing? I hear > the latest wireless-testing is borked so maybe try (git reset --hard > master-2010-09-29), its what I'm on. You are liable to hit a bunch of those crashes I've been reporting before you hit the DMA thing if you don't use latest (with Johanne's scan locking patch). I'm going to poke at IOMMU debugging and see what I find. I'll start a compile of vanilla wireless-testing + scan fix as well. Thanks, Ben > > Luis -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 18:45 ` Ben Greear @ 2010-10-07 19:14 ` Ben Greear 2010-10-07 19:17 ` Johannes Berg 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-07 19:14 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/07/2010 11:45 AM, Ben Greear wrote: > On 10/07/2010 11:42 AM, Luis R. Rodriguez wrote: >> On Thu, Oct 7, 2010 at 11:39 AM, Ben Greear<greearb@candelatech.com> >> wrote: >>> On 10/07/2010 11:29 AM, Luis R. Rodriguez wrote: >>>> >>>> On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg >>>> <johannes@sipsolutions.net> wrote: >>>>> >>>>> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote: >>>>>> >>>>>> In case it helps, here is a dump of where the corrupted SKB was >>>>>> deleted. >>>>> >>>>> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU >>>>> debugging into the mix could help you figure out if it's a DMA >>>>> problem. >>>> >>>> Ben, how much traffic are you RX'ing on these virtual interfaces? >>> >>> I disabled my user-space application, and this script alone can >>> reproduce >>> the problem fairly quickly on my system. You will need to change some >>> of those first variables. Just start it and wait a few minutes and >>> watch the splats show on the console :) >>> >>> Note that I am not generating any traffic, but the wpa_supplicants are >>> doing their thing of course... >>> >>> I'm using the kernel found here: >>> http://dmz2.candelatech.com/git/gitweb.cgi?p=linux.wireless-testing.ct/.git;a=summary >>> >>> >>> It's latest wireless-testing with some of my own patches, and some >>> I've gathered from here an there. I doubt I'm causing this problem, >>> but if you can't reproduce it with this script on your kernels, >>> I can try with base wireless-testing or whatever you are using. >> >> I'll run this now, but can you try a vanilla wireless-testing? I hear >> the latest wireless-testing is borked so maybe try (git reset --hard >> master-2010-09-29), its what I'm on. > > You are liable to hit a bunch of those crashes I've been reporting > before you hit the DMA thing if you don't use latest (with Johanne's scan > locking patch). > > I'm going to poke at IOMMU debugging and see what I find. > > I'll start a compile of vanilla wireless-testing + scan fix as well. Well, vanilla + scan patch locked pretty hard when I started the script. I was able to get sysrq to dump the locks, but it didn't seem to complete that and couldn't even dump more sysrq info after that. Might be something entirely different, of course, and no idea if this lock dump shows any real problem. Oct 7 12:08:43 localhost kernel: SysRq : Show Locks Held Oct 7 12:08:43 localhost kernel: Oct 7 12:08:43 localhost kernel: Showing all locks held in the system: Oct 7 12:08:43 localhost kernel: 3 locks held by kworker/0:0/4: Oct 7 12:08:43 localhost kernel: #0: (events){+.+.+.}, at: [<c0443cea>] process_one_work+0x145/0x295 Oct 7 12:08:43 localhost kernel: #1: ((linkwatch_work).work){+.+.+.}, at: [<c0443cea>] process_one_work+0x145/0x295 Oct 7 12:08:43 localhost kernel: #2: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11 Oct 7 12:08:43 localhost kernel: 3 locks held by kworker/u:1/38: Oct 7 12:08:43 localhost kernel: #0: ((wiphy_name(local->hw.wiphy))){+.+.+.}, at: [<c0443cea>] process_one_work+0x145/0x295 Oct 7 12:08:43 localhost kernel: #1: ((&sta->ampdu_mlme.work)){+.+...}, at: [<c0443cea>] process_one_work+0x145/0x295 Oct 7 12:08:43 localhost kernel: #2: (&sta->ampdu_mlme.mtx){+.+...}, at: [<f8887180>] ieee80211_ba_session_work+0x52/0xbe [mac80211] Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1584: Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1586: Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1589: Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1593: Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1596: Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed Oct 7 12:08:43 localhost kernel: 1 lock held by mingetty/1598: Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed Oct 7 12:08:43 localhost kernel: 1 lock held by bash/1683: Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed Oct 7 12:08:43 localhost kernel: 1 lock held by bash/1728: Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed Oct 7 12:08:43 localhost kernel: 1 lock held by bash/1752: Oct 7 12:08:43 localhost kernel: #0: (&tty->atomic_read_lock){+.+...}, at: [<c05f5e91>] n_tty_read+0x1d1/0x5ed Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2840: Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11 Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2842: Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11 Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2844: Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11 Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2846: Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11 Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2848: Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11 Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2850: Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11 Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2852: Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11 Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2854: Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11 Oct 7 12:08:43 localhost kernel: 1 lock held by wpa_supplicant/2856: Oct 7 12:08:43 localhost kernel: #0: (rtnl_mutex){+.+.+.}, at: [<c06da190>] rtnl_lock+0xf/0x11 OcSysRq : Show State Process wpa_supplicant (pid: 2904, ti=f3946000 task=f4124a60 task.ti=f3946000) Stack: Call Trace: Code: 73 18 8b 75 08 01 73 14 88 4c 02 1c 5b 5e 5d c3 55 89 e5 57 56 53 8b 5d 08 6b 72 04 3c ff 84 30 70 11 00 00 8b 79 10 6b 72 04 3c <8b> 7f 50 01 bc 30 SysRq : Show Locks Held Process wpa_supplicant (pid: 2904, ti=f3946000 task=f4124a60 task.ti=f3946000) Stack: Call Trace: Code: 73 18 8b 75 08 01 73 14 88 4c 02 1c 5b 5e 5d c3 55 89 e5 57 56 53 8b 5d 08 6b 72 04 3c ff 84 30 70 11 00 00 8b 79 10 6b 72 04 3c <8b> 7f 50 01 bc 30 Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 19:14 ` Ben Greear @ 2010-10-07 19:17 ` Johannes Berg 0 siblings, 0 replies; 84+ messages in thread From: Johannes Berg @ 2010-10-07 19:17 UTC (permalink / raw) To: Ben Greear; +Cc: Luis R. Rodriguez, linux-wireless@vger.kernel.org On Thu, 2010-10-07 at 12:14 -0700, Ben Greear wrote: > > I'll start a compile of vanilla wireless-testing + scan fix as well. > > Well, vanilla + scan patch locked pretty hard when I started the script. > > I was able to get sysrq to dump the locks, but it didn't seem to complete > that and couldn't even dump more sysrq info after that. > > Might be something entirely different, of course, and no idea if > this lock dump shows any real problem. Don't see any issues in this part of the dump -- heavy contention on the RTNL but that was expected with your usage. johannes ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 18:42 ` Luis R. Rodriguez 2010-10-07 18:45 ` Ben Greear @ 2010-10-07 19:22 ` Ben Greear 2010-10-07 19:27 ` Johannes Berg 1 sibling, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-07 19:22 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/07/2010 11:42 AM, Luis R. Rodriguez wrote: > On Thu, Oct 7, 2010 at 11:39 AM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/07/2010 11:29 AM, Luis R. Rodriguez wrote: >>> >>> On Thu, Oct 7, 2010 at 11:14 AM, Johannes Berg >>> <johannes@sipsolutions.net> wrote: >>>> >>>> On Thu, 2010-10-07 at 10:33 -0700, Ben Greear wrote: >>>>> >>>>> In case it helps, here is a dump of where the corrupted SKB was deleted. >>>> >>>> I wonder, do you have a machine with a decent IOMMU? Adding IOMMU >>>> debugging into the mix could help you figure out if it's a DMA problem. >>> >>> Ben, how much traffic are you RX'ing on these virtual interfaces? >> >> I disabled my user-space application, and this script alone can reproduce >> the problem fairly quickly on my system. You will need to change some >> of those first variables. Just start it and wait a few minutes and >> watch the splats show on the console :) >> >> Note that I am not generating any traffic, but the wpa_supplicants are >> doing their thing of course... >> >> I'm using the kernel found here: >> http://dmz2.candelatech.com/git/gitweb.cgi?p=linux.wireless-testing.ct/.git;a=summary >> >> It's latest wireless-testing with some of my own patches, and some >> I've gathered from here an there. I doubt I'm causing this problem, >> but if you can't reproduce it with this script on your kernels, >> I can try with base wireless-testing or whatever you are using. > > I'll run this now, but can you try a vanilla wireless-testing? I hear > the latest wireless-testing is borked so maybe try (git reset --hard > master-2010-09-29), its what I'm on. After reboot, and re-run of the script, I saw this in the logs, and shortly after, the SLUB poison warning dumped to screen. Maybe those DMA errors are serious? Also, I enabled CONFIG_DMA_API_DEBUG, but saw no indication it detected any problems. DDRCONF(NETDEV_UP): sta29: link is not ready ADDRCONF(NETDEV_UP): sta30: link is not ready ADDRCONF(NETDEV_UP): sta31: link is not ready sta0: authenticate with 00:14:d1:c6:d2:54 (try 1) sta0: authenticated ieee80211 phy0: device now idle ieee80211 phy0: device no longer idle - working sta0: associate with 00:14:d1:c6:d2:54 (try 1) sta18: authenticate with 00:14:d1:c6:d2:54 (try 1) sta0: associate with 00:14:d1:c6:d2:54 (try 2) sta18: authenticate with 00:14:d1:c6:d2:54 (try 2) sta0: associate with 00:14:d1:c6:d2:54 (try 3) sta18: authenticate with 00:14:d1:c6:d2:54 (try 3) sta0: association with 00:14:d1:c6:d2:54 timed out sta18: authentication with 00:14:d1:c6:d2:54 timed out ath: Failed to stop TX DMA in 100 msec after killing last frame ath: Failed to stop TX DMA. Resetting hardware! ieee80211 phy0: device now idle ieee80211 phy0: device no longer idle - scanning ieee80211 phy0: device now idle ieee80211 phy0: device no longer idle - working sta1: authenticate with 00:14:d1:c6:d2:54 (try 1) sta1: authenticated sta1: associate with 00:14:d1:c6:d2:54 (try 1) sta1: RX AssocResp from 00:14:d1:c6:d2:54 (capab=0x431 status=0 aid=18) sta1: associated ieee80211 phy0: Allocated STA 00:14:d1:c6:d2:54 ieee80211 phy0: Inserted STA 00:14:d1:c6:d2:54 ieee80211 phy0: WMM queue=2 aci=0 acm=0 aifs=3 cWmin=15 cWmax=1023 txop=0 uapsd=0 ieee80211 phy0: WMM queue=3 aci=1 acm=0 aifs=7 cWmin=15 cWmax=1023 txop=0 uapsd=0 ieee80211 phy0: WMM queue=1 aci=2 acm=0 aifs=2 cWmin=7 cWmax=15 txop=94 uapsd=0 ieee80211 phy0: WMM queue=0 aci=3 acm=0 aifs=2 cWmin=3 cWmax=7 txop=47 uapsd=0 Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 19:22 ` Ben Greear @ 2010-10-07 19:27 ` Johannes Berg 2010-10-07 21:31 ` Luis R. Rodriguez 2010-10-08 0:42 ` Bruno Randolf 0 siblings, 2 replies; 84+ messages in thread From: Johannes Berg @ 2010-10-07 19:27 UTC (permalink / raw) To: Ben Greear; +Cc: Luis R. Rodriguez, linux-wireless@vger.kernel.org On Thu, 2010-10-07 at 12:22 -0700, Ben Greear wrote: > After reboot, and re-run of the script, > I saw this in the logs, and shortly after, > the SLUB poison warning dumped to screen. > > Maybe those DMA errors are serious? > ath: Failed to stop TX DMA in 100 msec after killing last frame > ath: Failed to stop TX DMA. Resetting hardware! That's TX DMA, it can hardly result in invalid memory writes like the ones you've been seeing. I'm still convinced something is wrong with ath9k RX DMA, as you've seen the contents of frames written to already freed memory regions. Since I don't know anything about ath9k, you should probably not rely on me though :-) johannes ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 19:27 ` Johannes Berg @ 2010-10-07 21:31 ` Luis R. Rodriguez 2010-10-07 21:36 ` Luis R. Rodriguez 2010-10-07 21:52 ` Ben Greear 2010-10-08 0:42 ` Bruno Randolf 1 sibling, 2 replies; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-07 21:31 UTC (permalink / raw) To: Johannes Berg; +Cc: Ben Greear, linux-wireless@vger.kernel.org On Thu, Oct 7, 2010 at 12:27 PM, Johannes Berg <johannes@sipsolutions.net> wrote: > On Thu, 2010-10-07 at 12:22 -0700, Ben Greear wrote: > >> After reboot, and re-run of the script, >> I saw this in the logs, and shortly after, >> the SLUB poison warning dumped to screen. >> >> Maybe those DMA errors are serious? > >> ath: Failed to stop TX DMA in 100 msec after killing last frame >> ath: Failed to stop TX DMA. Resetting hardware! > > That's TX DMA, it can hardly result in invalid memory writes like the > ones you've been seeing. > > I'm still convinced something is wrong with ath9k RX DMA, as you've seen > the contents of frames written to already freed memory regions. Since I > don't know anything about ath9k, you should probably not rely on me > though :-) I'm on this now. Lets play. I had to remove /lib/udev/rules.d/75-persistent-net-generator.rules to avoid Ubuntu trying to remember the device names and it creating stax_rename names. I just ran your script with some modifications. You can find it here: http://www.kernel.org/pub/linux/kernel/people/mcgrof/scripts/poo.pl I then ran: for i in $(seq 0 31) ; do sudo dhclient seq$i; done It took about 10 minutes to get IP addresses for all interfaces but it got there eventually. Odd enough I was unable to ping the AP from any interface though. Not sure what that was about. But I got no oops, no slub dump. All I got was some more delba warnings which seems to indicate we haven't caught all the cases for its use: [ 3622.660344] addBA response timer expired on tid 0 [ 3622.660373] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0 [ 3622.680133] addBA response timer expired on tid 0 [ 3622.687196] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0 [ 3623.110077] addBA response timer expired on tid 0 [ 3623.110123] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0 [ 3628.935895] sta10: authenticate with 68:7f:74:3b:b1:10 (try 1) [ 3628.937194] switched off addBA timer for tid 0 [ 3628.937196] Aggregation is on for tid 0 [ 3628.937239] Stopping Tx BA session for 68:7f:74:3b:b1:0f tid 0 [ 3628.937243] ------------[ cut here ]------------ [ 3628.937263] WARNING: at include/net/mac80211.h:2694 rate_control_send_low+0xd3/0x140 [mac80211]() [ 3628.937265] Hardware name: 6460DWU [ 3628.937266] Modules linked in: binfmt_misc ppdev snd_hda_codec_analog rfcomm sco bridge joydev stp bnep l2cap nouveau ath9k snd_hda_intel mac80211 snd_hda_codec snd_hwdep snd_pcm ttm btusb ath9k_common thinkpad_acpi ath9k_hw bluetooth drm_kms_helper snd_seq_midi snd_rawmidi pcmcia snd_seq_midi_event drm snd_seq ath snd_timer snd_seq_device tpm_tis i2c_algo_bit cfg80211 snd nvram tpm tpm_bios yenta_socket pcmcia_rsrc video psmouse output pcmcia_core serio_raw soundcore snd_page_alloc intel_agp lp parport ohci1394 e1000e ieee1394 ahci libahci [ 3628.937307] Pid: 49, comm: kworker/u:3 Tainted: G W 2.6.36-rc6-wl+ #263 [ 3628.937310] Call Trace: [ 3628.937317] [<ffffffff8105ffcf>] warn_slowpath_common+0x7f/0xc0 [ 3628.937320] [<ffffffff8106002a>] warn_slowpath_null+0x1a/0x20 [ 3628.937329] [<ffffffffa032f603>] rate_control_send_low+0xd3/0x140 [mac80211] [ 3628.937336] [<ffffffffa038bfd8>] ath_get_rate+0x48/0x570 [ath9k] [ 3628.937340] [<ffffffff812b9f39>] ? put_dec+0x59/0x60 [ 3628.937349] [<ffffffffa032f42e>] rate_control_get_rate+0x8e/0x190 [mac80211] [ 3628.937360] [<ffffffffa0339928>] ieee80211_tx_h_rate_ctrl+0x1a8/0x4e0 [mac80211] [ 3628.937370] [<ffffffffa033a000>] invoke_tx_handlers+0x100/0x140 [mac80211] [ 3628.937379] [<ffffffffa033a0c5>] ieee80211_tx+0x85/0x240 [mac80211] [ 3628.937384] [<ffffffff8147b890>] ? skb_release_data+0xd0/0xe0 [ 3628.937386] [<ffffffff8147d72f>] ? pskb_expand_head+0x10f/0x1a0 [ 3628.937397] [<ffffffffa033a336>] ieee80211_xmit+0xb6/0x1d0 [mac80211] [ 3628.937399] [<ffffffff8147b9d3>] ? __alloc_skb+0x83/0x170 [ 3628.937409] [<ffffffffa033a4a4>] ieee80211_tx_skb+0x54/0x70 [mac80211] [ 3628.937418] [<ffffffffa03230ad>] ieee80211_send_delba+0x11d/0x190 [mac80211] [ 3628.937427] [<ffffffffa0323a18>] ieee80211_stop_tx_ba_cb+0x1b8/0x240 [mac80211] [ 3628.937431] [<ffffffff81036c89>] ? default_spin_lock_flags+0x9/0x10 [ 3628.937440] [<ffffffffa032e031>] ieee80211_iface_work+0x271/0x340 [mac80211] [ 3628.937450] [<ffffffffa032ddc0>] ? ieee80211_iface_work+0x0/0x340 [mac80211] [ 3628.937453] [<ffffffff8107a203>] process_one_work+0x123/0x440 [ 3628.937457] [<ffffffff8107c750>] worker_thread+0x170/0x400 [ 3628.937460] [<ffffffff8107c5e0>] ? worker_thread+0x0/0x400 [ 3628.937463] [<ffffffff81080b76>] kthread+0x96/0xa0 [ 3628.937466] [<ffffffff8100bea4>] kernel_thread_helper+0x4/0x10 [ 3628.937469] [<ffffffff81080ae0>] ? kthread+0x0/0xa0 [ 3628.937472] [<ffffffff8100bea0>] ? kernel_thread_helper+0x0/0x10 [ 3628.937474] ---[ end trace 9dd0d025ccb9b75c ]--- [ 3628.937980] switched off addBA timer for tid 0 [ 3628.937982] Aggregation is on for tid 0 But other than this I got nothing. I left the box sit there for about 1 hour and came back and it was still going with no issues. Mind you, I can't ping but that seems like another issue. You can find my logs here: http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/ Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 21:31 ` Luis R. Rodriguez @ 2010-10-07 21:36 ` Luis R. Rodriguez 2010-10-07 21:59 ` Luis R. Rodriguez 2010-10-07 21:52 ` Ben Greear 1 sibling, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-07 21:36 UTC (permalink / raw) To: Johannes Berg; +Cc: Ben Greear, linux-wireless@vger.kernel.org On Thu, Oct 7, 2010 at 2:31 PM, Luis R. Rodriguez <mcgrof@gmail.com> wrote: > On Thu, Oct 7, 2010 at 12:27 PM, Johannes Berg > <johannes@sipsolutions.net> wrote: >> On Thu, 2010-10-07 at 12:22 -0700, Ben Greear wrote: >> >>> After reboot, and re-run of the script, >>> I saw this in the logs, and shortly after, >>> the SLUB poison warning dumped to screen. >>> >>> Maybe those DMA errors are serious? >> >>> ath: Failed to stop TX DMA in 100 msec after killing last frame >>> ath: Failed to stop TX DMA. Resetting hardware! >> >> That's TX DMA, it can hardly result in invalid memory writes like the >> ones you've been seeing. >> >> I'm still convinced something is wrong with ath9k RX DMA, as you've seen >> the contents of frames written to already freed memory regions. Since I >> don't know anything about ath9k, you should probably not rely on me >> though :-) > > I'm on this now. Lets play. > > I had to remove /lib/udev/rules.d/75-persistent-net-generator.rules > to avoid Ubuntu trying to remember the device names and it creating > stax_rename names. > I just ran your script with some modifications. You can find it here: > > http://www.kernel.org/pub/linux/kernel/people/mcgrof/scripts/poo.pl > > I then ran: > > for i in $(seq 0 31) ; do sudo dhclient seq$i; done > > It took about 10 minutes to get IP addresses for all interfaces but it > got there eventually. Odd enough I was unable to ping the AP from any > interface though. Not sure what that was about. But I got no oops, no > slub dump. All I got was some more delba warnings which seems to > indicate we haven't caught all the cases for its use: > > [ 3622.660344] addBA response timer expired on tid 0 > [ 3622.660373] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0 > [ 3622.680133] addBA response timer expired on tid 0 > [ 3622.687196] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0 > [ 3623.110077] addBA response timer expired on tid 0 > [ 3623.110123] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0 > [ 3628.935895] sta10: authenticate with 68:7f:74:3b:b1:10 (try 1) > [ 3628.937194] switched off addBA timer for tid 0 > [ 3628.937196] Aggregation is on for tid 0 > [ 3628.937239] Stopping Tx BA session for 68:7f:74:3b:b1:0f tid 0 > [ 3628.937243] ------------[ cut here ]------------ > [ 3628.937263] WARNING: at include/net/mac80211.h:2694 > rate_control_send_low+0xd3/0x140 [mac80211]() > [ 3628.937265] Hardware name: 6460DWU > [ 3628.937266] Modules linked in: binfmt_misc ppdev > snd_hda_codec_analog rfcomm sco bridge joydev stp bnep l2cap nouveau > ath9k snd_hda_intel mac80211 snd_hda_codec snd_hwdep snd_pcm ttm btusb > ath9k_common thinkpad_acpi ath9k_hw bluetooth drm_kms_helper > snd_seq_midi snd_rawmidi pcmcia snd_seq_midi_event drm snd_seq ath > snd_timer snd_seq_device tpm_tis i2c_algo_bit cfg80211 snd nvram tpm > tpm_bios yenta_socket pcmcia_rsrc video psmouse output pcmcia_core > serio_raw soundcore snd_page_alloc intel_agp lp parport ohci1394 > e1000e ieee1394 ahci libahci > [ 3628.937307] Pid: 49, comm: kworker/u:3 Tainted: G W > 2.6.36-rc6-wl+ #263 > [ 3628.937310] Call Trace: > [ 3628.937317] [<ffffffff8105ffcf>] warn_slowpath_common+0x7f/0xc0 > [ 3628.937320] [<ffffffff8106002a>] warn_slowpath_null+0x1a/0x20 > [ 3628.937329] [<ffffffffa032f603>] rate_control_send_low+0xd3/0x140 [mac80211] > [ 3628.937336] [<ffffffffa038bfd8>] ath_get_rate+0x48/0x570 [ath9k] > [ 3628.937340] [<ffffffff812b9f39>] ? put_dec+0x59/0x60 > [ 3628.937349] [<ffffffffa032f42e>] rate_control_get_rate+0x8e/0x190 [mac80211] > [ 3628.937360] [<ffffffffa0339928>] > ieee80211_tx_h_rate_ctrl+0x1a8/0x4e0 [mac80211] > [ 3628.937370] [<ffffffffa033a000>] invoke_tx_handlers+0x100/0x140 [mac80211] > [ 3628.937379] [<ffffffffa033a0c5>] ieee80211_tx+0x85/0x240 [mac80211] > [ 3628.937384] [<ffffffff8147b890>] ? skb_release_data+0xd0/0xe0 > [ 3628.937386] [<ffffffff8147d72f>] ? pskb_expand_head+0x10f/0x1a0 > [ 3628.937397] [<ffffffffa033a336>] ieee80211_xmit+0xb6/0x1d0 [mac80211] > [ 3628.937399] [<ffffffff8147b9d3>] ? __alloc_skb+0x83/0x170 > [ 3628.937409] [<ffffffffa033a4a4>] ieee80211_tx_skb+0x54/0x70 [mac80211] > [ 3628.937418] [<ffffffffa03230ad>] ieee80211_send_delba+0x11d/0x190 [mac80211] > [ 3628.937427] [<ffffffffa0323a18>] > ieee80211_stop_tx_ba_cb+0x1b8/0x240 [mac80211] > [ 3628.937431] [<ffffffff81036c89>] ? default_spin_lock_flags+0x9/0x10 > [ 3628.937440] [<ffffffffa032e031>] ieee80211_iface_work+0x271/0x340 [mac80211] > [ 3628.937450] [<ffffffffa032ddc0>] ? ieee80211_iface_work+0x0/0x340 [mac80211] > [ 3628.937453] [<ffffffff8107a203>] process_one_work+0x123/0x440 > [ 3628.937457] [<ffffffff8107c750>] worker_thread+0x170/0x400 > [ 3628.937460] [<ffffffff8107c5e0>] ? worker_thread+0x0/0x400 > [ 3628.937463] [<ffffffff81080b76>] kthread+0x96/0xa0 > [ 3628.937466] [<ffffffff8100bea4>] kernel_thread_helper+0x4/0x10 > [ 3628.937469] [<ffffffff81080ae0>] ? kthread+0x0/0xa0 > [ 3628.937472] [<ffffffff8100bea0>] ? kernel_thread_helper+0x0/0x10 > [ 3628.937474] ---[ end trace 9dd0d025ccb9b75c ]--- > [ 3628.937980] switched off addBA timer for tid 0 > [ 3628.937982] Aggregation is on for tid 0 > > But other than this I got nothing. I left the box sit there for about > 1 hour and came back and it was still going with no issues. Mind you, > I can't ping but that seems like another issue. > > You can find my logs here: > > http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/ Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 21:36 ` Luis R. Rodriguez @ 2010-10-07 21:59 ` Luis R. Rodriguez 2010-10-11 20:51 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-07 21:59 UTC (permalink / raw) To: Johannes Berg; +Cc: Ben Greear, linux-wireless@vger.kernel.org On Thu, Oct 7, 2010 at 2:36 PM, Luis R. Rodriguez <mcgrof@gmail.com> wrote: > On Thu, Oct 7, 2010 at 2:31 PM, Luis R. Rodriguez <mcgrof@gmail.com> wrote: >> On Thu, Oct 7, 2010 at 12:27 PM, Johannes Berg >> <johannes@sipsolutions.net> wrote: >>> On Thu, 2010-10-07 at 12:22 -0700, Ben Greear wrote: >>> >>>> After reboot, and re-run of the script, >>>> I saw this in the logs, and shortly after, >>>> the SLUB poison warning dumped to screen. >>>> >>>> Maybe those DMA errors are serious? >>> >>>> ath: Failed to stop TX DMA in 100 msec after killing last frame >>>> ath: Failed to stop TX DMA. Resetting hardware! >>> >>> That's TX DMA, it can hardly result in invalid memory writes like the >>> ones you've been seeing. >>> >>> I'm still convinced something is wrong with ath9k RX DMA, as you've seen >>> the contents of frames written to already freed memory regions. Since I >>> don't know anything about ath9k, you should probably not rely on me >>> though :-) >> >> I'm on this now. Lets play. >> >> I had to remove /lib/udev/rules.d/75-persistent-net-generator.rules >> to avoid Ubuntu trying to remember the device names and it creating >> stax_rename names. >> I just ran your script with some modifications. You can find it here: >> >> http://www.kernel.org/pub/linux/kernel/people/mcgrof/scripts/poo.pl >> >> I then ran: >> >> for i in $(seq 0 31) ; do sudo dhclient seq$i; done >> >> It took about 10 minutes to get IP addresses for all interfaces but it >> got there eventually. Odd enough I was unable to ping the AP from any >> interface though. Not sure what that was about. But I got no oops, no >> slub dump. All I got was some more delba warnings which seems to >> indicate we haven't caught all the cases for its use: >> >> [ 3622.660344] addBA response timer expired on tid 0 >> [ 3622.660373] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0 >> [ 3622.680133] addBA response timer expired on tid 0 >> [ 3622.687196] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0 >> [ 3623.110077] addBA response timer expired on tid 0 >> [ 3623.110123] Tx BA session stop requested for 68:7f:74:3b:b1:0f tid 0 >> [ 3628.935895] sta10: authenticate with 68:7f:74:3b:b1:10 (try 1) >> [ 3628.937194] switched off addBA timer for tid 0 >> [ 3628.937196] Aggregation is on for tid 0 >> [ 3628.937239] Stopping Tx BA session for 68:7f:74:3b:b1:0f tid 0 >> [ 3628.937243] ------------[ cut here ]------------ >> [ 3628.937263] WARNING: at include/net/mac80211.h:2694 >> rate_control_send_low+0xd3/0x140 [mac80211]() >> [ 3628.937265] Hardware name: 6460DWU >> [ 3628.937266] Modules linked in: binfmt_misc ppdev >> snd_hda_codec_analog rfcomm sco bridge joydev stp bnep l2cap nouveau >> ath9k snd_hda_intel mac80211 snd_hda_codec snd_hwdep snd_pcm ttm btusb >> ath9k_common thinkpad_acpi ath9k_hw bluetooth drm_kms_helper >> snd_seq_midi snd_rawmidi pcmcia snd_seq_midi_event drm snd_seq ath >> snd_timer snd_seq_device tpm_tis i2c_algo_bit cfg80211 snd nvram tpm >> tpm_bios yenta_socket pcmcia_rsrc video psmouse output pcmcia_core >> serio_raw soundcore snd_page_alloc intel_agp lp parport ohci1394 >> e1000e ieee1394 ahci libahci >> [ 3628.937307] Pid: 49, comm: kworker/u:3 Tainted: G W >> 2.6.36-rc6-wl+ #263 >> [ 3628.937310] Call Trace: >> [ 3628.937317] [<ffffffff8105ffcf>] warn_slowpath_common+0x7f/0xc0 >> [ 3628.937320] [<ffffffff8106002a>] warn_slowpath_null+0x1a/0x20 >> [ 3628.937329] [<ffffffffa032f603>] rate_control_send_low+0xd3/0x140 [mac80211] >> [ 3628.937336] [<ffffffffa038bfd8>] ath_get_rate+0x48/0x570 [ath9k] >> [ 3628.937340] [<ffffffff812b9f39>] ? put_dec+0x59/0x60 >> [ 3628.937349] [<ffffffffa032f42e>] rate_control_get_rate+0x8e/0x190 [mac80211] >> [ 3628.937360] [<ffffffffa0339928>] >> ieee80211_tx_h_rate_ctrl+0x1a8/0x4e0 [mac80211] >> [ 3628.937370] [<ffffffffa033a000>] invoke_tx_handlers+0x100/0x140 [mac80211] >> [ 3628.937379] [<ffffffffa033a0c5>] ieee80211_tx+0x85/0x240 [mac80211] >> [ 3628.937384] [<ffffffff8147b890>] ? skb_release_data+0xd0/0xe0 >> [ 3628.937386] [<ffffffff8147d72f>] ? pskb_expand_head+0x10f/0x1a0 >> [ 3628.937397] [<ffffffffa033a336>] ieee80211_xmit+0xb6/0x1d0 [mac80211] >> [ 3628.937399] [<ffffffff8147b9d3>] ? __alloc_skb+0x83/0x170 >> [ 3628.937409] [<ffffffffa033a4a4>] ieee80211_tx_skb+0x54/0x70 [mac80211] >> [ 3628.937418] [<ffffffffa03230ad>] ieee80211_send_delba+0x11d/0x190 [mac80211] >> [ 3628.937427] [<ffffffffa0323a18>] >> ieee80211_stop_tx_ba_cb+0x1b8/0x240 [mac80211] >> [ 3628.937431] [<ffffffff81036c89>] ? default_spin_lock_flags+0x9/0x10 >> [ 3628.937440] [<ffffffffa032e031>] ieee80211_iface_work+0x271/0x340 [mac80211] >> [ 3628.937450] [<ffffffffa032ddc0>] ? ieee80211_iface_work+0x0/0x340 [mac80211] >> [ 3628.937453] [<ffffffff8107a203>] process_one_work+0x123/0x440 >> [ 3628.937457] [<ffffffff8107c750>] worker_thread+0x170/0x400 >> [ 3628.937460] [<ffffffff8107c5e0>] ? worker_thread+0x0/0x400 >> [ 3628.937463] [<ffffffff81080b76>] kthread+0x96/0xa0 >> [ 3628.937466] [<ffffffff8100bea4>] kernel_thread_helper+0x4/0x10 >> [ 3628.937469] [<ffffffff81080ae0>] ? kthread+0x0/0xa0 >> [ 3628.937472] [<ffffffff8100bea0>] ? kernel_thread_helper+0x0/0x10 >> [ 3628.937474] ---[ end trace 9dd0d025ccb9b75c ]--- >> [ 3628.937980] switched off addBA timer for tid 0 >> [ 3628.937982] Aggregation is on for tid 0 >> >> But other than this I got nothing. I left the box sit there for about >> 1 hour and came back and it was still going with no issues. Mind you, >> I can't ping but that seems like another issue. >> >> You can find my logs here: >> >> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/ > > Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that. Yay I can reproduce now. I'll be back, going to dig now. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 21:59 ` Luis R. Rodriguez @ 2010-10-11 20:51 ` Ben Greear 2010-10-12 1:03 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-11 20:51 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/07/2010 02:59 PM, Luis R. Rodriguez wrote: > On Thu, Oct 7, 2010 at 2:36 PM, Luis R. Rodriguez<mcgrof@gmail.com> wrote: >>> But other than this I got nothing. I left the box sit there for about >>> 1 hour and came back and it was still going with no issues. Mind you, >>> I can't ping but that seems like another issue. >>> >>> You can find my logs here: >>> >>> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/ >> >> Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that. > > Yay I can reproduce now. I'll be back, going to dig now. Any luck tracking this down? Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-11 20:51 ` Ben Greear @ 2010-10-12 1:03 ` Luis R. Rodriguez 2010-10-12 3:27 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-12 1:03 UTC (permalink / raw) To: Ben Greear; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On Mon, Oct 11, 2010 at 1:51 PM, Ben Greear <greearb@candelatech.com> wrote: > On 10/07/2010 02:59 PM, Luis R. Rodriguez wrote: >> >> On Thu, Oct 7, 2010 at 2:36 PM, Luis R. Rodriguez<mcgrof@gmail.com> >> wrote: > >>>> But other than this I got nothing. I left the box sit there for about >>>> 1 hour and came back and it was still going with no issues. Mind you, >>>> I can't ping but that seems like another issue. >>>> >>>> You can find my logs here: >>>> >>>> >>>> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/ >>> >>> Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that. >> >> Yay I can reproduce now. I'll be back, going to dig now. > > Any luck tracking this down? No, today for example I just finished reading e-mail and its already 6pm PST... But Friday I did get do do a lot of work and testing on this. The only pattern I see so far is that skb_copy() is used on the poison all the time. I am not sure if its because skb_copy() happens to run the poison check or what. I'll work on this tomorrow. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-12 1:03 ` Luis R. Rodriguez @ 2010-10-12 3:27 ` Ben Greear 2010-10-12 6:10 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-12 3:27 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/11/2010 06:03 PM, Luis R. Rodriguez wrote: > On Mon, Oct 11, 2010 at 1:51 PM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/07/2010 02:59 PM, Luis R. Rodriguez wrote: >>> >>> On Thu, Oct 7, 2010 at 2:36 PM, Luis R. Rodriguez<mcgrof@gmail.com> >>> wrote: >> >>>>> But other than this I got nothing. I left the box sit there for about >>>>> 1 hour and came back and it was still going with no issues. Mind you, >>>>> I can't ping but that seems like another issue. >>>>> >>>>> You can find my logs here: >>>>> >>>>> >>>>> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/ >>>> >>>> Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that. >>> >>> Yay I can reproduce now. I'll be back, going to dig now. >> >> Any luck tracking this down? > > No, today for example I just finished reading e-mail and its already > 6pm PST... But Friday I did get do do a lot of work and testing on > this. The only pattern I see so far is that skb_copy() is used on the > poison all the time. I am not sure if its because skb_copy() happens > to run the poison check or what. I'll work on this tomorrow. I know how that goes. Do you happen to have any magic tools that could be instrumented to show when DMA was happening in the chip, and to see if it somehow happens to dma to something after it is supposedly un-mapped? Another thing I was thinking about: Maybe the queue of skbs and dma addresses in ath9k is getting corrupted by multiple VIFs trying to write at once? Maybe some locking is needed in the xmit path? Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-12 3:27 ` Ben Greear @ 2010-10-12 6:10 ` Luis R. Rodriguez 2010-10-12 18:35 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-12 6:10 UTC (permalink / raw) To: Ben Greear; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear <greearb@candelatech.com> wrote: > On 10/11/2010 06:03 PM, Luis R. Rodriguez wrote: >> >> On Mon, Oct 11, 2010 at 1:51 PM, Ben Greear<greearb@candelatech.com> >> wrote: >>> >>> On 10/07/2010 02:59 PM, Luis R. Rodriguez wrote: >>>> >>>> On Thu, Oct 7, 2010 at 2:36 PM, Luis R. Rodriguez<mcgrof@gmail.com> >>>> wrote: >>> >>>>>> But other than this I got nothing. I left the box sit there for about >>>>>> 1 hour and came back and it was still going with no issues. Mind you, >>>>>> I can't ping but that seems like another issue. >>>>>> >>>>>> You can find my logs here: >>>>>> >>>>>> >>>>>> >>>>>> http://www.kernel.org/pub/linux/kernel/people/mcgrof/logs/2010/10-07-stress-sta-01/ >>>>> >>>>> Doh, I did not have CONFIG_SLUB_DEBUG_ON=y so building now with that. >>>> >>>> Yay I can reproduce now. I'll be back, going to dig now. >>> >>> Any luck tracking this down? >> >> No, today for example I just finished reading e-mail and its already >> 6pm PST... But Friday I did get do do a lot of work and testing on >> this. The only pattern I see so far is that skb_copy() is used on the >> poison all the time. I am not sure if its because skb_copy() happens >> to run the poison check or what. I'll work on this tomorrow. > > I know how that goes. > > Do you happen to have any magic tools that could be instrumented to show > when DMA was happening in the chip, and to see if it somehow happens to dma > to something after it is supposedly un-mapped? Um, not sure, I'd have to dig. But I was looking at this as an idea to borrow to test if its a DMA issue: https://patchwork.kernel.org/patch/22127/ However right now I'm thinking this is simply a free and then a race to try to use the free'd buffer. > Another thing I was thinking about: Maybe the queue of skbs and dma > addresses > in ath9k is getting corrupted by multiple VIFs trying to write at once? > Maybe > some locking is needed in the xmit path? That was my second hunch. My first shot was to use spin_lock_irqsave() over the the uses of the rxbuf list and that seemed to help but I still managed to get a poison eventually. My next item to check for is of the permissibility of creating too much pressure to the point we end up looping over the rxbuf list and race against mac80211 free'ing a buffer. Will test that tomorrow if nothing else comes up creeping my priority queue. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-12 6:10 ` Luis R. Rodriguez @ 2010-10-12 18:35 ` Ben Greear 2010-10-12 18:40 ` Luis R. Rodriguez 2010-10-13 5:31 ` Vasanthakumar Thiagarajan 0 siblings, 2 replies; 84+ messages in thread From: Ben Greear @ 2010-10-12 18:35 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: > On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> wrote: >> Another thing I was thinking about: Maybe the queue of skbs and dma >> addresses >> in ath9k is getting corrupted by multiple VIFs trying to write at once? >> Maybe >> some locking is needed in the xmit path? > > That was my second hunch. My first shot was to use spin_lock_irqsave() > over the the uses of the rxbuf list and that seemed to help but I > still managed to get a poison eventually. My next item to check for is > of the permissibility of creating too much pressure to the point we > end up looping over the rxbuf list and race against mac80211 free'ing > a buffer. Will test that tomorrow if nothing else comes up creeping my > priority queue. This code looks weird to me. One of the paprd branches deletes the skb, the other doesn't appear to. Neither null out bf->bf_mpdu, which would appear to leave a dangling pointer in at least the dev_kfree_skb_any() branch. ath_tx_complete frees it's skb in all cases, so another bf->bf_mpdu dangling pointer issue. Maybe at the least we should null out bf->bf_mpdu when skb is consumed? static void ath_tx_complete_buf(struct ath_softc *sc, struct ath_buf *bf, struct ath_txq *txq, struct list_head *bf_q, struct ath_tx_status *ts, int txok, int sendbar) { struct sk_buff *skb = bf->bf_mpdu; unsigned long flags; int tx_flags = 0; if (sendbar) tx_flags = ATH_TX_BAR; if (!txok) { tx_flags |= ATH_TX_ERROR; if (bf_isxretried(bf)) tx_flags |= ATH_TX_XRETRY; } dma_unmap_single(sc->dev, bf->bf_dmacontext, skb->len, DMA_TO_DEVICE); if (bf->bf_state.bfs_paprd) { if (time_after(jiffies, bf->bf_state.bfs_paprd_timestamp + msecs_to_jiffies(ATH_PAPRD_TIMEOUT))) dev_kfree_skb_any(skb); else complete(&sc->paprd_complete); } else { ath_tx_complete(sc, skb, bf->aphy, tx_flags); ath_debug_stat_tx(sc, txq, bf, ts); } > > Luis -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-12 18:35 ` Ben Greear @ 2010-10-12 18:40 ` Luis R. Rodriguez 2010-10-12 18:43 ` Ben Greear 2010-10-13 17:12 ` Ben Greear 2010-10-13 5:31 ` Vasanthakumar Thiagarajan 1 sibling, 2 replies; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-12 18:40 UTC (permalink / raw) To: Ben Greear; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear <greearb@candelatech.com> wrote: > On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: >> >> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> >> wrote: > >>> Another thing I was thinking about: Maybe the queue of skbs and dma >>> addresses >>> in ath9k is getting corrupted by multiple VIFs trying to write at once? >>> Maybe >>> some locking is needed in the xmit path? >> >> That was my second hunch. My first shot was to use spin_lock_irqsave() >> over the the uses of the rxbuf list and that seemed to help but I >> still managed to get a poison eventually. My next item to check for is >> of the permissibility of creating too much pressure to the point we >> end up looping over the rxbuf list and race against mac80211 free'ing >> a buffer. Will test that tomorrow if nothing else comes up creeping my >> priority queue. > > This code looks weird to me. One of the paprd branches > deletes the skb, the other doesn't appear to. Neither > null out bf->bf_mpdu, which would appear to leave a dangling > pointer in at least the dev_kfree_skb_any() branch. > > ath_tx_complete frees it's skb in all cases, so another > bf->bf_mpdu dangling pointer issue. > > Maybe at the least we should null out bf->bf_mpdu when > skb is consumed? You're reading my mind, that was what I was going to test today. Still doing e-mail sweep though. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-12 18:40 ` Luis R. Rodriguez @ 2010-10-12 18:43 ` Ben Greear 2010-10-12 19:51 ` Ben Greear 2010-10-13 17:12 ` Ben Greear 1 sibling, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-12 18:43 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote: > On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: >>> >>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> >>> wrote: >> >>>> Another thing I was thinking about: Maybe the queue of skbs and dma >>>> addresses >>>> in ath9k is getting corrupted by multiple VIFs trying to write at once? >>>> Maybe >>>> some locking is needed in the xmit path? >>> >>> That was my second hunch. My first shot was to use spin_lock_irqsave() >>> over the the uses of the rxbuf list and that seemed to help but I >>> still managed to get a poison eventually. My next item to check for is >>> of the permissibility of creating too much pressure to the point we >>> end up looping over the rxbuf list and race against mac80211 free'ing >>> a buffer. Will test that tomorrow if nothing else comes up creeping my >>> priority queue. >> >> This code looks weird to me. One of the paprd branches >> deletes the skb, the other doesn't appear to. Neither >> null out bf->bf_mpdu, which would appear to leave a dangling >> pointer in at least the dev_kfree_skb_any() branch. >> >> ath_tx_complete frees it's skb in all cases, so another >> bf->bf_mpdu dangling pointer issue. >> >> Maybe at the least we should null out bf->bf_mpdu when >> skb is consumed? > > You're reading my mind, that was what I was going to test today. Still > doing e-mail sweep though. I'm trying to put together a patch now..but the paprd branch makes no sense at all. Shouldn't it also free the skb in the complete(&sc->paprd_complete) branch? I don't see anything that uses the skb, and nothing that frees it. if (bf->bf_state.bfs_paprd) { if (time_after(jiffies, bf->bf_state.bfs_paprd_timestamp + msecs_to_jiffies(ATH_PAPRD_TIMEOUT))) dev_kfree_skb_any(skb); else complete(&sc->paprd_complete); Thanks, Ben > > Luis -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-12 18:43 ` Ben Greear @ 2010-10-12 19:51 ` Ben Greear 0 siblings, 0 replies; 84+ messages in thread From: Ben Greear @ 2010-10-12 19:51 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/12/2010 11:43 AM, Ben Greear wrote: > On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote: >> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<greearb@candelatech.com> >> wrote: >>> This code looks weird to me. One of the paprd branches >>> deletes the skb, the other doesn't appear to. Neither >>> null out bf->bf_mpdu, which would appear to leave a dangling >>> pointer in at least the dev_kfree_skb_any() branch. >>> >>> ath_tx_complete frees it's skb in all cases, so another >>> bf->bf_mpdu dangling pointer issue. >>> >>> Maybe at the least we should null out bf->bf_mpdu when >>> skb is consumed? >> >> You're reading my mind, that was what I was going to test today. Still >> doing e-mail sweep though. > > I'm trying to put together a patch now..but the paprd branch makes > no sense at all. > > Shouldn't it also free the skb in the complete(&sc->paprd_complete) branch? > I don't see anything that uses the skb, and nothing that frees it. > > if (bf->bf_state.bfs_paprd) { > if (time_after(jiffies, > bf->bf_state.bfs_paprd_timestamp + > msecs_to_jiffies(ATH_PAPRD_TIMEOUT))) > dev_kfree_skb_any(skb); > else > complete(&sc->paprd_complete); I tried this patch, but the memory poision warnings continue. Might still be a useful patch though... [greearb@build-32 mac80211]$ git diff diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c index 8656491..f43a004 100644 --- a/drivers/net/wireless/ath/ath9k/main.c +++ b/drivers/net/wireless/ath/ath9k/main.c @@ -398,7 +398,7 @@ void ath_paprd_calibrate(struct work_struct *work) "Timeout waiting for paprd training on " "TX chain %d\n", chain); - goto fail_paprd; + break; } if (!ar9003_paprd_is_done(ah)) @@ -416,7 +416,6 @@ void ath_paprd_calibrate(struct work_struct *work) ath_paprd_activate(sc); } -fail_paprd: ath9k_ps_restore(sc); } diff --git a/drivers/net/wireless/ath/ath9k/xmit.c b/drivers/net/wireless/ath/ath9k/xmit.c index 942be55..d39b4b5 100644 --- a/drivers/net/wireless/ath/ath9k/xmit.c +++ b/drivers/net/wireless/ath/ath9k/xmit.c @@ -1919,17 +1919,17 @@ static void ath_tx_complete_buf(struct ath_softc *sc, struct ath_buf *bf, dma_unmap_single(sc->dev, bf->bf_dmacontext, skb->len, DMA_TO_DEVICE); if (bf->bf_state.bfs_paprd) { - if (time_after(jiffies, - bf->bf_state.bfs_paprd_timestamp + - msecs_to_jiffies(ATH_PAPRD_TIMEOUT))) - dev_kfree_skb_any(skb); - else - complete(&sc->paprd_complete); + /* ath_paprd_calibrate owns the skb. */ + complete(&sc->paprd_complete); } else { - ath_tx_complete(sc, skb, bf->aphy, tx_flags); + /* stat_tx must be called first, it references skb. */ ath_debug_stat_tx(sc, txq, bf, ts); + ath_tx_complete(sc, skb, bf->aphy, tx_flags); } + /* At this point, skb is consumed one way or another */ + bf->bf_mpdu = NULL; + /* * Return the list of ath_buf of this mpdu to free queue */ Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-12 18:40 ` Luis R. Rodriguez 2010-10-12 18:43 ` Ben Greear @ 2010-10-13 17:12 ` Ben Greear 2010-10-13 17:29 ` Luis R. Rodriguez 1 sibling, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-13 17:12 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote: > On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: >>> >>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> >>> wrote: >> >>>> Another thing I was thinking about: Maybe the queue of skbs and dma >>>> addresses >>>> in ath9k is getting corrupted by multiple VIFs trying to write at once? >>>> Maybe >>>> some locking is needed in the xmit path? >>> >>> That was my second hunch. My first shot was to use spin_lock_irqsave() >>> over the the uses of the rxbuf list and that seemed to help but I >>> still managed to get a poison eventually. My next item to check for is >>> of the permissibility of creating too much pressure to the point we >>> end up looping over the rxbuf list and race against mac80211 free'ing >>> a buffer. Will test that tomorrow if nothing else comes up creeping my >>> priority queue. >> >> This code looks weird to me. One of the paprd branches >> deletes the skb, the other doesn't appear to. Neither >> null out bf->bf_mpdu, which would appear to leave a dangling >> pointer in at least the dev_kfree_skb_any() branch. >> >> ath_tx_complete frees it's skb in all cases, so another >> bf->bf_mpdu dangling pointer issue. >> >> Maybe at the least we should null out bf->bf_mpdu when >> skb is consumed? > > You're reading my mind, that was what I was going to test today. Still > doing e-mail sweep though. At least in the xmit path, it seems cards that have EDMA support do things a bit different. Out of curiosity, on the system(s), you reproduce this, are any of yours supporting EDMA? Mine appear to not support EDMA. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-13 17:12 ` Ben Greear @ 2010-10-13 17:29 ` Luis R. Rodriguez 2010-10-13 17:48 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-13 17:29 UTC (permalink / raw) To: Ben Greear; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On Wed, Oct 13, 2010 at 10:12 AM, Ben Greear <greearb@candelatech.com> wrote: > On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote: >> >> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<greearb@candelatech.com> >> wrote: >>> >>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: >>>> >>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> >>>> wrote: >>> >>>>> Another thing I was thinking about: Maybe the queue of skbs and dma >>>>> addresses >>>>> in ath9k is getting corrupted by multiple VIFs trying to write at once? >>>>> Maybe >>>>> some locking is needed in the xmit path? >>>> >>>> That was my second hunch. My first shot was to use spin_lock_irqsave() >>>> over the the uses of the rxbuf list and that seemed to help but I >>>> still managed to get a poison eventually. My next item to check for is >>>> of the permissibility of creating too much pressure to the point we >>>> end up looping over the rxbuf list and race against mac80211 free'ing >>>> a buffer. Will test that tomorrow if nothing else comes up creeping my >>>> priority queue. >>> >>> This code looks weird to me. One of the paprd branches >>> deletes the skb, the other doesn't appear to. Neither >>> null out bf->bf_mpdu, which would appear to leave a dangling >>> pointer in at least the dev_kfree_skb_any() branch. >>> >>> ath_tx_complete frees it's skb in all cases, so another >>> bf->bf_mpdu dangling pointer issue. >>> >>> Maybe at the least we should null out bf->bf_mpdu when >>> skb is consumed? >> >> You're reading my mind, that was what I was going to test today. Still >> doing e-mail sweep though. > > At least in the xmit path, it seems cards that have EDMA support do > things a bit different. Out of curiosity, on the system(s), you reproduce > this, are any of yours supporting EDMA? Mine appear to not support EDMA. EDMA is used on >= AR9003 families by Atheros. And no, I am not testing with an EDMA card, I am testing with an AR9002 family card, the AR9280 card. I am going to disregard the TX stuff as the bug is an RX issue :) I was able to more easily reproduce by doing an skb_copy() and free'ing the buffer right afterwards on the ath_send_to_mac80211() thingy, So it does appear that the poison check just happens more often when we do an skb_copy(). One reason this is easy to reproduce with multiple STAs is mac80211 uses skb_copy() to process each received skb for each STA. In my tests so far, protecting the rxbuf list with spin_lock_irqsave() did not help, and the wmb(); didn't either, something else is going on here. It would be nice to hack slab to keep an entire trace of the place the buffer was last free'd at instead of just the caller that freed it. I haven't yet found a pattern on how this happens yet. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-13 17:29 ` Luis R. Rodriguez @ 2010-10-13 17:48 ` Ben Greear 2010-10-14 21:25 ` Luis R. Rodriguez 2010-10-14 21:45 ` Johannes Berg 0 siblings, 2 replies; 84+ messages in thread From: Ben Greear @ 2010-10-13 17:48 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/13/2010 10:29 AM, Luis R. Rodriguez wrote: > On Wed, Oct 13, 2010 at 10:12 AM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote: >>> >>> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<greearb@candelatech.com> >>> wrote: >>>> >>>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: >>>>> >>>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> >>>>> wrote: >>>> >>>>>> Another thing I was thinking about: Maybe the queue of skbs and dma >>>>>> addresses >>>>>> in ath9k is getting corrupted by multiple VIFs trying to write at once? >>>>>> Maybe >>>>>> some locking is needed in the xmit path? >>>>> >>>>> That was my second hunch. My first shot was to use spin_lock_irqsave() >>>>> over the the uses of the rxbuf list and that seemed to help but I >>>>> still managed to get a poison eventually. My next item to check for is >>>>> of the permissibility of creating too much pressure to the point we >>>>> end up looping over the rxbuf list and race against mac80211 free'ing >>>>> a buffer. Will test that tomorrow if nothing else comes up creeping my >>>>> priority queue. >>>> >>>> This code looks weird to me. One of the paprd branches >>>> deletes the skb, the other doesn't appear to. Neither >>>> null out bf->bf_mpdu, which would appear to leave a dangling >>>> pointer in at least the dev_kfree_skb_any() branch. >>>> >>>> ath_tx_complete frees it's skb in all cases, so another >>>> bf->bf_mpdu dangling pointer issue. >>>> >>>> Maybe at the least we should null out bf->bf_mpdu when >>>> skb is consumed? >>> >>> You're reading my mind, that was what I was going to test today. Still >>> doing e-mail sweep though. >> >> At least in the xmit path, it seems cards that have EDMA support do >> things a bit different. Out of curiosity, on the system(s), you reproduce >> this, are any of yours supporting EDMA? Mine appear to not support EDMA. > > EDMA is used on>= AR9003 families by Atheros. And no, I am not > testing with an EDMA card, I am testing with an AR9002 family card, > the AR9280 card. I am going to disregard the TX stuff as the bug is an > RX issue :) I was able to more easily reproduce by doing an skb_copy() > and free'ing the buffer right afterwards on the ath_send_to_mac80211() > thingy, So it does appear that the poison check just happens more > often when we do an skb_copy(). One reason this is easy to reproduce > with multiple STAs is mac80211 uses skb_copy() to process each > received skb for each STA. > > In my tests so far, protecting the rxbuf list with spin_lock_irqsave() > did not help, and the wmb(); didn't either, something else is going on > here. It would be nice to hack slab to keep an entire trace of the > place the buffer was last free'd at instead of just the caller that > freed it. I instrumented slub a while back and got the backtrace. It was always in the same place for my testing. Here's the slub patch if you are interested in using it yourself: https://patchwork.kernel.org/patch/236921/ Are you able to reproduce this with a single STA interface? If so, we should be able to somewhat tie-break mac80211 by using another /n NIC, hopefully with similar AMPDU support, etc. [From a mail I sent on 10/7 in this thread] In case it helps, here is a dump of where the corrupted SKB was deleted. I added debugging to slub to get this information, but it looks like it's correct to me. Reading symbols from /home/greearb/kernel/2.6/wireless-testing-dbg.p4s/net/mac80211/mac80211.ko...done. (gdb) l *(ieee80211_rx+0x74d) 0x13751 is in ieee80211_rx (/home/greearb/git/linux.wireless-testing/include/linux/rcupdate.h:346). 341 * 342 * See rcu_read_lock() for more information. 343 */ 344 static inline void rcu_read_unlock(void) 345 { 346 rcu_read_release(); 347 __release(RCU); 348 __rcu_read_unlock(); 349 } 350 (gdb) # I don't really know what that second address means, but just in case it's useful, # I printed it out here: (gdb) l *(ieee80211_rx+0x7b4) 0x137b8 is in ieee80211_process_measurement_req (/home/greearb/git/linux.wireless-testing/net/mac80211/spectmgmt.c:74). 69 } 70 71 void ieee80211_process_measurement_req(struct ieee80211_sub_if_data *sdata, 72 struct ieee80211_mgmt *mgmt, 73 size_t len) 74 { 75 /* 76 * Ignoring measurement request is spec violation. 77 * Mandatory measurements must be reported optional 78 * measurements might be refused or reported incapable INFO: Freed in skb_release_data+0x8c/0x90 age=122 cpu=1 pid=0 set_track+0x3c/0x89 __slab_free+0x17f/0x1ba skb_release_data+0x8c/0x90 kfree+0xaf/0xdf skb_release_data+0x8c/0x90 skb_release_data+0x8c/0x90 skb_release_data+0x8c/0x90 __kfree_skb+0x12/0x6d consume_skb+0x2a/0x2c ieee80211_rx+0x74d/0x7b4 [mac80211] __kmalloc_track_caller+0xcd/0xf2 trace_hardirqs_on_caller+0xeb/0x125 ath_rx_send_to_mac80211+0x5a/0x60 [ath9k] trace_hardirqs_on+0xb/0xd -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-13 17:48 ` Ben Greear @ 2010-10-14 21:25 ` Luis R. Rodriguez 2010-10-14 21:31 ` Ben Greear 2010-10-14 21:45 ` Johannes Berg 1 sibling, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 21:25 UTC (permalink / raw) To: Ben Greear; +Cc: linux-wireless@vger.kernel.org On Wed, Oct 13, 2010 at 10:48 AM, Ben Greear <greearb@candelatech.com> wrote: > On 10/13/2010 10:29 AM, Luis R. Rodriguez wrote: >> >> On Wed, Oct 13, 2010 at 10:12 AM, Ben Greear<greearb@candelatech.com> >> wrote: >>> >>> On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote: >>>> >>>> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<greearb@candelatech.com> >>>> wrote: >>>>> >>>>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: >>>>>> >>>>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> >>>>>> wrote: >>>>> >>>>>>> Another thing I was thinking about: Maybe the queue of skbs and dma >>>>>>> addresses >>>>>>> in ath9k is getting corrupted by multiple VIFs trying to write at >>>>>>> once? >>>>>>> Maybe >>>>>>> some locking is needed in the xmit path? >>>>>> >>>>>> That was my second hunch. My first shot was to use spin_lock_irqsave() >>>>>> over the the uses of the rxbuf list and that seemed to help but I >>>>>> still managed to get a poison eventually. My next item to check for is >>>>>> of the permissibility of creating too much pressure to the point we >>>>>> end up looping over the rxbuf list and race against mac80211 free'ing >>>>>> a buffer. Will test that tomorrow if nothing else comes up creeping my >>>>>> priority queue. >>>>> >>>>> This code looks weird to me. One of the paprd branches >>>>> deletes the skb, the other doesn't appear to. Neither >>>>> null out bf->bf_mpdu, which would appear to leave a dangling >>>>> pointer in at least the dev_kfree_skb_any() branch. >>>>> >>>>> ath_tx_complete frees it's skb in all cases, so another >>>>> bf->bf_mpdu dangling pointer issue. >>>>> >>>>> Maybe at the least we should null out bf->bf_mpdu when >>>>> skb is consumed? >>>> >>>> You're reading my mind, that was what I was going to test today. Still >>>> doing e-mail sweep though. >>> >>> At least in the xmit path, it seems cards that have EDMA support do >>> things a bit different. Out of curiosity, on the system(s), you >>> reproduce >>> this, are any of yours supporting EDMA? Mine appear to not support EDMA. >> >> EDMA is used on>= AR9003 families by Atheros. And no, I am not >> testing with an EDMA card, I am testing with an AR9002 family card, >> the AR9280 card. I am going to disregard the TX stuff as the bug is an >> RX issue :) I was able to more easily reproduce by doing an skb_copy() >> and free'ing the buffer right afterwards on the ath_send_to_mac80211() >> thingy, So it does appear that the poison check just happens more >> often when we do an skb_copy(). One reason this is easy to reproduce >> with multiple STAs is mac80211 uses skb_copy() to process each >> received skb for each STA. >> >> In my tests so far, protecting the rxbuf list with spin_lock_irqsave() >> did not help, and the wmb(); didn't either, something else is going on >> here. It would be nice to hack slab to keep an entire trace of the >> place the buffer was last free'd at instead of just the caller that >> freed it. > > I instrumented slub a while back and got the backtrace. It > was always in the same place for my testing. > > Here's the slub patch if you are interested in using it yourself: > https://patchwork.kernel.org/patch/236921/ when compiling this patch I get: arch/x86/built-in.o: In function `store_stack': /home/mcgrof/wireless-testing/arch/x86/kernel/dumpstack.c:259: undefined reference to `store_trace' Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 21:25 ` Luis R. Rodriguez @ 2010-10-14 21:31 ` Ben Greear 2010-10-14 21:32 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-14 21:31 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-wireless@vger.kernel.org On 10/14/2010 02:25 PM, Luis R. Rodriguez wrote: > On Wed, Oct 13, 2010 at 10:48 AM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/13/2010 10:29 AM, Luis R. Rodriguez wrote: >>> >>> On Wed, Oct 13, 2010 at 10:12 AM, Ben Greear<greearb@candelatech.com> >>> wrote: >>>> >>>> On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote: >>>>> >>>>> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<greearb@candelatech.com> >>>>> wrote: >>>>>> >>>>>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: >>>>>>> >>>>>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> >>>>>>> wrote: >>>>>> >>>>>>>> Another thing I was thinking about: Maybe the queue of skbs and dma >>>>>>>> addresses >>>>>>>> in ath9k is getting corrupted by multiple VIFs trying to write at >>>>>>>> once? >>>>>>>> Maybe >>>>>>>> some locking is needed in the xmit path? >>>>>>> >>>>>>> That was my second hunch. My first shot was to use spin_lock_irqsave() >>>>>>> over the the uses of the rxbuf list and that seemed to help but I >>>>>>> still managed to get a poison eventually. My next item to check for is >>>>>>> of the permissibility of creating too much pressure to the point we >>>>>>> end up looping over the rxbuf list and race against mac80211 free'ing >>>>>>> a buffer. Will test that tomorrow if nothing else comes up creeping my >>>>>>> priority queue. >>>>>> >>>>>> This code looks weird to me. One of the paprd branches >>>>>> deletes the skb, the other doesn't appear to. Neither >>>>>> null out bf->bf_mpdu, which would appear to leave a dangling >>>>>> pointer in at least the dev_kfree_skb_any() branch. >>>>>> >>>>>> ath_tx_complete frees it's skb in all cases, so another >>>>>> bf->bf_mpdu dangling pointer issue. >>>>>> >>>>>> Maybe at the least we should null out bf->bf_mpdu when >>>>>> skb is consumed? >>>>> >>>>> You're reading my mind, that was what I was going to test today. Still >>>>> doing e-mail sweep though. >>>> >>>> At least in the xmit path, it seems cards that have EDMA support do >>>> things a bit different. Out of curiosity, on the system(s), you >>>> reproduce >>>> this, are any of yours supporting EDMA? Mine appear to not support EDMA. >>> >>> EDMA is used on>= AR9003 families by Atheros. And no, I am not >>> testing with an EDMA card, I am testing with an AR9002 family card, >>> the AR9280 card. I am going to disregard the TX stuff as the bug is an >>> RX issue :) I was able to more easily reproduce by doing an skb_copy() >>> and free'ing the buffer right afterwards on the ath_send_to_mac80211() >>> thingy, So it does appear that the poison check just happens more >>> often when we do an skb_copy(). One reason this is easy to reproduce >>> with multiple STAs is mac80211 uses skb_copy() to process each >>> received skb for each STA. >>> >>> In my tests so far, protecting the rxbuf list with spin_lock_irqsave() >>> did not help, and the wmb(); didn't either, something else is going on >>> here. It would be nice to hack slab to keep an entire trace of the >>> place the buffer was last free'd at instead of just the caller that >>> freed it. >> >> I instrumented slub a while back and got the backtrace. It >> was always in the same place for my testing. >> >> Here's the slub patch if you are interested in using it yourself: >> https://patchwork.kernel.org/patch/236921/ > > when compiling this patch I get: > > arch/x86/built-in.o: In function `store_stack': > /home/mcgrof/wireless-testing/arch/x86/kernel/dumpstack.c:259: > undefined reference to `store_trace' You are compiling on 32-bit system? I see the method in the patch, but probably only for 32-bit x86... Ben > > Luis -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 21:31 ` Ben Greear @ 2010-10-14 21:32 ` Luis R. Rodriguez 2010-10-14 21:39 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 21:32 UTC (permalink / raw) To: Ben Greear; +Cc: linux-wireless@vger.kernel.org On Thu, Oct 14, 2010 at 2:31 PM, Ben Greear <greearb@candelatech.com> wrote: > On 10/14/2010 02:25 PM, Luis R. Rodriguez wrote: >> >> On Wed, Oct 13, 2010 at 10:48 AM, Ben Greear<greearb@candelatech.com> >> wrote: >>> >>> On 10/13/2010 10:29 AM, Luis R. Rodriguez wrote: >>>> >>>> On Wed, Oct 13, 2010 at 10:12 AM, Ben Greear<greearb@candelatech.com> >>>> wrote: >>>>> >>>>> On 10/12/2010 11:40 AM, Luis R. Rodriguez wrote: >>>>>> >>>>>> On Tue, Oct 12, 2010 at 11:35 AM, Ben Greear<greearb@candelatech.com> >>>>>> wrote: >>>>>>> >>>>>>> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: >>>>>>>> >>>>>>>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> >>>>>>>> wrote: >>>>>>> >>>>>>>>> Another thing I was thinking about: Maybe the queue of skbs and >>>>>>>>> dma >>>>>>>>> addresses >>>>>>>>> in ath9k is getting corrupted by multiple VIFs trying to write at >>>>>>>>> once? >>>>>>>>> Maybe >>>>>>>>> some locking is needed in the xmit path? >>>>>>>> >>>>>>>> That was my second hunch. My first shot was to use >>>>>>>> spin_lock_irqsave() >>>>>>>> over the the uses of the rxbuf list and that seemed to help but I >>>>>>>> still managed to get a poison eventually. My next item to check for >>>>>>>> is >>>>>>>> of the permissibility of creating too much pressure to the point we >>>>>>>> end up looping over the rxbuf list and race against mac80211 >>>>>>>> free'ing >>>>>>>> a buffer. Will test that tomorrow if nothing else comes up creeping >>>>>>>> my >>>>>>>> priority queue. >>>>>>> >>>>>>> This code looks weird to me. One of the paprd branches >>>>>>> deletes the skb, the other doesn't appear to. Neither >>>>>>> null out bf->bf_mpdu, which would appear to leave a dangling >>>>>>> pointer in at least the dev_kfree_skb_any() branch. >>>>>>> >>>>>>> ath_tx_complete frees it's skb in all cases, so another >>>>>>> bf->bf_mpdu dangling pointer issue. >>>>>>> >>>>>>> Maybe at the least we should null out bf->bf_mpdu when >>>>>>> skb is consumed? >>>>>> >>>>>> You're reading my mind, that was what I was going to test today. Still >>>>>> doing e-mail sweep though. >>>>> >>>>> At least in the xmit path, it seems cards that have EDMA support do >>>>> things a bit different. Out of curiosity, on the system(s), you >>>>> reproduce >>>>> this, are any of yours supporting EDMA? Mine appear to not support >>>>> EDMA. >>>> >>>> EDMA is used on>= AR9003 families by Atheros. And no, I am not >>>> testing with an EDMA card, I am testing with an AR9002 family card, >>>> the AR9280 card. I am going to disregard the TX stuff as the bug is an >>>> RX issue :) I was able to more easily reproduce by doing an skb_copy() >>>> and free'ing the buffer right afterwards on the ath_send_to_mac80211() >>>> thingy, So it does appear that the poison check just happens more >>>> often when we do an skb_copy(). One reason this is easy to reproduce >>>> with multiple STAs is mac80211 uses skb_copy() to process each >>>> received skb for each STA. >>>> >>>> In my tests so far, protecting the rxbuf list with spin_lock_irqsave() >>>> did not help, and the wmb(); didn't either, something else is going on >>>> here. It would be nice to hack slab to keep an entire trace of the >>>> place the buffer was last free'd at instead of just the caller that >>>> freed it. >>> >>> I instrumented slub a while back and got the backtrace. It >>> was always in the same place for my testing. >>> >>> Here's the slub patch if you are interested in using it yourself: >>> https://patchwork.kernel.org/patch/236921/ >> >> when compiling this patch I get: >> >> arch/x86/built-in.o: In function `store_stack': >> /home/mcgrof/wireless-testing/arch/x86/kernel/dumpstack.c:259: >> undefined reference to `store_trace' > > You are compiling on 32-bit system? I see the method in > the patch, but probably only for 32-bit x86... Ah no I'm on 64-bit. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 21:32 ` Luis R. Rodriguez @ 2010-10-14 21:39 ` Ben Greear 0 siblings, 0 replies; 84+ messages in thread From: Ben Greear @ 2010-10-14 21:39 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-wireless@vger.kernel.org On 10/14/2010 02:32 PM, Luis R. Rodriguez wrote: > On Thu, Oct 14, 2010 at 2:31 PM, Ben Greear<greearb@candelatech.com> wrote: >>> arch/x86/built-in.o: In function `store_stack': >>> /home/mcgrof/wireless-testing/arch/x86/kernel/dumpstack.c:259: >>> undefined reference to `store_trace' >> >> You are compiling on 32-bit system? I see the method in >> the patch, but probably only for 32-bit x86... > > Ah no I'm on 64-bit. Probably you could cut-n-paste dump_trace in dumpstack_64.c, rename it store_trace, and hack it a bit as I did the one in dumpstack_32.c I don't currently have any ath9k systems running 64-bit, but I could put one together if needed. By the way, if you have any extra ideas of what to try next, I'll consider trying it. I am very low on ideas personally :P Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-13 17:48 ` Ben Greear 2010-10-14 21:25 ` Luis R. Rodriguez @ 2010-10-14 21:45 ` Johannes Berg 2010-10-14 21:47 ` Ben Greear 1 sibling, 1 reply; 84+ messages in thread From: Johannes Berg @ 2010-10-14 21:45 UTC (permalink / raw) To: Ben Greear; +Cc: Luis R. Rodriguez, linux-wireless@vger.kernel.org On Wed, 2010-10-13 at 10:48 -0700, Ben Greear wrote: > Here's the slub patch if you are interested in using it yourself: > https://patchwork.kernel.org/patch/236921/ That really really screams include/linux/stacktrace.h johannes ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 21:45 ` Johannes Berg @ 2010-10-14 21:47 ` Ben Greear 0 siblings, 0 replies; 84+ messages in thread From: Ben Greear @ 2010-10-14 21:47 UTC (permalink / raw) To: Johannes Berg; +Cc: Luis R. Rodriguez, linux-wireless@vger.kernel.org On 10/14/2010 02:45 PM, Johannes Berg wrote: > On Wed, 2010-10-13 at 10:48 -0700, Ben Greear wrote: > >> Here's the slub patch if you are interested in using it yourself: >> https://patchwork.kernel.org/patch/236921/ > > That really really screams include/linux/stacktrace.h Heh, that looks like it would have saved me a lot of pain :P Ben > > johannes -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-12 18:35 ` Ben Greear 2010-10-12 18:40 ` Luis R. Rodriguez @ 2010-10-13 5:31 ` Vasanthakumar Thiagarajan 2010-10-13 16:39 ` Ben Greear 1 sibling, 1 reply; 84+ messages in thread From: Vasanthakumar Thiagarajan @ 2010-10-13 5:31 UTC (permalink / raw) To: Ben Greear Cc: Luis R. Rodriguez, Johannes Berg, linux-wireless@vger.kernel.org On Wed, Oct 13, 2010 at 12:05:53AM +0530, Ben Greear wrote: > On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: > > On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> wrote: > > >> Another thing I was thinking about: Maybe the queue of skbs and dma > >> addresses > >> in ath9k is getting corrupted by multiple VIFs trying to write at once? > >> Maybe > >> some locking is needed in the xmit path? > > > > That was my second hunch. My first shot was to use spin_lock_irqsave() > > over the the uses of the rxbuf list and that seemed to help but I > > still managed to get a poison eventually. My next item to check for is > > of the permissibility of creating too much pressure to the point we > > end up looping over the rxbuf list and race against mac80211 free'ing > > a buffer. Will test that tomorrow if nothing else comes up creeping my > > priority queue. > > This code looks weird to me. One of the paprd branches > deletes the skb, the other doesn't appear to. Neither > null out bf->bf_mpdu, which would appear to leave a dangling > pointer in at least the dev_kfree_skb_any() branch. Single skb is (re)used for sending paprd training frames on more than one chains. This skb needs to be freed only when paprd fails on any of the chains or it succeeded on all the chains. The failure case is handled in ath_tx_complete_buf() and success case is in ath_paprd_calibrate(). > > ath_tx_complete frees it's skb in all cases, so another > bf->bf_mpdu dangling pointer issue. > > Maybe at the least we should null out bf->bf_mpdu when > skb is consumed? I dont see any point in NULLing out bf->bf_mpdu. bf is reclaimed onto a free tx buf pool as soon as it is done with the skb. bf_mpdu of any of the bf's is never accessed without any initialization (bf_ampdu = skb). Vasanth ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-13 5:31 ` Vasanthakumar Thiagarajan @ 2010-10-13 16:39 ` Ben Greear 2010-10-13 19:56 ` Björn Smedman 2010-10-14 5:37 ` Vasanthakumar Thiagarajan 0 siblings, 2 replies; 84+ messages in thread From: Ben Greear @ 2010-10-13 16:39 UTC (permalink / raw) To: Vasanthakumar Thiagarajan Cc: Luis R. Rodriguez, Johannes Berg, linux-wireless@vger.kernel.org On 10/12/2010 10:31 PM, Vasanthakumar Thiagarajan wrote: > On Wed, Oct 13, 2010 at 12:05:53AM +0530, Ben Greear wrote: >> On 10/11/2010 11:10 PM, Luis R. Rodriguez wrote: >>> On Mon, Oct 11, 2010 at 8:27 PM, Ben Greear<greearb@candelatech.com> wrote: >> >>>> Another thing I was thinking about: Maybe the queue of skbs and dma >>>> addresses >>>> in ath9k is getting corrupted by multiple VIFs trying to write at once? >>>> Maybe >>>> some locking is needed in the xmit path? >>> >>> That was my second hunch. My first shot was to use spin_lock_irqsave() >>> over the the uses of the rxbuf list and that seemed to help but I >>> still managed to get a poison eventually. My next item to check for is >>> of the permissibility of creating too much pressure to the point we >>> end up looping over the rxbuf list and race against mac80211 free'ing >>> a buffer. Will test that tomorrow if nothing else comes up creeping my >>> priority queue. >> >> This code looks weird to me. One of the paprd branches >> deletes the skb, the other doesn't appear to. Neither >> null out bf->bf_mpdu, which would appear to leave a dangling >> pointer in at least the dev_kfree_skb_any() branch. > > Single skb is (re)used for sending paprd training frames on more > than one chains. This skb needs to be freed only when paprd fails on > any of the chains or it succeeded on all the chains. The failure > case is handled in ath_tx_complete_buf() and success case is in > ath_paprd_calibrate(). >> >> ath_tx_complete frees it's skb in all cases, so another >> bf->bf_mpdu dangling pointer issue. >> >> Maybe at the least we should null out bf->bf_mpdu when >> skb is consumed? > > I dont see any point in NULLing out bf->bf_mpdu. bf is > reclaimed onto a free tx buf pool as soon as it is done > with the skb. bf_mpdu of any of the bf's is never accessed > without any initialization (bf_ampdu = skb). The code can use skb after its deleted currently, because ath_debug_stat_tx(sc, txq, bf, ts); references the bf_ampdu object (I think I added that reference lately..so it's really a bug that I caused). At the least, we should move the ath_debug_stat_tx logic before the ath_tx_complete() call. As for the paprd path, it looks racy to me: What if the paprd timer expires while the ath_tx_complete_buf logic is running? Either way, it seems safer to null out the bf_ampdu field after the memory is consumed..it could prevent some tricky bugs later. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-13 16:39 ` Ben Greear @ 2010-10-13 19:56 ` Björn Smedman 2010-10-13 20:03 ` Luis R. Rodriguez 2010-10-14 21:52 ` Björn Smedman 2010-10-14 5:37 ` Vasanthakumar Thiagarajan 1 sibling, 2 replies; 84+ messages in thread From: Björn Smedman @ 2010-10-13 19:56 UTC (permalink / raw) To: Ben Greear Cc: Vasanthakumar Thiagarajan, Luis R. Rodriguez, Johannes Berg, linux-wireless@vger.kernel.org Hi Ben, First of all keep up the good work. :) On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear <greearb@candelatech.com> wrote: [snip] > Either way, it seems safer to null out the bf_ampdu field after > the memory is consumed..it could prevent some tricky bugs later. I think this is a good idea. But it probably wont be enough to null out bf_mpdu. You also need to look at bf_buf_addr (which if I understand correctly is the physical address the DMA engine will actually write RXed frames to) and bf_dmacontext (which seems in most cases to hold an identical address and may in fact be where the DMA engine will really write the frame). /Björn ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-13 19:56 ` Björn Smedman @ 2010-10-13 20:03 ` Luis R. Rodriguez 2010-10-14 19:15 ` Ben Greear 2010-10-14 21:52 ` Björn Smedman 1 sibling, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-13 20:03 UTC (permalink / raw) To: Björn Smedman Cc: Ben Greear, Vasanthakumar Thiagarajan, Johannes Berg, linux-wireless@vger.kernel.org 2010/10/13 Björn Smedman <bjorn.smedman@venatech.se>: > Hi Ben, > > First of all keep up the good work. :) > > On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear <greearb@candelatech.com> wrote: > [snip] >> Either way, it seems safer to null out the bf_ampdu field after >> the memory is consumed..it could prevent some tricky bugs later. > > I think this is a good idea. But it probably wont be enough to null > out bf_mpdu. You also need to look at bf_buf_addr (which if I > understand correctly is the physical address the DMA engine will > actually write RXed frames to) and bf_dmacontext (which seems in most > cases to hold an identical address and may in fact be where the DMA > engine will really write the frame). See the TODO list for ath9k: * Remove this redundancy crap: bf->bf_buf_addr = bf->bf_dmacontext; http://wireless.kernel.org/en/users/Drivers/ath9k/todo Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-13 20:03 ` Luis R. Rodriguez @ 2010-10-14 19:15 ` Ben Greear 2010-10-14 19:17 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-14 19:15 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Björn Smedman, Vasanthakumar Thiagarajan, Johannes Berg, linux-wireless@vger.kernel.org On 10/13/2010 01:03 PM, Luis R. Rodriguez wrote: > 2010/10/13 Björn Smedman<bjorn.smedman@venatech.se>: >> Hi Ben, >> >> First of all keep up the good work. :) >> >> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<greearb@candelatech.com> wrote: >> [snip] >>> Either way, it seems safer to null out the bf_ampdu field after >>> the memory is consumed..it could prevent some tricky bugs later. >> >> I think this is a good idea. But it probably wont be enough to null >> out bf_mpdu. You also need to look at bf_buf_addr (which if I >> understand correctly is the physical address the DMA engine will >> actually write RXed frames to) and bf_dmacontext (which seems in most >> cases to hold an identical address and may in fact be where the DMA >> engine will really write the frame). > > See the TODO list for ath9k: > > * Remove this redundancy crap: bf->bf_buf_addr = bf->bf_dmacontext; > > http://wireless.kernel.org/en/users/Drivers/ath9k/todo I just posted a patch that attempts this. It doesn't fix the memory clobber issue, but maybe the code is a bit cleaner/safer at least... Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 19:15 ` Ben Greear @ 2010-10-14 19:17 ` Luis R. Rodriguez 0 siblings, 0 replies; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 19:17 UTC (permalink / raw) To: Ben Greear Cc: Björn Smedman, Vasanthakumar Thiagarajan, Johannes Berg, linux-wireless@vger.kernel.org On Thu, Oct 14, 2010 at 12:15 PM, Ben Greear <greearb@candelatech.com> wrote: > On 10/13/2010 01:03 PM, Luis R. Rodriguez wrote: >> >> 2010/10/13 Björn Smedman<bjorn.smedman@venatech.se>: >>> >>> Hi Ben, >>> >>> First of all keep up the good work. :) >>> >>> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<greearb@candelatech.com> >>> wrote: >>> [snip] >>>> >>>> Either way, it seems safer to null out the bf_ampdu field after >>>> the memory is consumed..it could prevent some tricky bugs later. >>> >>> I think this is a good idea. But it probably wont be enough to null >>> out bf_mpdu. You also need to look at bf_buf_addr (which if I >>> understand correctly is the physical address the DMA engine will >>> actually write RXed frames to) and bf_dmacontext (which seems in most >>> cases to hold an identical address and may in fact be where the DMA >>> engine will really write the frame). >> >> See the TODO list for ath9k: >> >> * Remove this redundancy crap: bf->bf_buf_addr = bf->bf_dmacontext; >> >> http://wireless.kernel.org/en/users/Drivers/ath9k/todo > > I just posted a patch that attempts this. It doesn't > fix the memory clobber issue, but maybe the code is a bit cleaner/safer > at least... Easier to read at least, thanks. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-13 19:56 ` Björn Smedman 2010-10-13 20:03 ` Luis R. Rodriguez @ 2010-10-14 21:52 ` Björn Smedman 2010-10-14 22:05 ` Ben Greear 2010-10-14 22:47 ` Ben Greear 1 sibling, 2 replies; 84+ messages in thread From: Björn Smedman @ 2010-10-14 21:52 UTC (permalink / raw) To: Ben Greear Cc: Vasanthakumar Thiagarajan, Luis R. Rodriguez, Johannes Berg, linux-wireless@vger.kernel.org 2010/10/13 Björn Smedman <bjorn.smedman@venatech.se>: > Hi Ben, > > First of all keep up the good work. :) > > On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear <greearb@candelatech.com> wrote: > [snip] >> Either way, it seems safer to null out the bf_ampdu field after >> the memory is consumed..it could prevent some tricky bugs later. > > I think this is a good idea. But it probably wont be enough to null > out bf_mpdu. You also need to look at bf_buf_addr (which if I > understand correctly is the physical address the DMA engine will > actually write RXed frames to) and bf_dmacontext (which seems in most > cases to hold an identical address and may in fact be where the DMA > engine will really write the frame). I took another look at the code. It turns out both bf_buf_addr and bf_dmacontext are in fact meaningless to the DMA. Instead each bf holds a pointer (bf_desc) to the real DMA descriptor which in turn holds the address (ds_data) where the DMA will really (really this time) write the frame. There is also a field to hold the virtual address of the same place (ds_vdata). It's a little too much work for me to set up the testbed you have Ben but would be interesting to see what happens if you set bf->bf_desc->ds_{data,vdata} = 0 as well. No? /Björn ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 21:52 ` Björn Smedman @ 2010-10-14 22:05 ` Ben Greear 2010-10-14 22:16 ` Luis R. Rodriguez 2010-10-14 22:47 ` Ben Greear 1 sibling, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-14 22:05 UTC (permalink / raw) To: Björn Smedman Cc: Vasanthakumar Thiagarajan, Luis R. Rodriguez, Johannes Berg, linux-wireless@vger.kernel.org On 10/14/2010 02:52 PM, Björn Smedman wrote: > 2010/10/13 Björn Smedman<bjorn.smedman@venatech.se>: >> Hi Ben, >> >> First of all keep up the good work. :) >> >> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<greearb@candelatech.com> wrote: >> [snip] >>> Either way, it seems safer to null out the bf_ampdu field after >>> the memory is consumed..it could prevent some tricky bugs later. >> >> I think this is a good idea. But it probably wont be enough to null >> out bf_mpdu. You also need to look at bf_buf_addr (which if I >> understand correctly is the physical address the DMA engine will >> actually write RXed frames to) and bf_dmacontext (which seems in most >> cases to hold an identical address and may in fact be where the DMA >> engine will really write the frame). > > I took another look at the code. It turns out both bf_buf_addr and > bf_dmacontext are in fact meaningless to the DMA. Instead each bf > holds a pointer (bf_desc) to the real DMA descriptor which in turn > holds the address (ds_data) where the DMA will really (really this > time) write the frame. There is also a field to hold the virtual > address of the same place (ds_vdata). > > It's a little too much work for me to set up the testbed you have Ben > but would be interesting to see what happens if you set > bf->bf_desc->ds_{data,vdata} = 0 as well. No? I'll investigate those suggestions. But setting up a test-bed is as easy as getting an ath9k NIC in a system, with a few APs around, and run the script below. You do not need any traffic generation, dhcp, etc...seems just beacons and whatever wpa_supplicant is doing is enough to hit the problem fast. (Make sure you are compiled to detect memory poisoning, of course). You'll need to fix the paths to the executables most likely. #!/usr/bin/perl use strict; my $iw = "./local/sbin/iw"; my $ip = "./local/sbin/ip"; my $wpa_s = "./local/bin/wpa_supplicant"; my $ssid = "candela-n"; my $key = "wpadmz123"; my $phy = "phy0"; my $max = 32; my $i; my $bmac = "00:01:02:03:04:"; my $cmd; # Create stations for ($i = 0; $i<$max; $i++) { runCmd("$iw phy $phy interface add sta$i type station"); my $mc5 = "$i"; if (length($mc5) == 1) { $mc5 = "0$mc5"; # pad mac octet } my $mac = "$bmac$mc5"; runCmd("$ip link set sta$i address $mac"); runCmd("$iw dev sta$i set power_save off"); } # Bring them up with WPA for ($i = 0; $i<$max; $i++) { open(FD, ">sta$i" . "_wpa.conf") || die("Couldn't open file: $!\n"); print FD " ctrl_interface=/var/run/wpa_supplicant fast_reauth=1 #can_scan_one=1 network={ ssid=\"$ssid\" proto=WPA key_mgmt=WPA-PSK psk=\"$key\" pairwise=TKIP CCMP group=TKIP CCMP } "; runCmd("$wpa_s -B -i sta$i -c sta$i" . "_wpa.conf -P sta$i" . "_wpa.pid -t -f sta$i" . "_wpa.log"); } sub runCmd { my $cmd = shift; print "$cmd\n"; `$cmd`; } Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 22:05 ` Ben Greear @ 2010-10-14 22:16 ` Luis R. Rodriguez 2010-10-14 22:29 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 22:16 UTC (permalink / raw) To: Ben Greear Cc: Björn Smedman, Vasanthakumar Thiagarajan, Johannes Berg, linux-wireless@vger.kernel.org 2010/10/14 Ben Greear <greearb@candelatech.com>: > On 10/14/2010 02:52 PM, Björn Smedman wrote: >> >> 2010/10/13 Björn Smedman<bjorn.smedman@venatech.se>: >>> >>> Hi Ben, >>> >>> First of all keep up the good work. :) >>> >>> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<greearb@candelatech.com> >>> wrote: >>> [snip] >>>> >>>> Either way, it seems safer to null out the bf_ampdu field after >>>> the memory is consumed..it could prevent some tricky bugs later. >>> >>> I think this is a good idea. But it probably wont be enough to null >>> out bf_mpdu. You also need to look at bf_buf_addr (which if I >>> understand correctly is the physical address the DMA engine will >>> actually write RXed frames to) and bf_dmacontext (which seems in most >>> cases to hold an identical address and may in fact be where the DMA >>> engine will really write the frame). >> >> I took another look at the code. It turns out both bf_buf_addr and >> bf_dmacontext are in fact meaningless to the DMA. Instead each bf >> holds a pointer (bf_desc) to the real DMA descriptor which in turn >> holds the address (ds_data) where the DMA will really (really this >> time) write the frame. There is also a field to hold the virtual >> address of the same place (ds_vdata). >> >> It's a little too much work for me to set up the testbed you have Ben >> but would be interesting to see what happens if you set >> bf->bf_desc->ds_{data,vdata} = 0 as well. No? > > I'll investigate those suggestions. > > But setting up a test-bed is as easy > as getting an ath9k NIC in a system, with a few APs around, and run the > script below. > > You do not need any traffic generation, dhcp, etc...seems just beacons and > whatever > wpa_supplicant is doing is enough to hit the problem fast. (Make sure > you are compiled to detect memory poisoning, of course). > > You'll need to fix the paths to the executables most likely. > You don't need such complicated scripts, I've managed to reproduce now by creating a lot of monitor interfaces and then looping with a regular interface issuing a scan command over and over. I suspect I'll be able to do this as well by changing channels instead of doing a scan. I believe the issue may be due to races in hardware on resets and enabling RX on an already freed buffer. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 22:16 ` Luis R. Rodriguez @ 2010-10-14 22:29 ` Luis R. Rodriguez 2010-10-14 22:35 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 22:29 UTC (permalink / raw) To: Ben Greear Cc: Björn Smedman, Vasanthakumar Thiagarajan, Johannes Berg, linux-wireless@vger.kernel.org On Thu, Oct 14, 2010 at 3:16 PM, Luis R. Rodriguez <mcgrof@gmail.com> wrote: > 2010/10/14 Ben Greear <greearb@candelatech.com>: >> On 10/14/2010 02:52 PM, Björn Smedman wrote: >>> >>> 2010/10/13 Björn Smedman<bjorn.smedman@venatech.se>: >>>> >>>> Hi Ben, >>>> >>>> First of all keep up the good work. :) >>>> >>>> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<greearb@candelatech.com> >>>> wrote: >>>> [snip] >>>>> >>>>> Either way, it seems safer to null out the bf_ampdu field after >>>>> the memory is consumed..it could prevent some tricky bugs later. >>>> >>>> I think this is a good idea. But it probably wont be enough to null >>>> out bf_mpdu. You also need to look at bf_buf_addr (which if I >>>> understand correctly is the physical address the DMA engine will >>>> actually write RXed frames to) and bf_dmacontext (which seems in most >>>> cases to hold an identical address and may in fact be where the DMA >>>> engine will really write the frame). >>> >>> I took another look at the code. It turns out both bf_buf_addr and >>> bf_dmacontext are in fact meaningless to the DMA. Instead each bf >>> holds a pointer (bf_desc) to the real DMA descriptor which in turn >>> holds the address (ds_data) where the DMA will really (really this >>> time) write the frame. There is also a field to hold the virtual >>> address of the same place (ds_vdata). >>> >>> It's a little too much work for me to set up the testbed you have Ben >>> but would be interesting to see what happens if you set >>> bf->bf_desc->ds_{data,vdata} = 0 as well. No? >> >> I'll investigate those suggestions. >> >> But setting up a test-bed is as easy >> as getting an ath9k NIC in a system, with a few APs around, and run the >> script below. >> >> You do not need any traffic generation, dhcp, etc...seems just beacons and >> whatever >> wpa_supplicant is doing is enough to hit the problem fast. (Make sure >> you are compiled to detect memory poisoning, of course). >> >> You'll need to fix the paths to the executables most likely. >> > > You don't need such complicated scripts, I've managed to reproduce now > by creating a lot of monitor interfaces and then looping with a > regular interface issuing a scan command over and over. I suspect > I'll be able to do this as well by changing channels instead of doing > a scan. I believe the issue may be due to races in hardware on resets > and enabling RX on an already freed buffer. Fun enough if I just create one monitor interface and loop quickly over some 2 GHz channels where I know I have traffic nearby I don't see the poison. So channel changes don't seem to do much because this is changing channels as fast as possible from userspace. I also can confirm that I see frames from the different channels as I move along. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 22:29 ` Luis R. Rodriguez @ 2010-10-14 22:35 ` Luis R. Rodriguez 2010-10-14 22:44 ` Ben Greear 2010-10-14 22:51 ` Luis R. Rodriguez 0 siblings, 2 replies; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 22:35 UTC (permalink / raw) To: Ben Greear; +Cc: linux-wireless, Luis R. Rodriguez On Thu, Oct 14, 2010 at 3:29 PM, Luis R. Rodriguez <mcgrof@gmail.com> wrote: > Fun enough if I just create one monitor interface and loop quickly > over some 2 GHz channels where I know I have traffic nearby I don't > see the poison. So channel changes don't seem to do much because this > is changing channels as fast as possible from userspace. I also can > confirm that I see frames from the different channels as I move along. Even forcing a band change doesn't help trigger it with just one mon0 and one regular device scanning in a loop; while true; do for i in 2412 5745 2417 5745 2422 5745 2427 5745 2432 5745 2442; do echo $i iw dev mon0 set freq $i; done; done while true; do iw dev wlan0 scan; done Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 22:35 ` Luis R. Rodriguez @ 2010-10-14 22:44 ` Ben Greear 2010-10-14 22:54 ` Luis R. Rodriguez 2010-10-14 22:51 ` Luis R. Rodriguez 1 sibling, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-14 22:44 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-wireless, Luis R. Rodriguez On 10/14/2010 03:35 PM, Luis R. Rodriguez wrote: > On Thu, Oct 14, 2010 at 3:29 PM, Luis R. Rodriguez<mcgrof@gmail.com> wrote: >> Fun enough if I just create one monitor interface and loop quickly >> over some 2 GHz channels where I know I have traffic nearby I don't >> see the poison. So channel changes don't seem to do much because this >> is changing channels as fast as possible from userspace. I also can >> confirm that I see frames from the different channels as I move along. > > Even forcing a band change doesn't help trigger it with just one mon0 > and one regular device scanning in a loop; > > while true; do for i in 2412 5745 2417 5745 2422 5745 2427 5745 2432 > 5745 2442; do echo $i iw dev mon0 set freq $i; done; done > while true; do iw dev wlan0 scan; done What if you just make a bunch of skb copies in ath9k before it sends them up the stack, and then delete them? (That's basically what a bunch of monitor devices would be doing, eh?) Thanks, Ben > > Luis -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 22:44 ` Ben Greear @ 2010-10-14 22:54 ` Luis R. Rodriguez 0 siblings, 0 replies; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 22:54 UTC (permalink / raw) To: Ben Greear, g; +Cc: Luis R. Rodriguez, linux-wireless, Luis Rodriguez On Thu, Oct 14, 2010 at 03:44:49PM -0700, Ben Greear wrote: > On 10/14/2010 03:35 PM, Luis R. Rodriguez wrote: > > On Thu, Oct 14, 2010 at 3:29 PM, Luis R. Rodriguez<mcgrof@gmail.com> wrote: > >> Fun enough if I just create one monitor interface and loop quickly > >> over some 2 GHz channels where I know I have traffic nearby I don't > >> see the poison. So channel changes don't seem to do much because this > >> is changing channels as fast as possible from userspace. I also can > >> confirm that I see frames from the different channels as I move along. > > > > Even forcing a band change doesn't help trigger it with just one mon0 > > and one regular device scanning in a loop; > > > > while true; do for i in 2412 5745 2417 5745 2422 5745 2427 5745 2432 > > 5745 2442; do echo $i iw dev mon0 set freq $i; done; done > > while true; do iw dev wlan0 scan; done > > What if you just make a bunch of skb copies in ath9k before it > sends them up the stack, and then delete them? (That's basically > what a bunch of monitor devices would be doing, eh?) Sure, as you can see from my patch I at least do it all the time now on RX and TX. The TX poison never shows though so currently I'm more suspicious about RX. The other idea I had was to just run check_bytes_and_report() often around the code, but haven't tried that yet. I'm also poking harware folks about some of the registers we use to better understand how DMA works here. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 22:35 ` Luis R. Rodriguez 2010-10-14 22:44 ` Ben Greear @ 2010-10-14 22:51 ` Luis R. Rodriguez 2010-10-14 23:19 ` Luis R. Rodriguez 1 sibling, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 22:51 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Ben Greear, linux-wireless, Luis Rodriguez On Thu, Oct 14, 2010 at 03:35:34PM -0700, Luis R. Rodriguez wrote: > On Thu, Oct 14, 2010 at 3:29 PM, Luis R. Rodriguez <mcgrof@gmail.com> wrote: > > Fun enough if I just create one monitor interface and loop quickly > > over some 2 GHz channels where I know I have traffic nearby I don't > > see the poison. So channel changes don't seem to do much because this > > is changing channels as fast as possible from userspace. I also can > > confirm that I see frames from the different channels as I move along. > > Even forcing a band change doesn't help trigger it with just one mon0 > and one regular device scanning in a loop; > > while true; do for i in 2412 5745 2417 5745 2422 5745 2427 5745 2432 > 5745 2442; do echo $i iw dev mon0 set freq $i; done; done > while true; do iw dev wlan0 scan; done OK so just so you know where I'm poking, this is what I have so far. The ath9k_hw_rxprocdesc() suggestion came from Jouni but it didn't seem to help. I'm disabling HT as I want to rule out things step by step. I haven't yet ruled out TX as haven't been able to trigger this poison yet just based on monitor interfaces and no frame TX's, I needed at probe requests sent by one STA. So the script I used was: #!/usr/bin/perl use strict; my $iw = "/usr/sbin/iw"; my $ip = "/sbin/ip"; my $phy = "phy0"; my $max = 300; my $i; my $cmd; # Create stations for ($i = 0; $i<$max; $i++) { runCmd("$iw phy $phy interface add mon$i type monitor"); runCmd("$ip link set dev mon$i up"); } sub runCmd { my $cmd = shift; print "$cmd\n"; `$cmd`; } And what I have on top of my tree right now, after your two new patches: I should note I never hit the WARN_ON() nor the printks, so that rules those out. diff --git a/drivers/net/wireless/ath/ath9k/init.c b/drivers/net/wireless/ath/ath9k/init.c index a4c5ed4..cd61727 100644 --- a/drivers/net/wireless/ath/ath9k/init.c +++ b/drivers/net/wireless/ath/ath9k/init.c @@ -192,6 +192,7 @@ static void setup_ht_cap(struct ath_softc *sc, int i, max_streams; ht_info->ht_supported = true; + ht_info->ht_supported = false; ht_info->cap = IEEE80211_HT_CAP_SUP_WIDTH_20_40 | IEEE80211_HT_CAP_SM_PS | IEEE80211_HT_CAP_SGI_40 | diff --git a/drivers/net/wireless/ath/ath9k/mac.c b/drivers/net/wireless/ath/ath9k/mac.c index 8c13479..a96327e 100644 --- a/drivers/net/wireless/ath/ath9k/mac.c +++ b/drivers/net/wireless/ath/ath9k/mac.c @@ -639,6 +639,10 @@ int ath9k_hw_rxprocdesc(struct ath_hw *ah, struct ath_desc *ds, if ((adsp->ds_rxstatus8 & AR_RxDone) == 0) return -EINPROGRESS; + ds->ds_data = 0; + ds->ds_vdata = 0; + wmb(); + ads.u.rx = adsp->u.rx; rs->rs_status = 0; diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c index bcd3892..b31b5fe 100644 --- a/drivers/net/wireless/ath/ath9k/main.c +++ b/drivers/net/wireless/ath/ath9k/main.c @@ -1243,6 +1243,10 @@ static int ath9k_tx(struct ieee80211_hw *hw, int padpos, padsize; struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data; int qnum; + struct sk_buff *tmp_skb; + + tmp_skb = skb_copy(skb, GFP_ATOMIC); + dev_kfree_skb_any(tmp_skb); if (aphy->state != ATH_WIPHY_ACTIVE && aphy->state != ATH_WIPHY_SCAN) { ath_print(common, ATH_DBG_XMIT, diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c index fe73fc5..8348199 100644 --- a/drivers/net/wireless/ath/ath9k/recv.c +++ b/drivers/net/wireless/ath/ath9k/recv.c @@ -502,6 +502,9 @@ int ath_startrecv(struct ath_softc *sc) goto start_recv; bf = list_first_entry(&sc->rx.rxbuf, struct ath_buf, list); + /* This is fishy, what if the bf->bf_daddr is not valid ? */ + if (!bf->bf_daddr) + printk("= hah bf->bf_daddr is 0!\n"); ath9k_hw_putrxbuf(ah, bf->bf_daddr); ath9k_hw_rxena(ah); @@ -663,6 +666,12 @@ static void ath_rx_send_to_mac80211(struct ieee80211_hw *hw, struct ieee80211_rx_status *rxs) { struct ieee80211_hdr *hdr; + struct sk_buff *tmp_skb; + + if (1) { + tmp_skb = skb_copy(skb, GFP_ATOMIC); + dev_kfree_skb_any(tmp_skb); + } hdr = (struct ieee80211_hdr *)skb->data; @@ -815,11 +821,17 @@ static struct ath_buf *ath_get_next_rx_buf(struct ath_softc *sc, ret = ath9k_hw_rxprocdesc(ah, tds, &trs, 0); if (ret == -EINPROGRESS) return NULL; + WARN_ON(1); } if (!bf->bf_mpdu) return bf; + if (!bf->bf_buf_addr) + printk("bf->bf_buf_addr = 0\n"); /* * Synchronize the DMA transfer with CPU before * 1. accessing the frame ^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 22:51 ` Luis R. Rodriguez @ 2010-10-14 23:19 ` Luis R. Rodriguez 2010-10-14 23:30 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 23:19 UTC (permalink / raw) To: Luis Rodriguez; +Cc: Luis R. Rodriguez, Ben Greear, linux-wireless Ok please try this patch, it cures it for me. Luis diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c index fe73fc5..e581b1f 100644 --- a/drivers/net/wireless/ath/ath9k/recv.c +++ b/drivers/net/wireless/ath/ath9k/recv.c @@ -306,10 +306,8 @@ static void ath_edma_start_recv(struct ath_softc *sc) static void ath_edma_stop_recv(struct ath_softc *sc) { - spin_lock_bh(&sc->rx.rxbuflock); ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP); ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP); - spin_unlock_bh(&sc->rx.rxbuflock); } int ath_rx_init(struct ath_softc *sc, int nbufs) @@ -518,6 +516,7 @@ bool ath_stoprecv(struct ath_softc *sc) struct ath_hw *ah = sc->sc_ah; bool stopped; + spin_lock_bh(&sc->rx.rxbuflock); ath9k_hw_stoppcurecv(ah); ath9k_hw_setrxfilter(ah, 0); stopped = ath9k_hw_stopdmarecv(ah); @@ -526,6 +525,7 @@ bool ath_stoprecv(struct ath_softc *sc) ath_edma_stop_recv(sc); else sc->rx.rxlink = NULL; + spin_unlock_bh(&sc->rx.rxbuflock); return stopped; } ^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 23:19 ` Luis R. Rodriguez @ 2010-10-14 23:30 ` Ben Greear 2010-10-14 23:39 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-14 23:30 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Luis Rodriguez, Luis R. Rodriguez, linux-wireless On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote: > Ok please try this patch, it cures it for me. Well, it got a lot further than normal, but it still hit the poison check after a few minutes. Current test case is my app loading 130 or so stations, each running wpa_supplicant. All were created, and quite a few had associated when the poison check hit. So, it definitely looks like a step in the right direction, but not fully fixed yet. I'll do some more testing with this patch applied and using just my perl script to make sure the problem is reproducible outside of my application. Thanks, Ben > > Luis > > diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c > index fe73fc5..e581b1f 100644 > --- a/drivers/net/wireless/ath/ath9k/recv.c > +++ b/drivers/net/wireless/ath/ath9k/recv.c > @@ -306,10 +306,8 @@ static void ath_edma_start_recv(struct ath_softc *sc) > > static void ath_edma_stop_recv(struct ath_softc *sc) > { > - spin_lock_bh(&sc->rx.rxbuflock); > ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP); > ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP); > - spin_unlock_bh(&sc->rx.rxbuflock); > } > > int ath_rx_init(struct ath_softc *sc, int nbufs) > @@ -518,6 +516,7 @@ bool ath_stoprecv(struct ath_softc *sc) > struct ath_hw *ah = sc->sc_ah; > bool stopped; > > + spin_lock_bh(&sc->rx.rxbuflock); > ath9k_hw_stoppcurecv(ah); > ath9k_hw_setrxfilter(ah, 0); > stopped = ath9k_hw_stopdmarecv(ah); > @@ -526,6 +525,7 @@ bool ath_stoprecv(struct ath_softc *sc) > ath_edma_stop_recv(sc); > else > sc->rx.rxlink = NULL; > + spin_unlock_bh(&sc->rx.rxbuflock); > > return stopped; > } -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 23:30 ` Ben Greear @ 2010-10-14 23:39 ` Luis R. Rodriguez 2010-10-14 23:48 ` Luis R. Rodriguez 2010-10-14 23:51 ` Ben Greear 0 siblings, 2 replies; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 23:39 UTC (permalink / raw) To: Ben Greear; +Cc: Luis Rodriguez, linux-wireless On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear <greearb@candelatech.com> wrote: > On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote: >> >> Ok please try this patch, it cures it for me. > > Well, it got a lot further than normal, but it still > hit the poison check after a few minutes. > > Current test case is my app loading 130 or so stations, each running > wpa_supplicant. All were created, and quite a few had associated > when the poison check hit. > > So, it definitely looks like a step in the right direction, but > not fully fixed yet. > > I'll do some more testing with this patch applied and using just my > perl script to make sure the problem is reproducible outside of my > application. Ok, whatever userspace does it should not corrupt to kernel, unless its poking /dev/mem Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 23:39 ` Luis R. Rodriguez @ 2010-10-14 23:48 ` Luis R. Rodriguez 2010-10-15 16:51 ` Ben Greear 2010-10-14 23:51 ` Ben Greear 1 sibling, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-14 23:48 UTC (permalink / raw) To: Ben Greear; +Cc: Luis Rodriguez, linux-wireless On Thu, Oct 14, 2010 at 04:39:45PM -0700, Luis R. Rodriguez wrote: > On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear <greearb@candelatech.com> wrote: > > On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote: > >> > >> Ok please try this patch, it cures it for me. > > > > Well, it got a lot further than normal, but it still > > hit the poison check after a few minutes. > > > > Current test case is my app loading 130 or so stations, each running > > wpa_supplicant. All were created, and quite a few had associated > > when the poison check hit. > > > > So, it definitely looks like a step in the right direction, but > > not fully fixed yet. > > > > I'll do some more testing with this patch applied and using just my > > perl script to make sure the problem is reproducible outside of my > > application. > > Ok, whatever userspace does it should not corrupt to kernel, unless > its poking /dev/mem Can also try this one instead, it will prevent any other instances we would not have caught on stopping and starting RX here. diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c index fe73fc5..db677c4 100644 --- a/drivers/net/wireless/ath/ath9k/recv.c +++ b/drivers/net/wireless/ath/ath9k/recv.c @@ -306,10 +306,8 @@ static void ath_edma_start_recv(struct ath_softc *sc) static void ath_edma_stop_recv(struct ath_softc *sc) { - spin_lock_bh(&sc->rx.rxbuflock); ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP); ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP); - spin_unlock_bh(&sc->rx.rxbuflock); } int ath_rx_init(struct ath_softc *sc, int nbufs) @@ -482,13 +480,14 @@ int ath_startrecv(struct ath_softc *sc) { struct ath_hw *ah = sc->sc_ah; struct ath_buf *bf, *tbf; + unsigned long flags; if (ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) { ath_edma_start_recv(sc); return 0; } - spin_lock_bh(&sc->rx.rxbuflock); + spin_lock_irqsave(&sc->rx.rxbuflock, flags); if (list_empty(&sc->rx.rxbuf)) goto start_recv; @@ -506,7 +505,7 @@ int ath_startrecv(struct ath_softc *sc) ath9k_hw_rxena(ah); start_recv: - spin_unlock_bh(&sc->rx.rxbuflock); + spin_unlock_irqrestore(&sc->rx.rxbuflock, flags); ath_opmode_init(sc); ath9k_hw_startpcureceive(ah, (sc->sc_flags & SC_OP_OFFCHANNEL)); @@ -517,7 +516,9 @@ bool ath_stoprecv(struct ath_softc *sc) { struct ath_hw *ah = sc->sc_ah; bool stopped; + unsigned long flags; + spin_lock_irqsave(&sc->rx.rxbuflock, flags); ath9k_hw_stoppcurecv(ah); ath9k_hw_setrxfilter(ah, 0); stopped = ath9k_hw_stopdmarecv(ah); @@ -526,6 +527,7 @@ bool ath_stoprecv(struct ath_softc *sc) ath_edma_stop_recv(sc); else sc->rx.rxlink = NULL; + spin_unlock_irqrestore(&sc->rx.rxbuflock, flags); return stopped; } ^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 23:48 ` Luis R. Rodriguez @ 2010-10-15 16:51 ` Ben Greear 2010-10-15 18:47 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-15 16:51 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Luis Rodriguez, linux-wireless On 10/14/2010 04:48 PM, Luis R. Rodriguez wrote: > On Thu, Oct 14, 2010 at 04:39:45PM -0700, Luis R. Rodriguez wrote: >> On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear<greearb@candelatech.com> wrote: >>> On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote: >>>> >>>> Ok please try this patch, it cures it for me. >>> >>> Well, it got a lot further than normal, but it still >>> hit the poison check after a few minutes. >>> >>> Current test case is my app loading 130 or so stations, each running >>> wpa_supplicant. All were created, and quite a few had associated >>> when the poison check hit. >>> >>> So, it definitely looks like a step in the right direction, but >>> not fully fixed yet. >>> >>> I'll do some more testing with this patch applied and using just my >>> perl script to make sure the problem is reproducible outside of my >>> application. >> >> Ok, whatever userspace does it should not corrupt to kernel, unless >> its poking /dev/mem > > Can also try this one instead, it will prevent any other instances > we would not have caught on stopping and starting RX here. It ran longer than before any of your locking patches (about 3 minutes), but it did hit the poison check. Before it did, I had a bunch of OOM errors trying to allocate skbs. I have 2GB of RAM on this system, but maybe it's not tuned properly, and not all of that can be used for networking on 32-bit kernels.... I have Felix's 3 ani patches from ~3 days ago applied, running 130 stations in WPA mode. I'm going to try running fewer to dodge the OOM case, but I have a few other things to take care of first. Thanks, Ben > > diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c > index fe73fc5..db677c4 100644 > --- a/drivers/net/wireless/ath/ath9k/recv.c > +++ b/drivers/net/wireless/ath/ath9k/recv.c > @@ -306,10 +306,8 @@ static void ath_edma_start_recv(struct ath_softc *sc) > > static void ath_edma_stop_recv(struct ath_softc *sc) > { > - spin_lock_bh(&sc->rx.rxbuflock); > ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP); > ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP); > - spin_unlock_bh(&sc->rx.rxbuflock); > } > > int ath_rx_init(struct ath_softc *sc, int nbufs) > @@ -482,13 +480,14 @@ int ath_startrecv(struct ath_softc *sc) > { > struct ath_hw *ah = sc->sc_ah; > struct ath_buf *bf, *tbf; > + unsigned long flags; > > if (ah->caps.hw_caps& ATH9K_HW_CAP_EDMA) { > ath_edma_start_recv(sc); > return 0; > } > > - spin_lock_bh(&sc->rx.rxbuflock); > + spin_lock_irqsave(&sc->rx.rxbuflock, flags); > if (list_empty(&sc->rx.rxbuf)) > goto start_recv; > > @@ -506,7 +505,7 @@ int ath_startrecv(struct ath_softc *sc) > ath9k_hw_rxena(ah); > > start_recv: > - spin_unlock_bh(&sc->rx.rxbuflock); > + spin_unlock_irqrestore(&sc->rx.rxbuflock, flags); > ath_opmode_init(sc); > ath9k_hw_startpcureceive(ah, (sc->sc_flags& SC_OP_OFFCHANNEL)); > > @@ -517,7 +516,9 @@ bool ath_stoprecv(struct ath_softc *sc) > { > struct ath_hw *ah = sc->sc_ah; > bool stopped; > + unsigned long flags; > > + spin_lock_irqsave(&sc->rx.rxbuflock, flags); > ath9k_hw_stoppcurecv(ah); > ath9k_hw_setrxfilter(ah, 0); > stopped = ath9k_hw_stopdmarecv(ah); > @@ -526,6 +527,7 @@ bool ath_stoprecv(struct ath_softc *sc) > ath_edma_stop_recv(sc); > else > sc->rx.rxlink = NULL; > + spin_unlock_irqrestore(&sc->rx.rxbuflock, flags); > > return stopped; > } -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 16:51 ` Ben Greear @ 2010-10-15 18:47 ` Luis R. Rodriguez 2010-10-15 19:36 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-15 18:47 UTC (permalink / raw) To: Ben Greear; +Cc: Luis Rodriguez, linux-wireless On Fri, Oct 15, 2010 at 9:51 AM, Ben Greear <greearb@candelatech.com> wrote: > On 10/14/2010 04:48 PM, Luis R. Rodriguez wrote: >> >> On Thu, Oct 14, 2010 at 04:39:45PM -0700, Luis R. Rodriguez wrote: >>> >>> On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear<greearb@candelatech.com> >>> wrote: >>>> >>>> On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote: >>>>> >>>>> Ok please try this patch, it cures it for me. >>>> >>>> Well, it got a lot further than normal, but it still >>>> hit the poison check after a few minutes. >>>> >>>> Current test case is my app loading 130 or so stations, each running >>>> wpa_supplicant. All were created, and quite a few had associated >>>> when the poison check hit. >>>> >>>> So, it definitely looks like a step in the right direction, but >>>> not fully fixed yet. >>>> >>>> I'll do some more testing with this patch applied and using just my >>>> perl script to make sure the problem is reproducible outside of my >>>> application. >>> >>> Ok, whatever userspace does it should not corrupt to kernel, unless >>> its poking /dev/mem >> >> Can also try this one instead, it will prevent any other instances >> we would not have caught on stopping and starting RX here. > > It ran longer than before any of your locking patches (about 3 minutes), but > it did hit the poison check. > > Before it did, I had a bunch of OOM errors trying to allocate > skbs. I have 2GB of RAM on this system, but maybe it's not tuned > properly, and not all of that can be used for networking on 32-bit > kernels.... > > I have Felix's 3 ani patches from ~3 days ago applied, running 130 stations > in WPA mode. > > I'm going to try running fewer to dodge the OOM case, > but I have a few other things to take care of first. Thanks for testing. So now I cannot reproduce the poison anymore, any other ideas what I can try? Does the perl script still give you poison or just with your Über proprietary application? Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 18:47 ` Luis R. Rodriguez @ 2010-10-15 19:36 ` Ben Greear 2010-10-15 21:07 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-15 19:36 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Luis Rodriguez, linux-wireless On 10/15/2010 11:47 AM, Luis R. Rodriguez wrote: > On Fri, Oct 15, 2010 at 9:51 AM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/14/2010 04:48 PM, Luis R. Rodriguez wrote: >>> >>> On Thu, Oct 14, 2010 at 04:39:45PM -0700, Luis R. Rodriguez wrote: >>>> >>>> On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear<greearb@candelatech.com> >>>> wrote: >>>>> >>>>> On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote: >>>>>> >>>>>> Ok please try this patch, it cures it for me. >>>>> >>>>> Well, it got a lot further than normal, but it still >>>>> hit the poison check after a few minutes. >>>>> >>>>> Current test case is my app loading 130 or so stations, each running >>>>> wpa_supplicant. All were created, and quite a few had associated >>>>> when the poison check hit. >>>>> >>>>> So, it definitely looks like a step in the right direction, but >>>>> not fully fixed yet. >>>>> >>>>> I'll do some more testing with this patch applied and using just my >>>>> perl script to make sure the problem is reproducible outside of my >>>>> application. >>>> >>>> Ok, whatever userspace does it should not corrupt to kernel, unless >>>> its poking /dev/mem >>> >>> Can also try this one instead, it will prevent any other instances >>> we would not have caught on stopping and starting RX here. >> >> It ran longer than before any of your locking patches (about 3 minutes), but >> it did hit the poison check. >> >> Before it did, I had a bunch of OOM errors trying to allocate >> skbs. I have 2GB of RAM on this system, but maybe it's not tuned >> properly, and not all of that can be used for networking on 32-bit >> kernels.... >> >> I have Felix's 3 ani patches from ~3 days ago applied, running 130 stations >> in WPA mode. >> >> I'm going to try running fewer to dodge the OOM case, >> but I have a few other things to take care of first. > > Thanks for testing. So now I cannot reproduce the poison anymore, any > other ideas what I can try? Does the perl script still give you poison > or just with your Über proprietary application? I was just writing that my script wouldn't reproduce it..but it did before I was done typing. Same looking poison exception as ever. I also notice my Trendnet AP is very un-happy with anything past around 30 STA devices associated, and according to it's status page..NONE of my STAs are associated, though things show up in /proc/net/wireless and I see auth/assoc messages in /var/log/messages on my STA system, so the AP may just be funky. On my system, most of the STAs are constantly trying to associate and being rejected by the AP. Here is updated script, creates 130 STAs and sleeps a bit between starting supplicants. It assumes you have a single PHY device, and if you do, it will use it's name regardless of how many times you reload the driver. Please forgive the lameness in the MAC creation logic..though it does at least appear to work :) Thanks, Ben #!/usr/bin/perl use strict; my $iw = "./local/sbin/iw"; my $ip = "./local/sbin/ip"; my $wpa_s = "./local/bin/wpa_supplicant"; my $ssid = "candela-n"; my $key = "wpadmz123"; my $phy = `cat /sys/class/ieee80211/*/name`; chomp($phy); my $max = 130; my $i; my $j = 4; my $k = 1; my $bmac = "00:01:02:03:"; my $cmd; # Create stations for ($i = 0; $i<$max; $i++) { runCmd("$iw phy $phy interface add sta$i type station"); if ($j > 99) { $j = 1; $k++; } my $mc5 = "$j"; if (length($mc5) == 1) { $mc5 = "0$mc5"; # pad mac octet } my $mc4 = "$k"; if (length($mc4) == 1) { $mc4 = "0$mc4"; # pad mac octet } $j++; my $mac = "$bmac$mc4:$mc5"; runCmd("$ip link set sta$i address $mac"); runCmd("$iw dev sta$i set power_save off"); } # Bring them up with WPA for ($i = 0; $i<$max; $i++) { open(FD, ">sta$i" . "_wpa.conf") || die("Couldn't open file: $!\n"); print FD " ctrl_interface=/var/run/wpa_supplicant fast_reauth=1 #can_scan_one=1 network={ ssid=\"$ssid\" proto=WPA key_mgmt=WPA-PSK psk=\"$key\" pairwise=TKIP CCMP group=TKIP CCMP } "; runCmd("$wpa_s -B -i sta$i -c sta$i" . "_wpa.conf -P sta$i" . "_wpa.pid -t -f sta$i" . "_wpa.log"); sleep(2); } sub runCmd { my $cmd = shift; print "$cmd\n"; `$cmd`; } > > Luis > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 19:36 ` Ben Greear @ 2010-10-15 21:07 ` Luis R. Rodriguez 2010-10-15 23:21 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-15 21:07 UTC (permalink / raw) To: Ben Greear; +Cc: Luis Rodriguez, linux-wireless On Fri, Oct 15, 2010 at 12:36:31PM -0700, Ben Greear wrote: > I was just writing that my script wouldn't reproduce it..but it did before I > was done typing. Same looking poison exception as ever. OK I was able to reproduce with the latest patch. > I also notice my Trendnet AP is very un-happy with anything past around 30 STA > devices associated, and according to it's status page..NONE of my STAs are associated, > though things show up in /proc/net/wireless and I see auth/assoc messages in > /var/log/messages on my STA system, so the AP may just be funky. Well we are stress testing the hell out of the AP too! > On my system, most of the STAs are constantly trying to associate and being > rejected by the AP. > > Here is updated script, creates 130 STAs and sleeps a bit between starting supplicants. > It assumes you have a single PHY device, and if you do, it will use it's name regardless > of how many times you reload the driver. > > Please forgive the lameness in the MAC creation logic..though it does at least appear > to work :) I'm now using this patch to force pressure: diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c index db677c4..2834c41 100644 --- a/drivers/net/wireless/ath/ath9k/recv.c +++ b/drivers/net/wireless/ath/ath9k/recv.c @@ -665,6 +665,13 @@ static void ath_rx_send_to_mac80211(struct ieee80211_hw *hw, struct ieee80211_rx_status *rxs) { struct ieee80211_hdr *hdr; + struct sk_buff *tmp_skb; + unsigned int i; + + for (i=0; i < 5; i++) { + tmp_skb = skb_copy(skb, GFP_ATOMIC); + dev_kfree_skb_any(tmp_skb); + } hdr = (struct ieee80211_hdr *)skb->data; And this script, slightly simplified mac address configuration. #!/usr/bin/perl use strict; my $iw = "/usr/sbin/iw"; my $ip = "/sbin/ip"; my $wpa_s = "/usr/local/sbin/wpa_supplicant"; my $ssid = "tesla-2g-bcm"; my $key = "stuff"; my $phy = "phy0"; my $max = 130; my $i; my $bmac = "00:01:02:03:04"; my $cmd; # Create stations for ($i = 0; $i<$max; $i++) { runCmd("$iw phy $phy interface add sta$i type station"); my $mc5 = sprintf("%02x", $i); my $mac = "$bmac:$mc5"; runCmd("$ip link set sta$i address $mac"); runCmd("$iw dev sta$i set power_save off"); } # Bring them up with WPA for ($i = 0; $i<$max; $i++) { open(FD, ">sta$i" . "_wpa.conf") || die("Couldn't open file: $!\n"); print FD " ctrl_interface=/var/run/wpa_supplicant fast_reauth=1 #can_scan_one=1 network={ ssid=\"$ssid\" proto=RSN key_mgmt=WPA-PSK psk=\"$key\" pairwise=CCMP group=CCMP } "; runCmd("$wpa_s -B -i sta$i -c sta$i" . "_wpa.conf -P sta$i" . "_wpa.pid -t "); sleep(2); } sub runCmd { my $cmd = shift; print "$cmd\n"; `$cmd`; } ^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 21:07 ` Luis R. Rodriguez @ 2010-10-15 23:21 ` Luis R. Rodriguez 2010-10-15 23:33 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-15 23:21 UTC (permalink / raw) To: Luis Rodriguez; +Cc: Ben Greear, linux-wireless Ben, please give this patch a shot. I addresses three races on the PCU: * When we were stopping the CPU for non-EDMA cards we never locked against anything starting the PCU again * ath9k_hw_startpcureceive() was being called without locking * Although we lock on the rxbuf lock for contention against starting/stopping the PCU, we also need to lock on the driver in locations where we start/stop the PCU within the same location otherwise we end up in inconsistant states and the hardware may end up proessing an incorrect buffer for DMA. To protect against this we use a new PCU lock on the main part of the driver to ensure each start/stop/reset operation is done atomically. And fixes one issue as a side effect: * No more packet loss on ping flood when you have one STA associated :) The only issue I see with this is I eventually run out of memory and my box becomes useless, unless I am mistaking that for some other issue. Please give this a shot and if it cures your woes I'll split it up into 3 separate patches, or maybe just two, one for the first two and one for the last issue. diff --git a/drivers/net/wireless/ath/ath9k/ath9k.h b/drivers/net/wireless/ath/ath9k/ath9k.h index 0c917e5..efee63c 100644 --- a/drivers/net/wireless/ath/ath9k/ath9k.h +++ b/drivers/net/wireless/ath/ath9k/ath9k.h @@ -312,6 +312,7 @@ struct ath_rx { unsigned int rxfilter; spinlock_t rxflushlock; spinlock_t rxbuflock; + spinlock_t pcu_lock; struct list_head rxbuf; struct ath_descdma rxdma; struct ath_buf *rx_bufptr; diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c index 62294da..6fbfe28 100644 --- a/drivers/net/wireless/ath/ath9k/main.c +++ b/drivers/net/wireless/ath/ath9k/main.c @@ -251,6 +251,9 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw, */ ath9k_hw_set_interrupts(ah, 0); ath_drain_all_txq(sc, false); + + spin_lock_bh(&sc->rx.pcu_lock); + stopped = ath_stoprecv(sc); /* XXX: do not flush receive queue here. We don't want @@ -278,6 +281,7 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw, "reset status %d\n", channel->center_freq, r); spin_unlock_bh(&sc->sc_resetlock); + spin_unlock_bh(&sc->rx.pcu_lock); goto ps_restore; } spin_unlock_bh(&sc->sc_resetlock); @@ -286,9 +290,12 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw, ath_print(common, ATH_DBG_FATAL, "Unable to restart recv logic\n"); r = -EIO; + spin_unlock_bh(&sc->rx.pcu_lock); goto ps_restore; } + spin_unlock_bh(&sc->rx.pcu_lock); + ath_cache_conf_rate(sc, &hw->conf); ath_update_txpow(sc); ath9k_hw_set_interrupts(ah, ah->imask); @@ -626,12 +633,16 @@ void ath9k_tasklet(unsigned long data) if (status & rxmask) { spin_lock_bh(&sc->rx.rxflushlock); + spin_lock_bh(&sc->rx.pcu_lock); + /* Check for high priority Rx first */ if ((ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) && (status & ATH9K_INT_RXHP)) ath_rx_tasklet(sc, 0, true); ath_rx_tasklet(sc, 0, false); + spin_unlock_bh(&sc->rx.pcu_lock); + spin_unlock_bh(&sc->rx.rxflushlock); } @@ -887,6 +898,7 @@ void ath_radio_enable(struct ath_softc *sc, struct ieee80211_hw *hw) if (!ah->curchan) ah->curchan = ath_get_curchannel(sc, sc->hw); + spin_lock_bh(&sc->rx.pcu_lock); spin_lock_bh(&sc->sc_resetlock); r = ath9k_hw_reset(ah, ah->curchan, ah->caldata, false); if (r) { @@ -901,8 +913,10 @@ void ath_radio_enable(struct ath_softc *sc, struct ieee80211_hw *hw) if (ath_startrecv(sc) != 0) { ath_print(common, ATH_DBG_FATAL, "Unable to restart recv logic\n"); + spin_unlock_bh(&sc->rx.pcu_lock); return; } + spin_unlock_bh(&sc->rx.pcu_lock); if (sc->sc_flags & SC_OP_BEACONS) ath_beacon_config(sc, NULL); /* restart beacons */ @@ -941,6 +955,9 @@ void ath_radio_disable(struct ath_softc *sc, struct ieee80211_hw *hw) ath9k_hw_set_interrupts(ah, 0); ath_drain_all_txq(sc, false); /* clear pending tx frames */ + + spin_lock_bh(&sc->rx.pcu_lock); + ath_stoprecv(sc); /* turn off frame recv */ ath_flushrecv(sc); /* flush recv queue */ @@ -958,6 +975,9 @@ void ath_radio_disable(struct ath_softc *sc, struct ieee80211_hw *hw) spin_unlock_bh(&sc->sc_resetlock); ath9k_hw_phy_disable(ah); + + spin_unlock_bh(&sc->rx.pcu_lock); + ath9k_hw_configpcipowersave(ah, 1, 1); ath9k_ps_restore(sc); ath9k_setpower(sc, ATH9K_PM_FULL_SLEEP); @@ -977,6 +997,9 @@ int ath_reset(struct ath_softc *sc, bool retry_tx) ath9k_hw_set_interrupts(ah, 0); ath_drain_all_txq(sc, retry_tx); + + spin_lock_bh(&sc->rx.pcu_lock); + ath_stoprecv(sc); ath_flushrecv(sc); @@ -991,6 +1014,8 @@ int ath_reset(struct ath_softc *sc, bool retry_tx) ath_print(common, ATH_DBG_FATAL, "Unable to start recv logic\n"); + spin_unlock_bh(&sc->rx.pcu_lock); + /* * We may be doing a reset in response to a request * that changes the channel so update any state that @@ -1155,6 +1180,7 @@ static int ath9k_start(struct ieee80211_hw *hw) * be followed by initialization of the appropriate bits * and then setup of the interrupt mask. */ + spin_lock_bh(&sc->rx.pcu_lock); spin_lock_bh(&sc->sc_resetlock); r = ath9k_hw_reset(ah, init_channel, ah->caldata, false); if (r) { @@ -1163,6 +1189,7 @@ static int ath9k_start(struct ieee80211_hw *hw) "(freq %u MHz)\n", r, curchan->center_freq); spin_unlock_bh(&sc->sc_resetlock); + spin_unlock_bh(&sc->rx.pcu_lock); goto mutex_unlock; } spin_unlock_bh(&sc->sc_resetlock); @@ -1184,8 +1211,10 @@ static int ath9k_start(struct ieee80211_hw *hw) ath_print(common, ATH_DBG_FATAL, "Unable to start recv logic\n"); r = -EIO; + spin_unlock_bh(&sc->rx.pcu_lock); goto mutex_unlock; } + spin_unlock_bh(&sc->rx.pcu_lock); /* Setup our intr mask. */ ah->imask = ATH9K_INT_TX | ATH9K_INT_RXEOL | @@ -1386,12 +1415,14 @@ static void ath9k_stop(struct ieee80211_hw *hw) * before setting the invalid flag. */ ath9k_hw_set_interrupts(ah, 0); + spin_lock_bh(&sc->rx.pcu_lock); if (!(sc->sc_flags & SC_OP_INVALID)) { ath_drain_all_txq(sc, false); ath_stoprecv(sc); ath9k_hw_phy_disable(ah); } else sc->rx.rxlink = NULL; + spin_unlock_bh(&sc->rx.pcu_lock); /* disable HAL and put h/w to sleep */ ath9k_hw_disable(ah); diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c index fe73fc5..20025a5 100644 --- a/drivers/net/wireless/ath/ath9k/recv.c +++ b/drivers/net/wireless/ath/ath9k/recv.c @@ -297,19 +297,17 @@ static void ath_edma_start_recv(struct ath_softc *sc) ath_rx_addbuffer_edma(sc, ATH9K_RX_QUEUE_LP, sc->rx.rx_edma[ATH9K_RX_QUEUE_LP].rx_fifo_hwsize); - spin_unlock_bh(&sc->rx.rxbuflock); - ath_opmode_init(sc); ath9k_hw_startpcureceive(sc->sc_ah, (sc->sc_flags & SC_OP_OFFCHANNEL)); + + spin_unlock_bh(&sc->rx.rxbuflock); } static void ath_edma_stop_recv(struct ath_softc *sc) { - spin_lock_bh(&sc->rx.rxbuflock); ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP); ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP); - spin_unlock_bh(&sc->rx.rxbuflock); } int ath_rx_init(struct ath_softc *sc, int nbufs) @@ -321,6 +319,7 @@ int ath_rx_init(struct ath_softc *sc, int nbufs) spin_lock_init(&sc->rx.rxflushlock); sc->sc_flags &= ~SC_OP_RXFLUSH; + spin_lock_init(&sc->rx.pcu_lock); spin_lock_init(&sc->rx.rxbuflock); if (sc->sc_ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) { @@ -506,9 +505,9 @@ int ath_startrecv(struct ath_softc *sc) ath9k_hw_rxena(ah); start_recv: - spin_unlock_bh(&sc->rx.rxbuflock); ath_opmode_init(sc); ath9k_hw_startpcureceive(ah, (sc->sc_flags & SC_OP_OFFCHANNEL)); + spin_unlock_bh(&sc->rx.rxbuflock); return 0; } @@ -518,6 +517,7 @@ bool ath_stoprecv(struct ath_softc *sc) struct ath_hw *ah = sc->sc_ah; bool stopped; + spin_lock_bh(&sc->rx.rxbuflock); ath9k_hw_stoppcurecv(ah); ath9k_hw_setrxfilter(ah, 0); stopped = ath9k_hw_stopdmarecv(ah); @@ -526,7 +526,9 @@ bool ath_stoprecv(struct ath_softc *sc) ath_edma_stop_recv(sc); else sc->rx.rxlink = NULL; + spin_unlock_bh(&sc->rx.rxbuflock); + WARN_ON(!stopped); return stopped; } @@ -663,6 +665,13 @@ static void ath_rx_send_to_mac80211(struct ieee80211_hw *hw, struct ieee80211_rx_status *rxs) { struct ieee80211_hdr *hdr; + struct sk_buff *tmp_skb; + unsigned int j; + + for (j=0; j < 5; j++) { + tmp_skb = skb_copy(skb, GFP_ATOMIC); + dev_kfree_skb_any(tmp_skb); + } hdr = (struct ieee80211_hdr *)skb->data; ^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 23:21 ` Luis R. Rodriguez @ 2010-10-15 23:33 ` Ben Greear 2010-10-15 23:38 ` Luis R. Rodriguez 2010-10-15 23:39 ` Ben Greear 0 siblings, 2 replies; 84+ messages in thread From: Ben Greear @ 2010-10-15 23:33 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Luis Rodriguez, linux-wireless On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote: > Ben, please give this patch a shot. I addresses three races on the PCU: > > * When we were stopping the CPU for non-EDMA cards we never locked against > anything starting the PCU again > > * ath9k_hw_startpcureceive() was being called without locking > > * Although we lock on the rxbuf lock for contention against starting/stopping > the PCU, we also need to lock on the driver in locations where we start/stop > the PCU within the same location otherwise we end up in inconsistant states > and the hardware may end up proessing an incorrect buffer for DMA. To > protect against this we use a new PCU lock on the main part of the driver to > ensure each start/stop/reset operation is done atomically. > > And fixes one issue as a side effect: > > * No more packet loss on ping flood when you have one STA associated :) > > The only issue I see with this is I eventually run out of memory and my box > becomes useless, unless I am mistaking that for some other issue. > > Please give this a shot and if it cures your woes I'll split it up into > 3 separate patches, or maybe just two, one for the first two and one for > the last issue. Sounds good, but this lockdep splat happens almost immediately upon starting my app: ======================================================= [ INFO: possible circular locking dependency detected ] 2.6.36-rc8-wl+ #32 ------------------------------------------------------- swapper/0 is trying to acquire lock: (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k] but task is already holding lock: (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k] which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}: [<c0457639>] lock_acquire+0x5a/0x78 [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k] [<fa16dda2>] ath_radio_disable+0x83/0x143 [ath9k] [<fa16e370>] ath9k_config+0x3c3/0x3d8 [ath9k] [<fa09ca2e>] ieee80211_hw_config+0x11b/0x125 [mac80211] [<fa0a8edf>] ieee80211_do_open+0x3c5/0x466 [mac80211] [<fa0a8fdb>] ieee80211_open+0x5b/0x5e [mac80211] [<c06ce76b>] __dev_open+0x80/0xae [<c06cc99b>] __dev_change_flags+0xa0/0x115 [<c06ce6bf>] dev_change_flags+0x13/0x3f [<c06d7e78>] do_setlink+0x23a/0x51b [<c06d847c>] rtnl_newlink+0x269/0x431 [<c06d79e2>] rtnetlink_rcv_msg+0x182/0x198 [<c06e503c>] netlink_rcv_skb+0x30/0x77 [<c06d7859>] rtnetlink_rcv+0x1b/0x22 [<c06e4e77>] netlink_unicast+0xbe/0x119 [<c06e5a15>] netlink_sendmsg+0x234/0x24c [<c06bf93a>] __sock_sendmsg+0x51/0x5a [<c06bfba4>] sock_sendmsg+0x93/0xa7 [<c06bfd8c>] sys_sendmsg+0x149/0x193 [<c06c148b>] sys_socketcall+0x15e/0x1a5 [<c0402f1c>] sysenter_do_call+0x12/0x38 -> #0 (&(&sc->rx.pcu_lock)->rlock){+.-...}: [<c0457374>] __lock_acquire+0x921/0xb8c [<c0457639>] lock_acquire+0x5a/0x78 [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k] [<c0438fd1>] tasklet_action+0x73/0xc6 [<c043945f>] __do_softirq+0x86/0x111 [<c0439520>] do_softirq+0x36/0x5a [<c0439659>] irq_exit+0x35/0x69 [<c0403fb9>] do_IRQ+0x86/0x9a [<c04034ee>] common_interrupt+0x2e/0x40 [<c040227f>] cpu_idle+0x4e/0x6b [<c074b6e9>] rest_init+0x8d/0x92 [<c09758ea>] start_kernel+0x320/0x325 [<c09750d0>] i386_start_kernel+0xd0/0xd7 other info that might help us debug this: 1 lock held by swapper/0: #0: (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k] stack backtrace: Pid: 0, comm: swapper Not tainted 2.6.36-rc8-wl+ #32 Call Trace: [<c075d940>] ? printk+0xf/0x17 [<c04565af>] print_circular_bug+0x91/0x9d [<c0457374>] __lock_acquire+0x921/0xb8c [<c0457639>] lock_acquire+0x5a/0x78 [<fa16e5c7>] ? ath9k_tasklet+0x7e/0x140 [ath9k] [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f [<fa16e5c7>] ? ath9k_tasklet+0x7e/0x140 [ath9k] [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k] [<c0438fd1>] tasklet_action+0x73/0xc6 [<c043945f>] __do_softirq+0x86/0x111 [<c0439520>] do_softirq+0x36/0x5a [<c0439659>] irq_exit+0x35/0x69 [<c0403fb9>] do_IRQ+0x86/0x9a [<c04034ee>] common_interrupt+0x2e/0x40 [<c045007b>] ? do_adjtimex+0x223/0x55e [<c0408245>] ? mwait_idle+0x5c/0x6c [<c040227f>] cpu_idle+0x4e/0x6b [<c074b6e9>] rest_init+0x8d/0x92 [<c09758ea>] start_kernel+0x320/0x325 [<c09750d0>] i386_start_kernel+0xd0/0xd7 -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 23:33 ` Ben Greear @ 2010-10-15 23:38 ` Luis R. Rodriguez 2010-10-15 23:41 ` Luis R. Rodriguez 2010-10-15 23:42 ` Ben Greear 2010-10-15 23:39 ` Ben Greear 1 sibling, 2 replies; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-15 23:38 UTC (permalink / raw) To: Ben Greear; +Cc: Luis Rodriguez, linux-wireless On Fri, Oct 15, 2010 at 04:33:50PM -0700, Ben Greear wrote: > On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote: > > Ben, please give this patch a shot. I addresses three races on the PCU: > > > > * When we were stopping the CPU for non-EDMA cards we never locked against > > anything starting the PCU again > > > > * ath9k_hw_startpcureceive() was being called without locking > > > > * Although we lock on the rxbuf lock for contention against starting/stopping > > the PCU, we also need to lock on the driver in locations where we start/stop > > the PCU within the same location otherwise we end up in inconsistant states > > and the hardware may end up proessing an incorrect buffer for DMA. To > > protect against this we use a new PCU lock on the main part of the driver to > > ensure each start/stop/reset operation is done atomically. > > > > And fixes one issue as a side effect: > > > > * No more packet loss on ping flood when you have one STA associated :) > > > > The only issue I see with this is I eventually run out of memory and my box > > becomes useless, unless I am mistaking that for some other issue. > > > > Please give this a shot and if it cures your woes I'll split it up into > > 3 separate patches, or maybe just two, one for the first two and one for > > the last issue. > > Sounds good, but this lockdep splat happens almost immediately upon starting > my app: > > ======================================================= > [ INFO: possible circular locking dependency detected ] > 2.6.36-rc8-wl+ #32 > ------------------------------------------------------- > swapper/0 is trying to acquire lock: > (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k] > > but task is already holding lock: > (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k] > > which lock already depends on the new lock. > > > the existing dependency chain (in reverse order) is: > > -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}: > [<c0457639>] lock_acquire+0x5a/0x78 > [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f > [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k] Ah we just need to nuke the flush lock, one second. Also remove my skb_copy() otherwise you will really run out of memory quickly, unless you really want to test with it :) Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 23:38 ` Luis R. Rodriguez @ 2010-10-15 23:41 ` Luis R. Rodriguez 2010-10-16 0:07 ` Ben Greear 2010-10-15 23:42 ` Ben Greear 1 sibling, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-15 23:41 UTC (permalink / raw) To: Luis Rodriguez; +Cc: Ben Greear, linux-wireless On Fri, Oct 15, 2010 at 04:38:14PM -0700, Luis Rodriguez wrote: > On Fri, Oct 15, 2010 at 04:33:50PM -0700, Ben Greear wrote: > > On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote: > > > Ben, please give this patch a shot. I addresses three races on the PCU: > > > > > > * When we were stopping the CPU for non-EDMA cards we never locked against > > > anything starting the PCU again > > > > > > * ath9k_hw_startpcureceive() was being called without locking > > > > > > * Although we lock on the rxbuf lock for contention against starting/stopping > > > the PCU, we also need to lock on the driver in locations where we start/stop > > > the PCU within the same location otherwise we end up in inconsistant states > > > and the hardware may end up proessing an incorrect buffer for DMA. To > > > protect against this we use a new PCU lock on the main part of the driver to > > > ensure each start/stop/reset operation is done atomically. > > > > > > And fixes one issue as a side effect: > > > > > > * No more packet loss on ping flood when you have one STA associated :) > > > > > > The only issue I see with this is I eventually run out of memory and my box > > > becomes useless, unless I am mistaking that for some other issue. > > > > > > Please give this a shot and if it cures your woes I'll split it up into > > > 3 separate patches, or maybe just two, one for the first two and one for > > > the last issue. > > > > Sounds good, but this lockdep splat happens almost immediately upon starting > > my app: > > > > ======================================================= > > [ INFO: possible circular locking dependency detected ] > > 2.6.36-rc8-wl+ #32 > > ------------------------------------------------------- > > swapper/0 is trying to acquire lock: > > (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k] > > > > but task is already holding lock: > > (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k] > > > > which lock already depends on the new lock. > > > > > > the existing dependency chain (in reverse order) is: > > > > -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}: > > [<c0457639>] lock_acquire+0x5a/0x78 > > [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f > > [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k] > > Ah we just need to nuke the flush lock, one second. Also remove my > skb_copy() otherwise you will really run out of memory quickly, > unless you really want to test with it :) Try this new one, note I if (0)'d the skb_copy() set that to if (1) if you want to force memory clobber. diff --git a/drivers/net/wireless/ath/ath9k/ath9k.h b/drivers/net/wireless/ath/ath9k/ath9k.h index 0c917e5..5dc5421 100644 --- a/drivers/net/wireless/ath/ath9k/ath9k.h +++ b/drivers/net/wireless/ath/ath9k/ath9k.h @@ -310,8 +310,8 @@ struct ath_rx { u8 rxotherant; u32 *rxlink; unsigned int rxfilter; - spinlock_t rxflushlock; spinlock_t rxbuflock; + spinlock_t pcu_lock; struct list_head rxbuf; struct ath_descdma rxdma; struct ath_buf *rx_bufptr; diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c index 62294da..b06f074 100644 --- a/drivers/net/wireless/ath/ath9k/main.c +++ b/drivers/net/wireless/ath/ath9k/main.c @@ -251,6 +251,9 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw, */ ath9k_hw_set_interrupts(ah, 0); ath_drain_all_txq(sc, false); + + spin_lock_bh(&sc->rx.pcu_lock); + stopped = ath_stoprecv(sc); /* XXX: do not flush receive queue here. We don't want @@ -278,6 +281,7 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw, "reset status %d\n", channel->center_freq, r); spin_unlock_bh(&sc->sc_resetlock); + spin_unlock_bh(&sc->rx.pcu_lock); goto ps_restore; } spin_unlock_bh(&sc->sc_resetlock); @@ -286,9 +290,12 @@ int ath_set_channel(struct ath_softc *sc, struct ieee80211_hw *hw, ath_print(common, ATH_DBG_FATAL, "Unable to restart recv logic\n"); r = -EIO; + spin_unlock_bh(&sc->rx.pcu_lock); goto ps_restore; } + spin_unlock_bh(&sc->rx.pcu_lock); + ath_cache_conf_rate(sc, &hw->conf); ath_update_txpow(sc); ath9k_hw_set_interrupts(ah, ah->imask); @@ -624,7 +631,7 @@ void ath9k_tasklet(unsigned long data) rxmask = (ATH9K_INT_RX | ATH9K_INT_RXEOL | ATH9K_INT_RXORN); if (status & rxmask) { - spin_lock_bh(&sc->rx.rxflushlock); + spin_lock_bh(&sc->rx.pcu_lock); /* Check for high priority Rx first */ if ((ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) && @@ -632,7 +639,7 @@ void ath9k_tasklet(unsigned long data) ath_rx_tasklet(sc, 0, true); ath_rx_tasklet(sc, 0, false); - spin_unlock_bh(&sc->rx.rxflushlock); + spin_unlock_bh(&sc->rx.pcu_lock); } if (status & ATH9K_INT_TX) { @@ -887,6 +894,7 @@ void ath_radio_enable(struct ath_softc *sc, struct ieee80211_hw *hw) if (!ah->curchan) ah->curchan = ath_get_curchannel(sc, sc->hw); + spin_lock_bh(&sc->rx.pcu_lock); spin_lock_bh(&sc->sc_resetlock); r = ath9k_hw_reset(ah, ah->curchan, ah->caldata, false); if (r) { @@ -901,8 +909,10 @@ void ath_radio_enable(struct ath_softc *sc, struct ieee80211_hw *hw) if (ath_startrecv(sc) != 0) { ath_print(common, ATH_DBG_FATAL, "Unable to restart recv logic\n"); + spin_unlock_bh(&sc->rx.pcu_lock); return; } + spin_unlock_bh(&sc->rx.pcu_lock); if (sc->sc_flags & SC_OP_BEACONS) ath_beacon_config(sc, NULL); /* restart beacons */ @@ -941,6 +951,9 @@ void ath_radio_disable(struct ath_softc *sc, struct ieee80211_hw *hw) ath9k_hw_set_interrupts(ah, 0); ath_drain_all_txq(sc, false); /* clear pending tx frames */ + + spin_lock_bh(&sc->rx.pcu_lock); + ath_stoprecv(sc); /* turn off frame recv */ ath_flushrecv(sc); /* flush recv queue */ @@ -958,6 +971,9 @@ void ath_radio_disable(struct ath_softc *sc, struct ieee80211_hw *hw) spin_unlock_bh(&sc->sc_resetlock); ath9k_hw_phy_disable(ah); + + spin_unlock_bh(&sc->rx.pcu_lock); + ath9k_hw_configpcipowersave(ah, 1, 1); ath9k_ps_restore(sc); ath9k_setpower(sc, ATH9K_PM_FULL_SLEEP); @@ -977,6 +993,9 @@ int ath_reset(struct ath_softc *sc, bool retry_tx) ath9k_hw_set_interrupts(ah, 0); ath_drain_all_txq(sc, retry_tx); + + spin_lock_bh(&sc->rx.pcu_lock); + ath_stoprecv(sc); ath_flushrecv(sc); @@ -991,6 +1010,8 @@ int ath_reset(struct ath_softc *sc, bool retry_tx) ath_print(common, ATH_DBG_FATAL, "Unable to start recv logic\n"); + spin_unlock_bh(&sc->rx.pcu_lock); + /* * We may be doing a reset in response to a request * that changes the channel so update any state that @@ -1155,6 +1176,7 @@ static int ath9k_start(struct ieee80211_hw *hw) * be followed by initialization of the appropriate bits * and then setup of the interrupt mask. */ + spin_lock_bh(&sc->rx.pcu_lock); spin_lock_bh(&sc->sc_resetlock); r = ath9k_hw_reset(ah, init_channel, ah->caldata, false); if (r) { @@ -1163,6 +1185,7 @@ static int ath9k_start(struct ieee80211_hw *hw) "(freq %u MHz)\n", r, curchan->center_freq); spin_unlock_bh(&sc->sc_resetlock); + spin_unlock_bh(&sc->rx.pcu_lock); goto mutex_unlock; } spin_unlock_bh(&sc->sc_resetlock); @@ -1184,8 +1207,10 @@ static int ath9k_start(struct ieee80211_hw *hw) ath_print(common, ATH_DBG_FATAL, "Unable to start recv logic\n"); r = -EIO; + spin_unlock_bh(&sc->rx.pcu_lock); goto mutex_unlock; } + spin_unlock_bh(&sc->rx.pcu_lock); /* Setup our intr mask. */ ah->imask = ATH9K_INT_TX | ATH9K_INT_RXEOL | @@ -1386,12 +1411,14 @@ static void ath9k_stop(struct ieee80211_hw *hw) * before setting the invalid flag. */ ath9k_hw_set_interrupts(ah, 0); + spin_lock_bh(&sc->rx.pcu_lock); if (!(sc->sc_flags & SC_OP_INVALID)) { ath_drain_all_txq(sc, false); ath_stoprecv(sc); ath9k_hw_phy_disable(ah); } else sc->rx.rxlink = NULL; + spin_unlock_bh(&sc->rx.pcu_lock); /* disable HAL and put h/w to sleep */ ath9k_hw_disable(ah); diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c index fe73fc5..128035c 100644 --- a/drivers/net/wireless/ath/ath9k/recv.c +++ b/drivers/net/wireless/ath/ath9k/recv.c @@ -297,19 +297,17 @@ static void ath_edma_start_recv(struct ath_softc *sc) ath_rx_addbuffer_edma(sc, ATH9K_RX_QUEUE_LP, sc->rx.rx_edma[ATH9K_RX_QUEUE_LP].rx_fifo_hwsize); - spin_unlock_bh(&sc->rx.rxbuflock); - ath_opmode_init(sc); ath9k_hw_startpcureceive(sc->sc_ah, (sc->sc_flags & SC_OP_OFFCHANNEL)); + + spin_unlock_bh(&sc->rx.rxbuflock); } static void ath_edma_stop_recv(struct ath_softc *sc) { - spin_lock_bh(&sc->rx.rxbuflock); ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_HP); ath_rx_remove_buffer(sc, ATH9K_RX_QUEUE_LP); - spin_unlock_bh(&sc->rx.rxbuflock); } int ath_rx_init(struct ath_softc *sc, int nbufs) @@ -319,8 +317,8 @@ int ath_rx_init(struct ath_softc *sc, int nbufs) struct ath_buf *bf; int error = 0; - spin_lock_init(&sc->rx.rxflushlock); sc->sc_flags &= ~SC_OP_RXFLUSH; + spin_lock_init(&sc->rx.pcu_lock); spin_lock_init(&sc->rx.rxbuflock); if (sc->sc_ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) { @@ -506,9 +504,9 @@ int ath_startrecv(struct ath_softc *sc) ath9k_hw_rxena(ah); start_recv: - spin_unlock_bh(&sc->rx.rxbuflock); ath_opmode_init(sc); ath9k_hw_startpcureceive(ah, (sc->sc_flags & SC_OP_OFFCHANNEL)); + spin_unlock_bh(&sc->rx.rxbuflock); return 0; } @@ -518,6 +516,7 @@ bool ath_stoprecv(struct ath_softc *sc) struct ath_hw *ah = sc->sc_ah; bool stopped; + spin_lock_bh(&sc->rx.rxbuflock); ath9k_hw_stoppcurecv(ah); ath9k_hw_setrxfilter(ah, 0); stopped = ath9k_hw_stopdmarecv(ah); @@ -526,19 +525,19 @@ bool ath_stoprecv(struct ath_softc *sc) ath_edma_stop_recv(sc); else sc->rx.rxlink = NULL; + spin_unlock_bh(&sc->rx.rxbuflock); + WARN_ON(!stopped); return stopped; } void ath_flushrecv(struct ath_softc *sc) { - spin_lock_bh(&sc->rx.rxflushlock); sc->sc_flags |= SC_OP_RXFLUSH; if (sc->sc_ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) ath_rx_tasklet(sc, 1, true); ath_rx_tasklet(sc, 1, false); sc->sc_flags &= ~SC_OP_RXFLUSH; - spin_unlock_bh(&sc->rx.rxflushlock); } static bool ath_beacon_dtim_pending_cab(struct sk_buff *skb) @@ -663,6 +662,15 @@ static void ath_rx_send_to_mac80211(struct ieee80211_hw *hw, struct ieee80211_rx_status *rxs) { struct ieee80211_hdr *hdr; + struct sk_buff *tmp_skb; + unsigned int j; + + if (0) { + for (j=0; j < 5; j++) { + tmp_skb = skb_copy(skb, GFP_ATOMIC); + dev_kfree_skb_any(tmp_skb); + } + } hdr = (struct ieee80211_hdr *)skb->data; ^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 23:41 ` Luis R. Rodriguez @ 2010-10-16 0:07 ` Ben Greear 0 siblings, 0 replies; 84+ messages in thread From: Ben Greear @ 2010-10-16 0:07 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Luis Rodriguez, linux-wireless On 10/15/2010 04:41 PM, Luis R. Rodriguez wrote: > On Fri, Oct 15, 2010 at 04:38:14PM -0700, Luis Rodriguez wrote: >> On Fri, Oct 15, 2010 at 04:33:50PM -0700, Ben Greear wrote: >>> On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote: >>>> Ben, please give this patch a shot. I addresses three races on the PCU: >>>> >>>> * When we were stopping the CPU for non-EDMA cards we never locked against >>>> anything starting the PCU again >>>> >>>> * ath9k_hw_startpcureceive() was being called without locking >>>> >>>> * Although we lock on the rxbuf lock for contention against starting/stopping >>>> the PCU, we also need to lock on the driver in locations where we start/stop >>>> the PCU within the same location otherwise we end up in inconsistant states >>>> and the hardware may end up proessing an incorrect buffer for DMA. To >>>> protect against this we use a new PCU lock on the main part of the driver to >>>> ensure each start/stop/reset operation is done atomically. >>>> >>>> And fixes one issue as a side effect: >>>> >>>> * No more packet loss on ping flood when you have one STA associated :) >>>> >>>> The only issue I see with this is I eventually run out of memory and my box >>>> becomes useless, unless I am mistaking that for some other issue. >>>> >>>> Please give this a shot and if it cures your woes I'll split it up into >>>> 3 separate patches, or maybe just two, one for the first two and one for >>>> the last issue. >>> >>> Sounds good, but this lockdep splat happens almost immediately upon starting >>> my app: >>> >>> ======================================================= >>> [ INFO: possible circular locking dependency detected ] >>> 2.6.36-rc8-wl+ #32 >>> ------------------------------------------------------- >>> swapper/0 is trying to acquire lock: >>> (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k] >>> >>> but task is already holding lock: >>> (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k] >>> >>> which lock already depends on the new lock. >>> >>> >>> the existing dependency chain (in reverse order) is: >>> >>> -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}: >>> [<c0457639>] lock_acquire+0x5a/0x78 >>> [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f >>> [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k] >> >> Ah we just need to nuke the flush lock, one second. Also remove my >> skb_copy() otherwise you will really run out of memory quickly, >> unless you really want to test with it :) > > Try this new one, note I if (0)'d the skb_copy() set that to if (1) if you want > to force memory clobber. Ahh, getting better. Ran for around 10 minutes with my app, never saw the poison, but system got into a bad state. It could be some other bug exposed by my test that was previously hidden by the poison problem, or maybe still bugs with the locking. First, I saw this..then things seemed to recover for a bit.. I've seen these before... ieee80211 phy0: sta85: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 ieee80211 phy0: sta73: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 ieee80211 phy0: sta109: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 ath: Failed to stop TX DMA in 100 msec after killing last frame ath: Failed to stop TX DMA. Resetting hardware! ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 sta29: authenticate with 00:14:d1:c6:d2:54 (try 1) sta38: authenticate with 00:14:d1:c6:d2:54 (try 1) Then a little later, hardware seems to go bad and won't recover. Interesting that the e1000e driver is also complaining..and I seem to have lost network connectivity to the system... Oct 15 16:54:38 localhost kernel: ieee80211 phy0: sta131: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 Oct 15 16:54:38 localhost kernel: ieee80211 phy0: sta85: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 Oct 15 16:54:38 localhost kernel: ieee80211 phy0: sta73: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 Oct 15 16:54:38 localhost kernel: ieee80211 phy0: sta109: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 15 16:54:38 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 15 16:54:38 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 15 16:54:38 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 15 16:54:38 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 15 16:54:38 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! ... Oct 15 16:54:40 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 Oct 15 16:54:40 localhost kernel: e1000e 0000:06:00.0: eth0: Detected Hardware Unit Hang: Oct 15 16:54:40 localhost kernel: TDH <3d> Oct 15 16:54:40 localhost kernel: TDT <3f> Oct 15 16:54:40 localhost kernel: next_to_use <3f> Oct 15 16:54:40 localhost kernel: next_to_clean <3d> Oct 15 16:54:40 localhost kernel: buffer_info[next_to_clean]: Oct 15 16:54:40 localhost kernel: time_stamp <3225b> Oct 15 16:54:40 localhost kernel: next_to_watch <3e> Oct 15 16:54:40 localhost kernel: jiffies <32c84> Oct 15 16:54:40 localhost kernel: next_to_watch.status <0> Oct 15 16:54:40 localhost kernel: MAC Status <80080f83> Oct 15 16:54:40 localhost kernel: PHY Status <796d> Oct 15 16:54:40 localhost kernel: PHY 1000BASE-T Status <7c00> Oct 15 16:54:40 localhost kernel: PHY Extended Status <3000> Oct 15 16:54:40 localhost kernel: PCI Status <4010> Oct 15 16:54:40 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 15 16:54:40 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! ... And then HD complains: Oct 15 16:55:18 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 Oct 15 16:55:18 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 15 16:55:18 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 15 16:55:18 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 Oct 15 16:55:18 localhost kernel: ata1: device not ready (errno=-16), forcing hardreset Oct 15 16:55:18 localhost kernel: ata1: soft resetting link Oct 15 16:55:18 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 15 16:55:18 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Maybe IRQs were disabled and not re-enabled somehow? There were no lockdep splats.. Here is a dump of the locks held... Oct 15 17:01:29 localhost kernel: SysRq : Show Locks Held Oct 15 17:01:29 localhost kernel: Oct 15 17:01:29 localhost kernel: Showing all locks held in the system: Oct 15 17:01:29 localhost kernel: 4 locks held by flush-253:0/502: Oct 15 17:01:29 localhost kernel: #0: (&type->s_umount_key#22){++++..}, at: [<c04cf1e0>] writeback_inodes_wb+0x95/0xf5 Oct 15 17:01:29 localhost kernel: #1: (jbd2_handle){+.+...}, at: [<c053814d>] start_this_handle+0x4b6/0x4ec Oct 15 17:01:29 localhost kernel: #2: (&ei->i_data_sem){++++..}, at: [<c051549a>] ext4_map_blocks+0xab/0x164 Oct 15 17:01:29 localhost kernel: #3: (&lg->lg_mutex){+.+...}, at: [<c0529792>] ext4_mb_initialize_context+0x1db/0x1e5 Oct 15 17:01:29 localhost kernel: 1 lock held by flush-253:2/1148: Oct 15 17:01:29 localhost kernel: #0: (&type->s_umount_key#22){++++..}, at: [<c04cf1e0>] writeback_inodes_wb+0x95/0xf5 Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1620: Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1622: Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1624: Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1626: Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1628: Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed Oct 15 17:01:29 localhost kernel: 1 lock held by mingetty/1630: Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed Oct 15 17:01:29 localhost kernel: 3 locks held by kworker/0:2/1632: Oct 15 17:01:29 localhost kernel: #0: (events){+.+.+.}, at: [<c0443cf6>] process_one_work+0x145/0x295 Oct 15 17:01:29 localhost kernel: #1: ((&ap->scsi_rescan_task)){+.+.+.}, at: [<c0443cf6>] process_one_work+0x145/0x295 Oct 15 17:01:29 localhost kernel: #2: (&ap->scsi_scan_mutex){+.+.+.}, at: [<c063f3c6>] ata_scsi_dev_rescan+0x1e/0xb6 Oct 15 17:01:29 localhost kernel: 1 lock held by bash/1814: Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed Oct 15 17:01:29 localhost kernel: 1 lock held by bash/1863: Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed Oct 15 17:01:29 localhost kernel: 1 lock held by bash/1935: Oct 15 17:01:29 localhost kernel: #0: (&tty->atomic_read_lock){+.+.+.}, at: [<c05f384d>] n_tty_read+0x1d1/0x5ed Oct 15 17:01:29 localhost kernel: 2 locks held by btserver/2422: Oct 15 17:01:29 localhost kernel: #0: (&sb->s_type->i_mutex_key#12){+.+.+.}, at: [<c04871bf>] generic_file_aio_write+0x4d/0xa6 Oct 15 17:01:29 localhost kernel: #1: (jbd2_handle){+.+...}, at: [<c053814d>] start_this_handle+0x4b6/0x4ec Oct 15 17:01:29 localhost kernel: 2 locks held by ip/23410: Oct 15 17:01:29 localhost kernel: #0: (&sb->s_type->i_mutex_key#12){+.+.+.}, at: [<c04871bf>] generic_file_aio_write+0x4d/0xa6 Oct 15 17:01:29 localhost kernel: #1: (jbd2_handle){+.+...}, at: [<c053814d>] start_this_handle+0x4b6/0x4ec Oct 15 17:01:29 localhost kernel: 4 locks held by sh/23416: Oct 15 17:01:29 localhost kernel: #0: (&sb->s_type->i_mutex_key#12){+.+.+.}, at: [<c04b6c36>] do_truncate+0x5a/0x7d Oct 15 17:01:29 localhost kernel: #1: (&sb->s_type->i_alloc_sem_key#4){+.+...}, at: [<c04c82e2>] notify_change+0x12a/0x218 Oct 15 17:01:29 localhost kernel: #2: (jbd2_handle){+.+...}, at: [<c053814d>] start_this_handle+0x4b6/0x4ec Oct 15 17:01:29 localhost kernel: #3: (&sbi->s_orphan_lock){+.+...}, at: [<c051975f>] ext4_orphan_add+0x1d8/0x1f7 Oct 15 17:01:29 localhost kernel: Oct 15 17:01:29 localhost kernel: ============================================= Can't get processor info from sysrq..don't know why: Oct 15 17:04:04 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 SysRq : Show backtrace of all active CPUs Oct 15 17:04:14 localhost kernel: SysRq : Show backtrace of all active CPUs Oct 15 17:04:14 localhost kernel: sending NMI to all CPUs: Oct 15 17:04:14 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 23:38 ` Luis R. Rodriguez 2010-10-15 23:41 ` Luis R. Rodriguez @ 2010-10-15 23:42 ` Ben Greear 2010-10-15 23:57 ` Luis R. Rodriguez 1 sibling, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-15 23:42 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Luis Rodriguez, linux-wireless On 10/15/2010 04:38 PM, Luis R. Rodriguez wrote: > On Fri, Oct 15, 2010 at 04:33:50PM -0700, Ben Greear wrote: >> On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote: >>> Ben, please give this patch a shot. I addresses three races on the PCU: >>> >>> * When we were stopping the CPU for non-EDMA cards we never locked against >>> anything starting the PCU again >>> >>> * ath9k_hw_startpcureceive() was being called without locking >>> >>> * Although we lock on the rxbuf lock for contention against starting/stopping >>> the PCU, we also need to lock on the driver in locations where we start/stop >>> the PCU within the same location otherwise we end up in inconsistant states >>> and the hardware may end up proessing an incorrect buffer for DMA. To >>> protect against this we use a new PCU lock on the main part of the driver to >>> ensure each start/stop/reset operation is done atomically. >>> >>> And fixes one issue as a side effect: >>> >>> * No more packet loss on ping flood when you have one STA associated :) >>> >>> The only issue I see with this is I eventually run out of memory and my box >>> becomes useless, unless I am mistaking that for some other issue. >>> >>> Please give this a shot and if it cures your woes I'll split it up into >>> 3 separate patches, or maybe just two, one for the first two and one for >>> the last issue. >> >> Sounds good, but this lockdep splat happens almost immediately upon starting >> my app: >> >> ======================================================= >> [ INFO: possible circular locking dependency detected ] >> 2.6.36-rc8-wl+ #32 >> ------------------------------------------------------- >> swapper/0 is trying to acquire lock: >> (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] ath9k_tasklet+0x7e/0x140 [ath9k] >> >> but task is already holding lock: >> (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] ath9k_tasklet+0x70/0x140 [ath9k] >> >> which lock already depends on the new lock. >> >> >> the existing dependency chain (in reverse order) is: >> >> -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}: >> [<c0457639>] lock_acquire+0x5a/0x78 >> [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f >> [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k] > > Ah we just need to nuke the flush lock, one second. Also remove my > skb_copy() otherwise you will really run out of memory quickly, > unless you really want to test with it :) Since you free the pkt, it shouldn't really consume, eh? Seems like we should be able to handle an extra 5 skbs floating around the system.. I'll try your new patch now... Thanks, Ben > > Luis -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 23:42 ` Ben Greear @ 2010-10-15 23:57 ` Luis R. Rodriguez 2010-10-17 19:44 ` Ben Greear 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-15 23:57 UTC (permalink / raw) To: Ben Greear; +Cc: Luis Rodriguez, linux-wireless On Fri, Oct 15, 2010 at 4:42 PM, Ben Greear <greearb@candelatech.com> wrote: > On 10/15/2010 04:38 PM, Luis R. Rodriguez wrote: >> >> On Fri, Oct 15, 2010 at 04:33:50PM -0700, Ben Greear wrote: >>> >>> On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote: >>>> >>>> Ben, please give this patch a shot. I addresses three races on the PCU: >>>> >>>> * When we were stopping the CPU for non-EDMA cards we never locked >>>> against >>>> anything starting the PCU again >>>> >>>> * ath9k_hw_startpcureceive() was being called without locking >>>> >>>> * Although we lock on the rxbuf lock for contention against >>>> starting/stopping >>>> the PCU, we also need to lock on the driver in locations where we >>>> start/stop >>>> the PCU within the same location otherwise we end up in >>>> inconsistant states >>>> and the hardware may end up proessing an incorrect buffer for DMA. >>>> To >>>> protect against this we use a new PCU lock on the main part of the >>>> driver to >>>> ensure each start/stop/reset operation is done atomically. >>>> >>>> And fixes one issue as a side effect: >>>> >>>> * No more packet loss on ping flood when you have one STA associated >>>> :) >>>> >>>> The only issue I see with this is I eventually run out of memory and my >>>> box >>>> becomes useless, unless I am mistaking that for some other issue. >>>> >>>> Please give this a shot and if it cures your woes I'll split it up into >>>> 3 separate patches, or maybe just two, one for the first two and one for >>>> the last issue. >>> >>> Sounds good, but this lockdep splat happens almost immediately upon >>> starting >>> my app: >>> >>> ======================================================= >>> [ INFO: possible circular locking dependency detected ] >>> 2.6.36-rc8-wl+ #32 >>> ------------------------------------------------------- >>> swapper/0 is trying to acquire lock: >>> (&(&sc->rx.pcu_lock)->rlock){+.-...}, at: [<fa16e5c7>] >>> ath9k_tasklet+0x7e/0x140 [ath9k] >>> >>> but task is already holding lock: >>> (&(&sc->rx.rxflushlock)->rlock){+.-...}, at: [<fa16e5b9>] >>> ath9k_tasklet+0x70/0x140 [ath9k] >>> >>> which lock already depends on the new lock. >>> >>> >>> the existing dependency chain (in reverse order) is: >>> >>> -> #1 (&(&sc->rx.rxflushlock)->rlock){+.-...}: >>> [<c0457639>] lock_acquire+0x5a/0x78 >>> [<c075f6ed>] _raw_spin_lock_bh+0x20/0x2f >>> [<fa170513>] ath_flushrecv+0x14/0x61 [ath9k] >> >> Ah we just need to nuke the flush lock, one second. Also remove my >> skb_copy() otherwise you will really run out of memory quickly, >> unless you really want to test with it :) > > Since you free the pkt, it shouldn't really consume, eh? What we want to do is to use a kernel routine that gets free memory *and* does a poison check against the memory area, so it does not matter if we free it immediately. > Seems like we should be able to handle an extra 5 skbs floating > around the system.. It does consume 5 times more for each RX's buffer, so for 130 STAs it would consume about 650 skbs instead of 130 if each RX'd an skb. > I'll try your new patch now... Thanks, Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 23:57 ` Luis R. Rodriguez @ 2010-10-17 19:44 ` Ben Greear 2010-10-18 22:46 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-17 19:44 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Luis Rodriguez, linux-wireless I had a chance to try your latest patch on a different machine and AP. This time, I was using 130 or so stations, but no encryption (no supplicant). I saw these exceptions below. The warn-on hits the !stopped check in ath_stoprecv. The system didn't crash, but all of the STAs soon dis-associated because of "inactivity". I haven't checked if that is some issue with my AP or what.. bool ath_stoprecv(struct ath_softc *sc) { struct ath_hw *ah = sc->sc_ah; bool stopped; spin_lock_bh(&sc->rx.rxbuflock); ath9k_hw_stoppcurecv(ah); ath9k_hw_setrxfilter(ah, 0); stopped = ath9k_hw_stopdmarecv(ah); if (sc->sc_ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) ath_edma_stop_recv(sc); else sc->rx.rxlink = NULL; spin_unlock_bh(&sc->rx.rxbuflock); WARN_ON(!stopped); return stopped; } ADDRCONF(NETDEV_UP): sta130: link is not ready sta90: no IPv6 routers present ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x40000020 ------------[ cut here ]------------ WARNING: at /home/greearb/git/linux.wireless-testing/drivers/net/wireless/ath/ath9k/recv.c:532 ath_stoprecv+0x80/0x87 [ath9k]() Hardware name: 945GM Modules linked in: 8021q garp stp llc michael_mic macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfs lockd fscache nfs_acl auth_rpcgss sunrpc p4_clockmod ipv6 uinput arc4 ecb ath9k snd_intel8x0 mac80211 snd_ac97_codec ac97_bus snd_seq snd_seq_device ath9k_common snd_pcm ath9k_hw ath snd_timer microcode pcspkr cfg80211 i2c_i801 snd soundcore snd_page_alloc serio_raw e1000e iTCO_wdt iTCO_vendor_support yenta_socket floppy i915 drm_kms_helper drm i2c_algo_bit i2c_core video output [last unloaded: ipt_addrtype] Pid: 5, comm: kworker/u:0 Tainted: G W 2.6.36-rc8-wl+ #12 Call Trace: [<c0434647>] warn_slowpath_common+0x65/0x7a [<f9450e38>] ? ath_stoprecv+0x80/0x87 [ath9k] [<c043466b>] warn_slowpath_null+0xf/0x13 [<f9450e38>] ath_stoprecv+0x80/0x87 [ath9k] [<f944f580>] ath_set_channel+0x99/0x1ff [ath9k] [<f94502b2>] ath9k_config+0x305/0x3d8 [ath9k] [<f9246a2e>] ieee80211_hw_config+0x11b/0x125 [mac80211] [<f924aa51>] ieee80211_scan_work+0x29e/0x3ed [mac80211] [<c0443d72>] ? process_one_work+0x145/0x295 [<c0443dbc>] process_one_work+0x18f/0x295 [<c0443d72>] ? process_one_work+0x145/0x295 [<f924a7b3>] ? ieee80211_scan_work+0x0/0x3ed [mac80211] [<c04453e5>] worker_thread+0xf9/0x1b8 [<c04452ec>] ? worker_thread+0x0/0x1b8 [<c0447d2f>] kthread+0x62/0x67 [<c0447ccd>] ? kthread+0x0/0x67 [<c0403506>] kernel_thread_helper+0x6/0x1a ---[ end trace bc53fa727ee2ae42 ]--- ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x40000020 ------------[ cut here ]------------ WARNING: at /home/greearb/git/linux.wireless-testing/drivers/net/wireless/ath/ath9k/recv.c:532 ath_stoprecv+0x80/0x87 [ath9k]() Hardware name: 945GM Modules linked in: 8021q garp stp llc michael_mic macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfs lockd fscache nfs_acl auth_rpcgss sunrpc p4_clockmod ipv6 uinput arc4 ecb ath9k snd_intel8x0 mac80211 snd_ac97_codec ac97_bus snd_seq snd_seq_device ath9k_common snd_pcm ath9k_hw ath snd_timer microcode pcspkr cfg80211 i2c_i801 snd soundcore snd_page_alloc serio_raw e1000e iTCO_wdt iTCO_vendor_support yenta_socket floppy i915 drm_kms_helper drm i2c_algo_bit i2c_core video output [last unloaded: ipt_addrtype] Pid: 41, comm: kworker/u:2 Tainted: G W 2.6.36-rc8-wl+ #12 Call Trace: [<c0434647>] warn_slowpath_common+0x65/0x7a [<f9450e38>] ? ath_stoprecv+0x80/0x87 [ath9k] [<c043466b>] warn_slowpath_null+0xf/0x13 [<f9450e38>] ath_stoprecv+0x80/0x87 [ath9k] [<f944f580>] ath_set_channel+0x99/0x1ff [ath9k] [<f94502b2>] ath9k_config+0x305/0x3d8 [ath9k] [<f9246a2e>] ieee80211_hw_config+0x11b/0x125 [mac80211] [<f924aa51>] ieee80211_scan_work+0x29e/0x3ed [mac80211] [<c0443d72>] ? process_one_work+0x145/0x295 [<c0443dbc>] process_one_work+0x18f/0x295 [<c0443d72>] ? process_one_work+0x145/0x295 [<f924a7b3>] ? ieee80211_scan_work+0x0/0x3ed [mac80211] [<c04453e5>] worker_thread+0xf9/0x1b8 [<c04452ec>] ? worker_thread+0x0/0x1b8 [<c0447d2f>] kthread+0x62/0x67 [<c0447ccd>] ? kthread+0x0/0x67 [<c0403506>] kernel_thread_helper+0x6/0x1a ---[ end trace bc53fa727ee2ae43 ]--- ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 sta126: direct probe to 00:18:e7:cb:ad:6e (try 1) sta130: direct probe to 00:18:e7:cb:ad:6e (try 1) sta91: no IPv6 routers present sta126: direct probe to 00:18:e7:cb:ad:6e (try 2) sta130: direct probe to 00:18:e7:cb:ad:6e (try 2) sta126: direct probe to 00:18:e7:cb:ad:6e (try 3) sta130: direct probe to 00:18:e7:cb:ad:6e (try 3) sta126: direct probe to 00:18:e7:cb:ad:6e timed out sta130: direct probe to 00:18:e7:cb:ad:6e timed out sta92: no IPv6 routers present ... Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-17 19:44 ` Ben Greear @ 2010-10-18 22:46 ` Luis R. Rodriguez 0 siblings, 0 replies; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-18 22:46 UTC (permalink / raw) To: Ben Greear; +Cc: Luis Rodriguez, linux-wireless On Sun, Oct 17, 2010 at 12:44 PM, Ben Greear <greearb@candelatech.com> wrote: > I had a chance to try your latest patch on a different machine and AP. This > time, > I was using 130 or so stations, but no encryption (no supplicant). > > I saw these exceptions below. The warn-on hits the !stopped check in > ath_stoprecv. > > The system didn't crash, but all of the STAs soon dis-associated because of > "inactivity". > I haven't checked if that is some issue with my AP or what.. > > > bool ath_stoprecv(struct ath_softc *sc) > { > struct ath_hw *ah = sc->sc_ah; > bool stopped; > > spin_lock_bh(&sc->rx.rxbuflock); > ath9k_hw_stoppcurecv(ah); > ath9k_hw_setrxfilter(ah, 0); > stopped = ath9k_hw_stopdmarecv(ah); > > if (sc->sc_ah->caps.hw_caps & ATH9K_HW_CAP_EDMA) > ath_edma_stop_recv(sc); > else > sc->rx.rxlink = NULL; > spin_unlock_bh(&sc->rx.rxbuflock); > > WARN_ON(!stopped); > return stopped; > } > > > ADDRCONF(NETDEV_UP): sta130: link is not ready > sta90: no IPv6 routers present > ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x40000020 We should find out what happened here. > ------------[ cut here ]------------ > WARNING: at > /home/greearb/git/linux.wireless-testing/drivers/net/wireless/ath/ath9k/recv.c:532 > ath_stoprecv+0x80/0x87 [ath9k]() > Hardware name: 945GM > Modules linked in: 8021q garp stp llc michael_mic macvlan pktgen iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi nfs lockd fscache nfs_acl > auth_rpcgss sunrpc p4_clockmod ipv6 uinput arc4 ecb ath9k snd_intel8x0 > mac80211 snd_ac97_codec ac97_bus snd_seq snd_seq_device ath9k_common snd_pcm > ath9k_hw ath snd_timer microcode pcspkr cfg80211 i2c_i801 snd soundcore > snd_page_alloc serio_raw e1000e iTCO_wdt iTCO_vendor_support yenta_socket > floppy i915 drm_kms_helper drm i2c_algo_bit i2c_core video output [last > unloaded: ipt_addrtype] > Pid: 5, comm: kworker/u:0 Tainted: G W 2.6.36-rc8-wl+ #12 > Call Trace: > [<c0434647>] warn_slowpath_common+0x65/0x7a > [<f9450e38>] ? ath_stoprecv+0x80/0x87 [ath9k] > [<c043466b>] warn_slowpath_null+0xf/0x13 > [<f9450e38>] ath_stoprecv+0x80/0x87 [ath9k] > [<f944f580>] ath_set_channel+0x99/0x1ff [ath9k] > [<f94502b2>] ath9k_config+0x305/0x3d8 [ath9k] > [<f9246a2e>] ieee80211_hw_config+0x11b/0x125 [mac80211] > [<f924aa51>] ieee80211_scan_work+0x29e/0x3ed [mac80211] > [<c0443d72>] ? process_one_work+0x145/0x295 > [<c0443dbc>] process_one_work+0x18f/0x295 > [<c0443d72>] ? process_one_work+0x145/0x295 > [<f924a7b3>] ? ieee80211_scan_work+0x0/0x3ed [mac80211] > [<c04453e5>] worker_thread+0xf9/0x1b8 > [<c04452ec>] ? worker_thread+0x0/0x1b8 > [<c0447d2f>] kthread+0x62/0x67 > [<c0447ccd>] ? kthread+0x0/0x67 > [<c0403506>] kernel_thread_helper+0x6/0x1a > ---[ end trace bc53fa727ee2ae42 ]--- > ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 > ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x40000020 > ------------[ cut here ]------------ > WARNING: at > /home/greearb/git/linux.wireless-testing/drivers/net/wireless/ath/ath9k/recv.c:532 > ath_stoprecv+0x80/0x87 [ath9k]() > Hardware name: 945GM > Modules linked in: 8021q garp stp llc michael_mic macvlan pktgen iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi nfs lockd fscache nfs_acl > auth_rpcgss sunrpc p4_clockmod ipv6 uinput arc4 ecb ath9k snd_intel8x0 > mac80211 snd_ac97_codec ac97_bus snd_seq snd_seq_device ath9k_common snd_pcm > ath9k_hw ath snd_timer microcode pcspkr cfg80211 i2c_i801 snd soundcore > snd_page_alloc serio_raw e1000e iTCO_wdt iTCO_vendor_support yenta_socket > floppy i915 drm_kms_helper drm i2c_algo_bit i2c_core video output [last > unloaded: ipt_addrtype] > Pid: 41, comm: kworker/u:2 Tainted: G W 2.6.36-rc8-wl+ #12 > Call Trace: > [<c0434647>] warn_slowpath_common+0x65/0x7a > [<f9450e38>] ? ath_stoprecv+0x80/0x87 [ath9k] > [<c043466b>] warn_slowpath_null+0xf/0x13 > [<f9450e38>] ath_stoprecv+0x80/0x87 [ath9k] > [<f944f580>] ath_set_channel+0x99/0x1ff [ath9k] > [<f94502b2>] ath9k_config+0x305/0x3d8 [ath9k] > [<f9246a2e>] ieee80211_hw_config+0x11b/0x125 [mac80211] > [<f924aa51>] ieee80211_scan_work+0x29e/0x3ed [mac80211] > [<c0443d72>] ? process_one_work+0x145/0x295 > [<c0443dbc>] process_one_work+0x18f/0x295 > [<c0443d72>] ? process_one_work+0x145/0x295 > [<f924a7b3>] ? ieee80211_scan_work+0x0/0x3ed [mac80211] > [<c04453e5>] worker_thread+0xf9/0x1b8 > [<c04452ec>] ? worker_thread+0x0/0x1b8 > [<c0447d2f>] kthread+0x62/0x67 > [<c0447ccd>] ? kthread+0x0/0x67 > [<c0403506>] kernel_thread_helper+0x6/0x1a > ---[ end trace bc53fa727ee2ae43 ]--- > ath: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 > sta126: direct probe to 00:18:e7:cb:ad:6e (try 1) > sta130: direct probe to 00:18:e7:cb:ad:6e (try 1) > sta91: no IPv6 routers present > sta126: direct probe to 00:18:e7:cb:ad:6e (try 2) > sta130: direct probe to 00:18:e7:cb:ad:6e (try 2) > sta126: direct probe to 00:18:e7:cb:ad:6e (try 3) > sta130: direct probe to 00:18:e7:cb:ad:6e (try 3) > sta126: direct probe to 00:18:e7:cb:ad:6e timed out > sta130: direct probe to 00:18:e7:cb:ad:6e timed out > sta92: no IPv6 routers present > ... So I put the warning there for debugging purposes. The reason I put it is we have no gaurantee we've told hardware to stop writing to that area of memory so if we then race and start RX I think we can likely run into a situation where we may not know which buffer hardware will be writing to next. I think we should add the warning on the next kernel development cycle and address all of its causes, if hardware cannot be stopped I believe we run the potential to also run into this same poison issue. The RX poison issue is resolved then but the other issues are separate issues which need to be debugged further. For example the stop TX dma issue might be resolved by also preventing to start TX and stop TX atomically. Something like what I did could likely also be done for TX. When I get some time (I have other uber higher priority issues now as usual) I'll break this patch into a 3 or 2 patches and submit upstream. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-15 23:33 ` Ben Greear 2010-10-15 23:38 ` Luis R. Rodriguez @ 2010-10-15 23:39 ` Ben Greear 1 sibling, 0 replies; 84+ messages in thread From: Ben Greear @ 2010-10-15 23:39 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Luis Rodriguez, linux-wireless On 10/15/2010 04:33 PM, Ben Greear wrote: > On 10/15/2010 04:21 PM, Luis R. Rodriguez wrote: >> Ben, please give this patch a shot. I addresses three races on the PCU: >> >> * When we were stopping the CPU for non-EDMA cards we never locked >> against >> anything starting the PCU again >> >> * ath9k_hw_startpcureceive() was being called without locking >> >> * Although we lock on the rxbuf lock for contention against >> starting/stopping >> the PCU, we also need to lock on the driver in locations where we >> start/stop >> the PCU within the same location otherwise we end up in inconsistant >> states >> and the hardware may end up proessing an incorrect buffer for DMA. To >> protect against this we use a new PCU lock on the main part of the >> driver to >> ensure each start/stop/reset operation is done atomically. >> >> And fixes one issue as a side effect: >> >> * No more packet loss on ping flood when you have one STA associated :) >> >> The only issue I see with this is I eventually run out of memory and >> my box >> becomes useless, unless I am mistaking that for some other issue. >> >> Please give this a shot and if it cures your woes I'll split it up into >> 3 separate patches, or maybe just two, one for the first two and one for >> the last issue. > > Sounds good, but this lockdep splat happens almost immediately upon > starting > my app: > > ======================================================= > [ INFO: possible circular locking dependency detected ] > 2.6.36-rc8-wl+ #32 It ran for a bit..never did see the poison warning, but system hard-locked (sysrq b on the serial console got no response) after a bit. Hopefully just because it hit this potential deadlock. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 23:39 ` Luis R. Rodriguez 2010-10-14 23:48 ` Luis R. Rodriguez @ 2010-10-14 23:51 ` Ben Greear 1 sibling, 0 replies; 84+ messages in thread From: Ben Greear @ 2010-10-14 23:51 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Luis Rodriguez, linux-wireless On 10/14/2010 04:39 PM, Luis R. Rodriguez wrote: > On Thu, Oct 14, 2010 at 4:30 PM, Ben Greear<greearb@candelatech.com> wrote: >> On 10/14/2010 04:19 PM, Luis R. Rodriguez wrote: >>> >>> Ok please try this patch, it cures it for me. >> >> Well, it got a lot further than normal, but it still >> hit the poison check after a few minutes. >> >> Current test case is my app loading 130 or so stations, each running >> wpa_supplicant. All were created, and quite a few had associated >> when the poison check hit. >> >> So, it definitely looks like a step in the right direction, but >> not fully fixed yet. >> >> I'll do some more testing with this patch applied and using just my >> perl script to make sure the problem is reproducible outside of my >> application. > > Ok, whatever userspace does it should not corrupt to kernel, unless > its poking /dev/mem Sure, but it's nice to have a simpler test case. I ran my script with 32 stations for maybe 20 minutes...seemed to be doing OK as far as stability goes..then it crashed in the divide-by-zero ani thing. I have to run home now, but I'll try my script with more interfaces and/or some traffic mix, etc, to see if I can get another reasonable way to reproduce this w/out my big binary application. I can also work on the divide-by-zero bug if no one beats me to it. Thanks, Ben > > Luis -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 21:52 ` Björn Smedman 2010-10-14 22:05 ` Ben Greear @ 2010-10-14 22:47 ` Ben Greear 2010-10-14 23:46 ` Björn Smedman 1 sibling, 1 reply; 84+ messages in thread From: Ben Greear @ 2010-10-14 22:47 UTC (permalink / raw) To: Björn Smedman Cc: Vasanthakumar Thiagarajan, Luis R. Rodriguez, Johannes Berg, linux-wireless@vger.kernel.org On 10/14/2010 02:52 PM, Björn Smedman wrote: > 2010/10/13 Björn Smedman<bjorn.smedman@venatech.se>: >> Hi Ben, >> >> First of all keep up the good work. :) >> >> On Wed, Oct 13, 2010 at 6:39 PM, Ben Greear<greearb@candelatech.com> wrote: >> [snip] >>> Either way, it seems safer to null out the bf_ampdu field after >>> the memory is consumed..it could prevent some tricky bugs later. >> >> I think this is a good idea. But it probably wont be enough to null >> out bf_mpdu. You also need to look at bf_buf_addr (which if I >> understand correctly is the physical address the DMA engine will >> actually write RXed frames to) and bf_dmacontext (which seems in most >> cases to hold an identical address and may in fact be where the DMA >> engine will really write the frame). > > I took another look at the code. It turns out both bf_buf_addr and > bf_dmacontext are in fact meaningless to the DMA. Instead each bf > holds a pointer (bf_desc) to the real DMA descriptor which in turn > holds the address (ds_data) where the DMA will really (really this > time) write the frame. There is also a field to hold the virtual > address of the same place (ds_vdata). > > It's a little too much work for me to set up the testbed you have Ben > but would be interesting to see what happens if you set > bf->bf_desc->ds_{data,vdata} = 0 as well. No? I tried the patch below, and it didn't seem to help. Might even have hurt..as it died on divide-by-zero error: Call Trace: [<c075e490>] ? printk+0xf/0x17 [<c075e37e>] panic+0x50/0x153 [<c07619db>] oops_end+0x92/0xa1 [<c04051cc>] die+0x53/0x59 [<c07612a3>] do_trap+0x89/0xa2 [<c0403b12>] ? do_divide_error+0x0/0x78 [<c0403b80>] do_divide_error+0x6e/0x78 [<faef811e>] ? ath9k_hw_ani_monitor+0x37/0x40e [ath9k_hw] [<fb1f77d9>] ? ath9k_ioread32+0x25/0x5b [ath9k] [<c045553a>] ? trace_hardirqs_off+0xb/0xd [<c0581210>] ? trace_hardirqs_off_thunk+0xc/0x10 [<c076103f>] error_code+0x5f/0x70 [<c0403b12>] ? do_divide_error+0x0/0x78 [<faef811e>] ? ath9k_hw_ani_monitor+0x37/0x40e [ath9k_hw] [<fb1fa783>] ath_ani_calibrate+0x143/0x20b [ath9k] [<c043d58f>] run_timer_softirq+0x14f/0x1e7 That might be an existing bug, however... Thanks, Ben diff --git a/drivers/net/wireless/ath/ath9k/ath9k.h b/drivers/net/wireless/ath/ath9k/ath9k.h index 0c917e5..8fba13d 100644 --- a/drivers/net/wireless/ath/ath9k/ath9k.h +++ b/drivers/net/wireless/ath/ath9k/ath9k.h @@ -737,4 +737,6 @@ bool ath_mac80211_start_queue(struct ath_softc *sc, u16 skb_queue); void ath_start_rfkill_poll(struct ath_softc *sc); extern void ath9k_rfkill_poll_state(struct ieee80211_hw *hw); +void ath_clear_dma_ptrs(struct ath_buf *bf); + #endif /* ATH9K_H */ diff --git a/drivers/net/wireless/ath/ath9k/beacon.c b/drivers/net/wireless/ath/ath9k/beacon.c index 97d471f..cc406f9 100644 --- a/drivers/net/wireless/ath/ath9k/beacon.c +++ b/drivers/net/wireless/ath/ath9k/beacon.c @@ -139,7 +139,7 @@ static struct ath_buf *ath_beacon_generate(struct ieee80211_hw *hw, dma_unmap_single(sc->dev, bf->bf_buf_addr, skb->len, DMA_TO_DEVICE); dev_kfree_skb_any(skb); - bf->bf_buf_addr = 0; + ath_clear_dma_ptrs(bf); } /* Get a new beacon from mac80211 */ @@ -167,8 +167,7 @@ static struct ath_buf *ath_beacon_generate(struct ieee80211_hw *hw, skb->len, DMA_TO_DEVICE); if (unlikely(dma_mapping_error(sc->dev, bf->bf_buf_addr))) { dev_kfree_skb_any(skb); - bf->bf_mpdu = NULL; - bf->bf_buf_addr = 0; + ath_clear_dma_ptrs(bf); ath_print(common, ATH_DBG_FATAL, "dma_mapping_error on beaconing\n"); return NULL; @@ -256,8 +255,7 @@ int ath_beacon_alloc(struct ath_wiphy *aphy, struct ieee80211_vif *vif) dma_unmap_single(sc->dev, bf->bf_buf_addr, skb->len, DMA_TO_DEVICE); dev_kfree_skb_any(skb); - bf->bf_mpdu = NULL; - bf->bf_buf_addr = 0; + ath_clear_dma_ptrs(bf); } /* NB: the beacon data buffer must be 32-bit aligned. */ @@ -302,8 +300,7 @@ int ath_beacon_alloc(struct ath_wiphy *aphy, struct ieee80211_vif *vif) skb->len, DMA_TO_DEVICE); if (unlikely(dma_mapping_error(sc->dev, bf->bf_buf_addr))) { dev_kfree_skb_any(skb); - bf->bf_mpdu = NULL; - bf->bf_buf_addr = 0; + ath_clear_dma_ptrs(bf); ath_print(common, ATH_DBG_FATAL, "dma_mapping_error on beacon alloc\n"); return -ENOMEM; @@ -329,8 +326,7 @@ void ath_beacon_return(struct ath_softc *sc, struct ath_vif *avp) dma_unmap_single(sc->dev, bf->bf_buf_addr, skb->len, DMA_TO_DEVICE); dev_kfree_skb_any(skb); - bf->bf_mpdu = NULL; - bf->bf_buf_addr = 0; + ath_clear_dma_ptrs(bf); } list_add_tail(&bf->list, &sc->beacon.bbuf); diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c index bcd3892..1722a21 100644 --- a/drivers/net/wireless/ath/ath9k/main.c +++ b/drivers/net/wireless/ath/ath9k/main.c @@ -213,6 +213,17 @@ static void ath_update_survey_stats(struct ath_softc *sc) ath_update_survey_nf(sc, pos); } +void ath_clear_dma_ptrs(struct ath_buf *bf) +{ + struct ath_desc *ds = bf->bf_desc; + bf->bf_buf_addr = 0; + bf->bf_mpdu = NULL; + if (ds) { + ds->ds_data = 0; + ds->ds_vdata = NULL; + } +} + /* * Set/change channels. If the channel is really being changed, it's done * by reseting the chip. To accomplish this we must first cleanup any pending diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c index fd778d2..5afb46f 100644 --- a/drivers/net/wireless/ath/ath9k/recv.c +++ b/drivers/net/wireless/ath/ath9k/recv.c @@ -269,8 +269,7 @@ static int ath_rx_edma_init(struct ath_softc *sc, int nbufs) if (unlikely(dma_mapping_error(sc->dev, bf->bf_buf_addr))) { dev_kfree_skb_any(skb); - bf->bf_mpdu = NULL; - bf->bf_buf_addr = 0; + ath_clear_dma_ptrs(bf); ath_print(common, ATH_DBG_FATAL, "dma_mapping_error() on RX init\n"); error = -ENOMEM; @@ -360,8 +359,7 @@ int ath_rx_init(struct ath_softc *sc, int nbufs) if (unlikely(dma_mapping_error(sc->dev, bf->bf_buf_addr))) { dev_kfree_skb_any(skb); - bf->bf_mpdu = NULL; - bf->bf_buf_addr = 0; + ath_clear_dma_ptrs(bf); ath_print(common, ATH_DBG_FATAL, "dma_mapping_error() on RX init\n"); error = -ENOMEM; @@ -396,8 +394,7 @@ void ath_rx_cleanup(struct ath_softc *sc) common->rx_bufsize, DMA_FROM_DEVICE); dev_kfree_skb(skb); - bf->bf_buf_addr = 0; - bf->bf_mpdu = NULL; + ath_clear_dma_ptrs(bf); } } @@ -1741,8 +1738,7 @@ int ath_rx_tasklet(struct ath_softc *sc, int flush, bool hp) if (unlikely(dma_mapping_error(sc->dev, bf->bf_buf_addr))) { dev_kfree_skb_any(requeue_skb); - bf->bf_mpdu = NULL; - bf->bf_buf_addr = 0; + ath_clear_dma_ptrs(bf); ath_print(common, ATH_DBG_FATAL, "dma_mapping_error() on RX\n"); ath_rx_send_to_mac80211(hw, sc, skb, rxs); diff --git a/drivers/net/wireless/ath/ath9k/xmit.c b/drivers/net/wireless/ath/ath9k/xmit.c index a5e5f27..e86f59c 100644 --- a/drivers/net/wireless/ath/ath9k/xmit.c +++ b/drivers/net/wireless/ath/ath9k/xmit.c @@ -1644,8 +1644,7 @@ static int ath_tx_setup_buffer(struct ieee80211_hw *hw, struct ath_buf *bf, bf->bf_buf_addr = dma_map_single(sc->dev, skb->data, skb->len, DMA_TO_DEVICE); if (unlikely(dma_mapping_error(sc->dev, bf->bf_buf_addr))) { - bf->bf_mpdu = NULL; - bf->bf_buf_addr = 0; + ath_clear_dma_ptrs(bf); ath_print(ath9k_hw_common(sc->sc_ah), ATH_DBG_FATAL, "dma_mapping_error() on TX\n"); return -ENOMEM; @@ -1915,7 +1914,6 @@ static void ath_tx_complete_buf(struct ath_softc *sc, struct ath_buf *bf, } dma_unmap_single(sc->dev, bf->bf_buf_addr, skb->len, DMA_TO_DEVICE); - bf->bf_buf_addr = 0; if (bf->bf_state.bfs_paprd) { if (time_after(jiffies, @@ -1931,7 +1929,7 @@ static void ath_tx_complete_buf(struct ath_softc *sc, struct ath_buf *bf, /* At this point, skb (bf->bf_mpdu) is consumed...make sure we don't * accidentally reference it later. */ - bf->bf_mpdu = NULL; + ath_clear_dma_ptrs(bf); /* * Return the list of ath_buf of this mpdu to free queue > > /Björn > -- > To unsubscribe from this list: send the line "unsubscribe linux-wireless" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 22:47 ` Ben Greear @ 2010-10-14 23:46 ` Björn Smedman 2010-10-18 13:48 ` Björn Smedman 0 siblings, 1 reply; 84+ messages in thread From: Björn Smedman @ 2010-10-14 23:46 UTC (permalink / raw) To: Ben Greear Cc: Vasanthakumar Thiagarajan, Luis R. Rodriguez, Johannes Berg, linux-wireless@vger.kernel.org 2010/10/15 Ben Greear <greearb@candelatech.com>: > I tried the patch below, and it didn't seem to help. Might even > have hurt..as it died on divide-by-zero error: Hmm, looks like the ani code got a zero listen time from the hw... That just might mean that the DMA actually hits one of these descriptors. :) I'll send a patch for the div-by-zero in a minute, and then perhaps we can narrow it down to which one of these descriptors is actually being used by DMA when it really shouldn't be? /Björn ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-14 23:46 ` Björn Smedman @ 2010-10-18 13:48 ` Björn Smedman 2010-10-18 17:24 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Björn Smedman @ 2010-10-18 13:48 UTC (permalink / raw) To: Ben Greear, Luis R. Rodriguez Cc: Vasanthakumar Thiagarajan, Johannes Berg, linux-wireless@vger.kernel.org 2010/10/15 Björn Smedman <bjorn.smedman@venatech.se> > > 2010/10/15 Ben Greear <greearb@candelatech.com>: > > I tried the patch below, and it didn't seem to help. Might even > > have hurt..as it died on divide-by-zero error: > > Hmm, looks like the ani code got a zero listen time from the hw... > That just might mean that the DMA actually hits one of these > descriptors. :) Am I the only one worried about this? Leaving a DMA descriptor pointing at memory which has been passed on to somebody else... To me that's like pointing a loaded gun at someone (and it seems this particular gun can go off a little haphazardly). Luis, given how hard it seems to be to get that locking and skb shoveling right, are you sure you want to keep pointing that DMA engine on innocent people's data? /Björn ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-18 13:48 ` Björn Smedman @ 2010-10-18 17:24 ` Luis R. Rodriguez 2010-10-18 22:34 ` Björn Smedman 0 siblings, 1 reply; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-18 17:24 UTC (permalink / raw) To: Björn Smedman; +Cc: Ben Greear, linux-wireless@vger.kernel.org 2010/10/18 Björn Smedman <bjorn.smedman@venatech.se>: > 2010/10/15 Björn Smedman <bjorn.smedman@venatech.se> >> >> 2010/10/15 Ben Greear <greearb@candelatech.com>: >> > I tried the patch below, and it didn't seem to help. Might even >> > have hurt..as it died on divide-by-zero error: >> >> Hmm, looks like the ani code got a zero listen time from the hw... >> That just might mean that the DMA actually hits one of these >> descriptors. :) > > Am I the only one worried about this? Leaving a DMA descriptor > pointing at memory which has been passed on to somebody else... To me > that's like pointing a loaded gun at someone (and it seems this > particular gun can go off a little haphazardly). > > Luis, given how hard it seems to be to get that locking and skb > shoveling right, are you sure you want to keep pointing that DMA > engine on innocent people's data? This is why this issue is of high priority to me. I no longer get the poison nor does Ben, the RX poison issue is resolved as far as I can tell, I just need to split up the patches into easily reviewable chunks and get them upstream. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-18 17:24 ` Luis R. Rodriguez @ 2010-10-18 22:34 ` Björn Smedman 2010-10-18 22:41 ` Luis R. Rodriguez 0 siblings, 1 reply; 84+ messages in thread From: Björn Smedman @ 2010-10-18 22:34 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Ben Greear, linux-wireless@vger.kernel.org 2010/10/18 Luis R. Rodriguez <mcgrof@gmail.com>: > 2010/10/18 Björn Smedman <bjorn.smedman@venatech.se>: >> 2010/10/15 Björn Smedman <bjorn.smedman@venatech.se> >>> >>> 2010/10/15 Ben Greear <greearb@candelatech.com>: >>> > I tried the patch below, and it didn't seem to help. Might even >>> > have hurt..as it died on divide-by-zero error: >>> >>> Hmm, looks like the ani code got a zero listen time from the hw... >>> That just might mean that the DMA actually hits one of these >>> descriptors. :) >> >> Am I the only one worried about this? Leaving a DMA descriptor >> pointing at memory which has been passed on to somebody else... To me >> that's like pointing a loaded gun at someone (and it seems this >> particular gun can go off a little haphazardly). >> >> Luis, given how hard it seems to be to get that locking and skb >> shoveling right, are you sure you want to keep pointing that DMA >> engine on innocent people's data? > > This is why this issue is of high priority to me. I no longer get the > poison nor does Ben, the RX poison issue is resolved as far as I can > tell, I just need to split up the patches into easily reviewable > chunks and get them upstream. The locking issue you found looks like it could cause those overwritten poisons (as well as some weird problems I've seen in AP mode with lots of monitor interfaces). It's really great to see that problem go. All I'm saying is that this stuff is difficult and the next time we get it wrong we should try to avoid overwriting arbitrary kernel memory with our RXed frames (or TXing something sensitive). Will the DMA engine stop when it sees a zero ds_data? In that case I suggest we never keep an address there that we do not want to RX to or TX from. Also, is there some way to verify that we are not corrupting memory with the DMA? I mean the poison check is great but if I understand correctly it cannot detect if we overwrite active memory (i.e. before it is freed and marked with the poison). /Björn ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-18 22:34 ` Björn Smedman @ 2010-10-18 22:41 ` Luis R. Rodriguez 0 siblings, 0 replies; 84+ messages in thread From: Luis R. Rodriguez @ 2010-10-18 22:41 UTC (permalink / raw) To: Björn Smedman; +Cc: Ben Greear, linux-wireless@vger.kernel.org 2010/10/18 Björn Smedman <bjorn.smedman@venatech.se>: > 2010/10/18 Luis R. Rodriguez <mcgrof@gmail.com>: >> 2010/10/18 Björn Smedman <bjorn.smedman@venatech.se>: >>> 2010/10/15 Björn Smedman <bjorn.smedman@venatech.se> >>>> >>>> 2010/10/15 Ben Greear <greearb@candelatech.com>: >>>> > I tried the patch below, and it didn't seem to help. Might even >>>> > have hurt..as it died on divide-by-zero error: >>>> >>>> Hmm, looks like the ani code got a zero listen time from the hw... >>>> That just might mean that the DMA actually hits one of these >>>> descriptors. :) >>> >>> Am I the only one worried about this? Leaving a DMA descriptor >>> pointing at memory which has been passed on to somebody else... To me >>> that's like pointing a loaded gun at someone (and it seems this >>> particular gun can go off a little haphazardly). >>> >>> Luis, given how hard it seems to be to get that locking and skb >>> shoveling right, are you sure you want to keep pointing that DMA >>> engine on innocent people's data? >> >> This is why this issue is of high priority to me. I no longer get the >> poison nor does Ben, the RX poison issue is resolved as far as I can >> tell, I just need to split up the patches into easily reviewable >> chunks and get them upstream. > > The locking issue you found looks like it could cause those > overwritten poisons (as well as some weird problems I've seen in AP > mode with lots of monitor interfaces). It's really great to see that > problem go. :) > All I'm saying is that this stuff is difficult and the > next time we get it wrong we should try to avoid overwriting arbitrary > kernel memory with our RXed frames (or TXing something sensitive). Patches welcomed. > Will the DMA engine stop when it sees a zero ds_data? In that case I > suggest we never keep an address there that we do not want to RX to or > TX from. We will always have an skb available to DMA for RX, its part of the design on RX on ath9k. We simply do drop the frame we just got DMA'd if we cannot allocate a new skb from the kernel. So we should always be able to DMA over and over and over. The issue here was due to a race on stopping and starting the PCU, it got confused on which buffer to write to. > Also, is there some way to verify that we are not corrupting memory > with the DMA? I mean the poison check is great I'm only aware of the poison checks. > but if I understand > correctly it cannot detect if we overwrite active memory (i.e. before > it is freed and marked with the poison). Right, the best you can do is to understand the code. Detecting bogus writes from hardware to are hard to detect on already used memory given that you'd need to ensure you understand what a writer to that area of memory will do. Anyway if you find actual issues instead of pure speculation please let us know. Luis ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-13 16:39 ` Ben Greear 2010-10-13 19:56 ` Björn Smedman @ 2010-10-14 5:37 ` Vasanthakumar Thiagarajan 1 sibling, 0 replies; 84+ messages in thread From: Vasanthakumar Thiagarajan @ 2010-10-14 5:37 UTC (permalink / raw) To: Ben Greear Cc: Vasanth Thiagarajan, Luis R. Rodriguez, Johannes Berg, linux-wireless@vger.kernel.org > > > > I dont see any point in NULLing out bf->bf_mpdu. bf is > > reclaimed onto a free tx buf pool as soon as it is done > > with the skb. bf_mpdu of any of the bf's is never accessed > > without any initialization (bf_ampdu = skb). > > The code can use skb after its deleted currently, because > ath_debug_stat_tx(sc, txq, bf, ts); references the bf_ampdu > object (I think I added that reference lately..so it's really > a bug that I caused). At the least, we should move the ath_debug_stat_tx > logic before the ath_tx_complete() call. Yes, this is a serious issue irrespective of initializing bf_mpdu to NULL. > > As for the paprd path, it looks racy to me: What if the paprd timer > expires while the ath_tx_complete_buf logic is running? That is the goal here, if we timed out on paprd training , ath_tx_complete_buf() has to free the skb. At least I dont see any race here, can you elaborate on your finding?, remember ath_tx_complete_buf() is in tasklet context. Vasanth ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 21:31 ` Luis R. Rodriguez 2010-10-07 21:36 ` Luis R. Rodriguez @ 2010-10-07 21:52 ` Ben Greear 1 sibling, 0 replies; 84+ messages in thread From: Ben Greear @ 2010-10-07 21:52 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Johannes Berg, linux-wireless@vger.kernel.org On 10/07/2010 02:31 PM, Luis R. Rodriguez wrote: > On Thu, Oct 7, 2010 at 12:27 PM, Johannes Berg > <johannes@sipsolutions.net> wrote: >> On Thu, 2010-10-07 at 12:22 -0700, Ben Greear wrote: >> >>> After reboot, and re-run of the script, >>> I saw this in the logs, and shortly after, >>> the SLUB poison warning dumped to screen. >>> >>> Maybe those DMA errors are serious? >> >>> ath: Failed to stop TX DMA in 100 msec after killing last frame >>> ath: Failed to stop TX DMA. Resetting hardware! >> >> That's TX DMA, it can hardly result in invalid memory writes like the >> ones you've been seeing. >> >> I'm still convinced something is wrong with ath9k RX DMA, as you've seen >> the contents of frames written to already freed memory regions. Since I >> don't know anything about ath9k, you should probably not rely on me >> though :-) > > I'm on this now. Lets play. > > I had to remove /lib/udev/rules.d/75-persistent-net-generator.rules > to avoid Ubuntu trying to remember the device names and it creating > stax_rename names. Right, we disable udev for 'sta*' devices. > I just ran your script with some modifications. You can find it here: > > http://www.kernel.org/pub/linux/kernel/people/mcgrof/scripts/poo.pl Can you post your kernel .config somewhere, and confirm which kernel you are using? Also, what ath9k NIC, platform, etc? We see the problem on two different systems (haven't tried more). I can figure out the brands of the NICs if that helps, and have included lspci information below. I've uploaded my kernel config to here: http://www.candelatech.com/~greearb/ctwl_kernel.cfg * Dual core Intel Pentium-D 32-bit 2GB RAM Fedora 13 (but with custom compiled top-of-tree iw, hostap, libnl Atheros NIC: from lspci -vv: 08:01.0 Network controller: Atheros Communications Inc. AR922X Wireless Network Adapter (rev 01) Subsystem: Device 0777:4002 Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B+ DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 17 Region 0: Memory at d0300000 (32-bit, non-prefetchable) [size=64K] Capabilities: [44] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=100mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel modules: ath9k * Dual-core Intel Atom, 32-bit 1GB RAM Fedora 13 (custom compiled top-of-tree iw, hostap, libnl) Atheros mini-pcie NIC, from lspci -vv: 03:00.0 Network controller: Atheros Communications Inc. AR928X Wireless Network Adapter (PCI-Express) (rev 01) Subsystem: Device 0777:4e05 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 19 Region 0: Memory at febf0000 (64-bit, non-prefetchable) [size=64K] Capabilities: [40] Power Management version 2 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [60] Express (v1) Legacy Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM unknown, Latency L0 <512ns, L1 <64us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: [90] MSI-X: Enable- Count=1 Masked- Vector table: BAR=0 offset=00000000 PBA: BAR=0 offset=00000000 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntrySize=0 Arb: Fixed- WRR32- WRR64- WRR128- 100ns- onfig- - - TableOffset=0 Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Fixed- RR32- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: [160 v1] Device Serial Number 00-00-00-00-00-00-00-00 Kernel driver in use: ath9k Kernel modules: ath9k > > I then ran: > > for i in $(seq 0 31) ; do sudo dhclient seq$i; done > > It took about 10 minutes to get IP addresses for all interfaces but it > got there eventually. Odd enough I was unable to ping the AP from any > interface though. Not sure what that was about. But I got no oops, no > slub dump. All I got was some more delba warnings which seems to > indicate we haven't caught all the cases for its use: If you just create one or two interfaces, can you ping as expected? Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-07 19:27 ` Johannes Berg 2010-10-07 21:31 ` Luis R. Rodriguez @ 2010-10-08 0:42 ` Bruno Randolf 2010-10-08 2:30 ` Ben Greear 1 sibling, 1 reply; 84+ messages in thread From: Bruno Randolf @ 2010-10-08 0:42 UTC (permalink / raw) To: Johannes Berg Cc: Ben Greear, Luis R. Rodriguez, linux-wireless@vger.kernel.org On Fri October 8 2010 04:27:22 Johannes Berg wrote: > I'm still convinced something is wrong with ath9k RX DMA, as you've seen > the contents of frames written to already freed memory regions. Since I > don't know anything about ath9k, you should probably not rely on me > though :-) not sure if this is related or not, but it reminds me of: "ath9k: BUG kmalloc-8192: Poison overwritten" http://marc.info/?l=linux-kernel&m=127379077422354&w=2 and: "ath5k in monitor mode: Poison overwritten on buffers allocated from ath_rxbuf_alloc" https://bugzilla.kernel.org/show_bug.cgi?id=15861 bruno ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-08 0:42 ` Bruno Randolf @ 2010-10-08 2:30 ` Ben Greear 0 siblings, 0 replies; 84+ messages in thread From: Ben Greear @ 2010-10-08 2:30 UTC (permalink / raw) To: Bruno Randolf Cc: Johannes Berg, Luis R. Rodriguez, linux-wireless@vger.kernel.org On 10/07/2010 05:42 PM, Bruno Randolf wrote: > On Fri October 8 2010 04:27:22 Johannes Berg wrote: >> I'm still convinced something is wrong with ath9k RX DMA, as you've seen >> the contents of frames written to already freed memory regions. Since I >> don't know anything about ath9k, you should probably not rely on me >> though :-) > > not sure if this is related or not, but it reminds me of: > > "ath9k: BUG kmalloc-8192: Poison overwritten" > http://marc.info/?l=linux-kernel&m=127379077422354&w=2 This looks identical...some sort of scan results in poisoned buffer. > > and: > > "ath5k in monitor mode: Poison overwritten on buffers allocated from > ath_rxbuf_alloc" > https://bugzilla.kernel.org/show_bug.cgi?id=15861 We've been doing similar (to the ath9k issue) tests with ath5k and it's been much more stable. But, our kernel is currently totally larded down with debugging so we can't the NIC as hard as we'd like. I'll make sure we do some hard-core testing on ath5k soon. Thanks, Ben > > bruno -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: memory clobber in rx path, maybe related to ath9k. 2010-10-05 17:00 memory clobber in rx path, maybe related to ath9k Ben Greear 2010-10-05 17:16 ` Luis R. Rodriguez @ 2010-10-05 17:22 ` Johannes Berg 1 sibling, 0 replies; 84+ messages in thread From: Johannes Berg @ 2010-10-05 17:22 UTC (permalink / raw) To: Ben Greear; +Cc: linux-wireless@vger.kernel.org On Tue, 2010-10-05 at 10:00 -0700, Ben Greear wrote: > Oct 5 09:50:18 localhost kernel: Object 0xf5b18030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > Oct 5 09:50:18 localhost kernel: Object 0xf5b18040: 80 00 00 00 ff ff ff ff ff ff c8 4c 75 20 e8 fd ....ÿÿÿÿÿÿÈLu.èý > Oct 5 09:50:18 localhost kernel: Object 0xf5b18050: c8 4c 75 20 e8 fd d0 88 80 c1 e1 03 00 00 00 00 ÈLu.èýÐ..Áá..... > Oct 5 09:50:18 localhost kernel: Object 0xf5b18060: 90 01 21 04 00 09 63 69 73 63 6f 73 62 2d 31 01 ..!...ciscosb-1. Evidently, a beacon was received into an already freed SKB. johannes ^ permalink raw reply [flat|nested] 84+ messages in thread
end of thread, other threads:[~2010-10-18 22:47 UTC | newest] Thread overview: 84+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-10-05 17:00 memory clobber in rx path, maybe related to ath9k Ben Greear 2010-10-05 17:16 ` Luis R. Rodriguez 2010-10-05 17:24 ` Ben Greear 2010-10-05 17:36 ` Luis R. Rodriguez 2010-10-05 17:38 ` Ben Greear 2010-10-05 17:43 ` Luis R. Rodriguez 2010-10-05 17:47 ` Ben Greear 2010-10-05 17:55 ` Luis R. Rodriguez 2010-10-05 18:14 ` Ben Greear 2010-10-05 21:12 ` Ben Greear 2010-10-07 17:33 ` Ben Greear 2010-10-07 18:14 ` Johannes Berg 2010-10-07 18:29 ` Luis R. Rodriguez 2010-10-07 18:39 ` Ben Greear 2010-10-07 18:42 ` Luis R. Rodriguez 2010-10-07 18:45 ` Ben Greear 2010-10-07 19:14 ` Ben Greear 2010-10-07 19:17 ` Johannes Berg 2010-10-07 19:22 ` Ben Greear 2010-10-07 19:27 ` Johannes Berg 2010-10-07 21:31 ` Luis R. Rodriguez 2010-10-07 21:36 ` Luis R. Rodriguez 2010-10-07 21:59 ` Luis R. Rodriguez 2010-10-11 20:51 ` Ben Greear 2010-10-12 1:03 ` Luis R. Rodriguez 2010-10-12 3:27 ` Ben Greear 2010-10-12 6:10 ` Luis R. Rodriguez 2010-10-12 18:35 ` Ben Greear 2010-10-12 18:40 ` Luis R. Rodriguez 2010-10-12 18:43 ` Ben Greear 2010-10-12 19:51 ` Ben Greear 2010-10-13 17:12 ` Ben Greear 2010-10-13 17:29 ` Luis R. Rodriguez 2010-10-13 17:48 ` Ben Greear 2010-10-14 21:25 ` Luis R. Rodriguez 2010-10-14 21:31 ` Ben Greear 2010-10-14 21:32 ` Luis R. Rodriguez 2010-10-14 21:39 ` Ben Greear 2010-10-14 21:45 ` Johannes Berg 2010-10-14 21:47 ` Ben Greear 2010-10-13 5:31 ` Vasanthakumar Thiagarajan 2010-10-13 16:39 ` Ben Greear 2010-10-13 19:56 ` Björn Smedman 2010-10-13 20:03 ` Luis R. Rodriguez 2010-10-14 19:15 ` Ben Greear 2010-10-14 19:17 ` Luis R. Rodriguez 2010-10-14 21:52 ` Björn Smedman 2010-10-14 22:05 ` Ben Greear 2010-10-14 22:16 ` Luis R. Rodriguez 2010-10-14 22:29 ` Luis R. Rodriguez 2010-10-14 22:35 ` Luis R. Rodriguez 2010-10-14 22:44 ` Ben Greear 2010-10-14 22:54 ` Luis R. Rodriguez 2010-10-14 22:51 ` Luis R. Rodriguez 2010-10-14 23:19 ` Luis R. Rodriguez 2010-10-14 23:30 ` Ben Greear 2010-10-14 23:39 ` Luis R. Rodriguez 2010-10-14 23:48 ` Luis R. Rodriguez 2010-10-15 16:51 ` Ben Greear 2010-10-15 18:47 ` Luis R. Rodriguez 2010-10-15 19:36 ` Ben Greear 2010-10-15 21:07 ` Luis R. Rodriguez 2010-10-15 23:21 ` Luis R. Rodriguez 2010-10-15 23:33 ` Ben Greear 2010-10-15 23:38 ` Luis R. Rodriguez 2010-10-15 23:41 ` Luis R. Rodriguez 2010-10-16 0:07 ` Ben Greear 2010-10-15 23:42 ` Ben Greear 2010-10-15 23:57 ` Luis R. Rodriguez 2010-10-17 19:44 ` Ben Greear 2010-10-18 22:46 ` Luis R. Rodriguez 2010-10-15 23:39 ` Ben Greear 2010-10-14 23:51 ` Ben Greear 2010-10-14 22:47 ` Ben Greear 2010-10-14 23:46 ` Björn Smedman 2010-10-18 13:48 ` Björn Smedman 2010-10-18 17:24 ` Luis R. Rodriguez 2010-10-18 22:34 ` Björn Smedman 2010-10-18 22:41 ` Luis R. Rodriguez 2010-10-14 5:37 ` Vasanthakumar Thiagarajan 2010-10-07 21:52 ` Ben Greear 2010-10-08 0:42 ` Bruno Randolf 2010-10-08 2:30 ` Ben Greear 2010-10-05 17:22 ` Johannes Berg
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).