From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Greear Date: Mon, 04 Oct 2010 10:18:12 -0700 Subject: [ath9k-devel] Ath9k in funky state after adding 130 STA interfaces. In-Reply-To: <20101004170732.GH2105@tux> References: <4CA81265.8030403@candelatech.com> <20101004170732.GH2105@tux> Message-ID: <4CAA0C54.5050106@candelatech.com> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: ath9k-devel@lists.ath9k.org On 10/04/2010 10:07 AM, Luis R. Rodriguez wrote: > On Sat, Oct 02, 2010 at 10:19:33PM -0700, Ben Greear wrote: >> This on wireless-testing from Friday, plus a few debugfs patches I posted >> recently. >> >> I was running two STA just fine, and then tried to add 128 more. > > You're serious right? I mean I'm happy your serious, but whoa, you > want 130 STAs on one interface working fine? Oh, I'm hoping for 256+ :) Ath5k can do 128 ok, with very minimal traffic load, at least. Our primary interest is load testing APs and such, so the more the merrier. Ath9k seems a bit more touchy. I'm testing today on a different system, but still ath9k. The system has been panic-ing, and locking hard. Here is results of a mostly-hard-lock. I was hoping the 'deadbeef' registers indicated a particular error in the NIC that I might could use for further debugging. Oct 4 10:08:32 localhost kernel: sta25: authenticate with 00:14:d1:c6:d2:54 (try 1) Oct 4 10:08:32 localhost kernel: ath: timeout (100000 us) on reg 0x9860: 0x0000002f & 0x00000001 != 0x00000000 Oct 4 10:08:32 localhost kernel: ath: Unable to reset channel (2437 MHz), reset status -5 Oct 4 10:08:32 localhost kernel: ath: Unable to set channel Oct 4 10:08:32 localhost kernel: sta119: authenticate with 00:14:d1:c6:d2:54 (try 1) Oct 4 10:08:32 localhost kernel: sta25: authenticate with 00:14:d1:c6:d2:54 (try 2) Oct 4 10:08:32 localhost kernel: sta119: authenticate with 00:14:d1:c6:d2:54 (try 2) Oct 4 10:08:32 localhost kernel: sta25: authenticate with 00:14:d1:c6:d2:54 (try 3) Oct 4 10:08:32 localhost kernel: sta2: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta13: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta57: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta74: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta96: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta97: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta110: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta117: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta120: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta119: authenticate with 00:14:d1:c6:d2:54 (try 3) Oct 4 10:08:32 localhost kernel: sta25: authentication with 00:14:d1:c6:d2:54 timed out Oct 4 10:08:32 localhost kernel: sta119: authentication with 00:14:d1:c6:d2:54 timed out Oct 4 10:08:33 localhost kernel: sta2: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta13: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta57: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta74: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta96: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta97: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta110: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta117: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta120: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 4 10:08:33 localhost kernel: ath: timeout (100000 us) on reg 0x7000: 0xdeadbeef & 0x00000003 != 0x00000000 Oct 4 10:08:33 localhost kernel: ath: Chip reset failed Oct 4 10:08:33 localhost kernel: ath: Unable to reset hardware; reset status -22 Oct 4 10:08:33 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:08:33 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:08:33 localhost kernel: ath: timeout (100000 us) on reg 0x7000: 0xdeadbeef & 0x00000003 != 0x00000000 Oct 4 10:08:33 localhost kernel: ath: Chip reset failed Oct 4 10:08:33 localhost kernel: ath: Unable to reset channel (2412 MHz), reset status -22 Oct 4 10:08:33 localhost kernel: ath: Unable to set channel Oct 4 10:08:33 localhost kernel: sta97: deauthenticating from 00:14:d1:c6:d2:54 by local choice (reason=3) Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 4 10:08:34 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:08:34 localhost kernel: ath: timeout (100000 us) on reg 0x9860: 0xdeadbeef & 0x00000001 != 0x00000000 Oct 4 10:08:34 localhost kernel: ath: Unable to reset hardware; reset status -5 Oct 4 10:08:34 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:08:34 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:08:34 localhost kernel: ath: timeout (100000 us) on reg 0x7000: 0xdeadbeef & 0x00000003 != 0x00000000 Oct 4 10:08:34 localhost kernel: ath: Chip reset failed Oct 4 10:08:34 localhost kernel: ath: Unable to reset channel (2437 MHz), reset status -22 Oct 4 10:08:34 localhost kernel: ath: Unable to set channel Oct 4 10:08:34 localhost kernel: ieee80211 phy0: Removed STA 00:14:d1:c6:d2:54 Oct 4 10:08:34 localhost kernel: ieee80211 phy0: Destroyed STA 00:14:d1:c6:d2:54 ... Lots more spewage, then no more kernel spewage, ..seems all HD access is fried..tried to pipe dmesg to disk, but command hung. Oct 4 10:09:01 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:09:01 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:09:02 localhost kernel: ath: timeout (100000 us) on reg 0x7000: 0xdeadbeef & 0x00000003 != 0x00000000 Oct 4 10:09:02 localhost kernel: ath: Chip reset failed Oct 4 10:09:02 localhost kernel: ath: Unable to reset channel (2437 MHz), reset status -22 Oct 4 10:09:02 localhost kernel: ath: Unable to set channel Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 4 10:09:02 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:09:02 localhost kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen SysRq : Show Locks Held Thanks, Ben > > Luis -- Ben Greear Candela Technologies Inc http://www.candelatech.com