Re: Firmware crash/lockup on WLE900VX.

[Ben Greear <greearb@candelatech.com>]

On 03/17/2014 04:53 PM, Ben Greear wrote:
> This is with my somewhat modified ath10k driver (mostly just
> additional printks as far as this firmware is concerned),
> and standard upstream firmware running in AP mode.
> 
> I guess the interesting question to me is why we cannot read
> the firmware dump area address?
> 
> It all goes down hill from there and soft resets fail to
> resolve the issue.
> 
> [245273.132602] IPv6: ADDRCONF(NETDEV_UP): vap0: link is not ready
> [245273.159965] IPv6: ADDRCONF(NETDEV_CHANGE): vap1: link becomes ready
> [245273.259178] IPv6: ADDRCONF(NETDEV_CHANGE): vap0: link becomes ready
> [245498.059492] e1000e: eth1 NIC Link is Down
> [245500.455638] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> [246136.713414] ath10k: firmware crashed!
> [246136.715802] ath10k: hardware name qca988x hw2.0 version 0x4100016c
> [246136.720699] ath10k: firmware version: 10.1.467.1-1
> [246136.736226] ath10k: failed to read FW dump area address: -16
> [246451.587593] ath10k: key installation failed for vdev 0 peer 00:f0:21:0e:80:7f: -110
> [246451.587604] vap1: failed to remove key (0, 00:f0:21:0e:80:7f) from hardware (-110)
> [246454.593921] ath10k: could not remove peer wep key 0 (-11)
> [246454.593926] ath10k: could not clear all peer wep keys for vdev 0: -11
> [246454.593929] ath10k: Failed to disassociate station: 00:f0:21:0e:80:7f vdev 0 ret -11
> [246454.593931] ------------[ cut here ]------------
> [246454.593962] WARNING: CPU: 1 PID: 17319 at /mnt/sda/home/greearb/git/linux.ath/net/mac80211/sta_info.c:893 __sta_info_destroy_part2+0x62/0x15e [mac80211]()
> [246454.593964] Modules linked in: iptable_raw xt_CT nf_nat_ipv4 nf_nat 8021q garp stp mrp llc fuse macvlan pktgen ip6table_filter ip6_tables ebtable_nat
> ebtables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd sunrpc fscache pcspkr uinput ath10k_pci coretemp hwmon ath10k_core ath
> intel_powerclamp mac80211 kvm snd_hda_codec_realtek snd_hda_codec_generic e1000e cfg80211 snd_hda_intel snd_hda_codec ptp snd_hwdep pps_core gpio_ich ppdev
> iTCO_wdt shpchp iTCO_vendor_support parport_pc parport microcode serio_raw lpc_ich i2c_i801 snd_seq snd_seq_device snd_pcm snd_timer snd soundcore ipv6 i915
> i2c_algo_bit drm_kms_helper ata_generic pata_acpi drm i2c_core video [last unloaded: iptable_nat]
> [246454.594044] CPU: 1 PID: 17319 Comm: hostapd Tainted: G         C   3.14.0-rc6-wl-ath+ #3
> [246454.594045] Hardware name: To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M., BIOS 4.6.3 03/06/2012
> [246454.594047]  0000000000000009 ffff8800bb597918 ffffffff815aaef5 0000000000000006
> [246454.594050]  0000000000000000 ffff8800bb597958 ffffffff810c1aa8 ffff8800bb66c000
> [246454.594052]  ffffffffa056b1a3 ffff8800bb66c000 ffff8800c60e45e0 ffff8800373c6800
> [246454.594055] Call Trace:
> [246454.594061]  [<ffffffff815aaef5>] dump_stack+0x4e/0x71
> [246454.594067]  [<ffffffff810c1aa8>] warn_slowpath_common+0x77/0x91
> [246454.594075]  [<ffffffffa056b1a3>] ? __sta_info_destroy_part2+0x62/0x15e [mac80211]
> [246454.594078]  [<ffffffff810c1ad7>] warn_slowpath_null+0x15/0x17
> [246454.594084]  [<ffffffffa056b1a3>] __sta_info_destroy_part2+0x62/0x15e [mac80211]
> [246454.594091]  [<ffffffffa056b437>] __sta_info_destroy+0x25/0x2f [mac80211]
> [246454.594099]  [<ffffffffa056b53d>] sta_info_destroy_addr_bss+0x37/0x57 [mac80211]
> [246454.594110]  [<ffffffffa0579a1c>] ieee80211_del_station+0x18/0x25 [mac80211]
> [246454.594119]  [<ffffffffa0308411>] nl80211_del_station+0x55/0x65 [cfg80211]
> [246454.594124]  [<ffffffff81524c08>] genl_family_rcv_msg+0x246/0x2ba
> [246454.594126]  [<ffffffff8152054c>] ? sock_put+0x12/0x22
> [246454.594129]  [<ffffffff81524cb9>] genl_rcv_msg+0x3d/0x60
> [246454.594131]  [<ffffffff81524c7c>] ? genl_family_rcv_msg+0x2ba/0x2ba
> [246454.594133]  [<ffffffff81523a3a>] netlink_rcv_skb+0x3d/0x8a
> [246454.594135]  [<ffffffff81523c37>] genl_rcv+0x24/0x34
> [246454.594137]  [<ffffffff8152321a>] netlink_unicast+0xc6/0x147
> [246454.594139]  [<ffffffff81523839>] netlink_sendmsg+0x59e/0x5e4
> [246454.594141]  [<ffffffff81521a3b>] ? netlink_recvmsg+0x2ca/0x2e9
> [246454.594145]  [<ffffffff814ea860>] __sock_sendmsg_nosec+0x25/0x27
> [246454.594147]  [<ffffffff814ed1cf>] sock_sendmsg+0x5a/0x7b
> [246454.594149]  [<ffffffff814edc7b>] ? move_addr_to_kernel+0x35/0x62
> [246454.594152]  [<ffffffff814f7885>] ? verify_iovec+0x52/0xa5
> [246454.594153]  [<ffffffff814edba5>] ___sys_sendmsg+0x21e/0x2bf
> [246454.594156]  [<ffffffff814ea973>] ? sockfs_dname+0x1c/0x1c
> [246454.594159]  [<ffffffff811a31c1>] ? destroy_inode+0x36/0x4f
> [246454.594161]  [<ffffffff811a38a4>] ? evict+0x155/0x15d
> [246454.594163]  [<ffffffff811837eb>] ? kmem_cache_free+0x34/0xa7
> [246454.594165]  [<ffffffff8119fb43>] ? __d_free+0x4b/0x50
> [246454.594167]  [<ffffffff811a7265>] ? mntput_no_expire+0x2e/0x126
> [246454.594169]  [<ffffffff811a7385>] ? mntput+0x28/0x2a
> [246454.594172]  [<ffffffff811909a0>] ? __fput+0x190/0x1b0
> [246454.594175]  [<ffffffff814ee097>] __sys_sendmsg+0x3d/0x5b
> [246454.594177]  [<ffffffff814ee0c2>] SyS_sendmsg+0xd/0x19
> [246454.594181]  [<ffffffff815b59fd>] system_call_fastpath+0x1a/0x1f
> [246454.594182] ---[ end trace 9d9631e0aba2fb1a ]---
> [246457.594136] ath10k: Failed to delete peer 00:f0:21:0e:80:7f for vdev 0: -11
> [246457.594140] ------------[ cut here ]------------
> [246457.594171] WARNING: CPU: 1 PID: 17319 at /mnt/sda/home/greearb/git/linux.ath/net/mac80211/sta_info.c:901 __sta_info_destroy_part2+0xba/0x15e [mac80211]()
> [246457.594173] Modules linked in: iptable_raw xt_CT nf_nat_ipv4 nf_nat 8021q garp stp mrp llc fuse macvlan pktgen ip6table_filter ip6_tables ebtable_nat
> ebtables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd sunrpc fscache pcspkr uinput ath10k_pci coretemp hwmon ath10k_core ath
> intel_powerclamp mac80211 kvm snd_hda_codec_realtek snd_hda_codec_generic e1000e cfg80211 snd_hda_intel snd_hda_codec ptp snd_hwdep pps_core gpio_ich ppdev
> iTCO_wdt shpchp iTCO_vendor_support parport_pc parport microcode serio_raw lpc_ich i2c_i801 snd_seq snd_seq_device snd_pcm snd_timer snd soundcore ipv6 i915
> i2c_algo_bit drm_kms_helper ata_generic pata_acpi drm i2c_core video [last unloaded: iptable_nat]
> [246457.594250] CPU: 1 PID: 17319 Comm: hostapd Tainted: G        WC   3.14.0-rc6-wl-ath+ #3
> [246457.594251] Hardware name: To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M., BIOS 4.6.3 03/06/2012
> [246457.594253]  0000000000000009 ffff8800bb597918 ffffffff815aaef5 0000000000000006
> [246457.594256]  0000000000000000 ffff8800bb597958 ffffffff810c1aa8 ffff8800bb597968
> [246457.594258]  ffffffffa056b1fb ffff8800bb66c000 00000000fffffff5 ffff8800373c6800
> [246457.594261] Call Trace:
> [246457.594267]  [<ffffffff815aaef5>] dump_stack+0x4e/0x71
> [246457.594273]  [<ffffffff810c1aa8>] warn_slowpath_common+0x77/0x91
> [246457.594281]  [<ffffffffa056b1fb>] ? __sta_info_destroy_part2+0xba/0x15e [mac80211]
> [246457.594283]  [<ffffffff810c1ad7>] warn_slowpath_null+0x15/0x17
> [246457.594290]  [<ffffffffa056b1fb>] __sta_info_destroy_part2+0xba/0x15e [mac80211]
> [246457.594297]  [<ffffffffa056b437>] __sta_info_destroy+0x25/0x2f [mac80211]
> [246457.594304]  [<ffffffffa056b53d>] sta_info_destroy_addr_bss+0x37/0x57 [mac80211]
> [246457.594315]  [<ffffffffa0579a1c>] ieee80211_del_station+0x18/0x25 [mac80211]
> [246457.594324]  [<ffffffffa0308411>] nl80211_del_station+0x55/0x65 [cfg80211]
> [246457.594329]  [<ffffffff81524c08>] genl_family_rcv_msg+0x246/0x2ba
> [246457.594331]  [<ffffffff8152054c>] ? sock_put+0x12/0x22
> [246457.594334]  [<ffffffff81524cb9>] genl_rcv_msg+0x3d/0x60
> [246457.594336]  [<ffffffff81524c7c>] ? genl_family_rcv_msg+0x2ba/0x2ba
> [246457.594338]  [<ffffffff81523a3a>] netlink_rcv_skb+0x3d/0x8a
> [246457.594340]  [<ffffffff81523c37>] genl_rcv+0x24/0x34
> [246457.594342]  [<ffffffff8152321a>] netlink_unicast+0xc6/0x147
> [246457.594344]  [<ffffffff81523839>] netlink_sendmsg+0x59e/0x5e4
> [246457.594346]  [<ffffffff81521a3b>] ? netlink_recvmsg+0x2ca/0x2e9
> [246457.594352]  [<ffffffff814ea860>] __sock_sendmsg_nosec+0x25/0x27
> [246457.594354]  [<ffffffff814ed1cf>] sock_sendmsg+0x5a/0x7b
> [246457.594356]  [<ffffffff814edc7b>] ? move_addr_to_kernel+0x35/0x62
> [246457.594359]  [<ffffffff814f7885>] ? verify_iovec+0x52/0xa5
> [246457.594360]  [<ffffffff814edba5>] ___sys_sendmsg+0x21e/0x2bf
> [246457.594362]  [<ffffffff814ea973>] ? sockfs_dname+0x1c/0x1c
> [246457.594365]  [<ffffffff811a31c1>] ? destroy_inode+0x36/0x4f
> [246457.594367]  [<ffffffff811a38a4>] ? evict+0x155/0x15d
> [246457.594370]  [<ffffffff811837eb>] ? kmem_cache_free+0x34/0xa7
> [246457.594372]  [<ffffffff8119fb43>] ? __d_free+0x4b/0x50
> [246457.594374]  [<ffffffff811a7265>] ? mntput_no_expire+0x2e/0x126
> [246457.594376]  [<ffffffff811a7385>] ? mntput+0x28/0x2a
> [246457.594379]  [<ffffffff811909a0>] ? __fput+0x190/0x1b0
> [246457.594381]  [<ffffffff814ee097>] __sys_sendmsg+0x3d/0x5b
> [246457.594383]  [<ffffffff814ee0c2>] SyS_sendmsg+0xd/0x19
> [246457.594385]  [<ffffffff815b59fd>] system_call_fastpath+0x1a/0x1f
> [246457.594387] ---[ end trace 9d9631e0aba2fb1b ]---
> [246457.594389] vap1: sta-info-destroy: drv-sta-state error: -11, sta: 00:f0:21:0e:80:7f
> [246462.597586] ath10k: key installation failed for vdev 0 peer 04:f0:21:0e:38:43: -11
> [246462.597591] vap1: failed to remove key (0, 04:f0:21:0e:38:43) from hardware (-11)
> [246465.603855] ath10k: could not remove peer wep key 0 (-11)
> [246465.603860] ath10k: could not clear all peer wep keys for vdev 0: -11
> [246465.603863] ath10k: Failed to disassociate station: 04:f0:21:0e:38:43 vdev 0 ret -11
> [246468.604130] ath10k: Failed to delete peer 04:f0:21:0e:38:43 for vdev 0: -11
> [246468.604134] vap1: sta-info-destroy: drv-sta-state error: -11, sta: 04:f0:21:0e:38:43
> [246471.604400] ath10k: Failed to set beacon mode for vdev 0: -11
> [246474.604670] ath10k: Failed to recalculate rts/cts prot for vdev 0: -11
> [246477.604943] ath10k: Failed to set preamble for vdev 0: -11
> [246480.605216] ath10k: key installation failed for vdev 0 peer 04:f0:21:03:38:99: -11
> [246480.605222] vap1: failed to remove key (1, ff:ff:ff:ff:ff:ff) from hardware (-11)
> [246483.611487] ath10k: key installation failed for vdev 0 peer 04:f0:21:03:38:99: -11
> [246483.611491] vap1: failed to set key (1, ff:ff:ff:ff:ff:ff) to hardware (-11)
> 

I should add:  rmmod of ath10k_pci and modprobe and it still remains stuck:

[265106.855296] ath10k_pci 0000:04:00.0: BAR 0: assigned [mem 0xfb400000-0xfb5fffff 64bit]
[265106.855308] ath10k_pci 0000:04:00.0: BAR 0: error updating (0xfb400004 != 0xffffffff)
[265106.862830] ath10k_pci 0000:04:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[265106.881871] ath10k_pci 0000:04:00.0: Refused to change power state, currently in D3
[265106.887288] ath10k: target took longer 5000 us to wake up (awake count 1)
[265106.887290] ath10k: Failed to get chip id: -110
[265106.891578] ath10k_pci: probe of 0000:04:00.0 failed with error -110


Thanks,
Ben



-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k