ath9k driver may be broken on ARM64 ?

ath9k driver may be broken on ARM64 ?

[Ellie Revves]

Hi,
for the context part:

I've got a marvell espressobin board, to which I attached a mini-PCIE 
wireless card, atheros AR9565 aka. QCB335. I am using both archlinux-arm 
and a custom OS made with buildroot, and both have this problem.

Problem:

When the system boots up, the kernel recognizes the card, and then the 
ath9k driver gets loaded. From what someone on the irc channel of 
#linux-wireless determined, the driver ends up doing a sigbus while 
attempting to read from a specific address in memory, which of course 
makes it crash. We did check the page tables, and the memory seems to be 
mapped fine. So we're still clueless as to why it is not possible to 
read from that particular memory address, and why the driver would do 
so. Here I attach the kernel page tables, along with the dmesg output. 
The problematic line in the trace is x19.

I'd appreciate any help in fixing this issue which I think might be 
related to ARM64.

Note that I am no programmer, all the info I gathered so far were with 
someone else's help. But if you need some more debug output, files or 
anything else, just let me know. I can also apply patches to test 
things, or modify the code if I know exactly in what way, i.e: what to 
copy and past.

Dmesg:

http://ix.io/1pTK

kernel page tables -- from a later kernel build, but do notice that the 
memory is still mapped correctly even if the register changed:

http://ix.io/1pQh

Thanks a lot :)

Re: ath9k driver may be broken on ARM64 ?

[Tom Psyborg]

Hi

There was similar issue reported in forum recently:
https://forum.openwrt.org/t/wireless-card-atheros-ar9565/23608

Could you enable ath9k debug and post output?

Re: ath9k driver may be broken on ARM64 ?

[Ellie Revves]

Hi,
sure, I enabled atheros wireless debugging in my kernel and am 
rebuilding it. After this is done, should I set ath9k.debug= on the 
cmdline ? It is built as module, but my filesystem is read-only so no 
modprobe.d snippet can be added.
Thanks

Tom Psyborg a écrit :
> Hi
>
> There was similar issue reported in forum recently:
> https://forum.openwrt.org/t/wireless-card-atheros-ar9565/23608
>
> Could you enable ath9k debug and post output?

Re: ath9k driver may be broken on ARM64 ?

[Ellie Revves]

Hi,
well I enabled CONFIG_ATH_DEBUG and booted this kernel, sorry for the 
mess of escape sequence but I captured via serial console:
http://ix.io/1pWI
I don't see anything different even if I cranked up ath9k.debug= to the 
*any* value. I guess this thing happens so early at boot that nothing 
useful is yet initialized...
Thanks

Ellie Revves a écrit :
> Hi,
> sure, I enabled atheros wireless debugging in my kernel and am 
> rebuilding it. After this is done, should I set ath9k.debug= on the 
> cmdline ? It is built as module, but my filesystem is read-only so no 
> modprobe.d snippet can be added.
> Thanks
>
> Tom Psyborg a écrit :
>> Hi
>>
>> There was similar issue reported in forum recently:
>> https://forum.openwrt.org/t/wireless-card-atheros-ar9565/23608
>>
>> Could you enable ath9k debug and post output?
>

Re: ath9k driver may be broken on ARM64 ?

[Tom Psyborg]

Hi

Try selecting config option ATH9K_SUPPORT_PCOEM (Support chips used in
PC OEM cards) and rebuild your image.

I'm not familiar with your architecture but it seems you're not
getting debug output. It should print a lot of info right after the:
ath: phy0: WB335 2-ANT card detected

Re: ath9k driver may be broken on ARM64 ?

[Ellie Revves]

Hi,
CONFIG_ATH9K_PCOEM is already enabled in my kernel build. In fact if it 
is disabled my card does not show up at all.
Thanks

Tom Psyborg a écrit :
> Hi
>
> Try selecting config option ATH9K_SUPPORT_PCOEM (Support chips used in
> PC OEM cards) and rebuild your image.
>
> I'm not familiar with your architecture but it seems you're not
> getting debug output. It should print a lot of info right after the:
> ath: phy0: WB335 2-ANT card detected

Re: ath9k driver may be broken on ARM64 ?

[Tom Psyborg]

if your system is read-only then you should make a build with debug
option enabled:

inside your buildroot create this path: files/etc/modules.d/ and put
file named ath9k there which should contain "ath9k debug=0xffffffff"
(more info: https://openwrt.org/docs/guide-developer/build-system/use-buildsystem#custom_files)

rebuild and try boot again capturing bootlog

Re: ath9k driver may be broken on ARM64 ?

[Ellie Revves]

Here's another boot log along with some commands after the system booted 
up to show what I did. I don't think I did anything wrong, and yet this 
is no different, debug option appears to be completely useless.
http://ix.io/1pY0
Thanks

Tom Psyborg a écrit :
> if your system is read-only then you should make a build with debug
> option enabled:
>
> inside your buildroot create this path: files/etc/modules.d/ and put
> file named ath9k there which should contain "ath9k debug=0xffffffff"
> (more info: https://openwrt.org/docs/guide-developer/build-system/use-buildsystem#custom_files)
>
> rebuild and try boot again capturing bootlog

Re: ath9k driver may be broken on ARM64 ?

[Dominique Martinet]

Ellie Revves wrote on Thu, Oct 25, 2018:
> Here's another boot log along with some commands after the system
> booted up to show what I did. I don't think I did anything wrong,
> and yet this is no different, debug option appears to be completely
> useless.

I'm not familiar with ath at all, just happened to be free at the time
this was brought up on IRC (#linux-wireless), so can relate what we said
there after looking at the debug symbols to interprete the stack a bit
further. It seems to fail before any print, honestly.

In the stack trace, x17 (0x4020) is reg_offset, so looking from the
value I think it would be AR_INTR_SYNC_CAUSE which should be in some
path like this inlined in ath9k_hw_init -> __ath9k_hw_init ->
ath9k_hw_set_reset_reg -> ath9k_hw_set_reset_power_on ->
ath9k_hw_set_reset -> tmpReg = REG_READ(ah, AR_INTR_SYNC_CAUSE);


The only thing that would have happened before is the ath_pci_probe call
that sets up the memory but even that doesn't log anything.

What I find surprising is that the mapping for sc->mem is done (as shown
in the kernel page table dump), and some REG_WRITE worked before, but
the first REG_READ fails despite being an aligned address so something
is odd with how the memory is setup?


I think it would be interesting to add some debug statement early in
init (pci init to check how the memory is setup, if I'm not wrong about
it being pci, and along the path I described earlier) ; but my free time
ended long ago so don't think I'll be of much more use from now :p

-- 
Dominique Martinet | Asmadeus

Re: ath9k driver may be broken on ARM64 ?

[Dominique Martinet]

Dominique Martinet wrote on Thu, Oct 25, 2018:
> What I find surprising is that the mapping for sc->mem is done (as shown
> in the kernel page table dump), and some REG_WRITE worked before, but
> the first REG_READ fails despite being an aligned address so something
> is odd with how the memory is setup?
> 
> 
> I think it would be interesting to add some debug statement early in
> init (pci init to check how the memory is setup, if I'm not wrong about
> it being pci, and along the path I described earlier) ; but my free time
> ended long ago so don't think I'll be of much more use from now :p

(not sure if you got notified off list, but fwiw the bug is apparently
resolved by upgrading to a newer release of u-boot - same version, from
u-boot 2017.03 armada v17.06 to u-boot 2017.03 armada v17.10
I have no idea what they're doing in u-boot with the memory setup but I
guess the case is closed...)

-- 
Asmadeus | Dominique Martinet