Timeout Issues

Timeout Issues

[Pushpal Sidhu]

Hi,

I'm having issues with ath10k timing out when trying to communicate
with the wireless card (UNEX DAXA-O1 + have tried several others).
Below is what I grabbed from dmesg.

$ dmesg | grep ath10k
[    5.233469] ath10k_pci 0000:09:00.0: enabling device (0140 -> 0142)
[    5.575954] ath10k: pci irq legacy irq_mode 0 reset_mode 0
[    7.704403] ath10k: failed to receive control response completion, polling..
[    8.704379] ath10k: Service connect timeout: -110
[    8.709089] ath10k: failed to connect htt (-110)
[    9.020495] ath10k: could not init core (-110)
[    9.324371] ath10k: could not probe fw (-110)

This causes the card to not be registered, and thus is unusable (there
is no wlanX)

I am able to use the Intel 7260 card (another AC card) with their
firmware binary blob, so I don't believe this is a hardware
configuration issue with mini-pcie.

I am using the following software on an ARM board:
ath10k branch of linux kernel: 3.16.0-wl-ath-182333-g6a69f0c
latest git of hostapd
latest git of iw
latest git of wpa_supplicant

I have tried both firmware-2 binary blobs (the "AP" and "STA"
versions) as well as the newer firmware-3 binary with the same exact
messages being displayed.

Any help towards finding a solution is appreciated, thanks!

- Pushpal Sidhu

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Timeout Issues

[Ben Greear]

On 08/19/2014 11:27 AM, Pushpal Sidhu wrote:
> Hi,
> 
> I'm having issues with ath10k timing out when trying to communicate
> with the wireless card (UNEX DAXA-O1 + have tried several others).

Have you tried a WLE900 VX NIC?  Those have worked well for me.


> Below is what I grabbed from dmesg.
> 
> $ dmesg | grep ath10k
> [    5.233469] ath10k_pci 0000:09:00.0: enabling device (0140 -> 0142)
> [    5.575954] ath10k: pci irq legacy irq_mode 0 reset_mode 0
> [    7.704403] ath10k: failed to receive control response completion, polling..
> [    8.704379] ath10k: Service connect timeout: -110
> [    8.709089] ath10k: failed to connect htt (-110)
> [    9.020495] ath10k: could not init core (-110)
> [    9.324371] ath10k: could not probe fw (-110)
> 
> This causes the card to not be registered, and thus is unusable (there
> is no wlanX)
> 
> I am able to use the Intel 7260 card (another AC card) with their
> firmware binary blob, so I don't believe this is a hardware
> configuration issue with mini-pcie.
> 
> I am using the following software on an ARM board:
> ath10k branch of linux kernel: 3.16.0-wl-ath-182333-g6a69f0c
> latest git of hostapd
> latest git of iw
> latest git of wpa_supplicant
> 
> I have tried both firmware-2 binary blobs (the "AP" and "STA"
> versions) as well as the newer firmware-3 binary with the same exact
> messages being displayed.
> 
> Any help towards finding a solution is appreciated, thanks!

Probably this is not a firmware issue, but if you have nothing else
to try, you can try my version of the 10.1.x firmware:

http://www.candelatech.com/ath10k.php

and possibly my hacked 3.14.14 kernel, which may spew out some additional
debug info if you have something like OTP issues.

git clone git://dmz2.candelatech.com/linux-3.14.dev.y

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Timeout Issues

[Pushpal Sidhu]

On Tue, Aug 19, 2014 at 11:42 AM, Ben Greear <greearb@candelatech.com> wrote:
> Have you tried a WLE900 VX NIC?  Those have worked well for me.

I haven't yet, though I'll get my hands on one to try out.  I have
tried a WLE900N5-20 with the
same results below, however.

>> I am using the following software on an ARM board:
>> ath10k branch of linux kernel: 3.16.0-wl-ath-182333-g6a69f0c
>> latest git of hostapd
>> latest git of iw
>> latest git of wpa_supplicant
>>
>> I have tried both firmware-2 binary blobs (the "AP" and "STA"
>> versions) as well as the newer firmware-3 binary with the same exact
>> messages being displayed.
>
> Probably this is not a firmware issue, but if you have nothing else
> to try, you can try my version of the 10.1.x firmware:
>
> http://www.candelatech.com/ath10k.php
>
> and possibly my hacked 3.14.14 kernel, which may spew out some additional
> debug info if you have something like OTP issues.
>
> git clone git://dmz2.candelatech.com/linux-3.14.dev.y

With the Kernel I was using (the 3.16.0-wl-ath-x-x), your firmware
produced the following:
[    5.466186] ath10k: Unknown FW IE: 5

and then output the same timed out messages. The upstream ath10k
driver that I'm using is
looking for firmware-3.bin it seems.

I cloned your kernel and built it, but was having the same exact
errors I posted before (no extra
debug info). One interesting thing I found is if I use OpenWrt trunk,
ath10k seems to behave fine,
but another distro seems to break things. I'll need to dig further
into that, but in the mean time, do
you have any thoughts?

- Pushpal

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Timeout Issues

[Kalle Valo]

Pushpal Sidhu <psidhu@gateworks.com> writes:

> One interesting thing I found is if I use OpenWrt trunk, ath10k seems
> to behave fine, but another distro seems to break things. I'll need to
> dig further into that, but in the mean time, do you have any thoughts?

What do you mean? On the exact same board ath10k works with openwrt but
with another distribution it fails?

What's this another distribution?

-- 
Kalle Valo

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Timeout Issues

[Michal Kazior]

On 20 August 2014 07:40, Kalle Valo <kvalo@qca.qualcomm.com> wrote:
> Pushpal Sidhu <psidhu@gateworks.com> writes:
>
>> One interesting thing I found is if I use OpenWrt trunk, ath10k seems
>> to behave fine, but another distro seems to break things. I'll need to
>> dig further into that, but in the mean time, do you have any thoughts?
>
> What do you mean? On the exact same board ath10k works with openwrt but
> with another distribution it fails?
>
> What's this another distribution?

I'm rather interested in the toolchain differences. Perhaps the
compiler you used to build 3.16 reorders instructions so much it
breaks driver interaction with the hardware. This would suggests
ath10k is missing some compiler/memory barriers.

Can you apply the following patch and see if it changes anything, please?


From 3f5fc9bae19a0b429370025e781701472fc64529 Mon Sep 17 00:00:00 2001
From: Michal Kazior <michal.kazior@tieto.com>
Date: Wed, 20 Aug 2014 08:30:15 +0200
Subject: [PATCH] ath10k: add memory barries

Testing purposes.

Signed-off-by: Michal Kazior <michal.kazior@tieto.com>
---
 drivers/net/wireless/ath/ath10k/htc.c | 1 +
 drivers/net/wireless/ath/ath10k/pci.c | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/drivers/net/wireless/ath/ath10k/htc.c
b/drivers/net/wireless/ath/ath10k/htc.c
index 7e08bb3..a1eb639 100644
--- a/drivers/net/wireless/ath/ath10k/htc.c
+++ b/drivers/net/wireless/ath/ath10k/htc.c
@@ -682,6 +682,7 @@ int ath10k_htc_connect_service(struct ath10k_htc *htc,
        req_msg->service_id = __cpu_to_le16(conn_req->service_id);

        reinit_completion(&htc->ctl_resp);
+       mb();

        status = ath10k_htc_send(htc, ATH10K_HTC_EP_0, skb);
        if (status) {
diff --git a/drivers/net/wireless/ath/ath10k/pci.c
b/drivers/net/wireless/ath/ath10k/pci.c
index 98c029b..673de9f 100644
--- a/drivers/net/wireless/ath/ath10k/pci.c
+++ b/drivers/net/wireless/ath/ath10k/pci.c
@@ -1115,6 +1115,8 @@ static int ath10k_pci_hif_start(struct ath10k *ar)
                goto err_stop;
        }

+       mb();
+
        /* Post buffers once to start things off. */
        ret = ath10k_pci_post_rx(ar);
        if (ret) {
-- 
1.8.5.3

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Timeout Issues

[Pushpal Sidhu]

Sorry for the amazingly late reply, other things crept into my schedule.

Kalle Valo <kvalo@qca.qualcomm.com> writes:
>Have you verified that with this kernel version the PCI bus works
>correctly on your ARM board? Especially check that interrupts work.

Michal Kazior <michal.kazior@tieto.com> writes:
>Note the:
>
>[    7.704403] ath10k: failed to receive control response completion, polling..
>
>This suggests interrupts aren't delivered to the host system for some
>reason although firmware is operational (control response itself was
>delivered to the host).
>
>This particular warning was introduced as part of a workaround for a
>virtual machine PCI passthrough (with the suspicion iomem writes
>aren't propagated properly in the VM). However only control response
>was the subject of the missing interrupt. In your case subsequent
>communication is missing interrupts.
>
>Can you check if the interrupt line is shared with any other devices, please:
>
> lspci -vv | grep -e '..:..\..' -e IRQ
>  cat /proc/interrupts
>
>
>>
>> This causes the card to not be registered, and thus is unusable (there
>>is no wlanX)
>>
>> I am able to use the Intel 7260 card (another AC card) with their
>> firmware binary blob, so I don't believe this is a hardware
>> configuration issue with mini-pcie.
>
>I assume you have only 1 mini-pcie slot on the board and you've used
>it when testing both intel and ath devices or does it have more than 1
>slot and you've used different slots when testing both devices?
>
>
>Michał

The board has 9 port PCI-e switch behind the RC. I did, however, use
the same slot. I found that there was always 1 interrupt, but never
any more. This made me think a bit more and found that if I set
PCI_MSI=n in the kernel config, the problem 'goes away' and I see
increasing number of interrupts and do not see anymore of the '-110'
error messages. This issue, for me, is solved as I don't necessary
need mixed signaled interrupts, but I'm unsure of why this is
happening and what is different about the Intel 7260HMW card from the
Atheros cards or in their respective drivers.

- Pushpal

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Timeout Issues

[Adrian Chadd]

.. wasn't there some odd-ness with the QCA988x MSI stuff anyway?

I recall driver WAR's in the 1.0 world; I haven't checked what 2.0 fixed.


-a

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Timeout Issues

[Michal Kazior]

On 2 September 2014 20:51, Pushpal Sidhu <psidhu@gateworks.com> wrote:
> Sorry for the amazingly late reply, other things crept into my schedule.
[...]
>
> The board has 9 port PCI-e switch behind the RC. I did, however, use
> the same slot. I found that there was always 1 interrupt, but never
> any more. This made me think a bit more and found that if I set
> PCI_MSI=n in the kernel config, the problem 'goes away' and I see
> increasing number of interrupts and do not see anymore of the '-110'
> error messages. This issue, for me, is solved as I don't necessary
> need mixed signaled interrupts, but I'm unsure of why this is
> happening and what is different about the Intel 7260HMW card from the
> Atheros cards or in their respective drivers.

Hmm.. Your original logs suggest ath10k was already running in
legacy/shared irq mode before which means it already tried MSI and
failed:

> [    5.233469] ath10k_pci 0000:09:00.0: enabling device (0140 -> 0142)
> [    5.575954] ath10k: pci irq legacy irq_mode 0 reset_mode 0

I wonder if the Intel card succeeds to pci_enable_msi(). Perhaps this
is a bug with QCA988X failing to recover from a failed MSI negotiation
with the PCI-E controller?

You could try adding irq_mode=1 parameter to ath10k_pci module. This
way you'll force ath10k to use shared interrupts without even trying
to go through MSI->shared failover and if I'm right you won't need the
PCI_MSI=n in your kernel.


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Timeout Issues

[Kalle Valo]

Pushpal Sidhu <psidhu@gateworks.com> writes:

> I'm having issues with ath10k timing out when trying to communicate
> with the wireless card (UNEX DAXA-O1 + have tried several others).
> Below is what I grabbed from dmesg.
>
> $ dmesg | grep ath10k
> [    5.233469] ath10k_pci 0000:09:00.0: enabling device (0140 -> 0142)
> [    5.575954] ath10k: pci irq legacy irq_mode 0 reset_mode 0
> [    7.704403] ath10k: failed to receive control response completion, polling..
> [    8.704379] ath10k: Service connect timeout: -110
> [    8.709089] ath10k: failed to connect htt (-110)
> [    9.020495] ath10k: could not init core (-110)
> [    9.324371] ath10k: could not probe fw (-110)
>
> This causes the card to not be registered, and thus is unusable (there
> is no wlanX)

My first guess is a problem related with the PCI bus, but that's just a
guess. Can you get us a trace-cmd log:

http://wireless.kernel.org/en/users/Drivers/ath10k/debug#Tracing

> I am able to use the Intel 7260 card (another AC card) with their
> firmware binary blob, so I don't believe this is a hardware
> configuration issue with mini-pcie.

Did you try iwlwifi on the exact same kernel?

> I am using the following software on an ARM board:
> ath10k branch of linux kernel: 3.16.0-wl-ath-182333-g6a69f0c

Have you verified that with this kernel version the PCI bus works
correctly on your ARM board? Especially check that interrupts work.

-- 
Kalle Valo

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Timeout Issues

[Kalle Valo]

Kalle Valo <kvalo@qca.qualcomm.com> writes:

> Pushpal Sidhu <psidhu@gateworks.com> writes:
>
>> I'm having issues with ath10k timing out when trying to communicate
>> with the wireless card (UNEX DAXA-O1 + have tried several others).
>> Below is what I grabbed from dmesg.
>>
>> $ dmesg | grep ath10k
>> [    5.233469] ath10k_pci 0000:09:00.0: enabling device (0140 -> 0142)
>> [    5.575954] ath10k: pci irq legacy irq_mode 0 reset_mode 0
>> [    7.704403] ath10k: failed to receive control response completion, polling..
>> [    8.704379] ath10k: Service connect timeout: -110
>> [    8.709089] ath10k: failed to connect htt (-110)
>> [    9.020495] ath10k: could not init core (-110)
>> [    9.324371] ath10k: could not probe fw (-110)
>>
>> This causes the card to not be registered, and thus is unusable (there
>> is no wlanX)
>
> My first guess is a problem related with the PCI bus, but that's just a
> guess. Can you get us a trace-cmd log:
>
> http://wireless.kernel.org/en/users/Drivers/ath10k/debug#Tracing

Just realised that's not possible as you would have to run trace-cmd
running before ath10k is loaded and that won't work.

debug_mask with 0xffffffff is better:

http://wireless.kernel.org/en/users/Drivers/ath10k/debug#Debug_log_messages

-- 
Kalle Valo

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Timeout Issues

[Michal Kazior]

On 19 August 2014 20:27, Pushpal Sidhu <psidhu@gateworks.com> wrote:
> Hi,
>
> I'm having issues with ath10k timing out when trying to communicate
> with the wireless card (UNEX DAXA-O1 + have tried several others).
> Below is what I grabbed from dmesg.
>
> $ dmesg | grep ath10k
> [    5.233469] ath10k_pci 0000:09:00.0: enabling device (0140 -> 0142)
> [    5.575954] ath10k: pci irq legacy irq_mode 0 reset_mode 0
> [    7.704403] ath10k: failed to receive control response completion, polling..
> [    8.704379] ath10k: Service connect timeout: -110
> [    8.709089] ath10k: failed to connect htt (-110)
> [    9.020495] ath10k: could not init core (-110)
> [    9.324371] ath10k: could not probe fw (-110)

Note the:

[    7.704403] ath10k: failed to receive control response completion, polling..

This suggests interrupts aren't delivered to the host system for some
reason although firmware is operational (control response itself was
delivered to the host).

This particular warning was introduced as part of a workaround for a
virtual machine PCI passthrough (with the suspicion iomem writes
aren't propagated properly in the VM). However only control response
was the subject of the missing interrupt. In your case subsequent
communication is missing interrupts.

Can you check if the interrupt line is shared with any other devices, please:

  lspci -vv | grep -e '..:..\..' -e IRQ
  cat /proc/interrupts


>
> This causes the card to not be registered, and thus is unusable (there
> is no wlanX)
>
> I am able to use the Intel 7260 card (another AC card) with their
> firmware binary blob, so I don't believe this is a hardware
> configuration issue with mini-pcie.

I assume you have only 1 mini-pcie slot on the board and you've used
it when testing both intel and ath devices or does it have more than 1
slot and you've used different slots when testing both devices?


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k