htt rx stopped. cannot recover

htt rx stopped. cannot recover

[Pushpal Sidhu]

Hello everyone.

I am having issues that occur when I have heavy network traffic in a
wireless bridge. That is:
  {lan1}<--ether-->AP<--wifi-->STA<--ether-->{lan2}

Note that the above scenario requires 4addr to be turned on. Once I
associate the AP and STAtion and begin passing traffic through at a
rate of greater than ~250mbps TCP for several seconds, I see the below
on the STAtion:

> ...
> [   50.437806] wlan0: authenticate with 00:30:1a:4e:00:5d
> [   51.443940] wlan0: send auth to 00:30:1a:4e:00:5d (try 1/3)
> [   51.450692] wlan0: authenticated
> [   51.464272] wlan0: associate with 00:30:1a:4e:00:5d (try 1/3)
> [   51.471328] wlan0: RX AssocResp from 00:30:1a:4e:00:5d (capab=0x1 status=0 aid=1)
> [   51.992991] wlan0: associated
> [   51.996455] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
> [   52.005001] br-lan: port 2(wlan0) entered forwarding state
> [   52.010561] br-lan: port 2(wlan0) entered forwarding state
Start passing traffic through
> [   61.614230] br-lan: port 1(eth1) entered forwarding state
> [   63.331225] ath10k_pci 0000:07:00.0: htt rx received chained non A-MSDU frame
> [   67.054654] br-lan: port 2(wlan0) entered forwarding state
> [   68.173786] ath10k_pci 0000:07:00.0: htt rx received chained non A-MSDU frame
> [   76.679431] ath10k_pci 0000:07:00.0: htt rx stopped. cannot recover
> [   76.685770] ath10k_pci 0000:07:00.0: failed to pop amsdu from htt rx ring -1
> [   76.692844] ath10k_pci 0000:07:00.0: htt is confused. refusing rx
> [   76.699005] ath10k_pci 0000:07:00.0: failed to pop amsdu from htt rx ring -1
> [   76.706138] ath10k_pci 0000:07:00.0: htt is confused. refusing rx
> [   76.712256] ath10k_pci 0000:07:00.0: failed to pop amsdu from htt rx ring -1
> [   76.719377] ath10k_pci 0000:07:00.0: htt is confused. refusing rx
> ...

Firmware: 10.1.467.2-1 (I've also tried 10.2-00082-4-2)
Kernel: 3.17.0-rc1-193451-g707b1bbd (From ath-next; I've also tried
several other kernel versions)
ath10k source: Current ath-next
(707b1bbd7e6ec533cfa027054ccbeb90113a40b0; I've also tried
backports-20141023 and backports-20140808)
Arch: arm
Software: Latest (released) wpa_supplicant and latest (released) hostapd
Radios: seen on both the ACE-DB-3 and WLE900VX

I've also tried disabling A-MSDU via "echo "1 64" >
htt_max_amsdu_ampdu", but continue seeing the issue. Any help on this
is appreciated.

Thanks,
Pushpal

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Pushpal Sidhu]

On Mon, Nov 10, 2014 at 2:11 PM, Ben Greear <greearb@candelatech.com> wrote:
>
> You might try my 'CT' firmware?
>
> http://www.candelatech.com/ath10k.php
>
> I'd be curious to know if it helps or not...

I just tried the "12" firmware on the station, but to no avail.

- Pushpal

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Ben Greear]

On 11/10/2014 02:21 PM, Pushpal Sidhu wrote:
> On Mon, Nov 10, 2014 at 2:11 PM, Ben Greear <greearb@candelatech.com> wrote:
>>
>> You might try my 'CT' firmware?
>>
>> http://www.candelatech.com/ath10k.php
>>
>> I'd be curious to know if it helps or not...
> 
> I just tried the "12" firmware on the station, but to no avail.

Same problem, or totally not working?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Pushpal Sidhu]

On Mon, Nov 10, 2014 at 2:28 PM, Ben Greear <greearb@candelatech.com> wrote:
> Same problem, or totally not working?

Same problem. I start passing traffic through and once there is
constant traffic of > 250mbps for several seconds, the issue occurs.

I forgot to mention in my original email that traffic stops once the
problem occurs.

- Pushpal

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Michal Kazior]

On 10 November 2014 23:33, Pushpal Sidhu <psidhu@gateworks.com> wrote:
> On Mon, Nov 10, 2014 at 2:28 PM, Ben Greear <greearb@candelatech.com> wrote:
>> Same problem, or totally not working?
>
> Same problem. I start passing traffic through and once there is
> constant traffic of > 250mbps for several seconds, the issue occurs.
>
> I forgot to mention in my original email that traffic stops once the
> problem occurs.

That's a safety mechanism to prevent memory corruption on the host.
The error you see is most likely an indication fw/hw Rx reporting
broke, i.e. fw says to pop more frames from the Rx ring than it is
ready to be popped.


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Pushpal Sidhu]

On Tue, Nov 11, 2014 at 10:31 PM, Michal Kazior <michal.kazior@tieto.com> wrote:
> That's a safety mechanism to prevent memory corruption on the host.
> The error you see is most likely an indication fw/hw Rx reporting
> broke, i.e. fw says to pop more frames from the Rx ring than it is
> ready to be popped.

So if this seems to indicate a hw/fw issue, how should I go about
fixing it? If it's a fw issue, it just goes to show how keeping this
particular fw closed-source is hurting this project more than anything
in my opinion.

Has anyone else been able to recreate this problem? If not, there
might be something else at play, but until someone else can recreate
the issue, I can't tell exactly where the problem lies. At first I
thought it was a 4addr mode issue, but I performed the following
scenario and believe that it might be related to bridging.
 {lan1}<--ether-->AP<--wifi-->STA

In this scenario, the AP bridges eth+wlan, with 4addr turned off (the
STA has no bridge). When I pass traffic through from lan1 to the
STAtion, the same issue occurs on the AP, this time with traffic above
~220mbps TCP. Does anyone know of any specific kernel configs relating
to bridging that I might not have enabled?

I understand that stopping traffic is a safety mechanism, but is there
a better way to handle the situation? I don't believe resetting
hardware is the proper solution as this requires reauthentication
(which takes time), only for the problem to remanifest itself in only
a few seconds of passing traffic through.

Thanks for any input,
Pushpal

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Michal Kazior]

On 13 November 2014 20:09, Pushpal Sidhu <psidhu@gateworks.com> wrote:
> On Tue, Nov 11, 2014 at 10:31 PM, Michal Kazior <michal.kazior@tieto.com> wrote:
[...]
> Has anyone else been able to recreate this problem? If not, there
> might be something else at play, but until someone else can recreate
> the issue, I can't tell exactly where the problem lies. At first I
> thought it was a 4addr mode issue, but I performed the following
> scenario and believe that it might be related to bridging.
>  {lan1}<--ether-->AP<--wifi-->STA
>
> In this scenario, the AP bridges eth+wlan, with 4addr turned off (the
> STA has no bridge). When I pass traffic through from lan1 to the
> STAtion, the same issue occurs on the AP, this time with traffic above
> ~220mbps TCP. Does anyone know of any specific kernel configs relating
> to bridging that I might not have enabled?

I've used ath10k AP in a bridge many times and haven't seen this issue yet.

Putting an interface into a bridge enables promiscuous mode on it. In
ath10k this is handled by creating a monitor vdev to implicitly
influence Rx filters so that hw/fw pass everything up to host. I
suspect your RF environment may contain particular traffic patterns
which trigger the problem within firmware code related to monitor
vdev.

You could try hacking up ath10k to not create/start monitor vdev and
see if you can still reproduce the problem. Keep in mind bridging will
be crippled in some cases since firmware won't deliver some frames to
ath10k.

There's also new WMI_10X_PDEV_PARAM_RX_FILTER that was introduced in
10.2. I didn't have time to play with it yet. In theory it might be
able possible to use it instead of monitor vdev for promisc mode, at
least with 10.2. I'm not really fond of the idea though because this
will add more complexity and different codepaths to the driver
depending on firmware revision.


> I understand that stopping traffic is a safety mechanism, but is there
> a better way to handle the situation? I don't believe resetting
> hardware is the proper solution as this requires reauthentication
> (which takes time), only for the problem to remanifest itself in only
> a few seconds of passing traffic through.

Hmm.. Now that I think about it, after the recent Rx rework it might
be possible to circumvent the problem. ath10k uses very little data
from Rx indication event and instead Rx descriptor is used for most
things. Popping function would need some changes so that it can back
off safely if a frame buffer isn't ready. ath10k would probably need
to poll for Rx too.


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Pushpal Sidhu]

On Fri, Nov 14, 2014 at 12:31 AM, Michal Kazior <michal.kazior@tieto.com> wrote:
> I've used ath10k AP in a bridge many times and haven't seen this issue yet.

I just found that I can reproduce the problem passing traffic through
from AP <-wifi-> STA with no bridging. The reason I didn't see the
problem before is that it takes considerably longer to reproduce and I
wasn't necessarily looking for it (65 seconds vs 5 seconds).

> Putting an interface into a bridge enables promiscuous mode on it. In
> ath10k this is handled by creating a monitor vdev to implicitly
> influence Rx filters so that hw/fw pass everything up to host. I
> suspect your RF environment may contain particular traffic patterns
> which trigger the problem within firmware code related to monitor
> vdev.
>
> You could try hacking up ath10k to not create/start monitor vdev and
> see if you can still reproduce the problem. Keep in mind bridging will
> be crippled in some cases since firmware won't deliver some frames to
> ath10k.

I've actually been using attenuators for these tests, a little more
than 60dB on each chain and as I mentioned above, I don't believe it's
a vdev issue due to the fact that I can reproduce this without
bridging (which doesn't put the radio into promiscuous mode and thus
doesn't create a monitor vdev, correct?).

> Hmm.. Now that I think about it, after the recent Rx rework it might
> be possible to circumvent the problem. ath10k uses very little data
> from Rx indication event and instead Rx descriptor is used for most
> things. Popping function would need some changes so that it can back
> off safely if a frame buffer isn't ready. ath10k would probably need
> to poll for Rx too.

I saw those patches and I liked what I saw. When I tested with them in
place, I found it failed in the same 'test' of checking for MSDU_DONE
flag (duh, just wanted to make that clear, haha). I tried not setting
the confused flag when the issue was hit, but eventually a kernel
panic would occur.

Something very interesting that I found was I only see this issue when
the radio's are behind a PCIe bridge. That is, I tested on two set's
of boards with the same CPU/structure, with the one difference that
there is no PCIe bridge on one. I am currently running an extended
test on the board without PCIe to see if it isn't an issue that will
manifest itself at a later time, but as of now, it's looking like
that's no the case. Do you have any thoughts on this?

- Pushpal

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Michal Kazior]

On 18 November 2014 02:20, Pushpal Sidhu <psidhu@gateworks.com> wrote:
> On Fri, Nov 14, 2014 at 12:31 AM, Michal Kazior <michal.kazior@tieto.com> wrote:
>> I've used ath10k AP in a bridge many times and haven't seen this issue yet.
>
> I just found that I can reproduce the problem passing traffic through
> from AP <-wifi-> STA with no bridging. The reason I didn't see the
> problem before is that it takes considerably longer to reproduce and I
> wasn't necessarily looking for it (65 seconds vs 5 seconds).
>
>> Putting an interface into a bridge enables promiscuous mode on it. In
>> ath10k this is handled by creating a monitor vdev to implicitly
>> influence Rx filters so that hw/fw pass everything up to host. I
>> suspect your RF environment may contain particular traffic patterns
>> which trigger the problem within firmware code related to monitor
>> vdev.
>>
>> You could try hacking up ath10k to not create/start monitor vdev and
>> see if you can still reproduce the problem. Keep in mind bridging will
>> be crippled in some cases since firmware won't deliver some frames to
>> ath10k.
>
> I've actually been using attenuators for these tests, a little more
> than 60dB on each chain and as I mentioned above, I don't believe it's
> a vdev issue due to the fact that I can reproduce this without
> bridging (which doesn't put the radio into promiscuous mode and thus
> doesn't create a monitor vdev, correct?).

You can get super sure and take a look in dmesg/syslog to verify if
ath10k's wlan interface is in promisc mode or not. I guess it's not
though.


>> Hmm.. Now that I think about it, after the recent Rx rework it might
>> be possible to circumvent the problem. ath10k uses very little data
>> from Rx indication event and instead Rx descriptor is used for most
>> things. Popping function would need some changes so that it can back
>> off safely if a frame buffer isn't ready. ath10k would probably need
>> to poll for Rx too.
>
> I saw those patches and I liked what I saw. When I tested with them in
> place, I found it failed in the same 'test' of checking for MSDU_DONE
> flag (duh, just wanted to make that clear, haha). I tried not setting
> the confused flag when the issue was hit, but eventually a kernel
> panic would occur.

I'm glad you like them.

The result of not setting the confused flag in this case is pretty expected :-)


> Something very interesting that I found was I only see this issue when
> the radio's are behind a PCIe bridge. That is, I tested on two set's
> of boards with the same CPU/structure, with the one difference that
> there is no PCIe bridge on one. I am currently running an extended
> test on the board without PCIe to see if it isn't an issue that will
> manifest itself at a later time, but as of now, it's looking like
> that's no the case. Do you have any thoughts on this?

Hmm.. Perhaps there's some kind of race and the host main memory isn't
updated in a timely fashion? In that case ath10k could try to, instead
of immediatelly unmapping, keep on syncing dma to cpu and checking the
MSDU_DONE bit for a few msecs (what an ugly workaround..) before it
finally unmaps the frame.

I wonder if that's a bug in ath10k's target device PCIe controller or
the PCIe bridge?


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Pushpal Sidhu]

On Mon, Nov 17, 2014 at 10:56 PM, Michal Kazior <michal.kazior@tieto.com> wrote:
> You can get super sure and take a look in dmesg/syslog to verify if
> ath10k's wlan interface is in promisc mode or not. I guess it's not
> though.
It seems that there's a "vdev 0". I'll look into hacking this out
later when I get some more time, though it seems that whenever a
station is associated with an AP, a vdev is always created.

>> ...
>> I tried not setting
>> the confused flag when the issue was hit, but eventually a kernel
>> panic would occur.
>
> I'm glad you like them.
>
> The result of not setting the confused flag in this case is pretty expected :-)
>
>> Something very interesting that I found was I only see this issue when
>> the radio's are behind a PCIe bridge. That is, I tested on two set's
>> of boards with the same CPU/structure, with the one difference that
>> there is no PCIe bridge on one. I am currently running an extended
>> test on the board without PCIe to see if it isn't an issue that will
>> manifest itself at a later time, but as of now, it's looking like
>> that's no the case. Do you have any thoughts on this?
>
> Hmm.. Perhaps there's some kind of race and the host main memory isn't
> updated in a timely fashion? In that case ath10k could try to, instead
> of immediatelly unmapping, keep on syncing dma to cpu and checking the
> MSDU_DONE bit for a few msecs (what an ugly workaround..) before it
> finally unmaps the frame.
>
> I wonder if that's a bug in ath10k's target device PCIe controller or
> the PCIe bridge?

While running the extended test, at about 7 hours of continuous
testing with the two boards without a PCIe bridge (configured as a
wireless bridge e.g. 4addr=on), the STAtion has a kernel panic that
seems to be caused by a double free or something similar. Log is
located here (glad I set 'debug' in kernel cmdline):
http://dpaste.com/1G77X7N (most interesting info located between times
[25684.512971] - [25684.587585]).

This is basically the same kernel oops I saw earlier when I disabled
the confused flag (occurs on skb_release_data). I've ruled out a SMP
issue as I can recreate this with nosmp. Because I see this issue on a
board without a PCIe bridge, I'm leaning towards the root cause of the
issue to maybe be in the ath10k's target device PCIe controller (irq's
being fired is what lead to this code path).

When I get some time later, I'll try that workaround, though I don't
think that's the "right" way to fix this problem.

- Pushpal

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Michal Kazior]

On 19 November 2014 02:23, Pushpal Sidhu <psidhu@gateworks.com> wrote:
> On Mon, Nov 17, 2014 at 10:56 PM, Michal Kazior <michal.kazior@tieto.com> wrote:
>> You can get super sure and take a look in dmesg/syslog to verify if
>> ath10k's wlan interface is in promisc mode or not. I guess it's not
>> though.
> It seems that there's a "vdev 0". I'll look into hacking this out
> later when I get some more time, though it seems that whenever a
> station is associated with an AP, a vdev is always created.

This is fine. Each vif will have a vdev of an according type. Promisc
mode triggers creation of an additional, special vdev (which isn't
associated with any vif whatsoever) with type=monitor


[...]
> While running the extended test, at about 7 hours of continuous
> testing with the two boards without a PCIe bridge (configured as a
> wireless bridge e.g. 4addr=on), the STAtion has a kernel panic that
> seems to be caused by a double free or something similar. Log is
> located here (glad I set 'debug' in kernel cmdline):
> http://dpaste.com/1G77X7N (most interesting info located between times
> [25684.512971] - [25684.587585]).

Is this on vanilla ath10k code or did you ignore the rx_confused flag
during the test?

Anyway it'd be lovely if you could get the code line for
ath10k_htt_txrx_compl_task+0x100 (e.g. via `gdb` with `l *
ath10k_htt_txrx_compl_task+0x100)`.


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: htt rx stopped. cannot recover

[Pushpal Sidhu]

On Tue, Nov 18, 2014 at 10:44 PM, Michal Kazior <michal.kazior@tieto.com> wrote:
>> While running the extended test, at about 7 hours of continuous
>> testing with the two boards without a PCIe bridge (configured as a
>> wireless bridge e.g. 4addr=on), the STAtion has a kernel panic that
>> seems to be caused by a double free or something similar. Log is
>> located here (glad I set 'debug' in kernel cmdline):
>> http://dpaste.com/1G77X7N (most interesting info located between times
>> [25684.512971] - [25684.587585]).
>
> Is this on vanilla ath10k code or did you ignore the rx_confused flag
> during the test?

Vanilla code.

> Anyway it'd be lovely if you could get the code line for
> ath10k_htt_txrx_compl_task+0x100 (e.g. via `gdb` with `l *
> ath10k_htt_txrx_compl_task+0x100)`.

I'm on a slightly constrained system, but here's the info I could grab
(I couldn't get the exact code line):

(gdb) list *ath10k_htt_txrx_compl_task+0x100
0xca3c is in ath10k_htt_txrx_compl_task (include/linux/skbuff.h:987).
982     include/linux/skbuff.h: No such file or directory.
(gdb) list *ath10k_htt_txrx_compl_task+0x100/0x544
0xc93c is in ath10k_htt_txrx_compl_task (include/linux/spinlock.h:290).
285     include/linux/spinlock.h: No such file or directory.

However, if I were to make a guess, I would think the the oops
occurred in the rx_ring.lock when freeing.

From your comment a few emails back, you mentioned that a possible
workaround is to "keep on syncing dma to cpu and checking the
MSDU_DONE bit for a few msecs." I tried this out and found (waited an
infinite amount of time just to verify) that it would never quit out,
meaning the memory wasn't there in the first place.

- Pushpal

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k