[ath9k-devel] ath9k-issue with stop RX/TX DMA - correlation with WMM traffic classes?

[ath9k-devel] ath9k-issue with stop RX/TX DMA - correlation with WMM traffic classes?

[Rougu]

Hi everyone,

while I might be totally wrong with my observation, I still would like 
to share it, because this issue is lingering for years now and many 
users are waiting to a solid grip on it to fix it.

I'm running several freifunk-nodes in Cologne, Germany, where the 
firmware contains a ath9k-watchdog that restarts the node, once the 
failure occurs. One of my nodes is affected in varying intervals of 200 
.. 10000 seconds uptime, depending on WLAN activity.

(openwrt, attitude adjustment, kernel 3.3.8)

Looking at "/sys/kernel/debug/ieee80211/phy0/ath9k/xmit", it seems that
MPDUs in the highest priority queue "VO" have a lower rate of 
"completion" (90,5%) than those in "BE" (>99,9%).

 From my laymen's perspective I would expect "VO" to have an even better 
completion rate than "BE".

Any chance to see a correlation between WMM priority queue handling and 
DMA access problems?

Regards,

Rougu




root at 64700284ede8:~# uptime
  09:52:13 up 14:33,  load average: 0.00, 0.01, 0.04

root at 64700284ede8:~# cat /sys/kernel/debug/ieee80211/phy0/ath9k/xmit
Num-Tx-Queues: 10  tx-queues-setup: 0x10f poll-work-seen: 52392
                             BE         BK        VI        VO

MPDUs Queued:          1349013         22         0      8497
MPDUs Completed:       1348756         22         0      7694
MPDUs XRetried:            257          0         0       803
Aggregates:                116         13         0         0
AMPDUs Queued HW:         2448        316         0         0
AMPDUs Queued SW:          364         72         0         0
AMPDUs Completed:         2767        388         0         0
AMPDUs Retried:            281         24         0         0
AMPDUs XRetried:            44          0         0         0
FIFO Underrun:               0          0         0         0
TXOP Exceeded:               0          0         0         0
TXTIMER Expiry:              0          0         0         0
DESC CFG Error:              0          0         0         0
DATA Underrun:               0          0         0         0
DELIM Underrun:              0          0         0         0
TX-Pkts-All:           1351824        410         0      8497
TX-Bytes-All:        184784983     454112         0   1280336
hw-put-tx-buf:               1          1         0         2
hw-tx-start:           1351887        376         0      8497
hw-tx-proc-desc:       1351887        376         0      8497
TX-Failed:                   0          0         0         0
txq-memory-address:   81b7a1b4   81b7a230  81b7a138  81b7a0bc
axq-qnum:                    2          3         1         0
axq-depth:                   0          0         0         0
axq-ampdu_depth:             0          0         0         0
axq-stopped                  0          0         0         0
tx-in-progress               0          0         0         0
pending-frames               0          0         0         0
txq_headidx:                 0          0         0         0
txq_tailidx:                 0          0         0         0
axq_q empty:                   0          0         1         0
axq_acq empty:                 1          1         1         1
txq_fifo[0] empty:             1          1         1         1
txq_fifo[1] empty:             1          1         1         1
txq_fifo[2] empty:             1          1         1         1
txq_fifo[3] empty:             1          1         1         1
txq_fifo[4] empty:             1          1         1         1
txq_fifo[5] empty:             1          1         1         1
txq_fifo[6] empty:             1          1         1         1
txq_fifo[7] empty:             1          1         1         1

[ath9k-devel] ath9k-issue with stop RX/TX DMA - correlation with WMM traffic classes?

[Felix Fietkau]

On 2013-04-12 10:27 AM, Rougu wrote:
> Hi everyone,
> 
> while I might be totally wrong with my observation, I still would like 
> to share it, because this issue is lingering for years now and many 
> users are waiting to a solid grip on it to fix it.
> 
> I'm running several freifunk-nodes in Cologne, Germany, where the 
> firmware contains a ath9k-watchdog that restarts the node, once the 
> failure occurs. One of my nodes is affected in varying intervals of 200 
> .. 10000 seconds uptime, depending on WLAN activity.
> 
> (openwrt, attitude adjustment, kernel 3.3.8)
Please also run a test with OpenWrt trunk and compare stability.

> Looking at "/sys/kernel/debug/ieee80211/phy0/ath9k/xmit", it seems that
> MPDUs in the highest priority queue "VO" have a lower rate of 
> "completion" (90,5%) than those in "BE" (>99,9%).
> 
>  From my laymen's perspective I would expect "VO" to have an even better 
> completion rate than "BE".
What you're missing in that perspective is the fact that not all traffic
is created equal :)
The VO queue carries management traffic. Probe responses frequently end
up in XRetry because the number of retries is intentionally limited to 1
(some clients are often either too far away to ACK the probe response,
or do not stick around long enough to receive it).

> Any chance to see a correlation between WMM priority queue handling and 
> DMA access problems?
No, the data is too limited for that.

- Felix

[ath9k-devel] ath9k-issue with stop RX/TX DMA - correlation with WMM traffic classes?

[Jan Lühr]

Hallo Felix,

Am 12.04.2013 um 17:41 schrieb Felix Fietkau:
> On 2013-04-12 10:27 AM, Rougu wrote:
>> while I might be totally wrong with my observation, I still would like 
>> to share it, because this issue is lingering for years now and many 
>> users are waiting to a solid grip on it to fix it.
>> 
>> I'm running several freifunk-nodes in Cologne, Germany, where the 
>> firmware contains a ath9k-watchdog that restarts the node, once the 
>> failure occurs. One of my nodes is affected in varying intervals of 200 
>> .. 10000 seconds uptime, depending on WLAN activity.
>> 
>> (openwrt, attitude adjustment, kernel 3.3.8)
> Please also run a test with OpenWrt trunk and compare stability.

thanks for providing feedback on this issue. Please keep in mind that using OpenWRT-trunk is not that easy for node operators of freifunk networks. I hope Rouge is able to do so - at least we will provide help in our Freifunk-Community (kbu).

I'd really like to discuss this issue at the upcoming Wireless Community Weekend in Berlin - will I meet you there?
At the moment we really do suffer from excessive driver-woos - causing the need to reboot nodes:
~ 1450 incidents have been reported within one month (http://register.kbu.freifunk.net/watchdog_bites). Some nodes are almost DoS'ed by certain - yet undiscovered - clients.

Thanks,
yanosz

[ath9k-devel] ath9k-issue with stop RX/TX DMA - correlation with WMM traffic classes?

[Felix Fietkau]

On 2013-04-12 6:57 PM, Jan L?hr wrote:
> Hallo Felix,
> 
> Am 12.04.2013 um 17:41 schrieb Felix Fietkau:
>> On 2013-04-12 10:27 AM, Rougu wrote:
>>> while I might be totally wrong with my observation, I still would like 
>>> to share it, because this issue is lingering for years now and many 
>>> users are waiting to a solid grip on it to fix it.
>>> 
>>> I'm running several freifunk-nodes in Cologne, Germany, where the 
>>> firmware contains a ath9k-watchdog that restarts the node, once the 
>>> failure occurs. One of my nodes is affected in varying intervals of 200 
>>> .. 10000 seconds uptime, depending on WLAN activity.
>>> 
>>> (openwrt, attitude adjustment, kernel 3.3.8)
>> Please also run a test with OpenWrt trunk and compare stability.
> 
> thanks for providing feedback on this issue. Please keep in mind
> that using OpenWRT-trunk is not that easy for node operators of freifunk
> networks. I hope Rouge is able to do so - at least we will provide help
> in our Freifunk-Community (kbu).
I'm also considering backporting mac80211 from trunk to AA after the
release (meant for people that build the branch from source, not for a
follow-up release).

> I'd really like to discuss this issue at the upcoming Wireless
> Community Weekend in Berlin - will I meet you there?
Yes, definitely.

> At the moment we really do suffer from excessive driver-woos -
> causing the need to reboot nodes:
> ~ 1450 incidents have been reported within one month
> (http://register.kbu.freifunk.net/watchdog_bites). Some nodes are almost
> DoS'ed by certain - yet undiscovered - clients.
I'm really interested how such a setup behaves with either trunk or a
mac80211 backport to AA. Many fixes were added that could not easily be
backported to AA because they were too intrusive for the release.

- Felix