More on ath10k_flush.

More on ath10k_flush.

[Ben Greear]

So, I tried adding lots of debug to the ath10k_flush,
and I think I might have found the issue.

My test case is:  create 32 stations, start TCP on all of them,
then reset (effectively ifdown;ifup) the station vifs.

I get lots of spews because the ath10k_flush call thinks it is
hanging.  But, based on the debug, I think the problem is that
the tx logic is continuing to accept packets while the flush
is trying to happen.

If my debugging stats are correct, the NIC managed to send around 4000
frames during the flush attempt, and the ath10k driver processed lots of
tx completions as well.  But there are still lots of pkts
pending tx and the tx queue was never empty.

ath10k: failed to flush transmit queue (skip 0 ar-state 1 pending_tx 1009  pre-pending-tx: 1045): 0
ath10k: pre: htc-send-tot: 3423  htt-tx 2545397  tx-compl 2544355  mgt-tx 0  mgt-compl 0
ath10k: post: htc-send-tot: 3424  htt-tx 2630981  tx-compl 2630156  mgt-tx 0  mgt-compl 0
ath10k: pre: hw-queued: 124395  hw-reaped: 124394
ath10k: post: hw-queued: 128418  hw-reaped: 128417


Do we need to somehow shut down all tx logic during the ath10k_flush
so that (other?) stations cannot transmit?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: More on ath10k_flush.

[Michal Kazior]

On 1 April 2014 02:24, Ben Greear <greearb@candelatech.com> wrote:
> So, I tried adding lots of debug to the ath10k_flush,
> and I think I might have found the issue.
>
> My test case is:  create 32 stations, start TCP on all of them,
> then reset (effectively ifdown;ifup) the station vifs.
>
> I get lots of spews because the ath10k_flush call thinks it is
> hanging.  But, based on the debug, I think the problem is that
> the tx logic is continuing to accept packets while the flush
> is trying to happen.
>
> If my debugging stats are correct, the NIC managed to send around 4000
> frames during the flush attempt, and the ath10k driver processed lots of
> tx completions as well.  But there are still lots of pkts
> pending tx and the tx queue was never empty.
>
> ath10k: failed to flush transmit queue (skip 0 ar-state 1 pending_tx 1009  pre-pending-tx: 1045): 0
> ath10k: pre: htc-send-tot: 3423  htt-tx 2545397  tx-compl 2544355  mgt-tx 0  mgt-compl 0
> ath10k: post: htc-send-tot: 3424  htt-tx 2630981  tx-compl 2630156  mgt-tx 0  mgt-compl 0
> ath10k: pre: hw-queued: 124395  hw-reaped: 124394
> ath10k: post: hw-queued: 128418  hw-reaped: 128417

What do these prints reflect exactly? What base upstream ath10k commit
are you on?


> Do we need to somehow shut down all tx logic during the ath10k_flush
> so that (other?) stations cannot transmit?

This seems strange. drv_flush() is called with all tx queues being
stopped in mac80211 with IEEE80211_QUEUE_STOP_REASON_FLUSH. I guess
you can still receive a few frames (due to SMP with many interfaces),
but that's it (at most num_cpus-1 I guess). You shouldn't be seeing
4000 frames being queued while you flush.


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: More on ath10k_flush.

[Ben Greear]

On 03/31/2014 10:44 PM, Michal Kazior wrote:
> On 1 April 2014 02:24, Ben Greear <greearb@candelatech.com> wrote:
>> So, I tried adding lots of debug to the ath10k_flush,
>> and I think I might have found the issue.
>>
>> My test case is:  create 32 stations, start TCP on all of them,
>> then reset (effectively ifdown;ifup) the station vifs.
>>
>> I get lots of spews because the ath10k_flush call thinks it is
>> hanging.  But, based on the debug, I think the problem is that
>> the tx logic is continuing to accept packets while the flush
>> is trying to happen.
>>
>> If my debugging stats are correct, the NIC managed to send around 4000
>> frames during the flush attempt, and the ath10k driver processed lots of
>> tx completions as well.  But there are still lots of pkts
>> pending tx and the tx queue was never empty.
>>
>> ath10k: failed to flush transmit queue (skip 0 ar-state 1 pending_tx 1009  pre-pending-tx: 1045): 0
>> ath10k: pre: htc-send-tot: 3423  htt-tx 2545397  tx-compl 2544355  mgt-tx 0  mgt-compl 0
>> ath10k: post: htc-send-tot: 3424  htt-tx 2630981  tx-compl 2630156  mgt-tx 0  mgt-compl 0
>> ath10k: pre: hw-queued: 124395  hw-reaped: 124394
>> ath10k: post: hw-queued: 128418  hw-reaped: 128417
>
> What do these prints reflect exactly? What base upstream ath10k commit
> are you on?
>
>
>> Do we need to somehow shut down all tx logic during the ath10k_flush
>> so that (other?) stations cannot transmit?
>
> This seems strange. drv_flush() is called with all tx queues being
> stopped in mac80211 with IEEE80211_QUEUE_STOP_REASON_FLUSH. I guess
> you can still receive a few frames (due to SMP with many interfaces),
> but that's it (at most num_cpus-1 I guess). You shouldn't be seeing
> 4000 frames being queued while you flush.

Subsequent debugging makes me think the extra packets are not coming
from the upper stack.  Possibly there are ath10k<->firmware management messages
being queued or something like that?

I'm out of office today, will post the debug patch when I get a chance.

Thanks,
Ben

>
>
> Michał
>


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: More on ath10k_flush.

[Michal Kazior]

On 1 April 2014 15:12, Ben Greear <greearb@candelatech.com> wrote:
> On 03/31/2014 10:44 PM, Michal Kazior wrote:
>> On 1 April 2014 02:24, Ben Greear <greearb@candelatech.com> wrote:

[...]

>>> Do we need to somehow shut down all tx logic during the ath10k_flush
>>> so that (other?) stations cannot transmit?
>>
>>
>> This seems strange. drv_flush() is called with all tx queues being
>> stopped in mac80211 with IEEE80211_QUEUE_STOP_REASON_FLUSH. I guess
>> you can still receive a few frames (due to SMP with many interfaces),
>> but that's it (at most num_cpus-1 I guess). You shouldn't be seeing
>> 4000 frames being queued while you flush.
>
>
> Subsequent debugging makes me think the extra packets are not coming
> from the upper stack.  Possibly there are ath10k<->firmware management
> messages
> being queued or something like that?

ath10k_flush() waits only for HTT Tx. It doesn't even wait for
management Tx (it probably should..).

There's a limit to the number of outstanding HTT Tx requests - it is
1424 (+n_cpus - 1). So it is possible to see at most this much frames
being completed while waiting for a flush.


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k