system hang with backports-20150511/20150525

system hang with backports-20150511/20150525

[Marty Faltesek]

Starting with backports-20150511, and continuing with
backports-20150525, we see frequent system hangs. backports-20150424
had no issue.

After the freeze, the console is non-responsive, as well as the
network stack (ssh/ping does not work). Using sysrq, I can see log
messages continuing from ath10k_pci after the freeze, along with some
other threads as well.

mac80211/ath10k/cfg80211 are the only modules in use from backports,
so it seems like a deadlock  could possibly be with mac80211 or
ath10k.

LOCKDEP didn't reveal anything.

Using a 3.2.26 kernel on ARM. AP mode. No encryption.

I've collected ftrace events for sched mac80211 net napi cfg80211
workqueue, which are included in the dmesg you can find here because
of its size:

http://tinyurl.com/dmesg-ftrace

In the logs, the last timestamp that my test script wrote is:

[ 1021.291495] hbeat0352

I've captured  ftrace events before and after 1021.291495.

Thanks,
Marty Faltesek
Google Fiber

Re: system hang with backports-20150511/20150525

[Michal Kazior]

+ath10k list

On 1 June 2015 at 03:37, Marty Faltesek <mfaltesek@google.com> wrote:
> Starting with backports-20150511, and continuing with
> backports-20150525, we see frequent system hangs. backports-20150424
> had no issue.

I don't see such binary releases on
https://backports.wiki.kernel.org/index.php/Main_Page
Hence I don't know what kernel you've backported the drivers from and
I can't compare anything.

Can you provide more details, please?


> After the freeze, the console is non-responsive, as well as the
> network stack (ssh/ping does not work). Using sysrq, I can see log
> messages continuing from ath10k_pci after the freeze, along with some
> other threads as well.

You probably refer to:

[ 1026.951643] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0,
skipped old beacon
[ 1026.951674] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0,
skipped old beacon
[ 1026.951698] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0,
skipped old beacon

What's puzzling to me are these timestamps. SWBA events are generated
by firmware (and sent to host) every beacon interval which is ~100ms
in most cases. In your case however I can see a burst of at least 10
SWBA events within 1ms. Either top(irq) or bottom(tasklet) got stuck
for some time.

It could be useful if you could enable ath10k debugging with
debug_mask=0xffffff3f (this could generate a lot of messages if you're
running traffic through ath10k).


> mac80211/ath10k/cfg80211 are the only modules in use from backports,
> so it seems like a deadlock  could possibly be with mac80211 or
> ath10k.
>
> LOCKDEP didn't reveal anything.

You might want to try tune /proc/sys/kernel/hung_task_timeout_secs
down (e.g. 5 or 10 seconds) and see what happens when you hit the
problem.


> Using a 3.2.26 kernel on ARM. AP mode. No encryption.
>
> I've collected ftrace events for sched mac80211 net napi cfg80211
> workqueue, which are included in the dmesg you can find here because
> of its size:
>
> http://tinyurl.com/dmesg-ftrace
>
> In the logs, the last timestamp that my test script wrote is:
>
> [ 1021.291495] hbeat0352
>
> I've captured  ftrace events before and after 1021.291495.

Your dmesg looks really messy and I'm worried if SWBA events really
came in a burst or not.


Michał

Re: system hang with backports-20150511/20150525

[Kalle Valo]

Michal Kazior <michal.kazior@tieto.com> writes:

> +ath10k list
>
> On 1 June 2015 at 03:37, Marty Faltesek <mfaltesek@google.com> wrote:
>> Starting with backports-20150511, and continuing with
>> backports-20150525, we see frequent system hangs. backports-20150424
>> had no issue.
>
> I don't see such binary releases on
> https://backports.wiki.kernel.org/index.php/Main_Page
> Hence I don't know what kernel you've backported the drivers from and
> I can't compare anything.
>
> Can you provide more details, please?

I suspect it's from here:

https://www.kernel.org/pub/linux/kernel/projects/backports/2015/05/25/

The backports project pages are a bit confusing and that location is
hard to find.

-- 
Kalle Valo

Re: system hang with backports-20150511/20150525

[Michal Kazior]

On 1 June 2015 at 09:13, Kalle Valo <kvalo@qca.qualcomm.com> wrote:
> Michal Kazior <michal.kazior@tieto.com> writes:
>
>> +ath10k list
>>
>> On 1 June 2015 at 03:37, Marty Faltesek <mfaltesek@google.com> wrote:
>>> Starting with backports-20150511, and continuing with
>>> backports-20150525, we see frequent system hangs. backports-20150424
>>> had no issue.
>>
>> I don't see such binary releases on
>> https://backports.wiki.kernel.org/index.php/Main_Page
>> Hence I don't know what kernel you've backported the drivers from and
>> I can't compare anything.
>>
>> Can you provide more details, please?
>
> I suspect it's from here:
>
> https://www.kernel.org/pub/linux/kernel/projects/backports/2015/05/25/
>
> The backports project pages are a bit confusing and that location is
> hard to find.

Oh, thanks!

Hmm.. There was a ton of changes between 20150424 and 20150511. For
one, ath10k started to use chanctx API and FAST_XMIT. But it's not a
given these two are to blame.

The latter can be easily disabled by removing
IEEE80211_HW_SUPPORT_FAST_XMIT from ar->hw->flags in ath10k's mac.c.
The former.. not so easy. I'd be awesome if you could do a git bisect.
The commit ids are a3da0fb6(good) f17107c(bad) (you need linux-next
git repo including its tags for these ids to be resolvable).


Michał

Re: system hang with backports-20150511/20150525

[Marty Faltesek]

I disabled IEEE80211_HW_SUPPORT_FAST_XMIT before, and still saw the
hang. I repeated today to confirm.

I added the extra ath10k debug flags you requested, and it causes a
system reset without any messages, very soon after the last hbeat
timestamp. I've uploaded  log "crash.6.1.15.13.46" to
http://tinyurl.com/dmesg-ftrace.

Any advice on how to bisect when using backports?

On Mon, Jun 1, 2015 at 4:27 AM, Michal Kazior <michal.kazior@tieto.com> wrote:
> On 1 June 2015 at 09:13, Kalle Valo <kvalo@qca.qualcomm.com> wrote:
>> Michal Kazior <michal.kazior@tieto.com> writes:
>>
>>> +ath10k list
>>>
>>> On 1 June 2015 at 03:37, Marty Faltesek <mfaltesek@google.com> wrote:
>>>> Starting with backports-20150511, and continuing with
>>>> backports-20150525, we see frequent system hangs. backports-20150424
>>>> had no issue.
>>>
>>> I don't see such binary releases on
>>> https://backports.wiki.kernel.org/index.php/Main_Page
>>> Hence I don't know what kernel you've backported the drivers from and
>>> I can't compare anything.
>>>
>>> Can you provide more details, please?
>>
>> I suspect it's from here:
>>
>> https://www.kernel.org/pub/linux/kernel/projects/backports/2015/05/25/
>>
>> The backports project pages are a bit confusing and that location is
>> hard to find.
>
> Oh, thanks!
>
> Hmm.. There was a ton of changes between 20150424 and 20150511. For
> one, ath10k started to use chanctx API and FAST_XMIT. But it's not a
> given these two are to blame.
>
> The latter can be easily disabled by removing
> IEEE80211_HW_SUPPORT_FAST_XMIT from ar->hw->flags in ath10k's mac.c.
> The former.. not so easy. I'd be awesome if you could do a git bisect.
> The commit ids are a3da0fb6(good) f17107c(bad) (you need linux-next
> git repo including its tags for these ids to be resolvable).
>
>
> Michał

Re: system hang with backports-20150511/20150525

[Michal Kazior]

On 1 June 2015 at 21:42, Marty Faltesek <mfaltesek@google.com> wrote:
> I disabled IEEE80211_HW_SUPPORT_FAST_XMIT before, and still saw the
> hang. I repeated today to confirm.

Thanks for checking.


> I added the extra ath10k debug flags you requested, and it causes a
> system reset without any messages, very soon after the last hbeat
> timestamp. I've uploaded  log "crash.6.1.15.13.46" to
> http://tinyurl.com/dmesg-ftrace.

I guess serial console gave up which isn't really surprising :( Thanks
for checking anyway.


> Any advice on how to bisect when using backports?

Sure. Generally you'll need to do the bisect on your linux-next tree
which you use to generate backports:

 git clone git://git.kernel.org/pub/scm/linux/kernel/git/backports/backports.git
 git clone git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
 cd linux-next
 git bisect start
 git bisect good a3da0fb6
 git bisect bad f17107c
 # repeat:
 cd ../backports
 ./gentree.py --clean --git-revision HEAD ../linux-next ../backports-output/
 cd ../backports-output
 # configure, make, test
 cd ../linux-next
 git bisect <good|bad> # "good" if problem doesn't reproduce, "bad" if it does
 # goto repeat


Michał




>
> On Mon, Jun 1, 2015 at 4:27 AM, Michal Kazior <michal.kazior@tieto.com> wrote:
>> On 1 June 2015 at 09:13, Kalle Valo <kvalo@qca.qualcomm.com> wrote:
>>> Michal Kazior <michal.kazior@tieto.com> writes:
>>>
>>>> +ath10k list
>>>>
>>>> On 1 June 2015 at 03:37, Marty Faltesek <mfaltesek@google.com> wrote:
>>>>> Starting with backports-20150511, and continuing with
>>>>> backports-20150525, we see frequent system hangs. backports-20150424
>>>>> had no issue.
>>>>
>>>> I don't see such binary releases on
>>>> https://backports.wiki.kernel.org/index.php/Main_Page
>>>> Hence I don't know what kernel you've backported the drivers from and
>>>> I can't compare anything.
>>>>
>>>> Can you provide more details, please?
>>>
>>> I suspect it's from here:
>>>
>>> https://www.kernel.org/pub/linux/kernel/projects/backports/2015/05/25/
>>>
>>> The backports project pages are a bit confusing and that location is
>>> hard to find.
>>
>> Oh, thanks!
>>
>> Hmm.. There was a ton of changes between 20150424 and 20150511. For
>> one, ath10k started to use chanctx API and FAST_XMIT. But it's not a
>> given these two are to blame.
>>
>> The latter can be easily disabled by removing
>> IEEE80211_HW_SUPPORT_FAST_XMIT from ar->hw->flags in ath10k's mac.c.
>> The former.. not so easy. I'd be awesome if you could do a git bisect.
>> The commit ids are a3da0fb6(good) f17107c(bad) (you need linux-next
>> git repo including its tags for these ids to be resolvable).
>>
>>
>> Michał