[ath9k-devel] Linux bug 14267 "No probe response from AP after 500ms, disconnecting."

[ath9k-devel] Linux bug 14267 "No probe response from AP after 500ms, disconnecting."

[Peter Stuge]

Is there some more information that I can gather in order to resolve
this issue? Note that disabling power management is not a sufficient
workaround for me, but it makes the error much less frequent.

http://bugzilla.kernel.org/show_bug.cgi?id=14267#c55

#56 has a full debug log of the failure attached. I realize that
stuff going on _before_ the failure is more interesting but to my
untrained eye it looked quite like when things were working ok.

How often should there be a beacon? Ie. what's the beacon loss timer?


Thanks

//Peter

[ath9k-devel] Linux bug 14267 "No probe response from AP after 500ms, disconnecting."

[Luis R. Rodriguez]

On Wed, Dec 16, 2009 at 09:23:56AM -0800, Peter Stuge wrote:
> Is there some more information that I can gather in order to resolve
> this issue? Note that disabling power management is not a sufficient
> workaround for me, but it makes the error much less frequent.
> 
> http://bugzilla.kernel.org/show_bug.cgi?id=14267#c55
> 
> #56 has a full debug log of the failure attached. I realize that
> stuff going on _before_ the failure is more interesting but to my
> untrained eye it looked quite like when things were working ok.
> 
> How often should there be a beacon? Ie. what's the beacon loss timer?

That bug report is about issues (so far AR5416 only reported with the PS issue)
about disconnections with your AP if PS is left enabled by default which will
trigger dynamic PS to be enabled by by default, its all about 2.6.32. The fix
on 2.6.32 which should help AR5416 (so far concrete device with issues) is
to disable PS by default.

You can also disable PS manually if your kernel has it enabled:

iwconfig wlan0 power off

Your post on that bug report is very different, you say you have merged
wireless-testing into your kernel (why??), you also posted a log but
it doesn't contain the initailization part of ath9k so your card device
cannot be seen. What card is this with?

Please keep stable kernel bugs apart from wireless-testing bugs. Wireless-testing
is bleedinge edge and as such if you have issues you should use ath9k-devel or
the linux-wireless list (like this) and not post on bug reports for stable bugs.

  Luis

[ath9k-devel] Linux bug 14267 "No probe response from AP after 500ms, disconnecting."

[Peter Stuge]

Hi,

Luis R. Rodriguez wrote:
> > http://bugzilla.kernel.org/show_bug.cgi?id=14267#c55
> 
> That bug report is about issues (so far AR5416 only reported with
> the PS issue) about disconnections with your AP if PS is left
> enabled by default which will trigger dynamic PS to be enabled by
> by default, its all about 2.6.32.

I observe the exact same problem with my two kernels. One was
2.6.32-rc6 and now with wl-testing.


> The fix on 2.6.32 which should help AR5416 (so far concrete device
> with issues) is to disable PS by default.
> 
> You can also disable PS manually if your kernel has it enabled:
> 
> iwconfig wlan0 power off

This worked well for me during brief testing with the -rc6 kernel. I
then switched to wl-testing to be up to date. The fix to disable by
default is included in my kernel, and PS is off. I still observe the
problem, although it is much less frequent. Ie. the fix does not
solve the problem for me. I'm not saying that it neccessarily is
exactly the same problem, but the manifestation is; I am
disconnected. :)


> Your post on that bug report is very different, you say you have
> merged wireless-testing into your kernel (why??),

I was running drm-intel.git to get some bleeding edge KMS patches. I
had that git so it was easy to pull in wl-testing, and I thought it
would make debugging and working with the experts easier. (And to
rule out that it had been fixed, of course. :)


> you also posted a log but it doesn't contain the initailization
> part of ath9k so your card device cannot be seen. What card is this
> with?

I did not find this bug at first, and so I started posting
information in bug 14664, my init information and more background is
there: http://bugzilla.kernel.org/show_bug.cgi?id=14664#c1
After a while I found 14267 and realized that 14664 was a dup.


> Please keep stable kernel bugs apart from wireless-testing bugs.

It looks like it is actually one and the same bug.
It's not fixed for me by the committed workaround. It is of course
possible that it is in fact a different bug producing identical
symptoms, even more so since I don't have exactly the same hardware.


> Wireless-testing is bleedinge edge and as such if you have issues
> you should use ath9k-devel or the linux-wireless list (like this)
> and not post on bug reports for stable bugs.

Nod. Let's go. How can I help further? Since the attachment log I
posted I had another failure here, now I have a debug log also with
some more activity before the failure. Should I attach it?


Thanks!

//Peter

[ath9k-devel] Linux bug 14267 "No probe response from AP after 500ms, disconnecting."

[Luis R. Rodriguez]

On Wed, Dec 16, 2009 at 02:21:57PM -0800, Peter Stuge wrote:
> Hi,
> 
> Luis R. Rodriguez wrote:
> > > http://bugzilla.kernel.org/show_bug.cgi?id=14267#c55
> >
> > That bug report is about issues (so far AR5416 only reported with
> > the PS issue) about disconnections with your AP if PS is left
> > enabled by default which will trigger dynamic PS to be enabled by
> > by default, its all about 2.6.32.
> 
> I observe the exact same problem with my two kernels. One was
> 2.6.32-rc6 and now with wl-testing.

2.6.32-rc6 is ancient :)

> > The fix on 2.6.32 which should help AR5416 (so far concrete device
> > with issues) is to disable PS by default.
> >
> > You can also disable PS manually if your kernel has it enabled:
> >
> > iwconfig wlan0 power off
> 
> This worked well for me during brief testing with the -rc6 kernel. I
> then switched to wl-testing to be up to date.

That's indeed a good move to test.

> The fix to disable by
> default is included in my kernel, and PS is off. I still observe the
> problem, although it is much less frequent. Ie. the fix does not
> solve the problem for me. I'm not saying that it neccessarily is
> exactly the same problem, but the manifestation is; I am
> disconnected. :)

Well so depending on the device you have you may need some patches which
may or may not have been present on wireless-testing. Some recent fixes
for ath9k on 2.6.32 and wireless-testing are important, Sujith also posted
some recent fixes. They don't all pertain to power save but some do. On your
bug report you did not indicate if you tested 2.6.32 with the latest patches
I had suggested to Justin.

> > Your post on that bug report is very different, you say you have
> > merged wireless-testing into your kernel (why??),
> 
> I was running drm-intel.git to get some bleeding edge KMS patches. I
> had that git so it was easy to pull in wl-testing, and I thought it
> would make debugging and working with the experts easier. (And to
> rule out that it had been fixed, of course. :)

It does help, absolutely, its what we prefer for debugging.

> > you also posted a log but it doesn't contain the initailization
> > part of ath9k so your card device cannot be seen. What card is this
> > with?
> 
> I did not find this bug at first, and so I started posting
> information in bug 14664, my init information and more background is
> there: http://bugzilla.kernel.org/show_bug.cgi?id=14664#c1
> After a while I found 14267 and realized that 14664 was a dup.

OK you also have an AR5416, which is the first 11n chipset generation
for Atheros, the bug report was originally for AR9280. Justin also
has an AR5416.

Lets make sure to keep these separate.

> > Please keep stable kernel bugs apart from wireless-testing bugs.
> 
> It looks like it is actually one and the same bug.

No, its one whole hardware generation part, the symptoms are
certainly the same though but I have not gotten indication from
AR2980 users that the issues are present there -- and that's
frankly what we have mainly been testing with.

> It's not fixed for me by the committed workaround. It is of course
> possible that it is in fact a different bug producing identical
> symptoms, even more so since I don't have exactly the same hardware.

Right.

> > Wireless-testing is bleedinge edge and as such if you have issues
> > you should use ath9k-devel or the linux-wireless list (like this)
> > and not post on bug reports for stable bugs.
> 
> Nod. Let's go. How can I help further? Since the attachment log I
> posted I had another failure here, now I have a debug log also with
> some more activity before the failure. Should I attach it?

Try sucking in Sujith's recent posted patches, although none of those
are PS fixes, and you can also follow the instructions I gave Justin
to help debug things.

  Luis

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Peter Stuge]

(linux-wireless posters, please Cc me since I am only subscribed to
ath9k-devel.)


Hello,

I get the above error (and thus loss of connectivity) using
wireless-testing/master as of Dec 13.

phy0: Atheros AR5416 MAC/BB Rev:2 AR5133 RF Rev:81: mem=0xf8320000, irq=21

I've previously posted various other info starting at
http://bugzilla.kernel.org/show_bug.cgi?id=14664#c1
and
http://bugzilla.kernel.org/show_bug.cgi?id=14267#c53

I recently installed this card in my laptop. I had drm-intel.git at
2.6.32-rc6 at that time and after seeing the error there I merged
wireless-testing/master since I thought that was the most recent
ath9k source. I'm happy to switch to something else if that helps,
just tell me where to get it.

I searched for a bug to add my information and find hints. I found
the above bugs which describe this very symptom, but Luis asked me to
move over to mailing lists since I see this also in wireless-testing.

I think I've tried all suggestions in bug 14267, but the issue
remains.

Manually disabling power management for the interface (iwconfig eth1
power off) makes it much more stable but I've still seen the error
twice. The first time after about a day and then again after a few
hours. I've been running with power management off since then, a
couple of days, so far without seeing the problem again.

My attachment in bug 14267 is a log from the first occurence but it
does not have very many messages leading up to the error. I also have
a longer debug log from the second time it happened, with about 5000
lines before the disconnect: http://stuge.se/ath9kdisconn.txt


I have applied these 6 patches posted by Sujith this week:

ath9k: Fix bug in assigning sequence number
ath9k: Clarify Interrupt mitigation
ath9k: Stop ANI when doing a reset
ath9k: Remove ANI lock
ath9k: Fix TX poll routine
ath9k: Fix TX queue draining

I then enabled power management and was disconnected after the
interface had been up for no more than a few minutes. I am now
running with PM off again, so that I can use the interface. :)


Luis R. Rodriguez wrote:
> > > The fix on 2.6.32 which should help AR5416 (so far concrete
> > > device with issues) is to disable PS by default.
..
> > This worked well for me during brief testing with the -rc6 kernel. I
> > then switched to wl-testing to be up to date.
> 
> That's indeed a good move to test.

To clarify; I only tested with power management off on -rc6 for a few
minutes, and then I switched to the wireless-testing/master kernel
that I am running now.


> > The fix to disable by default is included in my kernel, and PS is
> > off. I still observe the problem,
> 
> Well so depending on the device you have you may need some patches
> which may or may not have been present on wireless-testing.

What exactly does "on wireless-testing" mean? Are they in the
ath9k-devel archive or the linux-wireless archive? I would prefer to
fetch from a git, but email works too. Are patches committed to a
branch on wireless-testing.git? Or is there an ath9k.git?


> Some recent fixes for ath9k on 2.6.32 and wireless-testing are
> important,

As I wrote in one bug comment; I was running a wireless-testing
kernel per commit c770b16cd572bd434f90794be03ae20f5974e6e9 from Dec
13, and I saw the issue twice also with power management disabled.

It seems to me (of course I don't know the internals though) that
power management is not the single factor in this issue.


> Sujith also posted some recent fixes. They don't all pertain to
> power save but some do.

I applied the above 6 patches from Sujith. It's difficult to know if
I got the ones you mean without a more specific description. :)

The patches posted by you to linux-wireless@ on 2009-12-16 are
included in my wireless-testing/master kernel already:

ath9k: Fix TX hang poll routine
commit 73803a9b535b76f36afba4881af22fe7b84f49c0
CommitDate: Fri Dec 4 16:12:31 2009 -0500

ath9k: fix processing of TX PS null data frames
commit 87340fcfc6ada956132878a72efdc75431a684b3
CommitDate: Fri Dec 4 16:15:41 2009 -0500

ath9k: Fix maximum tx fifo settings for single stream devices
commit 499e75e2c226aa49ba1e801462a0bee02756984a
CommitDate: Fri Dec 4 16:15:42 2009 -0500

ath9k: fix tx status reporting
commit e8c6342d989e241513baeba4b05a04b6b1f3bc8b
CommitDate: Mon Dec 7 17:05:40 2009 -0500

mac80211: Fix dynamic power save for scanning.
commit fba4a86f5b2652fac0c508968a3a4b4e03d6b661
CommitDate: Mon Dec 7 17:05:35 2009 -0500


> On your bug report you did not indicate if you tested 2.6.32 with
> the latest patches I had suggested to Justin.

I did not test them on top of the .32-rc6 kernel but again they're in
the current kernel and I still reproduce the issue quickly with power
management on, and have seen it twice with PM off.

So far I have not seen the issue with PM off and 6 above patches
applied, that is what I am running with right now and I'll let you
know what happens. (With PM on the issue is still frequent.)


> OK you also have an AR5416, which is the first 11n chipset
> generation for Atheros, the bug report was originally for AR9280.
> Justin also has an AR5416.
> 
> Lets make sure to keep these separate.

The failure mode is the same, AR9280 is PCIe, AR9220 may be too
uncommon to have any data points and I see the issue only very
infrequently with power management off on AR5416. I think all these
factors can support that it is a single issue. But they are in no way
conclusive! Hopefully it can be fixed somewhere so that there will be
more data.


> > Nod. Let's go. How can I help further?
> 
> Try sucking in Sujith's recent posted patches, although none of
> those are PS fixes,

Did I get the right ones?


> and you can also follow the instructions I gave Justin to help
> debug things.

I tried to do that already. The debug log I attached didn't have too
much info leading up to the disconnect though. Feel free to get the
longer one.

Is there anything else can I do?


//Peter

No probe response from AP after 500ms, disconnecting.

[Peter Stuge]

(linux-wireless posters, please Cc me since I am only subscribed to
ath9k-devel.)


Hello,

I get the above error (and thus loss of connectivity) using
wireless-testing/master as of Dec 13.

phy0: Atheros AR5416 MAC/BB Rev:2 AR5133 RF Rev:81: mem=0xf8320000, irq=21

I've previously posted various other info starting at
http://bugzilla.kernel.org/show_bug.cgi?id=14664#c1
and
http://bugzilla.kernel.org/show_bug.cgi?id=14267#c53

I recently installed this card in my laptop. I had drm-intel.git at
2.6.32-rc6 at that time and after seeing the error there I merged
wireless-testing/master since I thought that was the most recent
ath9k source. I'm happy to switch to something else if that helps,
just tell me where to get it.

I searched for a bug to add my information and find hints. I found
the above bugs which describe this very symptom, but Luis asked me to
move over to mailing lists since I see this also in wireless-testing.

I think I've tried all suggestions in bug 14267, but the issue
remains.

Manually disabling power management for the interface (iwconfig eth1
power off) makes it much more stable but I've still seen the error
twice. The first time after about a day and then again after a few
hours. I've been running with power management off since then, a
couple of days, so far without seeing the problem again.

My attachment in bug 14267 is a log from the first occurence but it
does not have very many messages leading up to the error. I also have
a longer debug log from the second time it happened, with about 5000
lines before the disconnect: http://stuge.se/ath9kdisconn.txt


I have applied these 6 patches posted by Sujith this week:

ath9k: Fix bug in assigning sequence number
ath9k: Clarify Interrupt mitigation
ath9k: Stop ANI when doing a reset
ath9k: Remove ANI lock
ath9k: Fix TX poll routine
ath9k: Fix TX queue draining

I then enabled power management and was disconnected after the
interface had been up for no more than a few minutes. I am now
running with PM off again, so that I can use the interface. :)


Luis R. Rodriguez wrote:
> > > The fix on 2.6.32 which should help AR5416 (so far concrete
> > > device with issues) is to disable PS by default.
..
> > This worked well for me during brief testing with the -rc6 kernel. I
> > then switched to wl-testing to be up to date.
> 
> That's indeed a good move to test.

To clarify; I only tested with power management off on -rc6 for a few
minutes, and then I switched to the wireless-testing/master kernel
that I am running now.


> > The fix to disable by default is included in my kernel, and PS is
> > off. I still observe the problem,
> 
> Well so depending on the device you have you may need some patches
> which may or may not have been present on wireless-testing.

What exactly does "on wireless-testing" mean? Are they in the
ath9k-devel archive or the linux-wireless archive? I would prefer to
fetch from a git, but email works too. Are patches committed to a
branch on wireless-testing.git? Or is there an ath9k.git?


> Some recent fixes for ath9k on 2.6.32 and wireless-testing are
> important,

As I wrote in one bug comment; I was running a wireless-testing
kernel per commit c770b16cd572bd434f90794be03ae20f5974e6e9 from Dec
13, and I saw the issue twice also with power management disabled.

It seems to me (of course I don't know the internals though) that
power management is not the single factor in this issue.


> Sujith also posted some recent fixes. They don't all pertain to
> power save but some do.

I applied the above 6 patches from Sujith. It's difficult to know if
I got the ones you mean without a more specific description. :)

The patches posted by you to linux-wireless@ on 2009-12-16 are
included in my wireless-testing/master kernel already:

ath9k: Fix TX hang poll routine
commit 73803a9b535b76f36afba4881af22fe7b84f49c0
CommitDate: Fri Dec 4 16:12:31 2009 -0500

ath9k: fix processing of TX PS null data frames
commit 87340fcfc6ada956132878a72efdc75431a684b3
CommitDate: Fri Dec 4 16:15:41 2009 -0500

ath9k: Fix maximum tx fifo settings for single stream devices
commit 499e75e2c226aa49ba1e801462a0bee02756984a
CommitDate: Fri Dec 4 16:15:42 2009 -0500

ath9k: fix tx status reporting
commit e8c6342d989e241513baeba4b05a04b6b1f3bc8b
CommitDate: Mon Dec 7 17:05:40 2009 -0500

mac80211: Fix dynamic power save for scanning.
commit fba4a86f5b2652fac0c508968a3a4b4e03d6b661
CommitDate: Mon Dec 7 17:05:35 2009 -0500


> On your bug report you did not indicate if you tested 2.6.32 with
> the latest patches I had suggested to Justin.

I did not test them on top of the .32-rc6 kernel but again they're in
the current kernel and I still reproduce the issue quickly with power
management on, and have seen it twice with PM off.

So far I have not seen the issue with PM off and 6 above patches
applied, that is what I am running with right now and I'll let you
know what happens. (With PM on the issue is still frequent.)


> OK you also have an AR5416, which is the first 11n chipset
> generation for Atheros, the bug report was originally for AR9280.
> Justin also has an AR5416.
> 
> Lets make sure to keep these separate.

The failure mode is the same, AR9280 is PCIe, AR9220 may be too
uncommon to have any data points and I see the issue only very
infrequently with power management off on AR5416. I think all these
factors can support that it is a single issue. But they are in no way
conclusive! Hopefully it can be fixed somewhere so that there will be
more data.


> > Nod. Let's go. How can I help further?
> 
> Try sucking in Sujith's recent posted patches, although none of
> those are PS fixes,

Did I get the right ones?


> and you can also follow the instructions I gave Justin to help
> debug things.

I tried to do that already. The debug log I attached didn't have too
much info leading up to the disconnect though. Feel free to get the
longer one.

Is there anything else can I do?


//Peter

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Luis R. Rodriguez]

On Fri, Dec 18, 2009 at 03:57:08AM -0800, Peter Stuge wrote:
> (linux-wireless posters, please Cc me since I am only subscribed to
> ath9k-devel.)
> 
> 
> Hello,
> 
> I get the above error (and thus loss of connectivity) using
> wireless-testing/master as of Dec 13.
> 
> phy0: Atheros AR5416 MAC/BB Rev:2 AR5133 RF Rev:81: mem=0xf8320000, irq=21
> 
> I've previously posted various other info starting at
> http://bugzilla.kernel.org/show_bug.cgi?id=14664#c1
> and
> http://bugzilla.kernel.org/show_bug.cgi?id=14267#c53
> 
> I recently installed this card in my laptop. I had drm-intel.git at
> 2.6.32-rc6 at that time and after seeing the error there I merged
> wireless-testing/master since I thought that was the most recent
> ath9k source. I'm happy to switch to something else if that helps,
> just tell me where to get it.
> 
> I searched for a bug to add my information and find hints. I found
> the above bugs which describe this very symptom, but Luis asked me to
> move over to mailing lists since I see this also in wireless-testing.
> 
> I think I've tried all suggestions in bug 14267, but the issue
> remains.

This seems like your old comments from an old e-mail, but I guess you
include them since you now include linux-wireless.

> Manually disabling power management for the interface (iwconfig eth1
> power off) makes it much more stable but I've still seen the error
> twice. The first time after about a day and then again after a few
> hours. I've been running with power management off since then, a
> couple of days, so far without seeing the problem again.

Neat.

> My attachment in bug 14267 is a log from the first occurence but it
> does not have very many messages leading up to the error. I also have
> a longer debug log from the second time it happened, with about 5000
> lines before the disconnect: http://stuge.se/ath9kdisconn.txt
> 
> 
> I have applied these 6 patches posted by Sujith this week:
> 
> ath9k: Fix bug in assigning sequence number
> ath9k: Clarify Interrupt mitigation
> ath9k: Stop ANI when doing a reset
> ath9k: Remove ANI lock
> ath9k: Fix TX poll routine
> ath9k: Fix TX queue draining
> 
> I then enabled power management and was disconnected after the
> interface had been up for no more than a few minutes. I am now
> running with PM off again, so that I can use the interface. :)

Thanks for testing.

> Luis R. Rodriguez wrote:
> > > > The fix on 2.6.32 which should help AR5416 (so far concrete
> > > > device with issues) is to disable PS by default.
> ..
> > > This worked well for me during brief testing with the -rc6 kernel. I
> > > then switched to wl-testing to be up to date.
> >
> > That's indeed a good move to test.
> 
> To clarify; I only tested with power management off on -rc6 for a few
> minutes, and then I switched to the wireless-testing/master kernel
> that I am running now.

Understood, thanks for the clarification.

> > > The fix to disable by default is included in my kernel, and PS is
> > > off. I still observe the problem,
> >
> > Well so depending on the device you have you may need some patches
> > which may or may not have been present on wireless-testing.
> 
> What exactly does "on wireless-testing" mean? Are they in the
> ath9k-devel archive or the linux-wireless archive? I would prefer to
> fetch from a git, but email works too. Are patches committed to a
> branch on wireless-testing.git? Or is there an ath9k.git?

Wireless-testing is that git tree you cloned. Bleedinge edge wireless
patches posted to linux-wireless get merged to there by John in
preparation for the next merge window.

> > Some recent fixes for ath9k on 2.6.32 and wireless-testing are
> > important,
> 
> As I wrote in one bug comment; I was running a wireless-testing
> kernel per commit c770b16cd572bd434f90794be03ae20f5974e6e9 from Dec
> 13, and I saw the issue twice also with power management disabled.
> 
> It seems to me (of course I don't know the internals though) that
> power management is not the single factor in this issue.

It should also be noted the issues are affecting AR5416 with PS enabled
by default, AR9280 and friends seem to be OK with the latest patches.

> > Sujith also posted some recent fixes. They don't all pertain to
> > power save but some do.
> 
> I applied the above 6 patches from Sujith. It's difficult to know if
> I got the ones you mean without a more specific description. :)
> 
> The patches posted by you to linux-wireless@ on 2009-12-16 are
> included in my wireless-testing/master kernel already:
> 
> ath9k: Fix TX hang poll routine
> commit 73803a9b535b76f36afba4881af22fe7b84f49c0
> CommitDate: Fri Dec 4 16:12:31 2009 -0500
> 
> ath9k: fix processing of TX PS null data frames
> commit 87340fcfc6ada956132878a72efdc75431a684b3
> CommitDate: Fri Dec 4 16:15:41 2009 -0500
> 
> ath9k: Fix maximum tx fifo settings for single stream devices
> commit 499e75e2c226aa49ba1e801462a0bee02756984a
> CommitDate: Fri Dec 4 16:15:42 2009 -0500
> 
> ath9k: fix tx status reporting
> commit e8c6342d989e241513baeba4b05a04b6b1f3bc8b
> CommitDate: Mon Dec 7 17:05:40 2009 -0500
> 
> mac80211: Fix dynamic power save for scanning.
> commit fba4a86f5b2652fac0c508968a3a4b4e03d6b661
> CommitDate: Mon Dec 7 17:05:35 2009 -0500

So these would apply to stable, but wireless-testing should have had
these already. The patche I was referring to from Sujith were posted
only to linux-wireless, these are now merged on wireless-testing.

> > On your bug report you did not indicate if you tested 2.6.32 with
> > the latest patches I had suggested to Justin.
> 
> I did not test them on top of the .32-rc6 kernel but again they're in
> the current kernel and I still reproduce the issue quickly with power
> management on, and have seen it twice with PM off.
> 
> So far I have not seen the issue with PM off and 6 above patches
> applied, that is what I am running with right now and I'll let you
> know what happens. (With PM on the issue is still frequent.)

OK so far this narrows down to a specific AR5416 issue with PM on
by default only.

> > OK you also have an AR5416, which is the first 11n chipset
> > generation for Atheros, the bug report was originally for AR9280.
> > Justin also has an AR5416.
> >
> > Lets make sure to keep these separate.
> 
> The failure mode is the same, AR9280 is PCIe, AR9220 may be too
> uncommon to have any data points and I see the issue only very
> infrequently with power management off on AR5416. I think all these
> factors can support that it is a single issue. But they are in no way
> conclusive! Hopefully it can be fixed somewhere so that there will be
> more data.

They are different hardware, newer hardware families (>= AR9280) are
single chip and quite a few changes went into them, so thinking of them
as equal would be wrong. They certain share a lot but for example the
radios are completely different. Now sure, the issue can be the similar
but it doesn't mean that it is not fixed for some chipsets, ignoring that
would be unfair for those users of the newer harware families.

> > > Nod. Let's go. How can I help further?
> >
> > Try sucking in Sujith's recent posted patches, although none of
> > those are PS fixes,
> 
> Did I get the right ones?

Those are indeed needed but Sujith posted some new patch fixes but
not related to PS. Some of these fixes are to be merged in the next
next 2.6.32.y so might as well go ahead and apply then if using stable
or even wireless-testing. So no, you mised them. Here they are:

http://marc.info/?l=linux-wireless&r=3&b=200912&w=2

Patchwork has them in git am'able form:

http://patchwork.kernel.org/project/linux-wireless/list/

> > and you can also follow the instructions I gave Justin to help
> > debug things.
> 
> I tried to do that already. The debug log I attached didn't have too
> much info leading up to the disconnect though. Feel free to get the
> longer one.
> 
> Is there anything else can I do?

Try with the above, although I doubt they will help AR5416.

  Luis

Re: [ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Luis R. Rodriguez]

On Fri, Dec 18, 2009 at 03:57:08AM -0800, Peter Stuge wrote:
> (linux-wireless posters, please Cc me since I am only subscribed to
> ath9k-devel.)
> 
> 
> Hello,
> 
> I get the above error (and thus loss of connectivity) using
> wireless-testing/master as of Dec 13.
> 
> phy0: Atheros AR5416 MAC/BB Rev:2 AR5133 RF Rev:81: mem=0xf8320000, irq=21
> 
> I've previously posted various other info starting at
> http://bugzilla.kernel.org/show_bug.cgi?id=14664#c1
> and
> http://bugzilla.kernel.org/show_bug.cgi?id=14267#c53
> 
> I recently installed this card in my laptop. I had drm-intel.git at
> 2.6.32-rc6 at that time and after seeing the error there I merged
> wireless-testing/master since I thought that was the most recent
> ath9k source. I'm happy to switch to something else if that helps,
> just tell me where to get it.
> 
> I searched for a bug to add my information and find hints. I found
> the above bugs which describe this very symptom, but Luis asked me to
> move over to mailing lists since I see this also in wireless-testing.
> 
> I think I've tried all suggestions in bug 14267, but the issue
> remains.

This seems like your old comments from an old e-mail, but I guess you
include them since you now include linux-wireless.

> Manually disabling power management for the interface (iwconfig eth1
> power off) makes it much more stable but I've still seen the error
> twice. The first time after about a day and then again after a few
> hours. I've been running with power management off since then, a
> couple of days, so far without seeing the problem again.

Neat.

> My attachment in bug 14267 is a log from the first occurence but it
> does not have very many messages leading up to the error. I also have
> a longer debug log from the second time it happened, with about 5000
> lines before the disconnect: http://stuge.se/ath9kdisconn.txt
> 
> 
> I have applied these 6 patches posted by Sujith this week:
> 
> ath9k: Fix bug in assigning sequence number
> ath9k: Clarify Interrupt mitigation
> ath9k: Stop ANI when doing a reset
> ath9k: Remove ANI lock
> ath9k: Fix TX poll routine
> ath9k: Fix TX queue draining
> 
> I then enabled power management and was disconnected after the
> interface had been up for no more than a few minutes. I am now
> running with PM off again, so that I can use the interface. :)

Thanks for testing.

> Luis R. Rodriguez wrote:
> > > > The fix on 2.6.32 which should help AR5416 (so far concrete
> > > > device with issues) is to disable PS by default.
> ..
> > > This worked well for me during brief testing with the -rc6 kernel. I
> > > then switched to wl-testing to be up to date.
> >
> > That's indeed a good move to test.
> 
> To clarify; I only tested with power management off on -rc6 for a few
> minutes, and then I switched to the wireless-testing/master kernel
> that I am running now.

Understood, thanks for the clarification.

> > > The fix to disable by default is included in my kernel, and PS is
> > > off. I still observe the problem,
> >
> > Well so depending on the device you have you may need some patches
> > which may or may not have been present on wireless-testing.
> 
> What exactly does "on wireless-testing" mean? Are they in the
> ath9k-devel archive or the linux-wireless archive? I would prefer to
> fetch from a git, but email works too. Are patches committed to a
> branch on wireless-testing.git? Or is there an ath9k.git?

Wireless-testing is that git tree you cloned. Bleedinge edge wireless
patches posted to linux-wireless get merged to there by John in
preparation for the next merge window.

> > Some recent fixes for ath9k on 2.6.32 and wireless-testing are
> > important,
> 
> As I wrote in one bug comment; I was running a wireless-testing
> kernel per commit c770b16cd572bd434f90794be03ae20f5974e6e9 from Dec
> 13, and I saw the issue twice also with power management disabled.
> 
> It seems to me (of course I don't know the internals though) that
> power management is not the single factor in this issue.

It should also be noted the issues are affecting AR5416 with PS enabled
by default, AR9280 and friends seem to be OK with the latest patches.

> > Sujith also posted some recent fixes. They don't all pertain to
> > power save but some do.
> 
> I applied the above 6 patches from Sujith. It's difficult to know if
> I got the ones you mean without a more specific description. :)
> 
> The patches posted by you to linux-wireless@ on 2009-12-16 are
> included in my wireless-testing/master kernel already:
> 
> ath9k: Fix TX hang poll routine
> commit 73803a9b535b76f36afba4881af22fe7b84f49c0
> CommitDate: Fri Dec 4 16:12:31 2009 -0500
> 
> ath9k: fix processing of TX PS null data frames
> commit 87340fcfc6ada956132878a72efdc75431a684b3
> CommitDate: Fri Dec 4 16:15:41 2009 -0500
> 
> ath9k: Fix maximum tx fifo settings for single stream devices
> commit 499e75e2c226aa49ba1e801462a0bee02756984a
> CommitDate: Fri Dec 4 16:15:42 2009 -0500
> 
> ath9k: fix tx status reporting
> commit e8c6342d989e241513baeba4b05a04b6b1f3bc8b
> CommitDate: Mon Dec 7 17:05:40 2009 -0500
> 
> mac80211: Fix dynamic power save for scanning.
> commit fba4a86f5b2652fac0c508968a3a4b4e03d6b661
> CommitDate: Mon Dec 7 17:05:35 2009 -0500

So these would apply to stable, but wireless-testing should have had
these already. The patche I was referring to from Sujith were posted
only to linux-wireless, these are now merged on wireless-testing.

> > On your bug report you did not indicate if you tested 2.6.32 with
> > the latest patches I had suggested to Justin.
> 
> I did not test them on top of the .32-rc6 kernel but again they're in
> the current kernel and I still reproduce the issue quickly with power
> management on, and have seen it twice with PM off.
> 
> So far I have not seen the issue with PM off and 6 above patches
> applied, that is what I am running with right now and I'll let you
> know what happens. (With PM on the issue is still frequent.)

OK so far this narrows down to a specific AR5416 issue with PM on
by default only.

> > OK you also have an AR5416, which is the first 11n chipset
> > generation for Atheros, the bug report was originally for AR9280.
> > Justin also has an AR5416.
> >
> > Lets make sure to keep these separate.
> 
> The failure mode is the same, AR9280 is PCIe, AR9220 may be too
> uncommon to have any data points and I see the issue only very
> infrequently with power management off on AR5416. I think all these
> factors can support that it is a single issue. But they are in no way
> conclusive! Hopefully it can be fixed somewhere so that there will be
> more data.

They are different hardware, newer hardware families (>= AR9280) are
single chip and quite a few changes went into them, so thinking of them
as equal would be wrong. They certain share a lot but for example the
radios are completely different. Now sure, the issue can be the similar
but it doesn't mean that it is not fixed for some chipsets, ignoring that
would be unfair for those users of the newer harware families.

> > > Nod. Let's go. How can I help further?
> >
> > Try sucking in Sujith's recent posted patches, although none of
> > those are PS fixes,
> 
> Did I get the right ones?

Those are indeed needed but Sujith posted some new patch fixes but
not related to PS. Some of these fixes are to be merged in the next
next 2.6.32.y so might as well go ahead and apply then if using stable
or even wireless-testing. So no, you mised them. Here they are:

http://marc.info/?l=linux-wireless&r=3&b=200912&w=2

Patchwork has them in git am'able form:

http://patchwork.kernel.org/project/linux-wireless/list/

> > and you can also follow the instructions I gave Justin to help
> > debug things.
> 
> I tried to do that already. The debug log I attached didn't have too
> much info leading up to the disconnect though. Feel free to get the
> longer one.
> 
> Is there anything else can I do?

Try with the above, although I doubt they will help AR5416.

  Luis

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Peter Stuge]

Hi Luis,

Thanks for the reply!


Luis R. Rodriguez wrote:
> This seems like your old comments from an old e-mail, but I guess
> you include them since you now include linux-wireless.

Yeah, I tried to summarize previous and new findings for new readers.


> > Manually disabling power management for the interface (iwconfig eth1
> > power off) makes it much more stable but I've still seen the error
> > twice. The first time after about a day and then again after a few
> > hours. I've been running with power management off since then, a
> > couple of days, so far without seeing the problem again.
> 
> Neat.

Well, I am in no way convinced that the problem is gone just because
I haven't seen it for a few days. I haven't rebooted this machine
since the last issues, only unloaded/reloaded the ath modules after
each patch/compile cycle, that might matter too..


> > I have applied these 6 patches posted by Sujith this week:
> > 
> > ath9k: Fix bug in assigning sequence number
> > ath9k: Clarify Interrupt mitigation
> > ath9k: Stop ANI when doing a reset
> > ath9k: Remove ANI lock
> > ath9k: Fix TX poll routine
> > ath9k: Fix TX queue draining

..
> > I applied the above 6 patches from Sujith. It's difficult to know if
> > I got the ones you mean without a more specific description. :)
> > 
> > The patches posted by you to linux-wireless@ on 2009-12-16 are
> > included in my wireless-testing/master kernel already:
> > 
> > ath9k: Fix TX hang poll routine
..
> > ath9k: fix processing of TX PS null data frames
..
> > ath9k: Fix maximum tx fifo settings for single stream devices
..
> > ath9k: fix tx status reporting
..
> > mac80211: Fix dynamic power save for scanning.
> 
> So these would apply to stable, but wireless-testing should have had
> these already.

Right; "are included in my wireless-testing/master kernel already"..


> > So far I have not seen the issue with PM off and 6 above patches
> > applied, that is what I am running with right now and I'll let you
> > know what happens. (With PM on the issue is still frequent.)
> 
> OK so far this narrows down to a specific AR5416 issue with PM on
> by default only.

I'm not sure about "by default". The kernel I am running has the
workaround committed which disables PM by default. If I manually
enable PM I will quickly see the issue.


> They are different hardware, newer hardware families (>= AR9280)
> are single chip and quite a few changes went into them, so thinking
> of them as equal would be wrong.

Right, no, they are certainly not equal. But parts of them are - or
maybe more importantly, parts of the driver are.


> They certain share a lot but for example the radios are completely
> different.

Yeah - I imagine there were some changes when the radios went onto
the same chip.

I don't know if this problem lies closer to RF or Linux. Can we
narrow it down somehow?


> Now sure, the issue can be the similar but it doesn't mean that it
> is not fixed for some chipsets, ignoring that would be unfair for
> those users of the newer harware families.

Oh yes - I didn't mean that progressive patches should be blocked in
any way, but just to keep an open mind until the issue is completely
solved. :)


> > > Try sucking in Sujith's recent posted patches, although none of
> > > those are PS fixes,
> > 
> > Did I get the right ones?
> 
> Those are indeed needed but Sujith posted some new patch fixes but
> not related to PS. Some of these fixes are to be merged in the next
> next 2.6.32.y so might as well go ahead and apply then if using stable
> or even wireless-testing. So no, you mised them. Here they are:
> 
> http://marc.info/?l=linux-wireless&r=3&b=200912&w=2
> 
> Patchwork has them in git am'able form:
> 
> http://patchwork.kernel.org/project/linux-wireless/list/

The latest patches from Sujith are dated 2009-12-14 and are the ANI,
TX queue, TX hang, etc patches that I listed above saying that I had
applied them. They are in my running driver.


> > > and you can also follow the instructions I gave Justin to help
> > > debug things.
> > 
> > I tried to do that already. The debug log I attached didn't have too
> > much info leading up to the disconnect though. Feel free to get the
> > longer one.
> > 
> > Is there anything else can I do?
> 
> Try with the above, although I doubt they will help AR5416.

So what could give us more information? If the debug output is not
enough I'm happy to sprinkle printks over the driver in strategic
places, but I need some hints on where to do it.

What is the general operation of the driver? (I have some experience
with writing Linux drivers so feel free to get technical.) RX
descriptors and DMA? Is beacon reception special in any way from
reception of other packets? Would it be useful to try monitor mode
with PM enabled?

I am continuously using low TX and moderate RX bandwidth (internet
radio over a TCP VPN) - would it be helpful to load the card with
exclusively unidirectional transfers such as UDP, to see if the
problem becomes more or less frequent?

Although the issue can be seen as missing beacons maybe it is a
general problem with RX that is only visible in the log when the time
comes to expect a beacon?


Where to look further?


//Peter

Re: [ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Peter Stuge]

Hi Luis,

Thanks for the reply!


Luis R. Rodriguez wrote:
> This seems like your old comments from an old e-mail, but I guess
> you include them since you now include linux-wireless.

Yeah, I tried to summarize previous and new findings for new readers.


> > Manually disabling power management for the interface (iwconfig eth1
> > power off) makes it much more stable but I've still seen the error
> > twice. The first time after about a day and then again after a few
> > hours. I've been running with power management off since then, a
> > couple of days, so far without seeing the problem again.
> 
> Neat.

Well, I am in no way convinced that the problem is gone just because
I haven't seen it for a few days. I haven't rebooted this machine
since the last issues, only unloaded/reloaded the ath modules after
each patch/compile cycle, that might matter too..


> > I have applied these 6 patches posted by Sujith this week:
> > 
> > ath9k: Fix bug in assigning sequence number
> > ath9k: Clarify Interrupt mitigation
> > ath9k: Stop ANI when doing a reset
> > ath9k: Remove ANI lock
> > ath9k: Fix TX poll routine
> > ath9k: Fix TX queue draining

..
> > I applied the above 6 patches from Sujith. It's difficult to know if
> > I got the ones you mean without a more specific description. :)
> > 
> > The patches posted by you to linux-wireless@ on 2009-12-16 are
> > included in my wireless-testing/master kernel already:
> > 
> > ath9k: Fix TX hang poll routine
..
> > ath9k: fix processing of TX PS null data frames
..
> > ath9k: Fix maximum tx fifo settings for single stream devices
..
> > ath9k: fix tx status reporting
..
> > mac80211: Fix dynamic power save for scanning.
> 
> So these would apply to stable, but wireless-testing should have had
> these already.

Right; "are included in my wireless-testing/master kernel already"..


> > So far I have not seen the issue with PM off and 6 above patches
> > applied, that is what I am running with right now and I'll let you
> > know what happens. (With PM on the issue is still frequent.)
> 
> OK so far this narrows down to a specific AR5416 issue with PM on
> by default only.

I'm not sure about "by default". The kernel I am running has the
workaround committed which disables PM by default. If I manually
enable PM I will quickly see the issue.


> They are different hardware, newer hardware families (>= AR9280)
> are single chip and quite a few changes went into them, so thinking
> of them as equal would be wrong.

Right, no, they are certainly not equal. But parts of them are - or
maybe more importantly, parts of the driver are.


> They certain share a lot but for example the radios are completely
> different.

Yeah - I imagine there were some changes when the radios went onto
the same chip.

I don't know if this problem lies closer to RF or Linux. Can we
narrow it down somehow?


> Now sure, the issue can be the similar but it doesn't mean that it
> is not fixed for some chipsets, ignoring that would be unfair for
> those users of the newer harware families.

Oh yes - I didn't mean that progressive patches should be blocked in
any way, but just to keep an open mind until the issue is completely
solved. :)


> > > Try sucking in Sujith's recent posted patches, although none of
> > > those are PS fixes,
> > 
> > Did I get the right ones?
> 
> Those are indeed needed but Sujith posted some new patch fixes but
> not related to PS. Some of these fixes are to be merged in the next
> next 2.6.32.y so might as well go ahead and apply then if using stable
> or even wireless-testing. So no, you mised them. Here they are:
> 
> http://marc.info/?l=linux-wireless&r=3&b=200912&w=2
> 
> Patchwork has them in git am'able form:
> 
> http://patchwork.kernel.org/project/linux-wireless/list/

The latest patches from Sujith are dated 2009-12-14 and are the ANI,
TX queue, TX hang, etc patches that I listed above saying that I had
applied them. They are in my running driver.


> > > and you can also follow the instructions I gave Justin to help
> > > debug things.
> > 
> > I tried to do that already. The debug log I attached didn't have too
> > much info leading up to the disconnect though. Feel free to get the
> > longer one.
> > 
> > Is there anything else can I do?
> 
> Try with the above, although I doubt they will help AR5416.

So what could give us more information? If the debug output is not
enough I'm happy to sprinkle printks over the driver in strategic
places, but I need some hints on where to do it.

What is the general operation of the driver? (I have some experience
with writing Linux drivers so feel free to get technical.) RX
descriptors and DMA? Is beacon reception special in any way from
reception of other packets? Would it be useful to try monitor mode
with PM enabled?

I am continuously using low TX and moderate RX bandwidth (internet
radio over a TCP VPN) - would it be helpful to load the card with
exclusively unidirectional transfers such as UDP, to see if the
problem becomes more or less frequent?

Although the issue can be seen as missing beacons maybe it is a
general problem with RX that is only visible in the log when the time
comes to expect a beacon?


Where to look further?


//Peter

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Vivek Natarajan]

On Sat, Dec 19, 2009 at 1:37 AM, Peter Stuge <peter@stuge.se> wrote:

> So what could give us more information? If the debug output is not
> enough I'm happy to sprinkle printks over the driver in strategic
> places, but I need some hints on where to do it.
>
> What is the general operation of the driver? (I have some experience
> with writing Linux drivers so feel free to get technical.) RX
> descriptors and DMA? Is beacon reception special in any way from
> reception of other packets? Would it be useful to try monitor mode
> with PM enabled?

If the issue is specific to AR5416 and power save, then the TIM_TIMER
interrupt might be causing the trouble. You can check whether any
ATH9K_INT_TIM_TIMER interrupt is received in the ath_isr when the
disconnection happens.
This is the hw timer used by the chip to wakeup for  every beacon listen
interval.

Vivek.

Re: [ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Vivek Natarajan]

On Sat, Dec 19, 2009 at 1:37 AM, Peter Stuge <peter@stuge.se> wrote:

> So what could give us more information? If the debug output is not
> enough I'm happy to sprinkle printks over the driver in strategic
> places, but I need some hints on where to do it.
>
> What is the general operation of the driver? (I have some experience
> with writing Linux drivers so feel free to get technical.) RX
> descriptors and DMA? Is beacon reception special in any way from
> reception of other packets? Would it be useful to try monitor mode
> with PM enabled?

If the issue is specific to AR5416 and power save, then the TIM_TIMER
interrupt might be causing the trouble. You can check whether any
ATH9K_INT_TIM_TIMER interrupt is received in the ath_isr when the
disconnection happens.
This is the hw timer used by the chip to wakeup for  every beacon listen
interval.

Vivek.

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Peter Stuge]

Vivek Natarajan wrote:
> > So what could give us more information?
> 
> If the issue is specific to AR5416 and power save,

I don't think it is. Tonight I saw the problem a third time using
wireless-testing with power management off for the interface. This
time I also had Sujith's 6 patches from 2009-12-14 applied.

5k lines of debug=0xffffffff log is available at:
http://stuge.se/ath9kdisconn3_wltest+sujith.txt


> then the TIM_TIMER interrupt might be causing the trouble. You can
> check whether any ATH9K_INT_TIM_TIMER interrupt is received in the
> ath_isr when the disconnection happens.

I added status debugging to the isr as such:

--- a/drivers/net/wireless/ath/ath9k/main.c
+++ b/drivers/net/wireless/ath/ath9k/main.c
@@ -587,6 +587,11 @@ irqreturn_t ath_isr(int irq, void *dev)
         * value to insure we only process bits we requested.
         */
        ath9k_hw_getisr(ah, &status);   /* NB: clears ISR too */
+       printk(KERN_INFO "ath: ath_isr status=%08x imask=%08x wecare=%08x",
+               status, sc->imask, status & sc->imask);
+       if (status & ATH9K_INT_TIM_TIMER)
+               printk(" INT_TIM_TIMER");
+       printk("\n");
        status &= sc->imask;    /* discard unasked-for bits */
 
Reloaded the module brought up my networking and manually enabled
power management on the interface, since that exhibits the problem
quickly. A few minutes later I had a new debug=0xffffffff log which
is available here:

http://stuge.se/ath9kdisconn4_wltest+sujith_isrdebug.txt

ATH9K_INT_TIM_TIMER is 0x100 and it may be noteworth that after
413479.000031 "detected beacon loss from AP - sending probe request"
the next time INT_TIM_TIMER is set, at 413479.378293, it is masked
off by sc->imask. Could that be part of the problem?


> This is the hw timer used by the chip to wakeup for  every beacon
> listen interval.

Thank you for the pointer!


//Peter

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Peter Stuge]

Hello,

Peter Stuge wrote:
> I don't think it is. Tonight I saw the problem a third time using
> wireless-testing with power management off for the interface. This
> time I also had Sujith's 6 patches from 2009-12-14 applied.
> 
> 5k lines of debug=0xffffffff log is available at:
> http://stuge.se/ath9kdisconn3_wltest+sujith.txt

I just arrived at a rather busy wireless network, and I have
experienced the problem several times already in about an hour.

What more can I can do to help debug this issue further?

Thanks!

//Peter

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Peter Stuge]

Hello again ath9k experts,

Peter Stuge wrote:
> > > So what could give us more information?
> > 
> > If the issue is specific to AR5416 and power save,
> 
> I don't think it is. Tonight I saw the problem a third time using
> wireless-testing with power management off for the interface. This
> time I also had Sujith's 6 patches from 2009-12-14 applied.
> 
> 5k lines of debug=0xffffffff log is available at:
> http://stuge.se/ath9kdisconn3_wltest+sujith.txt

I am not getting any feedback about this issue. What can I do to
help?

I have been travelling the last few weeks and this issue results in
what I would call severe connectivity issues.

I also had an opportunity to meet with Felix Fietkau and discuss the
problem with him. While he is very familiar with mac80211 in general
and rate control in particular he could not help me understand the RX
path of ath9k in detail.

I applied a patch by Felix from Dec 5 for a rate control bug that
affects master mode:

http://patchwork.kernel.org/patch/68401/
http://patchwork.kernel.org/patch/68512/

This turns out to have significant negative impact on my connectivity
in STA mode. The link rate constantly changes, and ping -f with the
patch loses quite a lot of packets and eventually fails completely,
while ping -f without the patch is perfectly reliable and will stay
usable much longer. This problem may be unrelated to the
disassociation/failure-to-reassociate issue, but maybe not.

Felix emphasized the importance to run wpa_supplicant even if no WPA
is being used, and pointed out that it is quite possible that a wifi
interface can be disassociated because of RF environment changes -
although we were both very surprised that the ath9k card has severe
disassociation issues where my previous ipw2200 card never exhibited
the same symptoms.

I believe that my original problem lies in the RX path. For some
reason, packets are not received, possibly by hardware, possibly
because of settings or tuning that is not quite right. I would like
some input from the experts before I pursue this further.

How can I debug the ath9k RX path, starting in the lowest level radio
layer and working up? Once in the PC I think it might be too late
already, or I expect that the problem would be much more widespread.

In the system where this occurs I am using mix-and-match antennae. I
retrofitted this card into a ThinkPad replacing the ipw2200 card. I
reused the old MAIN and AUX antennae and I added a third antenna
which was delivered with and designed for this card. (Card and third
antenna are original Apple parts.)

A new clue is that once the interface has disassociated from an
access point, it will never associate automatically again. I must
manually down/up the interface, and then it will associate
immediately.

Since I never saw this behavior in exactly the same kernel with
another mac80211 driver (ipw2200) the problem must be in the ath9k
driver or in my "AR5416 MAC/BB Rev:2 AR5133 RF Rev:81" hardware.

I will very much appreciate your input!


//Peter

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Felix Fietkau]

Hi Peter,

On 2010-01-11 4:03 PM, Peter Stuge wrote:
> Since I never saw this behavior in exactly the same kernel with
> another mac80211 driver (ipw2200) the problem must be in the ath9k
> driver or in my "AR5416 MAC/BB Rev:2 AR5133 RF Rev:81" hardware.
ipw2200 is not a mac80211 driver. It probably handles reconnects
internally inside the libipw stack, whereas mac80211 does not do this
and leaves it to user space instead.
The difference in behaviour thus does not necessarily rule out a problem
in the RF environment.

- Felix

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Peter Stuge]

Hi Felix,

Felix Fietkau wrote:
> > Since I never saw this behavior in exactly the same kernel with
> > another mac80211 driver (ipw2200) the problem must be in the ath9k
> > driver or in my "AR5416 MAC/BB Rev:2 AR5133 RF Rev:81" hardware.
> 
> ipw2200 is not a mac80211 driver.

Whoops! That was me not doing thorough research! Sorry about that. :\


> It probably handles reconnects internally inside the libipw stack,
> whereas mac80211 does not do this and leaves it to user space
> instead.

I am not 100% sure, but I believe wpa_supplicant does not actually
help me. I think I still need to down/up the interface to reconnect.
I will test this further to make sure.


> The difference in behaviour thus does not necessarily rule out a
> problem in the RF environment.

I've seen the problem in every RF environment I've visited with this
card. Most are in cities small and large, but one is in an office on
the outskirts of a small city, where the only wifi to be seen is the
two closest of the six-or-so WAP54 access points that we have set up
throughout the adjacent buildings. There's very little traffic in
that network and I still see disconnects.

In any case I would love to debug this from the bottom and up. Any
pointers on registers and settings that may be useful too look into,
in particular if some features of the hardware are not yet supported
by the driver, would be most welcome.


//Peter

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Sujith]

Peter Stuge wrote:
> I've seen the problem in every RF environment I've visited with this
> card. Most are in cities small and large, but one is in an office on
> the outskirts of a small city, where the only wifi to be seen is the
> two closest of the six-or-so WAP54 access points that we have set up
> throughout the adjacent buildings. There's very little traffic in
> that network and I still see disconnects.
> 
> In any case I would love to debug this from the bottom and up. Any
> pointers on registers and settings that may be useful too look into,
> in particular if some features of the hardware are not yet supported
> by the driver, would be most welcome.

Can you upgrade to the latest wireless-testing kernel ?
RX errors can be seen in the 'recv' debugfs file.
The contents of 'rcstat' and 'interrupt' would also be useful.

For more information: http://wireless.kernel.org/en/users/Drivers/ath9k/debug

Sujith

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Peter Stuge]

Sujith wrote:
> > In any case I would love to debug this from the bottom and up.
> 
> Can you upgrade to the latest wireless-testing kernel ?

No problem - I did so yesterday. The last commit I have is:

commit a30ce7f35fbb5643b1e67051b55fb874ed19936a
Merge: e384cc6 7284ce6
Author: John W. Linville <linville@tuxdriver.com>
Date:   Wed Jan 13 11:23:51 2010 -0500

    Merge git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

$ uname -r
2.6.33-rc4-wl

This includes Felix Fietkau's patch which checks for missed error
codes in the tx status, which I experience another issue with, but I
believe that is unrelated. (The rate is very unstable and TCP RTT
suffers, with noticable impact on interactive connections.)


I booted the new kernel last night, enabled power management (which
will usually lead to a disconnect quickly) and ran ping -f for a
fairly long time:

--- 192.168.5.1 ping statistics ---
23441260 packets transmitted, 23437392 received, 0% packet loss, time 45608941ms
rtt min/avg/max/mdev = 1.022/1.976/4171.631/10.415 ms, pipe 190, ipg/ewma 1.945/87.697 ms

3868 ping replies out of 23.4 million were lost. Encouraging but not
ideal. This is not a very noisy environment, with 4 other APs in
range; mine and another on channel 1, others on 6 and 11. If I revert
Felix' commit 5b479a07 the number of lost packets might go down. I
will test that and report in a separate thread.

After stopping the ping I disabled power management for a while and
read email on a remote host. So far no disconnect.

Then I enabled power management again, and started streaming an
internet radio station over TCP. Within a few minutes there was a
disconnect. Then I disabled power management and have now been
listening to the stream for 4? hours without AP disconnect.
(I have needed to restart the stream once, but I think that's because
of the changing rate, related to Felix' patch.)


> RX errors can be seen in the 'recv' debugfs file.

This is from shortly after the disconnect:
# grep -v ': *0$' recv 
           CRC ERR :       3163
           PHY ERR :         18
            LENGTH :         18


And now:
# grep -v ': *0$' recv 
           CRC ERR :       3249
           PHY ERR :         20
            LENGTH :         20


> The contents of 'rcstat' and 'interrupt' would also be useful.

These are from now, unfortunately I did not look at these around the
time of the disconnect.

# cat rcstat 
    HT    MCS   Rate    Success    Retries   XRetries        PER
                1.0:     362456     297267      60600        100
                2.0:     329726     764191     237458         87
                5.5:     218655     549660     188111        100
               11.0:     193089     400518     122522         66
                6.0:          0          0          0          0
                9.0:          0          0          0          0
               12.0:      11502      63178      45627         65
               18.0:       6536      19554       7523         43
               24.0:       5304      13720       4671          0
               36.0:       7234      15586       4956          0
               48.0:     520557      68142       6507          0
               54.0:   22331583    1823652      15689          0

# grep -v ': *0$' interrupt 
      RX:   24499620
      TX:   47898947
     MIB:      73328
   BMISS:        796
     GTT:       4153
   TOTAL:   72325765


Thank you for your attention to this!

Is there some particular information that you would like me to
collect in order to correlate with the interrupt counts?


//Peter

[ath9k-devel] No probe response from AP after 500ms, disconnecting.

[Peter Stuge]

Peter Stuge wrote:
> Then I disabled power management and have now been listening to the
> stream for 4? hours without AP disconnect.

Today I was disconnected twice! First after circa 42 hours, next time
after about two hours. Power management off all the time.

After the first disconnect:

# grep -v ': .*0$' recv 
           CRC ERR :       6461
           PHY ERR :         63
            LENGTH :         63

# grep -v ': .*0$' interrupt  (sorry, typo in this regex excluded MIB+BMISS)
      RX:   28932743
      TX:   52412917
     GTT:       4996
   TOTAL:   81101729

# cat rcstat 
    HT    MCS   Rate    Success    Retries   XRetries        PER
                1.0:     362457     297268      60600          3
                2.0:     329846     764225     237462          7
                5.5:     219008     549931     188156          7
               11.0:     196711     402241     122795         90
                6.0:          0          0          0          0
                9.0:          0          0          0          0
               12.0:      12113      64751      46316         26
               18.0:       7217      20698       7789         90
               24.0:       6486      15581       5042         26
               36.0:      24856      32793       6472         90
               48.0:     645223     156008      13290         87
               54.0:   24568115    2062086      26422         26


After the second:

# grep -v ': *0$' recv

# grep -v ': *0$' interrupt 
      RX:   29075342
      TX:   52578271
     MIB:      76270
   BMISS:        804
     GTT:       5012
   TOTAL:   81406425

# cat rcstat 
    HT    MCS   Rate    Success    Retries   XRetries        PER
                1.0:     362458     297284      60604        100
                2.0:     329847     764237     237465         90
                5.5:     219008     549935     188157         26
               11.0:     197562     402284     122798         90
                6.0:          0          0          0          0
                9.0:          0          0          0          0
               12.0:      12518      66438      46985         26
               18.0:       7590      21803       8180         90
               24.0:       6960      16927       5467          0
               36.0:      25759      34660       7026          0
               48.0:     647533     159724      14309          0
               54.0:   24644173    2069935      27561          0


Thanks!

//Peter