Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Avery Pennarun]

Version: 3.15-rc1 and ath10k-stable-3.11-8 (both via backports to kernel 3.2.26)
Firmware: 10.1.467.2-1

Steps:
- Configure ath10k as AP on channel 149, width 80 MHz, WPA2 encryption
- Connect my 2009 macbook
- Start a ping of 8.8.8.8 in the background from my macbook
- Generate some traffic.  The trigger varies, but running uTorrent or
quickly opening a lot of background tabs in Chrome usually seems to
set it off within a couple of minutes.  iperf and isoping don't seem
to cause any trouble.

Expected:
- ping keeps pinging

Actual:
- tcpdump on the ath10k host shows ICMP requests coming in, and
responses going out.
- tcpdump on the macbook shows ICMP requests going out, but no
responses coming back.
- tcpdump -I (radiotap mode) on the macbook is a little hard to
understand since it's encrypted, but it shows some packets coming out
of the ath10k (broadcasts, I think) but no unicast packets to the
macbook.
- tcpdump on the macbook *does* show broadcast packets arriving.  For
example, ARP requests and "ping -b 192.168.1.255" (my local subnet IP
address) get through.
- Disconnecting and then reconnecting the wifi on my macbook fixes the
problem until it next triggers.
- Other STAs connected to the AP are not affected when the one STA
isn't able to communicate (although each one has the potential to
trigger the problem)

Disabling encryption makes the problem go away permanently.

This is pretty quick for me to reproduce, but unfortunately I don't
have any steps to trigger it instantly, nor any command line tools
that seem to make it happen.  (For example I tried multiple parallel
'curl' processes in a loop, and no lock.)

Nothing interesting appears in the dmesg or hostapd logs at the time
of the problem.

This sounds like it could be a problem with crypto session keys, but I
don't understand why it would only be wrong in a single direction.  I
also don't think my keys are rotating this quickly, so this shouldn't
be a key rotation problem (though I don't understand very well how
that works).

It might be my imagination, but it's possible that this triggers more
quickly if my macbook has been connected for a longer period of time
before generating the traffic burst.

Anything I can check to help narrow this down?

Thanks,

Avery

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Dave Taht]

On Sun, May 11, 2014 at 6:57 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> Version: 3.15-rc1 and ath10k-stable-3.11-8 (both via backports to kernel 3.2.26)
> Firmware: 10.1.467.2-1
>
> Steps:
> - Configure ath10k as AP on channel 149, width 80 MHz, WPA2 encryption
> - Connect my 2009 macbook
> - Start a ping of 8.8.8.8 in the background from my macbook
> - Generate some traffic.  The trigger varies, but running uTorrent or
> quickly opening a lot of background tabs in Chrome usually seems to
> set it off within a couple of minutes.  iperf and isoping don't seem
> to cause any trouble.
>
> Expected:
> - ping keeps pinging
>
> Actual:
> - tcpdump on the ath10k host shows ICMP requests coming in, and
> responses going out.
> - tcpdump on the macbook shows ICMP requests going out, but no
> responses coming back.
> - tcpdump -I (radiotap mode) on the macbook is a little hard to
> understand since it's encrypted, but it shows some packets coming out
> of the ath10k (broadcasts, I think) but no unicast packets to the
> macbook.
> - tcpdump on the macbook *does* show broadcast packets arriving.  For
> example, ARP requests and "ping -b 192.168.1.255" (my local subnet IP
> address) get through.
> - Disconnecting and then reconnecting the wifi on my macbook fixes the
> problem until it next triggers.
> - Other STAs connected to the AP are not affected when the one STA
> isn't able to communicate (although each one has the potential to
> trigger the problem)
>
> Disabling encryption makes the problem go away permanently.
>
> This is pretty quick for me to reproduce, but unfortunately I don't
> have any steps to trigger it instantly, nor any command line tools
> that seem to make it happen.  (For example I tried multiple parallel
> 'curl' processes in a loop, and no lock.)
>
> Nothing interesting appears in the dmesg or hostapd logs at the time
> of the problem.
>
> This sounds like it could be a problem with crypto session keys, but I
> don't understand why it would only be wrong in a single direction.  I
> also don't think my keys are rotating this quickly, so this shouldn't
> be a key rotation problem (though I don't understand very well how
> that works).

I have been failing to find and fix a very similar problem on the
ath9k for many months now. What I see happening there is that one or
more of the
hardware queues locks up, and stops transmitting traffic. So, for
example I might get traffic destined for the BK (background queue,
traffic marked CS1) hung,
but BE remains fine. Most recently I was able to lock up the VO, VI
AND BK queues by exercising it overnight with multiple copies of the
rrul test.

I don't know much about how the hardware queues are configured on
ath10k, but you can land stuff in each queue by marking with CS0, CS1,
CS5, and CS7 (BE,BK,VI,VO) on mac80211 based devices.

I can make it happen more often, faster, if the associated station has
considerable distance and less signal strength than nearby.

My bug in the bug tracker is here:

http://www.bufferbloat.net/issues/442

> It might be my imagination, but it's possible that this triggers more
> quickly if my macbook has been connected for a longer period of time
> before generating the traffic burst.

>
> Anything I can check to help narrow this down?

Move it farther away.

Blow it up with netperf-wrappers -H someserver rrul...

> Thanks,
>
> Avery
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Avery Pennarun]

On Sun, May 11, 2014 at 10:07 PM, Dave Taht <dave.taht@gmail.com> wrote:
> I have been failing to find and fix a very similar problem on the
> ath9k for many months now. What I see happening there is that one or
> more of the
> hardware queues locks up, and stops transmitting traffic. So, for
> example I might get traffic destined for the BK (background queue,
> traffic marked CS1) hung,
> but BE remains fine. Most recently I was able to lock up the VO, VI
> AND BK queues by exercising it overnight with multiple copies of the
> rrul test.
>
> I don't know much about how the hardware queues are configured on
> ath10k, but you can land stuff in each queue by marking with CS0, CS1,
> CS5, and CS7 (BE,BK,VI,VO) on mac80211 based devices.

I think my problem may be something else.  In particular, it seems to
affect each station separately, and doesn't seem to happen if I
disable encryption.  (Does your ath9k problem trigger if encryption is
turned off?)  I also have an ath9k device in the same AP on 2.4 GHz,
and it doesn't trigger there either.  I haven't attempted to see if
your bug triggers on that one though :)

> I can make it happen more often, faster, if the associated station has
> considerable distance and less signal strength than nearby.

I just checked, and my bug seems to trigger more often when I'm at a
longer distance (my macbook says about -60 RSSI) and less often at a
closer distance (currently macbook reports RSSI of -41).  Not sure if
this is related to increased retransmits or decreased speed or
something else.

> Blow it up with netperf-wrappers -H someserver rrul...

That's not a bad idea... I really need to get netperf-wrappers going
for some stress testing :)

Have fun,

Avery

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Dave Taht]

On Sun, May 11, 2014 at 7:29 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Sun, May 11, 2014 at 10:07 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> I have been failing to find and fix a very similar problem on the
>> ath9k for many months now. What I see happening there is that one or
>> more of the
>> hardware queues locks up, and stops transmitting traffic. So, for
>> example I might get traffic destined for the BK (background queue,
>> traffic marked CS1) hung,
>> but BE remains fine. Most recently I was able to lock up the VO, VI
>> AND BK queues by exercising it overnight with multiple copies of the
>> rrul test.
>>
>> I don't know much about how the hardware queues are configured on
>> ath10k, but you can land stuff in each queue by marking with CS0, CS1,
>> CS5, and CS7 (BE,BK,VI,VO) on mac80211 based devices.
>
> I think my problem may be something else.  In particular, it seems to
> affect each station separately, and doesn't seem to happen if I
> disable encryption.  (Does your ath9k problem trigger if encryption is
> turned off?)

No. WPA2 only so far.

I will try multiple stations to see if I can get it to occur only on a
per-station basis. (there are hardware queues for multiple forms of
traffic not just the visible VO, VI, BE, and BK queues)

> I also have an ath9k device in the same AP on 2.4 GHz,
> and it doesn't trigger there either.  I haven't attempted to see if
> your bug triggers on that one though :)

It really takes work to trigger it, and I can can now do it on both
2.4ghz and 5. Getting it down to under 6 hours of high traffic
recently was an accomplishment.

>> I can make it happen more often, faster, if the associated station has
>> considerable distance and less signal strength than nearby.

There are not often executed code paths controlling how noise rejection
works, and all sorts of hardware issues on configuring it that vary between
chipset versions. Ton of patches had landed in head that had an update
to the ANI values
that worked on newer versions of the ath9k chipset that later had to be modified
to deal with older ath9k chipsets.

> I just checked, and my bug seems to trigger more often when I'm at a
> longer distance (my macbook says about -60 RSSI) and less often at a
> closer distance (currently macbook reports RSSI of -41).  Not sure if
> this is related to increased retransmits or decreased speed or
> something else.
>
>> Blow it up with netperf-wrappers -H someserver rrul...
>
> That's not a bad idea... I really need to get netperf-wrappers going
> for some stress testing :)

The hardware queues are rarely tested.

If you just want to blow up one queue at a time, the syntax for netperf is

You can also arbitrarily do tos-setting with iptables.

dnsmasq uses CS6 by default, btw, so it's DHCP packets land by default
in VO and then get shuffled over to the multicast hw queue.


>
> Have fun,
>
> Avery



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Dave Taht]

On Sun, May 11, 2014 at 7:42 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Sun, May 11, 2014 at 7:29 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>> On Sun, May 11, 2014 at 10:07 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>> I have been failing to find and fix a very similar problem on the
>>> ath9k for many months now. What I see happening there is that one or
>>> more of the
>>> hardware queues locks up, and stops transmitting traffic. So, for
>>> example I might get traffic destined for the BK (background queue,
>>> traffic marked CS1) hung,
>>> but BE remains fine. Most recently I was able to lock up the VO, VI
>>> AND BK queues by exercising it overnight with multiple copies of the
>>> rrul test.
>>>
>>> I don't know much about how the hardware queues are configured on
>>> ath10k, but you can land stuff in each queue by marking with CS0, CS1,
>>> CS5, and CS7 (BE,BK,VI,VO) on mac80211 based devices.
>>
>> I think my problem may be something else.  In particular, it seems to
>> affect each station separately, and doesn't seem to happen if I
>> disable encryption.  (Does your ath9k problem trigger if encryption is
>> turned off?)
>
> No. WPA2 only so far.
>
> I will try multiple stations to see if I can get it to occur only on a
> per-station basis. (there are hardware queues for multiple forms of
> traffic not just the visible VO, VI, BE, and BK queues)
>
>> I also have an ath9k device in the same AP on 2.4 GHz,
>> and it doesn't trigger there either.  I haven't attempted to see if
>> your bug triggers on that one though :)
>
> It really takes work to trigger it, and I can can now do it on both
> 2.4ghz and 5. Getting it down to under 6 hours of high traffic
> recently was an accomplishment.
>
>>> I can make it happen more often, faster, if the associated station has
>>> considerable distance and less signal strength than nearby.
>
> There are not often executed code paths controlling how noise rejection
> works, and all sorts of hardware issues on configuring it that vary between
> chipset versions. Ton of patches had landed in head that had an update
> to the ANI values
> that worked on newer versions of the ath9k chipset that later had to be modified
> to deal with older ath9k chipsets.
>
>> I just checked, and my bug seems to trigger more often when I'm at a
>> longer distance (my macbook says about -60 RSSI) and less often at a
>> closer distance (currently macbook reports RSSI of -41).  Not sure if
>> this is related to increased retransmits or decreased speed or
>> something else.
>>
>>> Blow it up with netperf-wrappers -H someserver rrul...
>>
>> That's not a bad idea... I really need to get netperf-wrappers going
>> for some stress testing :)
>
> The hardware queues are rarely tested.
>
> If you just want to blow up one queue at a time, the syntax for netperf is

netperf -H someserver -t the_test -Y CS1,CS1 # or CS5,CS5 or CS6, CS6

I have been flooding all the queues with both -t TCP_STREAM and TCP_MAERTS
to make it happen using the rrul test, but I have also made it happen
with BE only.

Getting one data point every day or so makes for slow debugging.

>
> You can also arbitrarily do tos-setting with iptables.
>
> dnsmasq uses CS6 by default, btw, so it's DHCP packets land by default
> in VO and then get shuffled over to the multicast hw queue.
>
>
>>
>> Have fun,
>>
>> Avery
>
>
>
> --
> Dave Täht
>
> NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Ben Greear]

On 05/11/2014 06:57 PM, Avery Pennarun wrote:
> Version: 3.15-rc1 and ath10k-stable-3.11-8 (both via backports to kernel 3.2.26)
> Firmware: 10.1.467.2-1
>
> Steps:
> - Configure ath10k as AP on channel 149, width 80 MHz, WPA2 encryption
> - Connect my 2009 macbook
> - Start a ping of 8.8.8.8 in the background from my macbook
> - Generate some traffic.  The trigger varies, but running uTorrent or
> quickly opening a lot of background tabs in Chrome usually seems to
> set it off within a couple of minutes.  iperf and isoping don't seem
> to cause any trouble.
>
> Expected:
> - ping keeps pinging
>
> Actual:
> - tcpdump on the ath10k host shows ICMP requests coming in, and
> responses going out.
> - tcpdump on the macbook shows ICMP requests going out, but no
> responses coming back.
> - tcpdump -I (radiotap mode) on the macbook is a little hard to
> understand since it's encrypted, but it shows some packets coming out
> of the ath10k (broadcasts, I think) but no unicast packets to the
> macbook.
> - tcpdump on the macbook *does* show broadcast packets arriving.  For
> example, ARP requests and "ping -b 192.168.1.255" (my local subnet IP
> address) get through.
> - Disconnecting and then reconnecting the wifi on my macbook fixes the
> problem until it next triggers.
> - Other STAs connected to the AP are not affected when the one STA
> isn't able to communicate (although each one has the potential to
> trigger the problem)

Can you reproduce with any other type of station
device?

Also, have you tried sniffing with a third device to see if
the AP actually puts the ICMP responses on the air?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Avery Pennarun]

On Sun, May 11, 2014 at 11:09 PM, Ben Greear <greearb@candelatech.com> wrote:
> Can you reproduce with any other type of station
> device?

I'll try.  It's a little tricky since it seems to involve running
Chrome, etc, rather than my usual command-line tests for which I have
a wider variety of clients.

> Also, have you tried sniffing with a third device to see if
> the AP actually puts the ICMP responses on the air?

I did try that.  As far as I can tell, the ICMP responses are simply
not being sent at all.

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Ben Greear]

On 05/11/2014 08:54 PM, Avery Pennarun wrote:
> On Sun, May 11, 2014 at 11:09 PM, Ben Greear <greearb@candelatech.com> wrote:
>> Can you reproduce with any other type of station
>> device?
>
> I'll try.  It's a little tricky since it seems to involve running
> Chrome, etc, rather than my usual command-line tests for which I have
> a wider variety of clients.
>
>> Also, have you tried sniffing with a third device to see if
>> the AP actually puts the ICMP responses on the air?
>
> I did try that.  As far as I can tell, the ICMP responses are simply
> not being sent at all.

I haven't dug into many of the stats yet, but it's possible the
ath10k debugfs file would show some types of transmit errors in this case?

Might be interesting to see how long it takes the AP to generate
the tx status response for the transmitted ICMP packets.  My firmware
has some extended tx status, but I'm not sure it has anything overly
useful for your case...I was mostly concerned with the tx rate reporting
when writing it.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Avery Pennarun]

On Mon, May 12, 2014 at 12:05 AM, Ben Greear <greearb@candelatech.com> wrote:
> On 05/11/2014 08:54 PM, Avery Pennarun wrote:
>> On Sun, May 11, 2014 at 11:09 PM, Ben Greear <greearb@candelatech.com>
>> wrote:
>>> Also, have you tried sniffing with a third device to see if
>>> the AP actually puts the ICMP responses on the air?
>>
>> I did try that.  As far as I can tell, the ICMP responses are simply
>> not being sent at all.

I was incorrect about this because I was looking at the wrong data.  I
tested again with a more obvious method, running "ping -i0.01
192.168.1.107" where 107 is the address of my macbook.  With 100
packets per second, they overwhelm the rest of the traffic so it's
easy to see whether they're coming through.

The AP definitely *is* transmitting the packets on the air.  Even my
macbook can see them in the radiotap tcpdump mode, but it doesn't see
them at the IP layer.  So they are either being encrypted wrong or my
macbook is decrypting them wrong, I guess.

I think I'll try using wireshark and see what it thinks...

> I haven't dug into many of the stats yet, but it's possible the
> ath10k debugfs file would show some types of transmit errors in this case?

Possibly.  Can you give me a hint of where to look?  I don't really
know what these files do.

> Might be interesting to see how long it takes the AP to generate
> the tx status response for the transmitted ICMP packets.  My firmware
> has some extended tx status, but I'm not sure it has anything overly
> useful for your case...I was mostly concerned with the tx rate reporting
> when writing it.

I can try it with your firmware if you think there is useful data to
gather, although since it turned out my earlier statement wasn't true
maybe this is less important :)

Thanks,

Avery

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Ben Greear]

On 05/11/2014 09:56 PM, Avery Pennarun wrote:
> On Mon, May 12, 2014 at 12:05 AM, Ben Greear <greearb@candelatech.com> wrote:
>> On 05/11/2014 08:54 PM, Avery Pennarun wrote:
>>> On Sun, May 11, 2014 at 11:09 PM, Ben Greear <greearb@candelatech.com>
>>> wrote:
>>>> Also, have you tried sniffing with a third device to see if
>>>> the AP actually puts the ICMP responses on the air?
>>>
>>> I did try that.  As far as I can tell, the ICMP responses are simply
>>> not being sent at all.
>
> I was incorrect about this because I was looking at the wrong data.  I
> tested again with a more obvious method, running "ping -i0.01
> 192.168.1.107" where 107 is the address of my macbook.  With 100
> packets per second, they overwhelm the rest of the traffic so it's
> easy to see whether they're coming through.
>
> The AP definitely *is* transmitting the packets on the air.  Even my
> macbook can see them in the radiotap tcpdump mode, but it doesn't see
> them at the IP layer.  So they are either being encrypted wrong or my
> macbook is decrypting them wrong, I guess.
>
> I think I'll try using wireshark and see what it thinks...
>
>> I haven't dug into many of the stats yet, but it's possible the
>> ath10k debugfs file would show some types of transmit errors in this case?
>
> Possibly.  Can you give me a hint of where to look?  I don't really
> know what these files do.
>
>> Might be interesting to see how long it takes the AP to generate
>> the tx status response for the transmitted ICMP packets.  My firmware
>> has some extended tx status, but I'm not sure it has anything overly
>> useful for your case...I was mostly concerned with the tx rate reporting
>> when writing it.
>
> I can try it with your firmware if you think there is useful data to
> gather, although since it turned out my earlier statement wasn't true
> maybe this is less important :)

If it's getting on the air, then I think the only way to figure out
what is wrong is to decode the packets and see if they are encrypted
properly or not.  I think there is a way to get wireshark to decode
pkts by feeding it the proper keys, but I have not ever actually tried
doing that.

If pkts do not get on the air, then possibly the tx status and/or tx
error counters could tell you why, but it seems that is not relevant
in this case.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Avery Pennarun]

On Mon, May 12, 2014 at 1:05 AM, Ben Greear <greearb@candelatech.com> wrote:
> If it's getting on the air, then I think the only way to figure out
> what is wrong is to decode the packets and see if they are encrypted
> properly or not.  I think there is a way to get wireshark to decode
> pkts by feeding it the proper keys, but I have not ever actually tried
> doing that.
>
> If pkts do not get on the air, then possibly the tx status and/or tx
> error counters could tell you why, but it seems that is not relevant
> in this case.

Okay, in fact I just learned how to use that wireshark feature last
week, so I tried it just now and it worked.  It clearly shows the
downstream packets from the AP *are* decryptable from wireshark, but
they get no replies from the macbook (and the macbook doesn't show
them at its IP layer).

Any guesses?  Does that mean somehow the macbook got the wrong
*decryption* keys or something?

I'm currently trying to wade through the giant wireshark capture
trying to find the actual point where the dropout occurred, but it
seems to slow down a bit with 1.5 million frames captured :)

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Avery Pennarun]

On Mon, May 12, 2014 at 1:05 AM, Ben Greear <greearb@candelatech.com> wrote:
> If it's getting on the air, then I think the only way to figure out
> what is wrong is to decode the packets and see if they are encrypted
> properly or not.  I think there is a way to get wireshark to decode
> pkts by feeding it the proper keys, but I have not ever actually tried
> doing that.

Okay, here is a fairly reduced capture of my wireshark trace:
http://apenwarr.ca/tmp/ath10k-utorrent-dropout-v2-reduced.pcapng.gz

To decode the wifi packets in wireshark, you need to follow these steps:
- open the pcap
- Edit | Preferences
- Protocols | IEEE 802.11
- Enable Decryption: checked
- Decryption Keys: Edit
- New
- Key type: wpa-pwd
- Key: my-password:my-ssid  (I'll email these to Ben privately; anyone
else interested, let me know)
- Ok
- Ok

The capture contains a lot of simultaneous TCP and UDP sessions, since
I can only trigger the problem when there is quite a bit of stuff
going on.  Luckily the air itself was relatively quiet so there isn't
too much noise other than my AP and laptop.

I believe I can narrow down the dropout to somewhere between rows
20035 and 20391.  20035 is a packet from a remote server that is ACKed
by my macbook.  20391 is a packet from the same remote server that is
*not* ACKed by my macbook, and then there are a bunch of retransmits
on that session after that.

I'm still looking at the capture to find any clues, but if anybody
else wants to take a peek, please do :)

Thanks,

Avery

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Avery Pennarun]

On Mon, May 12, 2014 at 3:07 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Mon, May 12, 2014 at 1:05 AM, Ben Greear <greearb@candelatech.com> wrote:
>> If it's getting on the air, then I think the only way to figure out
>> what is wrong is to decode the packets and see if they are encrypted
>> properly or not.  I think there is a way to get wireshark to decode
>> pkts by feeding it the proper keys, but I have not ever actually tried
>> doing that.
>
> Okay, here is a fairly reduced capture of my wireshark trace:
> http://apenwarr.ca/tmp/ath10k-utorrent-dropout-v2-reduced.pcapng.gz
> [...]

From what I can see, the most suspicious part of this trace is an
802.11 Action - Add Block Ack Request at time 181.1456.  In a longer
version of this trace, I can see these occurring sporadically.  Each
one has a different TI field:

Time 8.367: TID=0
Time 9.609: TID=6
Time 156.188: TID=1
Time 178.076: TID=2
Time 181.145: TID=3  <-- almost exactly the time of the failure
Time 190.127: TID=5

I don't know how 802.11 QoS works, but I don't think it makes sense to
keep negotiating block ack policies with different TIDs.  Also, the
Block Ack Starting Sequence Number field is always zero; I don't know
what that does, but it seems weird to me.

Nevertheless, I can certainly see things being ACKed in my wifi
traces.  I think maybe some incorrect Add Block Ack Request packets
may be triggering a bug in the wifi driver on my macbook?

IEEE 802.11 wireless LAN management frame
    Fixed parameters
        Category code: Block Ack (3)
        Action code: Add Block Ack Request (0x00)
        Dialog token: 0x01
        Block Ack Parameters: 0x1017, A-MSDUs, Block Ack Policy
            .... .... .... ...1 = A-MSDUs: Permitted in QoS Data MPDUs
            .... .... .... ..1. = Block Ack Policy: Immediate Block Ack
            .... .... ..01 01.. = Traffic Identifier: 0x0005
            0001 0000 00.. .... = Number of Buffers (1 Buffer = 2304 Bytes): 64
        Block Ack Timeout: 0x0000
        Block Ack Starting Sequence Control (SSC): 0x0000
            .... .... .... 0000 = Fragment: 0
            0000 0000 0000 .... = Starting Sequence Number: 0


Thanks,

Avery

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Ben Greear]

On 05/12/2014 01:21 AM, Avery Pennarun wrote:
> On Mon, May 12, 2014 at 3:07 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
>> On Mon, May 12, 2014 at 1:05 AM, Ben Greear <greearb@candelatech.com> wrote:
>>> If it's getting on the air, then I think the only way to figure out
>>> what is wrong is to decode the packets and see if they are encrypted
>>> properly or not.  I think there is a way to get wireshark to decode
>>> pkts by feeding it the proper keys, but I have not ever actually tried
>>> doing that.
>>
>> Okay, here is a fairly reduced capture of my wireshark trace:
>> http://apenwarr.ca/tmp/ath10k-utorrent-dropout-v2-reduced.pcapng.gz
>> [...]
> 
> From what I can see, the most suspicious part of this trace is an
> 802.11 Action - Add Block Ack Request at time 181.1456.  In a longer
> version of this trace, I can see these occurring sporadically.  Each
> one has a different TI field:
> 
> Time 8.367: TID=0
> Time 9.609: TID=6
> Time 156.188: TID=1
> Time 178.076: TID=2
> Time 181.145: TID=3  <-- almost exactly the time of the failure
> Time 190.127: TID=5
> 
> I don't know how 802.11 QoS works, but I don't think it makes sense to
> keep negotiating block ack policies with different TIDs.  Also, the
> Block Ack Starting Sequence Number field is always zero; I don't know
> what that does, but it seems weird to me.

I'm afraid I don't know much about blockack either.  I did fix an assert
and use-after-free bug in blockack code in my firmware (though, possibly
the use-after-free was introduced by my earlier assert fix), and possibly
neither of these fixes are needed in .467 firmware, as I was first developing
on earlier firmware.

I think my blockack fixes only applied to bringing stations up
and down, so probably not related to your issue.

Hopefully someone that knows more about BA can comment on your
trace.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Adrian Chadd]

Hi,

I've faced this in FreeBSD. It's only very recently that I found some
corner case block-ack window tracking bugs that only occured during
periods of extreme packet loss. What I would do when this happens:

* I'd dump out the entire software queue for the hung station and
hardware queue state. the hardware queue state tends to be quite small
on ath9k/freebsd as we artifically limit the queue depth
* I also hacked up a rolling log of all the transmit, transmit
completion, baw add, baw remove log entries, so I could go back in
history to find where the hole came from.

The last one I found took a day of active torrenting and around 500
million log line entries just to trigger. :-)

As for firmware, lemme respond to that separately.



-a

On 11 May 2014 19:46, Dave Taht <dave.taht@gmail.com> wrote:
> On Sun, May 11, 2014 at 7:42 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Sun, May 11, 2014 at 7:29 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>>> On Sun, May 11, 2014 at 10:07 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>>> I have been failing to find and fix a very similar problem on the
>>>> ath9k for many months now. What I see happening there is that one or
>>>> more of the
>>>> hardware queues locks up, and stops transmitting traffic. So, for
>>>> example I might get traffic destined for the BK (background queue,
>>>> traffic marked CS1) hung,
>>>> but BE remains fine. Most recently I was able to lock up the VO, VI
>>>> AND BK queues by exercising it overnight with multiple copies of the
>>>> rrul test.
>>>>
>>>> I don't know much about how the hardware queues are configured on
>>>> ath10k, but you can land stuff in each queue by marking with CS0, CS1,
>>>> CS5, and CS7 (BE,BK,VI,VO) on mac80211 based devices.
>>>
>>> I think my problem may be something else.  In particular, it seems to
>>> affect each station separately, and doesn't seem to happen if I
>>> disable encryption.  (Does your ath9k problem trigger if encryption is
>>> turned off?)
>>
>> No. WPA2 only so far.
>>
>> I will try multiple stations to see if I can get it to occur only on a
>> per-station basis. (there are hardware queues for multiple forms of
>> traffic not just the visible VO, VI, BE, and BK queues)
>>
>>> I also have an ath9k device in the same AP on 2.4 GHz,
>>> and it doesn't trigger there either.  I haven't attempted to see if
>>> your bug triggers on that one though :)
>>
>> It really takes work to trigger it, and I can can now do it on both
>> 2.4ghz and 5. Getting it down to under 6 hours of high traffic
>> recently was an accomplishment.
>>
>>>> I can make it happen more often, faster, if the associated station has
>>>> considerable distance and less signal strength than nearby.
>>
>> There are not often executed code paths controlling how noise rejection
>> works, and all sorts of hardware issues on configuring it that vary between
>> chipset versions. Ton of patches had landed in head that had an update
>> to the ANI values
>> that worked on newer versions of the ath9k chipset that later had to be modified
>> to deal with older ath9k chipsets.
>>
>>> I just checked, and my bug seems to trigger more often when I'm at a
>>> longer distance (my macbook says about -60 RSSI) and less often at a
>>> closer distance (currently macbook reports RSSI of -41).  Not sure if
>>> this is related to increased retransmits or decreased speed or
>>> something else.
>>>
>>>> Blow it up with netperf-wrappers -H someserver rrul...
>>>
>>> That's not a bad idea... I really need to get netperf-wrappers going
>>> for some stress testing :)
>>
>> The hardware queues are rarely tested.
>>
>> If you just want to blow up one queue at a time, the syntax for netperf is
>
> netperf -H someserver -t the_test -Y CS1,CS1 # or CS5,CS5 or CS6, CS6
>
> I have been flooding all the queues with both -t TCP_STREAM and TCP_MAERTS
> to make it happen using the rrul test, but I have also made it happen
> with BE only.
>
> Getting one data point every day or so makes for slow debugging.
>
>>
>> You can also arbitrarily do tos-setting with iptables.
>>
>> dnsmasq uses CS6 by default, btw, so it's DHCP packets land by default
>> in VO and then get shuffled over to the multicast hw queue.
>>
>>
>>>
>>> Have fun,
>>>
>>> Avery
>>
>>
>>
>> --
>> Dave Täht
>>
>> NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article
>
>
>
> --
> Dave Täht
>
> NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Adrian Chadd]

Hi,

So the keys aren't bidirectional. There'll be a set of transmit and
receive keys per station, as well as the multicast key.

If you're seeing traffic in other TIDs for a given station, then it's
likely a block-ack tracking issue.

If you only see multicast traffic to the station, but no unicast
traffic, then it could be all the HWQ's are stuck; but it also could
be a key exchange issue. Normally on FreeBSD I'd just do a monitor
mode trace and look at whether I was receiving anything; before I then
dug down to see if the encryption sequence was fine (ie, no CCMP
replays, the key is correct, etc.)

As for ath10k, it's been a while since I was deep in the firmware.
Ideally you'd like to get TX completion notifications from the chip up
to the firmware as that'll at least tell you the frame is making it
out to the air and being ACKed (and is thus not going to be a hung
hardware queue or TID.) I don't know if TX completion notifications
are completely working though; I'll check once I have firmware source
again.



-a


On 11 May 2014 18:57, Avery Pennarun <apenwarr@gmail.com> wrote:
> Version: 3.15-rc1 and ath10k-stable-3.11-8 (both via backports to kernel 3.2.26)
> Firmware: 10.1.467.2-1
>
> Steps:
> - Configure ath10k as AP on channel 149, width 80 MHz, WPA2 encryption
> - Connect my 2009 macbook
> - Start a ping of 8.8.8.8 in the background from my macbook
> - Generate some traffic.  The trigger varies, but running uTorrent or
> quickly opening a lot of background tabs in Chrome usually seems to
> set it off within a couple of minutes.  iperf and isoping don't seem
> to cause any trouble.
>
> Expected:
> - ping keeps pinging
>
> Actual:
> - tcpdump on the ath10k host shows ICMP requests coming in, and
> responses going out.
> - tcpdump on the macbook shows ICMP requests going out, but no
> responses coming back.
> - tcpdump -I (radiotap mode) on the macbook is a little hard to
> understand since it's encrypted, but it shows some packets coming out
> of the ath10k (broadcasts, I think) but no unicast packets to the
> macbook.
> - tcpdump on the macbook *does* show broadcast packets arriving.  For
> example, ARP requests and "ping -b 192.168.1.255" (my local subnet IP
> address) get through.
> - Disconnecting and then reconnecting the wifi on my macbook fixes the
> problem until it next triggers.
> - Other STAs connected to the AP are not affected when the one STA
> isn't able to communicate (although each one has the potential to
> trigger the problem)
>
> Disabling encryption makes the problem go away permanently.
>
> This is pretty quick for me to reproduce, but unfortunately I don't
> have any steps to trigger it instantly, nor any command line tools
> that seem to make it happen.  (For example I tried multiple parallel
> 'curl' processes in a loop, and no lock.)
>
> Nothing interesting appears in the dmesg or hostapd logs at the time
> of the problem.
>
> This sounds like it could be a problem with crypto session keys, but I
> don't understand why it would only be wrong in a single direction.  I
> also don't think my keys are rotating this quickly, so this shouldn't
> be a key rotation problem (though I don't understand very well how
> that works).
>
> It might be my imagination, but it's possible that this triggers more
> quickly if my macbook has been connected for a longer period of time
> before generating the traffic burst.
>
> Anything I can check to help narrow this down?
>
> Thanks,
>
> Avery
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Avery Pennarun]

On Mon, May 12, 2014 at 4:21 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
> Nevertheless, I can certainly see things being ACKed in my wifi
> traces.  I think maybe some incorrect Add Block Ack Request packets
> may be triggering a bug in the wifi driver on my macbook?
>
> IEEE 802.11 wireless LAN management frame
>     Fixed parameters
>         Category code: Block Ack (3)
>         Action code: Add Block Ack Request (0x00)
>         Dialog token: 0x01
>         Block Ack Parameters: 0x1017, A-MSDUs, Block Ack Policy
>             .... .... .... ...1 = A-MSDUs: Permitted in QoS Data MPDUs
>             .... .... .... ..1. = Block Ack Policy: Immediate Block Ack
>             .... .... ..01 01.. = Traffic Identifier: 0x0005
>             0001 0000 00.. .... = Number of Buffers (1 Buffer = 2304 Bytes): 64
>         Block Ack Timeout: 0x0000
>         Block Ack Starting Sequence Control (SSC): 0x0000
>             .... .... .... 0000 = Fragment: 0
>             0000 0000 0000 .... = Starting Sequence Number: 0

Replicated this a few more times.  Still on a Macbook, but a different
one (latest MacOS version this time) on a different setting.

Once again, the problem kicked in immediately following an ADDBA
request like the above.  The TID was this time equal to 3 (which I
think is the problematic one last time too; I accidentally pasted the
ADDBA for TID=5 but in my earlier email I called out TID=3 as the one
that triggered the problem).

In this new test case, interestingly, unicast continued to work as
long as I was sending packets off-LAN (ie. through a gateway to
somewhere on the Internet); packets could be exchanged in both
directions.  But I couldn't ping the gateway or any other
LAN-connected device.  This is fishy since I might have blamed a lack
of multicast/broadcast and thus a failure of ARP (unicast was
obviously working) but my ARP table definitely contained the IP of the
gateway.

Anyway, one new clue: the ADDBA was triggered by a TCP ACK packet
being sent by the SPDY port on a server somewhere with a particular
packet priority: diffserv field = 0x60 (class selector 3, no ECN).

I don't know why this would scramble my Macbook's wifi, since the
packets themselves looked fine, but it did.  This was definitely the
first SPDY packet on the session and it triggered the ADDBA and died
instantly.

I really should take Dave Taht's advice and get that netperf thing
going and test all the QoS queues... but meanwhile, I think I'm in a
big hurry so I probably want to just rip out all support for mapping
diffserv to QoS on the ath10k transmit side.  Does anyone have any
suggestions on where to look for my hack-and-slash operation?

Thanks,

Avery

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Dave Taht]

On Wed, May 14, 2014 at 12:07 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Mon, May 12, 2014 at 4:21 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
>> Nevertheless, I can certainly see things being ACKed in my wifi
>> traces.  I think maybe some incorrect Add Block Ack Request packets
>> may be triggering a bug in the wifi driver on my macbook?
>>
>> IEEE 802.11 wireless LAN management frame
>>     Fixed parameters
>>         Category code: Block Ack (3)
>>         Action code: Add Block Ack Request (0x00)
>>         Dialog token: 0x01
>>         Block Ack Parameters: 0x1017, A-MSDUs, Block Ack Policy
>>             .... .... .... ...1 = A-MSDUs: Permitted in QoS Data MPDUs
>>             .... .... .... ..1. = Block Ack Policy: Immediate Block Ack
>>             .... .... ..01 01.. = Traffic Identifier: 0x0005
>>             0001 0000 00.. .... = Number of Buffers (1 Buffer = 2304 Bytes): 64
>>         Block Ack Timeout: 0x0000
>>         Block Ack Starting Sequence Control (SSC): 0x0000
>>             .... .... .... 0000 = Fragment: 0
>>             0000 0000 0000 .... = Starting Sequence Number: 0
>
> Replicated this a few more times.  Still on a Macbook, but a different
> one (latest MacOS version this time) on a different setting.
>
> Once again, the problem kicked in immediately following an ADDBA
> request like the above.  The TID was this time equal to 3 (which I
> think is the problematic one last time too; I accidentally pasted the
> ADDBA for TID=5 but in my earlier email I called out TID=3 as the one
> that triggered the problem).
>
> In this new test case, interestingly, unicast continued to work as
> long as I was sending packets off-LAN (ie. through a gateway to
> somewhere on the Internet); packets could be exchanged in both
> directions.  But I couldn't ping the gateway or any other
> LAN-connected device.  This is fishy since I might have blamed a lack
> of multicast/broadcast and thus a failure of ARP (unicast was
> obviously working) but my ARP table definitely contained the IP of the
> gateway.
>
> Anyway, one new clue: the ADDBA was triggered by a TCP ACK packet
> being sent by the SPDY port on a server somewhere with a particular
> packet priority: diffserv field = 0x60 (class selector 3, no ECN).
>
> I don't know why this would scramble my Macbook's wifi, since the
> packets themselves looked fine, but it did.  This was definitely the
> first SPDY packet on the session and it triggered the ADDBA and died
> instantly.
>
> I really should take Dave Taht's advice and get that netperf thing
> going and test all the QoS queues... but meanwhile, I think I'm in a
> big hurry so I probably want to just rip out all support for mapping
> diffserv to QoS on the ath10k transmit side.  Does anyone have any
> suggestions on where to look for my hack-and-slash operation?

Tis easy to hack and slash it out of cfg80211_classify8021d in
net/wireless/util.c

Recently someone added a direct mapping from vlan priorities to wifi
queues here - which is stupid, as the VO queue does not behave
anything like the equivalent vlan priority queue does. I'd like to see
that removed or mapped properly, also.

Or you can try telling hostapd to never negotiate wmm/802.11e.

Last night I got it to blow up the vi, vo, and bk queues on the ath9k
in about 4 hours, and I just successfully in under 20 minutes got the
BK queue to misbehave, driven by an iwl driver on linux, on both ipv6
and ipv4. It is probable this is elsewhere in the stack than the
drivers themselves. (although things are so intertwined down here that
it's hard to tell). Perhaps running on a noisier network triggers it
faster.

I logged last nights progress on http://www.bufferbloat.net/issues/442
- with pictures and logs - there are packet captures on that bug now,
but not on the monitoring interface. Adding that now.

What I just saw looks like this.

Command: /usr/local/bin/netperf -P 0 -v 0 -D -0.2 -6 -Y CS1,CS1 -H
fdfe:7b16:f8e2::1 -t TCP_STREAM -l 300 -f m
Program output:
  netperf: send_omni: connect_data_socket failed: Connection timed out

Warning: Command produced no valid data.
Data series: Ping (ms) UDP BK
Runner: NetperfDemoRunner
Command: /usr/local/bin/netperf -P 0 -v 0 -D -0.2 -6 -Y CS1,CS1 -H
fdfe:7b16:f8e2::1 -t UDP_RR -l 310 -- -e 2
Standard error output:

+ :
+ expr 3 + 1
+ i=4
+ netperf-wrapper -l 300 -H 172.21.18.1 -t default-4 -o default-4.svg
-p all_scaled rrul
Warning: Program exited non-zero (1).
Command: /usr/local/bin/netperf -P 0 -v 0 -D -0.2 -4 -Y CS1,CS1 -H
172.21.18.1 -t TCP_MAERTS -l 300 -f m
Program output:
  netperf: send_omni: connect_data_socket failed: Connection timed out

Warning: Program exited non-zero (1).
Command: /usr/local/bin/netperf -P 0 -v 0 -D -0.2 -4 -Y CS1,CS1 -H
172.21.18.1 -t TCP_STREAM -l 300 -f m
Program output:
  netperf: send_omni: connect_data_socket failed: Connection timed out

The bk, be, and vi queues are currently still operational.
>
> Thanks,
>
> Avery
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Adrian Chadd]

[snip]

I found a lot of .. bad behaving WME/TID/QoS mapping code in freebsd's
atheros and net80211 code. It wouldn't surprise me to find the same in
Linux. There's not always a lot of driver / stack testing with
multiple overlapping TID traffic. (I found a lot out by actually
running lots of torrents whilst working; that triggers some bulk/best
effort/high priority concurrent traffic and it clued me in for how
screwed up it is.)

Ok, so the "seqno = 0" thing is something that various wireless stacks
do (including a couple of freebsd drivers, ugh) - when aggregation is
enabled, they reset the seqno back to 0 before continuing. It's kind
of odd, but it's typically because the non-aggregation seqno space was
done by the wireless stack (net80211 or derived) but then aggregation,
BAW tracking transmit/retransmit is done by the driver and/or
firmware. Thus it doesn't actually know about the sequence numbers in
question.

Yes, you negotiate a different block-ack setup with each TID. Same as
how each TID has a separate sequence number space. It gets more
interesting, as there's only one set of keys for all TIDs, yet each
TID can have a separate replay counter.

Anyway, lemme look at the trace. I'd like to see what's going on after
the negotitation.


-a

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Adrian Chadd]

Hi,

When you quote frame offsets in a pcap, can you next time use the frame number?

I think your'e referring to 20253, right? That looks like the right frame.



-a


On 12 May 2014 01:21, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Mon, May 12, 2014 at 3:07 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
>> On Mon, May 12, 2014 at 1:05 AM, Ben Greear <greearb@candelatech.com> wrote:
>>> If it's getting on the air, then I think the only way to figure out
>>> what is wrong is to decode the packets and see if they are encrypted
>>> properly or not.  I think there is a way to get wireshark to decode
>>> pkts by feeding it the proper keys, but I have not ever actually tried
>>> doing that.
>>
>> Okay, here is a fairly reduced capture of my wireshark trace:
>> http://apenwarr.ca/tmp/ath10k-utorrent-dropout-v2-reduced.pcapng.gz
>> [...]
>
> From what I can see, the most suspicious part of this trace is an
> 802.11 Action - Add Block Ack Request at time 181.1456.  In a longer
> version of this trace, I can see these occurring sporadically.  Each
> one has a different TI field:
>
> Time 8.367: TID=0
> Time 9.609: TID=6
> Time 156.188: TID=1
> Time 178.076: TID=2
> Time 181.145: TID=3  <-- almost exactly the time of the failure
> Time 190.127: TID=5
>
> I don't know how 802.11 QoS works, but I don't think it makes sense to
> keep negotiating block ack policies with different TIDs.  Also, the
> Block Ack Starting Sequence Number field is always zero; I don't know
> what that does, but it seems weird to me.
>
> Nevertheless, I can certainly see things being ACKed in my wifi
> traces.  I think maybe some incorrect Add Block Ack Request packets
> may be triggering a bug in the wifi driver on my macbook?
>
> IEEE 802.11 wireless LAN management frame
>     Fixed parameters
>         Category code: Block Ack (3)
>         Action code: Add Block Ack Request (0x00)
>         Dialog token: 0x01
>         Block Ack Parameters: 0x1017, A-MSDUs, Block Ack Policy
>             .... .... .... ...1 = A-MSDUs: Permitted in QoS Data MPDUs
>             .... .... .... ..1. = Block Ack Policy: Immediate Block Ack
>             .... .... ..01 01.. = Traffic Identifier: 0x0005
>             0001 0000 00.. .... = Number of Buffers (1 Buffer = 2304 Bytes): 64
>         Block Ack Timeout: 0x0000
>         Block Ack Starting Sequence Control (SSC): 0x0000
>             .... .... .... 0000 = Fragment: 0
>             0000 0000 0000 .... = Starting Sequence Number: 0
>
>
> Thanks,
>
> Avery
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Adrian Chadd]

ok, so I've taken a 30 second look.

The CCMP IV's used for TID 3 look creepy. They start at 0x300000000000
and go up. It's much higher than the rest of the IVs used for
transmitting on the other TIDs.

It'd be nice to see TID 3 traffic -before- it negotiates the ADDBA,
just to see what the heck is going on there.

What _should_ be happening!

* transmitting to a station should have exactly one CCMP IV, shared
across all frames being transmitted. It should be allocated in order,
regardless of the TID
* on the receiving side, there's tracking for the CCMP IV seen for
each TID, and checked to avoid replay attacks.

It's choosing a totally different CCMP IV space for transmitting on
TID 3. I don't know why.

(And I still don't have firmware source, so I can't tell you yet. grr.)

Ben - I'm punting this one to you!




-a

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Avery Pennarun]

On Wed, May 14, 2014 at 4:46 PM, Adrian Chadd <adrian@freebsd.org> wrote:
> It'd be nice to see TID 3 traffic -before- it negotiates the ADDBA,
> just to see what the heck is going on there.

I'd have to re-check the big trace you're looking at, but at least in
my new one from today, it actually does the ADDBA before *any* traffic is
sent on TID=3.

The IV being different from what it should be definitely sounds like
something that would explain the odd behaviour of wireshark being able
to decode it, but MacOS rejecting it.  I guess MacOS could be
rejecting unexpected IVs to avoid replay attacks.

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Avery Pennarun]

On Wed, May 14, 2014 at 3:26 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Wed, May 14, 2014 at 12:07 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>> but meanwhile, I think I'm in a
>> big hurry so I probably want to just rip out all support for mapping
>> diffserv to QoS on the ath10k transmit side.  Does anyone have any
>> suggestions on where to look for my hack-and-slash operation?
>
> Tis easy to hack and slash it out of cfg80211_classify8021d in
> net/wireless/util.c
>
> Recently someone added a direct mapping from vlan priorities to wifi
> queues here - which is stupid, as the VO queue does not behave
> anything like the equivalent vlan priority queue does. I'd like to see
> that removed or mapped properly, also.

Yup, replaced this function with 'return 0' and my problems are gone.
(Well okay, this problem is gone, and you're probably not interested
in my personal problems. :))

Obviously just completely disabling QoS multiple queues is not the
right long term fix, but as Dave pointed out, it wasn't clear the code
there was exactly what we want anyway, so I won't feel too guilty
about it.  Meanwhile, hopefully someone with deeper knowledge than me
can figure out what's with the IVs.

Thanks!

Avery

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Dave Taht]

On Wed, May 14, 2014 at 11:12 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Wed, May 14, 2014 at 3:26 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Wed, May 14, 2014 at 12:07 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>>> but meanwhile, I think I'm in a
>>> big hurry so I probably want to just rip out all support for mapping
>>> diffserv to QoS on the ath10k transmit side.  Does anyone have any
>>> suggestions on where to look for my hack-and-slash operation?
>>
>> Tis easy to hack and slash it out of cfg80211_classify8021d in
>> net/wireless/util.c
>>
>> Recently someone added a direct mapping from vlan priorities to wifi
>> queues here - which is stupid, as the VO queue does not behave
>> anything like the equivalent vlan priority queue does. I'd like to see
>> that removed or mapped properly, also.
>
> Yup, replaced this function with 'return 0' and my problems are gone.
> (Well okay, this problem is gone, and you're probably not interested
> in my personal problems. :))

The wifi alliance won't certify something that doesn't do 802.11e
"correctly". Not that I care.

A side effect of ripping this out is that some prioritized traffic, notably
multicast & dhcp (marked with CS6) now gets head of line blocked
in the BE queue until it can be delivered.

A possible problem with this "fix" is that the underlying hw queues are
still having some sort of problem in the driver, and doing a
multicast flood (say, mdns-scan or uftp) while doing other traffic might
trigger it again.

The linux 4 exposed queue problem should long ago have been
bumped to directly include a 5th queue observable and manageable
for multicast events.

> Obviously just completely disabling QoS multiple queues is not the
> right long term fix, but as Dave pointed out, it wasn't clear the code
> there was exactly what we want anyway, so I won't feel too guilty
> about it.

Well, after spending 4+ months trying to get to the bottom of this
myself...

... I just also ripped 802.11e out of cerowrt in this same file for more
testing.

802.11e in a packet aggregated world is both conceptually and
operationally broken, when the real resource limit is txops,
and cramming as many packets (sanely) into a txop is more the
right thing.

I've been showing that 4 hardware queues induces more latency
than 1 for a long time now, I am inspired (before I commit to
ripping out 802.11e entirely) to produce some graphs showing how
horrible flooding those queues is, compared to having one queue.

Now, getting away from doing QoS at this level opens up an opportunity
in that there are still 4 independently programmable tx queues (from
hostapd) and they could be used instead as a poor mans "mu-mimo" to
drive multiple stations more sanely.

> Meanwhile, hopefully someone with deeper knowledge than me
> can figure out what's with the IVs.

+10

> Thanks!
>
> Avery



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Adrian Chadd]

Hi,

When I get the firmware source from QCA and Ben, I'll dig into the IV
shit in more detail.

As for the 11e and txop stuff - absolutely. I've been wanting to add
some txop-time aware clue to the freebsd packet scheduler and rate
control code. aggregation and 11e work great, but you can't just
simply keep packing max-length aggregates in when your bulk-download
station is on MCS0. That's just plain dumb. Linux and FreeBSD don't
take that into account.

We can use AMPDU and AMSDU to better interface with the timing and
transmission constraints of doing things over the open air. We're just
not.

Now that I'm not working at QCA I do plan on adding that stuff to
FreeBSD's atheros driver and hopefully work with the ath9k team to get
that into the rate control code there too. As for the firmware, I'm
happy to also add that to firmware - but not until the firmware is
open sourced.


-a


On 16 May 2014 13:01, Dave Taht <dave.taht@gmail.com> wrote:
> On Wed, May 14, 2014 at 11:12 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>> On Wed, May 14, 2014 at 3:26 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>> On Wed, May 14, 2014 at 12:07 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>>>> but meanwhile, I think I'm in a
>>>> big hurry so I probably want to just rip out all support for mapping
>>>> diffserv to QoS on the ath10k transmit side.  Does anyone have any
>>>> suggestions on where to look for my hack-and-slash operation?
>>>
>>> Tis easy to hack and slash it out of cfg80211_classify8021d in
>>> net/wireless/util.c
>>>
>>> Recently someone added a direct mapping from vlan priorities to wifi
>>> queues here - which is stupid, as the VO queue does not behave
>>> anything like the equivalent vlan priority queue does. I'd like to see
>>> that removed or mapped properly, also.
>>
>> Yup, replaced this function with 'return 0' and my problems are gone.
>> (Well okay, this problem is gone, and you're probably not interested
>> in my personal problems. :))
>
> The wifi alliance won't certify something that doesn't do 802.11e
> "correctly". Not that I care.
>
> A side effect of ripping this out is that some prioritized traffic, notably
> multicast & dhcp (marked with CS6) now gets head of line blocked
> in the BE queue until it can be delivered.
>
> A possible problem with this "fix" is that the underlying hw queues are
> still having some sort of problem in the driver, and doing a
> multicast flood (say, mdns-scan or uftp) while doing other traffic might
> trigger it again.
>
> The linux 4 exposed queue problem should long ago have been
> bumped to directly include a 5th queue observable and manageable
> for multicast events.
>
>> Obviously just completely disabling QoS multiple queues is not the
>> right long term fix, but as Dave pointed out, it wasn't clear the code
>> there was exactly what we want anyway, so I won't feel too guilty
>> about it.
>
> Well, after spending 4+ months trying to get to the bottom of this
> myself...
>
> ... I just also ripped 802.11e out of cerowrt in this same file for more
> testing.
>
> 802.11e in a packet aggregated world is both conceptually and
> operationally broken, when the real resource limit is txops,
> and cramming as many packets (sanely) into a txop is more the
> right thing.
>
> I've been showing that 4 hardware queues induces more latency
> than 1 for a long time now, I am inspired (before I commit to
> ripping out 802.11e entirely) to produce some graphs showing how
> horrible flooding those queues is, compared to having one queue.
>
> Now, getting away from doing QoS at this level opens up an opportunity
> in that there are still 4 independently programmable tx queues (from
> hostapd) and they could be used instead as a poor mans "mu-mimo" to
> drive multiple stations more sanely.
>
>> Meanwhile, hopefully someone with deeper knowledge than me
>> can figure out what's with the IVs.
>
> +10
>
>> Thanks!
>>
>> Avery
>
>
>
> --
> Dave Täht
>
> NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Adrian Chadd]

Hi,

Bumping this up.

Please don't let us forget. once I get access to the firmware source
from Ben and QCA I'll do a hell of a lot more digging.

Thanks,



-a


On 14 May 2014 23:12, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Wed, May 14, 2014 at 3:26 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Wed, May 14, 2014 at 12:07 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>>> but meanwhile, I think I'm in a
>>> big hurry so I probably want to just rip out all support for mapping
>>> diffserv to QoS on the ath10k transmit side.  Does anyone have any
>>> suggestions on where to look for my hack-and-slash operation?
>>
>> Tis easy to hack and slash it out of cfg80211_classify8021d in
>> net/wireless/util.c
>>
>> Recently someone added a direct mapping from vlan priorities to wifi
>> queues here - which is stupid, as the VO queue does not behave
>> anything like the equivalent vlan priority queue does. I'd like to see
>> that removed or mapped properly, also.
>
> Yup, replaced this function with 'return 0' and my problems are gone.
> (Well okay, this problem is gone, and you're probably not interested
> in my personal problems. :))
>
> Obviously just completely disabling QoS multiple queues is not the
> right long term fix, but as Dave pointed out, it wasn't clear the code
> there was exactly what we want anyway, so I won't feel too guilty
> about it.  Meanwhile, hopefully someone with deeper knowledge than me
> can figure out what's with the IVs.
>
> Thanks!
>
> Avery
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Dave Taht]

from what I see from reports on the ath9k list, and openwrt
I tend to think this problem is endemic, and not driver specific.

On Mon, May 19, 2014 at 2:04 PM, Adrian Chadd <adrian@freebsd.org> wrote:
> Hi,
>
> Bumping this up.
>
> Please don't let us forget. once I get access to the firmware source
> from Ben and QCA I'll do a hell of a lot more digging.
>
> Thanks,
>
>
>
> -a
>
>
> On 14 May 2014 23:12, Avery Pennarun <apenwarr@gmail.com> wrote:
>> On Wed, May 14, 2014 at 3:26 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>> On Wed, May 14, 2014 at 12:07 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>>>> but meanwhile, I think I'm in a
>>>> big hurry so I probably want to just rip out all support for mapping
>>>> diffserv to QoS on the ath10k transmit side.  Does anyone have any
>>>> suggestions on where to look for my hack-and-slash operation?
>>>
>>> Tis easy to hack and slash it out of cfg80211_classify8021d in
>>> net/wireless/util.c
>>>
>>> Recently someone added a direct mapping from vlan priorities to wifi
>>> queues here - which is stupid, as the VO queue does not behave
>>> anything like the equivalent vlan priority queue does. I'd like to see
>>> that removed or mapped properly, also.
>>
>> Yup, replaced this function with 'return 0' and my problems are gone.
>> (Well okay, this problem is gone, and you're probably not interested
>> in my personal problems. :))
>>
>> Obviously just completely disabling QoS multiple queues is not the
>> right long term fix, but as Dave pointed out, it wasn't clear the code
>> there was exactly what we want anyway, so I won't feel too guilty
>> about it.  Meanwhile, hopefully someone with deeper knowledge than me
>> can figure out what's with the IVs.
>>
>> Thanks!
>>
>> Avery
>>
>> _______________________________________________
>> ath10k mailing list
>> ath10k@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/ath10k



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Adrian Chadd]

On 19 May 2014 14:07, Dave Taht <dave.taht@gmail.com> wrote:
> from what I see from reports on the ath9k list, and openwrt
> I tend to think this problem is endemic, and not driver specific.

I thought the IV for ath10k was handled in the firmware, rather than
setup from the host. I'll have to go digging.



-a

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Kalle Valo]

Hi Avery,

Avery Pennarun <apenwarr@gmail.com> writes:

> On Sun, May 11, 2014 at 11:09 PM, Ben Greear <greearb@candelatech.com> wrote:
>> Can you reproduce with any other type of station
>> device?
>
> I'll try.  It's a little tricky since it seems to involve running
> Chrome, etc, rather than my usual command-line tests for which I have
> a wider variety of clients.

did you have a chance to check with other clients? It would be good to
know if this is macbook specific interoparibility problem or something
else.

-- 
Kalle Valo

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unicast packets stop being transmitted to a particular station, under load, when WPA2 is enabled

[Ben Greear]

On 05/27/2014 02:53 AM, Kalle Valo wrote:
> Hi Avery,
>
> Avery Pennarun <apenwarr@gmail.com> writes:
>
>> On Sun, May 11, 2014 at 11:09 PM, Ben Greear <greearb@candelatech.com> wrote:
>>> Can you reproduce with any other type of station
>>> device?
>>
>> I'll try.  It's a little tricky since it seems to involve running
>> Chrome, etc, rather than my usual command-line tests for which I have
>> a wider variety of clients.
>
> did you have a chance to check with other clients? It would be good to
> know if this is macbook specific interoparibility problem or something
> else.
>

So, to resurrect an ancient thread....

I have a user (who wants to buy and sell lots and lots of QCA chipset NICS!)
of my 10.1 firmware hitting this problem.  They have a way to reproduce
it readily, though for one reason or another, I cannot reproduce it locally using what
should be an identical setup.

Their problem cannot be reproduced in a stock 10.2 firmware (firmware-3.bin, I think) on a newer kernel,
while that newer kernel does hit the problem with my 10.1 firmware.  So, it is
definitely something in the firmware.  Their system cannot use 10.2 firmware
at the moment because it is based on an older kernel.

So, I am curious if someone would like to give me a hint as to what was the actual
firmware fix for this issue.  I don't need code snippets since that stuff is
private...but just a method name and/or file or something and I should be able
to track it down and backport just that fix.  I've tried comparing obvious code
spots in 10.1 and 10.2 firmware, but nothing obvious jumps out.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k