[ath9k-devel] GTK/PTK problem - key.c magic-bitshift

[ath9k-devel] GTK/PTK problem - key.c magic-bitshift

[Antonio Quartulli]

Hello list,

I am debugging a strange problem which symptom is that the PTK of some station
is messed up right after hostapd has configured a new GTK. This happens both in
WPA2 only and WPA/WPA2 mixed mode. I have no real clue of what is going on and
so, after having given a look at the hostapd code, I am now checking the driver
(Ath9k).

While digging I found a piece of code in key.c which I can't understand.
Here is the snippet:

111                 macLo = get_unaligned_le32(mac);
112                 macHi = get_unaligned_le16(mac + 4);
113                 macLo >>= 1;
114                 macLo |= (macHi & 1) << 31;
115                 macHi >>= 1;

At drivers/net/wireless/ath/key.c:113 there is a right-bitshift operation
executed on macLo. Since macLo is a 4byte variable containing 4 out of 6
bytes of the MAC address passed as argument, it looks to me like the bitshift
is throwing away one bit that we will never recover.

I don't want to claim that this is necessarily connected to the problem I am
experiencing, but I'd like to get a better understanding of this part in order
to continue my research.

However the driver I am using is not the latest one, so if somebody knows
whether this GTK/PTK problem is known and fixed I would be glad if you could
tell me :-)


Thanks a lot for your time.

Regards,


-- 
Antonio Quartulli
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131117/ec3e92b1/attachment.pgp 

[ath9k-devel] GTK/PTK problem - key.c magic-bitshift

[Adrian Chadd]

oh goodie! goodie!

You're likely hitting the "keycache is corrupted!" bug.

The keycache contents is throwing away a bit you don't actually need.
Look at which bit that actually is in the OUI specification. It's
something very specific.

There's a chip bug that's been (as far as I could tell whilst I was at
QCA) around forever.

You need to add code to check for decryption failures and if so,
trigger a periodic job to read all the keycache slots and validate
them against what cached values you keep in userland.

If they don't match, log an error and reprogram the slot.

(I haven't yet done it for FreeBSD but it's on my "when I'm bored one
weekend soon" project lits.)

HTH,



-adrian



On 17 November 2013 03:08, Antonio Quartulli <antonio@open-mesh.com> wrote:
> Hello list,
>
> I am debugging a strange problem which symptom is that the PTK of some station
> is messed up right after hostapd has configured a new GTK. This happens both in
> WPA2 only and WPA/WPA2 mixed mode. I have no real clue of what is going on and
> so, after having given a look at the hostapd code, I am now checking the driver
> (Ath9k).
>
> While digging I found a piece of code in key.c which I can't understand.
> Here is the snippet:
>
> 111                 macLo = get_unaligned_le32(mac);
> 112                 macHi = get_unaligned_le16(mac + 4);
> 113                 macLo >>= 1;
> 114                 macLo |= (macHi & 1) << 31;
> 115                 macHi >>= 1;
>
> At drivers/net/wireless/ath/key.c:113 there is a right-bitshift operation
> executed on macLo. Since macLo is a 4byte variable containing 4 out of 6
> bytes of the MAC address passed as argument, it looks to me like the bitshift
> is throwing away one bit that we will never recover.
>
> I don't want to claim that this is necessarily connected to the problem I am
> experiencing, but I'd like to get a better understanding of this part in order
> to continue my research.
>
> However the driver I am using is not the latest one, so if somebody knows
> whether this GTK/PTK problem is known and fixed I would be glad if you could
> tell me :-)
>
>
> Thanks a lot for your time.
>
> Regards,
>
>
> --
> Antonio Quartulli
>
> _______________________________________________
> ath9k-devel mailing list
> ath9k-devel at lists.ath9k.org
> https://lists.ath9k.org/mailman/listinfo/ath9k-devel
>

[ath9k-devel] GTK/PTK problem - key.c magic-bitshift

[Antonio Quartulli]

Hello!

On Mon, Nov 18, 2013 at 01:29:14PM -0800, Adrian Chadd wrote:
> oh goodie! goodie!
> 
> You're likely hitting the "keycache is corrupted!" bug.

Yeah I had this feeling already and I have been discussing with Jouni in
the meanwhile (who confirmed my feeling) :-)

> 
> The keycache contents is throwing away a bit you don't actually need.
> Look at which bit that actually is in the OUI specification. It's
> something very specific.
> 

Yeah, I realised that after talking to Felix.

> There's a chip bug that's been (as far as I could tell whilst I was at
> QCA) around forever.

Oh. This is a really bad news..I hoped it was a bug in the driver....
However now I know there is no bug to look for in the code! :)


> 
> You need to add code to check for decryption failures and if so,
> trigger a periodic job to read all the keycache slots and validate
> them against what cached values you keep in userland.
> 
> If they don't match, log an error and reprogram the slot.
> 
> (I haven't yet done it for FreeBSD but it's on my "when I'm bored one
> weekend soon" project lits.)

Sounds ugly..but I understood what you mean.
But why a periodic job? Isn't it enough to run the task once when there is a
decryption problem ?

Thanks a lot for your feedback!

> 
> HTH,
> 
> 
> 
> -adrian
> 
> 
> 
> On 17 November 2013 03:08, Antonio Quartulli <antonio@open-mesh.com> wrote:
> > Hello list,
> >
> > I am debugging a strange problem which symptom is that the PTK of some station
> > is messed up right after hostapd has configured a new GTK. This happens both in
> > WPA2 only and WPA/WPA2 mixed mode. I have no real clue of what is going on and
> > so, after having given a look at the hostapd code, I am now checking the driver
> > (Ath9k).
> >
> > While digging I found a piece of code in key.c which I can't understand.
> > Here is the snippet:
> >
> > 111                 macLo = get_unaligned_le32(mac);
> > 112                 macHi = get_unaligned_le16(mac + 4);
> > 113                 macLo >>= 1;
> > 114                 macLo |= (macHi & 1) << 31;
> > 115                 macHi >>= 1;
> >
> > At drivers/net/wireless/ath/key.c:113 there is a right-bitshift operation
> > executed on macLo. Since macLo is a 4byte variable containing 4 out of 6
> > bytes of the MAC address passed as argument, it looks to me like the bitshift
> > is throwing away one bit that we will never recover.
> >
> > I don't want to claim that this is necessarily connected to the problem I am
> > experiencing, but I'd like to get a better understanding of this part in order
> > to continue my research.
> >
> > However the driver I am using is not the latest one, so if somebody knows
> > whether this GTK/PTK problem is known and fixed I would be glad if you could
> > tell me :-)
> >
> >
> > Thanks a lot for your time.
> >
> > Regards,
> >
> >
> > --
> > Antonio Quartulli
> >
> > _______________________________________________
> > ath9k-devel mailing list
> > ath9k-devel at lists.ath9k.org
> > https://lists.ath9k.org/mailman/listinfo/ath9k-devel
> >

-- 
Antonio Quartulli
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131118/f4cce8f6/attachment.pgp 

[ath9k-devel] GTK/PTK problem - key.c magic-bitshift

[Antonio Quartulli]

On Mon, Nov 18, 2013 at 01:29:14PM -0800, Adrian Chadd wrote:
> There's a chip bug that's been (as far as I could tell whilst I was at
> QCA) around forever.
> 

Do you know if this bug is triggered during write operations to other slots
only (in my case it seems the cache was getting corrupted after a GTK upload)
or if this can happen at any time?

In the first case I could check the cache for possible corruptions only when
ath_key_config() is invoked (I could compare the cache state before the upload
with what I have after it - as Jouni suggested) and call the "resync task" only
in that case.


Cheers,

-- 
Antonio Quartulli
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131118/85d9feb7/attachment.pgp 

[ath9k-devel] GTK/PTK problem - key.c magic-bitshift

[Adrian Chadd]

I don't know exactly what triggers it.

I suggested a periodic event so you only do it say, max every second
or two, but you trigger it when you get more than a bunch of decrypt
errors. That way if you keep getting decrypt errors, you only get it
scheduled every second or two, rather than on each decrypt error.


-a


On 18 November 2013 13:51, Antonio Quartulli <antonio@open-mesh.com> wrote:
> On Mon, Nov 18, 2013 at 01:29:14PM -0800, Adrian Chadd wrote:
>> There's a chip bug that's been (as far as I could tell whilst I was at
>> QCA) around forever.
>>
>
> Do you know if this bug is triggered during write operations to other slots
> only (in my case it seems the cache was getting corrupted after a GTK upload)
> or if this can happen at any time?
>
> In the first case I could check the cache for possible corruptions only when
> ath_key_config() is invoked (I could compare the cache state before the upload
> with what I have after it - as Jouni suggested) and call the "resync task" only
> in that case.
>
>
> Cheers,
>
> --
> Antonio Quartulli

[ath9k-devel] GTK/PTK problem - key.c magic-bitshift

[Antonio Quartulli]

On Wed, Nov 20, 2013 at 12:41:07AM -0800, Adrian Chadd wrote:
> I don't know exactly what triggers it.

Ok

> 
> I suggested a periodic event so you only do it say, max every second
> or two, but you trigger it when you get more than a bunch of decrypt
> errors. That way if you keep getting decrypt errors, you only get it
> scheduled every second or two, rather than on each decrypt error.
> 

Yeah, sounds good. Right now I am simply trying to "refresh" the key cache right
after a key is uploaded and if I observe decryption error I disable the HW
acceleration for that STA (waiting for the next GTK rekey to refresh the cache
and re-enable the accel again).

However there is something I don't understand. In the packet dump I obtained
from the network where I observe the bug, I see that after this possible
"cache corruption
event" ARP requests from the STA to the AP are properly decrypted
(only ARP replies going in the other direction are not).

If the cache is really
compromised, how can this happen? I would expect the AP to not be able to
decrypt the requests as well...don't you think so?


Cheers,


-- 
Antonio Quartulli
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131122/51140b96/attachment.pgp 

[ath9k-devel] GTK/PTK problem - key.c magic-bitshift

[Adrian Chadd]

Hm? There are separate transmit and receive keys negotiated.



-adrian


On 22 November 2013 01:18, Antonio Quartulli <antonio@open-mesh.com> wrote:
> On Wed, Nov 20, 2013 at 12:41:07AM -0800, Adrian Chadd wrote:
>> I don't know exactly what triggers it.
>
> Ok
>
>>
>> I suggested a periodic event so you only do it say, max every second
>> or two, but you trigger it when you get more than a bunch of decrypt
>> errors. That way if you keep getting decrypt errors, you only get it
>> scheduled every second or two, rather than on each decrypt error.
>>
>
> Yeah, sounds good. Right now I am simply trying to "refresh" the key cache right
> after a key is uploaded and if I observe decryption error I disable the HW
> acceleration for that STA (waiting for the next GTK rekey to refresh the cache
> and re-enable the accel again).
>
> However there is something I don't understand. In the packet dump I obtained
> from the network where I observe the bug, I see that after this possible
> "cache corruption
> event" ARP requests from the STA to the AP are properly decrypted
> (only ARP replies going in the other direction are not).
>
> If the cache is really
> compromised, how can this happen? I would expect the AP to not be able to
> decrypt the requests as well...don't you think so?
>
>
> Cheers,
>
>
> --
> Antonio Quartulli

[ath9k-devel] GTK/PTK problem - key.c magic-bitshift

[Antonio Quartulli]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 22/11/13 10:42, Adrian Chadd wrote:


(sorry for re-sending but my mail server did not want to deliver to all..)

Adrian, sorry for the delay.

You are right. With TKIP we have different keys..but at this point how
can I detect a PTK-TX corruption? It may be the case that received
frames are decrypted properly (PTK-RX is ok)...but still..the PTX-TX
may have been corrupted. Right?


When should I trigger the periodic worker in this case?




- -- 
Antonio Quartulli
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBCAAGBQJSlZXPAAoJEADl0hg6qKeOYbYQAKZcAR0yLd7NJcmc30/ekcM5
pfkT3wY7fnkWz9Fq2M7OgEprFX91cz44qrokh+wNikFHZlDg3qIAv3Ti+FbbHiJ+
nTHu/2INGGD1Me27e9Ts6AWLALHcQsumJhzi96uRPEna0gr4oJeHWId84mT5tUDh
rO5yCFhiZiSMmnqvlG9PxIifXJo9QgQpjRy4tSRkDu9dIWrv+1F0+KLwcFGhpLCV
RUR8KpXqFHFMgrMuRaiFO23U1T1yZnllEiA1eQO7urMhlBnEJ0yODHj5VDcivEWe
obFzJYxG/aH+51m0y1KOcWm5C/fyvMUBitU+kwfV5SByRN4Tr50DdBQ/DCI4qR2R
7iJ/gTJcampKCRDTNUiRaUY2KYIxSEumAbgr5sRbFq2udUl8PJbEM/nhmdlMyG8N
HYPN9cgzeANvvvo30fpCyJukWCp8nefhtq3FYSeQpm/16rbNak3vffzgTXcOtAVJ
CHaBHcYweofmzeWtFOPTFEuLkCKA/ZZ0LzP+pv3Zmfe5Whb3LfSEcQRuwfhexb/6
SsG3omRYqn1rDxlV9MAajTv22KhJ26QIy9jp9qQDUOib4r/sleOcuVOc4CLxuaoc
DSAibUDZaMFPWnn7XtSbtDdZtDVc4XZ33xyJ9ad1IyvIh6M8YVgA2QCA2DJ5Zk9R
0uwdqKx4VM7aRgfFXd8h
=NqhI
-----END PGP SIGNATURE-----

[ath9k-devel] Key Cache corruption (was: GTK/PTK problem - key.c magic-bitshift)

[Antonio Quartulli]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi list, hi Adrian, hi Sujith (just added to the cc list),

sorry for the silence, but I have been busy testing different
approaches on how to solve this.

On 22/11/13 10:42, Adrian Chadd wrote:


Mh, that is true with TKIP.

This means we have no way to recognize a TX key corruption.
For this reason a periodic worker that unconditionally checks the
cache and possibly refresh it looked like the best solution.

However, after testing this strategy I realised that the worker is not
able to see the cache corruption: what it reads from the memory always
matches what we have in mac80211 (but I am sure we had a corruption).

My opinion is that the corruption is really happening at a low-low
level and so the memory content is not even altered. This may be one
of the reasons why this bug is so difficult to catch.

The only solution I see to this problem is to make the periodic worker
refresh the key cache every second, no matter if a corruption is found
or not.

However it does not sound like a good solution..any thought?



Another experiment I have done was to refresh the whole key cache each
time a new key is pushed into ath9k. I did this because I observed
that the corruption was triggered upon GTK upload (on re-keying).
This fix seemed to make things better.

So another plausible strategy could be a mix of the above solutions:

1) refresh the entire cache on each set_key(cmd=SET_KEY) call
2) upon detection of a number of RX decrypt errors schedule a worker
that refresh the entire cache (not the single slot...because that
could be the cause for another corruption...).


Still we have the problem of the TX key corruption with TKIP. If this
happens we can't detect it and we need to wait for the next set_key()
before the key can be properly restored.....


Any better idea?


Regards,


- -- 
Antonio Quartulli
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBCAAGBQJSnyzgAAoJEEKTMo6mOh1VxggP/jzp+UB8lPiFItwMPXwob8ik
qNd1WcQhcShft8MrmK9kfjzp91t/uBby/QVsqwxKu8sVIpuQJaGI4ikOtIZ6/8b6
FGkn+JouKY+gmEsT6U4Hfq6v4Ri/xKkUt5JfPN7moQr3OszitbnDGlO3gr9qY3Vv
YjDGkZPlAmGCpnLUY1nHiBrEhj0WX2GrFXQC06wskk5UYEyvE5PDy9LUt1ASQkxe
qa6rBWaanVT0YpLhoHXNldSu6aF1r1HXaG+lKnBMgJlBcLSBta7oxzYjxj6oPtLQ
J+QYkdkkwrEGqVR8JL+54tAYkJ0TZvw+DKuJ6PcWE/Z74LiiVczrEZ0yf5oQm8Ki
EgQUay8X3gOT9ZFIrnYUK3i2PdA8UsBv/xmIOhlRHqx2Au7bcekM+9c7tIZjxz0V
QVspKLaRefvP4thcB0Sni56D05vHdV69wFPui+d+Y2Z/i72NjP4x2AXylfOmVNsV
9tI135f8see5wQOQN7WJFuM67h7GlyeSIeAZWFjwhdDBWSmWDtal9w9930MZgAVe
Oh1oLCoboi3gywiN4bpL59Thvt164prDQA6aA9uu90Ytbzxept0ZUITd+z5XQVbb
BLBWLSrZme0BDCqkJw55Z/ypl4TlXIIdWK935YDNMPRl1RGo86KUDSoCbFZNjc4J
w8WBV7CS7jmU+E2DjLzy
=gnO4
-----END PGP SIGNATURE-----

[ath9k-devel] Key Cache corruption (was: GTK/PTK problem - key.c magic-bitshift)

[Adrian Chadd]

Hi,

I'll see if I can organise a chat with the MAC guys this week at
Atheros and ask them about this. Stay tuned...


-a

On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Hi list, hi Adrian, hi Sujith (just added to the cc list),
>
> sorry for the silence, but I have been busy testing different
> approaches on how to solve this.
>
> On 22/11/13 10:42, Adrian Chadd wrote:
>
>
> Mh, that is true with TKIP.
>
> This means we have no way to recognize a TX key corruption.
> For this reason a periodic worker that unconditionally checks the
> cache and possibly refresh it looked like the best solution.
>
> However, after testing this strategy I realised that the worker is not
> able to see the cache corruption: what it reads from the memory always
> matches what we have in mac80211 (but I am sure we had a corruption).
>
> My opinion is that the corruption is really happening at a low-low
> level and so the memory content is not even altered. This may be one
> of the reasons why this bug is so difficult to catch.
>
> The only solution I see to this problem is to make the periodic worker
> refresh the key cache every second, no matter if a corruption is found
> or not.
>
> However it does not sound like a good solution..any thought?
>
>
>
> Another experiment I have done was to refresh the whole key cache each
> time a new key is pushed into ath9k. I did this because I observed
> that the corruption was triggered upon GTK upload (on re-keying).
> This fix seemed to make things better.
>
> So another plausible strategy could be a mix of the above solutions:
>
> 1) refresh the entire cache on each set_key(cmd=SET_KEY) call
> 2) upon detection of a number of RX decrypt errors schedule a worker
> that refresh the entire cache (not the single slot...because that
> could be the cause for another corruption...).
>
>
> Still we have the problem of the TX key corruption with TKIP. If this
> happens we can't detect it and we need to wait for the next set_key()
> before the key can be properly restored.....
>
>
> Any better idea?
>
>
> Regards,
>
>
> - --
> Antonio Quartulli
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.22 (GNU/Linux)
>
> iQIcBAEBCAAGBQJSnyzgAAoJEEKTMo6mOh1VxggP/jzp+UB8lPiFItwMPXwob8ik
> qNd1WcQhcShft8MrmK9kfjzp91t/uBby/QVsqwxKu8sVIpuQJaGI4ikOtIZ6/8b6
> FGkn+JouKY+gmEsT6U4Hfq6v4Ri/xKkUt5JfPN7moQr3OszitbnDGlO3gr9qY3Vv
> YjDGkZPlAmGCpnLUY1nHiBrEhj0WX2GrFXQC06wskk5UYEyvE5PDy9LUt1ASQkxe
> qa6rBWaanVT0YpLhoHXNldSu6aF1r1HXaG+lKnBMgJlBcLSBta7oxzYjxj6oPtLQ
> J+QYkdkkwrEGqVR8JL+54tAYkJ0TZvw+DKuJ6PcWE/Z74LiiVczrEZ0yf5oQm8Ki
> EgQUay8X3gOT9ZFIrnYUK3i2PdA8UsBv/xmIOhlRHqx2Au7bcekM+9c7tIZjxz0V
> QVspKLaRefvP4thcB0Sni56D05vHdV69wFPui+d+Y2Z/i72NjP4x2AXylfOmVNsV
> 9tI135f8see5wQOQN7WJFuM67h7GlyeSIeAZWFjwhdDBWSmWDtal9w9930MZgAVe
> Oh1oLCoboi3gywiN4bpL59Thvt164prDQA6aA9uu90Ytbzxept0ZUITd+z5XQVbb
> BLBWLSrZme0BDCqkJw55Z/ypl4TlXIIdWK935YDNMPRl1RGo86KUDSoCbFZNjc4J
> w8WBV7CS7jmU+E2DjLzy
> =gnO4
> -----END PGP SIGNATURE-----

[ath9k-devel] Key Cache corruption

[Antonio Quartulli]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hoi Adrian,

I don't want to push (I am just polling here ;-)), but have you been
able to squeeze any information out of the MAC guys?

Cheers,

On 04/12/13 18:08, Adrian Chadd wrote:
> Hi,
> 
> I'll see if I can organise a chat with the MAC guys this week at 
> Atheros and ask them about this. Stay tuned...
> 
> 
> -a
> 
> On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com>
> wrote: Hi list, hi Adrian, hi Sujith (just added to the cc list),
> 
> sorry for the silence, but I have been busy testing different 
> approaches on how to solve this.
> 
> On 22/11/13 10:42, Adrian Chadd wrote:
> 
> 
> Mh, that is true with TKIP.
> 
> This means we have no way to recognize a TX key corruption. For
> this reason a periodic worker that unconditionally checks the cache
> and possibly refresh it looked like the best solution.
> 
> However, after testing this strategy I realised that the worker is
> not able to see the cache corruption: what it reads from the memory
> always matches what we have in mac80211 (but I am sure we had a
> corruption).
> 
> My opinion is that the corruption is really happening at a low-low 
> level and so the memory content is not even altered. This may be
> one of the reasons why this bug is so difficult to catch.
> 
> The only solution I see to this problem is to make the periodic
> worker refresh the key cache every second, no matter if a
> corruption is found or not.
> 
> However it does not sound like a good solution..any thought?
> 
> 
> 
> Another experiment I have done was to refresh the whole key cache
> each time a new key is pushed into ath9k. I did this because I
> observed that the corruption was triggered upon GTK upload (on
> re-keying). This fix seemed to make things better.
> 
> So another plausible strategy could be a mix of the above
> solutions:
> 
> 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2)
> upon detection of a number of RX decrypt errors schedule a worker 
> that refresh the entire cache (not the single slot...because that 
> could be the cause for another corruption...).
> 
> 
> Still we have the problem of the TX key corruption with TKIP. If
> this happens we can't detect it and we need to wait for the next
> set_key() before the key can be properly restored.....
> 
> 
> Any better idea?
> 
> 
> Regards,
> 
> 

- -- 
Antonio Quartulli
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBCAAGBQJSpZitAAoJEEKTMo6mOh1VSLYQAIPCr6j6YJd0WiVeO23xGS75
hO7kfSm2y+zMJvl8WYFY5mKrUUssXYfp9YvDOUZV1KSK3cxHt93dzlEnvyhLCAQc
RpHsbg0DPcuxIKzH4hQVl6mYyzAqMm2/icRXome+6H2Fz8fcGNy+4FGwiAhvAZdx
3WIj773EwE8m2/M4Kpa5HHIrcmJ305EFopAXZMEP9N37V1ANTGkJDhWI9gU3sqC/
XQmeNsTY3m0LG/CTyhoT0RvkW284pGxQkVPd29QDQFYKAQv0DcrPmohcprPcfQmA
Bf1MIKTV734RQ+bU/TPw5lpCpUnGPhILwiijsnwNWGKSBEgjydS7LXp9/F+QMHHi
inUOY9zVJHvETCJ+4cI/xKOYU1UCAsqnKldBmY63BrJr/MhJ1JiPsKVbd6rIyGmP
KuQ9J4y+yP+y6TOYFoyFNlLFWRS+tbY2XPm51nV7aaYvmKZi+GgA8i1per3xmNAl
28Qwqnerjl+gyF0+IpU+61uMkx9wg3jIs2PtW9Ek7tSLOPb5vjyCmRvw8N+Djyax
fr57lMwqV/yKZJlv5NA6i04WoocHn3IM2WaS7BeO/7h4sDKY2BOprpc4YxFw9rPF
ZwhDNrVpMOWJzpnBwFyNNjXofQ6iVdtDmzHuUG98Y+WvEbMMV4Rq5sMu7rq03Mz7
w2IPW9twgtvcXUc2V7R3
=GZj5
-----END PGP SIGNATURE-----

[ath9k-devel] Key Cache corruption

[Adrian Chadd]

Hi!

I've emailed them - they're tossing it around internally.


-a

On 9 December 2013 02:17, Antonio Quartulli <antonio@open-mesh.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Hoi Adrian,
>
> I don't want to push (I am just polling here ;-)), but have you been
> able to squeeze any information out of the MAC guys?
>
> Cheers,
>
> On 04/12/13 18:08, Adrian Chadd wrote:
>> Hi,
>>
>> I'll see if I can organise a chat with the MAC guys this week at
>> Atheros and ask them about this. Stay tuned...
>>
>>
>> -a
>>
>> On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com>
>> wrote: Hi list, hi Adrian, hi Sujith (just added to the cc list),
>>
>> sorry for the silence, but I have been busy testing different
>> approaches on how to solve this.
>>
>> On 22/11/13 10:42, Adrian Chadd wrote:
>>
>>
>> Mh, that is true with TKIP.
>>
>> This means we have no way to recognize a TX key corruption. For
>> this reason a periodic worker that unconditionally checks the cache
>> and possibly refresh it looked like the best solution.
>>
>> However, after testing this strategy I realised that the worker is
>> not able to see the cache corruption: what it reads from the memory
>> always matches what we have in mac80211 (but I am sure we had a
>> corruption).
>>
>> My opinion is that the corruption is really happening at a low-low
>> level and so the memory content is not even altered. This may be
>> one of the reasons why this bug is so difficult to catch.
>>
>> The only solution I see to this problem is to make the periodic
>> worker refresh the key cache every second, no matter if a
>> corruption is found or not.
>>
>> However it does not sound like a good solution..any thought?
>>
>>
>>
>> Another experiment I have done was to refresh the whole key cache
>> each time a new key is pushed into ath9k. I did this because I
>> observed that the corruption was triggered upon GTK upload (on
>> re-keying). This fix seemed to make things better.
>>
>> So another plausible strategy could be a mix of the above
>> solutions:
>>
>> 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2)
>> upon detection of a number of RX decrypt errors schedule a worker
>> that refresh the entire cache (not the single slot...because that
>> could be the cause for another corruption...).
>>
>>
>> Still we have the problem of the TX key corruption with TKIP. If
>> this happens we can't detect it and we need to wait for the next
>> set_key() before the key can be properly restored.....
>>
>>
>> Any better idea?
>>
>>
>> Regards,
>>
>>
>
> - --
> Antonio Quartulli
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.22 (GNU/Linux)
>
> iQIcBAEBCAAGBQJSpZitAAoJEEKTMo6mOh1VSLYQAIPCr6j6YJd0WiVeO23xGS75
> hO7kfSm2y+zMJvl8WYFY5mKrUUssXYfp9YvDOUZV1KSK3cxHt93dzlEnvyhLCAQc
> RpHsbg0DPcuxIKzH4hQVl6mYyzAqMm2/icRXome+6H2Fz8fcGNy+4FGwiAhvAZdx
> 3WIj773EwE8m2/M4Kpa5HHIrcmJ305EFopAXZMEP9N37V1ANTGkJDhWI9gU3sqC/
> XQmeNsTY3m0LG/CTyhoT0RvkW284pGxQkVPd29QDQFYKAQv0DcrPmohcprPcfQmA
> Bf1MIKTV734RQ+bU/TPw5lpCpUnGPhILwiijsnwNWGKSBEgjydS7LXp9/F+QMHHi
> inUOY9zVJHvETCJ+4cI/xKOYU1UCAsqnKldBmY63BrJr/MhJ1JiPsKVbd6rIyGmP
> KuQ9J4y+yP+y6TOYFoyFNlLFWRS+tbY2XPm51nV7aaYvmKZi+GgA8i1per3xmNAl
> 28Qwqnerjl+gyF0+IpU+61uMkx9wg3jIs2PtW9Ek7tSLOPb5vjyCmRvw8N+Djyax
> fr57lMwqV/yKZJlv5NA6i04WoocHn3IM2WaS7BeO/7h4sDKY2BOprpc4YxFw9rPF
> ZwhDNrVpMOWJzpnBwFyNNjXofQ6iVdtDmzHuUG98Y+WvEbMMV4Rq5sMu7rq03Mz7
> w2IPW9twgtvcXUc2V7R3
> =GZj5
> -----END PGP SIGNATURE-----

[ath9k-devel] Key Cache corruption

[Antonio Quartulli]

Hi Adrian!

but as far as you know, isn't the internal Atheros driver applying any
workaround or strange hack to circumvent this problem?
Or simply there is no "clear" workaround at all? (maybe this is what you
expect to be told from the MAC guys?)



Cheers,


On 09/12/13 20:38, Adrian Chadd wrote:
> Hi!
> 
> I've emailed them - they're tossing it around internally.
> 
> 
> -a
> 
> On 9 December 2013 02:17, Antonio Quartulli <antonio@open-mesh.com> wrote:
> Hoi Adrian,
> 
> I don't want to push (I am just polling here ;-)), but have you been
> able to squeeze any information out of the MAC guys?
> 
> Cheers,
> 
> On 04/12/13 18:08, Adrian Chadd wrote:
>>>> Hi,
>>>>
>>>> I'll see if I can organise a chat with the MAC guys this week at
>>>> Atheros and ask them about this. Stay tuned...
>>>>
>>>>
>>>> -a
>>>>
>>>> On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com>
>>>> wrote: Hi list, hi Adrian, hi Sujith (just added to the cc list),
>>>>
>>>> sorry for the silence, but I have been busy testing different
>>>> approaches on how to solve this.
>>>>
>>>> On 22/11/13 10:42, Adrian Chadd wrote:
>>>>
>>>>
>>>> Mh, that is true with TKIP.
>>>>
>>>> This means we have no way to recognize a TX key corruption. For
>>>> this reason a periodic worker that unconditionally checks the cache
>>>> and possibly refresh it looked like the best solution.
>>>>
>>>> However, after testing this strategy I realised that the worker is
>>>> not able to see the cache corruption: what it reads from the memory
>>>> always matches what we have in mac80211 (but I am sure we had a
>>>> corruption).
>>>>
>>>> My opinion is that the corruption is really happening at a low-low
>>>> level and so the memory content is not even altered. This may be
>>>> one of the reasons why this bug is so difficult to catch.
>>>>
>>>> The only solution I see to this problem is to make the periodic
>>>> worker refresh the key cache every second, no matter if a
>>>> corruption is found or not.
>>>>
>>>> However it does not sound like a good solution..any thought?
>>>>
>>>>
>>>>
>>>> Another experiment I have done was to refresh the whole key cache
>>>> each time a new key is pushed into ath9k. I did this because I
>>>> observed that the corruption was triggered upon GTK upload (on
>>>> re-keying). This fix seemed to make things better.
>>>>
>>>> So another plausible strategy could be a mix of the above
>>>> solutions:
>>>>
>>>> 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2)
>>>> upon detection of a number of RX decrypt errors schedule a worker
>>>> that refresh the entire cache (not the single slot...because that
>>>> could be the cause for another corruption...).
>>>>
>>>>
>>>> Still we have the problem of the TX key corruption with TKIP. If
>>>> this happens we can't detect it and we need to wait for the next
>>>> set_key() before the key can be properly restored.....
>>>>
>>>>
>>>> Any better idea?
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
> 

-- 
Antonio Quartulli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131212/915710c2/attachment.pgp 

[ath9k-devel] Key Cache corruption

[Adrian Chadd]

There's a work around, yeah. Much like I've/we've described.



-adrian

On 12 December 2013 01:17, Antonio Quartulli <antonio@open-mesh.com> wrote:
> Hi Adrian!
>
> but as far as you know, isn't the internal Atheros driver applying any
> workaround or strange hack to circumvent this problem?
> Or simply there is no "clear" workaround at all? (maybe this is what you
> expect to be told from the MAC guys?)
>
>
>
> Cheers,
>
>
> On 09/12/13 20:38, Adrian Chadd wrote:
>> Hi!
>>
>> I've emailed them - they're tossing it around internally.
>>
>>
>> -a
>>
>> On 9 December 2013 02:17, Antonio Quartulli <antonio@open-mesh.com> wrote:
>> Hoi Adrian,
>>
>> I don't want to push (I am just polling here ;-)), but have you been
>> able to squeeze any information out of the MAC guys?
>>
>> Cheers,
>>
>> On 04/12/13 18:08, Adrian Chadd wrote:
>>>>> Hi,
>>>>>
>>>>> I'll see if I can organise a chat with the MAC guys this week at
>>>>> Atheros and ask them about this. Stay tuned...
>>>>>
>>>>>
>>>>> -a
>>>>>
>>>>> On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com>
>>>>> wrote: Hi list, hi Adrian, hi Sujith (just added to the cc list),
>>>>>
>>>>> sorry for the silence, but I have been busy testing different
>>>>> approaches on how to solve this.
>>>>>
>>>>> On 22/11/13 10:42, Adrian Chadd wrote:
>>>>>
>>>>>
>>>>> Mh, that is true with TKIP.
>>>>>
>>>>> This means we have no way to recognize a TX key corruption. For
>>>>> this reason a periodic worker that unconditionally checks the cache
>>>>> and possibly refresh it looked like the best solution.
>>>>>
>>>>> However, after testing this strategy I realised that the worker is
>>>>> not able to see the cache corruption: what it reads from the memory
>>>>> always matches what we have in mac80211 (but I am sure we had a
>>>>> corruption).
>>>>>
>>>>> My opinion is that the corruption is really happening at a low-low
>>>>> level and so the memory content is not even altered. This may be
>>>>> one of the reasons why this bug is so difficult to catch.
>>>>>
>>>>> The only solution I see to this problem is to make the periodic
>>>>> worker refresh the key cache every second, no matter if a
>>>>> corruption is found or not.
>>>>>
>>>>> However it does not sound like a good solution..any thought?
>>>>>
>>>>>
>>>>>
>>>>> Another experiment I have done was to refresh the whole key cache
>>>>> each time a new key is pushed into ath9k. I did this because I
>>>>> observed that the corruption was triggered upon GTK upload (on
>>>>> re-keying). This fix seemed to make things better.
>>>>>
>>>>> So another plausible strategy could be a mix of the above
>>>>> solutions:
>>>>>
>>>>> 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2)
>>>>> upon detection of a number of RX decrypt errors schedule a worker
>>>>> that refresh the entire cache (not the single slot...because that
>>>>> could be the cause for another corruption...).
>>>>>
>>>>>
>>>>> Still we have the problem of the TX key corruption with TKIP. If
>>>>> this happens we can't detect it and we need to wait for the next
>>>>> set_key() before the key can be properly restored.....
>>>>>
>>>>>
>>>>> Any better idea?
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>
>
> --
> Antonio Quartulli
>

[ath9k-devel] Key Cache corruption

[Antonio Quartulli]

Oky!

Meanwhile you get a reply from the MAC guys I will try to test again the
"check & fix" approach. It is really strange that I can't recognize any
corruption.

However, the problem of the TX key corruption still remains. It seems we
can't do much about that.


Cheers (and thanks a lot so far!),

On 13/12/13 13:35, Adrian Chadd wrote:
> There's a work around, yeah. Much like I've/we've described.
> 
> 
> 
> -adrian
> 
> On 12 December 2013 01:17, Antonio Quartulli <antonio@open-mesh.com> wrote:
>> Hi Adrian!
>>
>> but as far as you know, isn't the internal Atheros driver applying any
>> workaround or strange hack to circumvent this problem?
>> Or simply there is no "clear" workaround at all? (maybe this is what you
>> expect to be told from the MAC guys?)
>>
>>
>>
>> Cheers,
>>
>>
>> On 09/12/13 20:38, Adrian Chadd wrote:
>>> Hi!
>>>
>>> I've emailed them - they're tossing it around internally.
>>>
>>>
>>> -a
>>>
>>> On 9 December 2013 02:17, Antonio Quartulli <antonio@open-mesh.com> wrote:
>>> Hoi Adrian,
>>>
>>> I don't want to push (I am just polling here ;-)), but have you been
>>> able to squeeze any information out of the MAC guys?
>>>
>>> Cheers,
>>>
>>> On 04/12/13 18:08, Adrian Chadd wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'll see if I can organise a chat with the MAC guys this week at
>>>>>> Atheros and ask them about this. Stay tuned...
>>>>>>
>>>>>>
>>>>>> -a
>>>>>>
>>>>>> On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com>
>>>>>> wrote: Hi list, hi Adrian, hi Sujith (just added to the cc list),
>>>>>>
>>>>>> sorry for the silence, but I have been busy testing different
>>>>>> approaches on how to solve this.
>>>>>>
>>>>>> On 22/11/13 10:42, Adrian Chadd wrote:
>>>>>>
>>>>>>
>>>>>> Mh, that is true with TKIP.
>>>>>>
>>>>>> This means we have no way to recognize a TX key corruption. For
>>>>>> this reason a periodic worker that unconditionally checks the cache
>>>>>> and possibly refresh it looked like the best solution.
>>>>>>
>>>>>> However, after testing this strategy I realised that the worker is
>>>>>> not able to see the cache corruption: what it reads from the memory
>>>>>> always matches what we have in mac80211 (but I am sure we had a
>>>>>> corruption).
>>>>>>
>>>>>> My opinion is that the corruption is really happening at a low-low
>>>>>> level and so the memory content is not even altered. This may be
>>>>>> one of the reasons why this bug is so difficult to catch.
>>>>>>
>>>>>> The only solution I see to this problem is to make the periodic
>>>>>> worker refresh the key cache every second, no matter if a
>>>>>> corruption is found or not.
>>>>>>
>>>>>> However it does not sound like a good solution..any thought?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Another experiment I have done was to refresh the whole key cache
>>>>>> each time a new key is pushed into ath9k. I did this because I
>>>>>> observed that the corruption was triggered upon GTK upload (on
>>>>>> re-keying). This fix seemed to make things better.
>>>>>>
>>>>>> So another plausible strategy could be a mix of the above
>>>>>> solutions:
>>>>>>
>>>>>> 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2)
>>>>>> upon detection of a number of RX decrypt errors schedule a worker
>>>>>> that refresh the entire cache (not the single slot...because that
>>>>>> could be the cause for another corruption...).
>>>>>>
>>>>>>
>>>>>> Still we have the problem of the TX key corruption with TKIP. If
>>>>>> this happens we can't detect it and we need to wait for the next
>>>>>> set_key() before the key can be properly restored.....
>>>>>>
>>>>>>
>>>>>> Any better idea?
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>
>>
>> --
>> Antonio Quartulli
>>


-- 
Antonio Quartulli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131213/e7f2777d/attachment.pgp 

[ath9k-devel] Key Cache corruption

[Antonio Quartulli]

Hi Adrian!

Did 2014 bring any news about this issue? :)

I haven't done much progress myself, but I won't give up!

Cheers,


-- 
Antonio Quartulli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20140107/93f7b32e/attachment.pgp