* [ath9k-devel] GTK/PTK problem - key.c magic-bitshift @ 2013-11-17 11:08 Antonio Quartulli 2013-11-18 21:29 ` Adrian Chadd 0 siblings, 1 reply; 16+ messages in thread From: Antonio Quartulli @ 2013-11-17 11:08 UTC (permalink / raw) To: ath9k-devel Hello list, I am debugging a strange problem which symptom is that the PTK of some station is messed up right after hostapd has configured a new GTK. This happens both in WPA2 only and WPA/WPA2 mixed mode. I have no real clue of what is going on and so, after having given a look at the hostapd code, I am now checking the driver (Ath9k). While digging I found a piece of code in key.c which I can't understand. Here is the snippet: 111 macLo = get_unaligned_le32(mac); 112 macHi = get_unaligned_le16(mac + 4); 113 macLo >>= 1; 114 macLo |= (macHi & 1) << 31; 115 macHi >>= 1; At drivers/net/wireless/ath/key.c:113 there is a right-bitshift operation executed on macLo. Since macLo is a 4byte variable containing 4 out of 6 bytes of the MAC address passed as argument, it looks to me like the bitshift is throwing away one bit that we will never recover. I don't want to claim that this is necessarily connected to the problem I am experiencing, but I'd like to get a better understanding of this part in order to continue my research. However the driver I am using is not the latest one, so if somebody knows whether this GTK/PTK problem is known and fixed I would be glad if you could tell me :-) Thanks a lot for your time. Regards, -- Antonio Quartulli -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: Digital signature Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131117/ec3e92b1/attachment.pgp ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] GTK/PTK problem - key.c magic-bitshift 2013-11-17 11:08 [ath9k-devel] GTK/PTK problem - key.c magic-bitshift Antonio Quartulli @ 2013-11-18 21:29 ` Adrian Chadd 2013-11-18 21:44 ` Antonio Quartulli 2013-11-18 21:51 ` Antonio Quartulli 0 siblings, 2 replies; 16+ messages in thread From: Adrian Chadd @ 2013-11-18 21:29 UTC (permalink / raw) To: ath9k-devel oh goodie! goodie! You're likely hitting the "keycache is corrupted!" bug. The keycache contents is throwing away a bit you don't actually need. Look at which bit that actually is in the OUI specification. It's something very specific. There's a chip bug that's been (as far as I could tell whilst I was at QCA) around forever. You need to add code to check for decryption failures and if so, trigger a periodic job to read all the keycache slots and validate them against what cached values you keep in userland. If they don't match, log an error and reprogram the slot. (I haven't yet done it for FreeBSD but it's on my "when I'm bored one weekend soon" project lits.) HTH, -adrian On 17 November 2013 03:08, Antonio Quartulli <antonio@open-mesh.com> wrote: > Hello list, > > I am debugging a strange problem which symptom is that the PTK of some station > is messed up right after hostapd has configured a new GTK. This happens both in > WPA2 only and WPA/WPA2 mixed mode. I have no real clue of what is going on and > so, after having given a look at the hostapd code, I am now checking the driver > (Ath9k). > > While digging I found a piece of code in key.c which I can't understand. > Here is the snippet: > > 111 macLo = get_unaligned_le32(mac); > 112 macHi = get_unaligned_le16(mac + 4); > 113 macLo >>= 1; > 114 macLo |= (macHi & 1) << 31; > 115 macHi >>= 1; > > At drivers/net/wireless/ath/key.c:113 there is a right-bitshift operation > executed on macLo. Since macLo is a 4byte variable containing 4 out of 6 > bytes of the MAC address passed as argument, it looks to me like the bitshift > is throwing away one bit that we will never recover. > > I don't want to claim that this is necessarily connected to the problem I am > experiencing, but I'd like to get a better understanding of this part in order > to continue my research. > > However the driver I am using is not the latest one, so if somebody knows > whether this GTK/PTK problem is known and fixed I would be glad if you could > tell me :-) > > > Thanks a lot for your time. > > Regards, > > > -- > Antonio Quartulli > > _______________________________________________ > ath9k-devel mailing list > ath9k-devel at lists.ath9k.org > https://lists.ath9k.org/mailman/listinfo/ath9k-devel > ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] GTK/PTK problem - key.c magic-bitshift 2013-11-18 21:29 ` Adrian Chadd @ 2013-11-18 21:44 ` Antonio Quartulli 2013-11-18 21:51 ` Antonio Quartulli 1 sibling, 0 replies; 16+ messages in thread From: Antonio Quartulli @ 2013-11-18 21:44 UTC (permalink / raw) To: ath9k-devel Hello! On Mon, Nov 18, 2013 at 01:29:14PM -0800, Adrian Chadd wrote: > oh goodie! goodie! > > You're likely hitting the "keycache is corrupted!" bug. Yeah I had this feeling already and I have been discussing with Jouni in the meanwhile (who confirmed my feeling) :-) > > The keycache contents is throwing away a bit you don't actually need. > Look at which bit that actually is in the OUI specification. It's > something very specific. > Yeah, I realised that after talking to Felix. > There's a chip bug that's been (as far as I could tell whilst I was at > QCA) around forever. Oh. This is a really bad news..I hoped it was a bug in the driver.... However now I know there is no bug to look for in the code! :) > > You need to add code to check for decryption failures and if so, > trigger a periodic job to read all the keycache slots and validate > them against what cached values you keep in userland. > > If they don't match, log an error and reprogram the slot. > > (I haven't yet done it for FreeBSD but it's on my "when I'm bored one > weekend soon" project lits.) Sounds ugly..but I understood what you mean. But why a periodic job? Isn't it enough to run the task once when there is a decryption problem ? Thanks a lot for your feedback! > > HTH, > > > > -adrian > > > > On 17 November 2013 03:08, Antonio Quartulli <antonio@open-mesh.com> wrote: > > Hello list, > > > > I am debugging a strange problem which symptom is that the PTK of some station > > is messed up right after hostapd has configured a new GTK. This happens both in > > WPA2 only and WPA/WPA2 mixed mode. I have no real clue of what is going on and > > so, after having given a look at the hostapd code, I am now checking the driver > > (Ath9k). > > > > While digging I found a piece of code in key.c which I can't understand. > > Here is the snippet: > > > > 111 macLo = get_unaligned_le32(mac); > > 112 macHi = get_unaligned_le16(mac + 4); > > 113 macLo >>= 1; > > 114 macLo |= (macHi & 1) << 31; > > 115 macHi >>= 1; > > > > At drivers/net/wireless/ath/key.c:113 there is a right-bitshift operation > > executed on macLo. Since macLo is a 4byte variable containing 4 out of 6 > > bytes of the MAC address passed as argument, it looks to me like the bitshift > > is throwing away one bit that we will never recover. > > > > I don't want to claim that this is necessarily connected to the problem I am > > experiencing, but I'd like to get a better understanding of this part in order > > to continue my research. > > > > However the driver I am using is not the latest one, so if somebody knows > > whether this GTK/PTK problem is known and fixed I would be glad if you could > > tell me :-) > > > > > > Thanks a lot for your time. > > > > Regards, > > > > > > -- > > Antonio Quartulli > > > > _______________________________________________ > > ath9k-devel mailing list > > ath9k-devel at lists.ath9k.org > > https://lists.ath9k.org/mailman/listinfo/ath9k-devel > > -- Antonio Quartulli -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: Digital signature Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131118/f4cce8f6/attachment.pgp ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] GTK/PTK problem - key.c magic-bitshift 2013-11-18 21:29 ` Adrian Chadd 2013-11-18 21:44 ` Antonio Quartulli @ 2013-11-18 21:51 ` Antonio Quartulli 2013-11-20 8:41 ` Adrian Chadd 1 sibling, 1 reply; 16+ messages in thread From: Antonio Quartulli @ 2013-11-18 21:51 UTC (permalink / raw) To: ath9k-devel On Mon, Nov 18, 2013 at 01:29:14PM -0800, Adrian Chadd wrote: > There's a chip bug that's been (as far as I could tell whilst I was at > QCA) around forever. > Do you know if this bug is triggered during write operations to other slots only (in my case it seems the cache was getting corrupted after a GTK upload) or if this can happen at any time? In the first case I could check the cache for possible corruptions only when ath_key_config() is invoked (I could compare the cache state before the upload with what I have after it - as Jouni suggested) and call the "resync task" only in that case. Cheers, -- Antonio Quartulli -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: Digital signature Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131118/85d9feb7/attachment.pgp ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] GTK/PTK problem - key.c magic-bitshift 2013-11-18 21:51 ` Antonio Quartulli @ 2013-11-20 8:41 ` Adrian Chadd 2013-11-22 9:18 ` Antonio Quartulli 0 siblings, 1 reply; 16+ messages in thread From: Adrian Chadd @ 2013-11-20 8:41 UTC (permalink / raw) To: ath9k-devel I don't know exactly what triggers it. I suggested a periodic event so you only do it say, max every second or two, but you trigger it when you get more than a bunch of decrypt errors. That way if you keep getting decrypt errors, you only get it scheduled every second or two, rather than on each decrypt error. -a On 18 November 2013 13:51, Antonio Quartulli <antonio@open-mesh.com> wrote: > On Mon, Nov 18, 2013 at 01:29:14PM -0800, Adrian Chadd wrote: >> There's a chip bug that's been (as far as I could tell whilst I was at >> QCA) around forever. >> > > Do you know if this bug is triggered during write operations to other slots > only (in my case it seems the cache was getting corrupted after a GTK upload) > or if this can happen at any time? > > In the first case I could check the cache for possible corruptions only when > ath_key_config() is invoked (I could compare the cache state before the upload > with what I have after it - as Jouni suggested) and call the "resync task" only > in that case. > > > Cheers, > > -- > Antonio Quartulli ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] GTK/PTK problem - key.c magic-bitshift 2013-11-20 8:41 ` Adrian Chadd @ 2013-11-22 9:18 ` Antonio Quartulli 2013-11-22 9:42 ` Adrian Chadd 0 siblings, 1 reply; 16+ messages in thread From: Antonio Quartulli @ 2013-11-22 9:18 UTC (permalink / raw) To: ath9k-devel On Wed, Nov 20, 2013 at 12:41:07AM -0800, Adrian Chadd wrote: > I don't know exactly what triggers it. Ok > > I suggested a periodic event so you only do it say, max every second > or two, but you trigger it when you get more than a bunch of decrypt > errors. That way if you keep getting decrypt errors, you only get it > scheduled every second or two, rather than on each decrypt error. > Yeah, sounds good. Right now I am simply trying to "refresh" the key cache right after a key is uploaded and if I observe decryption error I disable the HW acceleration for that STA (waiting for the next GTK rekey to refresh the cache and re-enable the accel again). However there is something I don't understand. In the packet dump I obtained from the network where I observe the bug, I see that after this possible "cache corruption event" ARP requests from the STA to the AP are properly decrypted (only ARP replies going in the other direction are not). If the cache is really compromised, how can this happen? I would expect the AP to not be able to decrypt the requests as well...don't you think so? Cheers, -- Antonio Quartulli -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: Digital signature Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131122/51140b96/attachment.pgp ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] GTK/PTK problem - key.c magic-bitshift 2013-11-22 9:18 ` Antonio Quartulli @ 2013-11-22 9:42 ` Adrian Chadd 2013-11-27 6:48 ` Antonio Quartulli 2013-12-04 13:23 ` [ath9k-devel] Key Cache corruption (was: GTK/PTK problem - key.c magic-bitshift) Antonio Quartulli 0 siblings, 2 replies; 16+ messages in thread From: Adrian Chadd @ 2013-11-22 9:42 UTC (permalink / raw) To: ath9k-devel Hm? There are separate transmit and receive keys negotiated. -adrian On 22 November 2013 01:18, Antonio Quartulli <antonio@open-mesh.com> wrote: > On Wed, Nov 20, 2013 at 12:41:07AM -0800, Adrian Chadd wrote: >> I don't know exactly what triggers it. > > Ok > >> >> I suggested a periodic event so you only do it say, max every second >> or two, but you trigger it when you get more than a bunch of decrypt >> errors. That way if you keep getting decrypt errors, you only get it >> scheduled every second or two, rather than on each decrypt error. >> > > Yeah, sounds good. Right now I am simply trying to "refresh" the key cache right > after a key is uploaded and if I observe decryption error I disable the HW > acceleration for that STA (waiting for the next GTK rekey to refresh the cache > and re-enable the accel again). > > However there is something I don't understand. In the packet dump I obtained > from the network where I observe the bug, I see that after this possible > "cache corruption > event" ARP requests from the STA to the AP are properly decrypted > (only ARP replies going in the other direction are not). > > If the cache is really > compromised, how can this happen? I would expect the AP to not be able to > decrypt the requests as well...don't you think so? > > > Cheers, > > > -- > Antonio Quartulli ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] GTK/PTK problem - key.c magic-bitshift 2013-11-22 9:42 ` Adrian Chadd @ 2013-11-27 6:48 ` Antonio Quartulli 2013-12-04 13:23 ` [ath9k-devel] Key Cache corruption (was: GTK/PTK problem - key.c magic-bitshift) Antonio Quartulli 1 sibling, 0 replies; 16+ messages in thread From: Antonio Quartulli @ 2013-11-27 6:48 UTC (permalink / raw) To: ath9k-devel -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 22/11/13 10:42, Adrian Chadd wrote: (sorry for re-sending but my mail server did not want to deliver to all..) Adrian, sorry for the delay. You are right. With TKIP we have different keys..but at this point how can I detect a PTK-TX corruption? It may be the case that received frames are decrypted properly (PTK-RX is ok)...but still..the PTX-TX may have been corrupted. Right? When should I trigger the periodic worker in this case? - -- Antonio Quartulli -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBCAAGBQJSlZXPAAoJEADl0hg6qKeOYbYQAKZcAR0yLd7NJcmc30/ekcM5 pfkT3wY7fnkWz9Fq2M7OgEprFX91cz44qrokh+wNikFHZlDg3qIAv3Ti+FbbHiJ+ nTHu/2INGGD1Me27e9Ts6AWLALHcQsumJhzi96uRPEna0gr4oJeHWId84mT5tUDh rO5yCFhiZiSMmnqvlG9PxIifXJo9QgQpjRy4tSRkDu9dIWrv+1F0+KLwcFGhpLCV RUR8KpXqFHFMgrMuRaiFO23U1T1yZnllEiA1eQO7urMhlBnEJ0yODHj5VDcivEWe obFzJYxG/aH+51m0y1KOcWm5C/fyvMUBitU+kwfV5SByRN4Tr50DdBQ/DCI4qR2R 7iJ/gTJcampKCRDTNUiRaUY2KYIxSEumAbgr5sRbFq2udUl8PJbEM/nhmdlMyG8N HYPN9cgzeANvvvo30fpCyJukWCp8nefhtq3FYSeQpm/16rbNak3vffzgTXcOtAVJ CHaBHcYweofmzeWtFOPTFEuLkCKA/ZZ0LzP+pv3Zmfe5Whb3LfSEcQRuwfhexb/6 SsG3omRYqn1rDxlV9MAajTv22KhJ26QIy9jp9qQDUOib4r/sleOcuVOc4CLxuaoc DSAibUDZaMFPWnn7XtSbtDdZtDVc4XZ33xyJ9ad1IyvIh6M8YVgA2QCA2DJ5Zk9R 0uwdqKx4VM7aRgfFXd8h =NqhI -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] Key Cache corruption (was: GTK/PTK problem - key.c magic-bitshift) 2013-11-22 9:42 ` Adrian Chadd 2013-11-27 6:48 ` Antonio Quartulli @ 2013-12-04 13:23 ` Antonio Quartulli 2013-12-04 17:08 ` Adrian Chadd 1 sibling, 1 reply; 16+ messages in thread From: Antonio Quartulli @ 2013-12-04 13:23 UTC (permalink / raw) To: ath9k-devel -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi list, hi Adrian, hi Sujith (just added to the cc list), sorry for the silence, but I have been busy testing different approaches on how to solve this. On 22/11/13 10:42, Adrian Chadd wrote: Mh, that is true with TKIP. This means we have no way to recognize a TX key corruption. For this reason a periodic worker that unconditionally checks the cache and possibly refresh it looked like the best solution. However, after testing this strategy I realised that the worker is not able to see the cache corruption: what it reads from the memory always matches what we have in mac80211 (but I am sure we had a corruption). My opinion is that the corruption is really happening at a low-low level and so the memory content is not even altered. This may be one of the reasons why this bug is so difficult to catch. The only solution I see to this problem is to make the periodic worker refresh the key cache every second, no matter if a corruption is found or not. However it does not sound like a good solution..any thought? Another experiment I have done was to refresh the whole key cache each time a new key is pushed into ath9k. I did this because I observed that the corruption was triggered upon GTK upload (on re-keying). This fix seemed to make things better. So another plausible strategy could be a mix of the above solutions: 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2) upon detection of a number of RX decrypt errors schedule a worker that refresh the entire cache (not the single slot...because that could be the cause for another corruption...). Still we have the problem of the TX key corruption with TKIP. If this happens we can't detect it and we need to wait for the next set_key() before the key can be properly restored..... Any better idea? Regards, - -- Antonio Quartulli -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBCAAGBQJSnyzgAAoJEEKTMo6mOh1VxggP/jzp+UB8lPiFItwMPXwob8ik qNd1WcQhcShft8MrmK9kfjzp91t/uBby/QVsqwxKu8sVIpuQJaGI4ikOtIZ6/8b6 FGkn+JouKY+gmEsT6U4Hfq6v4Ri/xKkUt5JfPN7moQr3OszitbnDGlO3gr9qY3Vv YjDGkZPlAmGCpnLUY1nHiBrEhj0WX2GrFXQC06wskk5UYEyvE5PDy9LUt1ASQkxe qa6rBWaanVT0YpLhoHXNldSu6aF1r1HXaG+lKnBMgJlBcLSBta7oxzYjxj6oPtLQ J+QYkdkkwrEGqVR8JL+54tAYkJ0TZvw+DKuJ6PcWE/Z74LiiVczrEZ0yf5oQm8Ki EgQUay8X3gOT9ZFIrnYUK3i2PdA8UsBv/xmIOhlRHqx2Au7bcekM+9c7tIZjxz0V QVspKLaRefvP4thcB0Sni56D05vHdV69wFPui+d+Y2Z/i72NjP4x2AXylfOmVNsV 9tI135f8see5wQOQN7WJFuM67h7GlyeSIeAZWFjwhdDBWSmWDtal9w9930MZgAVe Oh1oLCoboi3gywiN4bpL59Thvt164prDQA6aA9uu90Ytbzxept0ZUITd+z5XQVbb BLBWLSrZme0BDCqkJw55Z/ypl4TlXIIdWK935YDNMPRl1RGo86KUDSoCbFZNjc4J w8WBV7CS7jmU+E2DjLzy =gnO4 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] Key Cache corruption (was: GTK/PTK problem - key.c magic-bitshift) 2013-12-04 13:23 ` [ath9k-devel] Key Cache corruption (was: GTK/PTK problem - key.c magic-bitshift) Antonio Quartulli @ 2013-12-04 17:08 ` Adrian Chadd 2013-12-09 10:17 ` [ath9k-devel] Key Cache corruption Antonio Quartulli 0 siblings, 1 reply; 16+ messages in thread From: Adrian Chadd @ 2013-12-04 17:08 UTC (permalink / raw) To: ath9k-devel Hi, I'll see if I can organise a chat with the MAC guys this week at Atheros and ask them about this. Stay tuned... -a On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hi list, hi Adrian, hi Sujith (just added to the cc list), > > sorry for the silence, but I have been busy testing different > approaches on how to solve this. > > On 22/11/13 10:42, Adrian Chadd wrote: > > > Mh, that is true with TKIP. > > This means we have no way to recognize a TX key corruption. > For this reason a periodic worker that unconditionally checks the > cache and possibly refresh it looked like the best solution. > > However, after testing this strategy I realised that the worker is not > able to see the cache corruption: what it reads from the memory always > matches what we have in mac80211 (but I am sure we had a corruption). > > My opinion is that the corruption is really happening at a low-low > level and so the memory content is not even altered. This may be one > of the reasons why this bug is so difficult to catch. > > The only solution I see to this problem is to make the periodic worker > refresh the key cache every second, no matter if a corruption is found > or not. > > However it does not sound like a good solution..any thought? > > > > Another experiment I have done was to refresh the whole key cache each > time a new key is pushed into ath9k. I did this because I observed > that the corruption was triggered upon GTK upload (on re-keying). > This fix seemed to make things better. > > So another plausible strategy could be a mix of the above solutions: > > 1) refresh the entire cache on each set_key(cmd=SET_KEY) call > 2) upon detection of a number of RX decrypt errors schedule a worker > that refresh the entire cache (not the single slot...because that > could be the cause for another corruption...). > > > Still we have the problem of the TX key corruption with TKIP. If this > happens we can't detect it and we need to wait for the next set_key() > before the key can be properly restored..... > > > Any better idea? > > > Regards, > > > - -- > Antonio Quartulli > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.22 (GNU/Linux) > > iQIcBAEBCAAGBQJSnyzgAAoJEEKTMo6mOh1VxggP/jzp+UB8lPiFItwMPXwob8ik > qNd1WcQhcShft8MrmK9kfjzp91t/uBby/QVsqwxKu8sVIpuQJaGI4ikOtIZ6/8b6 > FGkn+JouKY+gmEsT6U4Hfq6v4Ri/xKkUt5JfPN7moQr3OszitbnDGlO3gr9qY3Vv > YjDGkZPlAmGCpnLUY1nHiBrEhj0WX2GrFXQC06wskk5UYEyvE5PDy9LUt1ASQkxe > qa6rBWaanVT0YpLhoHXNldSu6aF1r1HXaG+lKnBMgJlBcLSBta7oxzYjxj6oPtLQ > J+QYkdkkwrEGqVR8JL+54tAYkJ0TZvw+DKuJ6PcWE/Z74LiiVczrEZ0yf5oQm8Ki > EgQUay8X3gOT9ZFIrnYUK3i2PdA8UsBv/xmIOhlRHqx2Au7bcekM+9c7tIZjxz0V > QVspKLaRefvP4thcB0Sni56D05vHdV69wFPui+d+Y2Z/i72NjP4x2AXylfOmVNsV > 9tI135f8see5wQOQN7WJFuM67h7GlyeSIeAZWFjwhdDBWSmWDtal9w9930MZgAVe > Oh1oLCoboi3gywiN4bpL59Thvt164prDQA6aA9uu90Ytbzxept0ZUITd+z5XQVbb > BLBWLSrZme0BDCqkJw55Z/ypl4TlXIIdWK935YDNMPRl1RGo86KUDSoCbFZNjc4J > w8WBV7CS7jmU+E2DjLzy > =gnO4 > -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] Key Cache corruption 2013-12-04 17:08 ` Adrian Chadd @ 2013-12-09 10:17 ` Antonio Quartulli 2013-12-09 19:38 ` Adrian Chadd 0 siblings, 1 reply; 16+ messages in thread From: Antonio Quartulli @ 2013-12-09 10:17 UTC (permalink / raw) To: ath9k-devel -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hoi Adrian, I don't want to push (I am just polling here ;-)), but have you been able to squeeze any information out of the MAC guys? Cheers, On 04/12/13 18:08, Adrian Chadd wrote: > Hi, > > I'll see if I can organise a chat with the MAC guys this week at > Atheros and ask them about this. Stay tuned... > > > -a > > On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com> > wrote: Hi list, hi Adrian, hi Sujith (just added to the cc list), > > sorry for the silence, but I have been busy testing different > approaches on how to solve this. > > On 22/11/13 10:42, Adrian Chadd wrote: > > > Mh, that is true with TKIP. > > This means we have no way to recognize a TX key corruption. For > this reason a periodic worker that unconditionally checks the cache > and possibly refresh it looked like the best solution. > > However, after testing this strategy I realised that the worker is > not able to see the cache corruption: what it reads from the memory > always matches what we have in mac80211 (but I am sure we had a > corruption). > > My opinion is that the corruption is really happening at a low-low > level and so the memory content is not even altered. This may be > one of the reasons why this bug is so difficult to catch. > > The only solution I see to this problem is to make the periodic > worker refresh the key cache every second, no matter if a > corruption is found or not. > > However it does not sound like a good solution..any thought? > > > > Another experiment I have done was to refresh the whole key cache > each time a new key is pushed into ath9k. I did this because I > observed that the corruption was triggered upon GTK upload (on > re-keying). This fix seemed to make things better. > > So another plausible strategy could be a mix of the above > solutions: > > 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2) > upon detection of a number of RX decrypt errors schedule a worker > that refresh the entire cache (not the single slot...because that > could be the cause for another corruption...). > > > Still we have the problem of the TX key corruption with TKIP. If > this happens we can't detect it and we need to wait for the next > set_key() before the key can be properly restored..... > > > Any better idea? > > > Regards, > > - -- Antonio Quartulli -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBCAAGBQJSpZitAAoJEEKTMo6mOh1VSLYQAIPCr6j6YJd0WiVeO23xGS75 hO7kfSm2y+zMJvl8WYFY5mKrUUssXYfp9YvDOUZV1KSK3cxHt93dzlEnvyhLCAQc RpHsbg0DPcuxIKzH4hQVl6mYyzAqMm2/icRXome+6H2Fz8fcGNy+4FGwiAhvAZdx 3WIj773EwE8m2/M4Kpa5HHIrcmJ305EFopAXZMEP9N37V1ANTGkJDhWI9gU3sqC/ XQmeNsTY3m0LG/CTyhoT0RvkW284pGxQkVPd29QDQFYKAQv0DcrPmohcprPcfQmA Bf1MIKTV734RQ+bU/TPw5lpCpUnGPhILwiijsnwNWGKSBEgjydS7LXp9/F+QMHHi inUOY9zVJHvETCJ+4cI/xKOYU1UCAsqnKldBmY63BrJr/MhJ1JiPsKVbd6rIyGmP KuQ9J4y+yP+y6TOYFoyFNlLFWRS+tbY2XPm51nV7aaYvmKZi+GgA8i1per3xmNAl 28Qwqnerjl+gyF0+IpU+61uMkx9wg3jIs2PtW9Ek7tSLOPb5vjyCmRvw8N+Djyax fr57lMwqV/yKZJlv5NA6i04WoocHn3IM2WaS7BeO/7h4sDKY2BOprpc4YxFw9rPF ZwhDNrVpMOWJzpnBwFyNNjXofQ6iVdtDmzHuUG98Y+WvEbMMV4Rq5sMu7rq03Mz7 w2IPW9twgtvcXUc2V7R3 =GZj5 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] Key Cache corruption 2013-12-09 10:17 ` [ath9k-devel] Key Cache corruption Antonio Quartulli @ 2013-12-09 19:38 ` Adrian Chadd 2013-12-12 9:17 ` Antonio Quartulli 0 siblings, 1 reply; 16+ messages in thread From: Adrian Chadd @ 2013-12-09 19:38 UTC (permalink / raw) To: ath9k-devel Hi! I've emailed them - they're tossing it around internally. -a On 9 December 2013 02:17, Antonio Quartulli <antonio@open-mesh.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hoi Adrian, > > I don't want to push (I am just polling here ;-)), but have you been > able to squeeze any information out of the MAC guys? > > Cheers, > > On 04/12/13 18:08, Adrian Chadd wrote: >> Hi, >> >> I'll see if I can organise a chat with the MAC guys this week at >> Atheros and ask them about this. Stay tuned... >> >> >> -a >> >> On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com> >> wrote: Hi list, hi Adrian, hi Sujith (just added to the cc list), >> >> sorry for the silence, but I have been busy testing different >> approaches on how to solve this. >> >> On 22/11/13 10:42, Adrian Chadd wrote: >> >> >> Mh, that is true with TKIP. >> >> This means we have no way to recognize a TX key corruption. For >> this reason a periodic worker that unconditionally checks the cache >> and possibly refresh it looked like the best solution. >> >> However, after testing this strategy I realised that the worker is >> not able to see the cache corruption: what it reads from the memory >> always matches what we have in mac80211 (but I am sure we had a >> corruption). >> >> My opinion is that the corruption is really happening at a low-low >> level and so the memory content is not even altered. This may be >> one of the reasons why this bug is so difficult to catch. >> >> The only solution I see to this problem is to make the periodic >> worker refresh the key cache every second, no matter if a >> corruption is found or not. >> >> However it does not sound like a good solution..any thought? >> >> >> >> Another experiment I have done was to refresh the whole key cache >> each time a new key is pushed into ath9k. I did this because I >> observed that the corruption was triggered upon GTK upload (on >> re-keying). This fix seemed to make things better. >> >> So another plausible strategy could be a mix of the above >> solutions: >> >> 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2) >> upon detection of a number of RX decrypt errors schedule a worker >> that refresh the entire cache (not the single slot...because that >> could be the cause for another corruption...). >> >> >> Still we have the problem of the TX key corruption with TKIP. If >> this happens we can't detect it and we need to wait for the next >> set_key() before the key can be properly restored..... >> >> >> Any better idea? >> >> >> Regards, >> >> > > - -- > Antonio Quartulli > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.22 (GNU/Linux) > > iQIcBAEBCAAGBQJSpZitAAoJEEKTMo6mOh1VSLYQAIPCr6j6YJd0WiVeO23xGS75 > hO7kfSm2y+zMJvl8WYFY5mKrUUssXYfp9YvDOUZV1KSK3cxHt93dzlEnvyhLCAQc > RpHsbg0DPcuxIKzH4hQVl6mYyzAqMm2/icRXome+6H2Fz8fcGNy+4FGwiAhvAZdx > 3WIj773EwE8m2/M4Kpa5HHIrcmJ305EFopAXZMEP9N37V1ANTGkJDhWI9gU3sqC/ > XQmeNsTY3m0LG/CTyhoT0RvkW284pGxQkVPd29QDQFYKAQv0DcrPmohcprPcfQmA > Bf1MIKTV734RQ+bU/TPw5lpCpUnGPhILwiijsnwNWGKSBEgjydS7LXp9/F+QMHHi > inUOY9zVJHvETCJ+4cI/xKOYU1UCAsqnKldBmY63BrJr/MhJ1JiPsKVbd6rIyGmP > KuQ9J4y+yP+y6TOYFoyFNlLFWRS+tbY2XPm51nV7aaYvmKZi+GgA8i1per3xmNAl > 28Qwqnerjl+gyF0+IpU+61uMkx9wg3jIs2PtW9Ek7tSLOPb5vjyCmRvw8N+Djyax > fr57lMwqV/yKZJlv5NA6i04WoocHn3IM2WaS7BeO/7h4sDKY2BOprpc4YxFw9rPF > ZwhDNrVpMOWJzpnBwFyNNjXofQ6iVdtDmzHuUG98Y+WvEbMMV4Rq5sMu7rq03Mz7 > w2IPW9twgtvcXUc2V7R3 > =GZj5 > -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] Key Cache corruption 2013-12-09 19:38 ` Adrian Chadd @ 2013-12-12 9:17 ` Antonio Quartulli 2013-12-13 12:35 ` Adrian Chadd 0 siblings, 1 reply; 16+ messages in thread From: Antonio Quartulli @ 2013-12-12 9:17 UTC (permalink / raw) To: ath9k-devel Hi Adrian! but as far as you know, isn't the internal Atheros driver applying any workaround or strange hack to circumvent this problem? Or simply there is no "clear" workaround at all? (maybe this is what you expect to be told from the MAC guys?) Cheers, On 09/12/13 20:38, Adrian Chadd wrote: > Hi! > > I've emailed them - they're tossing it around internally. > > > -a > > On 9 December 2013 02:17, Antonio Quartulli <antonio@open-mesh.com> wrote: > Hoi Adrian, > > I don't want to push (I am just polling here ;-)), but have you been > able to squeeze any information out of the MAC guys? > > Cheers, > > On 04/12/13 18:08, Adrian Chadd wrote: >>>> Hi, >>>> >>>> I'll see if I can organise a chat with the MAC guys this week at >>>> Atheros and ask them about this. Stay tuned... >>>> >>>> >>>> -a >>>> >>>> On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com> >>>> wrote: Hi list, hi Adrian, hi Sujith (just added to the cc list), >>>> >>>> sorry for the silence, but I have been busy testing different >>>> approaches on how to solve this. >>>> >>>> On 22/11/13 10:42, Adrian Chadd wrote: >>>> >>>> >>>> Mh, that is true with TKIP. >>>> >>>> This means we have no way to recognize a TX key corruption. For >>>> this reason a periodic worker that unconditionally checks the cache >>>> and possibly refresh it looked like the best solution. >>>> >>>> However, after testing this strategy I realised that the worker is >>>> not able to see the cache corruption: what it reads from the memory >>>> always matches what we have in mac80211 (but I am sure we had a >>>> corruption). >>>> >>>> My opinion is that the corruption is really happening at a low-low >>>> level and so the memory content is not even altered. This may be >>>> one of the reasons why this bug is so difficult to catch. >>>> >>>> The only solution I see to this problem is to make the periodic >>>> worker refresh the key cache every second, no matter if a >>>> corruption is found or not. >>>> >>>> However it does not sound like a good solution..any thought? >>>> >>>> >>>> >>>> Another experiment I have done was to refresh the whole key cache >>>> each time a new key is pushed into ath9k. I did this because I >>>> observed that the corruption was triggered upon GTK upload (on >>>> re-keying). This fix seemed to make things better. >>>> >>>> So another plausible strategy could be a mix of the above >>>> solutions: >>>> >>>> 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2) >>>> upon detection of a number of RX decrypt errors schedule a worker >>>> that refresh the entire cache (not the single slot...because that >>>> could be the cause for another corruption...). >>>> >>>> >>>> Still we have the problem of the TX key corruption with TKIP. If >>>> this happens we can't detect it and we need to wait for the next >>>> set_key() before the key can be properly restored..... >>>> >>>> >>>> Any better idea? >>>> >>>> >>>> Regards, >>>> >>>> > -- Antonio Quartulli -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131212/915710c2/attachment.pgp ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] Key Cache corruption 2013-12-12 9:17 ` Antonio Quartulli @ 2013-12-13 12:35 ` Adrian Chadd 2013-12-13 12:55 ` Antonio Quartulli 2014-01-07 15:23 ` Antonio Quartulli 0 siblings, 2 replies; 16+ messages in thread From: Adrian Chadd @ 2013-12-13 12:35 UTC (permalink / raw) To: ath9k-devel There's a work around, yeah. Much like I've/we've described. -adrian On 12 December 2013 01:17, Antonio Quartulli <antonio@open-mesh.com> wrote: > Hi Adrian! > > but as far as you know, isn't the internal Atheros driver applying any > workaround or strange hack to circumvent this problem? > Or simply there is no "clear" workaround at all? (maybe this is what you > expect to be told from the MAC guys?) > > > > Cheers, > > > On 09/12/13 20:38, Adrian Chadd wrote: >> Hi! >> >> I've emailed them - they're tossing it around internally. >> >> >> -a >> >> On 9 December 2013 02:17, Antonio Quartulli <antonio@open-mesh.com> wrote: >> Hoi Adrian, >> >> I don't want to push (I am just polling here ;-)), but have you been >> able to squeeze any information out of the MAC guys? >> >> Cheers, >> >> On 04/12/13 18:08, Adrian Chadd wrote: >>>>> Hi, >>>>> >>>>> I'll see if I can organise a chat with the MAC guys this week at >>>>> Atheros and ask them about this. Stay tuned... >>>>> >>>>> >>>>> -a >>>>> >>>>> On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com> >>>>> wrote: Hi list, hi Adrian, hi Sujith (just added to the cc list), >>>>> >>>>> sorry for the silence, but I have been busy testing different >>>>> approaches on how to solve this. >>>>> >>>>> On 22/11/13 10:42, Adrian Chadd wrote: >>>>> >>>>> >>>>> Mh, that is true with TKIP. >>>>> >>>>> This means we have no way to recognize a TX key corruption. For >>>>> this reason a periodic worker that unconditionally checks the cache >>>>> and possibly refresh it looked like the best solution. >>>>> >>>>> However, after testing this strategy I realised that the worker is >>>>> not able to see the cache corruption: what it reads from the memory >>>>> always matches what we have in mac80211 (but I am sure we had a >>>>> corruption). >>>>> >>>>> My opinion is that the corruption is really happening at a low-low >>>>> level and so the memory content is not even altered. This may be >>>>> one of the reasons why this bug is so difficult to catch. >>>>> >>>>> The only solution I see to this problem is to make the periodic >>>>> worker refresh the key cache every second, no matter if a >>>>> corruption is found or not. >>>>> >>>>> However it does not sound like a good solution..any thought? >>>>> >>>>> >>>>> >>>>> Another experiment I have done was to refresh the whole key cache >>>>> each time a new key is pushed into ath9k. I did this because I >>>>> observed that the corruption was triggered upon GTK upload (on >>>>> re-keying). This fix seemed to make things better. >>>>> >>>>> So another plausible strategy could be a mix of the above >>>>> solutions: >>>>> >>>>> 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2) >>>>> upon detection of a number of RX decrypt errors schedule a worker >>>>> that refresh the entire cache (not the single slot...because that >>>>> could be the cause for another corruption...). >>>>> >>>>> >>>>> Still we have the problem of the TX key corruption with TKIP. If >>>>> this happens we can't detect it and we need to wait for the next >>>>> set_key() before the key can be properly restored..... >>>>> >>>>> >>>>> Any better idea? >>>>> >>>>> >>>>> Regards, >>>>> >>>>> >> > > -- > Antonio Quartulli > ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] Key Cache corruption 2013-12-13 12:35 ` Adrian Chadd @ 2013-12-13 12:55 ` Antonio Quartulli 2014-01-07 15:23 ` Antonio Quartulli 1 sibling, 0 replies; 16+ messages in thread From: Antonio Quartulli @ 2013-12-13 12:55 UTC (permalink / raw) To: ath9k-devel Oky! Meanwhile you get a reply from the MAC guys I will try to test again the "check & fix" approach. It is really strange that I can't recognize any corruption. However, the problem of the TX key corruption still remains. It seems we can't do much about that. Cheers (and thanks a lot so far!), On 13/12/13 13:35, Adrian Chadd wrote: > There's a work around, yeah. Much like I've/we've described. > > > > -adrian > > On 12 December 2013 01:17, Antonio Quartulli <antonio@open-mesh.com> wrote: >> Hi Adrian! >> >> but as far as you know, isn't the internal Atheros driver applying any >> workaround or strange hack to circumvent this problem? >> Or simply there is no "clear" workaround at all? (maybe this is what you >> expect to be told from the MAC guys?) >> >> >> >> Cheers, >> >> >> On 09/12/13 20:38, Adrian Chadd wrote: >>> Hi! >>> >>> I've emailed them - they're tossing it around internally. >>> >>> >>> -a >>> >>> On 9 December 2013 02:17, Antonio Quartulli <antonio@open-mesh.com> wrote: >>> Hoi Adrian, >>> >>> I don't want to push (I am just polling here ;-)), but have you been >>> able to squeeze any information out of the MAC guys? >>> >>> Cheers, >>> >>> On 04/12/13 18:08, Adrian Chadd wrote: >>>>>> Hi, >>>>>> >>>>>> I'll see if I can organise a chat with the MAC guys this week at >>>>>> Atheros and ask them about this. Stay tuned... >>>>>> >>>>>> >>>>>> -a >>>>>> >>>>>> On 4 December 2013 05:23, Antonio Quartulli <antonio@open-mesh.com> >>>>>> wrote: Hi list, hi Adrian, hi Sujith (just added to the cc list), >>>>>> >>>>>> sorry for the silence, but I have been busy testing different >>>>>> approaches on how to solve this. >>>>>> >>>>>> On 22/11/13 10:42, Adrian Chadd wrote: >>>>>> >>>>>> >>>>>> Mh, that is true with TKIP. >>>>>> >>>>>> This means we have no way to recognize a TX key corruption. For >>>>>> this reason a periodic worker that unconditionally checks the cache >>>>>> and possibly refresh it looked like the best solution. >>>>>> >>>>>> However, after testing this strategy I realised that the worker is >>>>>> not able to see the cache corruption: what it reads from the memory >>>>>> always matches what we have in mac80211 (but I am sure we had a >>>>>> corruption). >>>>>> >>>>>> My opinion is that the corruption is really happening at a low-low >>>>>> level and so the memory content is not even altered. This may be >>>>>> one of the reasons why this bug is so difficult to catch. >>>>>> >>>>>> The only solution I see to this problem is to make the periodic >>>>>> worker refresh the key cache every second, no matter if a >>>>>> corruption is found or not. >>>>>> >>>>>> However it does not sound like a good solution..any thought? >>>>>> >>>>>> >>>>>> >>>>>> Another experiment I have done was to refresh the whole key cache >>>>>> each time a new key is pushed into ath9k. I did this because I >>>>>> observed that the corruption was triggered upon GTK upload (on >>>>>> re-keying). This fix seemed to make things better. >>>>>> >>>>>> So another plausible strategy could be a mix of the above >>>>>> solutions: >>>>>> >>>>>> 1) refresh the entire cache on each set_key(cmd=SET_KEY) call 2) >>>>>> upon detection of a number of RX decrypt errors schedule a worker >>>>>> that refresh the entire cache (not the single slot...because that >>>>>> could be the cause for another corruption...). >>>>>> >>>>>> >>>>>> Still we have the problem of the TX key corruption with TKIP. If >>>>>> this happens we can't detect it and we need to wait for the next >>>>>> set_key() before the key can be properly restored..... >>>>>> >>>>>> >>>>>> Any better idea? >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> >>> >> >> -- >> Antonio Quartulli >> -- Antonio Quartulli -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20131213/e7f2777d/attachment.pgp ^ permalink raw reply [flat|nested] 16+ messages in thread
* [ath9k-devel] Key Cache corruption 2013-12-13 12:35 ` Adrian Chadd 2013-12-13 12:55 ` Antonio Quartulli @ 2014-01-07 15:23 ` Antonio Quartulli 1 sibling, 0 replies; 16+ messages in thread From: Antonio Quartulli @ 2014-01-07 15:23 UTC (permalink / raw) To: ath9k-devel Hi Adrian! Did 2014 bring any news about this issue? :) I haven't done much progress myself, but I won't give up! Cheers, -- Antonio Quartulli -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20140107/93f7b32e/attachment.pgp ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2014-01-07 15:23 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-11-17 11:08 [ath9k-devel] GTK/PTK problem - key.c magic-bitshift Antonio Quartulli 2013-11-18 21:29 ` Adrian Chadd 2013-11-18 21:44 ` Antonio Quartulli 2013-11-18 21:51 ` Antonio Quartulli 2013-11-20 8:41 ` Adrian Chadd 2013-11-22 9:18 ` Antonio Quartulli 2013-11-22 9:42 ` Adrian Chadd 2013-11-27 6:48 ` Antonio Quartulli 2013-12-04 13:23 ` [ath9k-devel] Key Cache corruption (was: GTK/PTK problem - key.c magic-bitshift) Antonio Quartulli 2013-12-04 17:08 ` Adrian Chadd 2013-12-09 10:17 ` [ath9k-devel] Key Cache corruption Antonio Quartulli 2013-12-09 19:38 ` Adrian Chadd 2013-12-12 9:17 ` Antonio Quartulli 2013-12-13 12:35 ` Adrian Chadd 2013-12-13 12:55 ` Antonio Quartulli 2014-01-07 15:23 ` Antonio Quartulli
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.