Potential Broadcast Issues After GTK Key Exchange on ath11k with IPQ8072A (QCN5024/QCN5054)

Potential Broadcast Issues After GTK Key Exchange on ath11k with IPQ8072A (QCN5024/QCN5054)

[Steffen Moser]

Hello everyone,

I've encountered a possible issue in a DD-WRT [1] setup where broadcast 
packets stop being delivered after a GTK (Group Temporal Key) exchange. 
This issue occurs on a system with the following hardware:

     Access Point Hardware: DynaLink DL-WRX36
     Router Software: DD-WRT v3.0-r58819 std (12/13/24)
     CPU: Qualcomm IPQ8072A
     WiFi Chips: Qualcomm QCN5024 and Qualcomm QCN5054
     WiFi Driver: ath11k
     Firmware: WLAN.HK.2.12-01460-QCAHKSWPL_SILICONZ-1
     NSS FW version: NSS.FW.12.5-210-HK.R
     Kernel: Linux WL-AP-EG 6.6.64-rt29 #1791 SMP Thu Dec 12 16:41:51 
+07 2024 aarch64 DD-WRT

The behavior is such that after a GTK exchange, the AP can get into a 
"weird state". When being there, broadcast frames like ARP or mDNS are 
no longer reliably delivered to connected clients while unicasts come 
still through. In this "weird state", the channel quality (active time 
vs. busy time) goes down and latencies to the still reachable WIFI 
clients rise.

I've come across a related bug report on GitHub that describes a similar 
issue: 
https://github.com/openwrt/openwrt/issues/9555#issuecomment-2433857175

Unfortunately, the GitHub discussion drifted towards various other 
possible bugs.

In the meantime, I have a done a lot of additional debugging, but I am 
coming to a dead end due to limited knowledge of the ath11k driver and 
firmware internals. Interestingly, the AP can get back from "weird 
state" to "normal state" after another GTK rekey event. I've seen this 
behavior only in the 5 GHz band, yet (using non-DFS-channels).

My questions to the Linux wireless experts and developers in this community:

  · Is such a behavior known with ath11k on IPQ8072A or on the mentioned 
WiFi chips (QCN5024/QCN5054)?

  · Could this be a driver or firmware issue that specifically arises 
after a GTK or even GMK exchange?

  · What can I do to debug it further? I've switched on debugging in 
"hostapd" in order to see the keying events. Are there more lower-level 
logs I can get from the WiFi chip and match to the latency and key 
exchange observations?

  · Are there any additional information I can/should deliver to give 
the devs more insight about this issue?

When exchanging the DynaLink DL-WRX36 AP by a Netgear R7800 AP (CPU: QCA 
IPQ8065), its predecessor, the problem is gone without touching any of 
the clients.

Thank you in advance for any insights or experiences regarding this issue.

Best regards,
Steffen

[1] https://dd-wrt.com/


-- 
✂-----------------------------------------------------------------------
Dipl.-Inf. Steffen Moser                  Tel (Office): +49.731.50.32407
School of Advanced Professional Studies       Ulm University, Room: 1013
https://wissenschaftliche-weiterbildung.org/    Oberberghof 7, 89081 Ulm
https://saps.uni-ulm.de/                                         Germany

Re: Potential Broadcast Issues After GTK Key Exchange on ath11k with IPQ8072A (QCN5024/QCN5054)

[Remi Pommarel]

Hi Steffen.

On Thu, Dec 19, 2024 at 04:02:30PM +0100, Steffen Moser wrote:
> Hello everyone,
> 
> I've encountered a possible issue in a DD-WRT [1] setup where broadcast
> packets stop being delivered after a GTK (Group Temporal Key) exchange. This
> issue occurs on a system with the following hardware:
> 
>     Access Point Hardware: DynaLink DL-WRX36
>     Router Software: DD-WRT v3.0-r58819 std (12/13/24)
>     CPU: Qualcomm IPQ8072A
>     WiFi Chips: Qualcomm QCN5024 and Qualcomm QCN5054
>     WiFi Driver: ath11k
>     Firmware: WLAN.HK.2.12-01460-QCAHKSWPL_SILICONZ-1
>     NSS FW version: NSS.FW.12.5-210-HK.R
>     Kernel: Linux WL-AP-EG 6.6.64-rt29 #1791 SMP Thu Dec 12 16:41:51 +07
> 2024 aarch64 DD-WRT
> 
> The behavior is such that after a GTK exchange, the AP can get into a "weird
> state". When being there, broadcast frames like ARP or mDNS are no longer
> reliably delivered to connected clients while unicasts come still through.
> In this "weird state", the channel quality (active time vs. busy time) goes
> down and latencies to the still reachable WIFI clients rise.

This looks a lot like an issue we hit a while back. There is this patch
[0] from Qualcomm's wlan-open repository. It is a revert of [1]. Using
that the issue was never reproduced. Maybe this can help.

Also adding ath11k list.

Regards.

-- 
Remi

[0]: https://git.codelinaro.org/clo/qsdk/oss/system/feeds/wlan-open/-/blob/win.wlan_host_opensource.3.0.r24/patches/ath11k/350-ath11k-Revert-clear-the-keys-properly-when-DISABLE_K.patch
[1]: commit 436a4e886598 ("ath11k: clear the keys properly via DISABLE_KEY")

> 
> I've come across a related bug report on GitHub that describes a similar
> issue:
> https://github.com/openwrt/openwrt/issues/9555#issuecomment-2433857175
> 
> Unfortunately, the GitHub discussion drifted towards various other possible
> bugs.
> 
> In the meantime, I have a done a lot of additional debugging, but I am
> coming to a dead end due to limited knowledge of the ath11k driver and
> firmware internals. Interestingly, the AP can get back from "weird state" to
> "normal state" after another GTK rekey event. I've seen this behavior only
> in the 5 GHz band, yet (using non-DFS-channels).
> 
> My questions to the Linux wireless experts and developers in this community:
> 
>  · Is such a behavior known with ath11k on IPQ8072A or on the mentioned WiFi
> chips (QCN5024/QCN5054)?
> 
>  · Could this be a driver or firmware issue that specifically arises after a
> GTK or even GMK exchange?
> 
>  · What can I do to debug it further? I've switched on debugging in
> "hostapd" in order to see the keying events. Are there more lower-level logs
> I can get from the WiFi chip and match to the latency and key exchange
> observations?
> 
>  · Are there any additional information I can/should deliver to give the
> devs more insight about this issue?
> 
> When exchanging the DynaLink DL-WRX36 AP by a Netgear R7800 AP (CPU: QCA
> IPQ8065), its predecessor, the problem is gone without touching any of the
> clients.
> 
> Thank you in advance for any insights or experiences regarding this issue.
> 
> Best regards,
> Steffen
> 
> [1] https://dd-wrt.com/
> 
> 
> -- 
> ✂-----------------------------------------------------------------------
> Dipl.-Inf. Steffen Moser                  Tel (Office): +49.731.50.32407
> School of Advanced Professional Studies       Ulm University, Room: 1013
> https://wissenschaftliche-weiterbildung.org/    Oberberghof 7, 89081 Ulm
> https://saps.uni-ulm.de/                                         Germany
> 

Re: Potential Broadcast Issues After GTK Key Exchange on ath11k with IPQ8072A (QCN5024/QCN5054)

[Sebastian Gottschall]

Hi Remi

Am 19.12.2024 um 16:35 schrieb Remi Pommarel:
> Hi Steffen.
>
> On Thu, Dec 19, 2024 at 04:02:30PM +0100, Steffen Moser wrote:
>> Hello everyone,
>>
>> I've encountered a possible issue in a DD-WRT [1] setup where broadcast
>> packets stop being delivered after a GTK (Group Temporal Key) exchange. This
>> issue occurs on a system with the following hardware:
>>
>>      Access Point Hardware: DynaLink DL-WRX36
>>      Router Software: DD-WRT v3.0-r58819 std (12/13/24)
>>      CPU: Qualcomm IPQ8072A
>>      WiFi Chips: Qualcomm QCN5024 and Qualcomm QCN5054
>>      WiFi Driver: ath11k
>>      Firmware: WLAN.HK.2.12-01460-QCAHKSWPL_SILICONZ-1
>>      NSS FW version: NSS.FW.12.5-210-HK.R
>>      Kernel: Linux WL-AP-EG 6.6.64-rt29 #1791 SMP Thu Dec 12 16:41:51 +07
>> 2024 aarch64 DD-WRT
>>
>> The behavior is such that after a GTK exchange, the AP can get into a "weird
>> state". When being there, broadcast frames like ARP or mDNS are no longer
>> reliably delivered to connected clients while unicasts come still through.
>> In this "weird state", the channel quality (active time vs. busy time) goes
>> down and latencies to the still reachable WIFI clients rise.
> This looks a lot like an issue we hit a while back. There is this patch
> [0] from Qualcomm's wlan-open repository. It is a revert of [1]. Using
> that the issue was never reproduced. Maybe this can help.
>
> Also adding ath11k list.
>
> Regards.

thank you for that hint. i  will integrate it into dd-wrt asap

Sebastian

>

Re: Potential Broadcast Issues After GTK Key Exchange on ath11k with IPQ8072A (QCN5024/QCN5054)

[Steffen Moser]

Dear Remi,

thank you very much for the pointer to the patch. Sebastian integrated 
it into DD-WRT. Now the DynaLink DL-WRX36 runs absolutely smoothly, the 
WLAN links are stable, the packet loss is gone. No weird states anymore, 
independent from the group key exchange interval:

https://data.saps.uni-ulm.de/index.php/s/NLNpWqjc8iGsaEM

So your idea was a direct hit! Thank you very, very much. Several months 
of debugging have come to an end...

Thank you very much and all the best for 2025!

Kind regards,
Steffen

On 19.12.24 4:35 PM, Remi Pommarel wrote:
> Hi Steffen.
> 
> On Thu, Dec 19, 2024 at 04:02:30PM +0100, Steffen Moser wrote:
>> Hello everyone,
>> 
>> I've encountered a possible issue in a DD-WRT [1] setup where broadcast
>> packets stop being delivered after a GTK (Group Temporal Key) exchange. This
>> issue occurs on a system with the following hardware:
>> 
>>     Access Point Hardware: DynaLink DL-WRX36
>>     Router Software: DD-WRT v3.0-r58819 std (12/13/24)
>>     CPU: Qualcomm IPQ8072A
>>     WiFi Chips: Qualcomm QCN5024 and Qualcomm QCN5054
>>     WiFi Driver: ath11k
>>     Firmware: WLAN.HK.2.12-01460-QCAHKSWPL_SILICONZ-1
>>     NSS FW version: NSS.FW.12.5-210-HK.R
>>     Kernel: Linux WL-AP-EG 6.6.64-rt29 #1791 SMP Thu Dec 12 16:41:51 +07
>> 2024 aarch64 DD-WRT
>> 
>> The behavior is such that after a GTK exchange, the AP can get into a "weird
>> state". When being there, broadcast frames like ARP or mDNS are no longer
>> reliably delivered to connected clients while unicasts come still through.
>> In this "weird state", the channel quality (active time vs. busy time) goes
>> down and latencies to the still reachable WIFI clients rise.
> 
> This looks a lot like an issue we hit a while back. There is this patch
> [0] from Qualcomm's wlan-open repository. It is a revert of [1]. Using
> that the issue was never reproduced. Maybe this can help.
> 
> Also adding ath11k list.
> 
> Regards.
> 

Re: Potential Broadcast Issues After GTK Key Exchange on ath11k with IPQ8072A (QCN5024/QCN5054)

[Nicolas Escande]

On Sat Dec 28, 2024 at 11:13 AM CET, Steffen Moser wrote:
> Dear Remi,
>
> thank you very much for the pointer to the patch. Sebastian integrated 
> it into DD-WRT. Now the DynaLink DL-WRX36 runs absolutely smoothly, the 
> WLAN links are stable, the packet loss is gone. No weird states anymore, 
> independent from the group key exchange interval:
>
> https://data.saps.uni-ulm.de/index.php/s/NLNpWqjc8iGsaEM
>
> So your idea was a direct hit! Thank you very, very much. Several months 
> of debugging have come to an end...

So this is at least the second time this commit breaks a setup.
 
@ath11k why isn't this pushed to mainline ?
This seems to be a clear regression, so even if there is no need to rush things
in the long run this still needs to to reverted mainline right ?

>
> Thank you very much and all the best for 2025!
>
> Kind regards,
> Steffen
>
> On 19.12.24 4:35 PM, Remi Pommarel wrote:
> > Hi Steffen.
> > 
> > On Thu, Dec 19, 2024 at 04:02:30PM +0100, Steffen Moser wrote:
> >> Hello everyone,
> >> 
> >> I've encountered a possible issue in a DD-WRT [1] setup where broadcast
> >> packets stop being delivered after a GTK (Group Temporal Key) exchange. This
> >> issue occurs on a system with the following hardware:
> >> 
> >>     Access Point Hardware: DynaLink DL-WRX36
> >>     Router Software: DD-WRT v3.0-r58819 std (12/13/24)
> >>     CPU: Qualcomm IPQ8072A
> >>     WiFi Chips: Qualcomm QCN5024 and Qualcomm QCN5054
> >>     WiFi Driver: ath11k
> >>     Firmware: WLAN.HK.2.12-01460-QCAHKSWPL_SILICONZ-1
> >>     NSS FW version: NSS.FW.12.5-210-HK.R
> >>     Kernel: Linux WL-AP-EG 6.6.64-rt29 #1791 SMP Thu Dec 12 16:41:51 +07
> >> 2024 aarch64 DD-WRT
> >> 
> >> The behavior is such that after a GTK exchange, the AP can get into a "weird
> >> state". When being there, broadcast frames like ARP or mDNS are no longer
> >> reliably delivered to connected clients while unicasts come still through.
> >> In this "weird state", the channel quality (active time vs. busy time) goes
> >> down and latencies to the still reachable WIFI clients rise.
> > 
> > This looks a lot like an issue we hit a while back. There is this patch
> > [0] from Qualcomm's wlan-open repository. It is a revert of [1]. Using
> > that the issue was never reproduced. Maybe this can help.
> > 
> > Also adding ath11k list.
> > 
> > Regards.
> > 

Re: Potential Broadcast Issues After GTK Key Exchange on ath11k with IPQ8072A (QCN5024/QCN5054)

[Kalle Valo]

"Nicolas Escande" <nico.escande@gmail.com> writes:

> On Sat Dec 28, 2024 at 11:13 AM CET, Steffen Moser wrote:
>> Dear Remi,
>>
>> thank you very much for the pointer to the patch. Sebastian integrated 
>> it into DD-WRT. Now the DynaLink DL-WRX36 runs absolutely smoothly, the 
>> WLAN links are stable, the packet loss is gone. No weird states anymore, 
>> independent from the group key exchange interval:
>>
>> https://data.saps.uni-ulm.de/index.php/s/NLNpWqjc8iGsaEM
>>
>> So your idea was a direct hit! Thank you very, very much. Several months 
>> of debugging have come to an end...
>
> So this is at least the second time this commit breaks a setup.
>  
> @ath11k why isn't this pushed to mainline ?
> This seems to be a clear regression, so even if there is no need to rush things
> in the long run this still needs to to reverted mainline right ?

Good question, I don't have an answer to that. Could someone (also
outside of Qualcomm) send a proper patch ASAP so that we can solve this?
And it's good ot include the link to this discussion and describe the
symptoms the revert is fixing.

This is the patch in question:

https://git.codelinaro.org/clo/qsdk/oss/system/feeds/wlan-open/-/blob/win.wlan_host_opensource.3.0.r24/patches/ath11k/350-ath11k-Revert-clear-the-keys-properly-when-DISABLE_K.patch

-- 
https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

Re: Potential Broadcast Issues After GTK Key Exchange on ath11k with IPQ8072A (QCN5024/QCN5054)

[Nicolas Escande]

On Thu Jan 9, 2025 at 2:25 PM CET, Kalle Valo wrote:
[...]
>
> Good question, I don't have an answer to that. Could someone (also
> outside of Qualcomm) send a proper patch ASAP so that we can solve this?
> And it's good ot include the link to this discussion and describe the
> symptoms the revert is fixing.
>
> This is the patch in question:
>
> https://git.codelinaro.org/clo/qsdk/oss/system/feeds/wlan-open/-/blob/win.wlan_host_opensource.3.0.r24/patches/ath11k/350-ath11k-Revert-clear-the-keys-properly-when-DISABLE_K.patch

I'm not so sure as how to proceed to post a patch from someone else while
editing the commit message to add more info...
Rameshkumar Sundaram is surely the best person to do it, no ? 

Thanks

Re: Potential Broadcast Issues After GTK Key Exchange on ath11k with IPQ8072A (QCN5024/QCN5054)

[Nicolas Escande]

On Thu Jan 9, 2025 at 2:25 PM CET, Kalle Valo wrote:
> "Nicolas Escande" <nico.escande@gmail.com> writes:
>
> > On Sat Dec 28, 2024 at 11:13 AM CET, Steffen Moser wrote:
> >> Dear Remi,
> >>
> >> thank you very much for the pointer to the patch. Sebastian integrated 
> >> it into DD-WRT. Now the DynaLink DL-WRX36 runs absolutely smoothly, the 
> >> WLAN links are stable, the packet loss is gone. No weird states anymore, 
> >> independent from the group key exchange interval:
> >>
> >> https://data.saps.uni-ulm.de/index.php/s/NLNpWqjc8iGsaEM
> >>
> >> So your idea was a direct hit! Thank you very, very much. Several months 
> >> of debugging have come to an end...
> >
> > So this is at least the second time this commit breaks a setup.
> >  
> > @ath11k why isn't this pushed to mainline ?
> > This seems to be a clear regression, so even if there is no need to rush things
> > in the long run this still needs to to reverted mainline right ?
>
> Good question, I don't have an answer to that. Could someone (also
> outside of Qualcomm) send a proper patch ASAP so that we can solve this?
> And it's good ot include the link to this discussion and describe the
> symptoms the revert is fixing.
>
> This is the patch in question:
>
> https://git.codelinaro.org/clo/qsdk/oss/system/feeds/wlan-open/-/blob/win.wlan_host_opensource.3.0.r24/patches/ath11k/350-ath11k-Revert-clear-the-keys-properly-when-DISABLE_K.patch

Not sure if I did it the right way but it at least is here :
https://lore.kernel.org/linux-wireless/20250117191455.3395145-1-nico.escande@gmail.com/

Re: Potential Broadcast Issues After GTK Key Exchange on ath11k with IPQ8072A (QCN5024/QCN5054)

[Steffen Moser]

Hi,

I’m not a kernel developer myself, but from a formal perspective, it 
looks good to me. Unfortunately, I don't really know what the patch was 
for originally. Maybe reverting it brakes something.

Best regards,
Steffen


On 17.01.25 8:17 PM, Nicolas Escande wrote:
> On Thu Jan 9, 2025 at 2:25 PM CET, Kalle Valo wrote:
>> "Nicolas Escande" <nico.escande@gmail.com> writes:
>>
>> > On Sat Dec 28, 2024 at 11:13 AM CET, Steffen Moser wrote:
>> >> Dear Remi,
>> >>
>> >> thank you very much for the pointer to the patch. Sebastian integrated 
>> >> it into DD-WRT. Now the DynaLink DL-WRX36 runs absolutely smoothly, the 
>> >> WLAN links are stable, the packet loss is gone. No weird states anymore, 
>> >> independent from the group key exchange interval:
>> >>
>> >> https://data.saps.uni-ulm.de/index.php/s/NLNpWqjc8iGsaEM
>> >>
>> >> So your idea was a direct hit! Thank you very, very much. Several months 
>> >> of debugging have come to an end...
>> >
>> > So this is at least the second time this commit breaks a setup.
>> >  
>> > @ath11k why isn't this pushed to mainline ?
>> > This seems to be a clear regression, so even if there is no need to rush things
>> > in the long run this still needs to to reverted mainline right ?
>>
>> Good question, I don't have an answer to that. Could someone (also
>> outside of Qualcomm) send a proper patch ASAP so that we can solve this?
>> And it's good ot include the link to this discussion and describe the
>> symptoms the revert is fixing.
>>
>> This is the patch in question:
>>
>> https://git.codelinaro.org/clo/qsdk/oss/system/feeds/wlan-open/-/blob/win.wlan_host_opensource.3.0.r24/patches/ath11k/350-ath11k-Revert-clear-the-keys-properly-when-DISABLE_K.patch
> 
> Not sure if I did it the right way but it at least is here :
> https://lore.kernel.org/linux-wireless/20250117191455.3395145-1-nico.escande@gmail.com/
>