Re: Connection loss (IWD HEAD with latest OWE / BSS selection patches) - brcmfmac driver

[James Prestwood <prestwoj@gmail.com>]

On 11/3/24 3:13 PM, Martin Petzold wrote:
> Dear James,
>
> Am 25.10.24 um 17:17 schrieb James Prestwood:
>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I open a new thread for this one: During the last weeks I 
>>>>>>>>>> have seen connection losses for 30+ minutes, sometimes even 
>>>>>>>>>> hours or just now even forever (IWD HEAD with v2 OWE / BSS 
>>>>>>>>>> selection patches). Driver is brcmfmac (NXP 6.1.36 kernel) 
>>>>>>>>>> and chip is BCM4339 (Laird LWB5).
>>>>>>>>>>
>>>>>>>>>> It happens in a) single router environment (WPA2-PSK; 
>>>>>>>>>> Touchstone TG3442DE), and b) router + repeater environment 
>>>>>>>>>> (WPA2 CCMP; Fritz!Box + Fritz!Repeater), and maybe also in 
>>>>>>>>>> the WPA3 OWE Transition network (yesterday lost a connection 
>>>>>>>>>> again).
>>>>>>>>>
>>>>>>>>> I lost now again 2 of 10 devices in the WPA3 OWE network (with 
>>>>>>>>> roaming). However, now they don't disappear all after a 
>>>>>>>>> shorter while. It seems to be later.
>>>>>>>>>
>>>>>>>>> I also lost one device in a Router+Repeater WPA2 (CCMP) 
>>>>>>>>> network. It is confirmed here on router side, that the device 
>>>>>>>>> is disconnected. Since more than a day.
>>>>>>>>
>>>>>>>> We can't do anything without logs. If you suspect its the 
>>>>>>>> blacklist you can lower the blacklist time down in main.conf:
>>>>>>>>
>>>>>>>> [
>
> I am still losing devices. Sometimes they come back again, but mostly 
> do not re-connect. I have observed the following:
>
> - Connection exists for several hours until about one day, or two. 
> Then gone for several hours or mostly forever.
> - For FritzBox+FritzRepeater I have seen the connection coming back 
> after like a day (here connection loss was also confirmed on router 
> side!)
> - For the Aruba enterprise environment the connection never came back 
> (until now no AP logs - waiting for an answer)
> - After reboot the connection comes back
> - It occurs only in an environment with multiple APs with same SSID 
> (i.e. roaming environment), however my single AP environments have all 
> strong signal
> - Some devices with identical configuration in this environment DO NOT 
> get lost, those seem to have quite strong signal (maybe they don't roam)
> - Other devices in the same environment work without any problems 
> (Intel+NetworkManager) and the APs are Aruba enterprise grade
> - I see almost the same in the Aruba enterprise environment, but ALSO 
> in a FritzBox + FritzRepeater environment
> - We had a bug in our web socket connection, causing to many IWD 
> requests. However, this was fixed. And why are all the other devices 
> okay? Maybe co-incidence with roaming and anything related to dropping 
> and re-connecting web socket connection.
>
> Please find attached my currently available debug logs (they are a few 
> days old, but I am quite sure this is the connection loss situation). 
> These logs are from the FritzBox+FritzRepeater environment. There are 
> no brcmfmac messages (but also no special debug level configured here)!
>
> I have now also disabled WiFi power saving and will deploy to the 
> environment...hoping the best.
>
> Maybe you could check the logs and have an idea?

Looks like the same thing as the last logs you sent. IWD tries to 
connect (sends CMD_CONNECT to the kernel) but gets no associated 
CMD_CONNECT event after that which causes IWD to wait indefinitely for 
that event. This, again, appears like a driver problem because its 
expected that the kernel tells userspace the result of the CMD_CONNECT 
request.

Only similarity I can see between the two sets of logs is there is a 
failed connection just prior to the hang. IWD then attempts to connect 
again but the 4-way handshake is never started and this results in a 
failure with status 16 (group key handshake timeout). In your latest set 
of logs IWD actually again tries to connect to a different BSS and gets 
status 16 before trying yet again and hanging.

This actually seems similar to an issue I encountered with ath10k where 
the network interface would time out being brought up. Retrying would 
succeed but the driver would be in a similar state where IWD could 
authenticate/associate but no data frames (i.e. 4-way handshake) would 
be passed to userspace. Only solution (until upstream fixed the bug) was 
to unload/reload the driver when we detected this condition.

If you are able to physically attach to a device currently in this state 
you may be able to get more info. For example if IWD is stuck like this 
try disconnecting/reconnecting with iwctl or restarting IWD to see what 
happens. If you end up in the same state right away I'm 99.9% sure the 
driver is the entire reason your running into this.

Thanks,

James