Re: Connection loss (IWD HEAD with latest OWE / BSS selection patches) - brcmfmac driver

Martin Petzold
2024-11-05T15:17:04Z

Dear James, dear Arend,

Am 05.11.24 um 14:14 schrieb James Prestwood:
> Hi Martin,
>
> On 11/4/24 3:20 PM, James Prestwood wrote:
>> Hi Martin,
>>
>> On 11/4/24 2:42 PM, Martin Petzold wrote:
>>> Dear James,
>>>
>>> Am 04.11.24 um 13:36 schrieb James Prestwood:
>>>>
>>>> On 11/3/24 3:13 PM, Martin Petzold wrote:
>>>>> Dear James,
>>>>>
>>>>> Am 25.10.24 um 17:17 schrieb James Prestwood:
>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I open a new thread for this one: During the last weeks I
>>>>>>>>>>>>>> have seen connection losses for 30+ minutes, sometimes
>>>>>>>>>>>>>> even hours or just now even forever (IWD HEAD with v2 OWE
>>>>>>>>>>>>>> / BSS selection patches). Driver is brcmfmac (NXP 6.1.36
>>>>>>>>>>>>>> kernel) and chip is BCM4339 (Laird LWB5).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It happens in a) single router environment (WPA2-PSK;
>>>>>>>>>>>>>> Touchstone TG3442DE), and b) router + repeater
>>>>>>>>>>>>>> environment (WPA2 CCMP; Fritz!Box + Fritz!Repeater), and
>>>>>>>>>>>>>> maybe also in the WPA3 OWE Transition network (yesterday
>>>>>>>>>>>>>> lost a connection again).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I lost now again 2 of 10 devices in the WPA3 OWE network
>>>>>>>>>>>>> (with roaming). However, now they don't disappear all
>>>>>>>>>>>>> after a shorter while. It seems to be later.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I also lost one device in a Router+Repeater WPA2 (CCMP)
>>>>>>>>>>>>> network. It is confirmed here on router side, that the
>>>>>>>>>>>>> device is disconnected. Since more than a day.
>>>>>>>>>>>>
>>>>>>>>>>>> We can't do anything without logs. If you suspect its the
>>>>>>>>>>>> blacklist you can lower the blacklist time down in main.conf:
>>>>>>>>>>>>
>>>>>>>>>>>> [
>>>>>
>>>>> I am still losing devices. Sometimes they come back again, but
>>>>> mostly do not re-connect. I have observed the following:
>>>>>
>>>>> - Connection exists for several hours until about one day, or two.
>>>>> Then gone for several hours or mostly forever.
>>>>> - For FritzBox+FritzRepeater I have seen the connection coming
>>>>> back after like a day (here connection loss was also confirmed on
>>>>> router side!)
>>>>> - For the Aruba enterprise environment the connection never came
>>>>> back (until now no AP logs - waiting for an answer)
>>>>> - After reboot the connection comes back
>>>>> - It occurs only in an environment with multiple APs with same
>>>>> SSID (i.e. roaming environment), however my single AP environments
>>>>> have all strong signal
>>>>> - Some devices with identical configuration in this environment DO
>>>>> NOT get lost, those seem to have quite strong signal (maybe they
>>>>> don't roam)
>>>>> - Other devices in the same environment work without any problems
>>>>> (Intel+NetworkManager) and the APs are Aruba enterprise grade
>>>>> - I see almost the same in the Aruba enterprise environment, but
>>>>> ALSO in a FritzBox + FritzRepeater environment
>>>>> - We had a bug in our web socket connection, causing to many IWD
>>>>> requests. However, this was fixed. And why are all the other
>>>>> devices okay? Maybe co-incidence with roaming and anything related
>>>>> to dropping and re-connecting web socket connection.
>>>>>
>>>>> Please find attached my currently available debug logs (they are a
>>>>> few days old, but I am quite sure this is the connection loss
>>>>> situation). These logs are from the FritzBox+FritzRepeater
>>>>> environment. There are no brcmfmac messages (but also no special
>>>>> debug level configured here)!
>>>>>
>>>>> I have now also disabled WiFi power saving and will deploy to the
>>>>> environment...hoping the best.
>>>>>
>>>>> Maybe you could check the logs and have an idea?
>>>>
>>>> Looks like the same thing as the last logs you sent. IWD tries to
>>>> connect (sends CMD_CONNECT to the kernel) but gets no associated
>>>> CMD_CONNECT event after that which causes IWD to wait indefinitely
>>>> for that event. This, again, appears like a driver problem because
>>>> its expected that the kernel tells userspace the result of the
>>>> CMD_CONNECT request.
>>>>
>>>> Only similarity I can see between the two sets of logs is there is
>>>> a failed connection just prior to the hang. IWD then attempts to
>>>> connect again but the 4-way handshake is never started and this
>>>> results in a failure with status 16 (group key handshake timeout).
>>>> In your latest set of logs IWD actually again tries to connect to a
>>>> different BSS and gets status 16 before trying yet again and hanging.
>>>>
>>>> This actually seems similar to an issue I encountered with ath10k
>>>> where the network interface would time out being brought up.
>>>> Retrying would succeed but the driver would be in a similar state
>>>> where IWD could authenticate/associate but no data frames (i.e.
>>>> 4-way handshake) would be passed to userspace. Only solution (until
>>>> upstream fixed the bug) was to unload/reload the driver when we
>>>> detected this condition.
>>>>
>>>> If you are able to physically attach to a device currently in this
>>>> state you may be able to get more info. For example if IWD is stuck
>>>> like this try disconnecting/reconnecting with iwctl or restarting
>>>> IWD to see what happens. If you end up in the same state right away
>>>> I'm 99.9% sure the driver is the entire reason your running into this.
>>>
>>> Are you sure? Maybe you could double-check?
>> I'm sure.
>>>
>>> Because my SOM vendor (Variscite) selling a few hundred thousand of
>>> these do not report any issue with this kernel, firmware, and
>>> NetworkManager (wpa_supplicant)...
>>
>> Because wpa_supplicant sets internal timers for these commands in
>> case the driver is broken. I would expect you would see this exact
>> behavior with wpa_supplicant, it would just disconnect/reconnect
>> after 5 seconds of no response from the kernel. And something like
>> this either a) goes entirely unnoticed and/or b) works well enough
>> for a hardware vendor to ship it to customers and not care.
>>
>> This is the commit adding these timers to wpa_supplicant:
>>
>> commit e29853bbff1eef781099a9108e3b51f26b477ac3
>> Author: Ben Greear <greearb@candelatech.com>
>> Date:   Thu Feb 24 16:59:46 2011 +0200
>>
>>     SME: Add timers for authentication and asscoiation
>>
>>     mac80211 authentication or association operation may get stuck
>> for some
>>     reasons, so wpa_supplicant better use an internal timer to
>> recover from
>>     this.
>>
>>     Signed-off-by: Ben Greear <greearb@candelatech.com>
>>
>> I wish it surprised me that 13 years later this behavior still
>> happens... We don't like adding special driver workarounds like this
>> in IWD because a) it becomes difficult to maintain and b) it just
>> hides the root cause and nobody ever fixes it. But my opinions aside,
>> for a driver like brcmfmac which is very mainstream, I guess we have
>> no choice but to adapt IWD to work around it like wpa_supplicant does.
>>
>> Thanks,
>>
>> James
>
> I've sent a patch to the list which sets a timer within IWD in case
> the connect event never arrives. Note that I cannot test this beyond
> manually commenting out code to "trick" IWD into thinking this
> happens. Applying that patch to the brcmfmac client your using is
> going to be the true test.

I may be able to test it, however only together with wifi power save
disabled (I also prepared NetworkManager branch, because the customer
will kill me otherwise).

@Arent: Maybe you could check all this, as it seems to be related to
some brcmfmac state. Just now I cannot provide logs, and your debug
level for brcmfmac produces a lot lot of data, which I somehow also need
to handle (limited space).

Best regards,

Martin