Re: [BUG] mt7921e: Intermittent connection failure

Re: [BUG] mt7921e: Intermittent connection failure

[Sean Wang]

Hi,

On Tue, Mar 31, 2026 at 2:40 AM silverducks <silverducks@altlinux.org> wrote:
>
> Greetings!
>
> I apologize for poor formatting in the previous email. I did not realize all
> plain text files' contents would be visible on the mailing list.
> I am attaching an archive containing the same files as in previous email for
> convenience.
> Given compression, I can also avoid using external hosting, which I
> presume is
> preferred, so I am including all relevant logs in the archive as well.
> I am also including original email text just in case.

I think the current test setup is still mixing too many variables, so
it is hard to tell what is actually triggering the issue.

In particular, if the goal is to test the NetworkManager path, the
script should not also manually manage wpa_supplicant, and iwd should
not be part of the same test either. NetworkManager normally manages
the Wi-Fi backend itself, so mixing manual wpa_supplicant handling,
iwd, and NetworkManager in one setup makes the result difficult to
interpret.

Could you first simplify the setup and test one path at a time?

If you want to test NetworkManager, use only NetworkManager, for
example by using nmcli to explicitly control the connection steps.
If you want to test plain wpa_supplicant, stop NetworkManager
completely and use only wpa_supplicant + wpa_cli. I would suggest
starting with this path, since that is also the setup I usually use
for testing.
If you want to test iwd, please test it separately as well.
Also suggest to avoid suspend/resume or hibernation for now.
The log you shared includes a clear S4 resume path (ACPI: PM: Waking
up from system sleep state S4 and pci_pm_restore returns -110), which
does not match a simple reconnect or module reload test.

Re: [BUG] mt7921e: Intermittent connection failure

[silverducks]

[-- Attachment #1: Type: text/plain, Size: 4164 bytes --]

On 02/04/2026 01:58, Sean Wang wrote:
 > I think the current test setup is still mixing too many variables, so
 > it is hard to tell what is actually triggering the issue.
 >
 > In particular, if the goal is to test the NetworkManager path, the
 > script should not also manually manage wpa_supplicant, and iwd should
 > not be part of the same test either. NetworkManager normally manages
 > the Wi-Fi backend itself, so mixing manual wpa_supplicant handling,
 > iwd, and NetworkManager in one setup makes the result difficult to
 > interpret.
 > Could you first simplify the setup and test one path at a time?
 > If you want to test NetworkManager, use only NetworkManager, for
 > example by using nmcli to explicitly control the connection steps.
 > If you want to test plain wpa_supplicant, stop NetworkManager
 > completely and use only wpa_supplicant + wpa_cli. I would suggest
 > starting with this path, since that is also the setup I usually use
 > for testing.
 > If you want to test iwd, please test it separately as well.
Some confusion here: I have indeed tested them separately. I left in
wpa_supplicant start in that particular script by mistake. Extra stop 
commands
for iwd and supplicant at the end of the loop are also redundant. I 
don't think
either affects outcome of the tests, but just in case, I'll clean up the 
test
scripts and rerun.
I am also attaching all scripts used alongside the resulting logs for 
clarity.

 > Also suggest to avoid suspend/resume or hibernation for now.
 > The log you shared includes a clear S4 resume path (ACPI: PM: Waking
 > up from system sleep state S4 and pci_pm_restore returns -110), which
 > does not match a simple reconnect or module reload test.
Those are the logs I collected when originally diagnosing the issue. Scripts
reload kernel module instead because I since realised that this is enough to
trigger the bug and it's much faster to do it this way when searching for a
failstate.
I am attaching logs where module reload is used for all cases.

------------
New data
------------
After some additional testing, I noticed simply reconnecting enough 
times seems
to trigger the bug as well. I'll attach an additional script for this too.
Given this, I've looked through the code relevant to connection process. I
think I may have found something of interest. The following patch gets 
rid of
original lockup state, but introduces a new error into dmesg.

---
  drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c 
b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c
index 462b4a68c4f0..47621ff13839 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c
@@ -1788,7 +1788,7 @@ int mt76_connac_mcu_cancel_hw_scan(struct mt76_phy 
*phy,
      }

      return mt76_mcu_send_msg(phy->dev, MCU_CE_CMD(CANCEL_HW_SCAN),
-                 &req, sizeof(req), false);
+                 &req, sizeof(req), true);
  }
  EXPORT_SYMBOL_GPL(mt76_connac_mcu_cancel_hw_scan);

-- 
2.50.1

Seems like making this call wait for the response is fixing the issue at 
least
partially. Judging from the relevant logs, it effectively makes the driver
reset on it's own whenever there's a problem.
Logs of two runs with a kernel compiled with this patch are attached.
One for reconnecting variant, one for module reload variant.

------------
Attachments
------------
I've structured the archive with a subfolder for each testing run, each
containing the relevant logs and the script, if applicable. Overview:
iwd and wpa_supplicant -only setups work as expected.
NetworkManager with module reload ends up in failstate.
NetworkManager with reconnection ends up in failstate.
Both of the latter start to have additional driver reloads instead of 
locking
when run with patched kernel.
Finally, I am also attaching a case where problem occurred immediately
after boot.

All failed cases were ended after confirming the failstate.
All successful cases were interrupted manually after a few minutes.

[-- Attachment #2: attachments.tar.gz --]
[-- Type: application/gzip, Size: 712828 bytes --]

Re: [BUG] mt7921e: Intermittent connection failure

[silverducks]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

Greetings!

Update:
New version of the script is less consistent at reproducing the bug.
I've also noticed that on patched kernel it triggers timeout way
more often than older script. I've rolled back to the older one,
which relies on the NM autoconnect feature, though retained cleanup
changes.
I'm guessing direct nmcli command is often preventing NM from doing
whatever it is usually doing that triggers the error.
Script and reruns with it attached.

[-- Attachment #2: attachments.tar.gz --]
[-- Type: application/gzip, Size: 158890 bytes --]

Re: [BUG] mt7921e: Intermittent connection failure

[Sean Wang]

Hi,

Thanks for the update and for sharing the revised script and rerun logs.
I tried to reproduce the issue with your script on an mt7921u, but I
could not reproduce the timeout on my side.
In my run, I got 398 reload cycles and 398 successful reconnects:
grep Reloading test_log.txt | wc -l
388
grep Connection test_log.txt | wc -l
398
The log is consistently like this after module reload [1]
If this were a generic race in the common MCU command layer, I would
expect it to reproduce on mt7921u as well.
Also, could you record and save the "iw event" log when you run the
test? That would help show what userspace-triggered
activities are happening. we can mimic the same sequence using wpa_cli

[1]
Starting WiFi mt7921u reset loop (using NetworkManager)
Making sure network services are down...
----------------------------------------
Reloading module mt7921u...
Starting network services...
Checking for connection...
Checking for connection...
Connected.
Connection succeeded. Cleaning up.
----------------------------------------
Reloading module mt7921u...
Starting network services...
Checking for connection...
Checking for connection...
Connected.
Connection succeeded. Cleaning up.
----------------------------------------
Reloading module mt7921u...
Starting network services...
Checking for connection...
Checking for connection...
Connected.
Connection succeeded. Cleaning up.
----------------------------------------
Reloading module mt7921u...
Starting network services...
Checking for connection...
Checking for connection...
Status: Connecting
Checking for connection...
Connected.
Connection succeeded. Cleaning up.
----------------------------------------
Reloading module mt7921u...
Starting network services...
Checking for connection...
Checking for connection...
Connected.
Connection succeeded. Cleaning up.
----------------------------------------
Reloading module mt7921u...
Starting network services...
Checking for connection...
Checking for connection...
Status: Connecting
Checking for connection...
Connected.
Connection succeeded. Cleaning up.
----------------------------------------
Reloading module mt7921u...
Starting network services...
Checking for connection...
Checking for connection...
Connected.
Connection succeeded. Cleaning up.
----------------------------------------
Reloading module mt7921u...
Starting network services...
Checking for connection...
Checking for connection...
Connected.
Connection succeeded. Cleaning up.
----------------------------------------
Reloading module mt7921u...
Starting network services...
Checking for connection...
Checking for connection...
Connected.
Connection succeeded. Cleaning up.
----------------------------------------
Reloading module mt7921u...
Starting network services...
Checking for connection...
Checking for connection...
Connected.
Connection succeeded. Cleaning up.
----------------------------------------
...
and so on

On Tue, Apr 14, 2026 at 1:22 AM silverducks <silverducks@altlinux.org> wrote:
>
> Greetings!
>
> Update:
> New version of the script is less consistent at reproducing the bug.
> I've also noticed that on patched kernel it triggers timeout way
> more often than older script. I've rolled back to the older one,
> which relies on the NM autoconnect feature, though retained cleanup
> changes.
> I'm guessing direct nmcli command is often preventing NM from doing
> whatever it is usually doing that triggers the error.
> Script and reruns with it attached.

Re: [BUG] mt7921e: Intermittent connection failure

[silverducks]

[-- Attachment #1: Type: text/plain, Size: 3853 bytes --]

Greetings!

On 17/04/2026 00:59, Sean Wang wrote:
 > I tried to reproduce the issue with your script on an mt7921u, but I
 > could not reproduce the timeout on my side.
 > <...>
 > If this were a generic race in the common MCU command layer, I would
 > expect it to reproduce on mt7921u as well.
It sounds like you are referring to the original bug as timeout here.
I may not have been clear about this, let me restate it more
precisely. My apologies if this is redundant.
--------------
mt7921e 0000:02:00.0: Message 0004001b (seq 9) timeout
--------------
This timeout error appears with aforementioned patch applied.
It seems to "replace" the bug during NM-autoconnect-reliant runs -
it's occurrence rate is the same and the original bug no longer
triggers. It causes the driver to reset:
--------------
[  124.601521] mt7921e 0000:02:00.0: Message 0004001b (seq 5) timeout
[  124.689461] mt7921e 0000:02:00.0: HW/SW Version: 0x8a108a10, Build 
Time: 20250625153620a

[  124.726504] mt7921e 0000:02:00.0: WM Firmware Version: ____010000, 
Build Time: 20250625153703
--------------
On the other hand, the original bug makes the device unresponsive,
but does not cause an error message or cause the driver to reset [1].

If I use nmcli to connect instead (as in previous revision of the
script) with the patch applied, this timeout starts to appear
significantly more frequently. It's not impossible that it's a
statistical outlier, but I did run this a number of times.

Without the patch, using nmcli to connect mostly gets rid of the bug:
occurrence rate drops considerably, though not to zero.
I think it simply allows less time for NM to trigger the bug.

 > If this were a generic race in the common MCU command layer, I would
 > expect it to reproduce on mt7921u as well.
Perhaps it could still exist in the common layer, if the USB version
is not triggering it due to slower bus speed or latency, or
some other difference?
Though, I'm not saying that aforementioned patch fixes the core issue.
More likely it only prevents the trigger from occurring.

 > Also, could you record and save the "iw event" log when you run the
 > test? That would help show what userspace-triggered
 > activities are happening. we can mimic the same sequence using wpa_cli
Of course. Attaching logs for module reload and reconnecting cases.

Thank you for your time.

[1]
<cut>
[  208.992595] wlp2s0: deauthenticating from 34:60:f9:99:08:38 by local 
choice (Reason: 3=DEAUTH_LEAVING)
[  209.611215] mt7921e 0000:02:00.0: ASIC revision: 79610010
[  209.697087] mt7921e 0000:02:00.0: HW/SW Version: 0x8a108a10, Build 
Time: 20250625153620a

[  209.734750] mt7921e 0000:02:00.0: WM Firmware Version: ____010000, 
Build Time: 20250625153703
[  210.579495] mt7921e 0000:02:00.0 wlp2s0: renamed from wlan0
[  215.603597] wlp2s0: authenticate with 34:60:f9:99:08:38 (local 
address=14:5a:fc:77:82:1f)
[  215.718389] wlp2s0: send auth to 34:60:f9:99:08:38 (try 1/3)
[  218.790084] wlp2s0: send auth to 34:60:f9:99:08:38 (try 2/3)
[  220.723047] wlp2s0: aborting authentication with 34:60:f9:99:08:38 by 
local choice (Reason: 3=DEAUTH_LEAVING)
[  234.992512] wlp2s0: authenticate with 34:60:f9:99:08:38 (local 
address=14:5a:fc:77:82:1f)
[  236.005909] wlp2s0: send auth to 34:60:f9:99:08:38 (try 1/3)
[  239.078085] wlp2s0: send auth to 34:60:f9:99:08:38 (try 2/3)
[  240.695078] wlp2s0: aborting authentication with 34:60:f9:99:08:38 by 
local choice (Reason: 3=DEAUTH_LEAVING)
[  254.926181] wlp2s0: authenticate with 34:60:f9:99:08:38 (local 
address=14:5a:fc:77:82:1f)
[  255.973995] wlp2s0: send auth to 34:60:f9:99:08:38 (try 1/3)
[  257.061953] wlp2s0: send auth to 34:60:f9:99:08:38 (try 2/3)
[  258.149899] wlp2s0: send auth to 34:60:f9:99:08:38 (try 3/3)
[  258.173202] wlp2s0: authentication with 34:60:f9:99:08:38 timed out
<end of log>


[-- Attachment #2: attachments.tar.gz --]
[-- Type: application/gzip, Size: 169901 bytes --]

Re: [BUG] mt7921e: Intermittent connection failure

[silverducks]

Greetings!

Update:
As of kernel version 7.1-rc4 the error is no longer reproduced.