[bug] Triple whammy of mt7996e bugs on 7.2+

[bug] Triple whammy of mt7996e bugs on 7.2+

[Ilya K]

Hey folks,

Not really sure where to report this, but I've encountered some "interesting" issues on my bpi-r4 running kernel 7.2 and up.

The first issue is that after a while the firmware on the mt7996e sometimes seemingly just locks up. 
As it usually is with those things, there's not much debug info available, so all I get is

[58764.004954] mt7996e 0000:01:00.0: Message 00130022 (seq 12) timeout

This has happened before, but very rarely, and now it seems to happen consistently at least once a day or so, on
the exact same setup that is stable for weeks on end with 7.1. I have not attempted a bisect yet because
1) I kinda need my network to actually work reliably and 2) it happens so inconsistently that it's hard to
pin down a good commit.

The other issue is that the kernel then fully explodes trying to recover from this, in two different ways 
depending on whether WED is enabled or disabled: with WED on, it tries to double free a page somewhere on the 
teardown path, and with WED off it just wedges seemingly all of nl80211 and any process that happened to be talking to it.

I'm attaching the logs for both cases, hopefully this helps figure out what's going on, and happy to do more testing if
needed.

For posterity, my current kernel tree is basically vanilla + just enough changes to get bpi-r4 10G ports working on NixOS:
- 7.1 branch: https://github.com/K900/linux/tree/bpi-r4-71
- 7.2 branch: https://github.com/K900/linux/tree/bpi-r4-72

(there are exactly zero *80211/mt76/mt7996e changes).

Oh, I've also tried pulling the latest firmware from vendor OpenWRT feed and that did not help matters at all.

Any help in figuring this out would be greatly appreciated!

- Ilya

Re: [bug] Triple whammy of mt7996e bugs on 7.2+

[Ben Greear]

On 9/4/26 07:28, Ilya K wrote:
> Hey folks,
> 
> Not really sure where to report this, but I've encountered some "interesting" issues on my bpi-r4 running kernel 7.2 and up.

Hello Ilya,

In case you feel like experimenting, I'm interested to know if this kernel
works better for you.  We have some tx-hang work-arounds for mtk7996, among
other things:

https://github.com/greearb/linux-ct-7.2

We have not tested on bpi-r4 anytime recently, however, and never used WED as far
as I am aware.

Thanks,
Ben

> 
> The first issue is that after a while the firmware on the mt7996e sometimes seemingly just locks up.
> As it usually is with those things, there's not much debug info available, so all I get is
> 
> [58764.004954] mt7996e 0000:01:00.0: Message 00130022 (seq 12) timeout
> 
> This has happened before, but very rarely, and now it seems to happen consistently at least once a day or so, on
> the exact same setup that is stable for weeks on end with 7.1. I have not attempted a bisect yet because
> 1) I kinda need my network to actually work reliably and 2) it happens so inconsistently that it's hard to
> pin down a good commit.
> 
> The other issue is that the kernel then fully explodes trying to recover from this, in two different ways
> depending on whether WED is enabled or disabled: with WED on, it tries to double free a page somewhere on the
> teardown path, and with WED off it just wedges seemingly all of nl80211 and any process that happened to be talking to it.
> 
> I'm attaching the logs for both cases, hopefully this helps figure out what's going on, and happy to do more testing if
> needed.
> 
> For posterity, my current kernel tree is basically vanilla + just enough changes to get bpi-r4 10G ports working on NixOS:
> - 7.1 branch: https://github.com/K900/linux/tree/bpi-r4-71
> - 7.2 branch: https://github.com/K900/linux/tree/bpi-r4-72
> 
> (there are exactly zero *80211/mt76/mt7996e changes).
> 
> Oh, I've also tried pulling the latest firmware from vendor OpenWRT feed and that did not help matters at all.
> 
> Any help in figuring this out would be greatly appreciated!
> 
> - Ilya
> 
> 

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [bug] Triple whammy of mt7996e bugs on 7.2+

[Ilya K]

> https://github.com/greearb/linux-ct-7.2
Hoo boy that's a lot of patches. Anything specific in there, or do you want me to just rebase my changes on that entire branch? I can try, I guess, but probably not today...

Re: [bug] Triple whammy of mt7996e bugs on 7.2+

[Ben Greear]

On 9/4/26 09:11, Ilya K wrote:
>> https://github.com/greearb/linux-ct-7.2
> Hoo boy that's a lot of patches. Anything specific in there, or do you want me to just rebase my changes on that entire branch? I can try, I guess, but probably not today...
> 

I don't have a good answer for you...it *is* a lot of patches, and I'm not sure if any of it
would actually help your system.  We do have this kernel working for an ARM based AP that uses
mtk7996 radio, and at one time it compiled for bpi4, but I'm not sure bpi4 still
works as we do not test it.

If you can get it to compile for your system by adding any needed patches on top, that is likely best
approach, and if that actually solves the problem, we'd be happy to consider adding any bpi4 patches
to our tree.

./build_bpi4.bash may be of some interest...that is how we built it back when we did try to support bpi4.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re:[bug] Triple whammy of mt7996e bugs on 7.2+

[Ilya K]

> If you can get it to compile for your system by adding any needed patches on top, that is likely best
approach, and if that actually solves the problem, we'd be happy to consider adding any bpi4 patches
to our tree.

OK, so curiosity got the better of me, and I ended up with https://github.com/K900/linux/tree/bpi-r4-72-ct, which is kind of a horrible frankentree of my changes rebased on (your tree rebased on 7.2.3 minus bpi-r4 changes).

More specifically, here's the commits from your side that got dropped (courtesy of git range-diff):

 234:  846c09f2d40e <    -:  ------------ arm: dts: mt7623: swap mmc and put uart2 first
 235:  0c7af9986c06 <    -:  ------------ utils: firmware: add r3mini phy firmware
 236:  e977a0c57579 <    -:  ------------ net: sfp: Fixup for OEM SFP-2.5G-T(-R-RM) module
 238:  2510cb2b3cf1 <    -:  ------------ dt-bindings: phy: mediatek,xfi-tphy: add new bindings
 239:  3ae8358d3a61 <    -:  ------------ dt-bindings: mt76: support setting per-band MAC address
 240:  654e6eeb87d9 <    -:  ------------ hwrng: add driver for MediaTek TRNG SMC
 244:  c5a3d787c53f <    -:  ------------ dt-bindings: net: mediatek: remove wrongly added clocks and SerDes
 245:  8387ae94bbd7 <    -:  ------------ dt-bindings: net: mediatek,net: fix and complete mt7988-eth binding
 247:  5ddb838fc0b1 <    -:  ------------ Add some build scripts for combining Adtran SDG-8734v and SDG-8733v in a single fit-multi.itb file
 249:  e34f830292ad <    -:  ------------ net: ethernet: mtk_eth_soc: Enable legacy SRAM behavior for v3 devices
 250:  3a1f3125a1cd <    -:  ------------ arm64: dts: mt7622: add pstore reserved memory
 251:  8e9dd16ebaae <    -:  ------------ arm64: dts: mediatek: add symbols to dtb
 252:  8585e7d77269 <    -:  ------------ Example files to copy the boot and modules.
 253:  64a808bc3c0c <    -:  ------------ Scripts to make the .itb boot file. Place the bpi-r4.itb in the /boot directory of the BPI-R4. The .itb file contains the Linux kernel and Device tree files.
 254:  3f4e69730a4f <    -:  ------------ Some example .config files to use when building the kernel.
 255:  5bd6a36c14b1 <    -:  ------------ build: add the .its files that are used to make the .itb boot file for the bpi-r4
 256:  3a252445e488 <    -:  ------------ build: Add a README.BPI file with instructions on how to build the Linux kernel for the BPI-R4.
 257:  9538b07c0635 <    -:  ------------ CT: Add BPI4 kernel .config file.
 258:  c6a7e4315557 <    -:  ------------ BPI4: Update readme with a few pkgs needed for building on Fedora.
 259:  b13df740e737 <    -:  ------------ BPI4: Add packaging script for bpi4 kernel and modules.
 260:  4c9c807e3d41 <    -:  ------------ BPI: Package mtk and intel firmware in kernel pkg.
 264:  92df0e28ed0b <    -:  ------------ arch: arm64: boot: dts: mediatek: Add .dts files for adtran wifi router.
 265:  d6d28e6f0a3d <    -:  ------------ arch: arm64: boot: dts: mediatek: Add .dts file for Adtran SDG-8734v wifi router.
 266:  7956efe1b195 <    -:  ------------ Add "config-sdg-8734v" and "make-itb-adtran.sh"
 267:  7a0b10a1a8cb <    -:  ------------ Add "adtran.its"
 268:  0abd83c3500e <    -:  ------------ arch: arm64: boot: dts: mediatek: Add some more items to the .dts file for Adtran SDG-8734v wifi router.
 269:  35e22fd5a813 <    -:  ------------ arch: arm64: boot: dts: mediatek: Add some more items around the fxs fon ports to the .dts file for Adtran SDG-8734v wifi router.
 270:  9a308285aee4 <    -:  ------------ Adtran kernel pkg script.
 271:  11894e8c8b42 <    -:  ------------ pkg: Fix up adtran AP pkg script.
 272:  1c98cd76a009 <    -:  ------------ Add some build scripts for Adtran SDG-8734v and SDG-8733v
 273:  d1433e1eb59a <    -:  ------------ arch: arm64: boot: dts: mediatek: Add some ethernet port labels and add file for Adtran SDG-8733v wifi router.
 274:  eb8730a4900e <    -:  ------------ Build multi-target adtran itb, and check for build failures.
 275:  09624f3c3c3b <    -:  ------------ adtran: Add dts file for 8733 (no v) platform.
 276:  e8b33db1b68b <    -:  ------------ mediatek: dts: Hack in mt7988a from our 6.11 kernel.
 905:  95d57af187cf <    -:  ------------ cfg: Update bpi4 cfg for 6.10 kernel.
 906:  140e2059ed2b <    -:  ------------ bpi4 kernel config: Enable piles of USB device support.
 924:  6377fb541899 <    -:  ------------ bpi: Added environment setup instructions for Debian/Ubuntu users.
 935:  c7424426111a <    -:  ------------ Adtran sdg 8733 readme.
 942:  43e9a3e2f391 <    -:  ------------ adtran: Disable usb3 to work around RF interference.
 943:  068d8f8543e9 <    -:  ------------ adtran: force usb to 2.0 properly
 944:  493b2cab2a8b <    -:  ------------ adtran: Force usb2 for other variants
1013:  46f9ae768fc3 <    -:  ------------ net: ethernet: mtk_wed: add ppe drop
1014:  0b8db8e83a54 <    -:  ------------ mtk7996: Fix pppe drop build issue.
1077:  93d44e506bc3 <    -:  ------------ Revert "net: ethernet: mtk_eth_soc: Enable legacy SRAM behavior for v3 devices"
1078:  4a591d110b55 <    -:  ------------ Revert "net: ethernet: mtk_eth_soc: use generic allocator for SRAM"
1165:  1113c075457f <    -:  ------------ pkg:  Ensure adtran image shows current build date.

The adtran stuff didn't want to compile so I just deleted as much of it as I could find and crossed my fingers, that was probably unnecessary.

The dropping of mtk_eth_soc stuff _was_ necessary to get Ethernet to probe at all.

I also had to enable CONFIG_ATH10K_DEBUGFS, CONFIG_IWLWIFI_DEBUGFS and CONFIG_NL80211_TESTMODE in the kernel config
to get some bits to build (I'm building this with a distro defconfig that enables way too many things and I have yet to find time to trim it down;
maybe one day).

This abomination is now booted on my hardware and I'll get back to you if it explodes again, which I'd expect to happen at some point
during the weekend at worst (and if it survives longer, that's already an improvement).

Also, I _just_ realized that the logs from the original message did not make it to the list, so here they are in gist form instead: https://gist.github.com/K900/a0a27551d5955fefe2976bda3816f021