Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ben Greear <greearb@candelatech.com>]

On 9/8/26 12:52, Ömer Mete Kaya wrote:
> 
> 
> On 9/8/26 19:50, Ben Greear wrote:
>> On 9/8/26 01:58, Ömer Mete Kaya wrote:
>>>
>>>
>>> On 9/7/26 19:38, Ben Greear wrote:
>>>> On 9/5/26 5:25 PM, Ömer Mete Kaya wrote:
>>>>> reg_check_chans_work() holds rtnl_mutex for the entire duration of
>>>>> iterating over all registered devices and calling cfg80211_leave() on
>>>>> each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
>>>>> leave mesh), causing rtnl_mutex starvation when many wireless
>>>>> interfaces
>>>>> are present. This results in tasks waiting for rtnl_mutex for longer
>>>>> than hung_task_timeout_secs:
>>>>>
>>>>>      INFO: task hung in inet_rtm_newaddr
>>>>>      INFO: task hung in inet6_rtm_newaddr
>>>>>      INFO: task hung in nsim_destroy
>>>>>      INFO: task hung in tun_chr_close
>>>>>      INFO: task hung in switchdev_deferred_process_work
>>>>
>>>> Hello Omer,
>>>>
>>>> Considering that maybe something has mis-diagnosed the problem, could
>>>> you share details of the stack traces of
>>>> the hung processes and lockdep output to see if the hang is actually
>>>> elsewhere?  What kernel version are you
>>>> testing?
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>
>>>
>>> Hi Ben,
>>>
>>> Here is the evidence with stack traces and lockdep output. My kernel
>>> version is 7.2.0-02677-g544d85de4dc2 (net/main HEAD) and both unpatched
>>> and patched kernels were tested on the same setup(52 mac80211_hwsim
>>> radios (mac80211_hwsim.radios=51)).
>>>
>>> Unpatched kernel:
>>>
>>> The lockdep output shows reg_check_chans_work as the rtnl_mutex holder
>>> and ip as the waiter:
>>>
>>> locks held by kworker/0:2/11408: 3, on CPU#0:
>>> #1: (reg_check_chans).work
>>> #2: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
>>> at: reg_check_chans_work+0xad/0x1330
>>>
>>> locks held by ip/14180: 1, on CPU#0:
>>> #0: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
>>> at: rtnl_getlink+0xbfb/0x13b0
>>>
>>> Hung task call trace:
>>>
>>> INFO: task ip:14180 blocked for more than 5 seconds.
>>> Call Trace:
>>> __schedule+0x1cba/0x6a40
>>> schedule+0xe2/0x2e0
>>> schedule_preempt_disabled+0x13/0x30
>>> __mutex_lock+0x871/0x1cc0
>>> rtnl_getlink+0xbfb/0x13b0
>>> rtnetlink_rcv_msg+0x9a1/0xee0
>>> netlink_rcv_skb+0x186/0x450
>>> netlink_sendmsg+0x8e5/0xde0
>>>
>>> Kernel panic, not syncing: hung_task: blocked tasks
>>>
>>> I injected msleep(200) per wdev in reg_leave_invalid_chans() to model a
>>> slow cfg80211_leave() operation, to measure the hold duration. With 52
>>> interfaces this produced approximately 10.5 seconds of
>>> continuous rtnl_mutex hold:
>>>
>>> cfg80211: reg_check_chans_work: rtnl held for 10499 ms
>>>
>>> The structural problem is clear regardless of the exact duration:
>>> rtnl_mutex is held across all per-interface cleanup for every
>>> registered device in a single acquisition.
>>>
>>> Patched kernel:
>>>
>>> rtnl_mutex is acquired per-device, each hold is brief:
>>>
>>> cfg80211: reg_check_chans_work: device 0 rtnl held 0 ms
>>> cfg80211: reg_check_chans_work: device 1 rtnl held 0 ms
>>> ...
>>> cfg80211: reg_check_chans_work: device 51 rtnl held 0 ms
>>>
>>> No hung tasks were observed.
>>
>> At least my kernel tends to only warn about hung tasks that are blocked
>> for minutes.  Did you adjust your kernel to make this timeout smaller?
> 
> Yes, I reduced hung_task_timeout_secs to 5 seconds for the test.
> With the default 140-second timeout my reproduction would not have
> triggered a hang, though syzbot does with that timeout. Actually, I've
> started to have doubts about the extent of its real-world impact, it
> depends heavily on callback latency and number of active interfaces.
> 
>> Do you actually have 52 real devices that cause this problem?  If so, what
>> hardware is this?  Or you can only reproduce this with modified kernel that
>> injects a 200ms sleep?
> 
> I used mac80211_hwsim virtual radios,
> and the hang could only be reproduced with artificial delays.
> The 200ms value was chosen as a rough estimate of a slow
> driver callback, not based on measured real hardware latency.

So you created a fake problem to make it fit what something
thinks is the root cause.  I don't think that is helpful.

Possibly root cause below is that something else holds wiphy.mtx, so reg_check_chans_work is blocked
and cannot make progress.  Do you have full dmesg dump that includes all blocked processes
and full lockdep dump?  Are you actually able to reproduce the problem w/out adding fake sleeps?

> actual syzbot crash report :
> 
>    locks held by kworker running reg_check_chans_work:
>      #2: rtnl_mutex, at: reg_check_chans_work+0xac net/wireless/reg.c:2469
>      #3: wiphy.mtx,  at: reg_check_chans_work+0x1a1 net/wireless/reg.c:2472
> 
>    locks held by syz-executor (inet6_rtm_newaddr):
>      #0: rtnl_mutex, at: inet6_rtm_newaddr+0x65e
> 
>    INFO: task syz-executor blocked for more than 143 seconds.
> 
> This directly shows reg_check_chans_work holding rtnl_mutex while
> inet6_rtm_newaddr waits — 143 seconds, with the default timeout.

You or your LLM is overly confident of root cause in my opinion.

I will be happy to talk with a human about this since I am interested in
weird wifi problems at scale, but please do not send more LLM responses.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com