Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

["Ömer Mete Kaya" <omermetekaya0@gmail.com>]

On 9/8/26 19:50, Ben Greear wrote:
> On 9/8/26 01:58, Ömer Mete Kaya wrote:
>>
>>
>> On 9/7/26 19:38, Ben Greear wrote:
>>> On 9/5/26 5:25 PM, Ömer Mete Kaya wrote:
>>>> reg_check_chans_work() holds rtnl_mutex for the entire duration of
>>>> iterating over all registered devices and calling cfg80211_leave() on
>>>> each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
>>>> leave mesh), causing rtnl_mutex starvation when many wireless
>>>> interfaces
>>>> are present. This results in tasks waiting for rtnl_mutex for longer
>>>> than hung_task_timeout_secs:
>>>>
>>>>     INFO: task hung in inet_rtm_newaddr
>>>>     INFO: task hung in inet6_rtm_newaddr
>>>>     INFO: task hung in nsim_destroy
>>>>     INFO: task hung in tun_chr_close
>>>>     INFO: task hung in switchdev_deferred_process_work
>>>
>>> Hello Omer,
>>>
>>> Considering that maybe something has mis-diagnosed the problem, could
>>> you share details of the stack traces of
>>> the hung processes and lockdep output to see if the hang is actually
>>> elsewhere?  What kernel version are you
>>> testing?
>>>
>>> Thanks,
>>> Ben
>>>
>>
>>
>> Hi Ben,
>>
>> Here is the evidence with stack traces and lockdep output. My kernel
>> version is 7.2.0-02677-g544d85de4dc2 (net/main HEAD) and both unpatched
>> and patched kernels were tested on the same setup(52 mac80211_hwsim
>> radios (mac80211_hwsim.radios=51)).
>>
>> Unpatched kernel:
>>
>> The lockdep output shows reg_check_chans_work as the rtnl_mutex holder
>> and ip as the waiter:
>>
>> locks held by kworker/0:2/11408: 3, on CPU#0:
>> #1: (reg_check_chans).work
>> #2: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
>> at: reg_check_chans_work+0xad/0x1330
>>
>> locks held by ip/14180: 1, on CPU#0:
>> #0: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
>> at: rtnl_getlink+0xbfb/0x13b0
>>
>> Hung task call trace:
>>
>> INFO: task ip:14180 blocked for more than 5 seconds.
>> Call Trace:
>> __schedule+0x1cba/0x6a40
>> schedule+0xe2/0x2e0
>> schedule_preempt_disabled+0x13/0x30
>> __mutex_lock+0x871/0x1cc0
>> rtnl_getlink+0xbfb/0x13b0
>> rtnetlink_rcv_msg+0x9a1/0xee0
>> netlink_rcv_skb+0x186/0x450
>> netlink_sendmsg+0x8e5/0xde0
>>
>> Kernel panic, not syncing: hung_task: blocked tasks
>>
>> I injected msleep(200) per wdev in reg_leave_invalid_chans() to model a
>> slow cfg80211_leave() operation, to measure the hold duration. With 52
>> interfaces this produced approximately 10.5 seconds of
>> continuous rtnl_mutex hold:
>>
>> cfg80211: reg_check_chans_work: rtnl held for 10499 ms
>>
>> The structural problem is clear regardless of the exact duration:
>> rtnl_mutex is held across all per-interface cleanup for every
>> registered device in a single acquisition.
>>
>> Patched kernel:
>>
>> rtnl_mutex is acquired per-device, each hold is brief:
>>
>> cfg80211: reg_check_chans_work: device 0 rtnl held 0 ms
>> cfg80211: reg_check_chans_work: device 1 rtnl held 0 ms
>> ...
>> cfg80211: reg_check_chans_work: device 51 rtnl held 0 ms
>>
>> No hung tasks were observed.
> 
> At least my kernel tends to only warn about hung tasks that are blocked
> for minutes.  Did you adjust your kernel to make this timeout smaller?

Yes, I reduced hung_task_timeout_secs to 5 seconds for the test.
With the default 140-second timeout my reproduction would not have
triggered a hang, though syzbot does with that timeout. Actually, I've
started to have doubts about the extent of its real-world impact, it
depends heavily on callback latency and number of active interfaces.

> Do you actually have 52 real devices that cause this problem?  If so, what
> hardware is this?  Or you can only reproduce this with modified kernel that
> injects a 200ms sleep?

I used mac80211_hwsim virtual radios,
and the hang could only be reproduced with artificial delays.
The 200ms value was chosen as a rough estimate of a slow
driver callback, not based on measured real hardware latency.

actual syzbot crash report :

  locks held by kworker running reg_check_chans_work:
    #2: rtnl_mutex, at: reg_check_chans_work+0xac net/wireless/reg.c:2469
    #3: wiphy.mtx,  at: reg_check_chans_work+0x1a1 net/wireless/reg.c:2472

  locks held by syz-executor (inet6_rtm_newaddr):
    #0: rtnl_mutex, at: inet6_rtm_newaddr+0x65e

  INFO: task syz-executor blocked for more than 143 seconds.

This directly shows reg_check_chans_work holding rtnl_mutex while
inet6_rtm_newaddr waits — 143 seconds, with the default timeout.

Regarding my earlier test: with mac80211_hwsim virtual radios,
cfg80211_leave() takes 0 ms per interface. Syzbot also uses virtual
radios, why does it take 143 seconds there?
Maybe because syzbot creates hundreds of wireless
interfaces across many network namespaces simultaneously, making the
loop over all devices very long.

My analysis was primarily code-based, reg_check_chans_work() holds
rtnl_mutex across all devices in a single acquisition, and
cfg80211_leave() can invoke slow driver operations. The lockdep
output confirms the structural issue exists.

> As a note, we've tested with 50+ real wifi radios in a system, and while
> we've seen deadlocks
> due to something weird with CMA memory allocation in Intel be200 radios,
> our fix was
> different (and never accepted upstream).
> 
> Thanks,
> Ben
> 
Thank you too,
Ömer Mete