Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ben Greear <greearb@candelatech.com>]

On 9/8/26 01:58, Ömer Mete Kaya wrote:
> 
> 
> On 9/7/26 19:38, Ben Greear wrote:
>> On 9/5/26 5:25 PM, Ömer Mete Kaya wrote:
>>> reg_check_chans_work() holds rtnl_mutex for the entire duration of
>>> iterating over all registered devices and calling cfg80211_leave() on
>>> each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
>>> leave mesh), causing rtnl_mutex starvation when many wireless interfaces
>>> are present. This results in tasks waiting for rtnl_mutex for longer
>>> than hung_task_timeout_secs:
>>>
>>>     INFO: task hung in inet_rtm_newaddr
>>>     INFO: task hung in inet6_rtm_newaddr
>>>     INFO: task hung in nsim_destroy
>>>     INFO: task hung in tun_chr_close
>>>     INFO: task hung in switchdev_deferred_process_work
>>
>> Hello Omer,
>>
>> Considering that maybe something has mis-diagnosed the problem, could
>> you share details of the stack traces of
>> the hung processes and lockdep output to see if the hang is actually
>> elsewhere?  What kernel version are you
>> testing?
>>
>> Thanks,
>> Ben
>>
> 
> 
> Hi Ben,
> 
> Here is the evidence with stack traces and lockdep output. My kernel
> version is 7.2.0-02677-g544d85de4dc2 (net/main HEAD) and both unpatched
> and patched kernels were tested on the same setup(52 mac80211_hwsim
> radios (mac80211_hwsim.radios=51)).
> 
> Unpatched kernel:
> 
> The lockdep output shows reg_check_chans_work as the rtnl_mutex holder
> and ip as the waiter:
> 
> locks held by kworker/0:2/11408: 3, on CPU#0:
> #1: (reg_check_chans).work
> #2: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
> at: reg_check_chans_work+0xad/0x1330
> 
> locks held by ip/14180: 1, on CPU#0:
> #0: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
> at: rtnl_getlink+0xbfb/0x13b0
> 
> Hung task call trace:
> 
> INFO: task ip:14180 blocked for more than 5 seconds.
> Call Trace:
> __schedule+0x1cba/0x6a40
> schedule+0xe2/0x2e0
> schedule_preempt_disabled+0x13/0x30
> __mutex_lock+0x871/0x1cc0
> rtnl_getlink+0xbfb/0x13b0
> rtnetlink_rcv_msg+0x9a1/0xee0
> netlink_rcv_skb+0x186/0x450
> netlink_sendmsg+0x8e5/0xde0
> 
> Kernel panic, not syncing: hung_task: blocked tasks
> 
> I injected msleep(200) per wdev in reg_leave_invalid_chans() to model a
> slow cfg80211_leave() operation, to measure the hold duration. With 52
> interfaces this produced approximately 10.5 seconds of
> continuous rtnl_mutex hold:
> 
> cfg80211: reg_check_chans_work: rtnl held for 10499 ms
> 
> The structural problem is clear regardless of the exact duration:
> rtnl_mutex is held across all per-interface cleanup for every
> registered device in a single acquisition.
> 
> Patched kernel:
> 
> rtnl_mutex is acquired per-device, each hold is brief:
> 
> cfg80211: reg_check_chans_work: device 0 rtnl held 0 ms
> cfg80211: reg_check_chans_work: device 1 rtnl held 0 ms
> ...
> cfg80211: reg_check_chans_work: device 51 rtnl held 0 ms
> 
> No hung tasks were observed.

At least my kernel tends to only warn about hung tasks that are blocked
for minutes.  Did you adjust your kernel to make this timeout smaller?

Do you actually have 52 real devices that cause this problem?  If so, what
hardware is this?  Or you can only reproduce this with modified kernel that
injects a 200ms sleep?

As a note, we've tested with 50+ real wifi radios in a system, and while we've seen deadlocks
due to something weird with CMA memory allocation in Intel be200 radios, our fix was
different (and never accepted upstream).

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com