[PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ömer Mete Kaya]

reg_check_chans_work() holds rtnl_mutex for the entire duration of
iterating over all registered devices and calling cfg80211_leave() on
each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
leave mesh), causing rtnl_mutex starvation when many wireless interfaces
are present. This results in tasks waiting for rtnl_mutex for longer
than hung_task_timeout_secs:

  INFO: task hung in inet_rtm_newaddr
  INFO: task hung in inet6_rtm_newaddr
  INFO: task hung in nsim_destroy

Fix by walking cfg80211_rdev_list under RCU and acquiring rtnl per-device,
so other rtnl waiters get a chance to run between devices.

I could not add the Fixes: tag because this patch addresses three separate
hung task reports whose cause bisections all failed, making it impossible
to identify a single introducing commit.

Reported-by: syzbot+adeb8550754921fece20@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=adeb8550754921fece20
Reported-by: syzbot+101224300649c3eb8af4@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=101224300649c3eb8af4
Reported-by: syzbot+8141dcbd23a8f857798a@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=8141dcbd23a8f857798a
Signed-off-by: Ömer Mete Kaya <omermetekaya0@gmail.com>
---
 net/wireless/reg.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/net/wireless/reg.c b/net/wireless/reg.c
index a8336baf85dc..a2e0d1cf8317 100644
--- a/net/wireless/reg.c
+++ b/net/wireless/reg.c
@@ -2466,12 +2466,21 @@ static void reg_check_chans_work(struct work_struct *work)
 	struct cfg80211_registered_device *rdev;
 
 	pr_debug("Verifying active interfaces after reg change\n");
-	rtnl_lock();
-
-	for_each_rdev(rdev)
+	/*
+	 * Acquire rtnl per-device instead of holding it for the entire loop;
+	 * cfg80211_leave() can be slow and starve other rtnl waiters otherwise.
+	 * wiphy_unregister() holds rtnl across list_del_rcu() + synchronize_rcu(),
+	 * so rdev cannot be freed while we hold rtnl_lock() below.
+	 */
+	rcu_read_lock();
+	list_for_each_entry_rcu(rdev, &cfg80211_rdev_list, list) {
+		rcu_read_unlock();
+		rtnl_lock();
 		reg_leave_invalid_chans(&rdev->wiphy);
-
-	rtnl_unlock();
+		rtnl_unlock();
+		rcu_read_lock();
+	}
+	rcu_read_unlock();
 }
 
 void reg_check_channels(void)
-- 
2.55.0

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ömer Mete Kaya]

Apologies for the noise — I forgot to mark the previous send as v2.
This is v2 of the patch, extending the fix to cover two additional
hung task reports (tun_chr_close and switchdev_deferred_process_work)
that share the same root cause.

Changes in v2:
- Added Reported-by/Closes for tun_chr_close (b0ae8f1a)
- Added Reported-by/Closes for switchdev_deferred_process_work (d6bbe0f5)
- Added linux-kernel@vger.kernel.org to CC
- Updated "three separate" to "five separate" in commit message

In-Reply-To: <20260903111705.472491-1-omermetekaya0@gmail.com>

[PATCH v2] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ömer Mete Kaya]

reg_check_chans_work() holds rtnl_mutex for the entire duration of
iterating over all registered devices and calling cfg80211_leave() on
each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
leave mesh), causing rtnl_mutex starvation when many wireless interfaces
are present. This results in tasks waiting for rtnl_mutex for longer
than hung_task_timeout_secs:

  INFO: task hung in inet_rtm_newaddr
  INFO: task hung in inet6_rtm_newaddr
  INFO: task hung in nsim_destroy
  INFO: task hung in tun_chr_close
  INFO: task hung in switchdev_deferred_process_work

Fix by walking cfg80211_rdev_list under RCU and acquiring rtnl per-device,
so other rtnl waiters get a chance to run between devices.

I could not add the Fixes: tag because this patch addresses five separate
hung task reports whose cause bisections all failed, making it impossible
to identify a single introducing commit.

Reported-by: syzbot+adeb8550754921fece20@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=adeb8550754921fece20
Reported-by: syzbot+101224300649c3eb8af4@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=101224300649c3eb8af4
Reported-by: syzbot+8141dcbd23a8f857798a@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=8141dcbd23a8f857798a
Reported-by: syzbot+b0ae8f1abf7d891e0426@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=b0ae8f1abf7d891e0426
Reported-by: syzbot+d6bbe0f5705cb8a5aa2b@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=d6bbe0f5705cb8a5aa2b
Signed-off-by: Ömer Mete Kaya <omermetekaya0@gmail.com>
---
 net/wireless/reg.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/net/wireless/reg.c b/net/wireless/reg.c
index a8336baf85dc..a2e0d1cf8317 100644
--- a/net/wireless/reg.c
+++ b/net/wireless/reg.c
@@ -2466,12 +2466,21 @@ static void reg_check_chans_work(struct work_struct *work)
 	struct cfg80211_registered_device *rdev;
 
 	pr_debug("Verifying active interfaces after reg change\n");
-	rtnl_lock();
-
-	for_each_rdev(rdev)
+	/*
+	 * Acquire rtnl per-device instead of holding it for the entire loop;
+	 * cfg80211_leave() can be slow and starve other rtnl waiters otherwise.
+	 * wiphy_unregister() holds rtnl across list_del_rcu() + synchronize_rcu(),
+	 * so rdev cannot be freed while we hold rtnl_lock() below.
+	 */
+	rcu_read_lock();
+	list_for_each_entry_rcu(rdev, &cfg80211_rdev_list, list) {
+		rcu_read_unlock();
+		rtnl_lock();
 		reg_leave_invalid_chans(&rdev->wiphy);
-
-	rtnl_unlock();
+		rtnl_unlock();
+		rcu_read_lock();
+	}
+	rcu_read_unlock();
 }
 
 void reg_check_channels(void)
-- 
2.55.0

Re: [PATCH v2] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Simon Horman]

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
Full review at: https://sashiko.dev/#/patchset/20260903151542.486376-2-omermetekaya0@gmail.com?part=1
---
wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

This commit changes reg_check_chans_work() to iterate over cfg80211_rdev_list
using RCU instead of holding rtnl_mutex for the entire loop. It acquires
rtnl_mutex on a per-device basis to prevent task hangs.

> diff --git a/net/wireless/reg.c b/net/wireless/reg.c
> index a8336baf85dc6..a2e0d1cf8317c 100644
> --- a/net/wireless/reg.c
> +++ b/net/wireless/reg.c
> @@ -2466,12 +2466,21 @@ static void reg_check_chans_work(struct work_struct *work)
>  	struct cfg80211_registered_device *rdev;
>  
>  	pr_debug("Verifying active interfaces after reg change\n");
> -	rtnl_lock();
> -
> -	for_each_rdev(rdev)
> +	/*
> +	 * Acquire rtnl per-device instead of holding it for the entire loop;
> +	 * cfg80211_leave() can be slow and starve other rtnl waiters otherwise.
> +	 * wiphy_unregister() holds rtnl across list_del_rcu() + synchronize_rcu(),
> +	 * so rdev cannot be freed while we hold rtnl_lock() below.
> +	 */
> +	rcu_read_lock();
> +	list_for_each_entry_rcu(rdev, &cfg80211_rdev_list, list) {
> +		rcu_read_unlock();

[Severity: High]
Does dropping the RCU read lock here remove the lifetime protection of the
current rdev element while waiting for rtnl_lock()?

If a concurrent thread unregisters the device (for example, via
wiphy_unregister()), it could acquire rtnl_lock, remove the device with
list_del_rcu(), and call synchronize_rcu().

Because the RCU read lock was dropped by the work thread here,
synchronize_rcu() could complete and allow the unregister thread to free rdev
before this thread acquires the rtnl lock.

> +		rtnl_lock();
>  		reg_leave_invalid_chans(&rdev->wiphy);

[Severity: High]
Can this regression lead to a use-after-free?

If rdev was freed during the race window while waiting for rtnl_lock(),
accessing rdev->wiphy here would read from freed memory.

> -
> -	rtnl_unlock();
> +		rtnl_unlock();
> +		rcu_read_lock();
> +	}

[Severity: High]
Will the list_for_each_entry_rcu() macro implicitly dereference
rdev->list.next to continue the iteration after the loop body finishes?

If rdev was freed while the lock was dropped, does this result in another
use-after-free when advancing to the next list element?

> +	rcu_read_unlock();
>  }

Re: [PATCH v2] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ömer Mete Kaya]

Thank you for the review, Simon! You are correct, the previous v2
had a UAF window between rcu_read_unlock() and rtnl_lock().

v3 fixes this by taking a snapshot of cfg80211_rdev_list under RCU
with get_device() holding a reference on each rdev, preventing freeing
between per-device rtnl acquisitions. After acquiring rtnl per-device,
wiphy.registered is checked (set under rtnl in wiphy_unregister()) to
safely skip any unregistered device.

Changes in v3:
- Replace RCU drop/reacquire pattern with get_device() snapshot
- Add wiphy.registered check after per-device rtnl_lock()
- Add Fixes: f7e60032c661 tag
- Update commit message to describe new approach
 
In-Reply-To: <20260903151542.486376-2-omermetekaya0@gmail.com>

[PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ömer Mete Kaya]

reg_check_chans_work() holds rtnl_mutex for the entire duration of
iterating over all registered devices and calling cfg80211_leave() on
each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
leave mesh), causing rtnl_mutex starvation when many wireless interfaces
are present. This results in tasks waiting for rtnl_mutex for longer
than hung_task_timeout_secs:

  INFO: task hung in inet_rtm_newaddr
  INFO: task hung in inet6_rtm_newaddr
  INFO: task hung in nsim_destroy
  INFO: task hung in tun_chr_close
  INFO: task hung in switchdev_deferred_process_work

Fix by taking a snapshot of cfg80211_rdev_list under RCU, holding a
device reference (get_device/put_device) to prevent freeing, then
acquiring rtnl per-device so other rtnl waiters get a chance to run
between devices. After acquiring rtnl, wiphy.registered is checked to
skip any device that was unregistered in the meantime.

Fixes: f7e60032c661 ("wifi: cfg80211: fix locking in regulatory disconnect")
Reported-by: syzbot+adeb8550754921fece20@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=adeb8550754921fece20
Reported-by: syzbot+101224300649c3eb8af4@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=101224300649c3eb8af4
Reported-by: syzbot+8141dcbd23a8f857798a@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=8141dcbd23a8f857798a
Reported-by: syzbot+b0ae8f1abf7d891e0426@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=b0ae8f1abf7d891e0426
Reported-by: syzbot+d6bbe0f5705cb8a5aa2b@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=d6bbe0f5705cb8a5aa2b
Signed-off-by: Ömer Mete Kaya <omermetekaya0@gmail.com>
---
 net/wireless/reg.c | 44 +++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 39 insertions(+), 5 deletions(-)

diff --git a/net/wireless/reg.c b/net/wireless/reg.c
index a8336baf85dc..7a3f1e470e14 100644
--- a/net/wireless/reg.c
+++ b/net/wireless/reg.c
@@ -2463,15 +2463,49 @@ static void reg_leave_invalid_chans(struct wiphy *wiphy)
 
 static void reg_check_chans_work(struct work_struct *work)
 {
-	struct cfg80211_registered_device *rdev;
+	struct cfg80211_registered_device *rdev, **snap;
+	int i, n = 0;
 
 	pr_debug("Verifying active interfaces after reg change\n");
-	rtnl_lock();
 
-	for_each_rdev(rdev)
-		reg_leave_invalid_chans(&rdev->wiphy);
+	/*
+	 * Snapshot rdev pointers under RCU with a device reference so they
+	 * cannot be freed between per-device rtnl acquisitions. Using a
+	 * per-device rtnl_lock() instead of holding it across all devices
+	 * avoids starving other rtnl waiters when cfg80211_leave() is slow.
+	 */
+	rcu_read_lock();
+	list_for_each_entry_rcu(rdev, &cfg80211_rdev_list, list)
+		n++;
+	rcu_read_unlock();
 
-	rtnl_unlock();
+	if (!n)
+		return;
+
+	snap = kmalloc_array(n, sizeof(*snap), GFP_KERNEL);
+	if (!snap)
+		return;
+
+	i = 0;
+	rcu_read_lock();
+	list_for_each_entry_rcu(rdev, &cfg80211_rdev_list, list) {
+		if (i >= n)
+			break;
+		get_device(&rdev->wiphy.dev);
+		snap[i++] = rdev;
+	}
+	rcu_read_unlock();
+	n = i;
+
+	for (i = 0; i < n; i++) {
+		rtnl_lock();
+		if (snap[i]->wiphy.registered)
+			reg_leave_invalid_chans(&snap[i]->wiphy);
+		rtnl_unlock();
+		put_device(&snap[i]->wiphy.dev);
+	}
+
+	kfree(snap);
 }
 
 void reg_check_channels(void)
-- 
2.55.0

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Johannes Berg]

On Sun, 2026-09-06 at 03:25 +0300, Ömer Mete Kaya wrote:
> 
> +	/*
> +	 * Snapshot rdev pointers under RCU with a device reference so they
> +	 * cannot be freed between per-device rtnl acquisitions. Using a
> +	 * per-device rtnl_lock() instead of holding it across all devices
> +	 * avoids starving other rtnl waiters when cfg80211_leave() is slow.
> +	 */
> +	rcu_read_lock();
> +	list_for_each_entry_rcu(rdev, &cfg80211_rdev_list, list)
> +		n++;
> +	rcu_read_unlock();
>  
> -	rtnl_unlock();
> +	if (!n)
> +		return;
> +
> +	snap = kmalloc_array(n, sizeof(*snap), GFP_KERNEL);
> +	if (!snap)
> +		return;

I really don't know _what_ you're doing here, or more realistically what
exactly you're asking your LLM to do here, but it's pointless. Please
just stop, you're not helping in any way.

johannes

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ben Greear]

On 9/5/26 5:25 PM, Ömer Mete Kaya wrote:
> reg_check_chans_work() holds rtnl_mutex for the entire duration of
> iterating over all registered devices and calling cfg80211_leave() on
> each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
> leave mesh), causing rtnl_mutex starvation when many wireless interfaces
> are present. This results in tasks waiting for rtnl_mutex for longer
> than hung_task_timeout_secs:
> 
>    INFO: task hung in inet_rtm_newaddr
>    INFO: task hung in inet6_rtm_newaddr
>    INFO: task hung in nsim_destroy
>    INFO: task hung in tun_chr_close
>    INFO: task hung in switchdev_deferred_process_work

Hello Omer,

Considering that maybe something has mis-diagnosed the problem, could you share details of the stack traces of
the hung processes and lockdep output to see if the hang is actually elsewhere?  What kernel version are you
testing?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ömer Mete Kaya]

On 9/7/26 19:38, Ben Greear wrote:
> On 9/5/26 5:25 PM, Ömer Mete Kaya wrote:
>> reg_check_chans_work() holds rtnl_mutex for the entire duration of
>> iterating over all registered devices and calling cfg80211_leave() on
>> each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
>> leave mesh), causing rtnl_mutex starvation when many wireless interfaces
>> are present. This results in tasks waiting for rtnl_mutex for longer
>> than hung_task_timeout_secs:
>>
>>    INFO: task hung in inet_rtm_newaddr
>>    INFO: task hung in inet6_rtm_newaddr
>>    INFO: task hung in nsim_destroy
>>    INFO: task hung in tun_chr_close
>>    INFO: task hung in switchdev_deferred_process_work
> 
> Hello Omer,
> 
> Considering that maybe something has mis-diagnosed the problem, could
> you share details of the stack traces of
> the hung processes and lockdep output to see if the hang is actually
> elsewhere?  What kernel version are you
> testing?
> 
> Thanks,
> Ben
> 


Hi Ben,

Here is the evidence with stack traces and lockdep output. My kernel
version is 7.2.0-02677-g544d85de4dc2 (net/main HEAD) and both unpatched
and patched kernels were tested on the same setup(52 mac80211_hwsim
radios (mac80211_hwsim.radios=51)).

Unpatched kernel:

The lockdep output shows reg_check_chans_work as the rtnl_mutex holder
and ip as the waiter:

locks held by kworker/0:2/11408: 3, on CPU#0:
#1: (reg_check_chans).work
#2: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
at: reg_check_chans_work+0xad/0x1330

locks held by ip/14180: 1, on CPU#0:
#0: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
at: rtnl_getlink+0xbfb/0x13b0

Hung task call trace:

INFO: task ip:14180 blocked for more than 5 seconds.
Call Trace:
__schedule+0x1cba/0x6a40
schedule+0xe2/0x2e0
schedule_preempt_disabled+0x13/0x30
__mutex_lock+0x871/0x1cc0
rtnl_getlink+0xbfb/0x13b0
rtnetlink_rcv_msg+0x9a1/0xee0
netlink_rcv_skb+0x186/0x450
netlink_sendmsg+0x8e5/0xde0

Kernel panic, not syncing: hung_task: blocked tasks

I injected msleep(200) per wdev in reg_leave_invalid_chans() to model a
slow cfg80211_leave() operation, to measure the hold duration. With 52
interfaces this produced approximately 10.5 seconds of
continuous rtnl_mutex hold:

cfg80211: reg_check_chans_work: rtnl held for 10499 ms

The structural problem is clear regardless of the exact duration:
rtnl_mutex is held across all per-interface cleanup for every
registered device in a single acquisition.

Patched kernel:

rtnl_mutex is acquired per-device, each hold is brief:

cfg80211: reg_check_chans_work: device 0 rtnl held 0 ms
cfg80211: reg_check_chans_work: device 1 rtnl held 0 ms
...
cfg80211: reg_check_chans_work: device 51 rtnl held 0 ms

No hung tasks were observed.


Best regards,
Ömer Mete Kaya

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ben Greear]

On 9/8/26 01:58, Ömer Mete Kaya wrote:
> 
> 
> On 9/7/26 19:38, Ben Greear wrote:
>> On 9/5/26 5:25 PM, Ömer Mete Kaya wrote:
>>> reg_check_chans_work() holds rtnl_mutex for the entire duration of
>>> iterating over all registered devices and calling cfg80211_leave() on
>>> each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
>>> leave mesh), causing rtnl_mutex starvation when many wireless interfaces
>>> are present. This results in tasks waiting for rtnl_mutex for longer
>>> than hung_task_timeout_secs:
>>>
>>>     INFO: task hung in inet_rtm_newaddr
>>>     INFO: task hung in inet6_rtm_newaddr
>>>     INFO: task hung in nsim_destroy
>>>     INFO: task hung in tun_chr_close
>>>     INFO: task hung in switchdev_deferred_process_work
>>
>> Hello Omer,
>>
>> Considering that maybe something has mis-diagnosed the problem, could
>> you share details of the stack traces of
>> the hung processes and lockdep output to see if the hang is actually
>> elsewhere?  What kernel version are you
>> testing?
>>
>> Thanks,
>> Ben
>>
> 
> 
> Hi Ben,
> 
> Here is the evidence with stack traces and lockdep output. My kernel
> version is 7.2.0-02677-g544d85de4dc2 (net/main HEAD) and both unpatched
> and patched kernels were tested on the same setup(52 mac80211_hwsim
> radios (mac80211_hwsim.radios=51)).
> 
> Unpatched kernel:
> 
> The lockdep output shows reg_check_chans_work as the rtnl_mutex holder
> and ip as the waiter:
> 
> locks held by kworker/0:2/11408: 3, on CPU#0:
> #1: (reg_check_chans).work
> #2: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
> at: reg_check_chans_work+0xad/0x1330
> 
> locks held by ip/14180: 1, on CPU#0:
> #0: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
> at: rtnl_getlink+0xbfb/0x13b0
> 
> Hung task call trace:
> 
> INFO: task ip:14180 blocked for more than 5 seconds.
> Call Trace:
> __schedule+0x1cba/0x6a40
> schedule+0xe2/0x2e0
> schedule_preempt_disabled+0x13/0x30
> __mutex_lock+0x871/0x1cc0
> rtnl_getlink+0xbfb/0x13b0
> rtnetlink_rcv_msg+0x9a1/0xee0
> netlink_rcv_skb+0x186/0x450
> netlink_sendmsg+0x8e5/0xde0
> 
> Kernel panic, not syncing: hung_task: blocked tasks
> 
> I injected msleep(200) per wdev in reg_leave_invalid_chans() to model a
> slow cfg80211_leave() operation, to measure the hold duration. With 52
> interfaces this produced approximately 10.5 seconds of
> continuous rtnl_mutex hold:
> 
> cfg80211: reg_check_chans_work: rtnl held for 10499 ms
> 
> The structural problem is clear regardless of the exact duration:
> rtnl_mutex is held across all per-interface cleanup for every
> registered device in a single acquisition.
> 
> Patched kernel:
> 
> rtnl_mutex is acquired per-device, each hold is brief:
> 
> cfg80211: reg_check_chans_work: device 0 rtnl held 0 ms
> cfg80211: reg_check_chans_work: device 1 rtnl held 0 ms
> ...
> cfg80211: reg_check_chans_work: device 51 rtnl held 0 ms
> 
> No hung tasks were observed.

At least my kernel tends to only warn about hung tasks that are blocked
for minutes.  Did you adjust your kernel to make this timeout smaller?

Do you actually have 52 real devices that cause this problem?  If so, what
hardware is this?  Or you can only reproduce this with modified kernel that
injects a 200ms sleep?

As a note, we've tested with 50+ real wifi radios in a system, and while we've seen deadlocks
due to something weird with CMA memory allocation in Intel be200 radios, our fix was
different (and never accepted upstream).

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ömer Mete Kaya]

On 9/8/26 19:50, Ben Greear wrote:
> On 9/8/26 01:58, Ömer Mete Kaya wrote:
>>
>>
>> On 9/7/26 19:38, Ben Greear wrote:
>>> On 9/5/26 5:25 PM, Ömer Mete Kaya wrote:
>>>> reg_check_chans_work() holds rtnl_mutex for the entire duration of
>>>> iterating over all registered devices and calling cfg80211_leave() on
>>>> each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
>>>> leave mesh), causing rtnl_mutex starvation when many wireless
>>>> interfaces
>>>> are present. This results in tasks waiting for rtnl_mutex for longer
>>>> than hung_task_timeout_secs:
>>>>
>>>>     INFO: task hung in inet_rtm_newaddr
>>>>     INFO: task hung in inet6_rtm_newaddr
>>>>     INFO: task hung in nsim_destroy
>>>>     INFO: task hung in tun_chr_close
>>>>     INFO: task hung in switchdev_deferred_process_work
>>>
>>> Hello Omer,
>>>
>>> Considering that maybe something has mis-diagnosed the problem, could
>>> you share details of the stack traces of
>>> the hung processes and lockdep output to see if the hang is actually
>>> elsewhere?  What kernel version are you
>>> testing?
>>>
>>> Thanks,
>>> Ben
>>>
>>
>>
>> Hi Ben,
>>
>> Here is the evidence with stack traces and lockdep output. My kernel
>> version is 7.2.0-02677-g544d85de4dc2 (net/main HEAD) and both unpatched
>> and patched kernels were tested on the same setup(52 mac80211_hwsim
>> radios (mac80211_hwsim.radios=51)).
>>
>> Unpatched kernel:
>>
>> The lockdep output shows reg_check_chans_work as the rtnl_mutex holder
>> and ip as the waiter:
>>
>> locks held by kworker/0:2/11408: 3, on CPU#0:
>> #1: (reg_check_chans).work
>> #2: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
>> at: reg_check_chans_work+0xad/0x1330
>>
>> locks held by ip/14180: 1, on CPU#0:
>> #0: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
>> at: rtnl_getlink+0xbfb/0x13b0
>>
>> Hung task call trace:
>>
>> INFO: task ip:14180 blocked for more than 5 seconds.
>> Call Trace:
>> __schedule+0x1cba/0x6a40
>> schedule+0xe2/0x2e0
>> schedule_preempt_disabled+0x13/0x30
>> __mutex_lock+0x871/0x1cc0
>> rtnl_getlink+0xbfb/0x13b0
>> rtnetlink_rcv_msg+0x9a1/0xee0
>> netlink_rcv_skb+0x186/0x450
>> netlink_sendmsg+0x8e5/0xde0
>>
>> Kernel panic, not syncing: hung_task: blocked tasks
>>
>> I injected msleep(200) per wdev in reg_leave_invalid_chans() to model a
>> slow cfg80211_leave() operation, to measure the hold duration. With 52
>> interfaces this produced approximately 10.5 seconds of
>> continuous rtnl_mutex hold:
>>
>> cfg80211: reg_check_chans_work: rtnl held for 10499 ms
>>
>> The structural problem is clear regardless of the exact duration:
>> rtnl_mutex is held across all per-interface cleanup for every
>> registered device in a single acquisition.
>>
>> Patched kernel:
>>
>> rtnl_mutex is acquired per-device, each hold is brief:
>>
>> cfg80211: reg_check_chans_work: device 0 rtnl held 0 ms
>> cfg80211: reg_check_chans_work: device 1 rtnl held 0 ms
>> ...
>> cfg80211: reg_check_chans_work: device 51 rtnl held 0 ms
>>
>> No hung tasks were observed.
> 
> At least my kernel tends to only warn about hung tasks that are blocked
> for minutes.  Did you adjust your kernel to make this timeout smaller?

Yes, I reduced hung_task_timeout_secs to 5 seconds for the test.
With the default 140-second timeout my reproduction would not have
triggered a hang, though syzbot does with that timeout. Actually, I've
started to have doubts about the extent of its real-world impact, it
depends heavily on callback latency and number of active interfaces.

> Do you actually have 52 real devices that cause this problem?  If so, what
> hardware is this?  Or you can only reproduce this with modified kernel that
> injects a 200ms sleep?

I used mac80211_hwsim virtual radios,
and the hang could only be reproduced with artificial delays.
The 200ms value was chosen as a rough estimate of a slow
driver callback, not based on measured real hardware latency.

actual syzbot crash report :

  locks held by kworker running reg_check_chans_work:
    #2: rtnl_mutex, at: reg_check_chans_work+0xac net/wireless/reg.c:2469
    #3: wiphy.mtx,  at: reg_check_chans_work+0x1a1 net/wireless/reg.c:2472

  locks held by syz-executor (inet6_rtm_newaddr):
    #0: rtnl_mutex, at: inet6_rtm_newaddr+0x65e

  INFO: task syz-executor blocked for more than 143 seconds.

This directly shows reg_check_chans_work holding rtnl_mutex while
inet6_rtm_newaddr waits — 143 seconds, with the default timeout.

Regarding my earlier test: with mac80211_hwsim virtual radios,
cfg80211_leave() takes 0 ms per interface. Syzbot also uses virtual
radios, why does it take 143 seconds there?
Maybe because syzbot creates hundreds of wireless
interfaces across many network namespaces simultaneously, making the
loop over all devices very long.

My analysis was primarily code-based, reg_check_chans_work() holds
rtnl_mutex across all devices in a single acquisition, and
cfg80211_leave() can invoke slow driver operations. The lockdep
output confirms the structural issue exists.

> As a note, we've tested with 50+ real wifi radios in a system, and while
> we've seen deadlocks
> due to something weird with CMA memory allocation in Intel be200 radios,
> our fix was
> different (and never accepted upstream).
> 
> Thanks,
> Ben
> 
Thank you too,
Ömer Mete

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ben Greear]

On 9/8/26 12:52, Ömer Mete Kaya wrote:
> 
> 
> On 9/8/26 19:50, Ben Greear wrote:
>> On 9/8/26 01:58, Ömer Mete Kaya wrote:
>>>
>>>
>>> On 9/7/26 19:38, Ben Greear wrote:
>>>> On 9/5/26 5:25 PM, Ömer Mete Kaya wrote:
>>>>> reg_check_chans_work() holds rtnl_mutex for the entire duration of
>>>>> iterating over all registered devices and calling cfg80211_leave() on
>>>>> each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
>>>>> leave mesh), causing rtnl_mutex starvation when many wireless
>>>>> interfaces
>>>>> are present. This results in tasks waiting for rtnl_mutex for longer
>>>>> than hung_task_timeout_secs:
>>>>>
>>>>>      INFO: task hung in inet_rtm_newaddr
>>>>>      INFO: task hung in inet6_rtm_newaddr
>>>>>      INFO: task hung in nsim_destroy
>>>>>      INFO: task hung in tun_chr_close
>>>>>      INFO: task hung in switchdev_deferred_process_work
>>>>
>>>> Hello Omer,
>>>>
>>>> Considering that maybe something has mis-diagnosed the problem, could
>>>> you share details of the stack traces of
>>>> the hung processes and lockdep output to see if the hang is actually
>>>> elsewhere?  What kernel version are you
>>>> testing?
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>
>>>
>>> Hi Ben,
>>>
>>> Here is the evidence with stack traces and lockdep output. My kernel
>>> version is 7.2.0-02677-g544d85de4dc2 (net/main HEAD) and both unpatched
>>> and patched kernels were tested on the same setup(52 mac80211_hwsim
>>> radios (mac80211_hwsim.radios=51)).
>>>
>>> Unpatched kernel:
>>>
>>> The lockdep output shows reg_check_chans_work as the rtnl_mutex holder
>>> and ip as the waiter:
>>>
>>> locks held by kworker/0:2/11408: 3, on CPU#0:
>>> #1: (reg_check_chans).work
>>> #2: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
>>> at: reg_check_chans_work+0xad/0x1330
>>>
>>> locks held by ip/14180: 1, on CPU#0:
>>> #0: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
>>> at: rtnl_getlink+0xbfb/0x13b0
>>>
>>> Hung task call trace:
>>>
>>> INFO: task ip:14180 blocked for more than 5 seconds.
>>> Call Trace:
>>> __schedule+0x1cba/0x6a40
>>> schedule+0xe2/0x2e0
>>> schedule_preempt_disabled+0x13/0x30
>>> __mutex_lock+0x871/0x1cc0
>>> rtnl_getlink+0xbfb/0x13b0
>>> rtnetlink_rcv_msg+0x9a1/0xee0
>>> netlink_rcv_skb+0x186/0x450
>>> netlink_sendmsg+0x8e5/0xde0
>>>
>>> Kernel panic, not syncing: hung_task: blocked tasks
>>>
>>> I injected msleep(200) per wdev in reg_leave_invalid_chans() to model a
>>> slow cfg80211_leave() operation, to measure the hold duration. With 52
>>> interfaces this produced approximately 10.5 seconds of
>>> continuous rtnl_mutex hold:
>>>
>>> cfg80211: reg_check_chans_work: rtnl held for 10499 ms
>>>
>>> The structural problem is clear regardless of the exact duration:
>>> rtnl_mutex is held across all per-interface cleanup for every
>>> registered device in a single acquisition.
>>>
>>> Patched kernel:
>>>
>>> rtnl_mutex is acquired per-device, each hold is brief:
>>>
>>> cfg80211: reg_check_chans_work: device 0 rtnl held 0 ms
>>> cfg80211: reg_check_chans_work: device 1 rtnl held 0 ms
>>> ...
>>> cfg80211: reg_check_chans_work: device 51 rtnl held 0 ms
>>>
>>> No hung tasks were observed.
>>
>> At least my kernel tends to only warn about hung tasks that are blocked
>> for minutes.  Did you adjust your kernel to make this timeout smaller?
> 
> Yes, I reduced hung_task_timeout_secs to 5 seconds for the test.
> With the default 140-second timeout my reproduction would not have
> triggered a hang, though syzbot does with that timeout. Actually, I've
> started to have doubts about the extent of its real-world impact, it
> depends heavily on callback latency and number of active interfaces.
> 
>> Do you actually have 52 real devices that cause this problem?  If so, what
>> hardware is this?  Or you can only reproduce this with modified kernel that
>> injects a 200ms sleep?
> 
> I used mac80211_hwsim virtual radios,
> and the hang could only be reproduced with artificial delays.
> The 200ms value was chosen as a rough estimate of a slow
> driver callback, not based on measured real hardware latency.

So you created a fake problem to make it fit what something
thinks is the root cause.  I don't think that is helpful.

Possibly root cause below is that something else holds wiphy.mtx, so reg_check_chans_work is blocked
and cannot make progress.  Do you have full dmesg dump that includes all blocked processes
and full lockdep dump?  Are you actually able to reproduce the problem w/out adding fake sleeps?

> actual syzbot crash report :
> 
>    locks held by kworker running reg_check_chans_work:
>      #2: rtnl_mutex, at: reg_check_chans_work+0xac net/wireless/reg.c:2469
>      #3: wiphy.mtx,  at: reg_check_chans_work+0x1a1 net/wireless/reg.c:2472
> 
>    locks held by syz-executor (inet6_rtm_newaddr):
>      #0: rtnl_mutex, at: inet6_rtm_newaddr+0x65e
> 
>    INFO: task syz-executor blocked for more than 143 seconds.
> 
> This directly shows reg_check_chans_work holding rtnl_mutex while
> inet6_rtm_newaddr waits — 143 seconds, with the default timeout.

You or your LLM is overly confident of root cause in my opinion.

I will be happy to talk with a human about this since I am interested in
weird wifi problems at scale, but please do not send more LLM responses.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ömer Mete Kaya]

On 9/8/26 23:09, Ben Greear wrote:
> On 9/8/26 12:52, Ömer Mete Kaya wrote:
>>
>>
>> On 9/8/26 19:50, Ben Greear wrote:
>>> On 9/8/26 01:58, Ömer Mete Kaya wrote:
>>>>
>>>>
>>>> On 9/7/26 19:38, Ben Greear wrote:
>>>>> On 9/5/26 5:25 PM, Ömer Mete Kaya wrote:
>>>>>> reg_check_chans_work() holds rtnl_mutex for the entire duration of
>>>>>> iterating over all registered devices and calling cfg80211_leave() on
>>>>>> each invalid wdev. cfg80211_leave() can be slow (disconnect, stop AP,
>>>>>> leave mesh), causing rtnl_mutex starvation when many wireless
>>>>>> interfaces
>>>>>> are present. This results in tasks waiting for rtnl_mutex for longer
>>>>>> than hung_task_timeout_secs:
>>>>>>
>>>>>>      INFO: task hung in inet_rtm_newaddr
>>>>>>      INFO: task hung in inet6_rtm_newaddr
>>>>>>      INFO: task hung in nsim_destroy
>>>>>>      INFO: task hung in tun_chr_close
>>>>>>      INFO: task hung in switchdev_deferred_process_work
>>>>>
>>>>> Hello Omer,
>>>>>
>>>>> Considering that maybe something has mis-diagnosed the problem, could
>>>>> you share details of the stack traces of
>>>>> the hung processes and lockdep output to see if the hang is actually
>>>>> elsewhere?  What kernel version are you
>>>>> testing?
>>>>>
>>>>> Thanks,
>>>>> Ben
>>>>>
>>>>
>>>>
>>>> Hi Ben,
>>>>
>>>> Here is the evidence with stack traces and lockdep output. My kernel
>>>> version is 7.2.0-02677-g544d85de4dc2 (net/main HEAD) and both unpatched
>>>> and patched kernels were tested on the same setup(52 mac80211_hwsim
>>>> radios (mac80211_hwsim.radios=51)).
>>>>
>>>> Unpatched kernel:
>>>>
>>>> The lockdep output shows reg_check_chans_work as the rtnl_mutex holder
>>>> and ip as the waiter:
>>>>
>>>> locks held by kworker/0:2/11408: 3, on CPU#0:
>>>> #1: (reg_check_chans).work
>>>> #2: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
>>>> at: reg_check_chans_work+0xad/0x1330
>>>>
>>>> locks held by ip/14180: 1, on CPU#0:
>>>> #0: ffffffff91118180 (rtnl_mutex){+.+.}-{4:4},
>>>> at: rtnl_getlink+0xbfb/0x13b0
>>>>
>>>> Hung task call trace:
>>>>
>>>> INFO: task ip:14180 blocked for more than 5 seconds.
>>>> Call Trace:
>>>> __schedule+0x1cba/0x6a40
>>>> schedule+0xe2/0x2e0
>>>> schedule_preempt_disabled+0x13/0x30
>>>> __mutex_lock+0x871/0x1cc0
>>>> rtnl_getlink+0xbfb/0x13b0
>>>> rtnetlink_rcv_msg+0x9a1/0xee0
>>>> netlink_rcv_skb+0x186/0x450
>>>> netlink_sendmsg+0x8e5/0xde0
>>>>
>>>> Kernel panic, not syncing: hung_task: blocked tasks
>>>>
>>>> I injected msleep(200) per wdev in reg_leave_invalid_chans() to model a
>>>> slow cfg80211_leave() operation, to measure the hold duration. With 52
>>>> interfaces this produced approximately 10.5 seconds of
>>>> continuous rtnl_mutex hold:
>>>>
>>>> cfg80211: reg_check_chans_work: rtnl held for 10499 ms
>>>>
>>>> The structural problem is clear regardless of the exact duration:
>>>> rtnl_mutex is held across all per-interface cleanup for every
>>>> registered device in a single acquisition.
>>>>
>>>> Patched kernel:
>>>>
>>>> rtnl_mutex is acquired per-device, each hold is brief:
>>>>
>>>> cfg80211: reg_check_chans_work: device 0 rtnl held 0 ms
>>>> cfg80211: reg_check_chans_work: device 1 rtnl held 0 ms
>>>> ...
>>>> cfg80211: reg_check_chans_work: device 51 rtnl held 0 ms
>>>>
>>>> No hung tasks were observed.
>>>
>>> At least my kernel tends to only warn about hung tasks that are blocked
>>> for minutes.  Did you adjust your kernel to make this timeout smaller?
>>
>> Yes, I reduced hung_task_timeout_secs to 5 seconds for the test.
>> With the default 140-second timeout my reproduction would not have
>> triggered a hang, though syzbot does with that timeout. Actually, I've
>> started to have doubts about the extent of its real-world impact, it
>> depends heavily on callback latency and number of active interfaces.
>>
>>> Do you actually have 52 real devices that cause this problem?  If so,
>>> what
>>> hardware is this?  Or you can only reproduce this with modified
>>> kernel that
>>> injects a 200ms sleep?
>>
>> I used mac80211_hwsim virtual radios,
>> and the hang could only be reproduced with artificial delays.
>> The 200ms value was chosen as a rough estimate of a slow
>> driver callback, not based on measured real hardware latency.
> 
> So you created a fake problem to make it fit what something
> thinks is the root cause.  I don't think that is helpful.

Agreed, but I already pointed out "my analysis was primarily code-based".

> Possibly root cause below is that something else holds wiphy.mtx, so
> reg_check_chans_work is blocked
> and cannot make progress. 

Actually a possibility, I wont be oppose since I cannot prove what
absolutely causes that 143 second timeout.

 Do you have full dmesg dump that includes all
> blocked processes
> and full lockdep dump?  Are you actually able to reproduce the problem
> w/out adding fake sleeps?

I cant produce from my own machine without fake sleeps because I lack
the actual hardware or a fuzzing ability as syzbot, its impossible with
a laptop and virtual radios. But according to syzbot's full crash report:

 INFO: task syz-executor:5585 blocked for more than 143 seconds.
      Not tainted syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor    state:D stack:25304 pid:5585  tgid:5585  ppid:1
 task_flags:0x400140 flags:0x00080002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5510 [inline]
 __schedule+0x17d9/0x56c0 kernel/sched/core.c:7234
 __schedule_loop kernel/sched/core.c:7311 [inline]
 schedule+0x164/0x2b0 kernel/sched/core.c:7326
 schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:7383
 __mutex_lock_common kernel/locking/mutex.c:726 [inline]
 __mutex_lock+0x7bf/0x1550 kernel/locking/mutex.c:821
 rtnl_net_lock include/linux/rtnetlink.h:130 [inline]
 inet_rtm_newaddr+0x464/0x19e0 net/ipv4/devinet.c:978
 rtnetlink_rcv_msg+0x802/0xc00 net/core/rtnetlink.c:7076
 netlink_rcv_skb+0x226/0x4a0 net/netlink/af_netlink.c:2556
 netlink_unicast_kernel net/netlink/af_netlink.c:1319 [inline]
 netlink_unicast+0x7bb/0x940 net/netlink/af_netlink.c:1345
 netlink_sendmsg+0x813/0xb40 net/netlink/af_netlink.c:1900
 sock_sendmsg_nosec+0x13a/0x180 net/socket.c:775
 __sock_sendmsg net/socket.c:790 [inline]
 __sys_sendto+0x408/0x5a0 net/socket.c:2252
 __do_sys_sendto net/socket.c:2259 [inline]
 __se_sys_sendto net/socket.c:2255 [inline]
 __x64_sys_sendto+0xde/0x100 net/socket.c:2255
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0x174/0x580 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f0ee475d68e
RSP: 002b:00007ffe75fcc6c8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 000055559003e500 RCX: 00007f0ee475d68e
RDX: 0000000000000028 RSI: 00007f0ee5544670 RDI: 0000000000000003
RBP: 0000000000000001 R08: 00007ffe75fcc744 R09: 000000000000000c
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003
R13: 0000000000000000 R14: 00007f0ee5544670 R15: 0000000000000000
 </TASK>
Showing all locks held in the system:
4 locks held by kworker/0:1/10:
 #0: ffff88801aca5540
((wq_completion)events_power_efficient){+.+.}-{0:0}, at:
process_one_work kernel/workqueue.c:3297 [inline]
 #0: ffff88801aca5540
((wq_completion)events_power_efficient){+.+.}-{0:0}, at:
process_scheduled_works+0xa20/0x14e0 kernel/workqueue.c:3405
 #1: ffffc9000023fc40 ((reg_check_chans).work){+.+.}-{0:0}, at:
process_one_work kernel/workqueue.c:3297 [inline]
 #1: ffffc9000023fc40 ((reg_check_chans).work){+.+.}-{0:0}, at:
process_scheduled_works+0xa20/0x14e0 kernel/workqueue.c:3405
 #2: ffffffff8fded300 (rtnl_mutex){+.+.}-{4:4}, at:
reg_check_chans_work+0xac/0x1110 net/wireless/reg.c:2469
 #3: ffff8880129707a0 (&rdev->wiphy.mtx){+.+.}-{4:4}, at:
class_wiphy_constructor include/net/cfg80211.h:6884 [inline]
 #3: ffff8880129707a0 (&rdev->wiphy.mtx){+.+.}-{4:4}, at:
reg_leave_invalid_chans net/wireless/reg.c:2457 [inline]
 #3: ffff8880129707a0 (&rdev->wiphy.mtx){+.+.}-{4:4}, at:
reg_check_chans_work+0x1a1/0x1110 net/wireless/reg.c:2472
1 lock held by khungtaskd/25:
 #0: ffffffff8e959c20 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire
include/linux/rcupdate.h:300 [inline]
 #0: ffffffff8e959c20 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock
include/linux/rcupdate.h:840 [inline]
 #0: ffffffff8e959c20 (rcu_read_lock){....}-{1:3}, at:
debug_show_all_locks+0x2e/0x180 kernel/locking/lockdep.c:6775
2 locks held by kworker/u4:6/160:
1 lock held by klogd/4693:
1 lock held by dhcpcd/5000:
 #0: ffffffff8fded300 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_net_lock
include/linux/rtnetlink.h:130 [inline]
 #0: ffffffff8fded300 (rtnl_mutex){+.+.}-{4:4}, at:
inet6_rtm_newaddr+0x65e/0xdf0 net/ipv6/addrconf.c:5062
2 locks held by getty/5093:
 #0: ffff88803fac20a0 (&tty->ldisc_sem){++++}-{0:0}, at:
tty_ldisc_ref_wait+0x25/0x70 drivers/tty/tty_ldisc.c:243
 #1: ffffc90000ad82e8 (&ldata->atomic_read_lock){+.+.}-{4:4}, at:
n_tty_read+0x45a/0x1360 drivers/tty/n_tty.c:2211
3 locks held by kworker/u4:0/5354:
6 locks held by kworker/u4:1/5428:
 #0: ffff88801beb4940 ((wq_completion)netns){+.+.}-{0:0}, at:
process_one_work kernel/workqueue.c:3297 [inline]
 #0: ffff88801beb4940 ((wq_completion)netns){+.+.}-{0:0}, at:
process_scheduled_works+0xa20/0x14e0 kernel/workqueue.c:3405
 #1: ffffc90002467c40 (net_cleanup_work){+.+.}-{0:0}, at:
process_one_work kernel/workqueue.c:3297 [inline]
 #1: ffffc90002467c40 (net_cleanup_work){+.+.}-{0:0}, at:
process_scheduled_works+0xa20/0x14e0 kernel/workqueue.c:3405
 #2: ffffffff8fdde748 (pernet_ops_rwsem){++++}-{4:4}, at:
cleanup_net+0xf5/0x810 net/core/net_namespace.c:673
 #3: ffff8880122f4128 (&dev->mutex){....}-{4:4}, at: device_lock
include/linux/device.h:1102 [inline]
 #3: ffff8880122f4128 (&dev->mutex){....}-{4:4}, at: devl_dev_lock
net/devlink/devl_internal.h:124 [inline]
 #3: ffff8880122f4128 (&dev->mutex){....}-{4:4}, at:
devlink_pernet_pre_exit+0x129/0x420 net/devlink/core.c:557

Especially this:
#2: ffffffff8fded300 (rtnl_mutex){+.+.}-{4:4}, at:
reg_check_chans_work+0xac/0x1110 net/wireless/reg.c:2469
#3: ffff8880129707a0 (&rdev->wiphy.mtx){+.+.}-{4:4}, at:
reg_leave_invalid_chans net/wireless/reg.c:2457 [inline]
#3: ffff8880129707a0 (&rdev->wiphy.mtx){+.+.}-{4:4}, at:
reg_check_chans_work+0x1a1/0x1110 net/wireless/reg.c:2472

> 
>> actual syzbot crash report :
>>
>>    locks held by kworker running reg_check_chans_work:
>>      #2: rtnl_mutex, at: reg_check_chans_work+0xac net/wireless/
>> reg.c:2469
>>      #3: wiphy.mtx,  at: reg_check_chans_work+0x1a1 net/wireless/
>> reg.c:2472
>>
>>    locks held by syz-executor (inet6_rtm_newaddr):
>>      #0: rtnl_mutex, at: inet6_rtm_newaddr+0x65e
>>
>>    INFO: task syz-executor blocked for more than 143 seconds.
>>
>> This directly shows reg_check_chans_work holding rtnl_mutex while
>> inet6_rtm_newaddr waits — 143 seconds, with the default timeout.
> 
> You or your LLM is overly confident of root cause in my opinion.

I am not confident, I can stop defending it immediately if I see a
reason to the contrary, I just couldnt find another cause yet. If you
did, please let me know. You said you made a different fix before,
perhaps showing it might open my eyes.

> I will be happy to talk with a human about this since I am interested in
> weird wifi problems at scale, but please do not send more LLM responses.

Was never the case, probably a misunderstanding.

> Thanks,
> Ben
> 
Thanks,
Ömer Mete

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ben Greear]

On 9/8/26 3:32 PM, Ömer Mete Kaya wrote:


> I cant produce from my own machine without fake sleeps because I lack
> the actual hardware or a fuzzing ability as syzbot, its impossible with
> a laptop and virtual radios. But according to syzbot's full crash report:

Assuming you are talking about this:

https://syzkaller.appspot.com/bug?extid=a2de4763f84f61499210

It is interesting.  I don't think it has anything to do with
your attempt to fix, but it certainly shows a lot of tasks that might be blocked,
one of which has rdev->mutex held while trying to acquire more locks.  I do not
see an obvious deadlock, but maybe it is there.

If you decide to post more fixes for random syscaller stuff, at least link the
source material so one can take a look at the larger picture.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[Ömer Mete Kaya]

On 9/9/26 02:40, Ben Greear wrote:
> If you decide to post more fixes for random syscaller stuff, at least
> link the
> source material so one can take a look at the larger picture.

Agreed, thank you for your suggestion.

Good days!

Regards,
Ömer Mete

Re: [PATCH] wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 3 potential
issue(s) to consider:

Critical: 0 · High: 2 · Medium: 1 · Low: 0

- [High] The patch only splits the rtnl critical section at wiphy
  granularity: rtnl_lock() is taken once per snapshot entry and held…
- [High] If a new wiphy device is registered between the two RCU read
  sections, the array population loop will break early and silently skip…
- [Medium] reg_check_chans_work() (net/wireless/reg.c:2464) newly depends
  on a kmalloc_array() allocation and silently abandons the entire…

--

wifi: cfg80211: avoid holding rtnl_mutex across all cfg80211_leave() calls

This changes reg_check_chans_work() to snapshot cfg80211_rdev_list under RCU
with a device reference on each entry, then take rtnl_lock() once per device
instead of once for the whole walk.  A wiphy.registered check is added after
each rtnl acquisition to skip devices that went away in the meantime.

Closes: https://syzkaller.appspot.com/bug?extid=adeb8550754921fece20
Closes: https://syzkaller.appspot.com/bug?extid=101224300649c3eb8af4
Closes: https://syzkaller.appspot.com/bug?extid=8141dcbd23a8f857798a
Closes: https://syzkaller.appspot.com/bug?extid=b0ae8f1abf7d891e0426
Closes: https://syzkaller.appspot.com/bug?extid=d6bbe0f5705cb8a5aa2b

> diff --git a/net/wireless/reg.c b/net/wireless/reg.c
> index a8336baf85dc6..7a3f1e470e143 100644
> --- a/net/wireless/reg.c
> +++ b/net/wireless/reg.c
> @@ -2463,15 +2463,49 @@ static void reg_leave_invalid_chans(struct wiphy *wiphy)
>  
>  static void reg_check_chans_work(struct work_struct *work)
>  {
> -	struct cfg80211_registered_device *rdev;
> +	struct cfg80211_registered_device *rdev, **snap;
> +	int i, n = 0;
>  
>  	pr_debug("Verifying active interfaces after reg change\n");
> -	rtnl_lock();
>  
> -	for_each_rdev(rdev)
> -		reg_leave_invalid_chans(&rdev->wiphy);
> +	/*
> +	 * Snapshot rdev pointers under RCU with a device reference so they
> +	 * cannot be freed between per-device rtnl acquisitions. Using a
> +	 * per-device rtnl_lock() instead of holding it across all devices
> +	 * avoids starving other rtnl waiters when cfg80211_leave() is slow.
> +	 */
> +	rcu_read_lock();
> +	list_for_each_entry_rcu(rdev, &cfg80211_rdev_list, list)
> +		n++;
> +	rcu_read_unlock();
>  
> -	rtnl_unlock();
> +	if (!n)
> +		return;
> +
> +	snap = kmalloc_array(n, sizeof(*snap), GFP_KERNEL);
> +	if (!snap)
> +		return;

[Severity: Medium]
Does this silently drop the whole regulatory enforcement pass when the
allocation fails?

Before this change reg_check_chans_work() could not fail: it was just
rtnl_lock() + for_each_rdev() + reg_leave_invalid_chans().  Now a transient
kmalloc_array() failure returns without touching any wiphy, so
reg_leave_invalid_chans() never runs for any device in this round.

The only place that queues the work is reg_check_channels():

net/wireless/reg.c:reg_check_channels() {
	mod_delayed_work(system_power_efficient_wq,
			 &reg_check_chans,
			 msecs_to_jiffies(REG_ENFORCE_GRACE_MS));
}

and its callers are all regulatory-change events (set_reg request, country
IE, driver hint, indoor flag).  Nothing re-arms the work on this failure
path, so interfaces that the newly applied regdomain forbids (AP beaconing,
associated STA, joined mesh) would keep running until some unrelated later
regdomain change happens to queue the work again.

Would rescheduling the work on allocation failure, or avoiding the
snapshot array entirely (for example restarting from the list head using
the rdev list generation counter), be preferable here?

> +
> +	i = 0;
> +	rcu_read_lock();
> +	list_for_each_entry_rcu(rdev, &cfg80211_rdev_list, list) {
> +		if (i >= n)
> +			break;
> +		get_device(&rdev->wiphy.dev);
> +		snap[i++] = rdev;
> +	}
> +	rcu_read_unlock();
> +	n = i;
> +
> +	for (i = 0; i < n; i++) {
> +		rtnl_lock();
> +		if (snap[i]->wiphy.registered)
> +			reg_leave_invalid_chans(&snap[i]->wiphy);
> +		rtnl_unlock();
> +		put_device(&snap[i]->wiphy.dev);
> +	}

[Severity: High]
The count pass and the fill pass are two separate RCU read
sections with nothing held in between, so n can be stale by the
time snap[] is populated - and the truncation is silent.

wiphy_register() publishes a new radio with
list_add_rcu(&rdev->list, &cfg80211_rdev_list), i.e. at the head
of the list.  If a device is registered (USB adapter plugged in, a
new radio created) between the two rcu_read_lock() sections, the
second list_for_each_entry_rcu() starts at that brand new rdev,
fills snap[] with the first n entries and then hits the i >= n
break.  The entries that fall off are the ones at the *tail*, i.e.
the oldest radios - exactly the ones most likely to have
associated STAs, beaconing APs or joined meshes that the new
regdomain forbids.  reg_leave_invalid_chans() is never called for
them in this round, and since reg_check_channels() is only armed
from regulatory-change events, nothing retries; those interfaces
keep operating on channels the new regdomain disallows.

Note this is a functional regression against the previous
for_each_rdev() walk under rtnl, which by construction saw a
consistent list.

cfg80211_rdev_list is written under rtnl, so would it be simpler
to build the snapshot in a single pass under rtnl_lock() (counting
and get_device() together, or via cfg80211_rdev_list_generation),
then drop rtnl and re-acquire it per device for the
reg_leave_invalid_chans() calls?  If you want to keep the RCU
two-pass form, what is the plan for the list growing between the
passes - re-count and retry, or over-allocate and re-check?

[Severity: High]
Does this actually bound the rtnl hold time?  The split here is at wiphy
granularity only: rtnl is taken once and held across the complete
reg_leave_invalid_chans() pass for that wiphy.

net/wireless/reg.c:reg_leave_invalid_chans() {
	list_for_each_entry(wdev, &rdev->wiphy.wdev_list, list) {
		bool valid;

		scoped_guard(wiphy, wiphy)
			valid = reg_wdev_chan_valid(wiphy, wdev);
		if (!valid)
			cfg80211_leave(rdev, wdev, -1);
	}
}

That inner loop is unchanged and can call cfg80211_leave() an unbounded
number of times, and each call goes into driver ops:

reg_leave_invalid_chans() -> cfg80211_leave() -> cfg80211_leave_locked() ->
cfg80211_disconnect() / cfg80211_stop_ap() / cfg80211_leave_mesh() /
cfg80211_stop_nan()

with cfg80211_leave() itself requiring rtnl:

net/wireless/core.c:cfg80211_leave() {
	ASSERT_RTNL();
	...
	guard(wiphy)(&rdev->wiphy);

	cfg80211_leave_locked(rdev, wdev, link_id);
}

So when the interfaces are concentrated on one wiphy (many wdevs on a single
radio, or one slow leave op), the longest contiguous rtnl section is the same
as before, and rtnl waiters such as inet_rtm_newaddr, tun_chr_close and
nsim_destroy can still exceed hung_task_timeout_secs.  Is that consistent
with the commit message claim that the reported hangs are closed?

Wouldn't a fix need to drop and re-acquire rtnl between wdevs, with
per-wdev re-validation after re-acquiring (re-lookup by wiphy_idx plus a
wdev registered re-check), rather than only between devices?

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20260906002657.620076-2-omermetekaya0%40gmail.com