[tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread

[tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread

[kernel test robot]

Hello,


as we understand, this commit is not the root cause of the
possible_recursive_locking_detected issue.
but due to the renaming, the detail stats change from form (1) to (2).

we failed to bisect to real first bad commit for
"possible_recursive_locking_detected" issue. so just make out this report FYI
that there is this issue caused by related code.


=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/runtime/test/torture_type:
  vm-snb/rcutorture/debian-11.1-i386-20220923.cgz/i386-randconfig-062-20251022/clang-20/300s/cpuhotplug/tasks-tracing


abfc01077df66593 b079d93796528053cde322f2ca8
---------------- ---------------------------
       fail:runs  %reproduction    fail:runs
           |             |             |
          6:6            0%           6:6     dmesg.WARNING:possible_recursive_locking_detected
          6:6         -100%            :6     dmesg.WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:do_set_cpus_allowed_but_task_is_already_holding_lock:at:cpu_stopper_thread          <-------- (1)
           :6          100%           6:6     dmesg.WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread       <-------- (2)



kernel test robot noticed "WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread" on:

commit: b079d93796528053cde322f2ca838c2d21c297e7 ("sched: Rename do_set_cpus_allowed()")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core

[test failed on linux-next/master 72fb0170ef1f45addf726319c52a0562b6913707]

in testcase: rcutorture
version: 
with following parameters:

	runtime: 300s
	test: cpuhotplug
	torture_type: tasks-tracing



config: i386-randconfig-062-20251022
compiler: clang-20
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G

(please refer to attached dmesg/kmsg for entire log/backtrace)



If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202510271206.24495a68-lkp@intel.com


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20251027/202510271206.24495a68-lkp@intel.com


[  116.814009][   T21] 
[  116.814488][   T21] ============================================
[  116.815227][   T21] WARNING: possible recursive locking detected
[  116.815957][   T21] 6.18.0-rc1-00014-gb079d9379652 #1 Tainted: G S                 
[  116.816878][   T21] --------------------------------------------
[  116.817602][   T21] migration/1/21 is trying to acquire lock:
[  116.818301][   T21] ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: set_cpus_allowed_force+0x3c/0xc0
[  116.820432][   T21] 
[  116.820432][   T21] but task is already holding lock:
[  116.821314][   T21] ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: cpu_stopper_thread+0x93/0x170
[  116.822291][   T21] 
[  116.822291][   T21] other info that might help us debug this:
[  116.826420][   T21]  Possible unsafe locking scenario:
[  116.826420][   T21] 
[  116.836196][   T21]        CPU0
[  116.836895][   T21]        ----
[  116.837592][   T21]   lock(&rq->__lock);
[  116.838388][   T21]   lock(&rq->__lock);
[  116.839558][   T21] 
[  116.839558][   T21]  *** DEADLOCK ***
[  116.839558][   T21] 
[  116.841003][   T21]  May be due to missing lock nesting notation
[  116.841003][   T21] 
[  116.842427][   T21] 2 locks held by migration/1/21:
[  116.843393][   T21]  #0: b92d06dc (&p->pi_lock){-.-.}-{2:2}, at: __balance_push_cpu_stop+0x28/0x2b0
[  116.845044][   T21]  #1: ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: cpu_stopper_thread+0x93/0x170
[  116.846669][   T21] 
[  116.846669][   T21] stack backtrace:
[  116.847890][   T21] CPU: 1 UID: 0 PID: 21 Comm: migration/1 Tainted: G S                  6.18.0-rc1-00014-gb079d9379652 #1 NONE  6d63d2e836521c1c681a07c673117fb98e4815ab
[  116.847897][   T21] Tainted: [S]=CPU_OUT_OF_SPEC
[  116.847898][   T21] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[  116.847901][   T21] Stopper: __balance_push_cpu_stop+0x0/0x2b0 <- finish_lock_switch+0x7d/0xd0
[  116.847909][   T21] Call Trace:
[  116.847914][   T21]  ? dump_stack_lvl+0xa4/0xdc
[  116.847919][   T21]  ? print_deadlock_bug+0x2df/0x300
[  116.847925][   T21]  ? __lock_acquire+0x268c/0x2ce0
[  116.847929][   T21]  ? __lock_acquire+0x601/0x2ce0
[  116.847933][   T21]  ? __lock_acquire+0x601/0x2ce0
[  116.847939][   T21]  ? lock_acquire+0xc3/0x1f0
[  116.847943][   T21]  ? set_cpus_allowed_force+0x3c/0xc0
[  116.847947][   T21]  ? lock_acquire+0xc3/0x1f0
[  116.847952][   T21]  ? __task_rq_lock+0x73/0x1d0
[  116.847955][   T21]  ? set_cpus_allowed_force+0x3c/0xc0
[  116.847959][   T21]  ? set_cpus_allowed_force+0x3c/0xc0
[  116.847962][   T21]  ? __balance_push_cpu_stop+0x136/0x2b0
[  116.847966][   T21]  ? select_fallback_rq+0x148/0x230
[  116.847970][   T21]  ? __balance_push_cpu_stop+0x163/0x2b0
[  116.847974][   T21]  ? cpu_stopper_thread+0x93/0x170
[  116.847978][   T21]  ? raw_spin_rq_lock_nested+0xb0/0xb0
[  116.847982][   T21]  ? smpboot_thread_fn+0x11b/0x260
[  116.847986][   T21]  ? kthread+0x2ef/0x330
[  116.847992][   T21]  ? trace_hardirqs_on+0x76/0xe0
[  116.847996][   T21]  ? kthreadd+0x2a0/0x2a0
[  116.847999][   T21]  ? __smpboot_create_thread+0x1c0/0x1c0
[  116.848003][   T21]  ? schedule_tail+0xa6/0x100
[  116.848006][   T21]  ? kthreadd+0x2a0/0x2a0
[  116.848009][   T21]  ? kthreadd+0x2a0/0x2a0
[  116.848012][   T21]  ? ret_from_fork+0x1cd/0x290
[  116.848017][   T21]  ? kthreadd+0x2a0/0x2a0
[  116.848020][   T21]  ? ret_from_fork_asm+0x12/0x18
[  116.848023][   T21]  ? entry_INT80_32+0xf0/0xf0


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread

[Peter Zijlstra]

On Mon, Oct 27, 2025 at 01:14:09PM +0800, kernel test robot wrote:

> kernel test robot noticed "WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread" on:
> 
> commit: b079d93796528053cde322f2ca838c2d21c297e7 ("sched: Rename do_set_cpus_allowed()")
> https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core

Your biscect went sideways, it is, as Jan correctly found:

  abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")


Anyway, this was helpful:

> [  116.814488][   T21] ============================================
> [  116.815227][   T21] WARNING: possible recursive locking detected
> [  116.815957][   T21] 6.18.0-rc1-00014-gb079d9379652 #1 Tainted: G S                 
> [  116.816878][   T21] --------------------------------------------
> [  116.817602][   T21] migration/1/21 is trying to acquire lock:
> [  116.818301][   T21] ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: set_cpus_allowed_force+0x3c/0xc0
> [  116.820432][   T21] 
> [  116.820432][   T21] but task is already holding lock:
> [  116.821314][   T21] ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: cpu_stopper_thread+0x93/0x170

> [  116.841003][   T21] 
> [  116.842427][   T21] 2 locks held by migration/1/21:
> [  116.843393][   T21]  #0: b92d06dc (&p->pi_lock){-.-.}-{2:2}, at: __balance_push_cpu_stop+0x28/0x2b0
> [  116.845044][   T21]  #1: ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: cpu_stopper_thread+0x93/0x170
> [  116.846669][   T21] 
> [  116.846669][   T21] stack backtrace:
> [  116.847890][   T21] CPU: 1 UID: 0 PID: 21 Comm: migration/1 Tainted: G S                  6.18.0-rc1-00014-gb079d9379652 #1 NONE  6d63d2e836521c1c681a07c673117fb98e4815ab
> [  116.847897][   T21] Tainted: [S]=CPU_OUT_OF_SPEC
> [  116.847898][   T21] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [  116.847901][   T21] Stopper: __balance_push_cpu_stop+0x0/0x2b0 <- finish_lock_switch+0x7d/0xd0
> [  116.847909][   T21] Call Trace:

> [  116.847939][   T21]  ? lock_acquire+0xc3/0x1f0
> [  116.847943][   T21]  ? set_cpus_allowed_force+0x3c/0xc0
> [  116.847947][   T21]  ? lock_acquire+0xc3/0x1f0
> [  116.847952][   T21]  ? __task_rq_lock+0x73/0x1d0
> [  116.847955][   T21]  ? set_cpus_allowed_force+0x3c/0xc0
> [  116.847959][   T21]  ? set_cpus_allowed_force+0x3c/0xc0
> [  116.847962][   T21]  ? __balance_push_cpu_stop+0x136/0x2b0
> [  116.847966][   T21]  ? select_fallback_rq+0x148/0x230
> [  116.847970][   T21]  ? __balance_push_cpu_stop+0x163/0x2b0
> [  116.847974][   T21]  ? cpu_stopper_thread+0x93/0x170

Clearly I missed that case :/

---
Subject: sched: Fix the do_set_cpus_allowed() locking fix

Commit abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
overlooked that __balance_push_cpu_stop() calls select_fallback_rq()
with rq->lock held. This makes that set_cpus_allowed_force() will
recursively take rq->lock and the machine locks up.

Run select_fallback_rq() earlier, without holding rq->lock. This opens
up a race window where a task could get migrated out from under us, but
that is harmless, we want the task migrated.

select_fallback_rq() itself will not be subject to concurrency as it
will be fully serialized by p->pi_lock, so there is no chance of
set_cpus_allowed_force() getting called with different arguments and
selecting different fallback CPUs for one task.

Fixes: abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
Reported-by: Jan Polensky <japo@linux.ibm.com>
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Closes: https://lore.kernel.org/oe-lkp/202510271206.24495a68-lkp@intel.com
---
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1842285eac1e..67b5f2faab36 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8044,18 +8044,15 @@ static int __balance_push_cpu_stop(void *arg)
 	struct rq_flags rf;
 	int cpu;
 
-	raw_spin_lock_irq(&p->pi_lock);
-	rq_lock(rq, &rf);
-
-	update_rq_clock(rq);
-
-	if (task_rq(p) == rq && task_on_rq_queued(p)) {
+	scoped_guard (raw_spinlock_irq, &p->pi_lock) {
 		cpu = select_fallback_rq(rq->cpu, p);
-		rq = __migrate_task(rq, &rf, p, cpu);
-	}
 
-	rq_unlock(rq, &rf);
-	raw_spin_unlock_irq(&p->pi_lock);
+		rq_lock(rq, &rf);
+		update_rq_clock(rq);
+		if (task_rq(p) == rq && task_on_rq_queued(p))
+			rq = __migrate_task(rq, &rf, p, cpu);
+		rq_unlock(rq, &rf);
+	}
 
 	put_task_struct(p);
 

Re: [tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread

[Peter Zijlstra]

On Mon, Oct 27, 2025 at 12:01:33PM +0100, Peter Zijlstra wrote:

Could someone confirm this fixes the problem?

> ---
> Subject: sched: Fix the do_set_cpus_allowed() locking fix
> 
> Commit abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
> overlooked that __balance_push_cpu_stop() calls select_fallback_rq()
> with rq->lock held. This makes that set_cpus_allowed_force() will
> recursively take rq->lock and the machine locks up.
> 
> Run select_fallback_rq() earlier, without holding rq->lock. This opens
> up a race window where a task could get migrated out from under us, but
> that is harmless, we want the task migrated.
> 
> select_fallback_rq() itself will not be subject to concurrency as it
> will be fully serialized by p->pi_lock, so there is no chance of
> set_cpus_allowed_force() getting called with different arguments and
> selecting different fallback CPUs for one task.
> 
> Fixes: abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
> Reported-by: Jan Polensky <japo@linux.ibm.com>
> Reported-by: kernel test robot <oliver.sang@intel.com>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Closes: https://lore.kernel.org/oe-lkp/202510271206.24495a68-lkp@intel.com
> ---
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 1842285eac1e..67b5f2faab36 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -8044,18 +8044,15 @@ static int __balance_push_cpu_stop(void *arg)
>  	struct rq_flags rf;
>  	int cpu;
>  
> -	raw_spin_lock_irq(&p->pi_lock);
> -	rq_lock(rq, &rf);
> -
> -	update_rq_clock(rq);
> -
> -	if (task_rq(p) == rq && task_on_rq_queued(p)) {
> +	scoped_guard (raw_spinlock_irq, &p->pi_lock) {
>  		cpu = select_fallback_rq(rq->cpu, p);
> -		rq = __migrate_task(rq, &rf, p, cpu);
> -	}
>  
> -	rq_unlock(rq, &rf);
> -	raw_spin_unlock_irq(&p->pi_lock);
> +		rq_lock(rq, &rf);
> +		update_rq_clock(rq);
> +		if (task_rq(p) == rq && task_on_rq_queued(p))
> +			rq = __migrate_task(rq, &rf, p, cpu);
> +		rq_unlock(rq, &rf);
> +	}
>  
>  	put_task_struct(p);
>  

Re: [tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread

[Jan Polensky]

On Tue, Oct 28, 2025 at 10:03:24AM +0100, Peter Zijlstra wrote:
> On Mon, Oct 27, 2025 at 12:01:33PM +0100, Peter Zijlstra wrote:
>
> Could someone confirm this fixes the problem?
>
Tested-by: Jan Polensky <japo@linux.ibm.com>

Thank you for the quick fix. I’ve verified the patch on s390x and can confirm that it resolves the
spin lock issue.

[tip: sched/core] sched: Fix the do_set_cpus_allowed() locking fix

[tip-bot2 for Peter Zijlstra]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     321f8bdcec1fe81b8ff6d558b1f9f25b751116a5
Gitweb:        https://git.kernel.org/tip/321f8bdcec1fe81b8ff6d558b1f9f25b751116a5
Author:        Peter Zijlstra <peterz@infradead.org>
AuthorDate:    Mon, 27 Oct 2025 12:01:33 +01:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 28 Oct 2025 12:37:38 +01:00

sched: Fix the do_set_cpus_allowed() locking fix

Commit abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
overlooked that __balance_push_cpu_stop() calls select_fallback_rq()
with rq->lock held. This makes that set_cpus_allowed_force() will
recursively take rq->lock and the machine locks up.

Run select_fallback_rq() earlier, without holding rq->lock. This opens
up a race window where a task could get migrated out from under us, but
that is harmless, we want the task migrated.

select_fallback_rq() itself will not be subject to concurrency as it
will be fully serialized by p->pi_lock, so there is no chance of
set_cpus_allowed_force() getting called with different arguments and
selecting different fallback CPUs for one task.

Fixes: abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
Reported-by: Jan Polensky <japo@linux.ibm.com>
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Jan Polensky <japo@linux.ibm.com>
Closes: https://lore.kernel.org/oe-lkp/202510271206.24495a68-lkp@intel.com
Link: https://patch.msgid.link/20251027110133.GI3245006@noisy.programming.kicks-ass.net
---
 kernel/sched/core.c | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1842285..67b5f2f 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8044,18 +8044,15 @@ static int __balance_push_cpu_stop(void *arg)
 	struct rq_flags rf;
 	int cpu;
 
-	raw_spin_lock_irq(&p->pi_lock);
-	rq_lock(rq, &rf);
-
-	update_rq_clock(rq);
-
-	if (task_rq(p) == rq && task_on_rq_queued(p)) {
+	scoped_guard (raw_spinlock_irq, &p->pi_lock) {
 		cpu = select_fallback_rq(rq->cpu, p);
-		rq = __migrate_task(rq, &rf, p, cpu);
-	}
 
-	rq_unlock(rq, &rf);
-	raw_spin_unlock_irq(&p->pi_lock);
+		rq_lock(rq, &rf);
+		update_rq_clock(rq);
+		if (task_rq(p) == rq && task_on_rq_queued(p))
+			rq = __migrate_task(rq, &rf, p, cpu);
+		rq_unlock(rq, &rf);
+	}
 
 	put_task_struct(p);
 

[tip: sched/core] sched: Fix the do_set_cpus_allowed() locking fix

[tip-bot2 for Peter Zijlstra]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     af13e5e437dc2eb8a3291aad70fc80d9cc78bc73
Gitweb:        https://git.kernel.org/tip/af13e5e437dc2eb8a3291aad70fc80d9cc78bc73
Author:        Peter Zijlstra <peterz@infradead.org>
AuthorDate:    Mon, 27 Oct 2025 12:01:33 +01:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 28 Oct 2025 15:00:48 +01:00

sched: Fix the do_set_cpus_allowed() locking fix

Commit abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
overlooked that __balance_push_cpu_stop() calls select_fallback_rq()
with rq->lock held. This makes that set_cpus_allowed_force() will
recursively take rq->lock and the machine locks up.

Run select_fallback_rq() earlier, without holding rq->lock. This opens
up a race window where a task could get migrated out from under us, but
that is harmless, we want the task migrated.

select_fallback_rq() itself will not be subject to concurrency as it
will be fully serialized by p->pi_lock, so there is no chance of
set_cpus_allowed_force() getting called with different arguments and
selecting different fallback CPUs for one task.

Fixes: abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
Reported-by: Jan Polensky <japo@linux.ibm.com>
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Jan Polensky <japo@linux.ibm.com>
Closes: https://lore.kernel.org/oe-lkp/202510271206.24495a68-lkp@intel.com
Link: https://patch.msgid.link/20251027110133.GI3245006@noisy.programming.kicks-ass.net
---
 kernel/sched/core.c | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 096e8d0..fd9ff69 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8044,18 +8044,15 @@ static int __balance_push_cpu_stop(void *arg)
 	struct rq_flags rf;
 	int cpu;
 
-	raw_spin_lock_irq(&p->pi_lock);
-	rq_lock(rq, &rf);
-
-	update_rq_clock(rq);
-
-	if (task_rq(p) == rq && task_on_rq_queued(p)) {
+	scoped_guard (raw_spinlock_irq, &p->pi_lock) {
 		cpu = select_fallback_rq(rq->cpu, p);
-		rq = __migrate_task(rq, &rf, p, cpu);
-	}
 
-	rq_unlock(rq, &rf);
-	raw_spin_unlock_irq(&p->pi_lock);
+		rq_lock(rq, &rf);
+		update_rq_clock(rq);
+		if (task_rq(p) == rq && task_on_rq_queued(p))
+			rq = __migrate_task(rq, &rf, p, cpu);
+		rq_unlock(rq, &rf);
+	}
 
 	put_task_struct(p);