* [tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread
@ 2025-10-27 5:14 kernel test robot
2025-10-27 11:01 ` Peter Zijlstra
0 siblings, 1 reply; 4+ messages in thread
From: kernel test robot @ 2025-10-27 5:14 UTC (permalink / raw)
To: Peter Zijlstra
Cc: oe-lkp, lkp, linux-kernel, x86, Juri Lelli, Tejun Heo,
Vincent Guittot, cgroups, aubrey.li, yu.c.chen, oliver.sang
Hello,
as we understand, this commit is not the root cause of the
possible_recursive_locking_detected issue.
but due to the renaming, the detail stats change from form (1) to (2).
we failed to bisect to real first bad commit for
"possible_recursive_locking_detected" issue. so just make out this report FYI
that there is this issue caused by related code.
=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/runtime/test/torture_type:
vm-snb/rcutorture/debian-11.1-i386-20220923.cgz/i386-randconfig-062-20251022/clang-20/300s/cpuhotplug/tasks-tracing
abfc01077df66593 b079d93796528053cde322f2ca8
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
6:6 0% 6:6 dmesg.WARNING:possible_recursive_locking_detected
6:6 -100% :6 dmesg.WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:do_set_cpus_allowed_but_task_is_already_holding_lock:at:cpu_stopper_thread <-------- (1)
:6 100% 6:6 dmesg.WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread <-------- (2)
kernel test robot noticed "WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread" on:
commit: b079d93796528053cde322f2ca838c2d21c297e7 ("sched: Rename do_set_cpus_allowed()")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core
[test failed on linux-next/master 72fb0170ef1f45addf726319c52a0562b6913707]
in testcase: rcutorture
version:
with following parameters:
runtime: 300s
test: cpuhotplug
torture_type: tasks-tracing
config: i386-randconfig-062-20251022
compiler: clang-20
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202510271206.24495a68-lkp@intel.com
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20251027/202510271206.24495a68-lkp@intel.com
[ 116.814009][ T21]
[ 116.814488][ T21] ============================================
[ 116.815227][ T21] WARNING: possible recursive locking detected
[ 116.815957][ T21] 6.18.0-rc1-00014-gb079d9379652 #1 Tainted: G S
[ 116.816878][ T21] --------------------------------------------
[ 116.817602][ T21] migration/1/21 is trying to acquire lock:
[ 116.818301][ T21] ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: set_cpus_allowed_force+0x3c/0xc0
[ 116.820432][ T21]
[ 116.820432][ T21] but task is already holding lock:
[ 116.821314][ T21] ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: cpu_stopper_thread+0x93/0x170
[ 116.822291][ T21]
[ 116.822291][ T21] other info that might help us debug this:
[ 116.826420][ T21] Possible unsafe locking scenario:
[ 116.826420][ T21]
[ 116.836196][ T21] CPU0
[ 116.836895][ T21] ----
[ 116.837592][ T21] lock(&rq->__lock);
[ 116.838388][ T21] lock(&rq->__lock);
[ 116.839558][ T21]
[ 116.839558][ T21] *** DEADLOCK ***
[ 116.839558][ T21]
[ 116.841003][ T21] May be due to missing lock nesting notation
[ 116.841003][ T21]
[ 116.842427][ T21] 2 locks held by migration/1/21:
[ 116.843393][ T21] #0: b92d06dc (&p->pi_lock){-.-.}-{2:2}, at: __balance_push_cpu_stop+0x28/0x2b0
[ 116.845044][ T21] #1: ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: cpu_stopper_thread+0x93/0x170
[ 116.846669][ T21]
[ 116.846669][ T21] stack backtrace:
[ 116.847890][ T21] CPU: 1 UID: 0 PID: 21 Comm: migration/1 Tainted: G S 6.18.0-rc1-00014-gb079d9379652 #1 NONE 6d63d2e836521c1c681a07c673117fb98e4815ab
[ 116.847897][ T21] Tainted: [S]=CPU_OUT_OF_SPEC
[ 116.847898][ T21] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 116.847901][ T21] Stopper: __balance_push_cpu_stop+0x0/0x2b0 <- finish_lock_switch+0x7d/0xd0
[ 116.847909][ T21] Call Trace:
[ 116.847914][ T21] ? dump_stack_lvl+0xa4/0xdc
[ 116.847919][ T21] ? print_deadlock_bug+0x2df/0x300
[ 116.847925][ T21] ? __lock_acquire+0x268c/0x2ce0
[ 116.847929][ T21] ? __lock_acquire+0x601/0x2ce0
[ 116.847933][ T21] ? __lock_acquire+0x601/0x2ce0
[ 116.847939][ T21] ? lock_acquire+0xc3/0x1f0
[ 116.847943][ T21] ? set_cpus_allowed_force+0x3c/0xc0
[ 116.847947][ T21] ? lock_acquire+0xc3/0x1f0
[ 116.847952][ T21] ? __task_rq_lock+0x73/0x1d0
[ 116.847955][ T21] ? set_cpus_allowed_force+0x3c/0xc0
[ 116.847959][ T21] ? set_cpus_allowed_force+0x3c/0xc0
[ 116.847962][ T21] ? __balance_push_cpu_stop+0x136/0x2b0
[ 116.847966][ T21] ? select_fallback_rq+0x148/0x230
[ 116.847970][ T21] ? __balance_push_cpu_stop+0x163/0x2b0
[ 116.847974][ T21] ? cpu_stopper_thread+0x93/0x170
[ 116.847978][ T21] ? raw_spin_rq_lock_nested+0xb0/0xb0
[ 116.847982][ T21] ? smpboot_thread_fn+0x11b/0x260
[ 116.847986][ T21] ? kthread+0x2ef/0x330
[ 116.847992][ T21] ? trace_hardirqs_on+0x76/0xe0
[ 116.847996][ T21] ? kthreadd+0x2a0/0x2a0
[ 116.847999][ T21] ? __smpboot_create_thread+0x1c0/0x1c0
[ 116.848003][ T21] ? schedule_tail+0xa6/0x100
[ 116.848006][ T21] ? kthreadd+0x2a0/0x2a0
[ 116.848009][ T21] ? kthreadd+0x2a0/0x2a0
[ 116.848012][ T21] ? ret_from_fork+0x1cd/0x290
[ 116.848017][ T21] ? kthreadd+0x2a0/0x2a0
[ 116.848020][ T21] ? ret_from_fork_asm+0x12/0x18
[ 116.848023][ T21] ? entry_INT80_32+0xf0/0xf0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread
2025-10-27 5:14 [tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread kernel test robot
@ 2025-10-27 11:01 ` Peter Zijlstra
2025-10-28 9:03 ` Peter Zijlstra
0 siblings, 1 reply; 4+ messages in thread
From: Peter Zijlstra @ 2025-10-27 11:01 UTC (permalink / raw)
To: kernel test robot, japo
Cc: oe-lkp, lkp, linux-kernel, x86, Juri Lelli, Tejun Heo,
Vincent Guittot, cgroups, aubrey.li, yu.c.chen
On Mon, Oct 27, 2025 at 01:14:09PM +0800, kernel test robot wrote:
> kernel test robot noticed "WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread" on:
>
> commit: b079d93796528053cde322f2ca838c2d21c297e7 ("sched: Rename do_set_cpus_allowed()")
> https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core
Your biscect went sideways, it is, as Jan correctly found:
abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
Anyway, this was helpful:
> [ 116.814488][ T21] ============================================
> [ 116.815227][ T21] WARNING: possible recursive locking detected
> [ 116.815957][ T21] 6.18.0-rc1-00014-gb079d9379652 #1 Tainted: G S
> [ 116.816878][ T21] --------------------------------------------
> [ 116.817602][ T21] migration/1/21 is trying to acquire lock:
> [ 116.818301][ T21] ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: set_cpus_allowed_force+0x3c/0xc0
> [ 116.820432][ T21]
> [ 116.820432][ T21] but task is already holding lock:
> [ 116.821314][ T21] ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: cpu_stopper_thread+0x93/0x170
> [ 116.841003][ T21]
> [ 116.842427][ T21] 2 locks held by migration/1/21:
> [ 116.843393][ T21] #0: b92d06dc (&p->pi_lock){-.-.}-{2:2}, at: __balance_push_cpu_stop+0x28/0x2b0
> [ 116.845044][ T21] #1: ee7f1930 (&rq->__lock){-.-.}-{2:2}, at: cpu_stopper_thread+0x93/0x170
> [ 116.846669][ T21]
> [ 116.846669][ T21] stack backtrace:
> [ 116.847890][ T21] CPU: 1 UID: 0 PID: 21 Comm: migration/1 Tainted: G S 6.18.0-rc1-00014-gb079d9379652 #1 NONE 6d63d2e836521c1c681a07c673117fb98e4815ab
> [ 116.847897][ T21] Tainted: [S]=CPU_OUT_OF_SPEC
> [ 116.847898][ T21] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [ 116.847901][ T21] Stopper: __balance_push_cpu_stop+0x0/0x2b0 <- finish_lock_switch+0x7d/0xd0
> [ 116.847909][ T21] Call Trace:
> [ 116.847939][ T21] ? lock_acquire+0xc3/0x1f0
> [ 116.847943][ T21] ? set_cpus_allowed_force+0x3c/0xc0
> [ 116.847947][ T21] ? lock_acquire+0xc3/0x1f0
> [ 116.847952][ T21] ? __task_rq_lock+0x73/0x1d0
> [ 116.847955][ T21] ? set_cpus_allowed_force+0x3c/0xc0
> [ 116.847959][ T21] ? set_cpus_allowed_force+0x3c/0xc0
> [ 116.847962][ T21] ? __balance_push_cpu_stop+0x136/0x2b0
> [ 116.847966][ T21] ? select_fallback_rq+0x148/0x230
> [ 116.847970][ T21] ? __balance_push_cpu_stop+0x163/0x2b0
> [ 116.847974][ T21] ? cpu_stopper_thread+0x93/0x170
Clearly I missed that case :/
---
Subject: sched: Fix the do_set_cpus_allowed() locking fix
Commit abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
overlooked that __balance_push_cpu_stop() calls select_fallback_rq()
with rq->lock held. This makes that set_cpus_allowed_force() will
recursively take rq->lock and the machine locks up.
Run select_fallback_rq() earlier, without holding rq->lock. This opens
up a race window where a task could get migrated out from under us, but
that is harmless, we want the task migrated.
select_fallback_rq() itself will not be subject to concurrency as it
will be fully serialized by p->pi_lock, so there is no chance of
set_cpus_allowed_force() getting called with different arguments and
selecting different fallback CPUs for one task.
Fixes: abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
Reported-by: Jan Polensky <japo@linux.ibm.com>
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Closes: https://lore.kernel.org/oe-lkp/202510271206.24495a68-lkp@intel.com
---
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1842285eac1e..67b5f2faab36 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8044,18 +8044,15 @@ static int __balance_push_cpu_stop(void *arg)
struct rq_flags rf;
int cpu;
- raw_spin_lock_irq(&p->pi_lock);
- rq_lock(rq, &rf);
-
- update_rq_clock(rq);
-
- if (task_rq(p) == rq && task_on_rq_queued(p)) {
+ scoped_guard (raw_spinlock_irq, &p->pi_lock) {
cpu = select_fallback_rq(rq->cpu, p);
- rq = __migrate_task(rq, &rf, p, cpu);
- }
- rq_unlock(rq, &rf);
- raw_spin_unlock_irq(&p->pi_lock);
+ rq_lock(rq, &rf);
+ update_rq_clock(rq);
+ if (task_rq(p) == rq && task_on_rq_queued(p))
+ rq = __migrate_task(rq, &rf, p, cpu);
+ rq_unlock(rq, &rf);
+ }
put_task_struct(p);
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread
2025-10-27 11:01 ` Peter Zijlstra
@ 2025-10-28 9:03 ` Peter Zijlstra
2025-10-28 11:29 ` Jan Polensky
0 siblings, 1 reply; 4+ messages in thread
From: Peter Zijlstra @ 2025-10-28 9:03 UTC (permalink / raw)
To: kernel test robot, japo
Cc: oe-lkp, lkp, linux-kernel, x86, Juri Lelli, Tejun Heo,
Vincent Guittot, cgroups, aubrey.li, yu.c.chen
On Mon, Oct 27, 2025 at 12:01:33PM +0100, Peter Zijlstra wrote:
Could someone confirm this fixes the problem?
> ---
> Subject: sched: Fix the do_set_cpus_allowed() locking fix
>
> Commit abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
> overlooked that __balance_push_cpu_stop() calls select_fallback_rq()
> with rq->lock held. This makes that set_cpus_allowed_force() will
> recursively take rq->lock and the machine locks up.
>
> Run select_fallback_rq() earlier, without holding rq->lock. This opens
> up a race window where a task could get migrated out from under us, but
> that is harmless, we want the task migrated.
>
> select_fallback_rq() itself will not be subject to concurrency as it
> will be fully serialized by p->pi_lock, so there is no chance of
> set_cpus_allowed_force() getting called with different arguments and
> selecting different fallback CPUs for one task.
>
> Fixes: abfc01077df6 ("sched: Fix do_set_cpus_allowed() locking")
> Reported-by: Jan Polensky <japo@linux.ibm.com>
> Reported-by: kernel test robot <oliver.sang@intel.com>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Closes: https://lore.kernel.org/oe-lkp/202510271206.24495a68-lkp@intel.com
> ---
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 1842285eac1e..67b5f2faab36 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -8044,18 +8044,15 @@ static int __balance_push_cpu_stop(void *arg)
> struct rq_flags rf;
> int cpu;
>
> - raw_spin_lock_irq(&p->pi_lock);
> - rq_lock(rq, &rf);
> -
> - update_rq_clock(rq);
> -
> - if (task_rq(p) == rq && task_on_rq_queued(p)) {
> + scoped_guard (raw_spinlock_irq, &p->pi_lock) {
> cpu = select_fallback_rq(rq->cpu, p);
> - rq = __migrate_task(rq, &rf, p, cpu);
> - }
>
> - rq_unlock(rq, &rf);
> - raw_spin_unlock_irq(&p->pi_lock);
> + rq_lock(rq, &rf);
> + update_rq_clock(rq);
> + if (task_rq(p) == rq && task_on_rq_queued(p))
> + rq = __migrate_task(rq, &rf, p, cpu);
> + rq_unlock(rq, &rf);
> + }
>
> put_task_struct(p);
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread
2025-10-28 9:03 ` Peter Zijlstra
@ 2025-10-28 11:29 ` Jan Polensky
0 siblings, 0 replies; 4+ messages in thread
From: Jan Polensky @ 2025-10-28 11:29 UTC (permalink / raw)
To: Peter Zijlstra
Cc: oe-lkp, lkp, linux-kernel, x86, Juri Lelli, Tejun Heo,
Vincent Guittot, cgroups, aubrey.li, yu.c.chen
On Tue, Oct 28, 2025 at 10:03:24AM +0100, Peter Zijlstra wrote:
> On Mon, Oct 27, 2025 at 12:01:33PM +0100, Peter Zijlstra wrote:
>
> Could someone confirm this fixes the problem?
>
Tested-by: Jan Polensky <japo@linux.ibm.com>
Thank you for the quick fix. I’ve verified the patch on s390x and can confirm that it resolves the
spin lock issue.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-10-28 11:30 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-10-27 5:14 [tip:sched/core] [sched] b079d93796: WARNING:possible_recursive_locking_detected_migration_is_trying_to_acquire_lock:at:set_cpus_allowed_force_but_task_is_already_holding_lock:at:cpu_stopper_thread kernel test robot
2025-10-27 11:01 ` Peter Zijlstra
2025-10-28 9:03 ` Peter Zijlstra
2025-10-28 11:29 ` Jan Polensky
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).