[PATCH] sched/core: Fix potential deadlock on rq lock

public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed

* [PATCH] sched/core: Fix potential deadlock on rq lock
@ 2025-09-11 12:42 Wang Tao
  2025-09-11 13:53 ` Peter Zijlstra
  0 siblings, 1 reply; 16+ messages in thread
From: Wang Tao @ 2025-09-11 12:42 UTC (permalink / raw)
  To: stable
  Cc: mingo, peterz, juri.lelli, vincent.guittot, dietmar.eggemann,
	rostedt, bsegall, mgorman, bristot, tglx, frederic, linux-kernel,
	tanghui20, zhangqiao22

When CPU 1 enters the nohz_full state, and the kworker on CPU 0 executes
the function sched_tick_remote, holding the lock on CPU1's rq
and triggering the warning WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3).
This leads to the process of printing the warning message, where the
console_sem semaphore is held. At this point, the print task on the
CPU1's rq cannot acquire the console_sem and joins the wait queue,
entering the UNINTERRUPTIBLE state. It waits for the console_sem to be
released and then wakes up. After the task on CPU 0 releases
the console_sem, it wakes up the waiting console_sem task.
In try_to_wake_up, it attempts to acquire the lock on CPU1's rq again,
resulting in a deadlock.

The triggering scenario is as follows:

CPU0								CPU1
sched_tick_remote
WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3)

report_bug							con_write
printk

console_unlock
								do_con_write
								console_lock
								down(&console_sem)
								list_add_tail(&waiter.list, &sem->wait_list);
up(&console_sem)
wake_up_q(&wake_q)
try_to_wake_up
__task_rq_lock
_raw_spin_lock

This patch fixes the issue by deffering all printk console printing
during the lock holding period.

Fixes: d84b31313ef8 ("sched/isolation: Offload residual 1Hz scheduler tick")
Signed-off-by: Wang Tao <wangtao554@huawei.com>
---
 kernel/sched/core.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index be00629f0ba4..8b2d5b5bfb93 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5723,8 +5723,10 @@ static void sched_tick_remote(struct work_struct *work)
 				 * Make sure the next tick runs within a
 				 * reasonable amount of time.
 				 */
+				printk_deferred_enter();
 				u64 delta = rq_clock_task(rq) - curr->se.exec_start;
 				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
+				printk_deferred_exit();
 			}
 			curr->sched_class->task_tick(rq, curr, 0);

-- 
2.34.1

^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched/core: Fix potential deadlock on rq lock
  2025-09-11 12:42 [PATCH] sched/core: Fix potential deadlock on rq lock Wang Tao
@ 2025-09-11 13:53 ` Peter Zijlstra
  2025-09-11 15:02   ` Frederic Weisbecker
  0 siblings, 1 reply; 16+ messages in thread
From: Peter Zijlstra @ 2025-09-11 13:53 UTC (permalink / raw)
  To: Wang Tao
  Cc: stable, mingo, juri.lelli, vincent.guittot, dietmar.eggemann,
	rostedt, bsegall, mgorman, bristot, tglx, frederic, linux-kernel,
	tanghui20, zhangqiao22

On Thu, Sep 11, 2025 at 12:42:49PM +0000, Wang Tao wrote:
> When CPU 1 enters the nohz_full state, and the kworker on CPU 0 executes
> the function sched_tick_remote, holding the lock on CPU1's rq
> and triggering the warning WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3).
> This leads to the process of printing the warning message, where the
> console_sem semaphore is held. At this point, the print task on the
> CPU1's rq cannot acquire the console_sem and joins the wait queue,
> entering the UNINTERRUPTIBLE state. It waits for the console_sem to be
> released and then wakes up. After the task on CPU 0 releases
> the console_sem, it wakes up the waiting console_sem task.
> In try_to_wake_up, it attempts to acquire the lock on CPU1's rq again,
> resulting in a deadlock.
> 
> The triggering scenario is as follows:
> 
> CPU0								CPU1
> sched_tick_remote
> WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3)
> 
> report_bug							con_write
> printk
> 
> console_unlock
> 								do_con_write
> 								console_lock
> 								down(&console_sem)
> 								list_add_tail(&waiter.list, &sem->wait_list);
> up(&console_sem)
> wake_up_q(&wake_q)
> try_to_wake_up
> __task_rq_lock
> _raw_spin_lock
> 
> This patch fixes the issue by deffering all printk console printing
> during the lock holding period.
> 
> Fixes: d84b31313ef8 ("sched/isolation: Offload residual 1Hz scheduler tick")
> Signed-off-by: Wang Tao <wangtao554@huawei.com>

I fundamentally hate that deferred thing and consider it a printk bug.

But really, if you trip that WARN, fix it and the problem goes away.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched/core: Fix potential deadlock on rq lock
  2025-09-11 13:53 ` Peter Zijlstra
@ 2025-09-11 15:02   ` Frederic Weisbecker
  2025-09-11 15:14     ` Phil Auld
  0 siblings, 1 reply; 16+ messages in thread
From: Frederic Weisbecker @ 2025-09-11 15:02 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Wang Tao, stable, mingo, juri.lelli, vincent.guittot,
	dietmar.eggemann, rostedt, bsegall, mgorman, bristot, tglx,
	linux-kernel, tanghui20, zhangqiao22

Le Thu, Sep 11, 2025 at 03:53:58PM +0200, Peter Zijlstra a écrit :
> On Thu, Sep 11, 2025 at 12:42:49PM +0000, Wang Tao wrote:
> > When CPU 1 enters the nohz_full state, and the kworker on CPU 0 executes
> > the function sched_tick_remote, holding the lock on CPU1's rq
> > and triggering the warning WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3).
> > This leads to the process of printing the warning message, where the
> > console_sem semaphore is held. At this point, the print task on the
> > CPU1's rq cannot acquire the console_sem and joins the wait queue,
> > entering the UNINTERRUPTIBLE state. It waits for the console_sem to be
> > released and then wakes up. After the task on CPU 0 releases
> > the console_sem, it wakes up the waiting console_sem task.
> > In try_to_wake_up, it attempts to acquire the lock on CPU1's rq again,
> > resulting in a deadlock.
> > 
> > The triggering scenario is as follows:
> > 
> > CPU0								CPU1
> > sched_tick_remote
> > WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3)
> > 
> > report_bug							con_write
> > printk
> > 
> > console_unlock
> > 								do_con_write
> > 								console_lock
> > 								down(&console_sem)
> > 								list_add_tail(&waiter.list, &sem->wait_list);
> > up(&console_sem)
> > wake_up_q(&wake_q)
> > try_to_wake_up
> > __task_rq_lock
> > _raw_spin_lock
> > 
> > This patch fixes the issue by deffering all printk console printing
> > during the lock holding period.
> > 
> > Fixes: d84b31313ef8 ("sched/isolation: Offload residual 1Hz scheduler tick")
> > Signed-off-by: Wang Tao <wangtao554@huawei.com>
> 
> I fundamentally hate that deferred thing and consider it a printk bug.
> 
> But really, if you trip that WARN, fix it and the problem goes away.

And probably it triggers a lot of false positives. An overloaded housekeeping
CPU can easily be off for 2 seconds. We should make it 30 seconds.

Thanks.

-- 
Frederic Weisbecker
SUSE Labs

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched/core: Fix potential deadlock on rq lock
  2025-09-11 15:02   ` Frederic Weisbecker
@ 2025-09-11 15:14     ` Phil Auld
  2025-09-11 15:38       ` Frederic Weisbecker
  0 siblings, 1 reply; 16+ messages in thread
From: Phil Auld @ 2025-09-11 15:14 UTC (permalink / raw)
  To: Frederic Weisbecker
  Cc: Peter Zijlstra, Wang Tao, stable, mingo, juri.lelli,
	vincent.guittot, dietmar.eggemann, rostedt, bsegall, mgorman,
	tglx, linux-kernel, tanghui20, zhangqiao22

On Thu, Sep 11, 2025 at 05:02:45PM +0200 Frederic Weisbecker wrote:
> Le Thu, Sep 11, 2025 at 03:53:58PM +0200, Peter Zijlstra a écrit :
> > On Thu, Sep 11, 2025 at 12:42:49PM +0000, Wang Tao wrote:
> > > When CPU 1 enters the nohz_full state, and the kworker on CPU 0 executes
> > > the function sched_tick_remote, holding the lock on CPU1's rq
> > > and triggering the warning WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3).
> > > This leads to the process of printing the warning message, where the
> > > console_sem semaphore is held. At this point, the print task on the
> > > CPU1's rq cannot acquire the console_sem and joins the wait queue,
> > > entering the UNINTERRUPTIBLE state. It waits for the console_sem to be
> > > released and then wakes up. After the task on CPU 0 releases
> > > the console_sem, it wakes up the waiting console_sem task.
> > > In try_to_wake_up, it attempts to acquire the lock on CPU1's rq again,
> > > resulting in a deadlock.
> > > 
> > > The triggering scenario is as follows:
> > > 
> > > CPU0								CPU1
> > > sched_tick_remote
> > > WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3)
> > > 
> > > report_bug							con_write
> > > printk
> > > 
> > > console_unlock
> > > 								do_con_write
> > > 								console_lock
> > > 								down(&console_sem)
> > > 								list_add_tail(&waiter.list, &sem->wait_list);
> > > up(&console_sem)
> > > wake_up_q(&wake_q)
> > > try_to_wake_up
> > > __task_rq_lock
> > > _raw_spin_lock
> > > 
> > > This patch fixes the issue by deffering all printk console printing
> > > during the lock holding period.
> > > 
> > > Fixes: d84b31313ef8 ("sched/isolation: Offload residual 1Hz scheduler tick")
> > > Signed-off-by: Wang Tao <wangtao554@huawei.com>
> > 
> > I fundamentally hate that deferred thing and consider it a printk bug.
> > 
> > But really, if you trip that WARN, fix it and the problem goes away.
> 
> And probably it triggers a lot of false positives. An overloaded housekeeping
> CPU can easily be off for 2 seconds. We should make it 30 seconds.
>

It does trigger pretty easily. We've done some work to try to make better
(spreading HK work around for example) but you can still hit it. Especially,
if there are virtualization layers involved...

Increasing that time a bit would be great :)

Cheers,
Phil


> Thanks.
> 
> -- 
> Frederic Weisbecker
> SUSE Labs
> 

-- 


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched/core: Fix potential deadlock on rq lock
  2025-09-11 15:14     ` Phil Auld
@ 2025-09-11 15:38       ` Frederic Weisbecker
  2025-09-11 16:13         ` [PATCH] sched: Increase sched_tick_remote timeout Phil Auld
  0 siblings, 1 reply; 16+ messages in thread
From: Frederic Weisbecker @ 2025-09-11 15:38 UTC (permalink / raw)
  To: Phil Auld
  Cc: Peter Zijlstra, Wang Tao, stable, mingo, juri.lelli,
	vincent.guittot, dietmar.eggemann, rostedt, bsegall, mgorman,
	tglx, linux-kernel, tanghui20, zhangqiao22

Le Thu, Sep 11, 2025 at 11:14:06AM -0400, Phil Auld a écrit :
> On Thu, Sep 11, 2025 at 05:02:45PM +0200 Frederic Weisbecker wrote:
> > Le Thu, Sep 11, 2025 at 03:53:58PM +0200, Peter Zijlstra a écrit :
> > > On Thu, Sep 11, 2025 at 12:42:49PM +0000, Wang Tao wrote:
> > > > When CPU 1 enters the nohz_full state, and the kworker on CPU 0 executes
> > > > the function sched_tick_remote, holding the lock on CPU1's rq
> > > > and triggering the warning WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3).
> > > > This leads to the process of printing the warning message, where the
> > > > console_sem semaphore is held. At this point, the print task on the
> > > > CPU1's rq cannot acquire the console_sem and joins the wait queue,
> > > > entering the UNINTERRUPTIBLE state. It waits for the console_sem to be
> > > > released and then wakes up. After the task on CPU 0 releases
> > > > the console_sem, it wakes up the waiting console_sem task.
> > > > In try_to_wake_up, it attempts to acquire the lock on CPU1's rq again,
> > > > resulting in a deadlock.
> > > > 
> > > > The triggering scenario is as follows:
> > > > 
> > > > CPU0								CPU1
> > > > sched_tick_remote
> > > > WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3)
> > > > 
> > > > report_bug							con_write
> > > > printk
> > > > 
> > > > console_unlock
> > > > 								do_con_write
> > > > 								console_lock
> > > > 								down(&console_sem)
> > > > 								list_add_tail(&waiter.list, &sem->wait_list);
> > > > up(&console_sem)
> > > > wake_up_q(&wake_q)
> > > > try_to_wake_up
> > > > __task_rq_lock
> > > > _raw_spin_lock
> > > > 
> > > > This patch fixes the issue by deffering all printk console printing
> > > > during the lock holding period.
> > > > 
> > > > Fixes: d84b31313ef8 ("sched/isolation: Offload residual 1Hz scheduler tick")
> > > > Signed-off-by: Wang Tao <wangtao554@huawei.com>
> > > 
> > > I fundamentally hate that deferred thing and consider it a printk bug.
> > > 
> > > But really, if you trip that WARN, fix it and the problem goes away.
> > 
> > And probably it triggers a lot of false positives. An overloaded housekeeping
> > CPU can easily be off for 2 seconds. We should make it 30 seconds.
> >
> 
> It does trigger pretty easily. We've done some work to try to make better
> (spreading HK work around for example) but you can still hit it. Especially,
> if there are virtualization layers involved...
> 
> Increasing that time a bit would be great :)

Interested in sending the patch? :-)

Thanks.

> 
> Cheers,
> Phil
> 
> 
> > Thanks.
> > 
> > -- 
> > Frederic Weisbecker
> > SUSE Labs
> > 
> 
> -- 
> 

-- 
Frederic Weisbecker
SUSE Labs

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [PATCH] sched: Increase sched_tick_remote timeout
  2025-09-11 15:38       ` Frederic Weisbecker
@ 2025-09-11 16:13         ` Phil Auld
  2025-09-11 16:29           ` Frederic Weisbecker
                             ` (4 more replies)
  0 siblings, 5 replies; 16+ messages in thread
From: Phil Auld @ 2025-09-11 16:13 UTC (permalink / raw)
  To: frederic
  Cc: bsegall, dietmar.eggemann, juri.lelli, linux-kernel, mgorman,
	mingo, pauld, peterz, rostedt, tanghui20, tglx, vincent.guittot,
	wangtao554, zhangqiao22, Waiman Long

Increase the sched_tick_remote WARN_ON timeout to remove false
positives due to temporarily busy HK cpus. The suggestion
was 30 seconds to catch really stuck remote tick processing
but not trigger it too easily.

Signed-off-by: Phil Auld <pauld@redhat.com>
Suggested-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Frederic Weisbecker <frederic@kernel.org>
---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index be00629f0ba4..ef90d358252d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5724,7 +5724,7 @@ static void sched_tick_remote(struct work_struct *work)
 				 * reasonable amount of time.
 				 */
 				u64 delta = rq_clock_task(rq) - curr->se.exec_start;
-				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
+				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 30);
 			}
 			curr->sched_class->task_tick(rq, curr, 0);
 
-- 
2.51.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched: Increase sched_tick_remote timeout
  2025-09-11 16:13         ` [PATCH] sched: Increase sched_tick_remote timeout Phil Auld
@ 2025-09-11 16:29           ` Frederic Weisbecker
  2025-09-17  6:26             ` wangtao (EQ)
  2025-09-16  8:44           ` wangtao (EQ)
                             ` (3 subsequent siblings)
  4 siblings, 1 reply; 16+ messages in thread
From: Frederic Weisbecker @ 2025-09-11 16:29 UTC (permalink / raw)
  To: Phil Auld
  Cc: bsegall, dietmar.eggemann, juri.lelli, linux-kernel, mgorman,
	mingo, peterz, rostedt, tanghui20, tglx, vincent.guittot,
	wangtao554, zhangqiao22, Waiman Long

Le Thu, Sep 11, 2025 at 12:13:00PM -0400, Phil Auld a écrit :
> Increase the sched_tick_remote WARN_ON timeout to remove false
> positives due to temporarily busy HK cpus. The suggestion
> was 30 seconds to catch really stuck remote tick processing
> but not trigger it too easily.
> 
> Signed-off-by: Phil Auld <pauld@redhat.com>
> Suggested-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Frederic Weisbecker <frederic@kernel.org>

Acked-by: Frederic Weisbecker <frederic@kernel.org>

-- 
Frederic Weisbecker
SUSE Labs

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched: Increase sched_tick_remote timeout
  2025-09-11 16:29           ` Frederic Weisbecker
@ 2025-09-17  6:26             ` wangtao (EQ)
  0 siblings, 0 replies; 16+ messages in thread
From: wangtao (EQ) @ 2025-09-17  6:26 UTC (permalink / raw)
  To: Frederic Weisbecker, Phil Auld
  Cc: bsegall, dietmar.eggemann, juri.lelli, linux-kernel, mgorman,
	mingo, peterz, rostedt, tanghui20, tglx, vincent.guittot,
	zhangqiao22, Waiman Long

Do we have plans to merge this patch into the mainline?

Thanks,

Tao

在 2025/9/12 0:29, Frederic Weisbecker 写道:
> Le Thu, Sep 11, 2025 at 12:13:00PM -0400, Phil Auld a écrit :
>> Increase the sched_tick_remote WARN_ON timeout to remove false
>> positives due to temporarily busy HK cpus. The suggestion
>> was 30 seconds to catch really stuck remote tick processing
>> but not trigger it too easily.
>>
>> Signed-off-by: Phil Auld <pauld@redhat.com>
>> Suggested-by: Frederic Weisbecker <frederic@kernel.org>
>> Cc: Peter Zijlstra <peterz@infradead.org>
>> Cc: Frederic Weisbecker <frederic@kernel.org>
> Acked-by: Frederic Weisbecker <frederic@kernel.org>
>

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched: Increase sched_tick_remote timeout
  2025-09-11 16:13         ` [PATCH] sched: Increase sched_tick_remote timeout Phil Auld
  2025-09-11 16:29           ` Frederic Weisbecker
@ 2025-09-16  8:44           ` wangtao (EQ)
  2025-09-16 12:49             ` Phil Auld
  2025-09-23 10:47           ` Phil Auld
                             ` (2 subsequent siblings)
  4 siblings, 1 reply; 16+ messages in thread
From: wangtao (EQ) @ 2025-09-16  8:44 UTC (permalink / raw)
  To: Phil Auld, frederic
  Cc: bsegall, dietmar.eggemann, juri.lelli, linux-kernel, mgorman,
	mingo, peterz, rostedt, tanghui20, tglx, vincent.guittot,
	zhangqiao22, Waiman Long

Increasing timeout alerts can reduce the probability of deadlocks. However, in the 'sched_tick_remote' method, there are 'WARN_ON_ONCE(rq->curr!= rq->donor)' and 'assert_clock_updated' in 'rq_clock_task'. Regardless of why these alerts are triggered, once they are triggered, 'printk' is called, which still leaves potential deadlock issues. Is there a better way to address these problems?

在 2025/9/12 0:13, Phil Auld 写道:
> Increase the sched_tick_remote WARN_ON timeout to remove false
> positives due to temporarily busy HK cpus. The suggestion
> was 30 seconds to catch really stuck remote tick processing
> but not trigger it too easily.
>
> Signed-off-by: Phil Auld <pauld@redhat.com>
> Suggested-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Frederic Weisbecker <frederic@kernel.org>
> ---
>   kernel/sched/core.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index be00629f0ba4..ef90d358252d 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -5724,7 +5724,7 @@ static void sched_tick_remote(struct work_struct *work)
>   				 * reasonable amount of time.
>   				 */
>   				u64 delta = rq_clock_task(rq) - curr->se.exec_start;
> -				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
> +				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 30);
>   			}
>   			curr->sched_class->task_tick(rq, curr, 0);
>   

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched: Increase sched_tick_remote timeout
  2025-09-16  8:44           ` wangtao (EQ)
@ 2025-09-16 12:49             ` Phil Auld
  0 siblings, 0 replies; 16+ messages in thread
From: Phil Auld @ 2025-09-16 12:49 UTC (permalink / raw)
  To: wangtao (EQ)
  Cc: frederic, bsegall, dietmar.eggemann, juri.lelli, linux-kernel,
	mgorman, mingo, peterz, rostedt, tanghui20, tglx, vincent.guittot,
	zhangqiao22, Waiman Long

On Tue, Sep 16, 2025 at 04:44:39PM +0800 wangtao (EQ) wrote:
> Increasing timeout alerts can reduce the probability of deadlocks. However, in the 'sched_tick_remote' method, there are 'WARN_ON_ONCE(rq->curr!= rq->donor)' and 'assert_clock_updated' in 'rq_clock_task'. Regardless of why these alerts are triggered, once they are triggered, 'printk' is called, which still leaves potential deadlock issues. Is there a better way to address these problems?
>

I'm not specically trying to solve the printk deadlock problem. My patch is
to make this particular warning go away by reducing the false positives.
That's tangential to your original posting. 

You can use the new printk mechanism with an atomic console to get around
the printk bug I think.

I think you could also use a serial console instead of a framebuffer based
console.


Cheers,
Phil



> 在 2025/9/12 0:13, Phil Auld 写道:
> > Increase the sched_tick_remote WARN_ON timeout to remove false
> > positives due to temporarily busy HK cpus. The suggestion
> > was 30 seconds to catch really stuck remote tick processing
> > but not trigger it too easily.
> > 
> > Signed-off-by: Phil Auld <pauld@redhat.com>
> > Suggested-by: Frederic Weisbecker <frederic@kernel.org>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Frederic Weisbecker <frederic@kernel.org>
> > ---
> >   kernel/sched/core.c | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index be00629f0ba4..ef90d358252d 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -5724,7 +5724,7 @@ static void sched_tick_remote(struct work_struct *work)
> >   				 * reasonable amount of time.
> >   				 */
> >   				u64 delta = rq_clock_task(rq) - curr->se.exec_start;
> > -				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
> > +				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 30);
> >   			}
> >   			curr->sched_class->task_tick(rq, curr, 0);
> 

-- 


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched: Increase sched_tick_remote timeout
  2025-09-11 16:13         ` [PATCH] sched: Increase sched_tick_remote timeout Phil Auld
  2025-09-11 16:29           ` Frederic Weisbecker
  2025-09-16  8:44           ` wangtao (EQ)
@ 2025-09-23 10:47           ` Phil Auld
  2025-10-10 12:13             ` Phil Auld
  2025-11-03 21:56             ` Phil Auld
  2025-11-14 12:19           ` [tip: sched/core] " tip-bot2 for Phil Auld
  2025-11-17 16:23           ` tip-bot2 for Phil Auld
  4 siblings, 2 replies; 16+ messages in thread
From: Phil Auld @ 2025-09-23 10:47 UTC (permalink / raw)
  To: frederic
  Cc: bsegall, dietmar.eggemann, juri.lelli, linux-kernel, mgorman,
	mingo, peterz, rostedt, tanghui20, tglx, vincent.guittot,
	wangtao554, zhangqiao22, Waiman Long

Hi,

On Thu, Sep 11, 2025 at 12:13:00PM -0400 Phil Auld wrote:
> Increase the sched_tick_remote WARN_ON timeout to remove false
> positives due to temporarily busy HK cpus. The suggestion
> was 30 seconds to catch really stuck remote tick processing
> but not trigger it too easily.
> 
> Signed-off-by: Phil Auld <pauld@redhat.com>
> Suggested-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Frederic Weisbecker <frederic@kernel.org>

Frederic ack'd this. Any other thoughts or opinions on this one
character patch?

Cheers,
Phil



> ---
>  kernel/sched/core.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index be00629f0ba4..ef90d358252d 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -5724,7 +5724,7 @@ static void sched_tick_remote(struct work_struct *work)
>  				 * reasonable amount of time.
>  				 */
>  				u64 delta = rq_clock_task(rq) - curr->se.exec_start;
> -				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
> +				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 30);
>  			}
>  			curr->sched_class->task_tick(rq, curr, 0);
>  
> -- 
> 2.51.0
> 

-- 


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched: Increase sched_tick_remote timeout
  2025-09-23 10:47           ` Phil Auld
@ 2025-10-10 12:13             ` Phil Auld
  2025-11-03 21:56             ` Phil Auld
  1 sibling, 0 replies; 16+ messages in thread
From: Phil Auld @ 2025-10-10 12:13 UTC (permalink / raw)
  To: frederic
  Cc: bsegall, dietmar.eggemann, juri.lelli, linux-kernel, mgorman,
	mingo, peterz, rostedt, tanghui20, tglx, vincent.guittot,
	wangtao554, zhangqiao22, Waiman Long

On Tue, Sep 23, 2025 at 06:47:39AM -0400 Phil Auld wrote:
> Hi,
> 
> On Thu, Sep 11, 2025 at 12:13:00PM -0400 Phil Auld wrote:
> > Increase the sched_tick_remote WARN_ON timeout to remove false
> > positives due to temporarily busy HK cpus. The suggestion
> > was 30 seconds to catch really stuck remote tick processing
> > but not trigger it too easily.
> > 
> > Signed-off-by: Phil Auld <pauld@redhat.com>
> > Suggested-by: Frederic Weisbecker <frederic@kernel.org>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Frederic Weisbecker <frederic@kernel.org>
> 
> Frederic ack'd this. Any other thoughts or opinions on this one
> character patch?

Ping...

> 
> Cheers,
> Phil
> 
> 
> 
> > ---
> >  kernel/sched/core.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index be00629f0ba4..ef90d358252d 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -5724,7 +5724,7 @@ static void sched_tick_remote(struct work_struct *work)
> >  				 * reasonable amount of time.
> >  				 */
> >  				u64 delta = rq_clock_task(rq) - curr->se.exec_start;
> > -				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
> > +				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 30);
> >  			}
> >  			curr->sched_class->task_tick(rq, curr, 0);
> >  
> > -- 
> > 2.51.0
> > 
> 
> -- 
> 
> 

-- 


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] sched: Increase sched_tick_remote timeout
  2025-09-23 10:47           ` Phil Auld
  2025-10-10 12:13             ` Phil Auld
@ 2025-11-03 21:56             ` Phil Auld
  1 sibling, 0 replies; 16+ messages in thread
From: Phil Auld @ 2025-11-03 21:56 UTC (permalink / raw)
  To: peterz
  Cc: bsegall, dietmar.eggemann, juri.lelli, linux-kernel, mgorman,
	mingo, frederic, rostedt, tanghui20, tglx, vincent.guittot,
	wangtao554, zhangqiao22, Waiman Long


Hi Peter,

On Tue, Sep 23, 2025 at 06:47:39AM -0400 Phil Auld wrote:
> Hi,
> 
> On Thu, Sep 11, 2025 at 12:13:00PM -0400 Phil Auld wrote:
> > Increase the sched_tick_remote WARN_ON timeout to remove false
> > positives due to temporarily busy HK cpus. The suggestion
> > was 30 seconds to catch really stuck remote tick processing
> > but not trigger it too easily.
> > 
> > Signed-off-by: Phil Auld <pauld@redhat.com>
> > Suggested-by: Frederic Weisbecker <frederic@kernel.org>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Frederic Weisbecker <frederic@kernel.org>
> 
> Frederic ack'd this. Any other thoughts or opinions on this one
> character patch?

Can we have this timeout increase, please? 


Thanks,
Phil

> 
> Cheers,
> Phil
> 
> 
> 
> > ---
> >  kernel/sched/core.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index be00629f0ba4..ef90d358252d 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -5724,7 +5724,7 @@ static void sched_tick_remote(struct work_struct *work)
> >  				 * reasonable amount of time.
> >  				 */
> >  				u64 delta = rq_clock_task(rq) - curr->se.exec_start;
> > -				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
> > +				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 30);
> >  			}
> >  			curr->sched_class->task_tick(rq, curr, 0);
> >  
> > -- 
> > 2.51.0
> > 
> 
> -- 
> 
> 

-- 


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [tip: sched/core] sched: Increase sched_tick_remote timeout
  2025-09-11 16:13         ` [PATCH] sched: Increase sched_tick_remote timeout Phil Auld
                             ` (2 preceding siblings ...)
  2025-09-23 10:47           ` Phil Auld
@ 2025-11-14 12:19           ` tip-bot2 for Phil Auld
  2025-11-14 13:07             ` Phil Auld
  2025-11-17 16:23           ` tip-bot2 for Phil Auld
  4 siblings, 1 reply; 16+ messages in thread
From: tip-bot2 for Phil Auld @ 2025-11-14 12:19 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: Frederic Weisbecker, Phil Auld, Peter Zijlstra (Intel), x86,
	linux-kernel

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     2616d12247639da40339757adc08c822147aa993
Gitweb:        https://git.kernel.org/tip/2616d12247639da40339757adc08c822147aa993
Author:        Phil Auld <pauld@redhat.com>
AuthorDate:    Thu, 11 Sep 2025 12:13:00 -04:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 14 Nov 2025 13:03:06 +01:00

sched: Increase sched_tick_remote timeout

Increase the sched_tick_remote WARN_ON timeout to remove false
positives due to temporarily busy HK cpus. The suggestion
was 30 seconds to catch really stuck remote tick processing
but not trigger it too easily.

Suggested-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Phil Auld <pauld@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Frederic Weisbecker <frederic@kernel.org>
Link: https://patch.msgid.link/20250911161300.437944-1-pauld@redhat.com
---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 68f19aa..699db3f 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5619,7 +5619,7 @@ static void sched_tick_remote(struct work_struct *work)
 				 * reasonable amount of time.
 				 */
 				u64 delta = rq_clock_task(rq) - curr->se.exec_start;
-				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
+				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 30);
 			}
 			curr->sched_class->task_tick(rq, curr, 0);
 

^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [tip: sched/core] sched: Increase sched_tick_remote timeout
  2025-11-14 12:19           ` [tip: sched/core] " tip-bot2 for Phil Auld
@ 2025-11-14 13:07             ` Phil Auld
  0 siblings, 0 replies; 16+ messages in thread
From: Phil Auld @ 2025-11-14 13:07 UTC (permalink / raw)
  To: linux-kernel
  Cc: linux-tip-commits, Frederic Weisbecker, Peter Zijlstra (Intel),
	x86

On Fri, Nov 14, 2025 at 12:19:06PM -0000 tip-bot2 for Phil Auld wrote:
> The following commit has been merged into the sched/core branch of tip:
> 
> Commit-ID:     2616d12247639da40339757adc08c822147aa993
> Gitweb:        https://git.kernel.org/tip/2616d12247639da40339757adc08c822147aa993
> Author:        Phil Auld <pauld@redhat.com>
> AuthorDate:    Thu, 11 Sep 2025 12:13:00 -04:00
> Committer:     Peter Zijlstra <peterz@infradead.org>
> CommitterDate: Fri, 14 Nov 2025 13:03:06 +01:00
>

Thanks Peter!  


> sched: Increase sched_tick_remote timeout
> 
> Increase the sched_tick_remote WARN_ON timeout to remove false
> positives due to temporarily busy HK cpus. The suggestion
> was 30 seconds to catch really stuck remote tick processing
> but not trigger it too easily.
> 
> Suggested-by: Frederic Weisbecker <frederic@kernel.org>
> Signed-off-by: Phil Auld <pauld@redhat.com>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Acked-by: Frederic Weisbecker <frederic@kernel.org>
> Link: https://patch.msgid.link/20250911161300.437944-1-pauld@redhat.com
> ---
>  kernel/sched/core.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 68f19aa..699db3f 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -5619,7 +5619,7 @@ static void sched_tick_remote(struct work_struct *work)
>  				 * reasonable amount of time.
>  				 */
>  				u64 delta = rq_clock_task(rq) - curr->se.exec_start;
> -				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
> +				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 30);
>  			}
>  			curr->sched_class->task_tick(rq, curr, 0);
>  
> 

-- 


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [tip: sched/core] sched: Increase sched_tick_remote timeout
  2025-09-11 16:13         ` [PATCH] sched: Increase sched_tick_remote timeout Phil Auld
                             ` (3 preceding siblings ...)
  2025-11-14 12:19           ` [tip: sched/core] " tip-bot2 for Phil Auld
@ 2025-11-17 16:23           ` tip-bot2 for Phil Auld
  4 siblings, 0 replies; 16+ messages in thread
From: tip-bot2 for Phil Auld @ 2025-11-17 16:23 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: Frederic Weisbecker, Phil Auld, Peter Zijlstra (Intel), x86,
	linux-kernel

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     aaab6bb54ab9bc4c37ff33b816031918d2760517
Gitweb:        https://git.kernel.org/tip/aaab6bb54ab9bc4c37ff33b816031918d2760517
Author:        Phil Auld <pauld@redhat.com>
AuthorDate:    Thu, 11 Sep 2025 12:13:00 -04:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Mon, 17 Nov 2025 17:13:15 +01:00

sched: Increase sched_tick_remote timeout

Increase the sched_tick_remote WARN_ON timeout to remove false
positives due to temporarily busy HK cpus. The suggestion
was 30 seconds to catch really stuck remote tick processing
but not trigger it too easily.

Suggested-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Phil Auld <pauld@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Frederic Weisbecker <frederic@kernel.org>
Link: https://patch.msgid.link/20250911161300.437944-1-pauld@redhat.com
---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 68f19aa..699db3f 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5619,7 +5619,7 @@ static void sched_tick_remote(struct work_struct *work)
 				 * reasonable amount of time.
 				 */
 				u64 delta = rq_clock_task(rq) - curr->se.exec_start;
-				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
+				WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 30);
 			}
 			curr->sched_class->task_tick(rq, curr, 0);
 

^ permalink raw reply related	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2025-11-17 16:23 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-09-11 12:42 [PATCH] sched/core: Fix potential deadlock on rq lock Wang Tao
2025-09-11 13:53 ` Peter Zijlstra
2025-09-11 15:02   ` Frederic Weisbecker
2025-09-11 15:14     ` Phil Auld
2025-09-11 15:38       ` Frederic Weisbecker
2025-09-11 16:13         ` [PATCH] sched: Increase sched_tick_remote timeout Phil Auld
2025-09-11 16:29           ` Frederic Weisbecker
2025-09-17  6:26             ` wangtao (EQ)
2025-09-16  8:44           ` wangtao (EQ)
2025-09-16 12:49             ` Phil Auld
2025-09-23 10:47           ` Phil Auld
2025-10-10 12:13             ` Phil Auld
2025-11-03 21:56             ` Phil Auld
2025-11-14 12:19           ` [tip: sched/core] " tip-bot2 for Phil Auld
2025-11-14 13:07             ` Phil Auld
2025-11-17 16:23           ` tip-bot2 for Phil Auld

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox