[PATCHv2 0/2]: kexec: Force kexec to proceed under heavy deadline load

[PATCHv2 0/2]: kexec: Force kexec to proceed under heavy deadline load

[Pingfan Liu]

During discussion of the scheduler deadline bug [1], Pierre Gondois
pointed out a potential issue during kexec: as CPUs are unplugged, the
available DL bandwidth of the root domain gradually decreases. At some
point, insufficient bandwidth triggers an overflow detection, causing
CPU hot-removal to fail and kexec to hang.[2]
    
I reproduced it on a system with 160 cpus with the following command
  seq 10 | xargs -I{} -P10 sh -c 'chrt -d -T 1000000 -P 1000000 0 yes > /dev/null &'
  kexec -e

The system hang during the kexec process.
 
This series skips the DL bandwidth check, SIGSTOP all DL tasks so that
the kexec process can proceed.

[1]: https://lore.kernel.org/all/20250929133602.32462-1-piliu@redhat.com/
[2]: https://lore.kernel.org/all/3408aca5-e6c9-434a-9950-82e9147fcbba@arm.com/

RFC -> v2:
Instead of migrating the DL tasks, SIGSTOP them.

Pingfan Liu (2):
  sched/deadline: Skip the deadline bandwidth check if kexec_in_progress
  kernel/kexec: Stop all userspace deadline tasks

 kernel/kexec_core.c     | 23 +++++++++++++++++++++++
 kernel/sched/deadline.c |  7 +++++++
 2 files changed, 30 insertions(+)

-- 
2.49.0

[PATCHv2 1/2] sched/deadline: Skip the deadline bandwidth check if kexec_in_progress

[Pingfan Liu]

During discussion of the scheduler deadline bug [1], Pierre Gondois
pointed out a potential issue during kexec: as CPUs are unplugged, the
available DL bandwidth of the root domain gradually decreases. At some
point, insufficient bandwidth triggers an overflow detection, causing
CPU hot-removal to fail and kexec to hang [2].

This can be reproduced by:
  chrt -d -T 1000000 -P 1000000 0 yes > /dev/null &
  kexec -e

Meeting deadline bandwidth requirements is unnecessary during the kexec
process. Skip DL bandwidth validation to allow kexec to proceed smoothly.

[1]: https://lore.kernel.org/all/20250929133602.32462-1-piliu@redhat.com/
[2]: https://lore.kernel.org/all/3408aca5-e6c9-434a-9950-82e9147fcbba@arm.com/

Reported-by: Pierre Gondois <pierre.gondois@arm.com>
Closes: https://lore.kernel.org/all/3408aca5-e6c9-434a-9950-82e9147fcbba@arm.com/
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Pierre Gondois <pierre.gondois@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Baoquan He <bhe@redhat.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Cc: Joel Granados <joel.granados@kernel.org>
To: kexec@lists.infradead.org
To: linux-kernel@vger.kernel.org
---
 kernel/kexec_core.c     | 6 ++++++
 kernel/sched/deadline.c | 7 +++++++
 2 files changed, 13 insertions(+)

diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index 31203f0bacafa..265de9d1ff5f5 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -1183,7 +1183,13 @@ int kernel_kexec(void)
 	} else
 #endif
 	{
+		/*
+		 * CPU hot-removal path refers to kexec_in_progress, it
+		 * requires a sync to ensure no in-flight hot-removing.
+		 */
+		cpu_hotplug_disable();
 		kexec_in_progress = true;
+		cpu_hotplug_enable();
 		kernel_restart_prepare("kexec reboot");
 		migrate_to_reboot_cpu();
 		syscore_shutdown();
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 72c1f72463c75..9db6f26b6cc81 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -18,6 +18,7 @@
 
 #include <linux/cpuset.h>
 #include <linux/sched/clock.h>
+#include <linux/kexec.h>
 #include <uapi/linux/sched/types.h>
 #include "sched.h"
 #include "pelt.h"
@@ -3484,6 +3485,12 @@ static int dl_bw_manage(enum dl_bw_request req, int cpu, u64 dl_bw)
 
 int dl_bw_deactivate(int cpu)
 {
+	/*
+	 * The system is shutting down and cannot roll back.  There is no point
+	 * in keeping track of bandwidth, which may fail hotplug.
+	 */
+	if (unlikely(kexec_in_progress))
+		return 0;
 	return dl_bw_manage(dl_bw_req_deactivate, cpu, 0);
 }
 
-- 
2.49.0

[PATCHv2 2/2] kernel/kexec: Stop all userspace deadline tasks

[Pingfan Liu]

Now that the deadline bandwidth check has been skipped, there is nothing
to prevent CPUs from being unplugged. But as deadline tasks are crowded
onto the remaining CPUs, they may starve normal tasks, especially the
hotplug kthreads. As a result, the kexec process will hang indefinitely.

Send SIGSTOP to all userspace deadline tasks at the beginning of kexec
to allow other tasks to run as CPUs are unplugged.

Signed-off-by: Pingfan Liu <piliu@redhat.com>
---
 kernel/kexec_core.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index 265de9d1ff5f5..090bb58797d12 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -1132,6 +1132,18 @@ bool kexec_load_permitted(int kexec_image_type)
 	return true;
 }
 
+static void stop_user_deadline_tasks(void)
+{
+	struct task_struct *task;
+
+	rcu_read_lock();
+	for_each_process(task) {
+		if (task->policy == SCHED_DEADLINE && task->mm)
+			send_sig(SIGSTOP, task, 1);
+	}
+	rcu_read_unlock();
+}
+
 /*
  * Move into place and start executing a preloaded standalone
  * executable.  If nothing was preloaded return an error.
@@ -1190,6 +1202,11 @@ int kernel_kexec(void)
 		cpu_hotplug_disable();
 		kexec_in_progress = true;
 		cpu_hotplug_enable();
+		/*
+		 * As CPU hot-removal, the crowed deadline task may starve other
+		 * tasks. So stop them.
+		 */
+		stop_user_deadline_tasks();
 		kernel_restart_prepare("kexec reboot");
 		migrate_to_reboot_cpu();
 		syscore_shutdown();
-- 
2.49.0

Re: [PATCHv2 2/2] kernel/kexec: Stop all userspace deadline tasks

[Andrew Morton]

On Tue, 28 Oct 2025 11:09:14 +0800 Pingfan Liu <piliu@redhat.com> wrote:

> Now that the deadline bandwidth check has been skipped, there is nothing
> to prevent CPUs from being unplugged. But as deadline tasks are crowded
> onto the remaining CPUs, they may starve normal tasks, especially the
> hotplug kthreads. As a result, the kexec process will hang indefinitely.
> 
> Send SIGSTOP to all userspace deadline tasks at the beginning of kexec
> to allow other tasks to run as CPUs are unplugged.

This all looks a bit hacky.

What's special about kexec?  If many CPUs are being unplugged via other
means, won't the kernel still hit the very problems which are being
addressed here for kexec?  If so, we should seek a general fix for these
issues rather than a kexec-specific one?

> --- a/kernel/kexec_core.c
> +++ b/kernel/kexec_core.c
> @@ -1132,6 +1132,18 @@ bool kexec_load_permitted(int kexec_image_type)
>  	return true;
>  }
>  
> +static void stop_user_deadline_tasks(void)
> +{
> +	struct task_struct *task;
> +
> +	rcu_read_lock();
> +	for_each_process(task) {
> +		if (task->policy == SCHED_DEADLINE && task->mm)
> +			send_sig(SIGSTOP, task, 1);
> +	}
> +	rcu_read_unlock();
> +}

If we can safely stop all SCHED_DEADLINE user tasks then presumably we
can safely stop all user tasks.  Why not do that?

>  /*
>   * Move into place and start executing a preloaded standalone
>   * executable.  If nothing was preloaded return an error.
> @@ -1190,6 +1202,11 @@ int kernel_kexec(void)
>  		cpu_hotplug_disable();
>  		kexec_in_progress = true;
>  		cpu_hotplug_enable();
> +		/*
> +		 * As CPU hot-removal, the crowed deadline task may starve other
> +		 * tasks. So stop them.
> +		 */
> +		stop_user_deadline_tasks();
>  		kernel_restart_prepare("kexec reboot");
>  		migrate_to_reboot_cpu();
>  		syscore_shutdown();
y

Re: [PATCHv2 2/2] kernel/kexec: Stop all userspace deadline tasks

[Pingfan Liu]

On Tue, Oct 28, 2025 at 04:39:11PM -0700, Andrew Morton wrote:
> On Tue, 28 Oct 2025 11:09:14 +0800 Pingfan Liu <piliu@redhat.com> wrote:
> 
> > Now that the deadline bandwidth check has been skipped, there is nothing
> > to prevent CPUs from being unplugged. But as deadline tasks are crowded
> > onto the remaining CPUs, they may starve normal tasks, especially the
> > hotplug kthreads. As a result, the kexec process will hang indefinitely.
> > 
> > Send SIGSTOP to all userspace deadline tasks at the beginning of kexec
> > to allow other tasks to run as CPUs are unplugged.
> 
> This all looks a bit hacky.
> 
> What's special about kexec?  If many CPUs are being unplugged via other
> means, won't the kernel still hit the very problems which are being
> addressed here for kexec?  If so, we should seek a general fix for these

No, they are in different situations. For normal CPU hot-removal, the
deadline bandwidth validation will prevent the CPU hot-removal if the
bandwidth requirements are not met. But in the kexec case, there is no
way to roll back, so it must try its best to proceed. That is why the
previous patch is introduced.

> issues rather than a kexec-specific one?
> 
> > --- a/kernel/kexec_core.c
> > +++ b/kernel/kexec_core.c
> > @@ -1132,6 +1132,18 @@ bool kexec_load_permitted(int kexec_image_type)
> >  	return true;
> >  }
> >  
> > +static void stop_user_deadline_tasks(void)
> > +{
> > +	struct task_struct *task;
> > +
> > +	rcu_read_lock();
> > +	for_each_process(task) {
> > +		if (task->policy == SCHED_DEADLINE && task->mm)
> > +			send_sig(SIGSTOP, task, 1);
> > +	}
> > +	rcu_read_unlock();
> > +}
> 
> If we can safely stop all SCHED_DEADLINE user tasks then presumably we
> can safely stop all user tasks.  Why not do that?
> 

I think it can be done that way.  Normal user tasks cannot block the CPU
hotplug kthread. Once the other CPUs are offlined, kexec can jump to the
new kernel immediately. However, DL tasks may starve the hotplug kthread
and kexec task, which is why I take this cautious approach.


Thanks,

Pingfan