[RFC 0/3] kexec: Force kexec to proceed under heavy deadline load

[RFC 0/3] kexec: Force kexec to proceed under heavy deadline load

[Pingfan Liu]

During discussion of the scheduler deadline bug [1], Pierre Gondois
pointed out a potential issue during kexec: as CPUs are unplugged, the
available DL bandwidth of the root domain gradually decreases. At some
point, insufficient bandwidth triggers an overflow detection, causing
CPU hot-removal to fail and kexec to hang.[2]
    
I reproduced it on a system with 160 cpus with the following command
  seq 10 | xargs -I{} -P10 sh -c 'chrt -d -T 1000000 -P 1000000 0 yes > /dev/null &'
  kexec -e

The system hang during the kexec process.
 
This series skips the DL bandwidth check, migrates the task from dying
CPU directly to the kexec CPU, and promotes the kexec to DL task. By
this way, the heavy deadline load will not starve the CPU hot-removal
kthread so that kexec task can move on.

In contrast to this series, an alternative aggressive approach is to
send SIGKILL to all DL tasks at the beginning of the kexec process.
Let us discuss how to resolve this issue.


[1]: https://lore.kernel.org/all/20250929133602.32462-1-piliu@redhat.com/
[2]: https://lore.kernel.org/all/3408aca5-e6c9-434a-9950-82e9147fcbba@arm.com/

Pingfan Liu (3):
  sched/deadline: Skip the deadline bandwidth check if kexec_in_progress
  kernel/cpu: Mark nonboot cpus as inactive when shutting down nonboot
    cpus
  kexec_core: Promote the kexec to DL task

 kernel/cpu.c            | 10 ++++++++++
 kernel/kexec_core.c     | 28 ++++++++++++++++++++++++++++
 kernel/sched/deadline.c |  7 +++++++
 3 files changed, 45 insertions(+)

-- 
2.49.0

[RFC 1/3] sched/deadline: Skip the deadline bandwidth check if kexec_in_progress

[Pingfan Liu]

During discussion of the scheduler deadline bug [1], Pierre Gondois
pointed out a potential issue during kexec: as CPUs are unplugged, the
available DL bandwidth of the root domain gradually decreases. At some
point, insufficient bandwidth triggers an overflow detection, causing
CPU hot-removal to fail and kexec to hang.

This can be reproduced by:
  chrt -d -T 1000000 -P 1000000 0 yes > /dev/null &
  kexec -e

This patch skips the DL bandwidth check if kexec is in progress.

[1]: https://lore.kernel.org/all/20250929133602.32462-1-piliu@redhat.com/

Reported-by: Pierre Gondois <pierre.gondois@arm.com>
Closes: https://lore.kernel.org/all/3408aca5-e6c9-434a-9950-82e9147fcbba@arm.com/
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Pierre Gondois <pierre.gondois@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Baoquan He <bhe@redhat.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Cc: Joel Granados <joel.granados@kernel.org>
To: kexec@lists.infradead.org
To: linux-kernel@vger.kernel.org
---
 kernel/kexec_core.c     | 6 ++++++
 kernel/sched/deadline.c | 7 +++++++
 2 files changed, 13 insertions(+)

diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index 31203f0bacafa..265de9d1ff5f5 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -1183,7 +1183,13 @@ int kernel_kexec(void)
 	} else
 #endif
 	{
+		/*
+		 * CPU hot-removal path refers to kexec_in_progress, it
+		 * requires a sync to ensure no in-flight hot-removing.
+		 */
+		cpu_hotplug_disable();
 		kexec_in_progress = true;
+		cpu_hotplug_enable();
 		kernel_restart_prepare("kexec reboot");
 		migrate_to_reboot_cpu();
 		syscore_shutdown();
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index a3a43baf4314e..cc864cc348b2c 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -18,6 +18,7 @@
 
 #include <linux/cpuset.h>
 #include <linux/sched/clock.h>
+#include <linux/kexec.h>
 #include <uapi/linux/sched/types.h>
 #include "sched.h"
 #include "pelt.h"
@@ -3502,6 +3503,12 @@ static int dl_bw_manage(enum dl_bw_request req, int cpu, u64 dl_bw)
 
 int dl_bw_deactivate(int cpu)
 {
+	/*
+	 * The system is shutting down and cannot roll back.  There is no point
+	 * in keeping track of bandwidth, which may fail hotplug.
+	 */
+	if (unlikely(kexec_in_progress))
+		return 0;
 	return dl_bw_manage(dl_bw_req_deactivate, cpu, 0);
 }
 
-- 
2.49.0

[RFC 2/3] kernel/cpu: Mark nonboot cpus as inactive when shutting down nonboot cpus

[Pingfan Liu]

The previous patch lifted the deadline bandwidth check during the kexec
process, which raises a potential issue: as the number of online CPUs
decreases, DL tasks may be crowded onto a few CPUs, which may starve the
CPU hotplug kthread. As a result, the hot-removal cannot proceed in
practice.  On the other hand, as CPUs are offlined one by one, all tasks
will eventually be migrated to the kexec CPU.

Therefore, this patch marks all other CPUs as inactive to signal the
scheduler to migrate tasks to the kexec CPU during hot-removal.

Signed-off-by: Pingfan Liu <piliu@redhat.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Pierre Gondois <pierre.gondois@arm.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: kexec@lists.infradead.org
To: linux-kernel@vger.kernel.org
---
 kernel/cpu.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/cpu.c b/kernel/cpu.c
index db9f6c539b28c..76aa0f784602b 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1546,6 +1546,16 @@ void smp_shutdown_nonboot_cpus(unsigned int primary_cpu)
 	if (!cpu_online(primary_cpu))
 		primary_cpu = cpumask_first(cpu_online_mask);
 
+	/*
+	 * Mark all other CPUs as inactive so the scheduler won't select them as
+	 * migration targets.
+	 */
+	for_each_online_cpu(cpu) {
+		if (cpu == primary_cpu)
+			continue;
+		set_cpu_active(cpu, false);
+	}
+
 	for_each_online_cpu(cpu) {
 		if (cpu == primary_cpu)
 			continue;
-- 
2.49.0

Re: [RFC 2/3] kernel/cpu: Mark nonboot cpus as inactive when shutting down nonboot cpus

[Thomas Gleixner]

On Wed, Oct 22 2025 at 20:13, Pingfan Liu wrote:
> The previous patch lifted the deadline bandwidth check during the kexec

Once this is applied 'The previous patch' is meaningless.

> process, which raises a potential issue: as the number of online CPUs
> decreases, DL tasks may be crowded onto a few CPUs, which may starve the
> CPU hotplug kthread. As a result, the hot-removal cannot proceed in
> practice.  On the other hand, as CPUs are offlined one by one, all tasks
> will eventually be migrated to the kexec CPU.
>
> Therefore, this patch marks all other CPUs as inactive to signal the

git grep "This patch" Documentation/process/

> scheduler to migrate tasks to the kexec CPU during hot-removal.

I'm not seeing what this solves. It just changes the timing of moving
tasks off to the boot CPU where they compete for the CPU for nothing.

When kexec() is in progress, then running user space tasks at all is a
completely pointless exercise.

So the obvious solution to the problem is to freeze all user space tasks
when kexec() is invoked. No horrible hacks in the deadline scheduler and
elsewhere required to make that work. No?

Thanks,

        tglx

Re: [RFC 2/3] kernel/cpu: Mark nonboot cpus as inactive when shutting down nonboot cpus

[Pingfan Liu]

On Mon, Oct 27, 2025 at 06:06:32PM +0100, Thomas Gleixner wrote:
> On Wed, Oct 22 2025 at 20:13, Pingfan Liu wrote:
> > The previous patch lifted the deadline bandwidth check during the kexec
> 
> Once this is applied 'The previous patch' is meaningless.
> 

I will rephrase it.
> > process, which raises a potential issue: as the number of online CPUs
> > decreases, DL tasks may be crowded onto a few CPUs, which may starve the
> > CPU hotplug kthread. As a result, the hot-removal cannot proceed in
> > practice.  On the other hand, as CPUs are offlined one by one, all tasks
> > will eventually be migrated to the kexec CPU.
> >
> > Therefore, this patch marks all other CPUs as inactive to signal the
> 
> git grep "This patch" Documentation/process/
> 

I will rephrase it.
> > scheduler to migrate tasks to the kexec CPU during hot-removal.
> 
> I'm not seeing what this solves. It just changes the timing of moving
> tasks off to the boot CPU where they compete for the CPU for nothing.
> 
> When kexec() is in progress, then running user space tasks at all is a
> completely pointless exercise.
> 
> So the obvious solution to the problem is to freeze all user space tasks

I agree, but what about a less intrusive approach? Simply stopping the 
DL tasks should suffice, as everything works correctly without them.

I have a draft patch ready. Let's discuss it and go from there.

> when kexec() is invoked. No horrible hacks in the deadline scheduler and
> elsewhere required to make that work. No?
> 

To clarify, skipping the dl_bw_deactivate() validation is necessary 
because it prevents CPU hot-removal.


Thanks,

Pingfan

Re: [RFC 2/3] kernel/cpu: Mark nonboot cpus as inactive when shutting down nonboot cpus

[Thomas Gleixner]

On Tue, Oct 28 2025 at 10:51, Pingfan Liu wrote:
> On Mon, Oct 27, 2025 at 06:06:32PM +0100, Thomas Gleixner wrote:
>> When kexec() is in progress, then running user space tasks at all is a
>> completely pointless exercise.
>> 
>> So the obvious solution to the problem is to freeze all user space tasks
>
> I agree, but what about a less intrusive approach? Simply stopping the 
> DL tasks should suffice, as everything works correctly without them.

What's intrusive about that? Task freezing exists already.

> I have a draft patch ready. Let's discuss it and go from there.
>
>> when kexec() is invoked. No horrible hacks in the deadline scheduler and
>> elsewhere required to make that work. No?
>
> To clarify, skipping the dl_bw_deactivate() validation is necessary 
> because it prevents CPU hot-removal.

If you freeze stuff there is nothing to do. Hibernation works exactly
that way without any magic hacks in a particular scheduling class, no?

Thanks,

        tglx

Re: [RFC 2/3] kernel/cpu: Mark nonboot cpus as inactive when shutting down nonboot cpus

[Pingfan Liu]

On Tue, Oct 28, 2025 at 01:59:11PM +0100, Thomas Gleixner wrote:
> On Tue, Oct 28 2025 at 10:51, Pingfan Liu wrote:
> > On Mon, Oct 27, 2025 at 06:06:32PM +0100, Thomas Gleixner wrote:
> >> When kexec() is in progress, then running user space tasks at all is a
> >> completely pointless exercise.
> >> 
> >> So the obvious solution to the problem is to freeze all user space tasks
> >
> > I agree, but what about a less intrusive approach? Simply stopping the 
> > DL tasks should suffice, as everything works correctly without them.
> 
> What's intrusive about that? Task freezing exists already.
> 

Thanks for your guidance. That's a good point -- system suspending is a
good analogy. I will check how PM handles it.

> > I have a draft patch ready. Let's discuss it and go from there.
> >
> >> when kexec() is invoked. No horrible hacks in the deadline scheduler and
> >> elsewhere required to make that work. No?
> >
> > To clarify, skipping the dl_bw_deactivate() validation is necessary 
> > because it prevents CPU hot-removal.
> 
> If you freeze stuff there is nothing to do. Hibernation works exactly
> that way without any magic hacks in a particular scheduling class, no?
> 

There is a nuance: DL bandwidth represents a commitment, not necessarily
the actual payload. Even a blocked DL task still occupies DL bandwidth.
The system's DL bandwidth remains unchanged as long as the CPUs stay
online, which is the case in hibernation.


Best Regards,

Pingfan

Re: [RFC 2/3] kernel/cpu: Mark nonboot cpus as inactive when shutting down nonboot cpus

[Thomas Gleixner]

On Wed, Oct 29 2025 at 19:36, Pingfan Liu wrote:
> On Tue, Oct 28, 2025 at 01:59:11PM +0100, Thomas Gleixner wrote:
>> If you freeze stuff there is nothing to do. Hibernation works exactly
>> that way without any magic hacks in a particular scheduling class, no?
>> 
>
> There is a nuance: DL bandwidth represents a commitment, not necessarily
> the actual payload. Even a blocked DL task still occupies DL bandwidth.
> The system's DL bandwidth remains unchanged as long as the CPUs stay
> online, which is the case in hibernation.

No. Hibernation brings the non-boot CPUs down in order to create the
disk image.

Thanks,

        tglx

Re: [RFC 2/3] kernel/cpu: Mark nonboot cpus as inactive when shutting down nonboot cpus

[Pingfan Liu]

On Wed, Oct 29, 2025 at 8:13 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> On Wed, Oct 29 2025 at 19:36, Pingfan Liu wrote:
> > On Tue, Oct 28, 2025 at 01:59:11PM +0100, Thomas Gleixner wrote:
> >> If you freeze stuff there is nothing to do. Hibernation works exactly
> >> that way without any magic hacks in a particular scheduling class, no?
> >>
> >
> > There is a nuance: DL bandwidth represents a commitment, not necessarily
> > the actual payload. Even a blocked DL task still occupies DL bandwidth.
> > The system's DL bandwidth remains unchanged as long as the CPUs stay
> > online, which is the case in hibernation.
>
> No. Hibernation brings the non-boot CPUs down in order to create the
> disk image.
>

Oh, I see. Since there are no DL tasks in the runqueue, no migration
occurs to activate the DL bandwidth. This approach, similar to PM, is
perfect for addressing this issue.


Thanks,

Pingfan

[RFC 3/3] kexec_core: Promote the kexec to DL task

[Pingfan Liu]

The previous patch lifted the deadline bandwidth check during the kexec
process. As a result, DL tasks may be crowded onto the kexec CPU, which
may starve the kexec task. At this point, the kexec task is the only
task needed to make the reboot proceed, hence promoting it to a deadline
task prevents this starvation.

Signed-off-by: Pingfan Liu <piliu@redhat.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Pierre Gondois <pierre.gondois@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Baoquan He <bhe@redhat.com>
To: kexec@lists.infradead.org
To: linux-kernel@vger.kernel.org
---
 kernel/kexec_core.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index 265de9d1ff5f5..0960bea1a8bab 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -41,6 +41,7 @@
 #include <linux/objtool.h>
 #include <linux/kmsg_dump.h>
 #include <linux/dma-map-ops.h>
+#include <uapi/linux/sched/types.h>
 
 #include <asm/page.h>
 #include <asm/sections.h>
@@ -1183,6 +1184,20 @@ int kernel_kexec(void)
 	} else
 #endif
 	{
+		struct sched_attr attr = {
+			.size		= sizeof(struct sched_attr),
+			.sched_policy	= SCHED_DEADLINE,
+			.sched_nice	= 0,
+			.sched_priority	= 0,
+			/*
+			 * Fake (unused) bandwidth; workaround to "fix"
+			 * priority inheritance.
+			 */
+			.sched_runtime	= NSEC_PER_MSEC,
+			.sched_deadline = 10 * NSEC_PER_MSEC,
+			.sched_period	= 10 * NSEC_PER_MSEC,
+		};
+
 		/*
 		 * CPU hot-removal path refers to kexec_in_progress, it
 		 * requires a sync to ensure no in-flight hot-removing.
@@ -1202,6 +1217,13 @@ int kernel_kexec(void)
 		 */
 		cpu_hotplug_enable();
 		pr_notice("Starting new kernel\n");
+		/*
+		 * During hot-removing cpu, all DL tasks will be migrated to
+		 * this cpu.  To prevent this task from starving, promoting it
+		 * to DL task. And soon, local interrupt will be disabled in
+		 * machine_kexec().
+		 */
+		sched_setattr_nocheck(current, &attr);
 		machine_shutdown();
 	}
 
-- 
2.49.0