* [PATCHv4 1/2] cpu/hotplug: Keep cpu hotplug disabled until the rebooting cpu is stable
[not found] <20220512030619.13426-1-kernelfans@gmail.com>
@ 2022-05-12 3:06 ` Pingfan Liu
2022-05-12 3:06 ` [PATCHv4 2/2] cpu/hotplug: Remove the input parameter primary_cpu of smp_shutdown_nonboot_cpus() Pingfan Liu
1 sibling, 0 replies; 3+ messages in thread
From: Pingfan Liu @ 2022-05-12 3:06 UTC (permalink / raw)
To: linuxppc-dev, linux-kernel
Cc: Pingfan Liu, Eric Biederman, Peter Zijlstra, Thomas Gleixner,
Vincent Donnefort, Ingo Molnar, Michael Ellerman, Mark Rutland,
YueHaibing, Baokun Li, Randy Dunlap, Valentin Schneider, kexec
smp_shutdown_nonboot_cpus() repeats the same code chunk as
migrate_to_reboot_cpu() to ensure that the rebooting happens on a valid
cpu.
if (!cpu_online(primary_cpu))
primary_cpu = cpumask_first(cpu_online_mask);
This is due to an unexpected cpu-down event like the following:
kernel_kexec()
migrate_to_reboot_cpu();
cpu_hotplug_enable();
-----------> comes a cpu_down(this_cpu) on other cpu
machine_shutdown();
smp_shutdown_nonboot_cpus(); which needs to re-check "if (!cpu_online(primary_cpu))"
Although the kexec-reboot task can get through a cpu_down() on its cpu,
this code looks a little confusing.
Tracing down the git history, the cpu_hotplug_enable() called by
kernel_kexec() is introduced by commit 011e4b02f1da ("powerpc, kexec:
Fix "Processor X is stuck" issue during kexec from ST mode"), which
wakes up all offline cpu by cpu_up(cpu). Later, it is required by the
architectures(arm/arm64/ia64/riscv) which resort to cpu hot-removing to
achieve kexec-reboot by
smp_shutdown_nonboot_cpus()->cpu_down_maps_locked().
Hence, the cpu_hotplug_enable() in kernel_kexec() is an architecture
requirement.
By deferring the cpu hotplug enable to a more proper point, where
smp_shutdown_nonboot_cpus() holds cpu_add_remove_lock, the
unexpected cpu-down event is squashed out and the rebooting cpu can keep
unchanged. (For powerpc, no gains from this change.)
As a result, the repeated code chunk can be removed and in [2/2], the
callsites of smp_shutdown_nonboot_cpus() can be consistent.
Signed-off-by: Pingfan Liu <kernelfans@gmail.com>
Cc: Eric Biederman <ebiederm@xmission.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Vincent Donnefort <vincent.donnefort@arm.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: YueHaibing <yuehaibing@huawei.com>
Cc: Baokun Li <libaokun1@huawei.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Valentin Schneider <valentin.schneider@arm.com>
Cc: kexec@lists.infradead.org
To: linuxppc-dev@lists.ozlabs.org
To: linux-kernel@vger.kernel.org
---
arch/powerpc/kexec/core_64.c | 1 +
kernel/cpu.c | 10 +++++-----
kernel/kexec_core.c | 11 +++++------
3 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/arch/powerpc/kexec/core_64.c b/arch/powerpc/kexec/core_64.c
index 6cc7793b8420..8ccf22197f08 100644
--- a/arch/powerpc/kexec/core_64.c
+++ b/arch/powerpc/kexec/core_64.c
@@ -224,6 +224,7 @@ static void wake_offline_cpus(void)
static void kexec_prepare_cpus(void)
{
+ cpu_hotplug_enable();
wake_offline_cpus();
smp_call_function(kexec_smp_down, NULL, /* wait */0);
local_irq_disable();
diff --git a/kernel/cpu.c b/kernel/cpu.c
index d0a9aa0b42e8..4415370f0e91 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1236,12 +1236,12 @@ void smp_shutdown_nonboot_cpus(unsigned int primary_cpu)
cpu_maps_update_begin();
/*
- * Make certain the cpu I'm about to reboot on is online.
- *
- * This is inline to what migrate_to_reboot_cpu() already do.
+ * At this point, the cpu hotplug is still disabled by
+ * migrate_to_reboot_cpu() to guarantee that the rebooting happens on
+ * the selected CPU. But cpu_down_maps_locked() returns -EBUSY, if
+ * cpu_hotplug_disabled. So re-enable CPU hotplug here.
*/
- if (!cpu_online(primary_cpu))
- primary_cpu = cpumask_first(cpu_online_mask);
+ __cpu_hotplug_enable();
for_each_online_cpu(cpu) {
if (cpu == primary_cpu)
diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index 68480f731192..1bd5a8c95a20 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -1168,14 +1168,13 @@ int kernel_kexec(void)
kexec_in_progress = true;
kernel_restart_prepare("kexec reboot");
migrate_to_reboot_cpu();
-
/*
- * migrate_to_reboot_cpu() disables CPU hotplug assuming that
- * no further code needs to use CPU hotplug (which is true in
- * the reboot case). However, the kexec path depends on using
- * CPU hotplug again; so re-enable it here.
+ * migrate_to_reboot_cpu() disables CPU hotplug and pin the
+ * rebooting thread on the selected CPU. If an architecture
+ * requires CPU hotplug to achieve kexec reboot, it should
+ * enable the hotplug in the architecture specific code
*/
- cpu_hotplug_enable();
+
pr_notice("Starting new kernel\n");
machine_shutdown();
}
--
2.31.1
^ permalink raw reply related [flat|nested] 3+ messages in thread* [PATCHv4 2/2] cpu/hotplug: Remove the input parameter primary_cpu of smp_shutdown_nonboot_cpus()
[not found] <20220512030619.13426-1-kernelfans@gmail.com>
2022-05-12 3:06 ` [PATCHv4 1/2] cpu/hotplug: Keep cpu hotplug disabled until the rebooting cpu is stable Pingfan Liu
@ 2022-05-12 3:06 ` Pingfan Liu
2022-05-15 10:03 ` Catalin Marinas
1 sibling, 1 reply; 3+ messages in thread
From: Pingfan Liu @ 2022-05-12 3:06 UTC (permalink / raw)
To: linux-ia64, linux-arm-kernel
Cc: Pingfan Liu, Thomas Gleixner, Geert Uytterhoeven, Peter Zijlstra,
Kees Cook, Jens Axboe, Russell King, Guenter Roeck, Arnd Bergmann,
Wolfram Sang, Lee Jones, Catalin Marinas, Will Deacon,
Mark Rutland, Peter Collingbourne, Marc Zyngier, linux-kernel
For all call sites of smp_shutdown_nonboot_cpus(),
$git grep smp_shutdown_nonboot_cpus -- arch | grep -v \*
arch/arm/kernel/reboot.c:94: smp_shutdown_nonboot_cpus(reboot_cpu);
arch/arm64/kernel/process.c:89: smp_shutdown_nonboot_cpus(reboot_cpu);
arch/ia64/kernel/process.c:578: smp_shutdown_nonboot_cpus(reboot_cpu);
arch/riscv/kernel/machine_kexec.c:135: smp_shutdown_nonboot_cpus(smp_processor_id());
As it shows, the input parameter seems a little arbitrary.
Since kernel_kexec()->migrate_to_reboot_cpu() has already pinned the
rebooting thread on the selected CPU and the CPU hotplug keeps disabled
before smp_shutdown_nonboot_cpus(). Let smp_shutdown_nonboot_cpus()
deduce the rebooting CPU by smp_processor_id(), instead of passing the
parameter primary_cpu to it.
As a result, all call sites look consistent.
Signed-off-by: Pingfan Liu <kernelfans@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Kees Cook <keescook@chromium.org>
Cc: Jens Axboe <axboe@kernel.dk>
To: linux-ia64@vger.kernel.org
Cc: Russell King <linux@armlinux.org.uk>
Cc: Guenter Roeck <linux@roeck-us.net>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Wolfram Sang <wsa+renesas@sang-engineering.com>
Cc: Lee Jones <lee.jones@linaro.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Collingbourne <pcc@google.com>
Cc: Marc Zyngier <maz@kernel.org>
To: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
---
arch/arm/kernel/reboot.c | 2 +-
arch/arm64/kernel/process.c | 2 +-
arch/ia64/kernel/process.c | 2 +-
arch/riscv/kernel/machine_kexec.c | 2 +-
include/linux/cpu.h | 4 ++--
kernel/cpu.c | 4 ++--
6 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/arm/kernel/reboot.c b/arch/arm/kernel/reboot.c
index 3044fcb8d073..98c961f0b377 100644
--- a/arch/arm/kernel/reboot.c
+++ b/arch/arm/kernel/reboot.c
@@ -91,7 +91,7 @@ void soft_restart(unsigned long addr)
*/
void machine_shutdown(void)
{
- smp_shutdown_nonboot_cpus(reboot_cpu);
+ smp_shutdown_nonboot_cpus();
}
/*
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 7fa97df55e3a..1f6498d44740 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -86,7 +86,7 @@ void arch_cpu_idle_dead(void)
*/
void machine_shutdown(void)
{
- smp_shutdown_nonboot_cpus(reboot_cpu);
+ smp_shutdown_nonboot_cpus();
}
/*
diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c
index d7a256bd9d6b..9af53ff43983 100644
--- a/arch/ia64/kernel/process.c
+++ b/arch/ia64/kernel/process.c
@@ -575,7 +575,7 @@ cpu_halt (void)
void machine_shutdown(void)
{
- smp_shutdown_nonboot_cpus(reboot_cpu);
+ smp_shutdown_nonboot_cpus();
#ifdef CONFIG_KEXEC
kexec_disable_iosapic();
diff --git a/arch/riscv/kernel/machine_kexec.c b/arch/riscv/kernel/machine_kexec.c
index cbef0fc73afa..a30c545714cc 100644
--- a/arch/riscv/kernel/machine_kexec.c
+++ b/arch/riscv/kernel/machine_kexec.c
@@ -132,7 +132,7 @@ void machine_shutdown(void)
local_irq_disable();
#if defined(CONFIG_HOTPLUG_CPU)
- smp_shutdown_nonboot_cpus(smp_processor_id());
+ smp_shutdown_nonboot_cpus();
#endif
}
diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index 54dc2f9a2d56..0506ba3ac85e 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -127,7 +127,7 @@ extern void cpu_hotplug_enable(void);
void clear_tasks_mm_cpumask(int cpu);
int remove_cpu(unsigned int cpu);
int cpu_device_down(struct device *dev);
-extern void smp_shutdown_nonboot_cpus(unsigned int primary_cpu);
+extern void smp_shutdown_nonboot_cpus(void);
#else /* CONFIG_HOTPLUG_CPU */
@@ -140,7 +140,7 @@ static inline void lockdep_assert_cpus_held(void) { }
static inline void cpu_hotplug_disable(void) { }
static inline void cpu_hotplug_enable(void) { }
static inline int remove_cpu(unsigned int cpu) { return -EPERM; }
-static inline void smp_shutdown_nonboot_cpus(unsigned int primary_cpu) { }
+static inline void smp_shutdown_nonboot_cpus(void) { }
#endif /* !CONFIG_HOTPLUG_CPU */
#ifdef CONFIG_PM_SLEEP_SMP
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 4415370f0e91..66b8af6b8a3e 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1228,7 +1228,7 @@ int remove_cpu(unsigned int cpu)
}
EXPORT_SYMBOL_GPL(remove_cpu);
-void smp_shutdown_nonboot_cpus(unsigned int primary_cpu)
+void smp_shutdown_nonboot_cpus(void)
{
unsigned int cpu;
int error;
@@ -1244,7 +1244,7 @@ void smp_shutdown_nonboot_cpus(unsigned int primary_cpu)
__cpu_hotplug_enable();
for_each_online_cpu(cpu) {
- if (cpu == primary_cpu)
+ if (cpu == smp_processor_id())
continue;
error = cpu_down_maps_locked(cpu, CPUHP_OFFLINE);
--
2.31.1
^ permalink raw reply related [flat|nested] 3+ messages in thread