* Re: [PATCH v2] pseries/kexec: skip resetting CPUs added by firmware but not started by the kernel
2026-03-30 6:22 [PATCH v2] pseries/kexec: skip resetting CPUs added by firmware but not started by the kernel Shivang Upadhyay
@ 2026-03-30 8:49 ` Shrikanth Hegde
2026-03-31 2:59 ` Shivang Upadhyay
2026-03-31 4:33 ` Srikar Dronamraju
2026-04-06 8:52 ` Vishal Chourasia
2 siblings, 1 reply; 8+ messages in thread
From: Shrikanth Hegde @ 2026-03-30 8:49 UTC (permalink / raw)
To: Shivang Upadhyay, linuxppc-dev, linux-kernel
Cc: Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy, Srikar Dronamraju, Nysal Jan K.A.,
Vishal Chourasia, Ritesh Harjani, Sourabh Jain, Anushree Mathur
Hi Shivang.
Few nitpicks.
On 3/30/26 11:52 AM, Shivang Upadhyay wrote:
> During DLPAR operations, the newly added CPUs start in halted mode.
> The kernel then takes some time to initialize those CPUs internally and
> start them using the "start-cpu" RTAS call. However, if a kexec crash
> occurs in this window (before the new CPU has been initialized),
> the kexec NMI will try to reset all other CPUs from the crashing CPU.
> This leads to firmware starting the uninitialized CPUs as well.
>
> This can cause the kdump kernel to hang during bring-up.
>
> Sample Log:
> [175993.028231][ T1502] NIP [00007fffb953f394] 0x7fffb953f394
> [175993.028314][ T1502] LR [00007fffb953f394] 0x7fffb953f394
> [175993.028390][ T1502] --- interrupt: 3000
> [ 5.519483][ T1] Processor 0 is stuck.
> [ 11.089481][ T1] Processor 1 is stuck.
>
> To fix this, only issue the system-reset hcall to CPUs that have
> actually been started by the kernel.
>
> Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Nicholas Piggin <npiggin@gmail.com>
> Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
> Cc: Srikar Dronamraju <srikar@linux.ibm.com>
> Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
> Cc: Nysal Jan K.A. <nysal@linux.ibm.com>
> Cc: Vishal Chourasia <vishalc@linux.ibm.com>
> Cc: Ritesh Harjani <ritesh.list@gmail.com>
> Cc: Sourabh Jain <sourabhjain@linux.ibm.com>
> Reported-by: Anushree Mathur <anushree.mathur@linux.vnet.ibm.com>
> Signed-off-by: Shivang Upadhyay <shivangu@linux.ibm.com>
> ---
> Changelog:
>
> V2:
> * added set_crash_nmi_ipi to saperate crash's case from other nmi_ipi
> users
>
> V1:
> * https://lore.kernel.org/all/20251205142825.44698-1-shivangu@linux.ibm.com/
> ---
> arch/powerpc/include/asm/smp.h | 1 +
> arch/powerpc/kernel/smp.c | 1 +
> arch/powerpc/platforms/pseries/smp.c | 29 +++++++++++++++++++++++++++-
> 3 files changed, 30 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h
> index e41b9ea42122..cb74201f5674 100644
> --- a/arch/powerpc/include/asm/smp.h
> +++ b/arch/powerpc/include/asm/smp.h
> @@ -47,6 +47,7 @@ struct smp_ops_t {
> void (*cause_ipi)(int cpu);
> #endif
> int (*cause_nmi_ipi)(int cpu);
> + void (*set_crash_nmi_ipi)(void);
> void (*probe)(void);
> int (*kick_cpu)(int nr);
> int (*prepare_cpu)(int nr);
> diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> index 3467f86fd78f..3390ee8adf79 100644
> --- a/arch/powerpc/kernel/smp.c
> +++ b/arch/powerpc/kernel/smp.c
> @@ -594,6 +594,7 @@ void crash_send_ipi(void (*crash_ipi_callback)(struct pt_regs *))
> {
> int cpu;
>
> + smp_ops->set_crash_nmi_ipi();
> smp_send_nmi_ipi(NMI_IPI_ALL_OTHERS, crash_ipi_callback, 1000000);
> if (kdump_in_progress() && crash_wake_offline) {
> for_each_present_cpu(cpu) {
> diff --git a/arch/powerpc/platforms/pseries/smp.c b/arch/powerpc/platforms/pseries/smp.c
> index db99725e752b..c6c2baacca9a 100644
> --- a/arch/powerpc/platforms/pseries/smp.c
> +++ b/arch/powerpc/platforms/pseries/smp.c
> @@ -51,6 +51,9 @@
> */
> static cpumask_var_t of_spin_mask;
>
> +
> +static int crash_nmi_ipi;
> +
> /* Query where a cpu is now. Return codes #defined in plpar_wrappers.h */
> int smp_query_cpu_stopped(unsigned int pcpu)
> {
> @@ -171,12 +174,35 @@ static void dbell_or_ic_cause_ipi(int cpu)
> ic_cause_ipi(cpu);
> }
>
> +static void pseries_set_crash_nmi_ipi(void)
> +{
> + crash_nmi_ipi = 1;
> +}
> +
> static int pseries_cause_nmi_ipi(int cpu)
> {
> int hwcpu;
> + int k, curcpu;
Please follow inverted christmas tree for variables.
Can the below block be re-written with less indentations?
One level indentation removal could be.
if (cpu == NMI_IPI_ALL_OTHERS && crash_nmi_ipi)
hwcpu = <>
else if (cpu == NMI_IPI_ALL_OTHERS)
hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
else
<existing remaining logic>
>
> + curcpu = smp_processor_id();
> if (cpu == NMI_IPI_ALL_OTHERS) {
> - hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
> + if (crash_nmi_ipi) {
> + for_each_present_cpu(k) {
> + if (k != curcpu) {
Maybe below to reduce one more indentation level.
if ( k == curcpu)
continue;
> + hwcpu = get_hard_smp_processor_id(k);
> +
> + /* it is possible that cpu is present,
> + * but not started yet.
> + */
> +
> + if (paca_ptrs[hwcpu]->cpu_start == 1) {
It either 1 or 0. So if(paca_ptrs[hwcpu]->cpu_start) is enough.
> + plpar_signal_sys_reset(hwcpu);
> + }
> + }
> + }
> + return 1;
> + } else
> + hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
> } else {
> if (cpu < 0) {
> WARN_ONCE(true, "incorrect cpu parameter %d", cpu);
> @@ -243,6 +269,7 @@ static struct smp_ops_t pseries_smp_ops = {
> .message_pass = NULL, /* Use smp_muxed_ipi_message_pass */
> .cause_ipi = NULL, /* Filled at runtime by pSeries_smp_probe() */
> .cause_nmi_ipi = pseries_cause_nmi_ipi,
> + .set_crash_nmi_ipi = pseries_set_crash_nmi_ipi,
> .probe = pSeries_smp_probe,
> .prepare_cpu = pseries_smp_prepare_cpu,
> .kick_cpu = smp_pSeries_kick_cpu,
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH v2] pseries/kexec: skip resetting CPUs added by firmware but not started by the kernel
2026-03-30 8:49 ` Shrikanth Hegde
@ 2026-03-31 2:59 ` Shivang Upadhyay
0 siblings, 0 replies; 8+ messages in thread
From: Shivang Upadhyay @ 2026-03-31 2:59 UTC (permalink / raw)
To: Shrikanth Hegde, linuxppc-dev, linux-kernel
Cc: Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy, Srikar Dronamraju, Nysal Jan K.A.,
Vishal Chourasia, Ritesh Harjani, Sourabh Jain, Anushree Mathur
Hi Shrikanth, Thanks for your review.
On Mon, 2026-03-30 at 14:19 +0530, Shrikanth Hegde wrote:
> Hi Shivang.
>
> Few nitpicks.
>
> On 3/30/26 11:52 AM, Shivang Upadhyay wrote:
> > During DLPAR operations, the newly added CPUs start in halted mode.
> > The kernel then takes some time to initialize those CPUs internally
> > and
> > start them using the "start-cpu" RTAS call. However, if a kexec
> > crash
> > occurs in this window (before the new CPU has been initialized),
> > the kexec NMI will try to reset all other CPUs from the crashing
> > CPU.
> > This leads to firmware starting the uninitialized CPUs as well.
> >
> > This can cause the kdump kernel to hang during bring-up.
> >
> > Sample Log:
> > [175993.028231][ T1502] NIP [00007fffb953f394] 0x7fffb953f394
> > [175993.028314][ T1502] LR [00007fffb953f394] 0x7fffb953f394
> > [175993.028390][ T1502] --- interrupt: 3000
> > [ 5.519483][ T1] Processor 0 is stuck.
> > [ 11.089481][ T1] Processor 1 is stuck.
> >
> > To fix this, only issue the system-reset hcall to CPUs that have
> > actually been started by the kernel.
> >
> > Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
> > Cc: Michael Ellerman <mpe@ellerman.id.au>
> > Cc: Nicholas Piggin <npiggin@gmail.com>
> > Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
> > Cc: Srikar Dronamraju <srikar@linux.ibm.com>
> > Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
> > Cc: Nysal Jan K.A. <nysal@linux.ibm.com>
> > Cc: Vishal Chourasia <vishalc@linux.ibm.com>
> > Cc: Ritesh Harjani <ritesh.list@gmail.com>
> > Cc: Sourabh Jain <sourabhjain@linux.ibm.com>
> > Reported-by: Anushree Mathur <anushree.mathur@linux.vnet.ibm.com>
> > Signed-off-by: Shivang Upadhyay <shivangu@linux.ibm.com>
> > ---
> > Changelog:
> >
> > V2:
> > * added set_crash_nmi_ipi to saperate crash's case from other
> > nmi_ipi
> > users
> >
> > V1:
> > *
> > https://lore.kernel.org/all/20251205142825.44698-1-shivangu@linux.ibm.com/
> > ---
> > arch/powerpc/include/asm/smp.h | 1 +
> > arch/powerpc/kernel/smp.c | 1 +
> > arch/powerpc/platforms/pseries/smp.c | 29
> > +++++++++++++++++++++++++++-
> > 3 files changed, 30 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/powerpc/include/asm/smp.h
> > b/arch/powerpc/include/asm/smp.h
> > index e41b9ea42122..cb74201f5674 100644
> > --- a/arch/powerpc/include/asm/smp.h
> > +++ b/arch/powerpc/include/asm/smp.h
> > @@ -47,6 +47,7 @@ struct smp_ops_t {
> > void (*cause_ipi)(int cpu);
> > #endif
> > int (*cause_nmi_ipi)(int cpu);
> > + void (*set_crash_nmi_ipi)(void);
> > void (*probe)(void);
> > int (*kick_cpu)(int nr);
> > int (*prepare_cpu)(int nr);
> > diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> > index 3467f86fd78f..3390ee8adf79 100644
> > --- a/arch/powerpc/kernel/smp.c
> > +++ b/arch/powerpc/kernel/smp.c
> > @@ -594,6 +594,7 @@ void crash_send_ipi(void
> > (*crash_ipi_callback)(struct pt_regs *))
> > {
> > int cpu;
> >
> > + smp_ops->set_crash_nmi_ipi();
> > smp_send_nmi_ipi(NMI_IPI_ALL_OTHERS, crash_ipi_callback,
> > 1000000);
> > if (kdump_in_progress() && crash_wake_offline) {
> > for_each_present_cpu(cpu) {
> > diff --git a/arch/powerpc/platforms/pseries/smp.c
> > b/arch/powerpc/platforms/pseries/smp.c
> > index db99725e752b..c6c2baacca9a 100644
> > --- a/arch/powerpc/platforms/pseries/smp.c
> > +++ b/arch/powerpc/platforms/pseries/smp.c
> > @@ -51,6 +51,9 @@
> > */
> > static cpumask_var_t of_spin_mask;
> >
> > +
> > +static int crash_nmi_ipi;
> > +
> > /* Query where a cpu is now. Return codes #defined in
> > plpar_wrappers.h */
> > int smp_query_cpu_stopped(unsigned int pcpu)
> > {
> > @@ -171,12 +174,35 @@ static void dbell_or_ic_cause_ipi(int cpu)
> > ic_cause_ipi(cpu);
> > }
> >
> > +static void pseries_set_crash_nmi_ipi(void)
> > +{
> > + crash_nmi_ipi = 1;
> > +}
> > +
> > static int pseries_cause_nmi_ipi(int cpu)
> > {
> > int hwcpu;
> > + int k, curcpu;
>
> Please follow inverted christmas tree for variables.
Ack.
>
>
> Can the below block be re-written with less indentations?
> One level indentation removal could be.
>
> if (cpu == NMI_IPI_ALL_OTHERS && crash_nmi_ipi)
> hwcpu = <>
> else if (cpu == NMI_IPI_ALL_OTHERS)
> hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
> else
> <existing remaining logic>
>
> >
> > + curcpu = smp_processor_id();
> > if (cpu == NMI_IPI_ALL_OTHERS) {
> > - hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
> > + if (crash_nmi_ipi) {
> > + for_each_present_cpu(k) {
> > + if (k != curcpu) {
>
> Maybe below to reduce one more indentation level.
>
> if ( k == curcpu)
> continue;
>
Sure, i can rewrite it that way.
>
> > + hwcpu =
> > get_hard_smp_processor_id(k);
> > +
> > + /* it is possible that cpu
> > is present,
> > + * but not started yet.
> > + */
> > +
> > + if (paca_ptrs[hwcpu]-
> > >cpu_start == 1) {
>
> It either 1 or 0. So if(paca_ptrs[hwcpu]->cpu_start) is enough.
Ack.
~Shivang.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] pseries/kexec: skip resetting CPUs added by firmware but not started by the kernel
2026-03-30 6:22 [PATCH v2] pseries/kexec: skip resetting CPUs added by firmware but not started by the kernel Shivang Upadhyay
2026-03-30 8:49 ` Shrikanth Hegde
@ 2026-03-31 4:33 ` Srikar Dronamraju
2026-03-31 6:44 ` Shivang Upadhyay
2026-04-06 8:52 ` Vishal Chourasia
2 siblings, 1 reply; 8+ messages in thread
From: Srikar Dronamraju @ 2026-03-31 4:33 UTC (permalink / raw)
To: Shivang Upadhyay
Cc: linuxppc-dev, linux-kernel, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy, Shrikanth Hegde,
Nysal Jan K.A., Vishal Chourasia, Ritesh Harjani, Sourabh Jain,
Anushree Mathur
* Shivang Upadhyay <shivangu@linux.ibm.com> [2026-03-30 11:52:06]:
> During DLPAR operations, the newly added CPUs start in halted mode.
> The kernel then takes some time to initialize those CPUs internally and
> start them using the "start-cpu" RTAS call. However, if a kexec crash
> occurs in this window (before the new CPU has been initialized),
> the kexec NMI will try to reset all other CPUs from the crashing CPU.
> This leads to firmware starting the uninitialized CPUs as well.
What would happen if an non kexec crash nmi was delivered when we did a
DLPAR operation and before the CPUs are initialized?
>
> This can cause the kdump kernel to hang during bring-up.
>
> Sample Log:
> [175993.028231][ T1502] NIP [00007fffb953f394] 0x7fffb953f394
> [175993.028314][ T1502] LR [00007fffb953f394] 0x7fffb953f394
> [175993.028390][ T1502] --- interrupt: 3000
> [ 5.519483][ T1] Processor 0 is stuck.
> [ 11.089481][ T1] Processor 1 is stuck.
>
> To fix this, only issue the system-reset hcall to CPUs that have
> actually been started by the kernel.
>
> Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Nicholas Piggin <npiggin@gmail.com>
> Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
> Cc: Srikar Dronamraju <srikar@linux.ibm.com>
> Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
> Cc: Nysal Jan K.A. <nysal@linux.ibm.com>
> Cc: Vishal Chourasia <vishalc@linux.ibm.com>
> Cc: Ritesh Harjani <ritesh.list@gmail.com>
> Cc: Sourabh Jain <sourabhjain@linux.ibm.com>
> Reported-by: Anushree Mathur <anushree.mathur@linux.vnet.ibm.com>
> Signed-off-by: Shivang Upadhyay <shivangu@linux.ibm.com>
> ---
> Changelog:
>
> V2:
> * added set_crash_nmi_ipi to saperate crash's case from other nmi_ipi
> users
>
> V1:
> * https://lore.kernel.org/all/20251205142825.44698-1-shivangu@linux.ibm.com/
> ---
> arch/powerpc/include/asm/smp.h | 1 +
> arch/powerpc/kernel/smp.c | 1 +
> arch/powerpc/platforms/pseries/smp.c | 29 +++++++++++++++++++++++++++-
> 3 files changed, 30 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h
> index e41b9ea42122..cb74201f5674 100644
> --- a/arch/powerpc/include/asm/smp.h
> +++ b/arch/powerpc/include/asm/smp.h
> @@ -47,6 +47,7 @@ struct smp_ops_t {
> void (*cause_ipi)(int cpu);
> #endif
> int (*cause_nmi_ipi)(int cpu);
> + void (*set_crash_nmi_ipi)(void);
> void (*probe)(void);
> int (*kick_cpu)(int nr);
> int (*prepare_cpu)(int nr);
> diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> index 3467f86fd78f..3390ee8adf79 100644
> --- a/arch/powerpc/kernel/smp.c
> +++ b/arch/powerpc/kernel/smp.c
> @@ -594,6 +594,7 @@ void crash_send_ipi(void (*crash_ipi_callback)(struct pt_regs *))
> {
> int cpu;
>
> + smp_ops->set_crash_nmi_ipi();
Shouldn't we be checking got smp_ops->set_crash_nmi_ipi being non-NULL
before calling. Dont we expect set_crash_nmi_ipi() to be NULL in PowerNV
after this patch? Or what would happen if set_crash_nmi_ipi() was called in
PowerNV code?
> smp_send_nmi_ipi(NMI_IPI_ALL_OTHERS, crash_ipi_callback, 1000000);
> if (kdump_in_progress() && crash_wake_offline) {
> for_each_present_cpu(cpu) {
> diff --git a/arch/powerpc/platforms/pseries/smp.c b/arch/powerpc/platforms/pseries/smp.c
> index db99725e752b..c6c2baacca9a 100644
> --- a/arch/powerpc/platforms/pseries/smp.c
> +++ b/arch/powerpc/platforms/pseries/smp.c
> @@ -51,6 +51,9 @@
> */
> static cpumask_var_t of_spin_mask;
>
> +
Nit: Are we adding an unnecessary newline?
> +static int crash_nmi_ipi;
> +
> /* Query where a cpu is now. Return codes #defined in plpar_wrappers.h */
> int smp_query_cpu_stopped(unsigned int pcpu)
> {
> @@ -171,12 +174,35 @@ static void dbell_or_ic_cause_ipi(int cpu)
> ic_cause_ipi(cpu);
> }
>
> +static void pseries_set_crash_nmi_ipi(void)
> +{
> + crash_nmi_ipi = 1;
> +}
> +
> static int pseries_cause_nmi_ipi(int cpu)
> {
> int hwcpu;
> + int k, curcpu;
>
> + curcpu = smp_processor_id();
> if (cpu == NMI_IPI_ALL_OTHERS) {
> - hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
> + if (crash_nmi_ipi) {
> + for_each_present_cpu(k) {
> + if (k != curcpu) {
> + hwcpu = get_hard_smp_processor_id(k);
> +
> + /* it is possible that cpu is present,
> + * but not started yet.
> + */
> +
> + if (paca_ptrs[hwcpu]->cpu_start == 1) {
> + plpar_signal_sys_reset(hwcpu);
> + }
> + }
> + }
> + return 1;
> + } else
> + hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
> } else {
> if (cpu < 0) {
> WARN_ONCE(true, "incorrect cpu parameter %d", cpu);
> @@ -243,6 +269,7 @@ static struct smp_ops_t pseries_smp_ops = {
> .message_pass = NULL, /* Use smp_muxed_ipi_message_pass */
> .cause_ipi = NULL, /* Filled at runtime by pSeries_smp_probe() */
> .cause_nmi_ipi = pseries_cause_nmi_ipi,
> + .set_crash_nmi_ipi = pseries_set_crash_nmi_ipi,
> .probe = pSeries_smp_probe,
> .prepare_cpu = pseries_smp_prepare_cpu,
> .kick_cpu = smp_pSeries_kick_cpu,
> --
> 2.53.0
>
--
Thanks and Regards
Srikar Dronamraju
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH v2] pseries/kexec: skip resetting CPUs added by firmware but not started by the kernel
2026-03-31 4:33 ` Srikar Dronamraju
@ 2026-03-31 6:44 ` Shivang Upadhyay
0 siblings, 0 replies; 8+ messages in thread
From: Shivang Upadhyay @ 2026-03-31 6:44 UTC (permalink / raw)
To: Srikar Dronamraju
Cc: linuxppc-dev, linux-kernel, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy, Shrikanth Hegde,
Nysal Jan K.A., Vishal Chourasia, Ritesh Harjani, Sourabh Jain,
Anushree Mathur
Hi Srikar,
Thanks for reviewing.
On Tue, 2026-03-31 at 10:03 +0530, Srikar Dronamraju wrote:
> * Shivang Upadhyay <shivangu@linux.ibm.com> [2026-03-30 11:52:06]:
>
> > During DLPAR operations, the newly added CPUs start in halted mode.
> > The kernel then takes some time to initialize those CPUs internally
> > and
> > start them using the "start-cpu" RTAS call. However, if a kexec
> > crash
> > occurs in this window (before the new CPU has been initialized),
> > the kexec NMI will try to reset all other CPUs from the crashing
> > CPU.
> > This leads to firmware starting the uninitialized CPUs as well.
>
> What would happen if an non kexec crash nmi was delivered when we did
> a
> DLPAR operation and before the CPUs are initialized?
As per my understanding, similar hang like situation should happen
there also. But I think this case would be highly unlikely. Crash's
case is little special, because this is called for offline cpus also.
>
> >
> > This can cause the kdump kernel to hang during bring-up.
> >
> > Sample Log:
> > [175993.028231][ T1502] NIP [00007fffb953f394] 0x7fffb953f394
> > [175993.028314][ T1502] LR [00007fffb953f394] 0x7fffb953f394
> > [175993.028390][ T1502] --- interrupt: 3000
> > [ 5.519483][ T1] Processor 0 is stuck.
> > [ 11.089481][ T1] Processor 1 is stuck.
> >
> > To fix this, only issue the system-reset hcall to CPUs that have
> > actually been started by the kernel.
> >
> > Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
> > Cc: Michael Ellerman <mpe@ellerman.id.au>
> > Cc: Nicholas Piggin <npiggin@gmail.com>
> > Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
> > Cc: Srikar Dronamraju <srikar@linux.ibm.com>
> > Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
> > Cc: Nysal Jan K.A. <nysal@linux.ibm.com>
> > Cc: Vishal Chourasia <vishalc@linux.ibm.com>
> > Cc: Ritesh Harjani <ritesh.list@gmail.com>
> > Cc: Sourabh Jain <sourabhjain@linux.ibm.com>
> > Reported-by: Anushree Mathur <anushree.mathur@linux.vnet.ibm.com>
> > Signed-off-by: Shivang Upadhyay <shivangu@linux.ibm.com>
> > ---
> > Changelog:
> >
> > V2:
> > * added set_crash_nmi_ipi to saperate crash's case from other
> > nmi_ipi
> > users
> >
> > V1:
> > *
> > https://lore.kernel.org/all/20251205142825.44698-1-shivangu@linux.ibm.com/
> > ---
> > arch/powerpc/include/asm/smp.h | 1 +
> > arch/powerpc/kernel/smp.c | 1 +
> > arch/powerpc/platforms/pseries/smp.c | 29
> > +++++++++++++++++++++++++++-
> > 3 files changed, 30 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/powerpc/include/asm/smp.h
> > b/arch/powerpc/include/asm/smp.h
> > index e41b9ea42122..cb74201f5674 100644
> > --- a/arch/powerpc/include/asm/smp.h
> > +++ b/arch/powerpc/include/asm/smp.h
> > @@ -47,6 +47,7 @@ struct smp_ops_t {
> > void (*cause_ipi)(int cpu);
> > #endif
> > int (*cause_nmi_ipi)(int cpu);
> > + void (*set_crash_nmi_ipi)(void);
> > void (*probe)(void);
> > int (*kick_cpu)(int nr);
> > int (*prepare_cpu)(int nr);
> > diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> > index 3467f86fd78f..3390ee8adf79 100644
> > --- a/arch/powerpc/kernel/smp.c
> > +++ b/arch/powerpc/kernel/smp.c
> > @@ -594,6 +594,7 @@ void crash_send_ipi(void
> > (*crash_ipi_callback)(struct pt_regs *))
> > {
> > int cpu;
> >
> > + smp_ops->set_crash_nmi_ipi();
>
> Shouldn't we be checking got smp_ops->set_crash_nmi_ipi being non-
> NULL
> before calling. Dont we expect set_crash_nmi_ipi() to be NULL in
> PowerNV
> after this patch? Or what would happen if set_crash_nmi_ipi() was
> called in
> PowerNV code?
>
Yeah, that should be a Bug, Ill fix this in new patch. Thanks for
catching this.
> > smp_send_nmi_ipi(NMI_IPI_ALL_OTHERS, crash_ipi_callback,
> > 1000000);
> > if (kdump_in_progress() && crash_wake_offline) {
> > for_each_present_cpu(cpu) {
> > diff --git a/arch/powerpc/platforms/pseries/smp.c
> > b/arch/powerpc/platforms/pseries/smp.c
> > index db99725e752b..c6c2baacca9a 100644
> > --- a/arch/powerpc/platforms/pseries/smp.c
> > +++ b/arch/powerpc/platforms/pseries/smp.c
> > @@ -51,6 +51,9 @@
> > */
> > static cpumask_var_t of_spin_mask;
> >
> > +
>
> Nit: Are we adding an unnecessary newline?
Ack.
>
> > +static int crash_nmi_ipi;
> > +
> > /* Query where a cpu is now. Return codes #defined in
> > plpar_wrappers.h */
> > int smp_query_cpu_stopped(unsigned int pcpu)
> > {
> > @@ -171,12 +174,35 @@ static void dbell_or_ic_cause_ipi(int cpu)
> > ic_cause_ipi(cpu);
> > }
> >
> > +static void pseries_set_crash_nmi_ipi(void)
> > +{
> > + crash_nmi_ipi = 1;
> > +}
> > +
> > static int pseries_cause_nmi_ipi(int cpu)
> > {
> > int hwcpu;
> > + int k, curcpu;
> >
> > + curcpu = smp_processor_id();
> > if (cpu == NMI_IPI_ALL_OTHERS) {
> > - hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
> > + if (crash_nmi_ipi) {
> > + for_each_present_cpu(k) {
> > + if (k != curcpu) {
> > + hwcpu =
> > get_hard_smp_processor_id(k);
> > +
> > + /* it is possible that cpu
> > is present,
> > + * but not started yet.
> > + */
> > +
> > + if (paca_ptrs[hwcpu]-
> > >cpu_start == 1) {
> > + plpar_signal_sys_r
> > eset(hwcpu);
> > + }
> > + }
> > + }
> > + return 1;
> > + } else
> > + hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
> > } else {
> > if (cpu < 0) {
> > WARN_ONCE(true, "incorrect cpu parameter
> > %d", cpu);
> > @@ -243,6 +269,7 @@ static struct smp_ops_t pseries_smp_ops = {
> > .message_pass = NULL, /* Use
> > smp_muxed_ipi_message_pass */
> > .cause_ipi = NULL, /* Filled at runtime by
> > pSeries_smp_probe() */
> > .cause_nmi_ipi = pseries_cause_nmi_ipi,
> > + .set_crash_nmi_ipi = pseries_set_crash_nmi_ipi,
> > .probe = pSeries_smp_probe,
> > .prepare_cpu = pseries_smp_prepare_cpu,
> > .kick_cpu = smp_pSeries_kick_cpu,
> > --
> > 2.53.0
> >
Thanks
~Shivang.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] pseries/kexec: skip resetting CPUs added by firmware but not started by the kernel
2026-03-30 6:22 [PATCH v2] pseries/kexec: skip resetting CPUs added by firmware but not started by the kernel Shivang Upadhyay
2026-03-30 8:49 ` Shrikanth Hegde
2026-03-31 4:33 ` Srikar Dronamraju
@ 2026-04-06 8:52 ` Vishal Chourasia
2026-04-07 10:19 ` Shivang Upadhyay
2 siblings, 1 reply; 8+ messages in thread
From: Vishal Chourasia @ 2026-04-06 8:52 UTC (permalink / raw)
To: Shivang Upadhyay, linuxppc-dev, linux-kernel
Cc: Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy, Srikar Dronamraju, Shrikanth Hegde,
Nysal Jan K.A., Ritesh Harjani, Sourabh Jain, Anushree Mathur
Hi Shivang,
Thanks for working on this issue.
A few questions and concerns about the approach:
1. Was this issue only observed with QEMU-based virtualization, or does
it also reproduce on PowerVM/phyp? The commit message and sample logs
don't clarify this. If this is QEMU-specific, I think we should fix this
in QEMU rather than working around it in the kernel.
2. The approach taken here moves away from the PAPR interface. The
kernel currently uses H_SIGNAL_SYS_RESET_ALL_OTHERS, which is the
architecturally defined hcall for this purpose. Replacing it with a
per-CPU loop that checks internal kernel state (paca cpu_start) breaks
the clean abstraction between guest and hypervisor.
QEMU's sPAPR implementation should behave the same way. The hypervisor
(QEMU) should maintain a list of CPUs that have been
activated/online/started and given to the guest. When
H_SIGNAL_SYS_RESET_ALL_OTHERS is called, QEMU should only reset those
CPUs that the guest has actually started. Unless the guest makes the
RTAS start-cpu call for a CPU, QEMU should not include that CPU in the
set of CPUs to be reset.
I think discussing this would help determine the right fix location.
Can you refer to the following commit in QEMU to see if help in this case.
commit fb802acdc8b162084e9e60d42aeba79097d14d2b
Author: Nicholas Piggin <npiggin@gmail.com>
Date: Tue Mar 18 15:03:48 2025 +1000
ppc/spapr: Fix RTAS stopped state
Thanks and Regards,
Vishal
On 30/03/26 11:52, Shivang Upadhyay wrote:
> During DLPAR operations, the newly added CPUs start in halted mode.
> The kernel then takes some time to initialize those CPUs internally and
> start them using the "start-cpu" RTAS call. However, if a kexec crash
> occurs in this window (before the new CPU has been initialized),
> the kexec NMI will try to reset all other CPUs from the crashing CPU.
> This leads to firmware starting the uninitialized CPUs as well.
>
> This can cause the kdump kernel to hang during bring-up.
>
> Sample Log:
> [175993.028231][ T1502] NIP [00007fffb953f394] 0x7fffb953f394
> [175993.028314][ T1502] LR [00007fffb953f394] 0x7fffb953f394
> [175993.028390][ T1502] --- interrupt: 3000
> [ 5.519483][ T1] Processor 0 is stuck.
> [ 11.089481][ T1] Processor 1 is stuck.
>
> To fix this, only issue the system-reset hcall to CPUs that have
> actually been started by the kernel.
>
> Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Nicholas Piggin <npiggin@gmail.com>
> Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
> Cc: Srikar Dronamraju <srikar@linux.ibm.com>
> Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
> Cc: Nysal Jan K.A. <nysal@linux.ibm.com>
> Cc: Vishal Chourasia <vishalc@linux.ibm.com>
> Cc: Ritesh Harjani <ritesh.list@gmail.com>
> Cc: Sourabh Jain <sourabhjain@linux.ibm.com>
> Reported-by: Anushree Mathur <anushree.mathur@linux.vnet.ibm.com>
> Signed-off-by: Shivang Upadhyay <shivangu@linux.ibm.com>
> ---
> Changelog:
>
> V2:
> * added set_crash_nmi_ipi to saperate crash's case from other nmi_ipi
> users
>
> V1:
> * https://lore.kernel.org/all/20251205142825.44698-1-shivangu@linux.ibm.com/
> ---
> arch/powerpc/include/asm/smp.h | 1 +
> arch/powerpc/kernel/smp.c | 1 +
> arch/powerpc/platforms/pseries/smp.c | 29 +++++++++++++++++++++++++++-
> 3 files changed, 30 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h
> index e41b9ea42122..cb74201f5674 100644
> --- a/arch/powerpc/include/asm/smp.h
> +++ b/arch/powerpc/include/asm/smp.h
> @@ -47,6 +47,7 @@ struct smp_ops_t {
> void (*cause_ipi)(int cpu);
> #endif
> int (*cause_nmi_ipi)(int cpu);
> + void (*set_crash_nmi_ipi)(void);
> void (*probe)(void);
> int (*kick_cpu)(int nr);
> int (*prepare_cpu)(int nr);
> diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> index 3467f86fd78f..3390ee8adf79 100644
> --- a/arch/powerpc/kernel/smp.c
> +++ b/arch/powerpc/kernel/smp.c
> @@ -594,6 +594,7 @@ void crash_send_ipi(void (*crash_ipi_callback)(struct pt_regs *))
> {
> int cpu;
>
> + smp_ops->set_crash_nmi_ipi();
> smp_send_nmi_ipi(NMI_IPI_ALL_OTHERS, crash_ipi_callback, 1000000);
> if (kdump_in_progress() && crash_wake_offline) {
> for_each_present_cpu(cpu) {
> diff --git a/arch/powerpc/platforms/pseries/smp.c b/arch/powerpc/platforms/pseries/smp.c
> index db99725e752b..c6c2baacca9a 100644
> --- a/arch/powerpc/platforms/pseries/smp.c
> +++ b/arch/powerpc/platforms/pseries/smp.c
> @@ -51,6 +51,9 @@
> */
> static cpumask_var_t of_spin_mask;
>
> +
> +static int crash_nmi_ipi;
> +
> /* Query where a cpu is now. Return codes #defined in plpar_wrappers.h */
> int smp_query_cpu_stopped(unsigned int pcpu)
> {
> @@ -171,12 +174,35 @@ static void dbell_or_ic_cause_ipi(int cpu)
> ic_cause_ipi(cpu);
> }
>
> +static void pseries_set_crash_nmi_ipi(void)
> +{
> + crash_nmi_ipi = 1;
> +}
> +
> static int pseries_cause_nmi_ipi(int cpu)
> {
> int hwcpu;
> + int k, curcpu;
>
> + curcpu = smp_processor_id();
> if (cpu == NMI_IPI_ALL_OTHERS) {
> - hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
> + if (crash_nmi_ipi) {
> + for_each_present_cpu(k) {
> + if (k != curcpu) {
> + hwcpu = get_hard_smp_processor_id(k);
> +
> + /* it is possible that cpu is present,
> + * but not started yet.
> + */
> +
> + if (paca_ptrs[hwcpu]->cpu_start == 1) {
> + plpar_signal_sys_reset(hwcpu);
> + }
> + }
> + }
> + return 1;
> + } else
> + hwcpu = H_SIGNAL_SYS_RESET_ALL_OTHERS;
> } else {
> if (cpu < 0) {
> WARN_ONCE(true, "incorrect cpu parameter %d", cpu);
> @@ -243,6 +269,7 @@ static struct smp_ops_t pseries_smp_ops = {
> .message_pass = NULL, /* Use smp_muxed_ipi_message_pass */
> .cause_ipi = NULL, /* Filled at runtime by pSeries_smp_probe() */
> .cause_nmi_ipi = pseries_cause_nmi_ipi,
> + .set_crash_nmi_ipi = pseries_set_crash_nmi_ipi,
> .probe = pSeries_smp_probe,
> .prepare_cpu = pseries_smp_prepare_cpu,
> .kick_cpu = smp_pSeries_kick_cpu,
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH v2] pseries/kexec: skip resetting CPUs added by firmware but not started by the kernel
2026-04-06 8:52 ` Vishal Chourasia
@ 2026-04-07 10:19 ` Shivang Upadhyay
2026-04-07 10:25 ` Vishal Chourasia
0 siblings, 1 reply; 8+ messages in thread
From: Shivang Upadhyay @ 2026-04-07 10:19 UTC (permalink / raw)
To: Vishal Chourasia, linuxppc-dev, linux-kernel
Cc: Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy, Srikar Dronamraju, Shrikanth Hegde,
Nysal Jan K.A., Ritesh Harjani, Sourabh Jain, Anushree Mathur
Hi,
Thanks for your review.
On Mon, 2026-04-06 at 14:22 +0530, Vishal Chourasia wrote:
> Hi Shivang,
>
> Thanks for working on this issue.
> A few questions and concerns about the approach:
>
> 1. Was this issue only observed with QEMU-based virtualization, or
> does
> it also reproduce on PowerVM/phyp? The commit message and sample logs
> don't clarify this. If this is QEMU-specific, I think we should fix
> this
> in QEMU rather than working around it in the kernel.
Currently this is only happening in Qemu (both tcg and kvm mode). But I
think this should be reproducible on phyp also. Ill confirm wheather it
is really the case or not.
> 2. The approach taken here moves away from the PAPR interface. The
> kernel currently uses H_SIGNAL_SYS_RESET_ALL_OTHERS, which is the
> architecturally defined hcall for this purpose. Replacing it with a
> per-CPU loop that checks internal kernel state (paca cpu_start)
> breaks
> the clean abstraction between guest and
> QEMU's sPAPR implementation should behave the same way. The
> hypervisor
Yeah it is a valid concern about ownership for this resets. Ill try to
see if this fix is possible in qemu itself.
> (QEMU) should maintain a list of CPUs that have been
> activated/online/started and given to the guest. When
> H_SIGNAL_SYS_RESET_ALL_OTHERS is called, QEMU should only reset those
> CPUs that the guest has actually started. Unless the guest makes the
> RTAS start-cpu call for a CPU, QEMU should not include that CPU in
> the
> set of CPUs to be reset.
>
> I think discussing this would help determine the right fix location.
>
> Can you refer to the following commit in QEMU to see if help in this
> case.
>
> commit fb802acdc8b162084e9e60d42aeba79097d14d2b
> Author: Nicholas Piggin <npiggin@gmail.com>
> Date: Tue Mar 18 15:03:48 2025 +1000
>
> ppc/spapr: Fix RTAS stopped state
>
Thanks for this reference. cpu->quiesced state was introduced in this
patch, for modelling "RTAS stopped" state.
as per the commit message:
A KVM spapr guest boots with all secondary CPUs defined to be in the
RTAS stopped" state. In this state, the CPU is only responsive to the
start-cpu RTAS call.
So, we should be able to use this to check wheather cpu is started or
not. Only other concern here would be about phyp's implementation for
this.
Thanks.
~Shivang.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] pseries/kexec: skip resetting CPUs added by firmware but not started by the kernel
2026-04-07 10:19 ` Shivang Upadhyay
@ 2026-04-07 10:25 ` Vishal Chourasia
0 siblings, 0 replies; 8+ messages in thread
From: Vishal Chourasia @ 2026-04-07 10:25 UTC (permalink / raw)
To: Shivang Upadhyay, linuxppc-dev, linux-kernel
Cc: Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy, Srikar Dronamraju, Shrikanth Hegde,
Nysal Jan K.A., Ritesh Harjani, Sourabh Jain, Anushree Mathur
On 07/04/26 15:49, Shivang Upadhyay wrote:
> Hi,
> Thanks for your review.
>
> On Mon, 2026-04-06 at 14:22 +0530, Vishal Chourasia wrote:
>> Hi Shivang,
>>
>> Thanks for working on this issue.
>> A few questions and concerns about the approach:
>>
>> 1. Was this issue only observed with QEMU-based virtualization, or
>> does
>> it also reproduce on PowerVM/phyp? The commit message and sample logs
>> don't clarify this. If this is QEMU-specific, I think we should fix
>> this
>> in QEMU rather than working around it in the kernel.
> Currently this is only happening in Qemu (both tcg and kvm mode). But I
> think this should be reproducible on phyp also. Ill confirm wheather it
> is really the case or not.
>
>> 2. The approach taken here moves away from the PAPR interface. The
>> kernel currently uses H_SIGNAL_SYS_RESET_ALL_OTHERS, which is the
>> architecturally defined hcall for this purpose. Replacing it with a
>> per-CPU loop that checks internal kernel state (paca cpu_start)
>> breaks
>> the clean abstraction between guest and
>> QEMU's sPAPR implementation should behave the same way. The
>> hypervisor
> Yeah it is a valid concern about ownership for this resets. Ill try to
> see if this fix is possible in qemu itself.
>
>> (QEMU) should maintain a list of CPUs that have been
>> activated/online/started and given to the guest. When
>> H_SIGNAL_SYS_RESET_ALL_OTHERS is called, QEMU should only reset those
>> CPUs that the guest has actually started. Unless the guest makes the
>> RTAS start-cpu call for a CPU, QEMU should not include that CPU in
>> the
>> set of CPUs to be reset.
>>
>> I think discussing this would help determine the right fix location.
>>
>> Can you refer to the following commit in QEMU to see if help in this
>> case.
>>
>> commit fb802acdc8b162084e9e60d42aeba79097d14d2b
>> Author: Nicholas Piggin <npiggin@gmail.com>
>> Date: Tue Mar 18 15:03:48 2025 +1000
>>
>> ppc/spapr: Fix RTAS stopped state
>>
> Thanks for this reference. cpu->quiesced state was introduced in this
> patch, for modelling "RTAS stopped" state.
>
> as per the commit message:
> A KVM spapr guest boots with all secondary CPUs defined to be in the
> RTAS stopped" state. In this state, the CPU is only responsive to the
> start-cpu RTAS call.
>
> So, we should be able to use this to check wheather cpu is started or
> not. Only other concern here would be about phyp's implementation for
> this.
Yes, something like this.
diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c
index 032805a8d0..8c51372cf8 100644
--- a/hw/ppc/spapr_hcall.c
+++ b/hw/ppc/spapr_hcall.c
@@ -1105,6 +1105,9 @@ static target_ulong h_signal_sys_reset(PowerPCCPU
*cpu,
continue;
}
}
+
+ if (c->env.quiesced) continue;
+
run_on_cpu(cs, spapr_do_system_reset_on_cpu, RUN_ON_CPU_NULL);
}
return H_SUCCESS;
>
> Thanks.
> ~Shivang.
>
>
>
^ permalink raw reply related [flat|nested] 8+ messages in thread