Re: sched: Unexpected reschedule of offline CPU#2!

Re: sched: Unexpected reschedule of offline CPU#2!

[Henning Schild]

Was this ever resolved and if so can someone please point me to the
patches? I started digging a bit but could not yet find how that
continued.

I am seeing similar or maybe the same problem on 4.19.192 with the
ipipe patch from the xenomai project applied.

regards,
Henning

Am Sat, 17 Aug 2019 22:21:48 +0200
schrieb Thomas Gleixner <tglx@linutronix.de>:

> On Fri, 16 Aug 2019, Guenter Roeck wrote:
> > On Fri, Aug 16, 2019 at 12:22:22PM +0200, Thomas Gleixner wrote:  
> > > diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> > > index 75fea0d48c0e..625627b1457c 100644
> > > --- a/arch/x86/kernel/process.c
> > > +++ b/arch/x86/kernel/process.c
> > > @@ -601,6 +601,7 @@ void stop_this_cpu(void *dummy)
> > >  	/*
> > >  	 * Remove this CPU:
> > >  	 */
> > > +	set_cpu_active(smp_processor_id(), false);
> > >  	set_cpu_online(smp_processor_id(), false);
> > >  	disable_local_APIC();
> > >  	mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
> > >   
> > No luck. The problem is still seen with this patch applied on top of
> > the mainline kernel (commit a69e90512d9def6).  
> 
> Yeah, was a bit too naive ....
> 
> We actually need to do the full cpuhotplug dance for a regular
> reboot. In the panic case, there is nothing we can do about. I'll
> have a look tomorrow.
> 
> Thanks,
> 
> 	tglx

Re: sched: Unexpected reschedule of offline CPU#2!

[Jan Kiszka]

[Henning, don't top-post ;)]

On 27.07.21 10:00, Henning Schild via Xenomai wrote:
> Was this ever resolved and if so can someone please point me to the
> patches? I started digging a bit but could not yet find how that
> continued.
> 
> I am seeing similar or maybe the same problem on 4.19.192 with the
> ipipe patch from the xenomai project applied.
> 

Before blaming the usual suspects, I have a general ordering question on 
mainline below.

> regards,
> Henning
> 
> Am Sat, 17 Aug 2019 22:21:48 +0200
> schrieb Thomas Gleixner <tglx@linutronix.de>:
> 
>> On Fri, 16 Aug 2019, Guenter Roeck wrote:
>>> On Fri, Aug 16, 2019 at 12:22:22PM +0200, Thomas Gleixner wrote:  
>>>> diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
>>>> index 75fea0d48c0e..625627b1457c 100644
>>>> --- a/arch/x86/kernel/process.c
>>>> +++ b/arch/x86/kernel/process.c
>>>> @@ -601,6 +601,7 @@ void stop_this_cpu(void *dummy)
>>>>  	/*
>>>>  	 * Remove this CPU:
>>>>  	 */
>>>> +	set_cpu_active(smp_processor_id(), false);
>>>>  	set_cpu_online(smp_processor_id(), false);
>>>>  	disable_local_APIC();
>>>>  	mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
>>>>   
>>> No luck. The problem is still seen with this patch applied on top of
>>> the mainline kernel (commit a69e90512d9def6).  
>>
>> Yeah, was a bit too naive ....
>>
>> We actually need to do the full cpuhotplug dance for a regular
>> reboot. In the panic case, there is nothing we can do about. I'll
>> have a look tomorrow.
>>

What is supposed to prevent the following in mainline:

CPU 0                   CPU 1                      CPU 2

native_stop_other_cpus                             <INTERRUPT>
  send_IPI_allbutself                              ...
                        <INTERRUPT>
                        sysvec_reboot
                          stop_this_cpu
                            set_cpu_online(false)
                                                   native_smp_send_reschedule(1)
                                                     if (cpu_is_offline(1)) ...

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: sched: Unexpected reschedule of offline CPU#2!

[guocai.he.cn]

if there have any updates or fixes about this issue?

we meet the similar issue, logs as following:

root@doon:~# poweroff
...............
...............

-----------[ cut here ]-----------

sched: Unexpected reschedule of offline CPU#10!

WARNING: CPU: 0 PID: 446324 at arch/x86/kernel/smp.c:126 native_smp_send_reschedule+0x3a/0x40

Modules linked in: vhost_net vhost macvtap tap xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_r

eject_ipv4 iptable_mangle iptable_nat linux_user_bde(PO) linux_kernel_bde(PO) xt_tcpudp bridge stp ll

c ip6table_filter ip6_tables iptable_filter ip_tables x_tables kvm_intel kvm vfio_pci vfio_virqfd vfi

o_iommu_type1 vfio pci_stub iavf uio_pci_hostif i40e(O) configfs qfx_pci_static_map(O) macvlan socktu

n(O) i2c_dev uio_fpga(O) uio intel_rapl_msr iTCO_wdt iTCO_vendor_support watchdog intel_rapl_common x

86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crct10dif_common aesni_intel aes_x86_6

4 glue_helper crypto_simd cryptd i2c_i801 igb(O) lpc_ich pcc_cpufreq sch_fq_codel nfsd openvswitch ns

h nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 irqbypass fuse [last unloaded: ebtab

les]

CPU: 0 PID: 446324 Comm: kworker/0:11 Tainted: P           O      5.2.60-rt15-LTS19 #1

Workqueue:  0x0 (rcu_gp)

RIP: 0010:native_smp_send_reschedule+0x3a/0x40

Code: 4f 9b 01 73 17 48 8b 05 34 1c 5b 01 be fd 00 00 00 48 8b 40 30 e8 a6 ac fb 00 5d c3 89 fe 48 c7

c7 28 08 b1 b8 e8 42 5c 02 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 8b 05 3d f8 ba 01 85 c0 0f 85 e1

RSP: 0018:ffff9dc940003c68 EFLAGS: 00010086

RAX: 0000000000000000 RBX: ffff9138c00a3400 RCX: 0000000000000006

RDX: 0000000000000007 RSI: 0000000000000003 RDI: ffff9138bfe16450

RBP: ffff9dc940003c68 R08: 0000099801c9bf9b R09: 0000000000000000

R10: ffff9dc940003a08 R11: 0000000000000002 R12: 000000000000000a

R13: ffff9dc940003d30 R14: 0000000000000000 R15: ffff9138c00a3400

FS:  0000000000000000(0000) GS:ffff9138bfe00000(0000) knlGS:0000000000000000

CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00000000000000b0 CR3: 0000000305364006 CR4: 00000000003606f0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Call Trace:

<IRQ>

resched_curr+0x69/0xc0

check_preempt_curr+0x54/0x90

ttwu_do_wakeup.isra.0+0x1e/0x150

ttwu_do_activate+0x5b/0x70

try_to_wake_up+0x224/0x570

? enqueue_task_fair+0x1f0/0xa70

? tracing_record_taskinfo_skip+0x3f/0x50

default_wake_function+0x12/0x20

autoremove_wake_function+0x12/0x40

__wake_up_common+0x7e/0x140

__wake_up_common_lock+0x7b/0xf0

__wake_up+0x13/0x20

wake_up_klogd_work_func+0x39/0x40

irq_work_run_list+0x4f/0x70

irq_work_tick+0x3b/0x50

update_process_times+0x65/0x70

tick_sched_timer+0x59/0x170

? tick_switch_to_oneshot.cold+0x79/0x79

__hrtimer_run_queues+0x10f/0x290

? recalibrate_cpu_khz+0x10/0x10

hrtimer_interrupt+0x109/0x220

smp_apic_timer_interrupt+0x76/0x150

apic_timer_interrupt+0xf/0x20

</IRQ>

RIP: 0010:finish_task_switch+0x87/0x280

Code: 85 c0 0f 8f dc 00 00 00 8b 05 e5 6b bd 01 85 c0 0f 8f e7 00 00 00 41 c7 45 40 00 00 00 00 41 c6

04 24 00 fb 8b 05 e9 f0 b5 01 <65> 48 8b 14 25 00 5d 01 00 85 c0 0f 8f 23 01 00 00 4d 85 f6 74 1d

RSP: 0018:ffff9dc94725bdf8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13

RAX: 0000000000000000 RBX: ffff9138a093c180 RCX: 0000000000000001

RDX: 0000000000000000 RSI: ffff9138a093c180 RDI: ffff9138b9298040

RBP: ffff9dc94725be20 R08: 000000000000022d R09: ffff9138a093c258

R10: 0000000000000000 R11: 0000000000000000 R12: ffff9138bfe23400

R13: ffff9138b9298040 R14: 0000000000000000 R15: 0000000000000002

? __switch_to_asm+0x34/0x70

__schedule+0x30b/0x690

schedule+0x42/0xb0

worker_thread+0xc1/0x3c0

kthread+0x106/0x140

? process_one_work+0x3f0/0x3f0

? kthread_park+0x90/0x90

ret_from_fork+0x35/0x40

--[ end trace 4ff5842bcc9fa5e0 ]

Re: sched: Unexpected reschedule of offline CPU#2!

[Thomas Gleixner]

On Tue, Jul 27 2021 at 10:46, Jan Kiszka wrote:

Picking up this dead thread again.

> What is supposed to prevent the following in mainline:
>
> CPU 0                   CPU 1                      CPU 2
>
> native_stop_other_cpus                             <INTERRUPT>
>   send_IPI_allbutself                              ...
>                         <INTERRUPT>
>                         sysvec_reboot
>                           stop_this_cpu
>                             set_cpu_online(false)
>                                                    native_smp_send_reschedule(1)
>                                                      if (cpu_is_offline(1)) ...

Nothing. And that's what probably happens if I read the stack trace
correctly.

But we can be slightly smarter about this for the reboot IPI (the NMI
case does not have that issue).

CPU 0                   CPU 1                      CPU 2

native_stop_other_cpus                             <INTERRUPT>
   send_IPI_allbutself                              ...
                         <IPI>
                         sysvec_reboot
                           wait_for_others();
                                                    </INTERRUPT>
                                                    <IPI>
                                                    sysvec_reboot
                                                    wait_for_others();
                           stop_this_cpu();           stop_this_cpu();
                             set_cpu_online(false);     set_cpu_online(false);

Something like the uncompiled below.

Thanks,

        tglx
---
--- a/arch/x86/include/asm/cpu.h
+++ b/arch/x86/include/asm/cpu.h
@@ -68,5 +68,6 @@ bool intel_find_matching_signature(void
 int intel_microcode_sanity_check(void *mc, bool print_err, int hdr_type);
 
 extern struct cpumask cpus_stop_mask;
+atomic_t cpus_stop_in_ipi;
 
 #endif /* _ASM_X86_CPU_H */
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -721,7 +721,7 @@ bool xen_set_default_idle(void);
 #define xen_set_default_idle 0
 #endif
 
-void __noreturn stop_this_cpu(void *dummy);
+void __noreturn stop_this_cpu(bool sync);
 void microcode_check(struct cpuinfo_x86 *prev_info);
 void store_cpu_caps(struct cpuinfo_x86 *info);
 
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -791,9 +791,10 @@ bool xen_set_default_idle(void)
 }
 #endif
 
+atomic_t cpus_stop_in_ipi;
 struct cpumask cpus_stop_mask;
 
-void __noreturn stop_this_cpu(void *dummy)
+void __noreturn stop_this_cpu(bool sync)
 {
 	struct cpuinfo_x86 *c = this_cpu_ptr(&cpu_info);
 	unsigned int cpu = smp_processor_id();
@@ -801,6 +802,16 @@ void __noreturn stop_this_cpu(void *dumm
 	local_irq_disable();
 
 	/*
+	 * Account this CPU and loop until the other CPUs reached this
+	 * point. If they don't react, the control CPU will raise an NMI.
+	 */
+	if (sync) {
+		atomic_dec(&cpus_stop_in_ipi);
+		while (atomic_read(&cpus_stop_in_ipi))
+			cpu_relax();
+	}
+
+	/*
 	 * Remove this CPU from the online mask and disable it
 	 * unconditionally. This might be redundant in case that the reboot
 	 * vector was handled late and stop_other_cpus() sent an NMI.
--- a/arch/x86/kernel/reboot.c
+++ b/arch/x86/kernel/reboot.c
@@ -788,7 +788,7 @@ static void native_machine_halt(void)
 
 	tboot_shutdown(TB_SHUTDOWN_HALT);
 
-	stop_this_cpu(NULL);
+	stop_this_cpu(false);
 }
 
 static void native_machine_power_off(void)
--- a/arch/x86/kernel/smp.c
+++ b/arch/x86/kernel/smp.c
@@ -125,7 +125,7 @@ static int smp_stop_nmi_callback(unsigne
 		return NMI_HANDLED;
 
 	cpu_emergency_disable_virtualization();
-	stop_this_cpu(NULL);
+	stop_this_cpu(false);
 
 	return NMI_HANDLED;
 }
@@ -137,7 +137,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_reboot)
 {
 	apic_eoi();
 	cpu_emergency_disable_virtualization();
-	stop_this_cpu(NULL);
+	stop_this_cpu(true);
 }
 
 static int register_stop_handler(void)
@@ -189,6 +189,7 @@ static void native_stop_other_cpus(int w
 	 */
 	cpumask_copy(&cpus_stop_mask, cpu_online_mask);
 	cpumask_clear_cpu(this_cpu, &cpus_stop_mask);
+	atomic_set(&cpus_stop_in_ipi, num_online_cpus() - 1);
 
 	if (!cpumask_empty(&cpus_stop_mask)) {
 		apic_send_IPI_allbutself(REBOOT_VECTOR);
@@ -235,10 +236,12 @@ static void native_stop_other_cpus(int w
 	local_irq_restore(flags);
 
 	/*
-	 * Ensure that the cpus_stop_mask cache lines are invalidated on
-	 * the other CPUs. See comment vs. SME in stop_this_cpu().
+	 * Ensure that the cpus_stop_mask and cpus_stop_in_ipi cache lines
+	 * are invalidated on the other CPUs. See comment vs. SME in
+	 * stop_this_cpu().
 	 */
 	cpumask_clear(&cpus_stop_mask);
+	atomic_set(&cpus_stop_in_ipi, 0);
 }
 
 /*

Re: sched: Unexpected reschedule of offline CPU#2!

[guocai he]

Thanks very much.

I will let our customer to try this patch and let you know the result.


-guocai

On 9/3/24 23:27, Thomas Gleixner wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> On Tue, Jul 27 2021 at 10:46, Jan Kiszka wrote:
>
> Picking up this dead thread again.
>
>> What is supposed to prevent the following in mainline:
>>
>> CPU 0                   CPU 1                      CPU 2
>>
>> native_stop_other_cpus                             <INTERRUPT>
>>    send_IPI_allbutself                              ...
>>                          <INTERRUPT>
>>                          sysvec_reboot
>>                            stop_this_cpu
>>                              set_cpu_online(false)
>>                                                     native_smp_send_reschedule(1)
>>                                                       if (cpu_is_offline(1)) ...
> Nothing. And that's what probably happens if I read the stack trace
> correctly.
>
> But we can be slightly smarter about this for the reboot IPI (the NMI
> case does not have that issue).
>
> CPU 0                   CPU 1                      CPU 2
>
> native_stop_other_cpus                             <INTERRUPT>
>     send_IPI_allbutself                              ...
>                           <IPI>
>                           sysvec_reboot
>                             wait_for_others();
>                                                      </INTERRUPT>
>                                                      <IPI>
>                                                      sysvec_reboot
>                                                      wait_for_others();
>                             stop_this_cpu();           stop_this_cpu();
>                               set_cpu_online(false);     set_cpu_online(false);
>
> Something like the uncompiled below.
>
> Thanks,
>
>          tglx
> ---
> --- a/arch/x86/include/asm/cpu.h
> +++ b/arch/x86/include/asm/cpu.h
> @@ -68,5 +68,6 @@ bool intel_find_matching_signature(void
>   int intel_microcode_sanity_check(void *mc, bool print_err, int hdr_type);
>
>   extern struct cpumask cpus_stop_mask;
> +atomic_t cpus_stop_in_ipi;
>
>   #endif /* _ASM_X86_CPU_H */
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -721,7 +721,7 @@ bool xen_set_default_idle(void);
>   #define xen_set_default_idle 0
>   #endif
>
> -void __noreturn stop_this_cpu(void *dummy);
> +void __noreturn stop_this_cpu(bool sync);
>   void microcode_check(struct cpuinfo_x86 *prev_info);
>   void store_cpu_caps(struct cpuinfo_x86 *info);
>
> --- a/arch/x86/kernel/process.c
> +++ b/arch/x86/kernel/process.c
> @@ -791,9 +791,10 @@ bool xen_set_default_idle(void)
>   }
>   #endif
>
> +atomic_t cpus_stop_in_ipi;
>   struct cpumask cpus_stop_mask;
>
> -void __noreturn stop_this_cpu(void *dummy)
> +void __noreturn stop_this_cpu(bool sync)
>   {
>          struct cpuinfo_x86 *c = this_cpu_ptr(&cpu_info);
>          unsigned int cpu = smp_processor_id();
> @@ -801,6 +802,16 @@ void __noreturn stop_this_cpu(void *dumm
>          local_irq_disable();
>
>          /*
> +        * Account this CPU and loop until the other CPUs reached this
> +        * point. If they don't react, the control CPU will raise an NMI.
> +        */
> +       if (sync) {
> +               atomic_dec(&cpus_stop_in_ipi);
> +               while (atomic_read(&cpus_stop_in_ipi))
> +                       cpu_relax();
> +       }
> +
> +       /*
>           * Remove this CPU from the online mask and disable it
>           * unconditionally. This might be redundant in case that the reboot
>           * vector was handled late and stop_other_cpus() sent an NMI.
> --- a/arch/x86/kernel/reboot.c
> +++ b/arch/x86/kernel/reboot.c
> @@ -788,7 +788,7 @@ static void native_machine_halt(void)
>
>          tboot_shutdown(TB_SHUTDOWN_HALT);
>
> -       stop_this_cpu(NULL);
> +       stop_this_cpu(false);
>   }
>
>   static void native_machine_power_off(void)
> --- a/arch/x86/kernel/smp.c
> +++ b/arch/x86/kernel/smp.c
> @@ -125,7 +125,7 @@ static int smp_stop_nmi_callback(unsigne
>                  return NMI_HANDLED;
>
>          cpu_emergency_disable_virtualization();
> -       stop_this_cpu(NULL);
> +       stop_this_cpu(false);
>
>          return NMI_HANDLED;
>   }
> @@ -137,7 +137,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_reboot)
>   {
>          apic_eoi();
>          cpu_emergency_disable_virtualization();
> -       stop_this_cpu(NULL);
> +       stop_this_cpu(true);
>   }
>
>   static int register_stop_handler(void)
> @@ -189,6 +189,7 @@ static void native_stop_other_cpus(int w
>           */
>          cpumask_copy(&cpus_stop_mask, cpu_online_mask);
>          cpumask_clear_cpu(this_cpu, &cpus_stop_mask);
> +       atomic_set(&cpus_stop_in_ipi, num_online_cpus() - 1);
>
>          if (!cpumask_empty(&cpus_stop_mask)) {
>                  apic_send_IPI_allbutself(REBOOT_VECTOR);
> @@ -235,10 +236,12 @@ static void native_stop_other_cpus(int w
>          local_irq_restore(flags);
>
>          /*
> -        * Ensure that the cpus_stop_mask cache lines are invalidated on
> -        * the other CPUs. See comment vs. SME in stop_this_cpu().
> +        * Ensure that the cpus_stop_mask and cpus_stop_in_ipi cache lines
> +        * are invalidated on the other CPUs. See comment vs. SME in
> +        * stop_this_cpu().
>           */
>          cpumask_clear(&cpus_stop_mask);
> +       atomic_set(&cpus_stop_in_ipi, 0);
>   }
>
>   /*
>

My branch is v5.2/standard/preempt-rt/intel-x86 and I make a patch according

[guocai.he.cn]

From cbf1606332c48b32c4bb8d61ac6911e3064e79fc Mon Sep 17 00:00:00 2001
From: Guocai He <guocai.he.cn@windriver.com>
Date: Wed, 4 Sep 2024 04:45:26 +0000
Subject: [PATCH] patch for poweroff

---
 arch/x86/include/asm/processor.h |  2 +-
 arch/x86/kernel/process.c        | 14 +++++++++++++-
 arch/x86/kernel/reboot.c         |  2 +-
 arch/x86/kernel/smp.c            |  9 ++++++---
 4 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 5f4e79d14613..4c1cf610807a 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -975,7 +975,7 @@ bool xen_set_default_idle(void);
 #define xen_set_default_idle 0
 #endif
 
-void stop_this_cpu(void *dummy);
+void stop_this_cpu(bool sync);
 void df_debug(struct pt_regs *regs, long error_code);
 void microcode_check(void);
 
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index 2243af6530f8..35d5cf73716e 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -590,9 +590,21 @@ bool xen_set_default_idle(void)
 }
 #endif
 
-void stop_this_cpu(void *dummy)
+atomic_t cpus_stop_in_ipi;
+void stop_this_cpu(bool sync)
 {
 	local_irq_disable();
+
+    /*
+     * Account this cpu and loop until the other cpus reached this
+     * point. If they don't react, the control cpu will raise an NMI.
+     */
+    if(sync) {
+             atomic_dec(&cpus_stop_in_ipi);
+             while (atomic_read(&cpus_stop_in_ipi))
+                   cpu_relax();
+    }
+
 	/*
 	 * Remove this CPU:
 	 */
diff --git a/arch/x86/kernel/reboot.c b/arch/x86/kernel/reboot.c
index 3f677832fc12..389643727e37 100644
--- a/arch/x86/kernel/reboot.c
+++ b/arch/x86/kernel/reboot.c
@@ -742,7 +742,7 @@ static void native_machine_halt(void)
 
 	tboot_shutdown(TB_SHUTDOWN_HALT);
 
-	stop_this_cpu(NULL);
+	stop_this_cpu(false);
 }
 
 static void native_machine_power_off(void)
diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
index f2a749586252..9dee65b96115 100644
--- a/arch/x86/kernel/smp.c
+++ b/arch/x86/kernel/smp.c
@@ -112,6 +112,7 @@
  *	about nothing of note with C stepping upwards.
  */
 
+extern atomic_t cpus_stop_in_ipi;
 static atomic_t stopping_cpu = ATOMIC_INIT(-1);
 static bool smp_no_nmi_ipi = false;
 
@@ -162,7 +163,7 @@ static int smp_stop_nmi_callback(unsigned int val, struct pt_regs *regs)
 		return NMI_HANDLED;
 
 	cpu_emergency_vmxoff();
-	stop_this_cpu(NULL);
+	stop_this_cpu(false);
 
 	return NMI_HANDLED;
 }
@@ -175,7 +176,7 @@ asmlinkage __visible void smp_reboot_interrupt(void)
 {
 	ipi_entering_ack_irq();
 	cpu_emergency_vmxoff();
-	stop_this_cpu(NULL);
+	stop_this_cpu(true);
 	irq_exit();
 }
 
@@ -192,7 +193,8 @@ static void native_stop_other_cpus(int wait)
 
 	if (reboot_force)
 		return;
-
+    
+    atomic_set(&cpus_stop_in_ipi, num_online_cpus() - 1);
 	/*
 	 * Use an own vector here because smp_call_function
 	 * does lots of things not suitable in a panic situation.
@@ -256,6 +258,7 @@ static void native_stop_other_cpus(int wait)
 	local_irq_save(flags);
 	disable_local_APIC();
 	mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
+    atomic_set(&cpus_stop_in_ipi, 0);
 	local_irq_restore(flags);
 }
 
-- 
2.25.1

[PATCH] patch for poweroff

[guocai.he.cn]

From: Guocai He <guocai.he.cn@windriver.com>

My branch is v5.2/standard/preempt-rt/intel-x86 and I make a patch according
to your adviced patch.
But it does not work.
Do you have more advice?
Thanks very much.

The following is my patch on v5.2:
---
 arch/x86/include/asm/processor.h |  2 +-
 arch/x86/kernel/process.c        | 14 +++++++++++++-
 arch/x86/kernel/reboot.c         |  2 +-
 arch/x86/kernel/smp.c            |  9 ++++++---
 4 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 5f4e79d14613..4c1cf610807a 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -975,7 +975,7 @@ bool xen_set_default_idle(void);
 #define xen_set_default_idle 0
 #endif
 
-void stop_this_cpu(void *dummy);
+void stop_this_cpu(bool sync);
 void df_debug(struct pt_regs *regs, long error_code);
 void microcode_check(void);
 
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index 2243af6530f8..35d5cf73716e 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -590,9 +590,21 @@ bool xen_set_default_idle(void)
 }
 #endif
 
-void stop_this_cpu(void *dummy)
+atomic_t cpus_stop_in_ipi;
+void stop_this_cpu(bool sync)
 {
 	local_irq_disable();
+
+    /*
+     * Account this cpu and loop until the other cpus reached this
+     * point. If they don't react, the control cpu will raise an NMI.
+     */
+    if(sync) {
+             atomic_dec(&cpus_stop_in_ipi);
+             while (atomic_read(&cpus_stop_in_ipi))
+                   cpu_relax();
+    }
+
 	/*
 	 * Remove this CPU:
 	 */
diff --git a/arch/x86/kernel/reboot.c b/arch/x86/kernel/reboot.c
index 3f677832fc12..389643727e37 100644
--- a/arch/x86/kernel/reboot.c
+++ b/arch/x86/kernel/reboot.c
@@ -742,7 +742,7 @@ static void native_machine_halt(void)
 
 	tboot_shutdown(TB_SHUTDOWN_HALT);
 
-	stop_this_cpu(NULL);
+	stop_this_cpu(false);
 }
 
 static void native_machine_power_off(void)
diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
index f2a749586252..9dee65b96115 100644
--- a/arch/x86/kernel/smp.c
+++ b/arch/x86/kernel/smp.c
@@ -112,6 +112,7 @@
  *	about nothing of note with C stepping upwards.
  */
 
+extern atomic_t cpus_stop_in_ipi;
 static atomic_t stopping_cpu = ATOMIC_INIT(-1);
 static bool smp_no_nmi_ipi = false;
 
@@ -162,7 +163,7 @@ static int smp_stop_nmi_callback(unsigned int val, struct pt_regs *regs)
 		return NMI_HANDLED;
 
 	cpu_emergency_vmxoff();
-	stop_this_cpu(NULL);
+	stop_this_cpu(false);
 
 	return NMI_HANDLED;
 }
@@ -175,7 +176,7 @@ asmlinkage __visible void smp_reboot_interrupt(void)
 {
 	ipi_entering_ack_irq();
 	cpu_emergency_vmxoff();
-	stop_this_cpu(NULL);
+	stop_this_cpu(true);
 	irq_exit();
 }
 
@@ -192,7 +193,8 @@ static void native_stop_other_cpus(int wait)
 
 	if (reboot_force)
 		return;
-
+    
+    atomic_set(&cpus_stop_in_ipi, num_online_cpus() - 1);
 	/*
 	 * Use an own vector here because smp_call_function
 	 * does lots of things not suitable in a panic situation.
@@ -256,6 +258,7 @@ static void native_stop_other_cpus(int wait)
 	local_irq_save(flags);
 	disable_local_APIC();
 	mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
+    atomic_set(&cpus_stop_in_ipi, 0);
 	local_irq_restore(flags);
 }
 
-- 
2.25.1

Re: sched: Unexpected reschedule of offline CPU#2!

[Phil Auld]

Hi Thomas,

On Tue, Sep 03, 2024 at 05:27:58PM +0200 Thomas Gleixner wrote:
> On Tue, Jul 27 2021 at 10:46, Jan Kiszka wrote:
> 
> Picking up this dead thread again.

Necro-ing this again...

I keep getting occasional reports of this case. Unfortunately
though, I've never been able to reproduce it myself.

Did the below patch ever go anywhere?

It seems to be stable in my testing with the addition of
an "extern" in asm/cpu.h to get it to build.

> 
> > What is supposed to prevent the following in mainline:
> >
> > CPU 0                   CPU 1                      CPU 2
> >
> > native_stop_other_cpus                             <INTERRUPT>
> >   send_IPI_allbutself                              ...
> >                         <INTERRUPT>
> >                         sysvec_reboot
> >                           stop_this_cpu
> >                             set_cpu_online(false)
> >                                                    native_smp_send_reschedule(1)
> >                                                      if (cpu_is_offline(1)) ...
> 
> Nothing. And that's what probably happens if I read the stack trace
> correctly.
> 
> But we can be slightly smarter about this for the reboot IPI (the NMI
> case does not have that issue).
> 
> CPU 0                   CPU 1                      CPU 2
> 
> native_stop_other_cpus                             <INTERRUPT>
>    send_IPI_allbutself                              ...
>                          <IPI>
>                          sysvec_reboot
>                            wait_for_others();
>                                                     </INTERRUPT>
>                                                     <IPI>
>                                                     sysvec_reboot
>                                                     wait_for_others();
>                            stop_this_cpu();           stop_this_cpu();
>                              set_cpu_online(false);     set_cpu_online(false);
> 
> Something like the uncompiled below.
> 
> Thanks,
> 
>         tglx
> ---
> --- a/arch/x86/include/asm/cpu.h
> +++ b/arch/x86/include/asm/cpu.h
> @@ -68,5 +68,6 @@ bool intel_find_matching_signature(void
>  int intel_microcode_sanity_check(void *mc, bool print_err, int hdr_type);
>  
>  extern struct cpumask cpus_stop_mask;
> +atomic_t cpus_stop_in_ipi;

extern 

>  
>  #endif /* _ASM_X86_CPU_H */
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -721,7 +721,7 @@ bool xen_set_default_idle(void);
>  #define xen_set_default_idle 0
>  #endif
>  
> -void __noreturn stop_this_cpu(void *dummy);
> +void __noreturn stop_this_cpu(bool sync);
>  void microcode_check(struct cpuinfo_x86 *prev_info);
>  void store_cpu_caps(struct cpuinfo_x86 *info);
>  
> --- a/arch/x86/kernel/process.c
> +++ b/arch/x86/kernel/process.c
> @@ -791,9 +791,10 @@ bool xen_set_default_idle(void)
>  }
>  #endif
>  
> +atomic_t cpus_stop_in_ipi;
>  struct cpumask cpus_stop_mask;
>  
> -void __noreturn stop_this_cpu(void *dummy)
> +void __noreturn stop_this_cpu(bool sync)
>  {
>  	struct cpuinfo_x86 *c = this_cpu_ptr(&cpu_info);
>  	unsigned int cpu = smp_processor_id();
> @@ -801,6 +802,16 @@ void __noreturn stop_this_cpu(void *dumm
>  	local_irq_disable();
>  
>  	/*
> +	 * Account this CPU and loop until the other CPUs reached this
> +	 * point. If they don't react, the control CPU will raise an NMI.
> +	 */
> +	if (sync) {
> +		atomic_dec(&cpus_stop_in_ipi);
> +		while (atomic_read(&cpus_stop_in_ipi))
> +			cpu_relax();
> +	}
> +
> +	/*
>  	 * Remove this CPU from the online mask and disable it
>  	 * unconditionally. This might be redundant in case that the reboot
>  	 * vector was handled late and stop_other_cpus() sent an NMI.
> --- a/arch/x86/kernel/reboot.c
> +++ b/arch/x86/kernel/reboot.c
> @@ -788,7 +788,7 @@ static void native_machine_halt(void)
>  
>  	tboot_shutdown(TB_SHUTDOWN_HALT);
>  
> -	stop_this_cpu(NULL);
> +	stop_this_cpu(false);
>  }
>  
>  static void native_machine_power_off(void)
> --- a/arch/x86/kernel/smp.c
> +++ b/arch/x86/kernel/smp.c
> @@ -125,7 +125,7 @@ static int smp_stop_nmi_callback(unsigne
>  		return NMI_HANDLED;
>  
>  	cpu_emergency_disable_virtualization();
> -	stop_this_cpu(NULL);
> +	stop_this_cpu(false);
>  
>  	return NMI_HANDLED;
>  }
> @@ -137,7 +137,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_reboot)
>  {
>  	apic_eoi();
>  	cpu_emergency_disable_virtualization();
> -	stop_this_cpu(NULL);
> +	stop_this_cpu(true);
>  }
>  
>  static int register_stop_handler(void)
> @@ -189,6 +189,7 @@ static void native_stop_other_cpus(int w
>  	 */
>  	cpumask_copy(&cpus_stop_mask, cpu_online_mask);
>  	cpumask_clear_cpu(this_cpu, &cpus_stop_mask);
> +	atomic_set(&cpus_stop_in_ipi, num_online_cpus() - 1);
>  
>  	if (!cpumask_empty(&cpus_stop_mask)) {
>  		apic_send_IPI_allbutself(REBOOT_VECTOR);
> @@ -235,10 +236,12 @@ static void native_stop_other_cpus(int w
>  	local_irq_restore(flags);
>  
>  	/*
> -	 * Ensure that the cpus_stop_mask cache lines are invalidated on
> -	 * the other CPUs. See comment vs. SME in stop_this_cpu().
> +	 * Ensure that the cpus_stop_mask and cpus_stop_in_ipi cache lines
> +	 * are invalidated on the other CPUs. See comment vs. SME in
> +	 * stop_this_cpu().
>  	 */
>  	cpumask_clear(&cpus_stop_mask);
> +	atomic_set(&cpus_stop_in_ipi, 0);
>  }
>  
>  /*
> 


Thanks,
Phil
-- 

Re: sched: Unexpected reschedule of offline CPU#2!

[Thomas Gleixner]

On Wed, Jul 09 2025 at 09:44, Phil Auld wrote:
> Hi Thomas,
>
> On Tue, Sep 03, 2024 at 05:27:58PM +0200 Thomas Gleixner wrote:
>> On Tue, Jul 27 2021 at 10:46, Jan Kiszka wrote:
>> 
>> Picking up this dead thread again.
>
> Necro-ing this again...
>
> I keep getting occasional reports of this case. Unfortunately
> though, I've never been able to reproduce it myself.
>
> Did the below patch ever go anywhere?

Nope. Guocai said it does not work and I have no reproducer either to
actually look at it deeper and there was further debug data provided.

> It seems to be stable in my testing with the addition of
> an "extern" in asm/cpu.h to get it to build.

I don't see why it wouldn't be stable, but as it does not seem to solve
the issue merging it as is does not make sense.

Thanks,

        tglx

Re: sched: Unexpected reschedule of offline CPU#2!

[Thomas Gleixner]

On Sat, Jul 19 2025 at 23:17, Thomas Gleixner wrote:
> On Wed, Jul 09 2025 at 09:44, Phil Auld wrote:
>> Hi Thomas,
>>
>> On Tue, Sep 03, 2024 at 05:27:58PM +0200 Thomas Gleixner wrote:
>>> On Tue, Jul 27 2021 at 10:46, Jan Kiszka wrote:
>>> 
>>> Picking up this dead thread again.
>>
>> Necro-ing this again...
>>
>> I keep getting occasional reports of this case. Unfortunately
>> though, I've never been able to reproduce it myself.
>>
>> Did the below patch ever go anywhere?
>
> Nope. Guocai said it does not work and I have no reproducer either to
> actually look at it deeper and there was further debug data provided.

Obviously:

        no further debug data was provided, so I can only tap in the
        dark.

Re: sched: Unexpected reschedule of offline CPU#2!

[Guenter Roeck]

On 7/20/25 03:47, Thomas Gleixner wrote:
> On Sat, Jul 19 2025 at 23:17, Thomas Gleixner wrote:
>> On Wed, Jul 09 2025 at 09:44, Phil Auld wrote:
>>> Hi Thomas,
>>>
>>> On Tue, Sep 03, 2024 at 05:27:58PM +0200 Thomas Gleixner wrote:
>>>> On Tue, Jul 27 2021 at 10:46, Jan Kiszka wrote:
>>>>
>>>> Picking up this dead thread again.
>>>
>>> Necro-ing this again...
>>>
>>> I keep getting occasional reports of this case. Unfortunately
>>> though, I've never been able to reproduce it myself.
>>>
>>> Did the below patch ever go anywhere?
>>
>> Nope. Guocai said it does not work and I have no reproducer either to
>> actually look at it deeper and there was further debug data provided.
> 
> Obviously:
> 
>          no further debug data was provided, so I can only tap in the
>          dark.
> 

FWIW, I have not seen this problem for a long time. I know it existed in
5.10, and I can find logs showing the problem in my company's bug system,
but only for 5.10 and older kernels. I don't see it in my test system either,
not even for 5.4.y or 5.10.y kernels.

Guenter

Re: sched: Unexpected reschedule of offline CPU#2!

[Phil Auld]

On Sun, Jul 20, 2025 at 12:47:24PM +0200 Thomas Gleixner wrote:
> On Sat, Jul 19 2025 at 23:17, Thomas Gleixner wrote:
> > On Wed, Jul 09 2025 at 09:44, Phil Auld wrote:
> >> Hi Thomas,
> >>
> >> On Tue, Sep 03, 2024 at 05:27:58PM +0200 Thomas Gleixner wrote:
> >>> On Tue, Jul 27 2021 at 10:46, Jan Kiszka wrote:
> >>> 
> >>> Picking up this dead thread again.
> >>
> >> Necro-ing this again...
> >>
> >> I keep getting occasional reports of this case. Unfortunately
> >> though, I've never been able to reproduce it myself.
> >>
> >> Did the below patch ever go anywhere?
> >
> > Nope. Guocai said it does not work and I have no reproducer either to
> > actually look at it deeper and there was further debug data provided.
> 
> Obviously:
> 
>         no further debug data was provided, so I can only tap in the
>         dark.
>

Fair enough, thanks. I did not see the "does not work" part :)


Cheers,
Phil


--