[PATCH] arm64: smp: distinguish secondary CPUs that hang after reaching head.S

[PATCH] arm64: smp: distinguish secondary CPUs that hang after reaching head.S

[Naman Jain]

When a secondary CPU fails to come online, __cpu_up() falls back to
__early_cpu_boot_status, but boot status 0x0 is ambiguous: it cannot
distinguish a CPU that never executed head.S (firmware/hypervisor never
dispatched it, so it never ran a single instruction) from one that
entered head.S, started executing, and then got stuck somewhere in kernel
bring-up. Add a change to let us tell those two cases apart, which
narrows down where to look when a CPU goes missing during boot.

Write a sentinel (CPU_BOOT_ASM_ENTERED, 0x10) as the first thing in
secondary_entry so the two cases can be told apart:

  0x0  - CPU never ran head.S
  0x10 - CPU entered head.S but did not come online

__cpu_up() resets the word before releasing each secondary, so a stale
value is not misattributed. Nothing clears the marker on the success
path. Reason being, the CPU instead sets CPU_BOOT_SUCCESS in
secondary_data.status, which __cpu_up() checks first, so
__early_cpu_boot_status is only read on failure. A stall before
CPU_BOOT_SUCCESS - including in secondary_start_kernel() - is therefore
reported as 0x10. The sentinel sits in the low byte and sets no
CPU_STUCK_REASON_* bits, so existing status decoding is unchanged.

Limitations: best-effort only. The status variable is global with no CPU
identity, and spin-table secondaries boot via secondary_holding_pen and
never write it, so 0x0 is a strong hint rather than a guarantee.

Signed-off-by: Naman Jain <namjain@linux.microsoft.com>
---
 arch/arm64/include/asm/smp.h | 10 ++++++++++
 arch/arm64/kernel/head.S     | 24 +++++++++++++++++++++++-
 arch/arm64/kernel/smp.c      | 21 +++++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index 10ea4f5430690..ca330cd1b0615 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -20,6 +20,16 @@
 /* Fatal system error detected by secondary CPU, crash the system */
 #define CPU_PANIC_KERNEL		(3)
 
+/*
+ * Best-effort marker written early in secondary_entry (PSCI/hotplug
+ * path) so __cpu_up() can tell whether a stuck CPU ever ran head.S.
+ * Not authoritative: spin-table secondaries use secondary_holding_pen
+ * and never write it, and the word has no CPU identity. Value must sit
+ * in the low byte, stay out of {0,1,2,3}, and set no CPU_STUCK_REASON_*
+ * bits.
+ */
+#define CPU_BOOT_ASM_ENTERED		(0x10)
+
 #define CPU_STUCK_REASON_52_BIT_VA	(UL(1) << CPU_STUCK_REASON_SHIFT)
 #define CPU_STUCK_REASON_NO_GRAN	(UL(2) << CPU_STUCK_REASON_SHIFT)
 
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 87a822e5c4ca8..8ac3c49b4daf7 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -353,6 +353,22 @@ SYM_FUNC_END(secondary_holding_pen)
 	 * be used where CPUs are brought online dynamically by the kernel.
 	 */
 SYM_FUNC_START(secondary_entry)
+	/*
+	 * Record that this CPU reached head.S, before init_kernel_el()
+	 * sets SCTLR_ELx.EE. A single-byte store to the reader's low
+	 * byte (__cpu_up() zeroed the word) is endian-agnostic, so a
+	 * big-endian kernel still reads 0x10. Flush it like
+	 * update_early_cpu_boot_status.
+	 */
+	adr_l	x2, __early_cpu_boot_status
+	mov	w1, #CPU_BOOT_ASM_ENTERED
+#ifdef CONFIG_CPU_BIG_ENDIAN
+	strb	w1, [x2, #7]	// low byte at top address
+#else
+	strb	w1, [x2]
+#endif
+	dmb	sy
+	dc	ivac, x2
 	mov	x0, xzr
 	bl	init_kernel_el			// w0=cpu_boot_mode
 	b	secondary_startup
@@ -386,7 +402,6 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
 	mov	x0, x20
 	bl	finalise_el2
 
-	str_l	xzr, __early_cpu_boot_status, x3
 	adr_l	x5, vectors
 	msr	vbar_el1, x5
 	isb
@@ -395,6 +410,13 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
 	ldr	x2, [x0, #CPU_BOOT_TASK]
 	cbz	x2, __secondary_too_slow
 
+	/*
+	 * Don't clear the marker here; nothing on the secondary clears
+	 * it. Once the CPU sets CPU_BOOT_SUCCESS in
+	 * secondary_data.status, __cpu_up() stops reading
+	 * __early_cpu_boot_status (and resets it before the next
+	 * release), so a stall before that still reports 0x10.
+	 */
 	init_cpu_task x2, x1, x3
 
 #ifdef CONFIG_ARM64_PTR_AUTH
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index cdcdd160e5b69..662205ae5a7a6 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -120,6 +120,17 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
 	secondary_data.task = idle;
 	update_cpu_boot_status(CPU_MMU_OFF);
 
+	/*
+	 * Reset the marker before releasing this secondary: head.S
+	 * never clears it, so without this a CPU that never reaches
+	 * head.S would inherit a stale 0x10 from a previous CPU. Flush
+	 * it so the MMU-off secondary sees the reset.
+	 */
+	WRITE_ONCE(__early_cpu_boot_status, 0);
+	dcache_clean_inval_poc((unsigned long)&__early_cpu_boot_status,
+			       (unsigned long)&__early_cpu_boot_status +
+			       sizeof(__early_cpu_boot_status));
+
 	/* Now bring the CPU into our world */
 	ret = boot_secondary(cpu, idle);
 	if (ret) {
@@ -145,10 +156,20 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
 
 	switch (status & CPU_BOOT_STATUS_MASK) {
 	default:
+		/*
+		 * 0x0 usually means the CPU never ran head.S, but
+		 * treat it as a hint: spin-table secondaries never
+		 * write the marker and the word has no CPU identity.
+		 */
 		pr_err("CPU%u: failed in unknown state : 0x%lx\n",
 		       cpu, status);
 		cpus_stuck_in_kernel++;
 		break;
+	case CPU_BOOT_ASM_ENTERED:
+		pr_err("CPU%u: entered head.S but not online : 0x%lx\n",
+		       cpu, status);
+		cpus_stuck_in_kernel++;
+		break;
 	case CPU_KILL_ME:
 		if (!op_cpu_kill(cpu)) {
 			pr_crit("CPU%u: died during early boot\n", cpu);

base-commit: 290aaf24a551d5a0dce037e3fab30820f9113a10
-- 
2.43.0

Re: [PATCH] arm64: smp: distinguish secondary CPUs that hang after reaching head.S

[Jinjie Ruan]

在 2026/7/22 19:30, Naman Jain 写道:
> When a secondary CPU fails to come online, __cpu_up() falls back to
> __early_cpu_boot_status, but boot status 0x0 is ambiguous: it cannot
> distinguish a CPU that never executed head.S (firmware/hypervisor never
> dispatched it, so it never ran a single instruction) from one that
> entered head.S, started executing, and then got stuck somewhere in kernel
> bring-up. Add a change to let us tell those two cases apart, which
> narrows down where to look when a CPU goes missing during boot.

I previously encountered this issue when debugging the parallel startup
of ARM64 secondary cores. It is difficult for the kernel to determine
whether the secondary core is hung in the firmware or whether it has not
executed a single instruction. So I think this motive is reasonable.

> 
> Write a sentinel (CPU_BOOT_ASM_ENTERED, 0x10) as the first thing in
> secondary_entry so the two cases can be told apart:
> 
>   0x0  - CPU never ran head.S
>   0x10 - CPU entered head.S but did not come online
> 
> __cpu_up() resets the word before releasing each secondary, so a stale
> value is not misattributed. Nothing clears the marker on the success
> path. Reason being, the CPU instead sets CPU_BOOT_SUCCESS in
> secondary_data.status, which __cpu_up() checks first, so
> __early_cpu_boot_status is only read on failure. A stall before
> CPU_BOOT_SUCCESS - including in secondary_start_kernel() - is therefore
> reported as 0x10. The sentinel sits in the low byte and sets no
> CPU_STUCK_REASON_* bits, so existing status decoding is unchanged.
> 
> Limitations: best-effort only. The status variable is global with no CPU
> identity, and spin-table secondaries boot via secondary_holding_pen and
> never write it, so 0x0 is a strong hint rather than a guarantee.
> 
> Signed-off-by: Naman Jain <namjain@linux.microsoft.com>
> ---
>  arch/arm64/include/asm/smp.h | 10 ++++++++++
>  arch/arm64/kernel/head.S     | 24 +++++++++++++++++++++++-
>  arch/arm64/kernel/smp.c      | 21 +++++++++++++++++++++
>  3 files changed, 54 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
> index 10ea4f5430690..ca330cd1b0615 100644
> --- a/arch/arm64/include/asm/smp.h
> +++ b/arch/arm64/include/asm/smp.h
> @@ -20,6 +20,16 @@
>  /* Fatal system error detected by secondary CPU, crash the system */
>  #define CPU_PANIC_KERNEL		(3)
>  
> +/*
> + * Best-effort marker written early in secondary_entry (PSCI/hotplug
> + * path) so __cpu_up() can tell whether a stuck CPU ever ran head.S.
> + * Not authoritative: spin-table secondaries use secondary_holding_pen
> + * and never write it, and the word has no CPU identity. Value must sit
> + * in the low byte, stay out of {0,1,2,3}, and set no CPU_STUCK_REASON_*
> + * bits.
> + */
> +#define CPU_BOOT_ASM_ENTERED		(0x10)
> +
>  #define CPU_STUCK_REASON_52_BIT_VA	(UL(1) << CPU_STUCK_REASON_SHIFT)
>  #define CPU_STUCK_REASON_NO_GRAN	(UL(2) << CPU_STUCK_REASON_SHIFT)
>  
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index 87a822e5c4ca8..8ac3c49b4daf7 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -353,6 +353,22 @@ SYM_FUNC_END(secondary_holding_pen)
>  	 * be used where CPUs are brought online dynamically by the kernel.
>  	 */
>  SYM_FUNC_START(secondary_entry)
> +	/*
> +	 * Record that this CPU reached head.S, before init_kernel_el()
> +	 * sets SCTLR_ELx.EE. A single-byte store to the reader's low
> +	 * byte (__cpu_up() zeroed the word) is endian-agnostic, so a
> +	 * big-endian kernel still reads 0x10. Flush it like
> +	 * update_early_cpu_boot_status.
> +	 */
> +	adr_l	x2, __early_cpu_boot_status
> +	mov	w1, #CPU_BOOT_ASM_ENTERED
> +#ifdef CONFIG_CPU_BIG_ENDIAN
> +	strb	w1, [x2, #7]	// low byte at top address
> +#else
> +	strb	w1, [x2]
> +#endif
> +	dmb	sy
> +	dc	ivac, x2
>  	mov	x0, xzr
>  	bl	init_kernel_el			// w0=cpu_boot_mode
>  	b	secondary_startup
> @@ -386,7 +402,6 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
>  	mov	x0, x20
>  	bl	finalise_el2
>  
> -	str_l	xzr, __early_cpu_boot_status, x3
>  	adr_l	x5, vectors
>  	msr	vbar_el1, x5
>  	isb
> @@ -395,6 +410,13 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
>  	ldr	x2, [x0, #CPU_BOOT_TASK]
>  	cbz	x2, __secondary_too_slow
>  
> +	/*
> +	 * Don't clear the marker here; nothing on the secondary clears
> +	 * it. Once the CPU sets CPU_BOOT_SUCCESS in
> +	 * secondary_data.status, __cpu_up() stops reading
> +	 * __early_cpu_boot_status (and resets it before the next
> +	 * release), so a stall before that still reports 0x10.
> +	 */
>  	init_cpu_task x2, x1, x3
>  
>  #ifdef CONFIG_ARM64_PTR_AUTH
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index cdcdd160e5b69..662205ae5a7a6 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -120,6 +120,17 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
>  	secondary_data.task = idle;
>  	update_cpu_boot_status(CPU_MMU_OFF);
>  
> +	/*
> +	 * Reset the marker before releasing this secondary: head.S
> +	 * never clears it, so without this a CPU that never reaches
> +	 * head.S would inherit a stale 0x10 from a previous CPU. Flush
> +	 * it so the MMU-off secondary sees the reset.
> +	 */
> +	WRITE_ONCE(__early_cpu_boot_status, 0);
> +	dcache_clean_inval_poc((unsigned long)&__early_cpu_boot_status,
> +			       (unsigned long)&__early_cpu_boot_status +
> +			       sizeof(__early_cpu_boot_status));
> +
>  	/* Now bring the CPU into our world */
>  	ret = boot_secondary(cpu, idle);
>  	if (ret) {
> @@ -145,10 +156,20 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
>  
>  	switch (status & CPU_BOOT_STATUS_MASK) {
>  	default:
> +		/*
> +		 * 0x0 usually means the CPU never ran head.S, but
> +		 * treat it as a hint: spin-table secondaries never
> +		 * write the marker and the word has no CPU identity.
> +		 */
>  		pr_err("CPU%u: failed in unknown state : 0x%lx\n",
>  		       cpu, status);
>  		cpus_stuck_in_kernel++;
>  		break;
> +	case CPU_BOOT_ASM_ENTERED:
> +		pr_err("CPU%u: entered head.S but not online : 0x%lx\n",
> +		       cpu, status);
> +		cpus_stuck_in_kernel++;
> +		break;
>  	case CPU_KILL_ME:
>  		if (!op_cpu_kill(cpu)) {
>  			pr_crit("CPU%u: died during early boot\n", cpu);
> 
> base-commit: 290aaf24a551d5a0dce037e3fab30820f9113a10

Re: [PATCH] arm64: smp: distinguish secondary CPUs that hang after reaching head.S

[Anshuman Khandual]

On 23/07/26 8:19 AM, Jinjie Ruan wrote:
> 
> 在 2026/7/22 19:30, Naman Jain 写道:
>> When a secondary CPU fails to come online, __cpu_up() falls back to
>> __early_cpu_boot_status, but boot status 0x0 is ambiguous: it cannot
>> distinguish a CPU that never executed head.S (firmware/hypervisor never
>> dispatched it, so it never ran a single instruction) from one that
>> entered head.S, started executing, and then got stuck somewhere in kernel
>> bring-up. Add a change to let us tell those two cases apart, which
>> narrows down where to look when a CPU goes missing during boot.
> I previously encountered this issue when debugging the parallel startup
> of ARM64 secondary cores. It is difficult for the kernel to determine
> whether the secondary core is hung in the firmware or whether it has not
> executed a single instruction. So I think this motive is reasonable.

Why should kernel determine the difference here ? Would not the firmware
know if it has started any secondary CPU for the kernel which must have
come inside head.S ? If the cpu gets hung inside firmware while starting
up then the debug responsibilities belong there instead.

Still wondering what's the rationale for this change.

Re: [PATCH] arm64: smp: distinguish secondary CPUs that hang after reaching head.S

[Naman Jain]

On 7/23/2026 11:26 AM, Anshuman Khandual wrote:
> 
> 
> On 23/07/26 8:19 AM, Jinjie Ruan wrote:
>>
>> 在 2026/7/22 19:30, Naman Jain 写道:
>>> When a secondary CPU fails to come online, __cpu_up() falls back to
>>> __early_cpu_boot_status, but boot status 0x0 is ambiguous: it cannot
>>> distinguish a CPU that never executed head.S (firmware/hypervisor never
>>> dispatched it, so it never ran a single instruction) from one that
>>> entered head.S, started executing, and then got stuck somewhere in kernel
>>> bring-up. Add a change to let us tell those two cases apart, which
>>> narrows down where to look when a CPU goes missing during boot.
>> I previously encountered this issue when debugging the parallel startup
>> of ARM64 secondary cores. It is difficult for the kernel to determine
>> whether the secondary core is hung in the firmware or whether it has not
>> executed a single instruction. So I think this motive is reasonable.
> 
> Why should kernel determine the difference here ? Would not the firmware
> know if it has started any secondary CPU for the kernel which must have
> come inside head.S ? If the cpu gets hung inside firmware while starting
> up then the debug responsibilities belong there instead.
> 
> Still wondering what's the rationale for this change.

Hello Anshuman,
This sounds fair to me. Let me elaborate the problem, beyond the scope 
of this patch. In production, we occasionally see these crashes where 
one of the CPU fails to bring up online, with 0x0 status code. 
Hypervisor may be missing the telemetry, but the problem is that we 
don't know if the secondary CPU ever started executing the instructions 
or is stuck somewhere between the start of head.S and marking itself 
online at the end of secondary_start_kernel().
There are couple of places, where we get those other status codes, but 
not everywhere. If the issue is not easily reproducible, experiments on 
local setups do not yield anything. That's where I am attempting to add 
some more information in kernel to debug these issues.

Regards,
Naman

Re: [PATCH] arm64: smp: distinguish secondary CPUs that hang after reaching head.S

[Anshuman Khandual]

On 24/07/26 10:16 AM, Naman Jain wrote:
> 
> 
> On 7/23/2026 11:26 AM, Anshuman Khandual wrote:
>>
>>
>> On 23/07/26 8:19 AM, Jinjie Ruan wrote:
>>>
>>> 在 2026/7/22 19:30, Naman Jain 写道:
>>>> When a secondary CPU fails to come online, __cpu_up() falls back to
>>>> __early_cpu_boot_status, but boot status 0x0 is ambiguous: it cannot
>>>> distinguish a CPU that never executed head.S (firmware/hypervisor never
>>>> dispatched it, so it never ran a single instruction) from one that
>>>> entered head.S, started executing, and then got stuck somewhere in kernel
>>>> bring-up. Add a change to let us tell those two cases apart, which
>>>> narrows down where to look when a CPU goes missing during boot.
>>> I previously encountered this issue when debugging the parallel startup
>>> of ARM64 secondary cores. It is difficult for the kernel to determine
>>> whether the secondary core is hung in the firmware or whether it has not
>>> executed a single instruction. So I think this motive is reasonable.
>>
>> Why should kernel determine the difference here ? Would not the firmware
>> know if it has started any secondary CPU for the kernel which must have
>> come inside head.S ? If the cpu gets hung inside firmware while starting
>> up then the debug responsibilities belong there instead.
>>
>> Still wondering what's the rationale for this change.
> 
> Hello Anshuman,
> This sounds fair to me. Let me elaborate the problem, beyond the scope of this patch. In production, we occasionally see these crashes where one of the CPU fails to bring up online, with 0x0 status code. Hypervisor may be missing the telemetry, but the problem is that we don't know if the secondary CPU ever started executing the instructions or is stuck somewhere between the start of head.S and marking itself online at the end of secondary_start_kernel().

Why that information is useful ? If firmware is sure to have dispatched
given CPU then should not the early kernel boot failure be debugged via
generally available methods. Is this trying create an alternative ?
> There are couple of places, where we get those other status codes, but not everywhere. If the issue is not easily reproducible, experiments on local setups do not yield anything. That's where I am attempting to add some more information in kernel to debug these issues.
But wondering if the intent is to further classify failures including
entered ASM, but failed else where during boot for ease in debugging
why not add more into CPU_STUCK_REASON_* ?