[PATCH v3 00/12] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs

[PATCH v3 00/12] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs

[Jinjie Ruan]

Support for parallel secondary CPU bringup is already utilized by x86,
MIPS, and RISC-V. This patch brings this capability to the arm64
architecture.

Introduce CONFIG_HOTPLUG_PARALLEL_SMT to avoid primary SMT threads
to boot first constraint.

And add a 'cpu' parameter to update_cpu_boot_status() to allow updating
the boot status at a per-CPU granularity during parallel bringup.

Rework the global `secondary_data` and `__early_cpu_boot_status` accessed
during early boot into per-CPU arrays to allow secondary CPUs to boot
in parallel.

And reuse `__cpu_logical_map` array in the early boot code in head.S
to resolve each secondary CPU's logical ID concurrently.

This series includes a subset of the refactoring patches proposed
by Will Deacon, with further adjustments.

Link: https://web.git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=cpu-hotplug

Bringup Time Comparison on real hardware(ms, lower is better):

 |     Platform                     | Baseline|   P=0   |   P=1  | Delta(%)|
 | -------------------------------- | ------- | ------- | ------ | ------- |
 | 192-core server(HIP12)           | 14619.2 | 14619.1 | 8589.4 | 41.21%  |
 | 32-core board                    | 2776.5  | 2881.0  | 1045.0 | 62.36%  |
 | 64-core board                    | 2297.0  |    /    | 814.4  | 64.5%   |

Below is the actual dmesg output demonstrating four concurrent boot
failures on different CPUs:

	CPU4 failed to report alive state
	CPU4: is stuck in kernel
	CPU4: does not support 52-bit VAs

	CPU6 failed to report alive state
	CPU6: is stuck in kernel
	CPU6: does not support 4K granule

	GICv3: CPU8: found redistributor 8 region 0:0x00000000081a0000
	GICv3: CPU8: using allocated LPI pending table @0x0000000100360000
	CPU8: Booted secondary processor 0x0000000008 [0x410fd034]
	...
	CPU16 failed to report alive state
	psci: CPU16 killed (polled 0 ms)
	CPU16: died during early boot

	CPU17: will not boot
	CPU17 failed to report alive state
	psci: CPU17 killed (polled 0 ms)
	CPU17: died during early boot

	CPU18 failed to report alive state
	Kernel panic - not syncing: CPU18 detected unsupported configuration
	CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 7.1.0-rc1-gdd2d3bbca3b5 #151 PREEMPT
	Hardware name: linux,dummy-virt (DT)
	Call trace:
	 show_stack+0x18/0x24 (C)
	 dump_stack_lvl+0x38/0xd0
	 dump_stack+0x18/0x24
	 vpanic+0x4f8/0x4fc
	 do_panic_on_target_cpu+0x0/0x1c
	 secondary_start_kernel+0x0/0x17c
	 cpuhp_bringup_ap+0x2cc/0x2dc
	 cpuhp_invoke_callback+0x168/0x2ac
	 __cpuhp_invoke_callback_range+0x90/0x118
	 _cpu_up+0x148/0x220
	 cpu_up+0xcc/0x158
	 cpuhp_bringup_mask.constprop.0+0x80/0xcc
	 bringup_nonboot_cpus+0x38/0x80
	 smp_init+0x30/0x8c
	 kernel_init_freeable+0x170/0x35c
	 kernel_init+0x24/0x1e0
	 ret_from_fork+0x10/0x20
	SMP: stopping secondary CPUs
	---[ end Kernel panic - not syncing: CPU18 detected unsupported configuration ]---

Changes in v3:
- Add necessary rework patches.
- Fix AI review issues in [2].
  1. Use lockdep_off() and lockdep_on to resolve the lockdep splat
     on failure paths, which avoid printk_deferred() and solve
     the pr_fmt() prefix issue or missing update.
  2. Ensure atomic updates to system_cpucaps bitmap for arm64 cpufeature.
  3. Use NR_CPUS __early_cpu_boot_status array in head.S and not clear
     __early_cpu_boot_status.
  4. Solve get_cpu_ops(0) which evaluates to the boot CPU issue in
    arch_cpuhp_init_parallel_bringup() with Will's rework patch.
- Handle `__early_cpu_boot_status` properly as Will pointed out with
  Will's patch.
- Implement arch_cpuhp_cleanup_kick_cpu() to cleanup for fail boot
  secondary CPUS and add support for error handling with Will's patch.
- Update the code as Thomas suggested. Rename PARALLEL_SMT_PRIMARY_FIRST
  to `HOTPLUG_PARALLEL_SMT` and not select it for RISC-V.
- Select HOTPLUG_PARALLEL if SMP as Thomas suggested.
- Rework early boot data into per-CPU arrays directly rather than using
  CONFIG_HOTPLUG_PARALLEL to differentiate code paths as Thomas suggested.
- Remove `cpu_running` and related complete.
- Add new test data on new hardware.

[2]: https://sashiko.dev/#/patchset/20260618092444.1316336-1-ruanjinjie%40huawei.com

Changes in v2:
- Remove RFC.
- Add Tested-by.
- Fix AI review issues in [1].
- Add arch_cpuhp_init_parallel_bringup() to check psci boot.
- Reuse `__cpu_logical_map` instead of a new aray.
- Defer rcutree_report_cpu_starting() until after
  check_local_cpu_capabilities() to prevent a potential control CPU
  deadlock if an early capability check fails.
- Move the assembly in head.S to a macro called `mpidr_to_cpuid`.
- Add `SECONDARY_DATA_SHIFT` for `lsl` to access `cpu_boot_data`.
- Add sizeof(struct secondary_data) power of 2 assert check.
- Expand testing with more data collected from real hardware.

[1] https://sashiko.dev/#/patchset/20260611133809.3854977-1-ruanjinjie%40huawei.com

Jinjie Ruan (5):
  cpu/hotplug: Introduce CONFIG_HOTPLUG_PARALLEL_SMT
  arm64: cpufeature: Ensure atomic updates to system_cpucaps bitmap
  arm64: smp: Pass CPU ID to update_cpu_boot_status()
  arm64: smp: Rework early boot data into per-CPU arrays
  arm64: Add HOTPLUG_PARALLEL support for secondary CPUs

Will Deacon (7):
  cpu/hotplug: Propagate bring-up status to
    arch_cpuhp_cleanup_kick_cpu()
  arm64: smp: Tidy up smp_prepare_cpus()
  arm64: smp: Tidy up cpuinfo init and cpufeature updates
  arm64: smp: Defer RCU registration during secondary CPU bringup
  arm64: smp: Use generic HOTPLUG_SPLIT_STARTUP machinery for CPU
    onlining
  arm64: cpu_ops: Make 'cpu_operations' pointer global instead of
    per-cpu
  arm64: cpu_ops: Introduce get_secondary_cpu_ops()

 arch/Kconfig                     |   5 ++
 arch/arm64/Kconfig               |   2 +-
 arch/arm64/include/asm/cpu.h     |   6 +-
 arch/arm64/include/asm/cpu_ops.h |   1 +
 arch/arm64/include/asm/smp.h     |  22 +++---
 arch/arm64/kernel/asm-offsets.c  |   2 +
 arch/arm64/kernel/cpu_ops.c      |  34 ++++++---
 arch/arm64/kernel/cpufeature.c   |  27 +++++--
 arch/arm64/kernel/cpuinfo.c      |  11 ---
 arch/arm64/kernel/head.S         |  44 ++++++++++--
 arch/arm64/kernel/setup.c        |   9 +++
 arch/arm64/kernel/smp.c          | 118 +++++++++++++++++--------------
 arch/arm64/mm/context.c          |   5 +-
 arch/arm64/mm/mmu.c              |   2 +-
 arch/mips/Kconfig                |   3 +-
 arch/x86/Kconfig                 |   2 +-
 arch/x86/kernel/smpboot.c        |   4 +-
 include/linux/cpuhotplug.h       |   2 +-
 kernel/cpu.c                     |   8 +--
 19 files changed, 189 insertions(+), 118 deletions(-)

-- 
2.34.1

[PATCH v3 01/12] cpu/hotplug: Introduce CONFIG_HOTPLUG_PARALLEL_SMT

[Jinjie Ruan]

During parallel CPU bringup, x86 requires primary SMT threads to boot
first to avoid siblings stopping during microcode updates. This constraint
is architecture-specific and unnecessary for other platforms
like arm64.

Introduce CONFIG_HOTPLUG_PARALLEL_SMT to decouple this constraint.
Platforms requiring this temporal order (e.g., x86) can select it
in Kconfig. Other architectures (e.g., arm64) can leave it unselected
to entirely bypass the SMT branch.

Suggested-by: Thomas Gleixner <tglx@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/Kconfig      | 5 +++++
 arch/mips/Kconfig | 3 +--
 arch/x86/Kconfig  | 2 +-
 kernel/cpu.c      | 4 ++--
 4 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/arch/Kconfig b/arch/Kconfig
index e86880045158..d25b61dc03b2 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -102,6 +102,11 @@ config HOTPLUG_PARALLEL
 	bool
 	select HOTPLUG_SPLIT_STARTUP
 
+config HOTPLUG_PARALLEL_SMT
+	bool
+	select HOTPLUG_PARALLEL
+	select HOTPLUG_SMT
+
 config GENERIC_IRQ_ENTRY
 	bool
 
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 4364f3dba688..8d9c57f3df23 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -660,7 +660,7 @@ config EYEQ
 	select USB_UHCI_BIG_ENDIAN_DESC if CPU_BIG_ENDIAN
 	select USB_UHCI_BIG_ENDIAN_MMIO if CPU_BIG_ENDIAN
 	select USE_OF
-	select HOTPLUG_PARALLEL if HOTPLUG_CPU
+	select HOTPLUG_PARALLEL_SMT if HOTPLUG_CPU
 	help
 	  Select this to build a kernel supporting EyeQ SoC from Mobileye.
 
@@ -2301,7 +2301,6 @@ config MIPS_CPS
 	select MIPS_CM
 	select MIPS_CPS_PM if HOTPLUG_CPU
 	select SMP
-	select HOTPLUG_SMT if HOTPLUG_PARALLEL
 	select HOTPLUG_CORE_SYNC_DEAD if HOTPLUG_CPU
 	select SYNC_R4K if (CEVT_R4K || CSRC_R4K)
 	select SYS_SUPPORTS_HOTPLUG_CPU
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index f3f7cb01d69d..2ea80da1e4f8 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -305,7 +305,7 @@ config X86
 	select HAVE_USER_RETURN_NOTIFIER
 	select HAVE_GENERIC_VDSO
 	select VDSO_GETRANDOM			if X86_64
-	select HOTPLUG_PARALLEL			if SMP && X86_64
+	select HOTPLUG_PARALLEL_SMT		if SMP && X86_64
 	select HOTPLUG_SMT			if SMP
 	select HOTPLUG_SPLIT_STARTUP		if SMP && X86_32
 	select IRQ_FORCED_THREADING
diff --git a/kernel/cpu.c b/kernel/cpu.c
index bc4f7a9ba64e..5a90f60ff60e 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1792,7 +1792,7 @@ static int __init parallel_bringup_parse_param(char *arg)
 }
 early_param("cpuhp.parallel", parallel_bringup_parse_param);
 
-#ifdef CONFIG_HOTPLUG_SMT
+#ifdef CONFIG_HOTPLUG_PARALLEL_SMT
 static inline bool cpuhp_smt_aware(void)
 {
 	return cpu_smt_max_threads > 1;
@@ -1811,7 +1811,7 @@ static inline const struct cpumask *cpuhp_get_primary_thread_mask(void)
 {
 	return cpu_none_mask;
 }
-#endif
+#endif /* CONFIG_HOTPLUG_PARALLEL_SMT */
 
 bool __weak arch_cpuhp_init_parallel_bringup(void)
 {
-- 
2.34.1

[PATCH v3 02/12] cpu/hotplug: Propagate bring-up status to arch_cpuhp_cleanup_kick_cpu()

[Jinjie Ruan]

From: Will Deacon <will@kernel.org>

In preparation for enabling the generic CPU hotplug machinery on arm64,
which has architecture-specific handling of early bringup failures,
extend arch_cpuhp_cleanup_kick_cpu() to take an additional argument
indicating whether or not the target AP reached the alive state.

Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/x86/kernel/smpboot.c  | 4 ++--
 include/linux/cpuhotplug.h | 2 +-
 kernel/cpu.c               | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 294a8ea60298..637660b15aee 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1057,7 +1057,7 @@ static int do_boot_cpu(u32 apicid, unsigned int cpu, struct task_struct *idle)
 
 	/* If the wakeup mechanism failed, cleanup the warm reset vector */
 	if (ret)
-		arch_cpuhp_cleanup_kick_cpu(cpu);
+		arch_cpuhp_cleanup_kick_cpu(cpu, false);
 	return ret;
 }
 
@@ -1105,7 +1105,7 @@ int arch_cpuhp_kick_ap_alive(unsigned int cpu, struct task_struct *tidle)
 	return smp_ops.kick_ap_alive(cpu, tidle);
 }
 
-void arch_cpuhp_cleanup_kick_cpu(unsigned int cpu)
+void arch_cpuhp_cleanup_kick_cpu(unsigned int cpu, bool is_alive)
 {
 	/* Cleanup possible dangling ends... */
 	if (smp_ops.kick_ap_alive == native_kick_ap && x86_platform.legacy.warm_reset)
diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
index 22ba327ec227..5c3b3e0bce47 100644
--- a/include/linux/cpuhotplug.h
+++ b/include/linux/cpuhotplug.h
@@ -511,7 +511,7 @@ struct task_struct;
 
 void cpuhp_ap_sync_alive(void);
 void arch_cpuhp_sync_state_poll(void);
-void arch_cpuhp_cleanup_kick_cpu(unsigned int cpu);
+void arch_cpuhp_cleanup_kick_cpu(unsigned int cpu, bool is_alive);
 int arch_cpuhp_kick_ap_alive(unsigned int cpu, struct task_struct *tidle);
 bool arch_cpuhp_init_parallel_bringup(void);
 
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 5a90f60ff60e..b0e31e624623 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -427,7 +427,7 @@ static bool cpuhp_can_boot_ap(unsigned int cpu)
 	return true;
 }
 
-void __weak arch_cpuhp_cleanup_kick_cpu(unsigned int cpu) { }
+void __weak arch_cpuhp_cleanup_kick_cpu(unsigned int cpu, bool is_alive) { }
 
 /*
  * Early CPU bringup synchronization point. Cannot use cpuhp_state::done_up
@@ -446,7 +446,7 @@ static int cpuhp_bp_sync_alive(unsigned int cpu)
 	}
 
 	/* Let the architecture cleanup the kick alive mechanics. */
-	arch_cpuhp_cleanup_kick_cpu(cpu);
+	arch_cpuhp_cleanup_kick_cpu(cpu, !ret);
 	return ret;
 }
 #else /* CONFIG_HOTPLUG_CORE_SYNC_FULL */
-- 
2.34.1

[PATCH v3 03/12] arm64: smp: Tidy up smp_prepare_cpus()

[Jinjie Ruan]

From: Will Deacon <will@kernel.org>

smp_prepare_cpus() is always run on the boot CPU (i.e. CPU 0) but goes
to great lengths to support running on a CPU where smp_processor_id()
is non-zero.

Clean up the code a little by hardcoding zero for the boot CPU ID.

Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/smp.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 1aa324104afb..e858d7d64d1f 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -772,16 +772,14 @@ void __init smp_init_cpus(void)
 void __init smp_prepare_cpus(unsigned int max_cpus)
 {
 	const struct cpu_operations *ops;
-	int err;
 	unsigned int cpu;
-	unsigned int this_cpu;
+	int err;
 
 	init_cpu_topology();
 
-	this_cpu = smp_processor_id();
-	store_cpu_topology(this_cpu);
-	numa_store_cpu_info(this_cpu);
-	numa_add_cpu(this_cpu);
+	store_cpu_topology(0);
+	numa_store_cpu_info(0);
+	numa_add_cpu(0);
 
 	/*
 	 * If UP is mandated by "nosmp" (which implies "maxcpus=0"), don't set
@@ -796,8 +794,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
 	 * secondaries from the bootloader.
 	 */
 	for_each_possible_cpu(cpu) {
-
-		if (cpu == smp_processor_id())
+		if (cpu == 0)
 			continue;
 
 		ops = get_cpu_ops(cpu);
-- 
2.34.1

[PATCH v3 04/12] arm64: smp: Tidy up cpuinfo init and cpufeature updates

[Jinjie Ruan]

From: Will Deacon <will@kernel.org>

Populating the 'cpuinfo_arm64' structure during CPU bringup and
subsequently checking/updating cpufeature structures is slightly
convoluted and differs unnecessarily between the boot CPU and secondary
CPUs.

Rework the code so that cpuinfo_store_cpu() is used to populate the
'cpuinfo_arm64' structure for each CPU, with secondary CPUs then calling
update_cpu_features() to update the global view of the available
features. This allows us to internalise the 'boot_cpu_data' in
cpufeature.c and paves the way for parallelising the ID register probing
during bring-up of secondary CPUs.

Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/cpu.h   |  6 ++----
 arch/arm64/kernel/cpufeature.c | 21 +++++++++++++++++----
 arch/arm64/kernel/cpuinfo.c    | 11 -----------
 arch/arm64/kernel/smp.c        |  3 ++-
 4 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/arch/arm64/include/asm/cpu.h b/arch/arm64/include/asm/cpu.h
index 71493b760b83..b77af3b1bde6 100644
--- a/arch/arm64/include/asm/cpu.h
+++ b/arch/arm64/include/asm/cpu.h
@@ -73,10 +73,8 @@ struct cpuinfo_arm64 {
 DECLARE_PER_CPU(struct cpuinfo_arm64, cpu_data);
 
 void cpuinfo_store_cpu(void);
-void __init cpuinfo_store_boot_cpu(void);
 
-void __init init_cpu_features(struct cpuinfo_arm64 *info);
-void update_cpu_features(int cpu, struct cpuinfo_arm64 *info,
-				 struct cpuinfo_arm64 *boot);
+void __init init_cpu_features(void);
+void update_cpu_features(int cpu);
 
 #endif /* __ASM_CPU_H */
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 6d53bb15cf7b..be75e60d56ca 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -117,6 +117,7 @@ EXPORT_SYMBOL(system_cpucaps);
 static struct arm64_cpu_capabilities const __ro_after_init *cpucap_ptrs[ARM64_NCAPS];
 
 DECLARE_BITMAP(boot_cpucaps, ARM64_NCAPS);
+static struct cpuinfo_arm64 boot_cpu_data;
 
 /*
  * arm64_use_ng_mappings must be placed in the .data section, otherwise it
@@ -1164,11 +1165,19 @@ static __init void detect_system_supports_pseudo_nmi(void)
 static inline void detect_system_supports_pseudo_nmi(void) { }
 #endif
 
-void __init init_cpu_features(struct cpuinfo_arm64 *info)
+void __init init_cpu_features(void)
 {
+	struct cpuinfo_arm64 *info = &per_cpu(cpu_data, 0);
+
 	/* Before we start using the tables, make sure it is sorted */
 	sort_ftr_regs();
 
+	/*
+	 * We keep a copy of the boot CPU registers so that physical hotplug
+	 * of CPU 0 can still be properly checked.
+	 */
+	boot_cpu_data = *info;
+
 	init_cpu_ftr_reg(SYS_CTR_EL0, info->reg_ctr);
 	init_cpu_ftr_reg(SYS_DCZID_EL0, info->reg_dczid);
 	init_cpu_ftr_reg(SYS_CNTFRQ_EL0, info->reg_cntfrq);
@@ -1363,12 +1372,14 @@ static int update_32bit_cpu_features(int cpu, struct cpuinfo_32bit *info,
  * non-boot CPU. Also performs SANITY checks to make sure that there
  * aren't any insane variations from that of the boot CPU.
  */
-void update_cpu_features(int cpu,
-			 struct cpuinfo_arm64 *info,
-			 struct cpuinfo_arm64 *boot)
+void update_cpu_features(int cpu)
 {
+	struct cpuinfo_arm64 *boot, *info;
 	int taint = 0;
 
+	boot = &boot_cpu_data;
+	info = per_cpu_ptr(&cpu_data, cpu);
+
 	/*
 	 * The kernel can handle differing I-cache policies, but otherwise
 	 * caches should look identical. Userspace JITs will make use of
@@ -3924,6 +3935,8 @@ static void __init setup_boot_cpu_capabilities(void)
 
 void __init setup_boot_cpu_features(void)
 {
+	init_cpu_features();
+
 	/*
 	 * Initialize the indirect array of CPU capabilities pointers before we
 	 * handle the boot CPU.
diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c
index 6149bc91251d..df740dc478b2 100644
--- a/arch/arm64/kernel/cpuinfo.c
+++ b/arch/arm64/kernel/cpuinfo.c
@@ -31,7 +31,6 @@
  * values depending on configuration at or after reset.
  */
 DEFINE_PER_CPU(struct cpuinfo_arm64, cpu_data);
-static struct cpuinfo_arm64 boot_cpu_data;
 
 static inline const char *icache_policy_str(int l1ip)
 {
@@ -523,14 +522,4 @@ void cpuinfo_store_cpu(void)
 {
 	struct cpuinfo_arm64 *info = this_cpu_ptr(&cpu_data);
 	__cpuinfo_store_cpu(info);
-	update_cpu_features(smp_processor_id(), info, &boot_cpu_data);
-}
-
-void __init cpuinfo_store_boot_cpu(void)
-{
-	struct cpuinfo_arm64 *info = &per_cpu(cpu_data, 0);
-	__cpuinfo_store_cpu(info);
-
-	boot_cpu_data = *info;
-	init_cpu_features(&boot_cpu_data);
 }
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index e858d7d64d1f..c14b179c595d 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -233,6 +233,7 @@ asmlinkage notrace void secondary_start_kernel(void)
 	 * Log the CPU info before it is marked online and might get read.
 	 */
 	cpuinfo_store_cpu();
+	update_cpu_features(cpu);
 	store_cpu_topology(cpu);
 
 	/*
@@ -453,7 +454,7 @@ void __init smp_prepare_boot_cpu(void)
 	 */
 	set_my_cpu_offset(per_cpu_offset(smp_processor_id()));
 
-	cpuinfo_store_boot_cpu();
+	cpuinfo_store_cpu();
 	setup_boot_cpu_features();
 
 	/* Conditionally switch to GIC PMR for interrupt masking */
-- 
2.34.1

[PATCH v3 05/12] arm64: smp: Defer RCU registration during secondary CPU bringup

[Jinjie Ruan]

From: Will Deacon <will@kernel.org>

Calling rcutree_report_cpu_starting() early during boot can lead to
livelocks with the generic CPU hotplug mechanism if the boot CPU blocks
on an RCU grace period while the CPU being onlined is spinning in
cpuhp_ap_sync_alive(). So cpuhp_ap_sync_alive() must be called
before rcutree_report_cpu_starting().

And to prevent a potential deadlock on the boot CPU,
check_local_cpu_capabilities() must be executed before
cpuhp_ap_sync_alive(). This ensures that if an early capability mismatch
occurs and the AP invokes cpu_die_early(), the boot CPU can detect
the boot timeout and proceed, rather than hanging indefinitely.

In preparation for enabling the generic CPU hotplug code on arm64, split
up the trace_hardirqs_off() call during secondary CPU bringup so that we
update lockdep early but defer the tracing updates until after
RCU is ready.

Furthermore, to support parallel bringup without triggering false RCU CPU
stall Warnings or deadlocks, the initialization order must be:

    secondary_start_kernel()
        -> lockdep_hardirqs_off()
        -> check_local_cpu_capabilities()
           -> cpuhp_ap_sync_alive()
        -> cpuhp_ap_sync_alive()
        -> rcutree_report_cpu_starting()
        -> trace_hardirqs_off_finish()

Because check_local_cpu_capabilities() must execute while RCU is still
offline on the local CPU, it normally triggers a false-positive lockdep
"suspicious RCU usage" splat during early lock acquisitions as commit
ce3d31ad3cac ("arm64/smp: Move rcu_cpu_starting() earlier") pointed out.

Resolve this lockdep splat by wrapping the early capability verification
path within lockdep_off() and lockdep_on(). This safely suppresses
false-positive RCU validation flags on the offline CPU while maintaining
the strictly mandated initialization order for race-free parallel bringup.

Signed-off-by: Will Deacon <will@kernel.org>
Co-developed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/smp.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index c14b179c595d..87f92cf9ffa8 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -35,6 +35,7 @@
 #include <linux/kgdb.h>
 #include <linux/kvm_host.h>
 #include <linux/nmi.h>
+#include <linux/lockdep.h>
 
 #include <asm/alternative.h>
 #include <asm/atomic.h>
@@ -215,15 +216,23 @@ asmlinkage notrace void secondary_start_kernel(void)
 	if (system_uses_irq_prio_masking())
 		init_gic_priority_masking();
 
-	rcutree_report_cpu_starting(cpu);
-	trace_hardirqs_off();
+	lockdep_hardirqs_off(CALLER_ADDR0);
 
+	/*
+	 * Since RCU is still offline on this CPU, any nested native printk
+	 * or lock acquisition would normally trigger a false-positive
+	 * "suspicious RCU usage" lockdep splat.
+	 */
+	lockdep_off();
 	/*
 	 * If the system has established the capabilities, make sure
 	 * this CPU ticks all of those. If it doesn't, the CPU will
 	 * fail to come online.
 	 */
 	check_local_cpu_capabilities();
+	lockdep_on();
+	rcutree_report_cpu_starting(cpu);
+	trace_hardirqs_off_finish();
 
 	ops = get_cpu_ops(cpu);
 	if (ops->cpu_postboot)
-- 
2.34.1

[PATCH v3 06/12] arm64: smp: Use generic HOTPLUG_SPLIT_STARTUP machinery for CPU onlining

[Jinjie Ruan]

From: Will Deacon <will@kernel.org>

In preparation for enabling parallel bringup of secondary CPUs
on arm64, take the baby step of moving from HOTPLUG_CORE_SYNC_DEAD
to HOTPLUG_SPLIT_STARTUP.

To fully enable HOTPLUG_SPLIT_STARTUP, this patch implements:

1) arch_cpuhp_kick_ap_alive(). Kick the secondary CPU via firmware
without blocking.

2) arch_cpuhp_cleanup_kick_cpu(). Extracts early boot telemetry upon
AP bringup timeouts.

3) Callbacks to cpuhp_ap_sync_alive() inside secondary_start_kernel().
Enforces the initial pre-online boot handshake from the secondary
CPU side.

Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/Kconfig      |  2 +-
 arch/arm64/kernel/smp.c | 39 +++++++++++++++++++--------------------
 2 files changed, 20 insertions(+), 21 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fe60738e5943..24496e9967a8 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -231,7 +231,7 @@ config ARM64
 	select HAVE_KPROBES
 	select HAVE_KRETPROBES
 	select HAVE_GENERIC_VDSO
-	select HOTPLUG_CORE_SYNC_DEAD if HOTPLUG_CPU
+	select HOTPLUG_SPLIT_STARTUP if SMP
 	select HOTPLUG_SMT if HOTPLUG_CPU
 	select IRQ_DOMAIN
 	select IRQ_FORCED_THREADING
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 87f92cf9ffa8..9482e8d38b98 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -107,12 +107,9 @@ static int boot_secondary(unsigned int cpu, struct task_struct *idle)
 	return -EOPNOTSUPP;
 }
 
-static DECLARE_COMPLETION(cpu_running);
-
-int __cpu_up(unsigned int cpu, struct task_struct *idle)
+int arch_cpuhp_kick_ap_alive(unsigned int cpu, struct task_struct *idle)
 {
 	int ret;
-	long status;
 
 	/*
 	 * We need to tell the secondary core where to find its stack and the
@@ -123,22 +120,22 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
 
 	/* Now bring the CPU into our world */
 	ret = boot_secondary(cpu, idle);
-	if (ret) {
-		if (ret != -EPERM)
-			pr_err("CPU%u: failed to boot: %d\n", cpu, ret);
-		return ret;
-	}
+	if (ret && ret != -EPERM)
+		pr_err("CPU%u: failed to boot: %d\n", cpu, ret);
+	return ret;
+}
+
+void arch_cpuhp_cleanup_kick_cpu(unsigned int cpu, bool is_alive)
+{
+	long status;
+
+	if (is_alive)
+		return;
 
 	/*
-	 * CPU was successfully started, wait for it to come online or
-	 * time out.
+	 * We failed to synchronise with the CPU, so check if it left us
+	 * any breadcrumbs.
 	 */
-	wait_for_completion_timeout(&cpu_running,
-				    msecs_to_jiffies(5000));
-	if (cpu_online(cpu))
-		return 0;
-
-	pr_crit("CPU%u: failed to come online\n", cpu);
 	secondary_data.task = NULL;
 	status = READ_ONCE(secondary_data.status);
 	if (status == CPU_MMU_OFF)
@@ -170,8 +167,6 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
 	case CPU_PANIC_KERNEL:
 		panic("CPU%u detected unsupported configuration\n", cpu);
 	}
-
-	return -EIO;
 }
 
 static void init_gic_priority_masking(void)
@@ -231,6 +226,11 @@ asmlinkage notrace void secondary_start_kernel(void)
 	 */
 	check_local_cpu_capabilities();
 	lockdep_on();
+	/*
+	 * Synchronise with the core bringing us online so that it knows
+	 * we made it into the kernel. We're still not 'online'.
+	 */
+	cpuhp_ap_sync_alive();
 	rcutree_report_cpu_starting(cpu);
 	trace_hardirqs_off_finish();
 
@@ -264,7 +264,6 @@ asmlinkage notrace void secondary_start_kernel(void)
 					 read_cpuid_id());
 	update_cpu_boot_status(CPU_BOOT_SUCCESS);
 	set_cpu_online(cpu, true);
-	complete(&cpu_running);
 
 	/*
 	 * Secondary CPUs enter the kernel with all DAIF exceptions masked.
-- 
2.34.1

[PATCH v3 07/12] arm64: cpu_ops: Make 'cpu_operations' pointer global instead of per-cpu

[Jinjie Ruan]

From: Will Deacon <will@kernel.org>

'cpu_ops' is an NR_CPUS-length array of 'cpu_operations' pointers, which
theoretically allows for different CPUs to have different bringup and
hotplug backends.

In reality, this complexity exists only to deal with the case where CPU0
is not hotpluggable, so replace the array with a single, global pointer
and record separately whether or not they apply to the boot CPU. Update
the logic in init_cpu_ops() to enforce that only a single set of
'cpu_ops' is required.

Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/cpu_ops.c | 29 ++++++++++++++++++++---------
 1 file changed, 20 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/kernel/cpu_ops.c b/arch/arm64/kernel/cpu_ops.c
index e133011f64b5..eacfb88a0c0c 100644
--- a/arch/arm64/kernel/cpu_ops.c
+++ b/arch/arm64/kernel/cpu_ops.c
@@ -20,7 +20,8 @@ extern const struct cpu_operations acpi_parking_protocol_ops;
 #endif
 extern const struct cpu_operations cpu_psci_ops;
 
-static const struct cpu_operations *cpu_ops[NR_CPUS] __ro_after_init;
+static const struct cpu_operations *cpu_ops __ro_after_init;
+static bool boot_cpu_has_enable_method __ro_after_init;
 
 static const struct cpu_operations *const dt_supported_cpu_ops[] __initconst = {
 	&smp_spin_table_ops,
@@ -40,6 +41,9 @@ static const struct cpu_operations * __init cpu_get_ops(const char *name)
 {
 	const struct cpu_operations *const *ops;
 
+	if (!name)
+		return NULL;
+
 	ops = acpi_disabled ? dt_supported_cpu_ops : acpi_supported_cpu_ops;
 
 	while (*ops) {
@@ -49,6 +53,7 @@ static const struct cpu_operations * __init cpu_get_ops(const char *name)
 		ops++;
 	}
 
+	pr_warn("Unsupported enable-method: %s\n", name);
 	return NULL;
 }
 
@@ -94,25 +99,31 @@ static const char *__init cpu_read_enable_method(int cpu)
 	return enable_method;
 }
 /*
- * Read a cpu's enable method and record it in cpu_ops.
+ * Read a cpu's enable method and update/check cpu_ops.
  */
 int __init init_cpu_ops(int cpu)
 {
 	const char *enable_method = cpu_read_enable_method(cpu);
+	const struct cpu_operations *ops = cpu_get_ops(enable_method);
 
-	if (!enable_method)
+	if (!ops)
 		return -ENODEV;
 
-	cpu_ops[cpu] = cpu_get_ops(enable_method);
-	if (!cpu_ops[cpu]) {
-		pr_warn("Unsupported enable-method: %s\n", enable_method);
-		return -EOPNOTSUPP;
-	}
+	if (!cpu_ops)
+		cpu_ops = ops;
+	else if (cpu_ops != ops)
+		return -EBUSY;
+
+	if (cpu == 0)
+		boot_cpu_has_enable_method = true;
 
 	return 0;
 }
 
 const struct cpu_operations *get_cpu_ops(int cpu)
 {
-	return cpu_ops[cpu];
+	if (cpu || boot_cpu_has_enable_method)
+		return cpu_ops;
+
+	return NULL;
 }
-- 
2.34.1

[PATCH v3 08/12] arm64: cpu_ops: Introduce get_secondary_cpu_ops()

[Jinjie Ruan]

From: Will Deacon <will@kernel.org>

Introduce get_secondary_cpu_ops() to retrieve a pointer to the
'cpu_operations' structure for the non-boot CPUs and use it instead of
get_cpu_ops() where we are dealing with secondary CPUs.

This is a pre-requisite for enabling parallel CPU bring-up.

Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/cpu_ops.h |  1 +
 arch/arm64/kernel/cpu_ops.c      |  5 +++++
 arch/arm64/kernel/smp.c          | 19 +++++++------------
 3 files changed, 13 insertions(+), 12 deletions(-)

diff --git a/arch/arm64/include/asm/cpu_ops.h b/arch/arm64/include/asm/cpu_ops.h
index a444c8915e88..cd298a8710d8 100644
--- a/arch/arm64/include/asm/cpu_ops.h
+++ b/arch/arm64/include/asm/cpu_ops.h
@@ -48,6 +48,7 @@ struct cpu_operations {
 
 int __init init_cpu_ops(int cpu);
 extern const struct cpu_operations *get_cpu_ops(int cpu);
+extern const struct cpu_operations *get_secondary_cpu_ops(void);
 
 static inline void __init init_bootcpu_ops(void)
 {
diff --git a/arch/arm64/kernel/cpu_ops.c b/arch/arm64/kernel/cpu_ops.c
index eacfb88a0c0c..7d183ca31dc8 100644
--- a/arch/arm64/kernel/cpu_ops.c
+++ b/arch/arm64/kernel/cpu_ops.c
@@ -127,3 +127,8 @@ const struct cpu_operations *get_cpu_ops(int cpu)
 
 	return NULL;
 }
+
+const struct cpu_operations *get_secondary_cpu_ops(void)
+{
+	return cpu_ops;
+}
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 9482e8d38b98..6b9586a69429 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -99,7 +99,7 @@ static inline int op_cpu_kill(unsigned int cpu)
  */
 static int boot_secondary(unsigned int cpu, struct task_struct *idle)
 {
-	const struct cpu_operations *ops = get_cpu_ops(cpu);
+	const struct cpu_operations *ops = get_secondary_cpu_ops();
 
 	if (ops->cpu_boot)
 		return ops->cpu_boot(cpu);
@@ -234,7 +234,7 @@ asmlinkage notrace void secondary_start_kernel(void)
 	rcutree_report_cpu_starting(cpu);
 	trace_hardirqs_off_finish();
 
-	ops = get_cpu_ops(cpu);
+	ops = get_secondary_cpu_ops();
 	if (ops->cpu_postboot)
 		ops->cpu_postboot();
 
@@ -334,7 +334,7 @@ int __cpu_disable(void)
 
 static int op_cpu_kill(unsigned int cpu)
 {
-	const struct cpu_operations *ops = get_cpu_ops(cpu);
+	const struct cpu_operations *ops = get_secondary_cpu_ops();
 
 	/*
 	 * If we have no means of synchronising with the dying CPU, then assume
@@ -375,7 +375,7 @@ void arch_cpuhp_cleanup_dead_cpu(unsigned int cpu)
 void __noreturn cpu_die(void)
 {
 	unsigned int cpu = smp_processor_id();
-	const struct cpu_operations *ops = get_cpu_ops(cpu);
+	const struct cpu_operations *ops = get_secondary_cpu_ops();
 
 	idle_task_exit();
 
@@ -501,7 +501,7 @@ static int __init smp_cpu_setup(int cpu)
 	if (init_cpu_ops(cpu))
 		return -ENODEV;
 
-	ops = get_cpu_ops(cpu);
+	ops = get_secondary_cpu_ops();
 	if (ops->cpu_init(cpu))
 		return -ENODEV;
 
@@ -780,7 +780,7 @@ void __init smp_init_cpus(void)
 
 void __init smp_prepare_cpus(unsigned int max_cpus)
 {
-	const struct cpu_operations *ops;
+	const struct cpu_operations *ops = get_secondary_cpu_ops();
 	unsigned int cpu;
 	int err;
 
@@ -806,10 +806,6 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
 		if (cpu == 0)
 			continue;
 
-		ops = get_cpu_ops(cpu);
-		if (!ops)
-			continue;
-
 		err = ops->cpu_prepare(cpu);
 		if (err)
 			continue;
@@ -1299,8 +1295,7 @@ bool smp_crash_stop_failed(void)
 static bool have_cpu_die(void)
 {
 #ifdef CONFIG_HOTPLUG_CPU
-	int any_cpu = raw_smp_processor_id();
-	const struct cpu_operations *ops = get_cpu_ops(any_cpu);
+	const struct cpu_operations *ops = get_secondary_cpu_ops();
 
 	if (ops && ops->cpu_die)
 		return true;
-- 
2.34.1

[PATCH v3 09/12] arm64: cpufeature: Ensure atomic updates to system_cpucaps bitmap

[Jinjie Ruan]

Parallel CPU bringup allows multiple secondary CPUs to concurrently
execute update_cpu_capabilities() during early boot.

The current non-atomic __set_bit() and __clear_bit() helpers perform
unserialized updates on the shared global bitmap, risking data races
and feature flag erasure.

Upgrade these operations to set_bit() and clear_bit() to ensure all
concurrent modifications are properly serialized via arm64 atomics.

Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/cpufeature.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index be75e60d56ca..a1a13f3e01ed 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -3548,7 +3548,7 @@ static void update_cpu_capabilities(u16 scope_mask)
 
 		if (!caps->matches(caps, cpucap_default_scope(caps))) {
 			if (match_all)
-				__clear_bit(caps->capability, system_cpucaps);
+				clear_bit(caps->capability, system_cpucaps);
 			continue;
 		}
 
@@ -3559,7 +3559,7 @@ static void update_cpu_capabilities(u16 scope_mask)
 		if (!match_all && caps->desc && !caps->cpus)
 			pr_info("detected: %s\n", caps->desc);
 
-		__set_bit(caps->capability, system_cpucaps);
+		set_bit(caps->capability, system_cpucaps);
 
 		if (boot_cpu && (caps->type & SCOPE_BOOT_CPU))
 			set_bit(caps->capability, boot_cpucaps);
-- 
2.34.1

[PATCH v3 10/12] arm64: smp: Pass CPU ID to update_cpu_boot_status()

[Jinjie Ruan]

To support CONFIG_HOTPLUG_PARALLEL, the CPU boot status tracking must
be refactored from a single global variable (secondary_data.status)
to a per-CPU tracking structure to prevent multi-core race conditions.

Add a 'cpu' parameter to update_cpu_boot_status() and update all its
callsites to pass the corresponding CPU ID. This allows updating the
boot status at a per-CPU granularity during parallel bringup.

Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/smp.h   | 6 +++---
 arch/arm64/kernel/cpufeature.c | 2 +-
 arch/arm64/kernel/smp.c        | 8 ++++----
 arch/arm64/mm/context.c        | 5 +++--
 4 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index 10ea4f543069..e2151a01731f 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -122,7 +122,7 @@ static inline void __noreturn cpu_park_loop(void)
 	}
 }
 
-static inline void update_cpu_boot_status(int val)
+static inline void update_cpu_boot_status(unsigned int cpu, int val)
 {
 	WRITE_ONCE(secondary_data.status, val);
 	/* Ensure the visibility of the status update */
@@ -134,9 +134,9 @@ static inline void update_cpu_boot_status(int val)
  * which calls for a kernel panic. Update the boot status and park the calling
  * CPU.
  */
-static inline void __noreturn cpu_panic_kernel(void)
+static inline void __noreturn cpu_panic_kernel(unsigned int cpu)
 {
-	update_cpu_boot_status(CPU_PANIC_KERNEL);
+	update_cpu_boot_status(cpu, CPU_PANIC_KERNEL);
 	cpu_park_loop();
 }
 
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index a1a13f3e01ed..abc3aef9c206 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -3685,7 +3685,7 @@ static void verify_local_cpu_caps(u16 scope_mask)
 			caps->desc, system_has_cap, cpu_has_cap);
 
 		if (cpucap_panic_on_conflict(caps))
-			cpu_panic_kernel();
+			cpu_panic_kernel(smp_processor_id());
 		else
 			cpu_die_early();
 	}
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 6b9586a69429..14b94df26b44 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -116,7 +116,7 @@ int arch_cpuhp_kick_ap_alive(unsigned int cpu, struct task_struct *idle)
 	 * page tables.
 	 */
 	secondary_data.task = idle;
-	update_cpu_boot_status(CPU_MMU_OFF);
+	update_cpu_boot_status(cpu, CPU_MMU_OFF);
 
 	/* Now bring the CPU into our world */
 	ret = boot_secondary(cpu, idle);
@@ -262,7 +262,7 @@ asmlinkage notrace void secondary_start_kernel(void)
 	pr_info("CPU%u: Booted secondary processor 0x%010lx [0x%08x]\n",
 					 cpu, (unsigned long)mpidr,
 					 read_cpuid_id());
-	update_cpu_boot_status(CPU_BOOT_SUCCESS);
+	update_cpu_boot_status(cpu, CPU_BOOT_SUCCESS);
 	set_cpu_online(cpu, true);
 
 	/*
@@ -420,11 +420,11 @@ void __noreturn cpu_die_early(void)
 	rcutree_report_cpu_dead();
 
 	if (IS_ENABLED(CONFIG_HOTPLUG_CPU)) {
-		update_cpu_boot_status(CPU_KILL_ME);
+		update_cpu_boot_status(cpu, CPU_KILL_ME);
 		__cpu_try_die(cpu);
 	}
 
-	update_cpu_boot_status(CPU_STUCK_IN_KERNEL);
+	update_cpu_boot_status(cpu, CPU_STUCK_IN_KERNEL);
 
 	cpu_park_loop();
 }
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index 0f4a28b87469..e78ae989ad57 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -63,6 +63,7 @@ static u32 get_cpu_asid_bits(void)
 /* Check if the current cpu's ASIDBits is compatible with asid_bits */
 void verify_cpu_asid_bits(void)
 {
+	unsigned int cpu = smp_processor_id();
 	u32 asid = get_cpu_asid_bits();
 
 	if (asid < asid_bits) {
@@ -71,8 +72,8 @@ void verify_cpu_asid_bits(void)
 		 * fewer ASID bits than the boot CPU.
 		 */
 		pr_crit("CPU%d: smaller ASID size(%u) than boot CPU (%u)\n",
-				smp_processor_id(), asid, asid_bits);
-		cpu_panic_kernel();
+			cpu, asid, asid_bits);
+		cpu_panic_kernel(cpu);
 	}
 }
 
-- 
2.34.1

[PATCH v3 11/12] arm64: smp: Rework early boot data into per-CPU arrays

[Jinjie Ruan]

Rework the global `secondary_data` and `__early_cpu_boot_status`
accessed during early boot into per-CPU arrays to allow secondary
CPUs to boot in parallel.

And reuse `__cpu_logical_map` array in the early boot code in head.S
to resolve each secondary CPU's logical ID concurrently.

Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/smp.h    | 16 ++++++------
 arch/arm64/kernel/asm-offsets.c |  2 ++
 arch/arm64/kernel/head.S        | 44 +++++++++++++++++++++++++++------
 arch/arm64/kernel/setup.c       |  9 +++++++
 arch/arm64/kernel/smp.c         | 11 +++++----
 arch/arm64/mm/mmu.c             |  2 +-
 6 files changed, 62 insertions(+), 22 deletions(-)

diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index e2151a01731f..7e69219e6e33 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -34,14 +34,9 @@
 /*
  * Logical CPU mapping.
  */
-extern u64 __cpu_logical_map[NR_CPUS];
+extern void set_cpu_logical_map(unsigned int cpu, u64 hwid);
 extern u64 cpu_logical_map(unsigned int cpu);
 
-static inline void set_cpu_logical_map(unsigned int cpu, u64 hwid)
-{
-	__cpu_logical_map[cpu] = hwid;
-}
-
 struct seq_file;
 
 /*
@@ -92,8 +87,11 @@ struct secondary_data {
 	long status;
 };
 
-extern struct secondary_data secondary_data;
-extern long __early_cpu_boot_status;
+static_assert((sizeof(struct secondary_data) & (sizeof(struct secondary_data) - 1)) == 0,
+	      "secondary_data size must be a power of 2 for assembly lsl assembly!");
+
+extern struct secondary_data cpu_boot_data[NR_CPUS];
+extern long __early_cpu_boot_status[NR_CPUS];
 extern void secondary_entry(void);
 
 extern void arch_send_call_function_single_ipi(int cpu);
@@ -124,7 +122,7 @@ static inline void __noreturn cpu_park_loop(void)
 
 static inline void update_cpu_boot_status(unsigned int cpu, int val)
 {
-	WRITE_ONCE(secondary_data.status, val);
+	WRITE_ONCE(cpu_boot_data[cpu].status, val);
 	/* Ensure the visibility of the status update */
 	dsb(ishst);
 }
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index b6367ff3a49c..566e2222af5b 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -11,6 +11,7 @@
 #include <linux/arm_sdei.h>
 #include <linux/sched.h>
 #include <linux/ftrace.h>
+#include <linux/log2.h>
 #include <linux/kexec.h>
 #include <linux/mm.h>
 #include <linux/kvm_host.h>
@@ -97,6 +98,7 @@ int main(void)
   BLANK();
 #endif
   DEFINE(CPU_BOOT_TASK,		offsetof(struct secondary_data, task));
+  DEFINE(SECONDARY_DATA_SHIFT,	ilog2(sizeof(struct secondary_data)));
   BLANK();
   DEFINE(FTR_OVR_VAL_OFFSET,	offsetof(struct arm64_ftr_override, val));
   DEFINE(FTR_OVR_MASK_OFFSET,	offsetof(struct arm64_ftr_override, mask));
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 87a822e5c4ca..f58de58c4edc 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -12,6 +12,7 @@
 #include <linux/linkage.h>
 #include <linux/init.h>
 #include <linux/pgtable.h>
+#include <linux/threads.h>
 
 #include <asm/asm_pointer_auth.h>
 #include <asm/assembler.h>
@@ -348,6 +349,31 @@ pen:	ldr	x4, [x3]
 	b	pen
 SYM_FUNC_END(secondary_holding_pen)
 
+	/*
+	 * Convert the physical MPIDR of the current secondary CPU
+	 * to its logical CPUID by traversing __cpu_logical_map
+	 * in parallel.
+	 */
+	.macro	mpidr_to_cpuid, mpidr, cpuid, tmp1, tmp2
+	mov_q	\tmp1, MPIDR_HWID_BITMASK
+	and	\mpidr, \mpidr, \tmp1
+
+	adr_l	\tmp1, __cpu_logical_map
+	mov	\cpuid, #0
+.Lfind_cpuid\@:
+	ldr	\tmp2, [\tmp1, \cpuid, lsl #3]
+	cmp	\tmp2, #-1
+	b.eq	.Lnext_cpu\@
+	cmp	\tmp2, \mpidr
+	b.eq	.Lfound_cpuid\@
+.Lnext_cpu\@:
+	add	\cpuid, \cpuid, #1
+	cmp	\cpuid, #NR_CPUS
+	b.ne	.Lfind_cpuid\@
+	b	__secondary_too_slow
+.Lfound_cpuid\@:
+	.endm
+
 	/*
 	 * Secondary entry point that jumps straight into the kernel. Only to
 	 * be used where CPUs are brought online dynamically by the kernel.
@@ -363,6 +389,8 @@ SYM_FUNC_START_LOCAL(secondary_startup)
 	 * Common entry point for secondary CPUs.
 	 */
 	mov	x20, x0				// preserve boot mode
+	mrs	x0, mpidr_el1
+	mpidr_to_cpuid mpidr=x0, cpuid=x19, tmp1=x1, tmp2=x3
 
 #ifdef CONFIG_ARM64_VA_BITS_52
 alternative_if ARM64_HAS_VA52
@@ -386,12 +414,12 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
 	mov	x0, x20
 	bl	finalise_el2
 
-	str_l	xzr, __early_cpu_boot_status, x3
 	adr_l	x5, vectors
 	msr	vbar_el1, x5
 	isb
 
-	adr_l	x0, secondary_data
+	adr_l	x0, cpu_boot_data
+	add	x0, x0, x19, lsl #SECONDARY_DATA_SHIFT
 	ldr	x2, [x0, #CPU_BOOT_TASK]
 	cbz	x2, __secondary_too_slow
 
@@ -430,13 +458,14 @@ SYM_FUNC_END(set_cpu_boot_mode_flag)
  *
  * update_early_cpu_boot_status tmp, status
  *  - Corrupts tmp1, tmp2
- *  - Writes 'status' to __early_cpu_boot_status and makes sure
+ *  - Writes 'status' to __early_cpu_boot_status[cpu] and makes sure
  *    it is committed to memory.
  */
 
-	.macro	update_early_cpu_boot_status status, tmp1, tmp2
-	mov	\tmp2, #\status
+	.macro	update_early_cpu_boot_status status, cpu_reg, tmp1, tmp2
 	adr_l	\tmp1, __early_cpu_boot_status
+	mov	\tmp2, #\status
+	add	\tmp1, \tmp1, \cpu_reg, lsl #3
 	str	\tmp2, [\tmp1]
 	dmb	sy
 	dc	ivac, \tmp1			// Invalidate potentially stale cache line
@@ -486,7 +515,7 @@ SYM_FUNC_START(__cpu_secondary_check52bitva)
 #endif
 
 	update_early_cpu_boot_status \
-		CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_52_BIT_VA, x0, x1
+		CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_52_BIT_VA, x19, x0, x1
 1:	wfe
 	wfi
 	b	1b
@@ -498,7 +527,7 @@ SYM_FUNC_END(__cpu_secondary_check52bitva)
 SYM_FUNC_START_LOCAL(__no_granule_support)
 	/* Indicate that this CPU can't boot and is stuck in the kernel */
 	update_early_cpu_boot_status \
-		CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_NO_GRAN, x1, x2
+		CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_NO_GRAN, x19, x1, x2
 1:
 	wfe
 	wfi
@@ -508,6 +537,7 @@ SYM_FUNC_END(__no_granule_support)
 SYM_FUNC_START_LOCAL(__primary_switch)
 	adrp	x1, reserved_pg_dir
 	adrp	x2, __pi_init_idmap_pg_dir
+	mov	x19, #0
 	bl	__enable_mmu
 
 	adrp	x1, early_init_stack
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 23c05dc7a8f2..856c00ab6f19 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -273,6 +273,15 @@ arch_initcall(reserve_memblock_reserved_regions);
 
 u64 __cpu_logical_map[NR_CPUS] = { [0 ... NR_CPUS-1] = INVALID_HWID };
 
+void set_cpu_logical_map(unsigned int cpu, u64 hwid)
+{
+	unsigned long start = (unsigned long)&__cpu_logical_map[cpu];
+	unsigned long end = start + sizeof(__cpu_logical_map[cpu]);
+
+	__cpu_logical_map[cpu] = hwid;
+	dcache_clean_inval_poc(start, end);
+}
+
 u64 cpu_logical_map(unsigned int cpu)
 {
 	return __cpu_logical_map[cpu];
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 14b94df26b44..98ddbe50081d 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -61,7 +61,8 @@
  * so we need some other way of telling a new secondary core
  * where to place its SVC stack
  */
-struct secondary_data secondary_data;
+struct secondary_data cpu_boot_data[NR_CPUS] ____cacheline_aligned;
+
 /* Number of CPUs which aren't online, but looping in kernel text. */
 static int cpus_stuck_in_kernel;
 
@@ -115,7 +116,7 @@ int arch_cpuhp_kick_ap_alive(unsigned int cpu, struct task_struct *idle)
 	 * We need to tell the secondary core where to find its stack and the
 	 * page tables.
 	 */
-	secondary_data.task = idle;
+	cpu_boot_data[cpu].task = idle;
 	update_cpu_boot_status(cpu, CPU_MMU_OFF);
 
 	/* Now bring the CPU into our world */
@@ -136,10 +137,10 @@ void arch_cpuhp_cleanup_kick_cpu(unsigned int cpu, bool is_alive)
 	 * We failed to synchronise with the CPU, so check if it left us
 	 * any breadcrumbs.
 	 */
-	secondary_data.task = NULL;
-	status = READ_ONCE(secondary_data.status);
+	cpu_boot_data[cpu].task = NULL;
+	status = READ_ONCE(cpu_boot_data[cpu].status);
 	if (status == CPU_MMU_OFF)
-		status = READ_ONCE(__early_cpu_boot_status);
+		status = READ_ONCE(__early_cpu_boot_status[cpu]);
 
 	switch (status & CPU_BOOT_STATUS_MASK) {
 	default:
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index dd85e093ffdb..71c160c6c383 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -62,7 +62,7 @@ static bool rodata_is_rw __ro_after_init = true;
  * The booting CPU updates the failed status @__early_cpu_boot_status,
  * with MMU turned off.
  */
-long __section(".mmuoff.data.write") __early_cpu_boot_status;
+long __section(".mmuoff.data.write") __early_cpu_boot_status[NR_CPUS];
 
 static DEFINE_SPINLOCK(swapper_pgdir_lock);
 static DEFINE_MUTEX(fixmap_lock);
-- 
2.34.1

[PATCH v3 12/12] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs

[Jinjie Ruan]

Support for parallel secondary CPU bringup is already utilized by x86,
MIPS and RISC-V. This patch brings this capability to the arm64
architecture.

To fully enable HOTPLUG_PARALLEL, this patch implements an arm64-specific
arch_cpuhp_init_parallel_bringup() handler.

In parallel bringup, early `set_cpu_present(cpu, 0)` inside
cpu_die_early() removes the secondary CPU prematurely, causing the primary
CPU's second-stage cpuhp_bringup_mask() sweep to skip it and drop
failure logs.

Remove this early unregistration from the secondary CPU, deferring the
set_cpu_present(cpu, 0) call to the primary CPU's cleanup path to ensure
robust parallel boot timeout detection.

Tested natively with ATF on QEMU arm64 virt machine with 64 cores
and also tested with KVM arm64 guest with 128 vCPUs.

Tested-by: Michael Kelley <mhklinux@outlook.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/Kconfig      |  2 +-
 arch/arm64/kernel/smp.c | 12 ++++++++++--
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 24496e9967a8..a9d8030e7492 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -231,7 +231,7 @@ config ARM64
 	select HAVE_KPROBES
 	select HAVE_KRETPROBES
 	select HAVE_GENERIC_VDSO
-	select HOTPLUG_SPLIT_STARTUP if SMP
+	select HOTPLUG_PARALLEL if SMP
 	select HOTPLUG_SMT if HOTPLUG_CPU
 	select IRQ_DOMAIN
 	select IRQ_FORCED_THREADING
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 98ddbe50081d..a973b2d3bab1 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -93,6 +93,15 @@ static inline int op_cpu_kill(unsigned int cpu)
 }
 #endif
 
+extern const struct cpu_operations cpu_psci_ops;
+
+/* Establish whether parallel bringup can be supported. */
+bool __init arch_cpuhp_init_parallel_bringup(void)
+{
+	const struct cpu_operations *ops = get_secondary_cpu_ops();
+
+	return ops == &cpu_psci_ops;
+}
 
 /*
  * Boot a secondary CPU, and assign it the specified idle task.
@@ -137,6 +146,7 @@ void arch_cpuhp_cleanup_kick_cpu(unsigned int cpu, bool is_alive)
 	 * We failed to synchronise with the CPU, so check if it left us
 	 * any breadcrumbs.
 	 */
+	set_cpu_present(cpu, 0);
 	cpu_boot_data[cpu].task = NULL;
 	status = READ_ONCE(cpu_boot_data[cpu].status);
 	if (status == CPU_MMU_OFF)
@@ -416,8 +426,6 @@ void __noreturn cpu_die_early(void)
 
 	pr_crit("CPU%d: will not boot\n", cpu);
 
-	/* Mark this CPU absent */
-	set_cpu_present(cpu, 0);
 	rcutree_report_cpu_dead();
 
 	if (IS_ENABLED(CONFIG_HOTPLUG_CPU)) {
-- 
2.34.1

Re: [PATCH v3 00/12] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs

[Will Deacon]

On Wed, Jun 24, 2026 at 05:25:25PM +0800, Jinjie Ruan wrote:
> Support for parallel secondary CPU bringup is already utilized by x86,
> MIPS, and RISC-V. This patch brings this capability to the arm64
> architecture.
> 
> Introduce CONFIG_HOTPLUG_PARALLEL_SMT to avoid primary SMT threads
> to boot first constraint.
> 
> And add a 'cpu' parameter to update_cpu_boot_status() to allow updating
> the boot status at a per-CPU granularity during parallel bringup.
> 
> Rework the global `secondary_data` and `__early_cpu_boot_status` accessed
> during early boot into per-CPU arrays to allow secondary CPUs to boot
> in parallel.
> 
> And reuse `__cpu_logical_map` array in the early boot code in head.S
> to resolve each secondary CPU's logical ID concurrently.
> 
> This series includes a subset of the refactoring patches proposed
> by Will Deacon, with further adjustments.

Sheesh, Jinjie, what are you doing?

I said yesterday that I would dust off the old series after the merge
window:

  https://lore.kernel.org/all/ajqYaklhIyvaNLlk@willie-the-truck/

"Please just give me a week or so to rebase my changes and send them out
 for discussion"

but instead, you've posted patches from me that are missing a bunch of
fixes that need to be folded back in:

  https://web.git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/commit/?h=cpu-hotplug&id=2d5b8df5d4e2bbc142e3b4f21cabbca96e3da79d

so now you're asking people to review incomplete patches from somebody
else.

Please just give me the time I asked for. If you want to help out in the
meantime, there are plenty of patches that need reviewing...

Will

Re: [PATCH v3 00/12] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs

[Borislav Petkov]

On Wed, Jun 24, 2026 at 01:16:16PM +0100, Will Deacon wrote:
> If you want to help out in the meantime, there are plenty of patches that
> need reviewing...

I would loooove it if git send-email speed and rate of sending mail is
proportional to how many other patches the sender has really reviewed...

:-P :-P :-P

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH v3 00/12] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs

[Jinjie Ruan]

On 6/24/2026 8:16 PM, Will Deacon wrote:
> On Wed, Jun 24, 2026 at 05:25:25PM +0800, Jinjie Ruan wrote:
>> Support for parallel secondary CPU bringup is already utilized by x86,
>> MIPS, and RISC-V. This patch brings this capability to the arm64
>> architecture.
>>
>> Introduce CONFIG_HOTPLUG_PARALLEL_SMT to avoid primary SMT threads
>> to boot first constraint.
>>
>> And add a 'cpu' parameter to update_cpu_boot_status() to allow updating
>> the boot status at a per-CPU granularity during parallel bringup.
>>
>> Rework the global `secondary_data` and `__early_cpu_boot_status` accessed
>> during early boot into per-CPU arrays to allow secondary CPUs to boot
>> in parallel.
>>
>> And reuse `__cpu_logical_map` array in the early boot code in head.S
>> to resolve each secondary CPU's logical ID concurrently.
>>
>> This series includes a subset of the refactoring patches proposed
>> by Will Deacon, with further adjustments.
> 
> Sheesh, Jinjie, what are you doing?
> 
> I said yesterday that I would dust off the old series after the merge
> window:
> 
>   https://lore.kernel.org/all/ajqYaklhIyvaNLlk@willie-the-truck/
> 
> "Please just give me a week or so to rebase my changes and send them out
>  for discussion"
> 
> but instead, you've posted patches from me that are missing a bunch of
> fixes that need to be folded back in:
> 
>   https://web.git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/commit/?h=cpu-hotplug&id=2d5b8df5d4e2bbc142e3b4f21cabbca96e3da79d
> 

Hi Will,

I am terribly sorry for my impatience and the confusion this has caused.
I completely misread the situation and shouldn't have posted those
incomplete patches, especially when you explicitly asked for a week to
rebase them properly.

> so now you're asking people to review incomplete patches from somebody
> else.
> 
> Please just give me the time I asked for. If you want to help out in the
> meantime, there are plenty of patches that need reviewing...


It was a mistake on my part, and I sincerely apologize for wasting
community review resources and disrupting your schedule.

I will absolutely back off now and wait for your official series. Lesson
learned. Thanks for your patience and for calling me out on this.

Best regards,
Jinjie

> 
> Will
>