Re: [PATCH RFC 3/3] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs

[Jinjie Ruan <ruanjinjie@huawei.com>]

On 6/23/2026 10:33 PM, Will Deacon wrote:
> On Mon, Jun 22, 2026 at 05:16:30PM +0800, Jinjie Ruan wrote:
>>
>>
>> On 6/18/2026 8:21 PM, Will Deacon wrote:
>>> Hi Jinjie,
>>>
>>> On Mon, Jun 15, 2026 at 04:51:48PM +0800, Jinjie Ruan wrote:
>>>> On 6/12/2026 11:45 PM, Michael Kelley wrote:
>>>>> From: Jinjie Ruan <ruanjinjie@huawei.com> Sent: Thursday, June 11, 2026 6:38 AM
>>>>>>
>>>>>> Support for parallel secondary CPU bringup is already utilized by x86,
>>>>>> MIPS, and RISC-V. This patch brings this capability to the arm64
>>>>>> architecture.
>>>>>>
>>>>>> Rework the global `secondary_data` accessed during early boot into
>>>>>> a per-CPU array. This array maps logical CPU IDs to MPIDR_EL1 values,
>>>>>> enabling the early boot code in head.S to resolve each secondary CPU's
>>>>>> logical ID concurrently.
>>>>>>
>>>>>> To fully enable HOTPLUG_PARALLEL, this patch implements:
>>>>>> 1) An arm64-specific arch_cpuhp_kick_ap_alive() handler.
>>>>>> 2) Callbacks to cpuhp_ap_sync_alive() inside secondary_start_kernel().
>>>>>>
>>>>>> Successfully tested on QEMU ARM64 virt machine (KVM on, 128 vCPUs).
>>>>>>
>>>>>> |     test kernel	   | secondary CPUs boot time |
>>>>>> |  ---------------------   |	--------------------  |
>>>>>> |   Without this patch     |		155.672	      |
>>>>>> |   cpuhp.parallel=0	   |		62.897	      |
>>>>>> |   cpuhp.parallel=1	   |		166.703	      |
>>>>>
>>>>> The last two rows seem mixed up. I would expect parallel=0 to
>>>>> result in a longer boot time.
>>>>
>>>> Hi, Michael,
>>>>
>>>> The results are correct and not mixed up.
>>>>
>>>> Compared to the original non‑HOTPLUG_PARALLEL approach, the advantage of
>>>> cpuhp.parallel=0 lies in its use of cpu_relax(`yield` on arm64) instead
>>>> of the wait_for_completion_timeout() mechanism (which may cause sleep
>>>> and context switching). This significantly reduces the overhead of VM
>>>> exits and context switches in a KVM guest, thereby cutting the secondary
>>>> CPU boot time by more than half.
>>>
>>> I don't think that's a particularly compelling reason to enable this for
>>> arm64, in all honesty. The yield instruction typically doesn't do
>>> anything on actual arm64 silicon, so this probably means that you're
>>> introducing busy-loops which tend to be bad for power and scalability.
>>>
>>> I implemented this a while ago [1] but didn't manage to see much in terms
>>> of performance improvement and so I didn't bother to send the patches out
>>> after talking about it at KVM forum [2]. However, as mentioned at the end
>>> of that talk, it _is_ still useful for confidential VMs using PSCI so
>>> let me dust off my old series and send it out to see what you think.
>>
>> Hi Will,
>>
>> Thanks for the insights! Your point about using PSCI v0.2's Context ID
>> to avoid the NR_CPUS array for input parameters (like
>> secondary_data.task) is incredibly elegant.
>>
>> However, if I understand your series correctly, it seems your approach
>> primarily targets preventing the concurrent use of secondary_data.task,
>> but it doesn't seem to account for the potential data trampling on
>> secondary_data.status when multiple secondary CPUs are brought up
>> simultaneously.
>>
>> update_cpu_boot_status()
>>   -> WRITE_ONCE(secondary_data.status.flags[val], 1)
>>
>> arch_cpuhp_cleanup_kick_cpu()
>>   -> status = READ_ONCE(secondary_data.status)
> 
> I need to dust it back off but IIRC I made that thing a byte array, with
> a separate byte for each failure reason.
> 
> Will

Hi, Will,

Thanks for the clarification. A byte array with a separate byte per
failure reason does prevent trampling between different failure types.

However, the issue arises if multiple secondary CPUs fail for the exact
same reason simultaneously. In that scenario, they will still attempt to
write to the same byte index at the same time. As a result, the primary
CPU reading the status later won't be able to distinguish which specific
CPUs encountered the problem, or how many of them failed.

I test your patch with error inject, which configures CPU4 and CPU6,
along with CPU16 and CPU18, to generate distinct boot failures, while
making CPU17 hit the same boot failure as CPU16. The output is not
correct as below:

[    0.332528] smp: Bringing up secondary CPUs ...
[   10.674114] CPU1 failed to report alive state
[   10.674392] CPU1 detected lack of support for 52-bit VAs
[   10.674610] CPUs may be stuck in kernel
[   21.016707] CPU2 failed to report alive state
[   31.357320] CPU3 failed to report alive state
[   41.693228] CPU4 failed to report alive state
[   52.033112] CPU5 failed to report alive state
[   62.378198] CPU6 failed to report alive state
[   72.716467] CPU7 failed to report alive state
[   83.046746] CPU8 failed to report alive state
[   93.338020] CPU9 failed to report alive state
[  103.591986] CPU10 failed to report alive state
[  113.893741] CPU11 failed to report alive state
[  124.230870] CPU12 failed to report alive state
[  134.567597] CPU13 failed to report alive state
[  144.905256] CPU14 failed to report alive state
[  155.247633] CPU15 failed to report alive state
[  165.584891] CPU16 failed to report alive state
[  175.920794] CPU17 failed to report alive state
[  186.256323] CPU18 failed to report alive state
[  196.596136] CPU19 failed to report alive state

The expected output is as below:

        CPU4 failed to report alive state
        CPU4: is stuck in kernel
        CPU4: does not support 52-bit VAs

        CPU6 failed to report alive state
        CPU6: is stuck in kernel
        CPU6: does not support 4K granule

        GICv3: CPU8: found redistributor 8 region 0:0x00000000081a0000
        GICv3: CPU8: using allocated LPI pending table @0x0000000100360000
        CPU8: Booted secondary processor 0x0000000008 [0x410fd034]
        ...
        CPU16 failed to report alive state
        psci: CPU16 killed (polled 0 ms)
        CPU16: died during early boot

        CPU17: will not boot
        CPU17 failed to report alive state
        psci: CPU17 killed (polled 0 ms)
        CPU17: died during early boot

        CPU18 failed to report alive state
        Kernel panic - not syncing: CPU18 detected unsupported configuration

Best regards,
Jinjie


>