From: Jinjie Ruan <ruanjinjie@huawei.com>
To: Michael Kelley <mhklinux@outlook.com>,
"catalin.marinas@arm.com" <catalin.marinas@arm.com>,
"will@kernel.org" <will@kernel.org>,
"tsbogend@alpha.franken.de" <tsbogend@alpha.franken.de>,
"pjw@kernel.org" <pjw@kernel.org>,
"palmer@dabbelt.com" <palmer@dabbelt.com>,
"aou@eecs.berkeley.edu" <aou@eecs.berkeley.edu>,
"alex@ghiti.fr" <alex@ghiti.fr>,
"tglx@kernel.org" <tglx@kernel.org>,
"mingo@redhat.com" <mingo@redhat.com>,
"bp@alien8.de" <bp@alien8.de>,
"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
"hpa@zytor.com" <hpa@zytor.com>,
"peterz@infradead.org" <peterz@infradead.org>,
"kees@kernel.org" <kees@kernel.org>,
"nathan@kernel.org" <nathan@kernel.org>,
"linusw@kernel.org" <linusw@kernel.org>,
"ojeda@kernel.org" <ojeda@kernel.org>,
"david.kaplan@amd.com" <david.kaplan@amd.com>,
"lukas.bulwahn@redhat.com" <lukas.bulwahn@redhat.com>,
"ryan.roberts@arm.com" <ryan.roberts@arm.com>,
"maz@kernel.org" <maz@kernel.org>,
"timothy.hayes@arm.com" <timothy.hayes@arm.com>,
"lpieralisi@kernel.org" <lpieralisi@kernel.org>,
"thuth@redhat.com" <thuth@redhat.com>,
"oupton@kernel.org" <oupton@kernel.org>,
"yeoreum.yun@arm.com" <yeoreum.yun@arm.com>,
"miko.lenczewski@arm.com" <miko.lenczewski@arm.com>,
"broonie@kernel.org" <broonie@kernel.org>,
"kevin.brodsky@arm.com" <kevin.brodsky@arm.com>,
"james.clark@linaro.org" <james.clark@linaro.org>,
"tabba@google.com" <tabba@google.com>,
"mrigendra.chaubey@gmail.com" <mrigendra.chaubey@gmail.com>,
"arnd@arndb.de" <arnd@arndb.de>,
"anshuman.khandual@arm.com" <anshuman.khandual@arm.com>,
"x86@kernel.org" <x86@kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-arm-kernel@lists.infradead.org"
<linux-arm-kernel@lists.infradead.org>,
"linux-mips@vger.kernel.org" <linux-mips@vger.kernel.org>,
"linux-riscv@lists.infradead.org"
<linux-riscv@lists.infradead.org>
Subject: Re: [PATCH RFC 3/3] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs
Date: Mon, 15 Jun 2026 17:57:22 +0800 [thread overview]
Message-ID: <4588efb4-e757-4ca7-9197-025b67ca9ef6@huawei.com> (raw)
In-Reply-To: <SN6PR02MB41575306521E6223561F476FD4182@SN6PR02MB4157.namprd02.prod.outlook.com>
On 6/12/2026 11:45 PM, Michael Kelley wrote:
> From: Jinjie Ruan <ruanjinjie@huawei.com> Sent: Thursday, June 11, 2026 6:38 AM
>>
>> Support for parallel secondary CPU bringup is already utilized by x86,
>> MIPS, and RISC-V. This patch brings this capability to the arm64
>> architecture.
>>
>> Rework the global `secondary_data` accessed during early boot into
>> a per-CPU array. This array maps logical CPU IDs to MPIDR_EL1 values,
>> enabling the early boot code in head.S to resolve each secondary CPU's
>> logical ID concurrently.
>>
>> To fully enable HOTPLUG_PARALLEL, this patch implements:
>> 1) An arm64-specific arch_cpuhp_kick_ap_alive() handler.
>> 2) Callbacks to cpuhp_ap_sync_alive() inside secondary_start_kernel().
>>
>> Successfully tested on QEMU ARM64 virt machine (KVM on, 128 vCPUs).
>>
>> | test kernel | secondary CPUs boot time |
>> | --------------------- | -------------------- |
>> | Without this patch | 155.672 |
>> | cpuhp.parallel=0 | 62.897 |
>> | cpuhp.parallel=1 | 166.703 |
>
> The last two rows seem mixed up. I would expect parallel=0 to
> result in a longer boot time.
Without this patch:
KVM event statistics (6 entries)
Event name Samples Sample% Time (ns) Time%
Mean Time (ns)
DABT_LOW 323112 75.00% 1669148000 17.00%
5165
WFx 85817 19.00% 723215800 7.00%
8427
SYS64 14914 3.00% 419934530 4.00%
28157
IRQ 5643 1.00% 6732439250 70.00%
1193060
HVC64 282 0.00% 35543970 0.00%
126042
IABT_LOW 1 0.00% 6130 0.00%
6130
cpuhp.parallel=0:
Event name Samples Sample% Time (ns) Time%
Mean Time (ns)
DABT_LOW 308175 80.00% 643628050 6.00%
2088
WFx 55208 14.00% 261925270 2.00%
4744
SYS64 14975 3.00% 155727880 1.00%
10399
IRQ 4755 1.00% 8496162210 88.00%
1786784
HVC64 280 0.00% 19429900 0.00%
69392
IABT_LOW 1 0.00% 5850 0.00%
5850
cpuhp.parallel=1:
Event name Samples Sample% Time (ns) Time%
Mean Time (ns)
DABT_LOW 307923 77.00% 692965050 2.00%
2250
WFx 59549 15.00% 287888960 0.00%
4834
SYS64 15127 3.00% 334366230 1.00%
22103
IRQ 12861 3.00% 29784004970 95.00%
2315838
HVC64 280 0.00% 21869940 0.00%
78106
IABT_LOW 1 0.00% 9320 0.00%
9320
- Default (no patch): Slowest HVC64 handling (126 μs), highest WFx count
(85k), and most total VM‑exits.
- cpuhp.parallel=1: HVC64 latency improved to 78 μs (close to
cpuhp.parallel=0), but IRQ exits increased dramatically (12.9k, 2.7×
that of `cpuhp.parallel=0`), accounting for 95% of event time and
becoming the new bottleneck.
- cpuhp.parallel=0: Fastest HVC64 (69 μs), lowest IRQ exits (4.8k), and
lowest total samples, delivering the best overall boot performance.
Therefor, `cpuhp.parallel=1` reduces HVC cost but suffers from a massive
increase in IRQ exits, while `cpuhp.parallel=0` avoids this interrupt
storm and therefore performs best in a KVM guest.
>
> Michael
>
>>
>> Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
>> ---
>> arch/arm64/Kconfig | 1 +
>> arch/arm64/include/asm/smp.h | 8 ++++++++
>> arch/arm64/kernel/head.S | 23 +++++++++++++++++++++++
>> arch/arm64/kernel/smp.c | 27 +++++++++++++++++++++++++++
>> 4 files changed, 59 insertions(+)
>>
>
>
next prev parent reply other threads:[~2026-06-15 9:57 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-06-11 13:38 [PATCH RFC 0/3] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs Jinjie Ruan
2026-06-11 13:38 ` [PATCH RFC 1/3] cpu/hotplug: Introduce CONFIG_PARALLEL_SMT_PRIMARY_FIRST Jinjie Ruan
2026-06-18 15:17 ` Thomas Gleixner
2026-06-11 13:38 ` [PATCH RFC 2/3] arm64: smp: Pass CPU ID to update_cpu_boot_status() Jinjie Ruan
2026-06-11 13:38 ` [PATCH RFC 3/3] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs Jinjie Ruan
2026-06-12 15:45 ` Michael Kelley
2026-06-15 8:51 ` Jinjie Ruan
2026-06-18 12:21 ` Will Deacon
2026-06-15 9:57 ` Jinjie Ruan [this message]
2026-06-18 15:53 ` Thomas Gleixner
2026-06-18 15:49 ` Thomas Gleixner
2026-06-12 15:51 ` [PATCH RFC 0/3] " Michael Kelley
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4588efb4-e757-4ca7-9197-025b67ca9ef6@huawei.com \
--to=ruanjinjie@huawei.com \
--cc=alex@ghiti.fr \
--cc=anshuman.khandual@arm.com \
--cc=aou@eecs.berkeley.edu \
--cc=arnd@arndb.de \
--cc=bp@alien8.de \
--cc=broonie@kernel.org \
--cc=catalin.marinas@arm.com \
--cc=dave.hansen@linux.intel.com \
--cc=david.kaplan@amd.com \
--cc=hpa@zytor.com \
--cc=james.clark@linaro.org \
--cc=kees@kernel.org \
--cc=kevin.brodsky@arm.com \
--cc=linusw@kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mips@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=lpieralisi@kernel.org \
--cc=lukas.bulwahn@redhat.com \
--cc=maz@kernel.org \
--cc=mhklinux@outlook.com \
--cc=miko.lenczewski@arm.com \
--cc=mingo@redhat.com \
--cc=mrigendra.chaubey@gmail.com \
--cc=nathan@kernel.org \
--cc=ojeda@kernel.org \
--cc=oupton@kernel.org \
--cc=palmer@dabbelt.com \
--cc=peterz@infradead.org \
--cc=pjw@kernel.org \
--cc=ryan.roberts@arm.com \
--cc=tabba@google.com \
--cc=tglx@kernel.org \
--cc=thuth@redhat.com \
--cc=timothy.hayes@arm.com \
--cc=tsbogend@alpha.franken.de \
--cc=will@kernel.org \
--cc=x86@kernel.org \
--cc=yeoreum.yun@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox