From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EC490CD98F2 for ; Thu, 18 Jun 2026 12:21:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To: Content-Transfer-Encoding:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Jxvhe6p1siX027yhrT2z3XtNk0V+WNmEv1s3Z1RR4ZI=; b=QSbw+7Ed9lw6P2T4Bp2f1M7BSI ZPC+pixtygy9OlFj15VGvlJAASVtGFW4/dlbAayWDAURqJgw6QWC0UmEh34AemAcpMb7YlCmp4yQV 01HwmscEdg/5NW9sC0m9MQOIDG+zrLToU2fjKCYgeigfPptbkKKNKqbTcDYl42Mevkw9gtVmZMc+K D24YJz0jhLis6H2MwmYsT9FuRlSU5NTo6Devmt5NG+nLQgBoXga28VXWvzM0PFPfL4sDKJKK7Y2T3 BaIkMZ3vL9IhO58K8xOoVZoFZIME9C5y+/DNX6JAR0GCM4ORGbv0qnb+SyiQG7+FdCRDIBj0xhHjQ 6UNLzQ+g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1waBkH-00000001Fyo-0wEm; Thu, 18 Jun 2026 12:21:21 +0000 Received: from tor.source.kernel.org ([172.105.4.254]) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1waBkG-00000001FyI-1oxb; Thu, 18 Jun 2026 12:21:20 +0000 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) by tor.source.kernel.org (Postfix) with ESMTP id 20C8960008; Thu, 18 Jun 2026 12:21:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 014171F000E9; Thu, 18 Jun 2026 12:21:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1781785278; bh=Jxvhe6p1siX027yhrT2z3XtNk0V+WNmEv1s3Z1RR4ZI=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=Abiv2iO4RiW//8pR9dLhCcfHcyT0i58RzDg5KzLOl4m1sOVrs9DawRFk1ryHLcLTZ GPY4ji0YfFluVDNoGFEQXe/s9wWAINc1y53L2TgIChzRQy+tck3KRiTB4YXjBtxuF/ dKOzf0YDV7JDGJYBomYM/m/3eJJw0ECQPGWyOA2dny3d1EqMKYJ5bv0qL/FdTFInjo UwObl+tH/TA8ZI2XIv81QMpK+UGTHEWY2yqpQg+12mo83OKk5o3NZmROcMePyhmjO+ piAkoCTIX25wwfovDDqzcGs4sdtJWDhEf0M8suoHHDosU2AvUXb3uErsuxUgwmrIPT xJGfjxrHWZ73A== Date: Thu, 18 Jun 2026 13:21:08 +0100 From: Will Deacon To: Jinjie Ruan Cc: Michael Kelley , "catalin.marinas@arm.com" , "tsbogend@alpha.franken.de" , "pjw@kernel.org" , "palmer@dabbelt.com" , "aou@eecs.berkeley.edu" , "alex@ghiti.fr" , "tglx@kernel.org" , "mingo@redhat.com" , "bp@alien8.de" , "dave.hansen@linux.intel.com" , "hpa@zytor.com" , "peterz@infradead.org" , "kees@kernel.org" , "nathan@kernel.org" , "linusw@kernel.org" , "ojeda@kernel.org" , "david.kaplan@amd.com" , "lukas.bulwahn@redhat.com" , "ryan.roberts@arm.com" , "maz@kernel.org" , "timothy.hayes@arm.com" , "lpieralisi@kernel.org" , "thuth@redhat.com" , "oupton@kernel.org" , "yeoreum.yun@arm.com" , "miko.lenczewski@arm.com" , "broonie@kernel.org" , "kevin.brodsky@arm.com" , "james.clark@linaro.org" , "tabba@google.com" , "mrigendra.chaubey@gmail.com" , "arnd@arndb.de" , "anshuman.khandual@arm.com" , "x86@kernel.org" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH RFC 3/3] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs Message-ID: References: <20260611133809.3854977-1-ruanjinjie@huawei.com> <20260611133809.3854977-4-ruanjinjie@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Jinjie, On Mon, Jun 15, 2026 at 04:51:48PM +0800, Jinjie Ruan wrote: > On 6/12/2026 11:45 PM, Michael Kelley wrote: > > From: Jinjie Ruan Sent: Thursday, June 11, 2026 6:38 AM > >> > >> Support for parallel secondary CPU bringup is already utilized by x86, > >> MIPS, and RISC-V. This patch brings this capability to the arm64 > >> architecture. > >> > >> Rework the global `secondary_data` accessed during early boot into > >> a per-CPU array. This array maps logical CPU IDs to MPIDR_EL1 values, > >> enabling the early boot code in head.S to resolve each secondary CPU's > >> logical ID concurrently. > >> > >> To fully enable HOTPLUG_PARALLEL, this patch implements: > >> 1) An arm64-specific arch_cpuhp_kick_ap_alive() handler. > >> 2) Callbacks to cpuhp_ap_sync_alive() inside secondary_start_kernel(). > >> > >> Successfully tested on QEMU ARM64 virt machine (KVM on, 128 vCPUs). > >> > >> | test kernel | secondary CPUs boot time | > >> | --------------------- | -------------------- | > >> | Without this patch | 155.672 | > >> | cpuhp.parallel=0 | 62.897 | > >> | cpuhp.parallel=1 | 166.703 | > > > > The last two rows seem mixed up. I would expect parallel=0 to > > result in a longer boot time. > > Hi, Michael, > > The results are correct and not mixed up. > > Compared to the original non‑HOTPLUG_PARALLEL approach, the advantage of > cpuhp.parallel=0 lies in its use of cpu_relax(`yield` on arm64) instead > of the wait_for_completion_timeout() mechanism (which may cause sleep > and context switching). This significantly reduces the overhead of VM > exits and context switches in a KVM guest, thereby cutting the secondary > CPU boot time by more than half. I don't think that's a particularly compelling reason to enable this for arm64, in all honesty. The yield instruction typically doesn't do anything on actual arm64 silicon, so this probably means that you're introducing busy-loops which tend to be bad for power and scalability. I implemented this a while ago [1] but didn't manage to see much in terms of performance improvement and so I didn't bother to send the patches out after talking about it at KVM forum [2]. However, as mentioned at the end of that talk, it _is_ still useful for confidential VMs using PSCI so let me dust off my old series and send it out to see what you think. It relies on PSCI v0.2, which means we don't need the NR_CPUS size array for secondary_data and I also have some support for error handling (it doesn't look like you handle __early_cpu_boot_status properly). It looks like I could include your first patch, though! Will [1] https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=cpu-hotplug [2] https://www.youtube.com/watch?v=Q6kOshnnQuE From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CF617CD98ED for ; Thu, 18 Jun 2026 12:21:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=E2IpG0oSgsbzrzjmUg7SuEE0nHkMlqBHj+6PH4jqKwc=; b=n4SdYHy9gNcaLo e1U4MF2FDCDySRqRi6JiTgKiXG0N9P3hC8scJiJKmuEYKVvGcIZozQjkii+6ZUNJW/4v5+khvfr6q ZwkUUwK0u2VlrJXEqoO8Od6w89nPxkaoxw8Ny9CB1mAL96xpm+JcZwOBfyMZo8mByhOb0nIAPa80n PpVQxIcWzEdEgr8GG97UYxagfpmKNpyqb+36E8JHa4AanOBIFsjGguMWqy1+VyDy+pv5OOOyJWoT4 I3j5JHturmv46J7MD4pzv3063vmSlXzn18BkrMQJBFjJtjbd3cJ//vbaeZeK+6PyoX3l3DoIQ9ovZ hYM0NFPm/Zmo5s8MM4Vw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1waBkH-00000001Fz5-1NRr; Thu, 18 Jun 2026 12:21:21 +0000 Received: from tor.source.kernel.org ([172.105.4.254]) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1waBkG-00000001FyI-1oxb; Thu, 18 Jun 2026 12:21:20 +0000 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) by tor.source.kernel.org (Postfix) with ESMTP id 20C8960008; Thu, 18 Jun 2026 12:21:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 014171F000E9; Thu, 18 Jun 2026 12:21:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1781785278; bh=Jxvhe6p1siX027yhrT2z3XtNk0V+WNmEv1s3Z1RR4ZI=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=Abiv2iO4RiW//8pR9dLhCcfHcyT0i58RzDg5KzLOl4m1sOVrs9DawRFk1ryHLcLTZ GPY4ji0YfFluVDNoGFEQXe/s9wWAINc1y53L2TgIChzRQy+tck3KRiTB4YXjBtxuF/ dKOzf0YDV7JDGJYBomYM/m/3eJJw0ECQPGWyOA2dny3d1EqMKYJ5bv0qL/FdTFInjo UwObl+tH/TA8ZI2XIv81QMpK+UGTHEWY2yqpQg+12mo83OKk5o3NZmROcMePyhmjO+ piAkoCTIX25wwfovDDqzcGs4sdtJWDhEf0M8suoHHDosU2AvUXb3uErsuxUgwmrIPT xJGfjxrHWZ73A== Date: Thu, 18 Jun 2026 13:21:08 +0100 From: Will Deacon To: Jinjie Ruan Cc: Michael Kelley , "catalin.marinas@arm.com" , "tsbogend@alpha.franken.de" , "pjw@kernel.org" , "palmer@dabbelt.com" , "aou@eecs.berkeley.edu" , "alex@ghiti.fr" , "tglx@kernel.org" , "mingo@redhat.com" , "bp@alien8.de" , "dave.hansen@linux.intel.com" , "hpa@zytor.com" , "peterz@infradead.org" , "kees@kernel.org" , "nathan@kernel.org" , "linusw@kernel.org" , "ojeda@kernel.org" , "david.kaplan@amd.com" , "lukas.bulwahn@redhat.com" , "ryan.roberts@arm.com" , "maz@kernel.org" , "timothy.hayes@arm.com" , "lpieralisi@kernel.org" , "thuth@redhat.com" , "oupton@kernel.org" , "yeoreum.yun@arm.com" , "miko.lenczewski@arm.com" , "broonie@kernel.org" , "kevin.brodsky@arm.com" , "james.clark@linaro.org" , "tabba@google.com" , "mrigendra.chaubey@gmail.com" , "arnd@arndb.de" , "anshuman.khandual@arm.com" , "x86@kernel.org" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH RFC 3/3] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs Message-ID: References: <20260611133809.3854977-1-ruanjinjie@huawei.com> <20260611133809.3854977-4-ruanjinjie@huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org SGkgSmluamllLAoKT24gTW9uLCBKdW4gMTUsIDIwMjYgYXQgMDQ6NTE6NDhQTSArMDgwMCwgSmlu amllIFJ1YW4gd3JvdGU6Cj4gT24gNi8xMi8yMDI2IDExOjQ1IFBNLCBNaWNoYWVsIEtlbGxleSB3 cm90ZToKPiA+IEZyb206IEppbmppZSBSdWFuIDxydWFuamluamllQGh1YXdlaS5jb20+IFNlbnQ6 IFRodXJzZGF5LCBKdW5lIDExLCAyMDI2IDY6MzggQU0KPiA+Pgo+ID4+IFN1cHBvcnQgZm9yIHBh cmFsbGVsIHNlY29uZGFyeSBDUFUgYnJpbmd1cCBpcyBhbHJlYWR5IHV0aWxpemVkIGJ5IHg4NiwK PiA+PiBNSVBTLCBhbmQgUklTQy1WLiBUaGlzIHBhdGNoIGJyaW5ncyB0aGlzIGNhcGFiaWxpdHkg dG8gdGhlIGFybTY0Cj4gPj4gYXJjaGl0ZWN0dXJlLgo+ID4+Cj4gPj4gUmV3b3JrIHRoZSBnbG9i YWwgYHNlY29uZGFyeV9kYXRhYCBhY2Nlc3NlZCBkdXJpbmcgZWFybHkgYm9vdCBpbnRvCj4gPj4g YSBwZXItQ1BVIGFycmF5LiBUaGlzIGFycmF5IG1hcHMgbG9naWNhbCBDUFUgSURzIHRvIE1QSURS X0VMMSB2YWx1ZXMsCj4gPj4gZW5hYmxpbmcgdGhlIGVhcmx5IGJvb3QgY29kZSBpbiBoZWFkLlMg dG8gcmVzb2x2ZSBlYWNoIHNlY29uZGFyeSBDUFUncwo+ID4+IGxvZ2ljYWwgSUQgY29uY3VycmVu dGx5Lgo+ID4+Cj4gPj4gVG8gZnVsbHkgZW5hYmxlIEhPVFBMVUdfUEFSQUxMRUwsIHRoaXMgcGF0 Y2ggaW1wbGVtZW50czoKPiA+PiAxKSBBbiBhcm02NC1zcGVjaWZpYyBhcmNoX2NwdWhwX2tpY2tf YXBfYWxpdmUoKSBoYW5kbGVyLgo+ID4+IDIpIENhbGxiYWNrcyB0byBjcHVocF9hcF9zeW5jX2Fs aXZlKCkgaW5zaWRlIHNlY29uZGFyeV9zdGFydF9rZXJuZWwoKS4KPiA+Pgo+ID4+IFN1Y2Nlc3Nm dWxseSB0ZXN0ZWQgb24gUUVNVSBBUk02NCB2aXJ0IG1hY2hpbmUgKEtWTSBvbiwgMTI4IHZDUFVz KS4KPiA+Pgo+ID4+IHwgICAgIHRlc3Qga2VybmVsCSAgIHwgc2Vjb25kYXJ5IENQVXMgYm9vdCB0 aW1lIHwKPiA+PiB8ICAtLS0tLS0tLS0tLS0tLS0tLS0tLS0gICB8CS0tLS0tLS0tLS0tLS0tLS0t LS0tICB8Cj4gPj4gfCAgIFdpdGhvdXQgdGhpcyBwYXRjaCAgICAgfAkJMTU1LjY3MgkgICAgICB8 Cj4gPj4gfCAgIGNwdWhwLnBhcmFsbGVsPTAJICAgfAkJNjIuODk3CSAgICAgIHwKPiA+PiB8ICAg Y3B1aHAucGFyYWxsZWw9MQkgICB8CQkxNjYuNzAzCSAgICAgIHwKPiA+IAo+ID4gVGhlIGxhc3Qg dHdvIHJvd3Mgc2VlbSBtaXhlZCB1cC4gSSB3b3VsZCBleHBlY3QgcGFyYWxsZWw9MCB0bwo+ID4g cmVzdWx0IGluIGEgbG9uZ2VyIGJvb3QgdGltZS4KPiAKPiBIaSwgTWljaGFlbCwKPiAKPiBUaGUg cmVzdWx0cyBhcmUgY29ycmVjdCBhbmQgbm90IG1peGVkIHVwLgo+IAo+IENvbXBhcmVkIHRvIHRo ZSBvcmlnaW5hbCBub27igJFIT1RQTFVHX1BBUkFMTEVMIGFwcHJvYWNoLCB0aGUgYWR2YW50YWdl IG9mCj4gY3B1aHAucGFyYWxsZWw9MCBsaWVzIGluIGl0cyB1c2Ugb2YgY3B1X3JlbGF4KGB5aWVs ZGAgb24gYXJtNjQpIGluc3RlYWQKPiBvZiB0aGUgd2FpdF9mb3JfY29tcGxldGlvbl90aW1lb3V0 KCkgbWVjaGFuaXNtICh3aGljaCBtYXkgY2F1c2Ugc2xlZXAKPiBhbmQgY29udGV4dCBzd2l0Y2hp bmcpLiBUaGlzIHNpZ25pZmljYW50bHkgcmVkdWNlcyB0aGUgb3ZlcmhlYWQgb2YgVk0KPiBleGl0 cyBhbmQgY29udGV4dCBzd2l0Y2hlcyBpbiBhIEtWTSBndWVzdCwgdGhlcmVieSBjdXR0aW5nIHRo ZSBzZWNvbmRhcnkKPiBDUFUgYm9vdCB0aW1lIGJ5IG1vcmUgdGhhbiBoYWxmLgoKSSBkb24ndCB0 aGluayB0aGF0J3MgYSBwYXJ0aWN1bGFybHkgY29tcGVsbGluZyByZWFzb24gdG8gZW5hYmxlIHRo aXMgZm9yCmFybTY0LCBpbiBhbGwgaG9uZXN0eS4gVGhlIHlpZWxkIGluc3RydWN0aW9uIHR5cGlj YWxseSBkb2Vzbid0IGRvCmFueXRoaW5nIG9uIGFjdHVhbCBhcm02NCBzaWxpY29uLCBzbyB0aGlz IHByb2JhYmx5IG1lYW5zIHRoYXQgeW91J3JlCmludHJvZHVjaW5nIGJ1c3ktbG9vcHMgd2hpY2gg dGVuZCB0byBiZSBiYWQgZm9yIHBvd2VyIGFuZCBzY2FsYWJpbGl0eS4KCkkgaW1wbGVtZW50ZWQg dGhpcyBhIHdoaWxlIGFnbyBbMV0gYnV0IGRpZG4ndCBtYW5hZ2UgdG8gc2VlIG11Y2ggaW4gdGVy bXMKb2YgcGVyZm9ybWFuY2UgaW1wcm92ZW1lbnQgYW5kIHNvIEkgZGlkbid0IGJvdGhlciB0byBz ZW5kIHRoZSBwYXRjaGVzIG91dAphZnRlciB0YWxraW5nIGFib3V0IGl0IGF0IEtWTSBmb3J1bSBb Ml0uIEhvd2V2ZXIsIGFzIG1lbnRpb25lZCBhdCB0aGUgZW5kCm9mIHRoYXQgdGFsaywgaXQgX2lz XyBzdGlsbCB1c2VmdWwgZm9yIGNvbmZpZGVudGlhbCBWTXMgdXNpbmcgUFNDSSBzbwpsZXQgbWUg ZHVzdCBvZmYgbXkgb2xkIHNlcmllcyBhbmQgc2VuZCBpdCBvdXQgdG8gc2VlIHdoYXQgeW91IHRo aW5rLgoKSXQgcmVsaWVzIG9uIFBTQ0kgdjAuMiwgd2hpY2ggbWVhbnMgd2UgZG9uJ3QgbmVlZCB0 aGUgTlJfQ1BVUyBzaXplIGFycmF5CmZvciBzZWNvbmRhcnlfZGF0YSBhbmQgSSBhbHNvIGhhdmUg c29tZSBzdXBwb3J0IGZvciBlcnJvciBoYW5kbGluZyAoaXQKZG9lc24ndCBsb29rIGxpa2UgeW91 IGhhbmRsZSBfX2Vhcmx5X2NwdV9ib290X3N0YXR1cyBwcm9wZXJseSkuCgpJdCBsb29rcyBsaWtl IEkgY291bGQgaW5jbHVkZSB5b3VyIGZpcnN0IHBhdGNoLCB0aG91Z2ghCgpXaWxsCgpbMV0gaHR0 cHM6Ly9naXQua2VybmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5lbC9naXQvd2lsbC9saW51eC5n aXQvbG9nLz9oPWNwdS1ob3RwbHVnClsyXSBodHRwczovL3d3dy55b3V0dWJlLmNvbS93YXRjaD92 PVE2a09zaG5uUXVFCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXwpsaW51eC1yaXNjdiBtYWlsaW5nIGxpc3QKbGludXgtcmlzY3ZAbGlzdHMuaW5mcmFkZWFk Lm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LXJp c2N2Cg==